Hello,
I just tried to setup biobakery workflows yesterday for the first time, and when I attempt to run the command below (modified pathways for privacy), I get the error " AttributeError: Can’t pickle local object ‘ArgumentParser.init..identity’ " listed below. Any help on getting past this would be greatly appreciated. Thanks!
Command:
biobakery_workflows wmgx --input /Volumes/mydrive/data/wc/test --output /Volumes/mydrive/data/wc/test/biobakery_output --local-jobs 2 --pair-identifier _R1_ --qc-options="--trf /Users/nyb/miniforge3/bin --threads 8 --processes 2 --reference-db ./databases/mouse_refdb --reference-db ./databases/human_refdb"
Error:
(Jan 22 11:33:24) [ 0/52 - 0.00%] **Ready ** Task 0: kneaddata____20171FL-03-01-788_S299_L007_R1_001
(Jan 22 11:33:24) [ 0/52 - 0.00%] **Ready ** Task 3: kneaddata____20171FL-03-01-788_S299_L007_R2_001
Traceback (most recent call last):
File "/Users/nyb/miniforge3/envs/biobakery/bin/wmgx.py", line 181, in <module>
workflow.go()
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/site-packages/anadama2/workflow.py", line 803, in go
_runner.run_tasks(task_idxs)
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/site-packages/anadama2/runners.py", line 273, in run_tasks
self._fill_work_q()
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/site-packages/anadama2/runners.py", line 330, in _fill_work_q
w.start()
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'
Worth noting, if I remove the --local-jobs 2 flag, I get a different error, which seems like a step forward.
Err: ERROR: Unable to find bowtie2 index files in directory: /Users/nyb/biobakery_workflows_databases/kneaddata_db_human_genome
(full error message shown at bottom)
The issue I then still have with this is that the kneaddata databases didn’t populate in the biobakery workflows databases folder to begin with after I ran biobakery_workflows_databases --install wmgx. I had to manually install them, and I can create a directory by that name and make sure the correct files are in it, and I can set the environmental database reference (KNEADDATA_DB_HUMAN_GENOME – except I don’t know where I’m supposed to set this?), but I want it to also use the mouse database as a reference genome too for removing contaminants, so how can I ensure it’s doing that if at least from what I can see at present, it’s ignoring my qc-options which tell it to go find those files in a different place? Can I just add the mouse genome files to the Kneaddata_DB_Human_Genome directory so it can find them?
Task 6 failed
Name: metaphlan____20171FL-03-01-788_S299_L007_R1_001.gz
Original error:
Task failed because parent task `0' failed
Task 5 failed
Name: kneaddata_read_count_table
Original error:
Task failed because parent task `0' failed
Task 8 failed
Name: metaphlan____20091FL-05-01-75_S143_L007_R2_001.gz
Original error:
Task failed because parent task `3' failed
Task 11 failed
Name: humann____20171FL-03-01-788_S299_L007_R1_001.gz
Original error:
Task failed because parent task `0' failed
Task 33 failed
Name: strainphlan_sample2markers____20171FL-03-01-788_S299_L007_R1_001
Original error:
Task failed because parent task `6' failed
Task 9 failed
Name: metaphlan_join_taxonomic_profiles
Original error:
Task failed because parent task `8' failed
Task 13 failed
Name: humann____20091FL-05-01-75_S143_L007_R2_001.gz
Original error:
Task failed because parent task `8' failed
Task 34 failed
Name: strainphlan_sample2markers____20091FL-05-01-75_S143_L007_R2_001
Original error:
Task failed because parent task `8' failed
Task 15 failed
Name: humann_regroup_UniRef2EC____20171FL-03-01-788_S299_L007_R1_001.gz
Original error:
Task failed because parent task `11' failed
Task 20 failed
Name: humann_renorm_genes_relab____20171FL-03-01-788_S299_L007_R1_001.gz
Original error:
Task failed because parent task `11' failed
Task 24 failed
Name: humann_renorm_pathways_relab____20171FL-03-01-788_S299_L007_R1_001.gz
Original error:
Task failed because parent task `11' failed
Task 10 failed
Name: metaphlan_count_species
Original error:
Task failed because parent task `9' failed
Task 14 failed
Name: humann_count_alignments_species
Original error:
Task failed because parent task `11' failed
Task 16 failed
Name: humann_regroup_UniRef2EC____20091FL-05-01-75_S143_L007_R2_001.gz
Original error:
Task failed because parent task `13' failed
Task 17 failed
Name: humann_join_tables_genefamilies
Original error:
Task failed because parent task `11' failed
Task 19 failed
Name: humann_join_tables_pathabundance
Original error:
Task failed because parent task `11' failed
Task 21 failed
Name: humann_renorm_genes_relab____20091FL-05-01-75_S143_L007_R2_001.gz
Original error:
Task failed because parent task `13' failed
Task 25 failed
Name: humann_renorm_pathways_relab____20091FL-05-01-75_S143_L007_R2_001.gz
Original error:
Task failed because parent task `13' failed
Task 35 failed
Name: strainphlan_print_clades
Original error:
Task failed because parent task `33' failed
Task 22 failed
Name: humann_renorm_ecs_relab____20171FL-03-01-788_S299_L007_R1_001.gz
Original error:
Task failed because parent task `15' failed
Task 18 failed
Name: humann_join_tables_ecs
Original error:
Task failed because parent task `16' failed
Task 23 failed
Name: humann_renorm_ecs_relab____20091FL-05-01-75_S143_L007_R2_001.gz
Original error:
Task failed because parent task `16' failed
Task 26 failed
Name: humann_join_tables_genes_relab
Original error:
Task failed because parent task `20' failed
Task 28 failed
Name: humann_join_tables_pathways_relab
Original error:
Task failed because parent task `24' failed
Task 36 failed
Name: Task36
Original error:
Task failed because parent task `9' failed
Task 27 failed
Name: humann_join_tables_ecs_relab
Original error:
Task failed because parent task `22' failed
Task 29 failed
Name: humann_count_features_genes
Original error:
Task failed because parent task `26' failed
Task 31 failed
Name: humann_count_features_pathways
Original error:
Task failed because parent task `28' failed
Task 37 failed
Name: strainphlan_clade_0
Original error:
Task failed because parent task `33' failed
Task 38 failed
Name: strainphlan_clade_1
Original error:
Task failed because parent task `33' failed
Task 39 failed
Name: strainphlan_clade_2
Original error:
Task failed because parent task `33' failed
Task 40 failed
Name: strainphlan_clade_3
Original error:
Task failed because parent task `33' failed
Task 41 failed
Name: strainphlan_clade_4
Original error:
Task failed because parent task `33' failed
Task 42 failed
Name: strainphlan_clade_5
Original error:
Task failed because parent task `33' failed
Task 43 failed
Name: strainphlan_clade_6
Original error:
Task failed because parent task `33' failed
Task 44 failed
Name: strainphlan_clade_7
Original error:
Task failed because parent task `33' failed
Task 45 failed
Name: strainphlan_clade_8
Original error:
Task failed because parent task `33' failed
Task 46 failed
Name: strainphlan_clade_9
Original error:
Task failed because parent task `33' failed
Task 47 failed
Name: strainphlan_clade_10
Original error:
Task failed because parent task `33' failed
Task 48 failed
Name: strainphlan_clade_11
Original error:
Task failed because parent task `33' failed
Task 49 failed
Name: strainphlan_clade_12
Original error:
Task failed because parent task `33' failed
Task 50 failed
Name: strainphlan_clade_13
Original error:
Task failed because parent task `33' failed
Task 51 failed
Name: strainphlan_clade_14
Original error:
Task failed because parent task `33' failed
Task 52 failed
Name: strainphlan_clade_15
Original error:
Task failed because parent task `33' failed
Task 53 failed
Name: strainphlan_clade_16
Original error:
Task failed because parent task `33' failed
Task 54 failed
Name: strainphlan_clade_17
Original error:
Task failed because parent task `33' failed
Task 55 failed
Name: strainphlan_clade_18
Original error:
Task failed because parent task `33' failed
Task 56 failed
Name: strainphlan_clade_19
Original error:
Task failed because parent task `33' failed
Task 30 failed
Name: humann_count_features_ecs
Original error:
Task failed because parent task `27' failed
Task 32 failed
Name: humann_merge_feature_counts
Original error:
Task failed because parent task `29' failed
Traceback (most recent call last):
File "/Users/nyb/miniforge3/envs/biobakery/bin/wmgx.py", line 181, in <module>
workflow.go()
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/site-packages/anadama2/workflow.py", line 804, in go
self._handle_finished()
File "/Users/nyb/miniforge3/envs/biobakery/lib/python3.11/site-packages/anadama2/workflow.py", line 836, in _handle_finished
raise RunFailed()
anadama2.workflow.RunFailed