Hi all, I am a beginner in using Biobakery Workflows.
I encountered an issue where tasks fail while running the wmgx Biobakery Workflow with tutorial files in Docker.
Command:
biobakery_workflows wmgx --input Tutorials --output output/test3 --functional-profiling-option='--bypass-translated-search' --bypass-strain-profiling
Error report:
(Feb 10 04:04:02) [ 0/56 - 0.00%] **Ready ** Task 3: kneaddata____HD32R1_subsample
(Feb 10 04:04:02) [ 0/56 - 0.00%] **Started ** Task 3: kneaddata____HD32R1_subsample
(Feb 10 04:04:02) [ 1/56 - 1.79%] **Failed ** Task 3: kneaddata____HD32R1_subsample
(Feb 10 04:04:02) [ 2/56 - 3.57%] **Failed ** Task 16: metaphlan____HD32R1_subsample
(Feb 10 04:04:02) [ 3/56 - 5.36%] **Failed ** Task 25: humann____HD32R1_subsample
(Feb 10 04:04:02) [ 4/56 - 7.14%] **Failed ** Task 32: humann_regroup_UniRef2EC____HD32R1_subsample
(Feb 10 04:04:02) [ 5/56 - 8.93%] **Failed ** Task 47: humann_renorm_ecs_relab____HD32R1_subsample
(Feb 10 04:04:02) [ 6/56 - 10.71%] **Failed ** Task 41: humann_renorm_genes_relab____HD32R1_subsample
(Feb 10 04:04:02) [ 7/56 - 12.50%] **Failed ** Task 53: humann_renorm_pathways_relab____HD32R1_subsample
(Feb 10 04:04:02) [ 7/56 - 12.50%] **Ready ** Task 5: kneaddata____HD48R4_subsample
(Feb 10 04:04:02) [ 7/56 - 12.50%] **Started ** Task 5: kneaddata____HD48R4_subsample
(Feb 10 04:04:02) [ 8/56 - 14.29%] **Failed ** Task 5: kneaddata____HD48R4_subsample
(Feb 10 04:04:02) [ 9/56 - 16.07%] **Failed ** Task 17: metaphlan____HD48R4_subsample
(Feb 10 04:04:02) [10/56 - 17.86%] **Failed ** Task 26: humann____HD48R4_subsample
(Feb 10 04:04:02) [11/56 - 19.64%] **Failed ** Task 33: humann_regroup_UniRef2EC____HD48R4_subsample
(Feb 10 04:04:02) [12/56 - 21.43%] **Failed ** Task 48: humann_renorm_ecs_relab____HD48R4_subsample
(Feb 10 04:04:02) [13/56 - 23.21%] **Failed ** Task 42: humann_renorm_genes_relab____HD48R4_subsample
(Feb 10 04:04:02) [14/56 - 25.00%] **Failed ** Task 54: humann_renorm_pathways_relab____HD48R4_subsample
(Feb 10 04:04:02) [14/56 - 25.00%] **Ready ** Task 7: kneaddata____LV16R4_subsample
(Feb 10 04:04:02) [14/56 - 25.00%] **Started ** Task 7: kneaddata____LV16R4_subsample
(Feb 10 04:04:03) [15/56 - 26.79%] **Failed ** Task 7: kneaddata____LV16R4_subsample
(Feb 10 04:04:03) [16/56 - 28.57%] **Failed ** Task 18: metaphlan____LV16R4_subsample
(Feb 10 04:04:03) [17/56 - 30.36%] **Failed ** Task 27: humann____LV16R4_subsample
(Feb 10 04:04:03) [18/56 - 32.14%] **Failed ** Task 34: humann_regroup_UniRef2EC____LV16R4_subsample
(Feb 10 04:04:03) [19/56 - 33.93%] **Failed ** Task 49: humann_renorm_ecs_relab____LV16R4_subsample
(Feb 10 04:04:03) [20/56 - 35.71%] **Failed ** Task 43: humann_renorm_genes_relab____LV16R4_subsample
(Feb 10 04:04:03) [21/56 - 37.50%] **Failed ** Task 55: humann_renorm_pathways_relab____LV16R4_subsample
(Feb 10 04:04:03) [21/56 - 37.50%] **Ready ** Task 9: kneaddata____LD96R2_subsample
(Feb 10 04:04:03) [21/56 - 37.50%] **Started ** Task 9: kneaddata____LD96R2_subsample
(Feb 10 04:04:03) [22/56 - 39.29%] **Failed ** Task 9: kneaddata____LD96R2_subsample
(Feb 10 04:04:03) [23/56 - 41.07%] **Failed ** Task 19: metaphlan____LD96R2_subsample
(Feb 10 04:04:03) [24/56 - 42.86%] **Failed ** Task 28: humann____LD96R2_subsample
(Feb 10 04:04:03) [25/56 - 44.64%] **Failed ** Task 35: humann_regroup_UniRef2EC____LD96R2_subsample
(Feb 10 04:04:03) [26/56 - 46.43%] **Failed ** Task 50: humann_renorm_ecs_relab____LD96R2_subsample
(Feb 10 04:04:03) [27/56 - 48.21%] **Failed ** Task 44: humann_renorm_genes_relab____LD96R2_subsample
(Feb 10 04:04:03) [28/56 - 50.00%] **Failed ** Task 56: humann_renorm_pathways_relab____LD96R2_subsample
(Feb 10 04:04:03) [28/56 - 50.00%] **Ready ** Task 11: kneaddata____LV20R4_subsample
(Feb 10 04:04:03) [28/56 - 50.00%] **Started ** Task 11: kneaddata____LV20R4_subsample
(Feb 10 04:04:03) [29/56 - 51.79%] **Failed ** Task 11: kneaddata____LV20R4_subsample
(Feb 10 04:04:03) [30/56 - 53.57%] **Failed ** Task 20: metaphlan____LV20R4_subsample
(Feb 10 04:04:03) [31/56 - 55.36%] **Failed ** Task 29: humann____LV20R4_subsample
(Feb 10 04:04:03) [32/56 - 57.14%] **Failed ** Task 36: humann_regroup_UniRef2EC____LV20R4_subsample
(Feb 10 04:04:03) [33/56 - 58.93%] **Failed ** Task 51: humann_renorm_ecs_relab____LV20R4_subsample
(Feb 10 04:04:03) [34/56 - 60.71%] **Failed ** Task 45: humann_renorm_genes_relab____LV20R4_subsample
(Feb 10 04:04:03) [35/56 - 62.50%] **Failed ** Task 57: humann_renorm_pathways_relab____LV20R4_subsample
(Feb 10 04:04:03) [35/56 - 62.50%] **Ready ** Task 0: kneaddata____HD42R4_subsample
(Feb 10 04:04:03) [35/56 - 62.50%] **Started ** Task 0: kneaddata____HD42R4_subsample
(Feb 10 04:04:03) [36/56 - 64.29%] **Failed ** Task 0: kneaddata____HD42R4_subsample
(Feb 10 04:04:03) [37/56 - 66.07%] **Failed ** Task 13: kneaddata_read_count_table
(Feb 10 04:04:03) [38/56 - 67.86%] **Failed ** Task 14: metaphlan____HD42R4_subsample
(Feb 10 04:04:03) [39/56 - 69.64%] **Failed ** Task 21: metaphlan_join_taxonomic_profiles
(Feb 10 04:04:03) [40/56 - 71.43%] **Failed ** Task 22: metaphlan_count_species
(Feb 10 04:04:03) [41/56 - 73.21%] **Failed ** Task 23: humann____HD42R4_subsample
(Feb 10 04:04:03) [42/56 - 75.00%] **Failed ** Task 30: humann_count_alignments_species
(Feb 10 04:04:03) [43/56 - 76.79%] **Failed ** Task 31: humann_regroup_UniRef2EC____HD42R4_subsample
(Feb 10 04:04:03) [44/56 - 78.57%] **Failed ** Task 38: humann_join_tables_ecs
(Feb 10 04:04:03) [45/56 - 80.36%] **Failed ** Task 46: humann_renorm_ecs_relab____HD42R4_subsample
(Feb 10 04:04:03) [46/56 - 82.14%] **Failed ** Task 59: humann_join_tables_ecs_relab
(Feb 10 04:04:03) [47/56 - 83.93%] **Failed ** Task 62: humann_count_features_ecs
(Feb 10 04:04:03) [48/56 - 85.71%] **Failed ** Task 37: humann_join_tables_genefamilies
(Feb 10 04:04:03) [49/56 - 87.50%] **Failed ** Task 39: humann_join_tables_pathabundance
(Feb 10 04:04:03) [50/56 - 89.29%] **Failed ** Task 40: humann_renorm_genes_relab____HD42R4_subsample
(Feb 10 04:04:03) [51/56 - 91.07%] **Failed ** Task 58: humann_join_tables_genes_relab
(Feb 10 04:04:03) [52/56 - 92.86%] **Failed ** Task 61: humann_count_features_genes
(Feb 10 04:04:03) [53/56 - 94.64%] **Failed ** Task 52: humann_renorm_pathways_relab____HD42R4_subsample
(Feb 10 04:04:03) [54/56 - 96.43%] **Failed ** Task 60: humann_join_tables_pathways_relab
(Feb 10 04:04:03) [55/56 - 98.21%] **Failed ** Task 63: humann_count_features_pathways
(Feb 10 04:04:03) [56/56 - 100.00%] **Failed ** Task 64: humann_merge_feature_counts
Run Finished
Task 3 failed
Name: kneaddata____HD32R1_subsample
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/anadama2/runners.py", line 201, in _run_task_locally
action_func(task)
File "/usr/local/lib/python3.6/dist-packages/anadama2/helpers.py", line 89, in actually_sh
ret = _sh(s, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/anadama2/util/__init__.py", line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `kneaddata --input /Tutorials/HD32R1_subsample.fastq.gz --output /output/test3/kneaddata/main --threads 1 --output-prefix HD32R1_subsample --reference-db /opt/biobakery_workflows_databases/kneaddata_db_human_genome --serial --run-trf && mv /output/test3/kneaddata/main/HD32R1_subsample.repeats.removed.fastq /output/test3/kneaddata/main/HD32R1_subsample.fastq' failed.
Out: b''
Err: b'ERROR: Unable to find bowtie2 index files in directory: /opt/biobakery_workflows_databases/kneaddata_db_human_genome\n'
>Task 16 failed
Name: metaphlan____HD32R1_subsample
Original error:
Task failed because parent task `3' failed
Task 25 failed
Name: humann____HD32R1_subsample
Original error:
Task failed because parent task `16' failed
Task 32 failed
Name: humann_regroup_UniRef2EC____HD32R1_subsample
Original error:
Task failed because parent task `25' failed
Task 47 failed
Name: humann_renorm_ecs_relab____HD32R1_subsample
Original error:
Task failed because parent task `32' failed
Task 41 failed
Name: humann_renorm_genes_relab____HD32R1_subsample
Original error:
Task failed because parent task `25' failed
Task 53 failed
Name: humann_renorm_pathways_relab____HD32R1_subsample
Original error:
Task failed because parent task `25' failed
Task 5 failed
Name: kneaddata____HD48R4_subsample
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/anadama2/runners.py", line 201, in _run_task_locally
action_func(task)
File "/usr/local/lib/python3.6/dist-packages/anadama2/helpers.py", line 89, in actually_sh
ret = _sh(s, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/anadama2/util/__init__.py", line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `kneaddata --input /Tutorials/HD48R4_subsample.fastq.gz --output /output/test3/kneaddata/main --threads 1 --output-prefix HD48R4_subsample --reference-db /opt/biobakery_workflows_databases/kneaddata_db_human_genome --serial --run-trf && mv /output/test3/kneaddata/main/HD48R4_subsample.repeats.removed.fastq /output/test3/kneaddata/main/HD48R4_subsample.fastq' failed.
Out: b''
Err: b'ERROR: Unable to find bowtie2 index files in directory: /opt/biobakery_workflows_databases/kneaddata_db_human_genome\n'
Task 17 failed
Name: metaphlan____HD48R4_subsample
Original error:
Task failed because parent task `5' failed
Task 26 failed
Name: humann____HD48R4_subsample
Original error:
Task failed because parent task `17' failed
Task 33 failed
Name: humann_regroup_UniRef2EC____HD48R4_subsample
Original error:
Task failed because parent task `26' failed
Task 48 failed
Name: humann_renorm_ecs_relab____HD48R4_subsample
Original error:
Task failed because parent task `33' failed
Task 42 failed
Name: humann_renorm_genes_relab____HD48R4_subsample
Original error:
Task failed because parent task `26' failed
Task 54 failed
Name: humann_renorm_pathways_relab____HD48R4_subsample
Original error:
Task failed because parent task `26' failed
Task 7 failed
Name: kneaddata____LV16R4_subsample
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/anadama2/runners.py", line 201, in _run_task_locally
action_func(task)
File "/usr/local/lib/python3.6/dist-packages/anadama2/helpers.py", line 89, in actually_sh
ret = _sh(s, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/anadama2/util/__init__.py", line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `kneaddata --input /Tutorials/LV16R4_subsample.fastq.gz --output /output/test3/kneaddata/main --threads 1 --output-prefix LV16R4_subsample --reference-db /opt/biobakery_workflows_databases/kneaddata_db_human_genome --serial --run-trf && mv /output/test3/kneaddata/main/LV16R4_subsample.repeats.removed.fastq /output/test3/kneaddata/main/LV16R4_subsample.fastq' failed.
Out: b''
Err: b'ERROR: Unable to find bowtie2 index files in directory: /opt/biobakery_workflows_databases/kneaddata_db_human_genome\n'
Task 18 failed
Name: metaphlan____LV16R4_subsample
Original error:
Task failed because parent task `7' failed
Task 27 failed
Name: humann____LV16R4_subsample
Original error:
Task failed because parent task `18' failed
Task 34 failed
Name: humann_regroup_UniRef2EC____LV16R4_subsample
Original error:
Task failed because parent task `27' failed
Task 49 failed
Name: humann_renorm_ecs_relab____LV16R4_subsample
Original error:
Task failed because parent task `34' failed
Task 43 failed
Name: humann_renorm_genes_relab____LV16R4_subsample
Original error:
Task failed because parent task `27' failed
Task 55 failed
Name: humann_renorm_pathways_relab____LV16R4_subsample
Original error:
Task failed because parent task `27' failed
Task 9 failed
Name: kneaddata____LD96R2_subsample
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/anadama2/runners.py", line 201, in _run_task_locally
action_func(task)
File "/usr/local/lib/python3.6/dist-packages/anadama2/helpers.py", line 89, in actually_sh
ret = _sh(s, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/anadama2/util/__init__.py", line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `kneaddata --input /Tutorials/LD96R2_subsample.fastq.gz --output /output/test3/kneaddata/main --threads 1 --output-prefix LD96R2_subsample --reference-db /opt/biobakery_workflows_databases/kneaddata_db_human_genome --serial --run-trf && mv /output/test3/kneaddata/main/LD96R2_subsample.repeats.removed.fastq /output/test3/kneaddata/main/LD96R2_subsample.fastq' failed.
Out: b''
Err: b'ERROR: Unable to find bowtie2 index files in directory: /opt/biobakery_workflows_databases/kneaddata_db_human_genome\n'
Task 19 failed
Name: metaphlan____LD96R2_subsample
Original error:
Task failed because parent task `9' failed
Task 28 failed
Name: humann____LD96R2_subsample
Original error:
Task failed because parent task `9' failed
Task 35 failed
Name: humann_regroup_UniRef2EC____LD96R2_subsample
Original error:
Task failed because parent task `28' failed
Task 50 failed
Name: humann_renorm_ecs_relab____LD96R2_subsample
Original error:
Task failed because parent task `35' failed
Task 44 failed
Name: humann_renorm_genes_relab____LD96R2_subsample
Original error:
Task failed because parent task `28' failed
Task 56 failed
Name: humann_renorm_pathways_relab____LD96R2_subsample
Original error:
Task failed because parent task `28' failed
Task 11 failed
Name: kneaddata____LV20R4_subsample
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/anadama2/runners.py", line 201, in _run_task_locally
action_func(task)
File "/usr/local/lib/python3.6/dist-packages/anadama2/helpers.py", line 89, in actually_sh
ret = _sh(s, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/anadama2/util/__init__.py", line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `kneaddata --input /Tutorials/LV20R4_subsample.fastq.gz --output /output/test3/kneaddata/main --threads 1 --output-prefix LV20R4_subsample --reference-db /opt/biobakery_workflows_databases/kneaddata_db_human_genome --serial --run-trf && mv /output/test3/kneaddata/main/LV20R4_subsample.repeats.removed.fastq /output/test3/kneaddata/main/LV20R4_subsample.fastq' failed.
Out: b''
Err: b'ERROR: Unable to find bowtie2 index files in directory: /opt/biobakery_workflows_databases/kneaddata_db_human_genome\n'
Task 20 failed
Name: metaphlan____LV20R4_subsample
Original error:
Task failed because parent task `11' failed
Task 29 failed
Name: humann____LV20R4_subsample
Original error:
Task failed because parent task `11' failed
Task 36 failed
Name: humann_regroup_UniRef2EC____LV20R4_subsample
Original error:
Task failed because parent task `29' failed
Task 51 failed
Name: humann_renorm_ecs_relab____LV20R4_subsample
Original error:
Task failed because parent task `36' failed
Task 45 failed
Name: humann_renorm_genes_relab____LV20R4_subsample
Original error:
Task failed because parent task `29' failed
Task 57 failed
Name: humann_renorm_pathways_relab____LV20R4_subsample
Original error:
Task failed because parent task `29' failed
Task 0 failed
Name: kneaddata____HD42R4_subsample
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/anadama2/runners.py", line 201, in _run_task_locally
action_func(task)
File "/usr/local/lib/python3.6/dist-packages/anadama2/helpers.py", line 89, in actually_sh
ret = _sh(s, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/anadama2/util/__init__.py", line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `kneaddata --input /Tutorials/HD42R4_subsample.fastq.gz --output /output/test3/kneaddata/main --threads 1 --output-prefix HD42R4_subsample --reference-db /opt/biobakery_workflows_databases/kneaddata_db_human_genome --serial --run-trf && mv /output/test3/kneaddata/main/HD42R4_subsample.repeats.removed.fastq /output/test3/kneaddata/main/HD42R4_subsample.fastq' failed.
Out: b''
Err: b'ERROR: Unable to find bowtie2 index files in directory: /opt/biobakery_workflows_databases/kneaddata_db_human_genome\n'
Task 13 failed
Name: kneaddata_read_count_table
Original error:
Task failed because parent task `0' failed
Task 14 failed
Name: metaphlan____HD42R4_subsample
Original error:
Task failed because parent task `0' failed
Task 21 failed
Name: metaphlan_join_taxonomic_profiles
Original error:
Task failed because parent task `14' failed
Task 22 failed
Name: metaphlan_count_species
Original error:
Task failed because parent task `21' failed
Task 23 failed
Name: humann____HD42R4_subsample
Original error:
Task failed because parent task `0' failed
Task 30 failed
Name: humann_count_alignments_species
Original error:
Task failed because parent task `23' failed
Task 31 failed
Name: humann_regroup_UniRef2EC____HD42R4_subsample
Original error:
Task failed because parent task `23' failed
Task 38 failed
Name: humann_join_tables_ecs
Original error:
Task failed because parent task `32' failed
Task 46 failed
Name: humann_renorm_ecs_relab____HD42R4_subsample
Original error:
Task failed because parent task `31' failed
Task 59 failed
Name: humann_join_tables_ecs_relab
Original error:
Task failed because parent task `46' failed
Task 62 failed
Name: humann_count_features_ecs
Original error:
Task failed because parent task `59' failed
Task 37 failed
Name: humann_join_tables_genefamilies
Original error:
Task failed because parent task `23' failed
Task 39 failed
Name: humann_join_tables_pathabundance
Original error:
Task failed because parent task `23' failed
Task 40 failed
Name: humann_renorm_genes_relab____HD42R4_subsample
Original error:
Task failed because parent task `23' failed
Task 58 failed
Name: humann_join_tables_genes_relab
Original error:
Task failed because parent task `40' failed
Task 61 failed
Name: humann_count_features_genes
Original error:
Task failed because parent task `58' failed
Task 52 failed
Name: humann_renorm_pathways_relab____HD42R4_subsample
Original error:
Task failed because parent task `23' failed
Task 60 failed
Name: humann_join_tables_pathways_relab
Original error:
Task failed because parent task `52' failed
Task 63 failed
Name: humann_count_features_pathways
Original error:
Task failed because parent task `60' failed
Task 64 failed
Name: humann_merge_feature_counts
Original error:
Task failed because parent task `61' failed
Traceback (most recent call last):
File "/usr/local/bin/wmgx.py", line 183, in <module>
workflow.go()
File "/usr/local/lib/python3.6/dist-packages/anadama2/workflow.py", line 800, in go
self._handle_finished()
File "/usr/local/lib/python3.6/dist-packages/anadama2/workflow.py", line 832, in _handle_finished
raise RunFailed()
anadama2.workflow.RunFailed
For your reference, these are the versions I used:
- Python 2.7.17
- Python3 3.6.9
- kneaddata v0.7.6
- humann v3.0.0.alpha.3
- metaphlan 3.0.1
Please help!