Thanks again
Yes the database was downloaded; but I can download it again. Sorry here what do you mean by custom folder. Do you mean in the biobakery workflow folder.
Secondly I have 32 gb ram only from that I am allocating about 28 Gb to virtual Box.
I want to try and latter on with multiple samples I can manage more ram.
Thanks and Regards
Muneer!
Hello, Yes sorry for any confusion. By custom folder I was referring to a folder different than the default one selected by the workflows.
Is it possible to allocate 30Gb to the Virtualbox? Just giving a little more memory might help that task complete.
Thanks!
Lauren
@lauren.j.mciver
Thank you so much for being so helpful and supportive.
Thanks again.
Few points to clear before I would run my samples:
- As mentioned previously, I am working with Soil and water samples. Do I need to change the database.
Thanks and Regards
Muneer!
Sure! There is only one HUMAnN and MetaPhlAn database for all sample types so you should be okay.
Thanks!
Lauren
Thanks again
Finally Let me start running my samples and hopefully after successful run,
Will update you with some good news.
Kind Regards
Muneer!
Hi again @lauren.j.mciver
As directed by you related to the ChocoPhlAn database.
I had the database in the default folder. However I downloaded it again and placed in a new folder with kneaddata, HumanN.
Kindly please guide me weather I am putting it in right folder or not?
Thanks and Regards
Muneer!
Hello, Run “$ humann_config --print” and this will list the locations of the expected HUMAnN databases. If you have placed it in the folder listed in the config you are set! If not you can use this tool to change the location of the expected database.
Thanks!
Lauren
H again @lauren.j.mciver
Thank you again
I ran the command, and I think I don’t have the full Chocophlan database available, as it reports as Demo
(qiime2-2021.4) qiime2@qiime2core2021-4:~$ humann_config --print
HUMAnN Configuration ( Section : Name = Value )
database_folders : nucleotide = /home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/humann/data/chocophlan_DEMO
database_folders : protein = /home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/humann/data/uniref_DEMO
database_folders : utility_mapping = /home/qiime2/biobakery_workflows_databases/humann/utility_mapping
run_modes : resume = False
run_modes : verbose = False
run_modes : bypass_prescreen = False
run_modes : bypass_nucleotide_index = False
run_modes : bypass_nucleotide_search = False
run_modes : bypass_translated_search = False
run_modes : threads = 1
alignment_settings : evalue_threshold = 1.0
alignment_settings : prescreen_threshold = 0.01
alignment_settings : translated_subject_coverage_threshold = 50.0
alignment_settings : translated_query_coverage_threshold = 90.0
alignment_settings : nucleotide_subject_coverage_threshold = 50.0
alignment_settings : nucleotide_query_coverage_threshold = 90.0
output_format : output_max_decimals = 10
output_format : remove_stratified_output = False
output_format : remove_column_description_output = False
Now I downloaded it manually, do I place it in biobakery_workflows_databases, where the humann/utility mapping is present
Thanks and Regards
Hello, You are right. Thanks for the detailed post. You want to install the full nucleotide and protein databases. Run these two commands and pick any folder and the script will update the config file for you.
$ humann_databases --download chocophlan full /home/qiime2/biobakery_workflows_databases/humann/nucleotide_database
$ humann_databases --download uniref uniref90_ec_filtered_diamond /home/qiime2/biobakery_workflows_databases/humann/diamond_database
I suggested the EC filtered translated search database as I think this will help you fit in the memory requirements of your VirtualBox system.
Thanks,
Lauren
Hi again
Thank you so much
I tried many times but the downloading gave errors and after multiples could only manage to download uniref90_ec_filtered_diamond.
The full chocophlan gave continuous errors after 50%.
The I downloaded it manually, and will untar it and place it like the same way you mentioned in the above command.
Would that workout, or I have to use any command.
Please update
Thanks and Regards
Muneer!
Hi, Sorry for the download issues. We are working on that on our end. You can manually download it and place it in the expected folder.
Thank you,
Lauren
Hi
Thank you so much.
I am creating the same path manually like in the command you mention.
Thank you so much for all the help.
Moving forward to run the sample, hopes for smooth running of the command.
Thanks again
Regards
Muneer
Hi @lauren.j.mciver
I trust you are well and keeping safe.
Firstly, thanks for all your help and support.
I started running my samples, however I had some issues:
- kneaddata ran successfully,
- Metaphlan ran successfully
- humann failed.
biobakery_workflows wmgx --input Biobakery_trial --bypass-strain-profiling --local-jobs 1 --output output_data
**(Sep 15 10:03:45) [ 0/28 - 0.00%] Ready ** Task 3: kneaddata____SJ-15_R2
**(Sep 15 10:03:45) [ 0/28 - 0.00%] Started ** Task 3: kneaddata____SJ-15_R2
(Sep 15 11:09:39) [ 1/28 - 3.57%] Completed Task 3: kneaddata____SJ-15_R2
**(Sep 15 11:09:39) [ 1/28 - 3.57%] Ready ** Task 8: metaphlan____SJ-15_R2.gz
**(Sep 15 11:09:39) [ 1/28 - 3.57%] Started ** Task 8: metaphlan____SJ-15_R2.gz
(Sep 15 12:04:37) [ 2/28 - 7.14%] Completed Task 8: metaphlan____SJ-15_R2.gz
**(Sep 15 12:04:37) [ 2/28 - 7.14%] Ready ** Task 13: humann____SJ-15_R2.gz
**(Sep 15 12:04:37) [ 2/28 - 7.14%] Started ** Task 13: humann____SJ-15_R2.gz
**(Sep 15 12:05:19) [ 3/28 - 10.71%] Failed ** Task 13: humann____SJ-15_R2.gz
**(Sep 15 12:05:19) [ 4/28 - 14.29%] Failed ** Task 16: humann_regroup_UniRef2EC____SJ-15_R2.gz
**(Sep 15 12:05:19) [ 5/28 - 17.86%] Failed ** Task 23: humann_renorm_ecs_relab____SJ-15_R2.gz
**(Sep 15 12:05:19) [ 6/28 - 21.43%] Failed ** Task 21: humann_renorm_genes_relab____SJ-15_R2.gz
**(Sep 15 12:05:19) [ 7/28 - 25.00%] Failed ** Task 25: humann_renorm_pathways_relab____SJ-15_R2.gz
**(Sep 15 12:05:19) [ 7/28 - 25.00%] Ready ** Task 0: kneaddata____SJ-15_R1
**(Sep 15 12:05:19) [ 7/28 - 25.00%] Started ** Task 0: kneaddata____SJ-15_R1
While the command is still running for the forward, but I am worried what caused this error
Kindly suggest
thanks and Regards
Muneer
Hi again,
Th command has completed and ended with error in humann analysis
The kneaddata and Metaphlan analysis completed successfully.
biobakery_workflows wmgx --input Biobakery_trial --bypass-strain-profiling --local-jobs 1 --output output_data
(Sep 15 10:03:45) [ 0/28 - 0.00%] **Ready ** Task 3: kneaddata____SJ-15_R2
(Sep 15 10:03:45) [ 0/28 - 0.00%] **Started ** Task 3: kneaddata____SJ-15_R2
(Sep 15 11:09:39) [ 1/28 - 3.57%] Completed Task 3: kneaddata____SJ-15_R2
(Sep 15 11:09:39) [ 1/28 - 3.57%] **Ready ** Task 8: metaphlan____SJ-15_R2.gz
(Sep 15 11:09:39) [ 1/28 - 3.57%] **Started ** Task 8: metaphlan____SJ-15_R2.gz
(Sep 15 12:04:37) [ 2/28 - 7.14%] Completed Task 8: metaphlan____SJ-15_R2.gz
(Sep 15 12:04:37) [ 2/28 - 7.14%] **Ready ** Task 13: humann____SJ-15_R2.gz
(Sep 15 12:04:37) [ 2/28 - 7.14%] **Started ** Task 13: humann____SJ-15_R2.gz
(Sep 15 12:05:19) [ 3/28 - 10.71%] **Failed ** Task 13: humann____SJ-15_R2.gz
(Sep 15 12:05:19) [ 4/28 - 14.29%] **Failed ** Task 16: humann_regroup_UniRef2EC____SJ-15_R2.gz
(Sep 15 12:05:19) [ 5/28 - 17.86%] **Failed ** Task 23: humann_renorm_ecs_relab____SJ-15_R2.gz
(Sep 15 12:05:19) [ 6/28 - 21.43%] **Failed ** Task 21: humann_renorm_genes_relab____SJ-15_R2.gz
(Sep 15 12:05:19) [ 7/28 - 25.00%] **Failed ** Task 25: humann_renorm_pathways_relab____SJ-15_R2.gz
(Sep 15 12:05:19) [ 7/28 - 25.00%] **Ready ** Task 0: kneaddata____SJ-15_R1
(Sep 15 12:05:19) [ 7/28 - 25.00%] **Started ** Task 0: kneaddata____SJ-15_R1
(Sep 15 12:51:26) [ 8/28 - 28.57%] Completed Task 0: kneaddata____SJ-15_R1
(Sep 15 12:51:26) [ 8/28 - 28.57%] **Ready ** Task 5: kneaddata_read_count_table
(Sep 15 12:51:26) [ 8/28 - 28.57%] **Started ** Task 5: kneaddata_read_count_table
(Sep 15 12:51:26) [ 9/28 - 32.14%] Completed Task 5: kneaddata_read_count_table
(Sep 15 12:51:26) [ 9/28 - 32.14%] **Ready ** Task 6: metaphlan____SJ-15_R1.gz
(Sep 15 12:51:26) [ 9/28 - 32.14%] **Started ** Task 6: metaphlan____SJ-15_R1.gz
(Sep 15 13:53:00) [10/28 - 35.71%] Completed Task 6: metaphlan____SJ-15_R1.gz
(Sep 15 13:53:00) [10/28 - 35.71%] **Ready ** Task 9: metaphlan_join_taxonomic_profiles
(Sep 15 13:53:00) [10/28 - 35.71%] **Started ** Task 9: metaphlan_join_taxonomic_profiles
(Sep 15 13:53:00) [11/28 - 39.29%] Completed Task 9: metaphlan_join_taxonomic_profiles
(Sep 15 13:53:00) [11/28 - 39.29%] **Ready ** Task 10: metaphlan_count_species
(Sep 15 13:53:00) [11/28 - 39.29%] **Started ** Task 10: metaphlan_count_species
(Sep 15 13:53:00) [12/28 - 42.86%] Completed Task 10: metaphlan_count_species
(Sep 15 13:53:00) [12/28 - 42.86%] **Ready ** Task 11: humann____SJ-15_R1.gz
(Sep 15 13:53:00) [12/28 - 42.86%] **Started ** Task 11: humann____SJ-15_R1.gz
**(Sep 15 13:53:35) [13/28 - 46.43%] Failed ** Task 11: humann____SJ-15_R1.gz
**(Sep 15 13:53:35) [14/28 - 50.00%] Failed ** Task 14: humann_count_alignments_species
**(Sep 15 13:53:35) [15/28 - 53.57%] Failed ** Task 15: humann_regroup_UniRef2EC____SJ-15_R1.gz
**(Sep 15 13:53:35) [16/28 - 57.14%] Failed ** Task 18: humann_join_tables_ecs
**(Sep 15 13:53:35) [17/28 - 60.71%] Failed ** Task 22: humann_renorm_ecs_relab____SJ-15_R1.gz
**(Sep 15 13:53:35) [18/28 - 64.29%] Failed ** Task 27: humann_join_tables_ecs_relab
**(Sep 15 13:53:35) [19/28 - 67.86%] Failed ** Task 30: humann_count_features_ecs
**(Sep 15 13:53:35) [20/28 - 71.43%] Failed ** Task 17: humann_join_tables_genefamilies
**(Sep 15 13:53:35) [21/28 - 75.00%] Failed ** Task 19: humann_join_tables_pathabundance
**(Sep 15 13:53:35) [22/28 - 78.57%] Failed ** Task 20: humann_renorm_genes_relab____SJ-15_R1.gz
**(Sep 15 13:53:35) [23/28 - 82.14%] Failed ** Task 26: humann_join_tables_genes_relab
**(Sep 15 13:53:35) [24/28 - 85.71%] Failed ** Task 29: humann_count_features_genes
**(Sep 15 13:53:35) [25/28 - 89.29%] Failed ** Task 24: humann_renorm_pathways_relab____SJ-15_R1.gz
**(Sep 15 13:53:35) [26/28 - 92.86%] Failed ** Task 28: humann_join_tables_pathways_relab
**(Sep 15 13:53:35) [27/28 - 96.43%] Failed ** Task 31: humann_count_features_pathways
**(Sep 15 13:53:35) [28/28 - 100.00%] Failed ** Task 32: humann_merge_feature_counts
Run Finished
Task 13 failed
Name: humann____SJ-15_R2.gz
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/runners.py”, line 201, in _run_task_locally
action_func(task)
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/helpers.py”, line 89, in actually_sh
ret = _sh(s, **kwargs)
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/util/init.py”, line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `humann --input /home/qiime2/output_data/kneaddata/main/SJ-15_R2.fastq.gz --output /home/qiime2/output_data/humann/main --o-log /home/qiime2/output_data/humann/main/SJ-15_R2.log --threads 1 --taxonomic-profile /home/qiime2/output_data/metaphlan/main/SJ-15_R2_taxonomic_profile.tsv ’ failed.
Out: b’Output files will be written to: /home/qiime2/output_data/humann/main\nDecompressing gzipped file …\n\n’
Err: b’ERROR: You are using the demo ChocoPhlAn database with a non-demo input file. If you have not already done so, please run humann_databases to download the full ChocoPhlAn database. If you have downloaded the full database, use the option --nucleotide-database to provide the location. You can also run humann_config to update the default database location. For additional information, please see the HUMAnN User Manual.\n’
Task 16 failed
Name: humann_regroup_UniRef2EC____SJ-15_R2.gz
Original error:
Task failed because parent task 13' failed Task 23 failed Name: humann_renorm_ecs_relab____SJ-15_R2.gz Original error: Task failed because parent task 16’ failed
Task 21 failed
Name: humann_renorm_genes_relab____SJ-15_R2.gz
Original error:
Task failed because parent task 13' failed Task 25 failed Name: humann_renorm_pathways_relab____SJ-15_R2.gz Original error: Task failed because parent task 13’ failed
Task 11 failed
Name: humann____SJ-15_R1.gz
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/runners.py”, line 201, in _run_task_locally
action_func(task)
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/helpers.py”, line 89, in actually_sh
ret = _sh(s, **kwargs)
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/util/init.py”, line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0], ret[1]))
anadama2.util.ShellException: [Errno 1] Command `humann --input /home/qiime2/output_data/kneaddata/main/SJ-15_R1.fastq.gz --output /home/qiime2/output_data/humann/main --o-log /home/qiime2/output_data/humann/main/SJ-15_R1.log --threads 1 --taxonomic-profile /home/qiime2/output_data/metaphlan/main/SJ-15_R1_taxonomic_profile.tsv ’ failed.
Out: b’Output files will be written to: /home/qiime2/output_data/humann/main\nDecompressing gzipped file …\n\n’
Err: b’ERROR: You are using the demo ChocoPhlAn database with a non-demo input file. If you have not already done so, please run humann_databases to download the full ChocoPhlAn database. If you have downloaded the full database, use the option --nucleotide-database to provide the location. You can also run humann_config to update the default database location. For additional information, please see the HUMAnN User Manual.\n’
Task 14 failed
Name: humann_count_alignments_species
Original error:
Task failed because parent task 11' failed Task 15 failed Name: humann_regroup_UniRef2EC____SJ-15_R1.gz Original error: Task failed because parent task 11’ failed
Task 18 failed
Name: humann_join_tables_ecs
Original error:
Task failed because parent task 16' failed Task 22 failed Name: humann_renorm_ecs_relab____SJ-15_R1.gz Original error: Task failed because parent task 15’ failed
Task 27 failed
Name: humann_join_tables_ecs_relab
Original error:
Task failed because parent task 22' failed Task 30 failed Name: humann_count_features_ecs Original error: Task failed because parent task 27’ failed
Task 17 failed
Name: humann_join_tables_genefamilies
Original error:
Task failed because parent task 11' failed Task 19 failed Name: humann_join_tables_pathabundance Original error: Task failed because parent task 11’ failed
Task 20 failed
Name: humann_renorm_genes_relab____SJ-15_R1.gz
Original error:
Task failed because parent task 11' failed Task 26 failed Name: humann_join_tables_genes_relab Original error: Task failed because parent task 20’ failed
Task 29 failed
Name: humann_count_features_genes
Original error:
Task failed because parent task 26' failed Task 24 failed Name: humann_renorm_pathways_relab____SJ-15_R1.gz Original error: Task failed because parent task 11’ failed
Task 28 failed
Name: humann_join_tables_pathways_relab
Original error:
Task failed because parent task 24' failed Task 31 failed Name: humann_count_features_pathways Original error: Task failed because parent task 28’ failed
Task 32 failed
Name: humann_merge_feature_counts
Original error:
Task failed because parent task `29’ failed
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2021.4/bin/wmgx.py”, line 181, in
workflow.go()
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/workflow.py”, line 801, in go
self._handle_finished()
File “/home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/anadama2/workflow.py”, line 833, in _handle_finished
raise RunFailed()
anadama2.workflow.RunFailed
Then I ran humann_config
humann_config
HUMAnN Configuration ( Section : Name = Value )
database_folders : nucleotide = /home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/humann/data/chocophlan_DEMO
database_folders : protein = /home/qiime2/biobakery_workflows_databases/humann/diamond_database/uniref
database_folders : utility_mapping = /home/qiime2/biobakery_workflows_databases/humann/utility_mapping
run_modes : resume = False
run_modes : verbose = False
run_modes : bypass_prescreen = False
run_modes : bypass_nucleotide_index = False
run_modes : bypass_nucleotide_search = False
run_modes : bypass_translated_search = False
run_modes : threads = 1
alignment_settings : evalue_threshold = 1.0
alignment_settings : prescreen_threshold = 0.01
alignment_settings : translated_subject_coverage_threshold = 50.0
alignment_settings : translated_query_coverage_threshold = 90.0
alignment_settings : nucleotide_subject_coverage_threshold = 50.0
alignment_settings : nucleotide_query_coverage_threshold = 90.0
output_format : output_max_decimals = 10
output_format : remove_stratified_output = False
output_format : remove_column_description_output = False
For me what I get from this error is that I have Demo Nucleotide database installed.
But I have installed the full chocophlan database and the same is located in home/qiime/biobakery_workflows_database/human
But I think it is not set as default, so what to do to set it as a path like utility mapping and uniref database
kindly suggest
thanks and Regards
Thanks for the full error message! I am glad to see that Kneaddata and MetaPhlAn finished without error. It looks like you need to install the nucleotide database for HUMAnN.
Try installing it with this command:
$ humann_databases --download chocophlan full /home/qiime2/biobakery_workflows_databases/humann/nucleotide_database
Thanks!
Lauren
Thank you again
I tried the downloading, but it fails again and again.
As mentioned previously I have downloaded the database manually and put in the path home/qiime2/biobakery_workflows_databases/humann/nucleotide_database
But it again gave the same error, is there any command by which I update the configuration or set the path.
Anyways kindly suggest, how to fix this error.
With Regards
Muneer!
Hi@lauren
I tried to download the database again and same downloading issues.
As I have previously downloaded it manually and placed on the folder as mentioned by you. However from the error if seems the command is setting different path such as
“nucleotide = /home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/humann/data/chocophlan_DEMO”
but not the path where it is placed. I then tried to copy the database and placed it in the above path as well, but it gave the same error.
May be any way to update the and configure the database and update the path.
Or Can I use --nucleotide-databse and put the exact path in the command.
Thanks and Regards
Hi@ Lauren
Download issues still persisting
what to do
Hi @mallamuneer , Sorry to hear you are having issues with the downloads. We are working on migrating our databases to a faster host.
Were you able to download the database manually without error? If so, can you update the humann_config again to the location of the current database? Make sure the new folder for the full database is not called “chocophlan_DEMO” as HUMAnN knows this as the location of the demo database.
Thank you,
Lauren
Hi @lauren.j.mciver
Thanks for all the help throughout
Yes, I downloaded the database manually with out any error, I placed it in the path and named as chocophlan
then I ran the command
humann_config
HUMAnN Configuration ( Section : Name = Value )
database_folders : nucleotide = /home/qiime2/miniconda/envs/qiime2-2021.4/lib/python3.8/site-packages/humann/data/chocophlan_DEMO
database_folders : protein = /home/qiime2/biobakery_workflows_databases/humann/diamond_database/uniref
database_folders : utility_mapping = /home/qiime2/biobakery_workflows_databases/humann/utility_mapping
run_modes : resume = False
run_modes : verbose = False
run_modes : bypass_prescreen = False
run_modes : bypass_nucleotide_index = False
run_modes : bypass_nucleotide_search = False
run_modes : bypass_translated_search = False
run_modes : threads = 1
alignment_settings : evalue_threshold = 1.0
alignment_settings : prescreen_threshold = 0.01
alignment_settings : translated_subject_coverage_threshold = 50.0
alignment_settings : translated_query_coverage_threshold = 90.0
alignment_settings : nucleotide_subject_coverage_threshold = 50.0
alignment_settings : nucleotide_query_coverage_threshold = 90.0
output_format : output_max_decimals = 10
output_format : remove_stratified_output = False
output_format : remove_column_description_output = False
It is still showing the cocophlan Demo
how to update it
Please advise what to do
Thanks and Regards