Humann can not run with metaphlan together

wenping · July 26, 2020, 1:08pm

hi,
I installed metaphlan whose database in /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan_databases/. When I run humann like following:
(py37) [zhangwenping@localhost try_humann]$ humann --input /data/liying_metagenome/clean_data_liying/SRR5130527_paired.1.fastq --output try_SRR5130527_paired.1 --metaphlan /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan
Output files will be written to: /data/try_humann/try_SRR5130527_paired.1
Removing spaces from identifiers in input file …

WARNING: Can not call software version for bowtie2

Running metaphlan …

CRITICAL ERROR: Error executing: /data/xmjd/miniconda2/envs/py37/bin/metaphlan /data/try_humann/try_SRR5130527_paired.1/SRR5130527_paired.1_humann_temp/tmpbs8jnhgf/tmpf8cqxgux -t rel_ab -o /data/try_humann/try_SRR5130527_paired.1/SRR5130527_paired.1_humann_temp/SRR5130527_paired.1_metaphlan_bugs_list.tsv --input_type fastq --bowtie2out /data/try_humann/try_SRR5130527_paired.1/SRR5130527_paired.1_humann_temp/SRR5130527_paired.1_metaphlan_bowtie2.txt

Error message returned from metaphlan :

Downloading https://www.dropbox.com/sh/7qze7m7g9fe2xjg/AAA4XDP85WHon_eHvztxkamTa/file_list.txt?dl=1

Warning: Unable to download https://www.dropbox.com/sh/7qze7m7g9fe2xjg/AAA4XDP85WHon_eHvztxkamTa/file_list.txt?dl=1
Traceback (most recent call last):
File “/data/xmjd/miniconda2/envs/py37/bin/metaphlan”, line 10, in
sys.exit(main())
File “/data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan.py”, line 916, in main
pars[‘index’] = check_and_install_database(pars[‘index’], pars[‘bowtie2db’], pars[‘bowtie2_build’], pars[‘nproc’], pars[‘force_download’])
File “/data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/init.py”, line 269, in check_and_install_database
index = resolve_latest_database(bowtie2_db, ls_f[‘mpa_latest’], force_redownload_latest)
UnboundLocalError: local variable ‘ls_f’ referenced before assignment

Can you tell me which options for humann will show the database of metaphlan locate in /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan_databases/?

Thank you very much!

wenping

lauren.j.mciver · July 28, 2020, 7:51pm

Hi Wenping, Thanks for all the info! The MetaPhlAn databases are usually installed by default in a subfolder of the MetaPhlAn executable. Sorry we don’t have an option in HUMAnN to provide the location of the MetaPhlAn database. Based on your info I think the databases would be installed in the directory you listed /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan_databases/ . It looks like the issue with installing the database might be resolved if you upgrade to a newer version of MetaPhlAn. Would you try upgrading the MetaPhlAn software and then rerun? If you continue to see errors please let us know.

Thanks!
Lauren

wenping · July 29, 2020, 4:45am

Hi Lauren,
Thank you for your reply. I upgraded the metaphlan and the version is MetaPhlAn version 3.0.1 (25 Jun 2020). However, it doesn’t work and metaphlan stiil want to download the database from dropbox that is not availble for us.

fbeghini · July 29, 2020, 7:35am

If Dropbox is not accessible you can try to download the database from Zenodo, here’s the link https://zenodo.org/record/3957592.
I’ve pushed a new release (3.0.2) in which the database automatically is retrieved from Zenodo if Dropbox is not reachable.

wenping · July 29, 2020, 7:54am

Hi Francesco,
Thank you very much for your reply. I downloaded the database of metaphlan and metaphlan can run succesfully when showing the pathway of the database. However, no options for running humann to show the pathway of database of metaphlan.

fbeghini · July 29, 2020, 8:08am

You can use the HUMAnN --metaphlan-options parameter to provide additional MetaPhlAn parameters for the internal exection (e.g. -x <dbname> --bowtie2db <db_path>)

wenping · July 29, 2020, 8:27am

Hi Francesco,
I run the following code, but it doesn’t work:

(py37) [zhangwenping@localhost try_humann]$ humann --input /data/liying_metagenome/clean_data_liying/SRR5130527_paired.1.fastq --output try_SRR5130527_paired.1 --metaphlan-options -x mpa_v30_CHOCOPhlAn_201901 --bowtie2db /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan_databases/
usage: humann [-h] [–version] [-v] [-r] [–bypass-prescreen]
[–bypass-nucleotide-index] [–bypass-translated-search]
[–bypass-nucleotide-search] -i <input.fastq> -o
[–nucleotide-database <nucleotide_database>]
[–annotation-gene-index <3>]
[–protein-database <protein_database>] [–evalue <1.0>]
[–search-mode {uniref50,uniref90}] [–metaphlan ]
[–metaphlan-options <metaphlan_options>]
[–diamond-options <diamond_options>]
[–bowtie-options <bowtie_options>] [–o-log <sample.log>]
[–log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
[–remove-temp-output] [–threads <1>]
[–prescreen-threshold <0.01>]
[–nucleotide-identity-threshold <0.0>]
[–translated-identity-threshold <Automatically: 50.0 or 80.0, Custom: 0.0-100.0>]
[–translated-subject-coverage-threshold <50.0>]
[–nucleotide-subject-coverage-threshold <50.0>]
[–translated-query-coverage-threshold <90.0>]
[–nucleotide-query-coverage-threshold <90.0>]
[–bowtie2 ] [–usearch ]
[–rapsearch ] [–diamond ]
[–taxonomic-profile <taxonomic_profile.tsv>]
[–id-mapping <id_mapping.tsv>]
[–translated-alignment {usearch,rapsearch,diamond}]
[–xipe {on,off}] [–minpath {on,off}] [–pick-frames {on,off}]
[–gap-fill {on,off}] [–output-format {tsv,biom}]
[–output-max-decimals <10>] [–output-basename <sample_name>]
[–remove-stratified-output]
[–remove-column-description-output]
[–input-format {fastq,fastq.gz,fasta,fasta.gz,sam,bam,blastm8,genetable,biom}]
[–pathways-database <pathways_database.tsv>]
[–pathways {metacyc,unipathway}]
[–memory-use {minimum,maximum}]
humann: error: argument --metaphlan-options: expected one argument

fbeghini · July 29, 2020, 8:40am

You are missing the quotes, it should be --metaphlan-options "-x mpa_v30_CHOCOPhlAn_201901 --bowtie2db /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan_databases/"

wenping · July 29, 2020, 9:23am

thank you very much. It works, but another error shows as following:

(py37) [zhangwenping@localhost try_humann]$ humann --input /data/liying_metagenome/clean_data_liying/SRR5130527_paired.1.fastq --output try_SRR5130527_paired.1 --metaphlan-options ‘-x mpa_v30_CHOCOPhlAn_201901 --bowtie2db /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan_databases/’
Output files will be written to: /data/try_humann/try_SRR5130527_paired.1
Removing spaces from identifiers in input file …

WARNING: Can not call software version for bowtie2

Running metaphlan …

CRITICAL ERROR: Error executing: /data/xmjd/miniconda2/envs/py37/bin/metaphlan /data/try_humann/try_SRR5130527_paired.1/SRR5130527_paired.1_humann_temp/tmpkcc2_rom/tmpw83r1n2g -x mpa_v30_CHOCOPhlAn_201901 --bowtie2db /data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan_databases/ -o /data/try_humann/try_SRR5130527_paired.1/SRR5130527_paired.1_humann_temp/SRR5130527_paired.1_metaphlan_bugs_list.tsv --input_type fastq --bowtie2out /data/try_humann/try_SRR5130527_paired.1/SRR5130527_paired.1_humann_temp/SRR5130527_paired.1_metaphlan_bowtie2.txt

Error message returned from metaphlan :
Use of uninitialized value $bt2_args[2] in join or string at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 423.
Use of uninitialized value bt2_args[3] in join or string at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 423. Use of uninitialized value [2] in string eq at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 360.
Use of uninitialized value $[3] in string eq at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 360.
Use of uninitialized value in exists at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 81.
Use of uninitialized value in exists at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 81.
Use of uninitialized value $bt2_args[2] in join or string at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 459.
Use of uninitialized value $bt2_args[3] in join or string at /data/xmjd/miniconda2/envs/py37/bin/bowtie2 line 459.
Traceback (most recent call last):
File “/data/xmjd/miniconda2/envs/py37/bin/metaphlan”, line 10, in
sys.exit(main())
File “/data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan.py”, line 991, in main
min_alignment_len=pars[‘min_alignment_len’], read_min_len=pars[‘read_min_len’], min_mapq_val=pars[‘min_mapq_val’])
File “/data/xmjd/miniconda2/envs/py37/lib/python3.7/site-packages/metaphlan/metaphlan.py”, line 429, in run_bowtie2
nreads = int(read_fastx_stderr[0])
IndexError: list index out of range

fbeghini · July 29, 2020, 3:25pm

Could you run read_fastx.py -l 70 <input file> > /dev/null on the input fastq and see if the total number of reads is printed on stderr?

wenping · July 30, 2020, 2:44am

I run the code and no message was shown as the following:

(py37) [zhangwenping@localhost clean_data_liying] read_fastx.py -l 70 SRR5130527_paired.1.fastq > /dev/null (py37) [zhangwenping@localhost clean_data_liying]

fbeghini · July 30, 2020, 8:55am

Are you sure the input file is not empty and well-formed?

wenping · July 30, 2020, 9:05am

The input file is as the following:
(py37) [zhangwenping@localhost clean_data_wuqi] read_fastx.py -l 70 CRR055205_paired.1.fastq > /dev/null (py37) [zhangwenping@localhost clean_data_wuqi] more CRR055205_paired.1.fastq
@HWI-ST531R:260:C4GKMACXX:3:1101:1954:1955 1:N:0:GTCTG
TAAAGTATACATTTATAATTAGTTAGGAGAAGAATTTATGTCGTTTTCAGC
+
CCCFFEDEHHHHGJJJIJJJJJHIJJJGGJIJIJJJJIIJHIJHIIJJIJI
@HWI-ST531R:260:C4GKMACXX:3:1101:2131:1942 1:N:0:GTCTG
TCANCAGTCATTATTGCTTCACCATGAACTTCAAAAAAATCACCCGTTG
+
@@@#4ADDHHHHHEIHEIJIIIJJJJJIJIJJIIJJIJJGBDFFHHAEH
@HWI-ST531R:260:C4GKMACXX:3:1101:2108:1942 1:N:0:GTCTG
CATNTTTATCATTAACTGGTCGAACAAGAATGTCAAATAATTGGGGATCGG
+
CCC#4ADFHHHHHJJJJJJHIJJJIJJJHIJJJJJJJJJJJJJJJIFGIIJ
@HWI-ST531R:260:C4GKMACXX:3:1101:2487:1994 1:N:0:GTCTG
GTCATAGATTAAGGCATGAAGAAAACTATTGTGAAGAAATTAATCTCAA
+
?@@DFDD?CFFFFHIIIGEHIIIIGEC@HICFDGHGEGHGHIICDG>DH
@HWI-ST531R:260:C4GKMACXX:3:1101:2647:1943 1:N:0:GTCTG
AGTNGTGTCGTGTGTTGAGGGTGTTATGGAACCTGTATTTATTGACATCTG
+
@@?#4=BDFFFFBGFFEEGII1CCFEGEIDFFIIFDEGIICGIIFFIEFII
@HWI-ST531R:260:C4GKMACXX:3:1101:2676:1948 1:N:0:GTCTG
CCGNTTGGTACAATAATTCATACAGGAGATTTTAAAATAGACTATACTCCA
+
@@C#4ADDBFHHHJJJJJIJDHJJJJIIFHGIJIIIJEEGHIHIJIJJIJJ
@HWI-ST531R:260:C4GKMACXX:3:1101:2694:1962 1:N:0:GTCTG
TTTAAGACAAATTCAAGATGGGATAGAGCTATTTATACTTGGACTAGGAAG
+
CCCFFFFFHHHHHJJIJIJJJJIJIJJJJJJJJJIGJJJJJJJJJIJJIII
@HWI-ST531R:260:C4GKMACXX:3:1101:2570:1982 1:N:0:GTCTG
CTATATCATTAAATCCATCCTTATTGAGAGTTTCTATGATTGCTGCAAAAACTTC
+
CCCFDFDFHHFHHIIHIIIGIJIHHJHIHIAHHGIGGIJIJIJJJJJIIHJJIJJ
(py37) [zhangwenping@localhost clean_data_wuqi]$

fbeghini · July 30, 2020, 9:27am

It looks like that all the reads may be filtered since the length is below 70bp. You should try adding --read_min_len 49 to --metaphlan-options

wenping · July 31, 2020, 4:56am

Hi, Francesco
Thank you very much, it works.