No MetaPhlAn BowTie2 database found (--index option)!

Hi Biobakery community!

I recently upgraded to HUMAnN v3.0 after having used HUMAnN2, but don’t seem to be able to run the demo because of a recurring error.
This text will be hidden
I installed it via conda in a contained environment, as suggested here: humann3 – The Huttenhower Lab

# create env 
conda create --name biobakery3 python=3.7
# activate env
conda activate biobakery3

# update channels
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --add channels biobakery

# install biobakery and metaphlan
conda install -c biobakery human
conda install -c bioconda metaphlan

# test installation
humann_test

All of the tests seemed to be fine. I then proceeded to run the demo:

humann -i demo.fastq -o demo_outputs

The first time I ran it, it took 4 hours, after which the job was killed on the cluster I was using.

After that, it returned the following errors:

CRITICAL ERROR: Error executing: /data/home/btx143/.conda/envs/biobakery3/bin/metaphlan /data/home/btx143/shotgun2/demo/data/demo.fastq -t rel_ab -o /data/home/btx143/shotgun2/demo/outputs/demo_output/demo_humann_temp/demo_metaphlan_bugs_list.tsv --input_type fastq --bowtie2out /data/home/btx143/shotgun2/demo/outputs/demo_output/demo_humann_temp/demo_metaphlan_bowtie2.txt

and

Error message returned from metaphlan :
No MetaPhlAn BowTie2 database found (–index option)!
Expecting location /data/home/btx143/.conda/envs/biobakery3/lib/python3.7/site-packages/metaphlan/metaphlan_databases/mpa_v30_CHOCOPhlAn_201901
Exiting…

The issue is that when I check in the directory where the database indices should be, then I find the following files:

mpa_latest
mpa_v30_CHOCOPhlAn_201901.1.bt2
mpa_v30_CHOCOPhlAn_201901.2.bt2
mpa_v30_CHOCOPhlAn_201901.3.bt2
mpa_v30_CHOCOPhlAn_201901.4.bt2
mpa_v30_CHOCOPhlAn_201901.fna
mpa_v30_CHOCOPhlAn_201901.fna.bz2
mpa_v30_CHOCOPhlAn_201901.md5
mpa_v30_CHOCOPhlAn_201901.pkl
mpa_v30_CHOCOPhlAn_201901.rev.1.bt2
mpa_v30_CHOCOPhlAn_201901.rev.2.bt2
mpa_v30_CHOCOPhlAn_201901.tar

And the expected “mpa_v30_CHOCOPhlAn_201901” is missing

I found a similar post in this forum and tried the following:

1. manually delete metaphlan/bowtie2 databases and re-run demo

From what I unnderstand, humann (through metaphlan and bowtie2) should automatically

2. manually delete metaphlan/bowtie2 databases, download the files and use bowtie2 to build the databases

tar -xvf mpa_v30_CHOCOPhlAn_201901.tar
bunzip2 *.bz2
bowtie2-build mpa_v30_CHOCOPhlAn_201901.fna mpa_v30_CHOCOPhlAn_201901

And even tried to run humann specifying the databases option:

humann -i demo.fastq -o sample_results --metaphlan-options “–bowtie2db {condapath}/{dbpa} -x mpa_v30_CHOCOPhlAn_201901”

After this, I still get the same annoying error:

No MetaPhlAn BowTie2 database found (–index option)!
Expecting location /data/home/btx143/.conda/envs/biobakery3/lib/python3.7/site-packages/metaphlan/metaphlan_databases/mpa_v30_CHOCOPhlAn_201901
Exiting…

Did anyone run into the same or similar error?

Cheers,

Giacomo

2 Likes

Hello, Sorry you are having issues running the demo. Thank you for your detailed and organized post! I think you might have an older version of MetaPhlAn installed. Can you check the version number? If it is an older version if you update to the latest version and then run $ metaphlan --install it should download and install the database. Once you have the database installed, you should hopefully be set! If you still see errors double check you have the latest version of HUMAnN installed. Sometimes conda does not always install the latest version but instead it installs an older version that is compatible with the current conda environment.

Thank you,
Lauren

Hi Lauren,

Thank you for your reply, and sorry for late answer!
I did have metaphlan v3.0.7 installed already. All it took was to run:

metaphlan --install

(as you sugested)
and the correct databases were installed!

Thank you so much!

Giacomo

Hi Giacomo, Awesome! Thanks for the follow up and glad to hear you are all set! Please post again if you have other issues.

Thank you,
Lauren

Hi there!

I’m having trouble running the demo as well. For context, I’m on linux server and I’ve tried the exact commands listed on the humann3 site (link here). I’ve made sure that metaphlan is correct version, as well as humann. I’ve tried both the conda installation and the pip installation but consistently run into the same error when I try to run the demo. For fun, I tried running metaphlan --install but got the same error again. The error reads as follows:

###########################################
Downloading MetaPhlAn database
Please note due to the size this might take a few minutes

File /home/kmorten/bin/anaconda3/envs/biobakery3/lib/python3.7/site-packages/metaphlan/metaphlan_databases/mpa_v30_CHOCOPhlAn_201901.tar already present!

File /home/kmorten/bin/anaconda3/envs/biobakery3/lib/python3.7/site-packages/metaphlan/metaphlan_databases/mpa_v30_CHOCOPhlAn_201901.md5 already present!
Traceback (most recent call last):
File “/home/kmorten/bin/anaconda3/envs/biobakery3/bin/metaphlan”, line 10, in
sys.exit(main())
File “/home/kmorten/bin/anaconda3/envs/biobakery3/lib/python3.7/site-packages/metaphlan/metaphlan.py”, line 925, in main
pars[‘index’] = check_and_install_database(pars[‘index’], pars[‘bowtie2db’], pars[‘bowtie2_build’], pars[‘nproc’], pars[‘force_download’])
File “/home/kmorten/bin/anaconda3/envs/biobakery3/lib/python3.7/site-packages/metaphlan/init.py”, line 301, in check_and_install_database
download_unpack_tar(index, bowtie2_db, bowtie2_build, nproc, use_zenodo)
File “/home/kmorten/bin/anaconda3/envs/biobakery3/lib/python3.7/site-packages/metaphlan/init.py”, line 180, in download_unpack_tar
bt2_build_output = subp.check_output([bowtie2_build, ‘–usage’], stderr=subp.STDOUT)
File “/home/kmorten/bin/anaconda3/envs/biobakery3/lib/python3.7/subprocess.py”, line 411, in check_output
**kwargs).stdout
File “/home/kmorten/bin/anaconda3/envs/biobakery3/lib/python3.7/subprocess.py”, line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘bowtie2-build’, ‘–usage’]’ returned non-zero exit status 127.
############################################

I’ve read a lot of the posts that seem to have traces of the issue I’m experiencing, but no luck. Any suggestions would be much appreciated!

Thank you!
Kate

I was getting this error when I was running HumanN3. I fixed it (and am now getting OP’s error) by running

conda install bowtie2

for some reason bowtie2 wasn’t installed.

Also I didn’t have tbb installed and had to do that as well

I’m getting this error and

metaphlan —install

does not solve it. I do have the latest versions of metaphlan and humann

I am also running into this error. Just installed today

It might be a fault of the new database.
I managed to trick metaphlan into downloading an earlier version, by editing the mpa_latest file in the databases folder. we’ll see how that goes

no luck at all

Like @Massoud_Maher, for me this error was due to the lack of a particular threading library for C++, called tbb2. I think Bowtie needs it. I couldn’t find tbb2 on Conda. I’m in an Ubuntu environment, so I installed it outside of conda with apt:

sudo apt-get install libtbb2

Hello biobakery community,
I used alpha version of humann3 and an older version of MetaPhlAn it worked.
However, I installed the newest version of both, installed the most up to date chocoplan, uniref databases, it shows the same error message.

Error message returned from metaphlan :
No MetaPhlAn BowTie2 database found (–index option)!
Expecting location bowtie2db

Is there a way to use the newer version and still be able to run?
Thanks!!
Yike

Hi!

Just wanted to chime in and say that im getting the same error as @YikeShen. Im using humann v3.0.1 and metaphlan 3.0.14.
Running metaphlan --install returns “The database is installed”. I checked the metaphlan_databases folder in my metaphlan installation on conda and the following files are there, so it looks ok:

README.txt
mpa_latest
mpa_v30_CHOCOPhlAn_201901.1.bt2.tmp
mpa_v30_CHOCOPhlAn_201901.2.bt2.tmp
mpa_v30_CHOCOPhlAn_201901.3.bt2.tmp
mpa_v30_CHOCOPhlAn_201901.4.bt2.tmp
mpa_v30_CHOCOPhlAn_201901.76.sa
mpa_v30_CHOCOPhlAn_201901.77.sa
mpa_v30_CHOCOPhlAn_201901.78.sa
mpa_v30_CHOCOPhlAn_201901.79.sa
mpa_v30_CHOCOPhlAn_201901.80.sa
mpa_v30_CHOCOPhlAn_201901.81.sa
mpa_v30_CHOCOPhlAn_201901.82.sa
mpa_v30_CHOCOPhlAn_201901.83.sa
mpa_v30_CHOCOPhlAn_201901.84.sa
mpa_v30_CHOCOPhlAn_201901.85.sa
mpa_v30_CHOCOPhlAn_201901.86.sa
mpa_v30_CHOCOPhlAn_201901.87.sa
mpa_v30_CHOCOPhlAn_201901.88.sa
mpa_v30_CHOCOPhlAn_201901.89.sa
mpa_v30_CHOCOPhlAn_201901.90.sa
mpa_v30_CHOCOPhlAn_201901.91.sa
mpa_v30_CHOCOPhlAn_201901.92.sa
mpa_v30_CHOCOPhlAn_201901.93.sa
mpa_v30_CHOCOPhlAn_201901.94.sa
mpa_v30_CHOCOPhlAn_201901.95.sa
mpa_v30_CHOCOPhlAn_201901.96.sa
mpa_v30_CHOCOPhlAn_201901.fna
mpa_v30_CHOCOPhlAn_201901.fna.bz2
mpa_v30_CHOCOPhlAn_201901.md5
mpa_v30_CHOCOPhlAn_201901.pkl
mpa_v30_CHOCOPhlAn_201901.tar

Ive tried manually specifying the --index and --bowtie2db paths in --metaphlan-options but it still returns the same error messsage:

Error message returned from metaphlan :
No MetaPhlAn BowTie2 database found (–index option)!
Expecting location bowtie2db

The exact call i used is the following:

humann --metaphlan-options "-t rel_ab --index mpa_v30_CHOCOPhlAn_201901 --bowtie2db /priv/c3-sysbio/sysbio/LeaversData_Omics/MetabogenMelanoma/ici_env/lib/python3.9/site-packages/metaphlan/metaphlan_databases" --input /priv/c3-sysbio/sysbio/LeaversData_Omics/MetabogenMelanoma/fastq_filt_cat/D9_FDPL210411324-1a_HWWWJDSX2_L3_cat.fq.gz --output /priv/c3-sysbio/sysbio/LeaversData_Omics/MetabogenMelanoma/test_out --threads 18

Best,
Angelo

I looked through the metaphlan.py source code now and it looks like the database files i have are incorrect. The first 4 should end with .bt2 and not .bt2.tmp, also the rev1.bt2 and rev2.bt2 are missing. Gonna see if i can re-download these somehow.

EDIT:
Managed to fix it now! simply re-downloaded the databases and re-built the bowtie2 index in a separate folder using metaphlan --install --bowtie2db /path/to/mpa_dbs

I managed to run the humann_test function a few days ago without any problems, using just the installation instructions on your website (humann3 – The Huttenhower Lab). Also, just runnning metaphlan --install returns that the databases have already been installed, even though in my case they still had the .tmp extension as well. Might be good to add an extra check in these functions in a future release.

For any of u having similar issues, maybe double check that the metaphlan_databases folder in your conda installation contains the following files:

mpa_v30_CHOCOPhlAn_201901.1.bt2
mpa_v30_CHOCOPhlAn_201901.2.bt2
mpa_v30_CHOCOPhlAn_201901.3.bt2
mpa_v30_CHOCOPhlAn_201901.4.bt2
mpa_v30_CHOCOPhlAn_201901.rev1.bt2
mpa_v30_CHOCOPhlAn_201901.rev2.bt2
mpa_v30_CHOCOPhlAn_201901.fna.bz2
mpa_v30_CHOCOPhlAn_201901.md5
mpa_v30_CHOCOPhlAn_201901.pkl
mpa_v30_CHOCOPhlAn_201901.tar

Hi everyone,

I am running HUMAnN3 on an ssh client and dealing with the same error when I run the demo:
Error message returned from metaphlan :
No MetaPhlAn BowTie2 database found (–index option)!
Expecting location bowtie2db

I have updated or tried updating humann, metaphlan, bowtie2, and tbb, and the issue still occurs when I have the latest version of all of these packages.

I attempted @angelolimeta’s solution to re-download the databases and rebuild the bowtie2 index using metaphlan --install --bowtie2db mpa_dbs and that partially failed (therefore not resolving the issue), with this error message:
Building Bowtie2 indexes
Fatal error running ‘bowtie2-build --quiet --threads 4 -f mpa_dbs/mpa_v30_CHOCOPhlAn_201901.fna mpa_dbs/mpa_v30_CHOCOPhlAn_201901’
Error message: ‘Command ‘[‘bowtie2-build’, ‘–quiet’, ‘–threads’, ‘4’, ‘-f’, ‘mpa_dbs/mpa_v30_CHOCOPhlAn_201901.fna’, ‘mpa_dbs/mpa_v30_CHOCOPhlAn_201901’]’ returned non-zero exit status 247.’

Any assistance would be very appreciated!

Those two versions I installed in December 2020 works for me.

I have encountered the same error and find that every time I try to manually download the databases, I get different results. Sometimes the mpa_v30_CHOCOPhlAn_201901.rev.1.bt2/ mpa_v30_CHOCOPhlAn_201901.rev.2.bt2 files are just not included. Sometimes one or more of the file sizes are zero. I have followed another thread (Humann 3 bowtie error - #9 by nasnuvens) that suggested deleting the .bt2 files and manually rebuilding the index. This seemed to work for me but I am not sure if the files are complete.

For someone who has successfully used these databases: what are the final sizes of each file in the metaphlan_databases directory?