Difficulty downloading databases in humann3

Microbial community profiling HUMAnN
1

Hello,

I am trying to download databases for humann3, using the provided ‘humann_databases’ command to do so. Using this command starts the download as expected, only to fail at about 5~7%.

command used: humann_databases --download uniref uniref90_diamond uniref_db/
error generated: CRITICAL ERROR: Unable to download and extract from URL: http://huttenhower.sph.harvard.edu/humann2_data/uniprot/uniref_annotated/uniref50_annotated_v201901.tar.gz

This error did not give me much to work with, so I started with some troubleshooting of my own. I have tried many of the easy solutions that come to mind: I run humann3 from an environment containing only humann3, python 3.7 and their dependencies; I have tried to run it from two different linux systems (both with a stable connection, and full read and write rights); I have tried to download other databases (the demo database and chocophlan work just fine); finally I also tried to download the databases using wget which cuts off the download at about the same progress.

As I don’t understand what’s going wrong, I would like your help with downloading the uniref90 database.

2

Hello, Thank you for the detailed post and sorry for any confusion. Our downloads are currently off-line at the moment due to some hardware updates on our end. Please try again tomorrow and it should hopefully be resolved.

Thank you,
Lauren

3

Hello humann3 team,

I am trying to download humann3 databases using the following command.
humann_databases --download uniref uniref90_diamond /home/sharmaa4/Databases --update-config yes

And I am getting the following error.
CRITICAL ERROR: Unable to download and extract from URL: http://huttenhower.sph.harvard.edu/humann2_data/uniprot/uniref_annotated/uniref90_annotated_v201901.tar.gz

Can you please help me to fix this issue?

Best,
Ashok

4

Hello, Sorry for the error. I just tried the download to confirm all seems to be well currently. You might have hit an intermittent issue where based on load our server is unable to serve the database. Please try again and let me know if it now works okay.

Thank you,
Lauren

5

Hi Lauren,

Thanks for your response. I tried and it failed again after downloading around ~1.3GB. This has given me the same error message.
CRITICAL ERROR: Unable to download and extract from URL: http://huttenhower.sph.harvard.edu/humann2_data/chocophlan/full_chocophlan.v296_201901.tar.gz

Looking forward to hearing from you.

6

Hi - Sorry to hear that did not fix it. If you would try downloading the file directly with a tool like wget and let me know if you see any additional error messages that would be helpful for debugging. I don’t see anything that appears to be off on our end but the download rate does appear to be slow which might be the cause of your error.

Thank you,
Lauren

7

Thank you so much. wget is quite fast. Once I will get these databases how should I add them in to the path so that humann will able to read? can you help me with this? Thank you.

8

Happy to hear wget worked okay! I was hoping it would provide us additional errors for debugging but if completed the full download that works too. All you would need to get setup is to decompress the tar archive $ tar xzvf <filename> and the run $ humann_config --update database_folders nucleotide <path> to point HUMAnN to the folder by default. Alternatively after the database tar archive is decompressed you can point HUMAnN to the folder with each run by adding the option --nucleotide-database <path>.

Thank you,
Lauren

9

Hello Lauren,
Thank you so much.
While using wget I guess these files are not completely downloaded. However from the file size it seems complete (uniref90_annotated_v201901.tar.gz is around ~19GB). When I decompressing the tar archive it is giving me the following error.

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

10

I agree with you that it looks like a partial download from the errors with tar. Do you see any errors during the download?

Thank you,
Lauren

11

I haven’t checked the log output file so I can’t say it for sure, But I have started the downloading again and will let you if there will be any error at the end of this download.

12

Hi Lauren,
Still the same error.
gzip: full_chocophlan.v296_201901.tar.gz: unexpected end of file

At the time of download, I can only see this **21-03-22 16:32:56 (6.30 KB/s) - Connection closed at byte 8776897958. Retrying multiple times.

I am not sure if there any other way by which I can get these databases. Looking forward to hearing from you.

13

I have a mirrored version of both the ChocoPhlAn database and the full UniRef90 on Dropbox. Can you try downloading it from full_chocophlan.v296_201901.tar.gz - Dropbox and uniref90_annotated_v201901.tar.gz - Dropbox?

1 Like
14

Thank you so much! Can you please upload this file also to dropbox and provide me a link?
full_mapping_v201901.tar.gz

15

@ashoks773 Thank you for the follow up. I checked and I don’t see any connectivity issues on our end. Did the dropbox links work okay for you?

@fbeghini Thank you!

Lauren

16

Yes, dropbox link worked and I have downloaded them.

17

Great! Thank you for letting us know. We will look into what we might be able to improve on our end to increase the download speeds.

Thank you,
Lauren

18

Dear Fbeghini，do you have “full_mapping_v201901.tar.gz” in your dropbox? Would you mind share with us, that would be very helpful to me.

19

Hi, yes, you can find it hosted also here http://cmprod1.cibio.unitn.it/databases/HUMAnN/full_mapping_v201901.tar.gz

21

you are so awesome!!!, thanks so much!