Difficulty downloading databases in humann3

Hello humann3 team,

I am trying to download humann3 databases using the following command.
humann_databases --download uniref uniref90_diamond /home/sharmaa4/Databases --update-config yes

And I am getting the following error.
CRITICAL ERROR: Unable to download and extract from URL: http://huttenhower.sph.harvard.edu/humann2_data/uniprot/uniref_annotated/uniref90_annotated_v201901.tar.gz

Can you please help me to fix this issue?

Best,
Ashok

Hello, Sorry for the error. I just tried the download to confirm all seems to be well currently. You might have hit an intermittent issue where based on load our server is unable to serve the database. Please try again and let me know if it now works okay.

Thank you,
Lauren

Hi Lauren,

Thanks for your response. I tried and it failed again after downloading around ~1.3GB. This has given me the same error message.
CRITICAL ERROR: Unable to download and extract from URL: http://huttenhower.sph.harvard.edu/humann2_data/chocophlan/full_chocophlan.v296_201901.tar.gz

Looking forward to hearing from you.

Hi - Sorry to hear that did not fix it. If you would try downloading the file directly with a tool like wget and let me know if you see any additional error messages that would be helpful for debugging. I don’t see anything that appears to be off on our end but the download rate does appear to be slow which might be the cause of your error.

Thank you,
Lauren

Thank you so much. wget is quite fast. Once I will get these databases how should I add them in to the path so that humann will able to read? can you help me with this? Thank you.

Happy to hear wget worked okay! I was hoping it would provide us additional errors for debugging but if completed the full download that works too. All you would need to get setup is to decompress the tar archive $ tar xzvf <filename> and the run $ humann_config --update database_folders nucleotide <path> to point HUMAnN to the folder by default. Alternatively after the database tar archive is decompressed you can point HUMAnN to the folder with each run by adding the option --nucleotide-database <path>.

Thank you,
Lauren

Hello Lauren,
Thank you so much.
While using wget I guess these files are not completely downloaded. However from the file size it seems complete (uniref90_annotated_v201901.tar.gz is around ~19GB). When I decompressing the tar archive it is giving me the following error.

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

I agree with you that it looks like a partial download from the errors with tar. Do you see any errors during the download?

Thank you,
Lauren

I haven’t checked the log output file so I can’t say it for sure, But I have started the downloading again and will let you if there will be any error at the end of this download.

Hi Lauren,
Still the same error.
gzip: full_chocophlan.v296_201901.tar.gz: unexpected end of file

At the time of download, I can only see this **21-03-22 16:32:56 (6.30 KB/s) - Connection closed at byte 8776897958. Retrying multiple times.

I am not sure if there any other way by which I can get these databases. Looking forward to hearing from you.

I have a mirrored version of both the ChocoPhlAn database and the full UniRef90 on Dropbox. Can you try downloading it from full_chocophlan.v296_201901.tar.gz - Dropbox and uniref90_annotated_v201901.tar.gz - Dropbox?

1 Like

Thank you so much! Can you please upload this file also to dropbox and provide me a link?
full_mapping_v201901.tar.gz

@ashoks773 Thank you for the follow up. I checked and I don’t see any connectivity issues on our end. Did the dropbox links work okay for you?

@fbeghini Thank you!

Lauren

Yes, dropbox link worked and I have downloaded them.

Great! Thank you for letting us know. We will look into what we might be able to improve on our end to increase the download speeds.

Thank you,
Lauren

Dear Fbeghini,do you have “full_mapping_v201901.tar.gz” in your dropbox? Would you mind share with us, that would be very helpful to me.

Hi, yes, you can find it hosted also here http://cmprod1.cibio.unitn.it/databases/HUMAnN/full_mapping_v201901.tar.gz

you are so awesome!!!, thanks so much!

The links are great!
Could we request for mirror links for Kneaddata database as well?
Link is currently down.

Download URL: http://huttenhower.sph.harvard.edu/kneadData_databases/Homo_sapiens_hg37_and_human_contamination_Bowtie2_v0.1.tar.gz
CRITICAL ERROR: Unable to download and extract from URL: http://huttenhower.sph.harvard.edu/kneadData_databases/Homo_sapiens_hg37_and_human_contamination_Bowtie2_v0.1.tar.gz

I’m sorry, but unfortunately I do not have a local copy of that database.