Disk space to run Humann (v3.0.0.alpha.3)

Microbial community profiling HUMAnN
1

Hi!

I am trying to run Humann3 using:
MetaPhlAn version 3.0.1 (25 Jun 2020)
Bowtie2 version 2.3.5.1

When I ran the pipeline using demo.fastq file the process is complete without problem but now I am using my own data and the process is killed while bowtie2 is running. Is a disk space problem but I have near to 500 gb of free space. How much space I need to run Humann?

The command I used is:

humann --threads 16 -i myfastq.fastq.gz -o humann_results/

(myfast.fastq.gz = ca. 16 gb)

The files created before the process is killed are:

-rw-rw-r-- 1 ubuntu ubuntu 753 Jul 21 15:31 35_merge.log
-rw-rw-r-- 1 ubuntu ubuntu 94636370802 Jul 17 20:52 35_merge_bowtie2_aligned.sam
-rw-rw-r-- 1 ubuntu ubuntu 699428317 Jul 21 14:22 35_merge_bowtie2_aligned.tsv
-rw-rw-r-- 1 ubuntu ubuntu 33883528 Jul 17 17:38 35_merge_bowtie2_index.1.bt2
-rw-rw-r-- 1 ubuntu ubuntu 12709036 Jul 17 17:38 35_merge_bowtie2_index.2.bt2
-rw-rw-r-- 1 ubuntu ubuntu 515843 Jul 17 17:38 35_merge_bowtie2_index.3.bt2
-rw-rw-r-- 1 ubuntu ubuntu 12709029 Jul 17 17:38 35_merge_bowtie2_index.4.bt2
-rw-rw-r-- 1 ubuntu ubuntu 33883528 Jul 17 17:39 35_merge_bowtie2_index.rev.1.bt2
-rw-rw-r-- 1 ubuntu ubuntu 12709036 Jul 17 17:39 35_merge_bowtie2_index.rev.2.bt2
-rw-rw-r-- 1 ubuntu ubuntu 32357776654 Jul 17 21:49 35_merge_bowtie2_unaligned.fa
-rw-rw-r-- 1 ubuntu ubuntu 63596850 Jul 17 17:38 35_merge_custom_chocophlan_database.ffn
-rw-rw-r-- 1 ubuntu ubuntu 18442082 Jul 17 17:37 35_merge_metaphlan_bowtie2.txt
-rw-rw-r-- 1 ubuntu ubuntu 8841 Jul 17 17:38 35_merge_metaphlan_bugs_list.tsv

Besides a directory called tmp844ry3z_ is created with huge files:
-rw------- 1 ubuntu ubuntu 23 Jul 17 17:38 bowtie2_stderr_gz0mtlu9
-rw------- 1 ubuntu ubuntu 5850 Jul 17 17:39 bowtie2_stdout_n5w2u0yg
-rw------- 1 ubuntu ubuntu 1844074034 Jul 17 21:49 temp_alignmentsz5gocnra
-rw------- 1 ubuntu ubuntu 84166608864 Jul 17 15:58 tmp1kqg2zip
-rw------- 1 ubuntu ubuntu 84394629336 Jul 17 15:45 tmpr7r_f6w6

Thanks!!

2

Hello, Thanks for the detailed file listing! It looks like the run you posted ran through the nucleotide search portion with bowtie2 as I see the *_aligned.[sam/tsv] files but likely got stuck in the next step in the workflow. I think those large tmp output files that you see are from the translated search portion with diamond. With a compressed input file at 16gb that is a lot of reads which is great! However, you might have a lot of alignments in the translated search portion of the run which could make those files very large (looks like they are ~80Gb). With 500gb of disk space if you would just run a few jobs at a time and also set the option to remove the intermediate output files, --remove-temp-output, if you do not need these intermediate alignment files for future reference, this should solve the issue with running out of disk space.

Thanks,
Lauren

3

Hi Lauren thank you for your answer. Unfortunatelly I achieve the same error.
I used:
humann --threads 16 -i myfastq.fastq.gz -o humann_results/ --remove-temp-output

and this I see on the screen:

Output files will be written to: /home/ubuntu/sample/35/humann_results
Decompressing gzipped file …

Removing spaces from identifiers in input file …

Running metaphlan …

Found g__Dactylococcopsis.s__Dactylococcopsis_salina : 54.70% of mapped reads
Found g__Paraburkholderia.s__Paraburkholderia_fungorum : 10.53% of mapped reads
Found g__Halorubrum.s__Halorubrum_sp_AJ67 : 8.76% of mapped reads
Found g__Paraburkholderia.s__Paraburkholderia_insulsa : 8.30% of mapped reads
Found g__Halorubrum.s__Halorubrum_tebenquichense : 7.33% of mapped reads
Found g__Cutibacterium.s__Cutibacterium_acnes : 7.09% of mapped reads
Found g__Halorubrum.s__Halorubrum_hochstenium : 1.09% of mapped reads
Found g__Phormidium.s__Phormidium_willei : 0.64% of mapped reads
Found g__Phormidium.s__Phormidium_sp_OSCR : 0.62% of mapped reads
Found g__Coleofasciculus.s__Coleofasciculus_chthonoplastes : 0.51% of mapped reads
Found g__Halothece.s__Halothece_sp_PCC_7418 : 0.45% of mapped reads

Total species selected from prescreen: 11

Selected species explain 100.00% of predicted community composition

Creating custom ChocoPhlAn database …

Running bowtie2-build …

Running bowtie2 …

Killed.

Despite I used --remove-temp-outout, the temp files anyway were created:
$ ls humann_results/
35_merge_humann_temp_mir4tggp

$ ls -l humann_results/35_merge_humann_temp_mir4tggp/
total 130091300
-rw-rw-r-- 1 ubuntu ubuntu 11895 Jul 22 21:47 35_merge.log
-rw-rw-r-- 1 ubuntu ubuntu 94636370834 Jul 22 21:17 35_merge_bowtie2_aligned.sam
-rw-rw-r-- 1 ubuntu ubuntu 6030850248 Jul 22 21:42 35_merge_bowtie2_aligned.tsv
-rw-rw-r-- 1 ubuntu ubuntu 33883528 Jul 22 20:50 35_merge_bowtie2_index.1.bt2
-rw-rw-r-- 1 ubuntu ubuntu 12709036 Jul 22 20:50 35_merge_bowtie2_index.2.bt2
-rw-rw-r-- 1 ubuntu ubuntu 515843 Jul 22 20:50 35_merge_bowtie2_index.3.bt2
-rw-rw-r-- 1 ubuntu ubuntu 12709029 Jul 22 20:50 35_merge_bowtie2_index.4.bt2
-rw-rw-r-- 1 ubuntu ubuntu 33883528 Jul 22 20:51 35_merge_bowtie2_index.rev.1.bt2
-rw-rw-r-- 1 ubuntu ubuntu 12709036 Jul 22 20:51 35_merge_bowtie2_index.rev.2.bt2
-rw-rw-r-- 1 ubuntu ubuntu 32357779725 Jul 22 22:13 35_merge_bowtie2_unaligned.fa
-rw-rw-r-- 1 ubuntu ubuntu 63596850 Jul 22 20:50 35_merge_custom_chocophlan_database.ffn
-rw-rw-r-- 1 ubuntu ubuntu 18442082 Jul 22 20:49 35_merge_metaphlan_bowtie2.txt
-rw-rw-r-- 1 ubuntu ubuntu 8878 Jul 22 20:50 35_merge_metaphlan_bugs_list.tsv
drwx------ 2 ubuntu ubuntu 137 Jul 22 21:47 tmpkkhj2mqe

$ ls -l humann_results/35_merge_humann_temp_mir4tggp/tmpkkhj2mqe/
total 166411460
-rw------- 1 ubuntu ubuntu 23 Jul 22 20:50 bowtie2_stderr_ughmyd5s
-rw------- 1 ubuntu ubuntu 5904 Jul 22 20:51 bowtie2_stdout_121156w0
-rw------- 1 ubuntu ubuntu 1844080034 Jul 22 22:13 temp_alignmentsxfvg2ax5
-rw------- 1 ubuntu ubuntu 84394629336 Jul 22 19:35 tmp9vlxgji_
-rw------- 1 ubuntu ubuntu 84166608864 Jul 22 19:49 tmpo9mtdn_f

Can you help me?
Thank you!

4

Maybe is not a disk space problem since after the process is killed I still have space:

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nvme1n1 1048064000 610119504 437944496 59% /home/ubuntu/sample

5

Hi - Thanks for the follow up info! I agree with you in that I don’t think it is a disk space issue. I think possibly your run is being killed because it is running out of memory when it is processing the bowtie2 results. If you are running a couple runs at once try just running one at a time and see if this helps!

Thanks,
Lauren