Metawibele preprocessing error

Microbial community profiling MetaWIBELE
1

Hi,

Thank you for developing the MetaWIBELE tool!
I installed the tool using conda along with all its dependencies as specified on the GitHub page. However, I was unable to install psortb as I don’t have root access on HPC. When I tested MetaWIBELE on my sample set, I encountered some errors.

More specifically, running the command:

metawibele preprocess  --input bowtie/ \
--output metaw_preprocess --output-basename test --extension-paired "_R1.fastq.gz,_R2.fastq.gz" \
--extension ".fastq.gz" \
--local-jobs 12 > test.metawibele.stdout

I received an error (full report attached):

09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 25, DPD_291__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 22, DPD_295__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 19, DPD_293__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 16, DPD_290__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 13, DPD_287__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 10, DPD_292__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 7, DPD_288__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 4, DPD_289__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 0, DPD_294__megahit : ready and waiting for resources 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 25, DPD_291__megahit : starting to run 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:12 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_291 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_291_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_291_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_291 --out-prefix DPD_291 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_291.log 2>&1
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 22, DPD_295__megahit : starting to run 
09/02/2024 09:56:12 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:12 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_295 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_295_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_295_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_295 --out-prefix DPD_295 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_295.log 2>&1
09/02/2024 09:56:12 AM - LoggerReporter - INFO: task 19, DPD_293__megahit : starting to run 
09/02/2024 09:56:13 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:13 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_293 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_293_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_293_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_293 --out-prefix DPD_293 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_293.log 2>&1
09/02/2024 09:56:13 AM - LoggerReporter - INFO: task 7, DPD_288__megahit : starting to run 
09/02/2024 09:56:13 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:13 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_288 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_288_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_288_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_288 --out-prefix DPD_288 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_288.log 2>&1
09/02/2024 09:56:13 AM - LoggerReporter - INFO: task 16, DPD_290__megahit : starting to run 
09/02/2024 09:56:13 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:13 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_290 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_290_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_290_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_290 --out-prefix DPD_290 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_290.log 2>&1
09/02/2024 09:56:13 AM - LoggerReporter - INFO: task 13, DPD_287__megahit : starting to run 
09/02/2024 09:56:13 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9
09/02/2024 09:56:13 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_287 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_287_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_287_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_287 --out-prefix DPD_287 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_287.log 2>&1
09/02/2024 09:56:13 AM - LoggerReporter - INFO: task 10, DPD_292__megahit : starting to run 
09/02/2024 09:56:14 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:14 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_292 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_292_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_292_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_292 --out-prefix DPD_292 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_292.log 2>&1
09/02/2024 09:56:14 AM - LoggerReporter - INFO: task 4, DPD_289__megahit : starting to run 
09/02/2024 09:56:14 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:14 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_289 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_289_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_289_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_289 --out-prefix DPD_289 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_289.log 2>&1
09/02/2024 09:56:14 AM - LoggerReporter - INFO: task 0, DPD_294__megahit : starting to run 
09/02/2024 09:56:14 AM - LoggerReporter - INFO: Tracked executable version:  MEGAHIT v1.2.9

09/02/2024 09:56:14 AM - LoggerReporter - INFO: Executing with shell:  rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_294 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_294_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_294_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_294 --out-prefix DPD_294 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_294.log 2>&1
09/02/2024 09:59:42 AM - LoggerReporter - ERROR: task 19, DPD_293__megahit :  Failed! Error message : Error executing action 0. Original Exception: 
Traceback (most recent call last):
  File "/home_beegfs/ditagu/miniconda3/envs/metawible/lib/python3.7/site-packages/anadama2/runners.py", line 200, in _run_task_locally
    action_func(task)
  File "/home_beegfs/ditagu/miniconda3/envs/metawible/lib/python3.7/site-packages/anadama2/helpers.py", line 107, in actually_sh
    ret = _sh(s, **kwargs)
  File "/home_beegfs/ditagu/miniconda3/envs/metawible/lib/python3.7/site-packages/anadama2/util/__init__.py", line 320, in sh
    raise ShellException(proc.returncode, msg.format(cmd, ret[0].decode('utf-8'), ret[1].decode('utf-8')))
anadama2.util.ShellException: [Errno 247] Command `rm -rf /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_293 && megahit -1 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_293_R1.fastq.gz -2 /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/bowtie/DPD_293_R2.fastq.gz -t 12 -o /mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_293 --out-prefix DPD_293 >/mnt/beegfs2/home/ditagu/projekti/DPD_metagen_24/BMC_L09/metaw_preprocess/assembly/DPD_293.log 2>&1' failed. 
Out: 
Err: 
 
[............]


09/02/2024 02:16:57 PM - LoggerReporter - ERROR: AnADAMA run finished with errors.
Traceback (most recent call last):
  File "/home_beegfs/ditagu/miniconda3/envs/metawible/bin/preprocess.py", line 175, in <module>
    main(parse_cli_arguments())
  File "/home_beegfs/ditagu/miniconda3/envs/metawible/bin/preprocess.py", line 172, in main
    workflow.go()
  File "/home_beegfs/ditagu/miniconda3/envs/metawible/lib/python3.7/site-packages/anadama2/workflow.py", line 804, in go
    self._handle_finished()
  File "/home_beegfs/ditagu/miniconda3/envs/metawible/lib/python3.7/site-packages/anadama2/workflow.py", line 836, in _handle_finished
    raise RunFailed()
anadama2.workflow.RunFailed

The input consists of human gut microbiome samples, each containing approximately 20 million filtered reads. Human genome sequences were removed before MetaWIBELE preprocessing. I’m not entirely sure what the error messages are indicating—could there be an issue with MEGAHIT? In the past, I’ve successfully assembled metagenomic reads using MetaSPAdes, which worked very well.

Hope that someone could help me to overcome this,
Dita

metawibele_log.txt (44.1 KB)

2

Hi there,

It looks like that some of your samples failed in assembly using MEGAHIT. Could you check the detailed assembly log files for these samples (e.g. metaw_preprocess/assembly/DPD_293.log) and see if you can find more information from there?

Thanks!
Yancong

3

Hi Yancong,

Thank you for the prompt reply!
I re-ran the command with higher resources (26 processors and 10GB per processor), and it completed successfully! It seems that the issue was likely due to insufficient memory allocation.

Thanks,
Dita

1 Like