Plans to port kneaddata to a workflow management system?

Infrastructure and utilities KneadData
1

Hi all,

I was wondering whether there are current plans to port kneaddata to workflow management system such as snakemake or nextflow?

I am a fan of the simplicity and user friendliness of kneaddata for users working with human-derived microbiome samples but I have the experience that the current implementation as a Python-only based pipeline running jobs sequentially does not work for very deeply-sequenced samples.

Due to the large uncompressed intermediated files, kneaddata needed more > 1 TB of memory for some of my deeply sequenced metagenomic samples and took multiple days to align the sequencing data on a single machine using multiple processors. To be able to process these samples, I implemented the steps of kneaddata (Trimmomatic, BowTie2, TRF) into a Snakemake-based workflow that allowed me to distribute the steps over a large computing cluster and reduce the amount of temporary files kept for each step.

I am happy to make this workflow available to others via GitHub, however, I was wondering whether there are already plans from the biobakery team to switch to a workflow management system? If this is the case, I would prefer to contribute such an attempt, if appreciated, rather than spending further time on a separate adaptation.

Best,
Alex

2

Hi Alex, Thank you for the post. You make a good point about the importance of workflow management systems for large scale data processing. We too run our large jobs on a SLURM cluster. To distribute our jobs efficiently we have developed our own workflow management system named AnADAMA2. We use it for our tools including our bioBakery workflows. The bioBakery workflows includes Kneaddata so through the workflows we run Kneaddata in a distributed manner on our cluster; We use this method to distribute the individual Kneaddata jobs instead of distributing the individual Kneaddata tasks. Please check out the docs and let us know if you have any questions!

Thank you,
Lauren

3

Thanks, Lauren, AnADAMA2 completely slipped, my apologies. I will definitely check it out .

Best,
Alex