Optimal thread count for KneadData, Bowtie2, MetaPhlAn, and HUMAnN on multi-core systems

Microbial community profiling HUMAnN
1

Hi BioBakery team and fellow users,

I’m running a metagenomic pipeline using KneadData, Bowtie2, MetaPhlAn, and HUMAnN on a multi-core system (dual-socket, 24 physical cores per socket).

Specifically, I’m curious:

  • For each tool (KneadData, Bowtie2, MetaPhlAn, HUMAnN), which thread count (e.g., 8, 16, 24, 32) generally gives the best balance of runtime efficiency and resource utilization?
  • At which thread point does performance plateau or degrade (i.e., adding more threads yields little to no speedup)?
  • Have others observed “sweet spots” for each tool on similar high-core-count servers (e.g. 48, 64, 96 threads)?

I’d appreciate sharing any benchmark data, personal experience, or guidance from the devs. Thank you for your help!

2

I tend to throw 8 threads at MetaPhlAn + HUMAnN. That definitely gives a nice performance boost over a single thread but I think it starts to degrade after that. It’s been a long time since I actually benchmarked it though!