BioBakery grid job options

Infrastructure and utilities bioBakery workflows
1

Currently using wmgx.py v0.1

I am setting up the workflows to use slurm with grid options. It works for the most part, but I’m having issues with the specifics. It seems that each step of the workflow is assigned memory and time resources automatically and I haven’t been able to override that. If this is possible to do in the command arguments, can someone direct me to instructions? If it isn’t possible, does anyone know where to edit source code to change the current memory allocations? I’ll insert below an example of my grid options.

biobakery_workflows wmgx --input ${rawseqs} --output ${outdir} --threads 10 --pair-identifier _R1_001 \
  --grid-jobs 20 --grid slurm --grid-scratch ${outdir}/scratch --grid-partition="defq" \ ## this all works
  --grid-options="--job-name=multinode" \ ## doesn't work
  --grid-options="--time=72:00:00" \ ## doesn't work
  --grid-options="--mem=10G" \ ## doesn't work
  --grid-environment=<with list of conda environment options I use> ## works
2

Hello, The grid jobs determine time and memory automatically based on the size of the input file. If the time and memory requirements are not enough for each job the workflow will resubmit a job at most 3x times, each time with 2x more time or memory depending on which resource was not enough in the prior run. There is not a way to edit the time and memory for each task but hopefully you should not need to do so. I would recommend each task get 8 threads because that is the amount of threads used for the benchmarking. Please post if you have any additional questions!

Thanks!
Lauren

1 Like