# Biobakery metagenomics memory storage requirements for running Biobakery wgmx pipeline **URL:** \ **Category:** bioBakery workflows\ **Created:** [November 2, 2023, 9:34pm UTC](https://forum.biobakery.org/t/biobakery-metagenomics-memory-storage-requirements-for-running-biobakery-wgmx-pipeline/6163 "2023-11-02T21:34:28Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Mak0130](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.biobakery.org/mak0130/32/2524_2.png) [@Mak0130](https://forum.biobakery.org/u/Mak0130)\ **Post date:** [November 2, 2023, 9:34pm UTC](https://forum.biobakery.org/t/biobakery-metagenomics-memory-storage-requirements-for-running-biobakery-wgmx-pipeline/6163/1 "2023-11-02T21:34:28Z")
Hello Biobakery admins and users, First time using the Biobakery pipeline for my metagenomics data. I have 160 fastaq.gz files (pair-end) for 80 samples. Each file is about 8 GB. I have 3TB of memory space available in my cluster lab directory to run the pipeline; the raw data itself is taking about ~1.3 TB of space, so I have ~1.7 TB of storage space. I ran the script I wrote for the Biobakery pipeline, but after running for 13 hrs, the script timed out due to it running out of disk space (when I checked the amount of memory available it said 0 TB). I wanted to ask if there was a way to determine, ahead of time, how much memory I’ll need to run the pipeline so that I can request an increase. Thank you. 🙂 ---
**Author:** ![Mak0130](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.biobakery.org/mak0130/32/2524_2.png) [@Mak0130](https://forum.biobakery.org/u/Mak0130)\ **Post date:** [November 3, 2023, 4:28pm UTC](https://forum.biobakery.org/t/biobakery-metagenomics-memory-storage-requirements-for-running-biobakery-wgmx-pipeline/6163/2 "2023-11-03T16:28:01Z")
Update, I’ve added the option “–remove-intermediate-output” into my script. I’d still like an answer to my question, but figured adding this option to my command should circumvent the issue of running out of disk space. ---
**Author:** ![Mak0130](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.biobakery.org/mak0130/32/2524_2.png) [@Mak0130](https://forum.biobakery.org/u/Mak0130)\ **Post date:** [November 6, 2023, 7:00pm UTC](https://forum.biobakery.org/t/biobakery-metagenomics-memory-storage-requirements-for-running-biobakery-wgmx-pipeline/6163/3 "2023-11-06T19:00:50Z")
Update, despite removing intermediate files I ran out of space. Before I request more storage space from our cluster, can someone provide an answer to my question? ---
**Author:** ![lauren.j.mciver](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@lauren.j.mciver](https://forum.biobakery.org/u/lauren.j.mciver)\ **Post date:** [November 9, 2023, 10:57pm UTC](https://forum.biobakery.org/t/biobakery-metagenomics-memory-storage-requirements-for-running-biobakery-wgmx-pipeline/6163/4 "2023-11-09T22:57:35Z")
Hi @Mak0130 , Yes, that is a great option to add. It will reduce the overall disk space needed a lot. In general while running the workflows I would estimate you would need about 3x the amount of disk space for intermediate files. So if you have ~1.2 Tb of input files I would estimate you would need about 3.8 Tb of disk space overall if you were to run all of your inputs files at once. Since you have 3 Tb in total of disk space I would suggest running your input files in batches to split up your runs into 3 or 4 sets. You can do this if you are using the biobakery workflows by adding the option `--local-jobs N` or `--grid-jobs N` and only `N` jobs will run at once. Please post if you run into any additional issues! Thanks! Lauren ---
**Author:** ![Mak0130](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.biobakery.org/mak0130/32/2524_2.png) [@Mak0130](https://forum.biobakery.org/u/Mak0130)\ **Post date:** [November 12, 2023, 6:44pm UTC](https://forum.biobakery.org/t/biobakery-metagenomics-memory-storage-requirements-for-running-biobakery-wgmx-pipeline/6163/5 "2023-11-12T18:44:57Z")
Thanks for the reply! I am running all my files at once, the “Kneaddata” part of the pipeline finished this morning, and it took 4.2 TB of memory space. Luckily I requested an increase in my memory storage space to about 7 TB. Currently using 5.6 TB of the 7 TB. Hopefully the 1.4 TB left is enough for the remaining portion. Will update when it’s finished running. If I run out of disk space again, I’ll split the analysis into batches. Thanks!