Warning in kneaddata.utilities, unable to remove file

Infrastructure and utilities KneadData
1

Hello,

I’m wondering if anyone can help with this error.

Here’s what I ran:

kneaddata
–input ./demo.fastq
–reference-db ~/private/Tools/human_ref_genome
–output qc_kneaddata_demo2
–trimmomatic /share/apps/Trimmomatic-0.39
–trimmomatic-options “SLIDINGWINDOW:4:20 MINLEN:50”
–bowtie2 /share/apps/bowtie2-2.4.2/bin
–bowtie2-options “–very-sensitive --dovetail”
–remove-intermediate-output
-v
-t 4 \

The log file includes the following warning message: WARNING: Unable to remove file: /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata_hg37dec_v0.1_bowtie2_clean.fastq

Here’s the complete log file:

01/14/2021 03:01:28 PM - kneaddata.knead_data - INFO: Running kneaddata v0.7.4

01/14/2021 03:01:28 PM - kneaddata.knead_data - INFO: Output files will be written to: /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2

01/14/2021 03:01:28 PM - kneaddata.knead_data - DEBUG: Running with the following arguments:

verbose = True

input = /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/demo.fastq

output_dir = /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2

reference_db = /home/xuanhe89/private/Tools/human_ref_genome/hg37dec_v0.1

bypass_trim = False

output_prefix = demo_kneaddata

threads = 4

processes = 1

trimmomatic_quality_scores = -phred33

bmtagger = False

trf = False

fastqc_start = False

fastqc_end = False

store_temp_output = False

remove_intermediate_output = True

cat_final_output = False

log_level = DEBUG

log = /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.log

trimmomatic_path = /share/apps/Trimmomatic-0.39/trimmomatic.jar

max_memory = 500m

trimmomatic_options = SLIDINGWINDOW:4:20 MINLEN:50

bowtie2_path = /share/apps/bowtie2-2.4.2/bin/bowtie2

bowtie2_options = --very-sensitive --dovetail --phred33

no_discordant = False

reorder = False

serial = False

bmtagger_path = None

trf_path = None

match = 2

mismatch = 7

delta = 7

pm = 80

pi = 10

minscore = 50

maxperiod = 500

fastqc_path = None

remove_temp_output = True

01/14/2021 03:01:28 PM - kneaddata.utilities - INFO: READ COUNT: raw single : Initial number of reads ( /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/demo.fastq ): 1075.0

01/14/2021 03:01:28 PM - kneaddata.utilities - DEBUG: Checking input file to Trimmomatic : /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/demo.fastq

01/14/2021 03:01:28 PM - kneaddata.utilities - INFO: Running Trimmomatic …

01/14/2021 03:01:28 PM - kneaddata.utilities - INFO: Execute command: java -Xmx500m -jar /share/apps/Trimmomatic-0.39/trimmomatic.jar SE -threads 4 -phred33 /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/demo.fastq /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.trimmed.fastq SLIDINGWINDOW:4:20 MINLEN:50

01/14/2021 03:01:29 PM - kneaddata.utilities - DEBUG: b’TrimmomaticSE: Started with arguments:\n -threads 4 -phred33 /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/demo.fastq /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.trimmed.fastq SLIDINGWINDOW:4:20 MINLEN:50\nInput Reads: 1075 Surviving: 1075 (100.00%) Dropped: 0 (0.00%)\nTrimmomaticSE: Completed successfully\n’

01/14/2021 03:01:29 PM - kneaddata.utilities - DEBUG: Checking output file from Trimmomatic : /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.trimmed.fastq

01/14/2021 03:01:29 PM - kneaddata.utilities - INFO: READ COUNT: trimmed single : Total reads after trimming ( /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.trimmed.fastq ): 1075.0

01/14/2021 03:01:29 PM - kneaddata.run - INFO: Decontaminating …

01/14/2021 03:01:29 PM - kneaddata.utilities - DEBUG: Checking input file to bowtie2 : /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.trimmed.fastq

01/14/2021 03:01:29 PM - kneaddata.utilities - INFO: Running bowtie2 …

01/14/2021 03:01:29 PM - kneaddata.utilities - INFO: Execute command: /share/apps/bowtie2-2.4.2/bin/bowtie2 --threads 4 --very-sensitive --dovetail --phred33 -x /home/xuanhe89/private/Tools/human_ref_genome/hg37dec_v0.1 -U /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.trimmed.fastq --un /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata_hg37dec_v0.1_bowtie2_clean.fastq --al /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata_hg37dec_v0.1_bowtie2_contam.fastq -S /dev/null

01/14/2021 03:01:37 PM - kneaddata.utilities - DEBUG: b’1075 reads; of these:\n 1075 (100.00%) were unpaired; of these:\n 1065 (99.07%) aligned 0 times\n 9 (0.84%) aligned exactly 1 time\n 1 (0.09%) aligned >1 times\n0.93% overall alignment rate\n’

01/14/2021 03:01:37 PM - kneaddata.utilities - DEBUG: Checking output file from bowtie2 : /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata_hg37dec_v0.1_bowtie2_clean.fastq

01/14/2021 03:01:37 PM - kneaddata.run - INFO: Total contaminate sequences in file ( /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata_hg37dec_v0.1_bowtie2_contam.fastq ) : 10.0

01/14/2021 03:01:37 PM - kneaddata.utilities - INFO: READ COUNT: decontaminated hg37dec_v0.1 single : Total reads after removing those found in reference database ( /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata_hg37dec_v0.1_bowtie2_clean.fastq ): 1065.0

01/14/2021 03:01:37 PM - kneaddata.utilities - INFO: READ COUNT: final single : Total reads after merging results from multiple databases ( /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.fastq ): 1065.0

01/14/2021 03:01:37 PM - kneaddata.utilities - WARNING: Unable to remove file: /home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata_hg37dec_v0.1_bowtie2_clean.fastq

01/14/2021 03:01:37 PM - kneaddata.knead_data - INFO:
Final output file created:
/home/xuanhe89/private/Lime_metagenomic/Batch1_test_set/qc_kneaddata_demo2/demo_kneaddata.fastq

Thank you so much for your help,

Xuan

2

Hey Xuan,

I’m not a developer but ran into the same issue recently. This error pops up because you include the flag --remove-intermediate-output. I believe the file that it is trying to remove (demo_kneaddata_hg37dec_v0.1_bowtie2_clean.fastq) is an intermediate file. For myself, the error occurs because of permission issues (ie kneaddata tries to remove the file but doesn’t end up having permissions to remove it). I get around this by manually removing any intermediate output I don’t want to keep, but overall, this error does not impact the final output from kneaddata. (sampleID_kneaddata.fastq)

3

Thank you so much for this! :grinning:

4

Thank you @ewissel ! You are 100% correct!

Thanks again,
Lauren

5

Hi Lauren, so does that mean that one should not use the --remove-intermediate parameter with the latest version? I ran into the same problem, there are about 60 files I am getting for each sample and that is taking up a lot of memory. Any help would be appreciated. Thank you.