# Kneaddata FASTQ header problem

**URL:** <https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482>\
**Category:** KneadData\
**Created:** [May 15, 2020, 4:29pm UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482 "2020-05-15T16:29:28Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![thad](https://avatars.discourse-cdn.com/v4/letter/t/48db29/32.png) [@thad](https://forum.biobakery.org/u/thad)\
**Post date:** [May 15, 2020, 4:29pm UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/1 "2020-05-15T16:29:29Z")

</div>

Version 0.7.5 of kneaddata does not handle FASTQ headers as seamlessly as earlier versions. With paired FASTQ files for which the headers for both read\_1.fastq and read\_2.fastq are,

'at’SRR7280791.1 1 length=101  
'at’SRR7280791.2 2 length=101  
'at’SRR7280791.3 3 length=101

where ‘at’ represents the at symbol. ( The forum script interprets the at symbol as a reference to another user. )  
kneaddata reports different read counts for paired output files kneaddata\_paired\_1.fastq and kneaddata\_paired\_2.fastq. After massaging the headers to,

'at’SRR7280791.1/1  
'at’SRR7280791.2/1  
'at’SRR7280791.3/1

for read\_1.fastq and,

'at’SRR7280791.1/2  
'at’SRR7280791.2/2  
'at’SRR7280791.3/2

for read\_2.fastq, kneaddata reports identical read counts for the paired output files.

---

<div class="post-metadata">

**Author:** ![ychauncey](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@ychauncey](https://forum.biobakery.org/u/ychauncey)\
**Post date:** [September 8, 2020, 2:32pm UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/2 "2020-09-08T14:32:59Z")

</div>

Hi, a similar problem was found in my project.  
The output file paired-1 and paired-2 have same counts, but the order of their header (or ID) is obviously different.  
paired-1

 ![1](https://canada1.discourse-cdn.com/flex027/uploads/biobakery/original/1X/82333f5aabdc01974b7b7cb39005f7737c3e8e2d.jpeg)  
paired-2  
 ![2](https://canada1.discourse-cdn.com/flex027/uploads/biobakery/original/1X/8349af8ea72eab62aa169c96acc539c50ff830bb.jpeg)  
It is serious problem to will cause the error of assembly and other workflow.  
If I reorder the ID of paired-1 and paired-2 that generated by kneaddata, whether it is right or not?

---

<div class="post-metadata">

**Author:** ![lauren.j.mciver](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@lauren.j.mciver](https://forum.biobakery.org/u/lauren.j.mciver)\
**Post date:** [September 8, 2020, 7:05pm UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/3 "2020-09-08T19:05:03Z")

</div>

Hi, Thank you for the detailed post. Kneaddata does not require the pairs be ordered. Also other tools in the bioBakery suite do not require the reads to be ordered (eg HUMAnN, MetaPhlAn, StrainPhlAn, PanPhlAn).

Thank you,  
Lauren

---

<div class="post-metadata">

**Author:** ![ychauncey](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@ychauncey](https://forum.biobakery.org/u/ychauncey)\
**Post date:** [September 9, 2020, 1:40am UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/4 "2020-09-09T01:40:49Z")

</div>

Thank you. indeed, the bioBakery suite do not require the reads to be ordered. But the assembly of paired reads might need the same header, such as megahit. If not, it could produce different assemlby.

Before performed by kneaddata, the paired reads, namely raw reads, have same counts and same order. however, after the paired reads were analyzed using the kneaddata (v0.7.10), the clean pairs count was same but the ID order of clean pairs was different.

Chauncey

---

<div class="post-metadata">

**Author:** ![lauren.j.mciver](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@lauren.j.mciver](https://forum.biobakery.org/u/lauren.j.mciver)\
**Post date:** [September 9, 2020, 7:55pm UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/5 "2020-09-09T19:55:45Z")

</div>

Hi Chauncey, Thank you for the follow up note. Good point in that assembly based methods will require the reads for the pairs to be in the same order. If you run kneaddata with the option `--reorder` it should keep the read pairs in the same order as the original input files. We did not set this to run by default because it does increase the runtime and we don’t always need the pairs to be ordered.

Thank you,  
Lauren

---

<div class="post-metadata">

**Author:** ![ychauncey](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@ychauncey](https://forum.biobakery.org/u/ychauncey)\
**Post date:** [September 10, 2020, 1:02am UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/6 "2020-09-10T01:02:31Z")

</div>

Thank you very much.

---

<div class="post-metadata">

**Author:** ![smb2020](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@smb2020](https://forum.biobakery.org/u/smb2020)\
**Post date:** [March 19, 2021, 6:41am UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/7 "2021-03-19T06:41:48Z")

</div>

Hi Lauren. I am struggling a bit because the headers have been a bit problematic for me. I download the runs from the SRA and get headers like this

forward  
@SRRxxxx.1 1 length=151  
@SRRxxxx.2 2 length=151  
@SRRxxxx.3 3 length=151

reverse  
@SRRxxxx.1 1 length=151  
@SRRxxxx.2 2 length=151  
@SRRxxxx.3 3 length=151

I am using kneaddata like this:

```
  kneaddata --threads xxx 
    --input SRRxxx_1.fastq --input SRRxxx_2.fastq \
    --output out \
    --reference-db databases/human_genome_index/ \
    --trimmomatic-options "ILLUMINACLIP:adapters.fa:2:30:10: SLIDINGWINDOW:4:20 MINLEN:50" --trimmomatic /data/\
    --bowtie2-options "--very-sensitive --dovetail"

```

I get different number of reads for the forward and reverse reads probably because of the read identifier issue. But I also cant have read identifiers because I use bwa down the line. Do you have any ideas on how to format the headers for preprocessing and also then how to change the headers again so I can use bwa which requires identical pair end reads?

---

<div class="post-metadata">

**Author:** ![smb2020](https://avatars.discourse-cdn.com/v4/letter/s/c6cbf5/32.png) [@smb2020](https://forum.biobakery.org/u/smb2020)\
**Post date:** [March 19, 2021, 6:56pm UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/8 "2021-03-19T18:56:36Z")

</div>

Would appreciate any guidance as i am truly clueless

---

<div class="post-metadata">

**Author:** ![lauren.j.mciver](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@lauren.j.mciver](https://forum.biobakery.org/u/lauren.j.mciver)\
**Post date:** [March 19, 2021, 8:35pm UTC](https://forum.biobakery.org/t/kneaddata-fastq-header-problem/482/9 "2021-03-19T20:35:08Z")

</div>

Hi - Thank you for your detailed post and sorry for any confusion about the sequence identifiers. We are working on the next release of Kneaddata which will handle the case of read pairs that do not have the expected identifiers for internal tracking.

Since the sequence identifiers on the forward and reverse strands are the same I would recommend adding a “/1” and “/2” to the ends of the forward and reverse identifiers, respectively. This way they should be tracked correctly and also not appear as the same read when run through bowtie2. Sorry for this inconvenience and please know you will not have to make this manual change with the next release of Kneaddata.

I am not familiar with bwa not allowing specific identifier formats. If you would post what specifically it does not allow I can help debug from there.

Thank you,  
Lauren