Kneaddata v.0.12.4
I am trying to use Kneaddata’s --max-memory flag for Trimmomatic, to avoid Java heap space errors for large fastq files (with threads set to 16, I got this error for a 9GB fastq.gz file).
I found that in order for the --max-memory flag to work, one must set the --trimmomatic PATH flag to the .jar file. However, when I set the path to something like “~/miniforge3/envs/Shotgun_shortread_env_NXXX_TEST_NAME/share/trimmomatic/trimmomatic.jar” I got an “Unable to list files in trimmomatic directory:” error. It turns out this comes from the find_dependency() function, in which it tries to list the files in path_provided - which obviously fails, because “trimmomatic.jar” is not a directory. But if I instead give the directory, the function finds both “trimmomatic” and “trimmomatic.jar”, because “trimmomatic_jar=“trimmomatic*”” in config.py, which unreliably selects the former instead of the latter.
EDIT: I originally said this was run due to bypass-trf. That was me misreading, so I have edited the post
EDIT2: For those having the same issue, a workaround is to set the Java option as an environment variable before running Kneaddata, e.g.:
export _JAVA_OPTIONS=“-Xmx{round(memory*0.75)}g”
(note, Trimmomatic specifically expects “_JAVA_OPTIONS”)
1 Like
As far as I can tell, java options such as max-memory can’t be set as part of the CL call for trimmomatic unless the .jar file is specified (even outside of kneaddata). The recommended way to do this otherwise is to set the environment variable as you figured out.
The find_dependency() issue is a bug though which we will fix. Thanks for pointing this out!
Actually, I was unable to replicate the bug. @RikkeML Are you putting “/share/trimmomatic/” as the --trimmomatic path or are you putting “/share/”? The former should work fine, unless you have a subdirectory also called “trimmomatic” but I don’t see why that would be the case.
I would change the default config.py file to be trimmomatic*.jar, but it looks like the .jar extension was removed previously to fix a different issue.
@tkuntz-hsph, I am putting “/share/trimmomatic/” as the --trimmomatic path flag. The problem is that that path has both the “trimmomatic.jar” file and a “trimmomatic” wrapper script that calls the “trimmomatic.jar” file. So when find_dependency() lists anything that matches “trimmomatic*” from that path, it gets both options, and chooses the first.
The reason you can’t reproduce this may be the trimmomatic version. The documentation states that Kneaddata requires trimmomatic=0.33, but the conda package installation uses the newest version of trimmomatic, which was v0.40 when we last installed it.
Just for extra clarification: Kneaddata’s trim() function specifically has this code to use the --max-memory flag (“java_memory” in trim()):
# Determine if the provided trimmomatic_path is a jar or an executable for conda install.
if trimmomatic_path.endswith('.jar'):
command = ["java", "-Xmx" + java_memory, "-jar", trimmomatic_path]
else:
# Assume it's a directly executable file
command = [trimmomatic_path]
but since trimmomatic_path will never end with “.jar” when there is a wrapper script called “trimmomatic” and it is impossible to give the “.jar” file as the --trimmomatic path flag due to find_dependency() the --max-memory flag is always ignored.
Hi. I think it’s not matching the path, it’s matching the wrapper script, which is very different. We don’t expect users to have both the trimmomatic jar and a wrapper in the same folder. It’d be similar as having multiple versions of trimmomatic in the same folder; neither is how the tool is normally installed. The issue with changing the search string is that in cases where only the wrapper is pointed to, then it won’t work at all with *.jar. As I said, I don’t think there is any way to pass a java option to a wrapper, even outside of kneaddata. That is something we can make more clear by adding some info to the readme. Kneaddata supports newer versions of trimmomatic as well, so if it says it has to be 0.33, that should be fixed. Thanks!
When installing Kneaddata/Trimmomatic using conda (miniforge3), the wrapper script is placed next to the .jar file by default, when installing trimmomatic in a clean environment. I have attached a screenshot below, to show the folder structure.
You are completely right that one cannot pass a java option to the wrapper. I believe the quick fix is to allow inputting the complete path to the .jar file and then making sure find_dependency() is either skipped or disregards the --trimmomatic parameter, if the path ends in “.jar”
ETA: I just double-checked and you can pass a memory option to the trimmomatic wrapper script using the flag -Xm*, e.g., “trimmomatic -Xmx20g”. This may only apply to the conda installation, though, so fixing the --trimmomatic/find_dependency() issue is probably the better option.
1 Like
Okay, yes I’ve replicated that from a conda install, I see what you mean. Thank you for pointing this out. I’ll see what we can do about a fix.
For passing the memory option to the wrapper script, if that is always possible, then I think it’s an easy fix. I’ll look into that as well.
1 Like
Yeah, the glob pattern being too broad is definitely the core issue here. Since Trimmomatic ships both the wrapper script and the jar in the same directory (depending on how it’s packaged), using just “trimmomatic*” is always gonna be ambiguous. If the fix needs to stay backwards-compatible with installs that don’t have the .jar extension for some reason, maybe the find_dependency() function should check the file type or extension explicitly rather than relying on glob ordering, since that’s not guaranteed to be consistent across systems.
