Cannot get biobakery_workflows vis to run

Infrastructure and utilities bioBakery workflows
1

Hi everybody,

I cannot get to run biobakery visualisation workflow. I tried the following command in the latest Docker image
biobakery_workflows wmgx_vis --input /opt/biobakery_result_files/ --output /opt/vizualisation/ --project-name TEST

Stdout :

(Nov 03 16:15:48) [0/2 - 0.00%] **Ready ** Task 0: document
(Nov 03 16:15:48) [0/2 - 0.00%] **Started ** Task 0: document
Exception:
list index out of range
Error messages will be included in output document
Traceback (most recent call last):
File “/usr/local/bin/wmgx_vis.py”, line 133, in
workflow.go()
File “/usr/local/lib/python3.6/dist-packages/anadama2/workflow.py”, line 804, in go
self._handle_finished()
File “/usr/local/lib/python3.6/dist-packages/anadama2/workflow.py”, line 836, in _handle_finished
raise RunFailed()
anadama2.workflow.RunFailed

And the content of anadama.log:

2022-11-03 16:15:51,496 LoggerReporter task_failed ERROR: task 0, document : Failed! Error message : Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/anadama2/runners.py”, line 200, in _run_task_locally
action_func(task)
File “/usr/local/lib/python3.6/dist-packages/anadama2/document.py”, line 315, in create
doc.weave(shell=PwebProcessorSpaces)
File “/usr/local/lib/python3.6/dist-packages/pweave/pweb.py”, line 198, in weave
self.run(shell)
File “/usr/local/lib/python3.6/dist-packages/pweave/pweb.py”, line 149, in run
runner.run()
File “/usr/local/lib/python3.6/dist-packages/pweave/processors.py”, line 65, in run
self.executed = list(map(self._runcode, self.parsed))
File “/usr/local/lib/python3.6/dist-packages/pweave/processors.py”, line 131, in _runcode
chunk[‘content’] = self.loadinline(chunk[‘content’])
File “/usr/local/lib/python3.6/dist-packages/anadama2/document.py”, line 259, in loadinline
result = self.loadstring(code_str).strip()
File “/usr/local/lib/python3.6/dist-packages/pweave/processors.py”, line 296, in loadstring
exec(compiled, scope)
File “chunk”, line 1, in
NameError: name ‘caption’ is not defined

I then tried a different approach.
Following the advice in this post from Biobakery’s forum, I tried to set up biobakery through conda:
conda create -c biobakery -n biobakery_3.6 python=3.6 biobakery_workflows tbb=2020.2 networkx=1.11 pweave=0.25 python-leveldb

Trying to run the vizualisation workflow failed:
biobakery_workflows vis --input run_test/output/ --output run_test/vis --log-level DEBUG --project-name TEST

Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery/bin/vis.py”, line 123, in
args.introduction_text = “{#id .class width=675px height=505px}\n\n”+intro_methods
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/biobakery_workflows/utilities.py”, line 1170, in get_package_file
found_files=list(filter(lambda file: os.path.isfile(os.path.join(package_install_folder,file)),os.listdir(package_install_folder)))
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/biobakery_workflows/…/images’`

I thave been able to circumvent this problem by doing
ln -s /home/theoscilicium/miniconda3/envs/biobakery/images/ /home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/images

I then ran again the vis workflow, here is the content of anadama.log:
2022-11-03 16:35:13,339 LoggerReporter started INFO: Beginning AnADAMA run with 6 tasks.

2022-11-03 16:35:13,340 LoggerReporter started INFO: Workflow description = A workflow for visualization
2022-11-03 16:35:13,340 LoggerReporter started INFO: Workflow version = 3.1
2022-11-03 16:35:13,340 LoggerReporter started INFO: Workflow configuration options
2022-11-03 16:35:13,340 LoggerReporter started INFO: input = run_test/output/
2022-11-03 16:35:13,340 LoggerReporter started INFO: project_name = TEST
2022-11-03 16:35:13,340 LoggerReporter started INFO: author_name =
2022-11-03 16:35:13,340 LoggerReporter started INFO: header_image =
2022-11-03 16:35:13,340 LoggerReporter started INFO: input_metadata =
2022-11-03 16:35:13,340 LoggerReporter started INFO: input_file_type =
2022-11-03 16:35:13,340 LoggerReporter started INFO: input_picard = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: input_picard_extension = quality_by_cycle_metrics
2022-11-03 16:35:13,340 LoggerReporter started INFO: metadata_categorical =
2022-11-03 16:35:13,340 LoggerReporter started INFO: metadata_continuous =
2022-11-03 16:35:13,340 LoggerReporter started INFO: metadata_exclude =
2022-11-03 16:35:13,340 LoggerReporter started INFO: min_abundance = 0.01
2022-11-03 16:35:13,340 LoggerReporter started INFO: min_samples = 3
2022-11-03 16:35:13,340 LoggerReporter started INFO: max_sets_heatmap = 25
2022-11-03 16:35:13,340 LoggerReporter started INFO: max_missing = 20.0
2022-11-03 16:35:13,340 LoggerReporter started INFO: max_sets_barplot = 15
2022-11-03 16:35:13,340 LoggerReporter started INFO: max_groups_barplot = 5
2022-11-03 16:35:13,340 LoggerReporter started INFO: correlation_threshold = 0.7
2022-11-03 16:35:13,340 LoggerReporter started INFO: format = html
2022-11-03 16:35:13,340 LoggerReporter started INFO: introduction_text =
2022-11-03 16:35:13,340 LoggerReporter started INFO: print_template = False
2022-11-03 16:35:13,340 LoggerReporter started INFO: use_template =
2022-11-03 16:35:13,340 LoggerReporter started INFO: output = run_test/vis
2022-11-03 16:35:13,340 LoggerReporter started INFO: config = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: jobs = 1
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_jobs = 0
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_partition =
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_benchmark = off
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_options = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_submit_sleep = 5
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_environment = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_scratch =
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_time_max = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: grid_mem_max = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: dry_run = False
2022-11-03 16:35:13,340 LoggerReporter started INFO: skip_nothing = False
2022-11-03 16:35:13,340 LoggerReporter started INFO: quit_early = False
2022-11-03 16:35:13,340 LoggerReporter started INFO: until_task = None
2022-11-03 16:35:13,340 LoggerReporter started INFO: exclude_task = None
2022-11-03 16:35:13,341 LoggerReporter started INFO: target = None
2022-11-03 16:35:13,341 LoggerReporter started INFO: exclude_target = None
2022-11-03 16:35:13,341 LoggerReporter started INFO: log_level = DEBUG
2022-11-03 16:35:13,359 anadama2.workflow go DEBUG: Sorting task_nos by network topology
2022-11-03 16:35:13,359 anadama2.workflow go DEBUG: Sorting complete
2022-11-03 16:35:13,359 anadama2.tracked any_different DEBUG: Dep ‘/home/theoscilicium/bb_docker_test/run_test/vis’ of type <class ‘anadama2.tracked.HugeTrackedFile’> changed: wasn’t previously saved in backend
2022-11-03 16:35:13,359 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 4 because of dep change
2022-11-03 16:35:13,359 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 5 because of dep change
2022-11-03 16:35:13,359 anadama2.tracked any_different DEBUG: Dep ‘/home/theoscilicium/bb_docker_test/run_test/vis.zip’ of type <class ‘anadama2.tracked.HugeTrackedFile’> changed: wasn’t previously saved in backend
2022-11-03 16:35:13,359 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 4 because of dep change
2022-11-03 16:35:13,359 anadama2.tracked any_different DEBUG: Dep ‘/home/theoscilicium/bb_docker_test/run_test/output/metaphlan/merged/metaphlan_taxonomic_profiles.tsv’ of type <class ‘anadama2.tracked.HugeTrackedFile’> changed: wasn’t previously saved in backend
2022-11-03 16:35:13,359 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 2 because of dep change
2022-11-03 16:35:13,359 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 3 because of dep change
2022-11-03 16:35:13,360 anadama2.tracked any_different DEBUG: Dep ‘/home/theoscilicium/bb_docker_test/run_test/vis/wmgx_report.html’ of type <class ‘anadama2.tracked.HugeTrackedFile’> changed: wasn’t previously saved in backend
2022-11-03 16:35:13,360 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 2 because of dep change
2022-11-03 16:35:13,360 anadama2.tracked any_different DEBUG: Dep ‘/home/theoscilicium/bb_docker_test/run_test/output/humann/merged/ecs_relab.tsv’ of type <class ‘anadama2.tracked.HugeTrackedFile’> changed: wasn’t previously saved in backend
2022-11-03 16:35:13,360 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 0 because of dep change
2022-11-03 16:35:13,360 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 1 because of dep change
2022-11-03 16:35:13,360 anadama2.tracked any_different DEBUG: Dep ‘/home/theoscilicium/bb_docker_test/run_test/vis/ecs/ecs_relab.tsv’ of type <class ‘anadama2.tracked.HugeTrackedFile’> changed: wasn’t previously saved in backend
2022-11-03 16:35:13,360 anadama2.workflow _filter_skipped_tasks DEBUG: Can’t skip task 0 because of dep change
2022-11-03 16:35:13,360 anadama2.runners run_tasks DEBUG: Running 6 tasks locally and serially
2022-11-03 16:35:13,360 LoggerReporter log_event DEBUG: task 5, Track pre-existing dependencies : ready and waiting for resources : 0 parents: . 1 children: [4].
2022-11-03 16:35:13,360 LoggerReporter log_event DEBUG: task 5, Track pre-existing dependencies : starting to run : 0 parents: . 1 children: [4].
2022-11-03 16:35:13,360 anadama2.runners _run_task_locally DEBUG: Executing task 5 action 0
2022-11-03 16:35:13,360 anadama2.runners _run_task_locally DEBUG: Completed executing task 5 action 0
2022-11-03 16:35:13,360 LoggerReporter log_event DEBUG: task 5, Track pre-existing dependencies : completed successfully
2022-11-03 16:35:13,360 LoggerReporter log_event DEBUG: task 3, Track pre-existing dependencies : ready and waiting for resources : 0 parents: . 1 children: [2].
2022-11-03 16:35:13,360 LoggerReporter log_event DEBUG: task 3, Track pre-existing dependencies : starting to run : 0 parents: . 1 children: [2].
2022-11-03 16:35:13,360 anadama2.runners _run_task_locally DEBUG: Executing task 3 action 0
2022-11-03 16:35:13,360 anadama2.runners _run_task_locally DEBUG: Completed executing task 3 action 0
2022-11-03 16:35:13,360 LoggerReporter log_event DEBUG: task 3, Track pre-existing dependencies : completed successfully
2022-11-03 16:35:13,360 LoggerReporter log_event DEBUG: task 1, Track pre-existing dependencies : ready and waiting for resources : 0 parents: . 1 children: [0].
2022-11-03 16:35:13,361 LoggerReporter log_event DEBUG: task 1, Track pre-existing dependencies : starting to run : 0 parents: . 1 children: [0].
2022-11-03 16:35:13,361 anadama2.runners _run_task_locally DEBUG: Executing task 1 action 0
2022-11-03 16:35:13,361 anadama2.runners _run_task_locally DEBUG: Completed executing task 1 action 0
2022-11-03 16:35:13,361 LoggerReporter log_event DEBUG: task 1, Track pre-existing dependencies : completed successfully
2022-11-03 16:35:13,361 LoggerReporter log_event INFO: task 0, humann_rename_table_ecs : ready and waiting for resources : 1 parents: [1]. 1 children: [2].
2022-11-03 16:35:13,361 LoggerReporter log_event INFO: task 0, humann_rename_table_ecs : starting to run : 1 parents: [1]. 1 children: [2].
2022-11-03 16:35:13,361 LoggerReporter task_command INFO: Executing with shell: mkdir -p run_test/vis/ecs && humann_rename_table --input /home/theoscilicium/bb_docker_test/run_test/output/humann/merged/ecs_relab.tsv --output /home/theoscilicium/bb_docker_test/run_test/vis/ecs/ecs_relab.tsv --names ec
2022-11-03 16:35:13,361 anadama2.runners _run_task_locally DEBUG: Executing task 0 action 0
2022-11-03 16:35:13,433 anadama2.helpers actually_sh INFO: Execution complete. Stdout:
Stderr: b"Loading table from: /home/theoscilicium/bb_docker_test/run_test/output/humann/merged/ecs_relab.tsv\n Treating /home/theoscilicium/bb_docker_test/run_test/output/humann/merged/ecs_relab.tsv as stratified output, e.g. [‘1.1.1.1’, ‘g__Acidaminococcus.s__Acidaminococcus_intestini’]\nLoading mapping file from: /home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/humann/tools/…/data/misc/map_level4ec_name.txt.gz\nRenamed 224 of 232 entries (96.55%)\n"
2022-11-03 16:35:13,433 anadama2.runners _run_task_locally DEBUG: Completed executing task 0 action 0
2022-11-03 16:35:13,433 LoggerReporter log_event INFO: task 0, humann_rename_table_ecs : completed successfully
2022-11-03 16:35:13,434 LoggerReporter log_event INFO: task 2, document : ready and waiting for resources : 2 parents: [3, 0]. 1 children: [4].
2022-11-03 16:35:13,434 LoggerReporter log_event INFO: task 2, document : starting to run : 2 parents: [3, 0]. 1 children: [4].
2022-11-03 16:35:13,434 anadama2.runners _run_task_locally DEBUG: Executing task 2 action 0
2022-11-03 16:35:13,441 anadama2.helpers actually_sh INFO: Executing with shell: pweave /home/theoscilicium/bb_docker_test/run_test/vis/tmpy_tmnit1/wmgx_report.pmd -o /home/theoscilicium/bb_docker_test/run_test/vis/tmpy_tmnit1/wmgx_report.md
2022-11-03 16:35:15,080 LoggerReporter task_failed ERROR: task 2, document : Failed! Error message : Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/anadama2/runners.py”, line 200, in _run_task_locally
action_func(task)
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/anadama2/document.py”, line 319, in create
sh(“pweave {0} -o {1}”.format(temp_template,intermediate_template),log_command=True)()
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/anadama2/helpers.py”, line 107, in actually_sh
ret = _sh(s, **kwargs)
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/anadama2/util/init.py”, line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0].decode(‘utf-8’), ret[1].decode(‘utf-8’)))
anadama2.util.ShellException: [Errno 1] Command ‘pweave /home/theoscilicium/bb_docker_test/run_test/vis/tmpy_tmnit1/wmgx_report.pmd -o /home/theoscilicium/bb_docker_test/run_test/vis/tmpy_tmnit1/wmgx_report.md’ failed.
Out: Error in trying to detect pandoc version
Your pandoc version is below 1.16, not setting figure size and id
Processing chunk 1 named None from line 12
Processing chunk 2 named read qc counts from line 27
Processing chunk 3 named write qc table paired from line 42
Processing chunk 4 named write qc table orphan from line 53
Processing chunk 5 named compute read ratios and show table from line 67
Processing chunk 6 named plot filtered read counts from line 88
Processing chunk 7 named read metaphlan profile and filter from line 119
Processing chunk 8 named compile species and genera and show table of counts from line 137
Processing chunk 9 named find alpha diversity plots from line 146
Processing chunk 10 named show alpha diversity boxplots from line 156
Processing chunk 11 named show alpha diversity scatterplots from line 162
Processing chunk 12 named show pcoa species from line 174

Err: Exception:
[Errno 2] No such file or directory: ‘R’: ‘R’
Error messages will be included in output document
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery/bin/pweave”, line 6, in
sys.exit(pweave.scripts.weave())
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/scripts.py”, line 55, in weave
pweave.weave(infile, **opts_dict)
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/init.py”, line 73, in weave
doc.weave(shell)
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/pweb.py”, line 198, in weave
self.run(shell)
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/pweb.py”, line 149, in run
runner.run()
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/processors.py”, line 65, in run
self.executed = list(map(self._runcode, self.parsed))
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/processors.py”, line 131, in _runcode
chunk[‘content’] = self.loadinline(chunk[‘content’])
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/processors.py”, line 331, in loadinline
result = self.loadstring(self.add_echo(code_str)).strip()
File “/home/theoscilicium/miniconda3/envs/biobakery/lib/python3.6/site-packages/pweave/processors.py”, line 296, in loadstring
exec(compiled, scope)
File “chunk”, line 1, in
NameError: name ‘caption’ is not defined

2022-11-03 16:35:15,080 LoggerReporter task_failed ERROR: task 4, archive : Failed! Error message : Task failed because parent task ‘2’ failed
2022-11-03 16:35:15,080 LoggerReporter finished ERROR: AnADAMA run finished with errors.

This problem of NameError: name ‘caption’ is not defined has been reported here in Aug. 2022 before, but I could not find any satisfactory workaround…
Any help, pointers or anything would all be very appreciated :).

2

New try with conda install and Python 3.7:
conda create -c biobakery -n biobakery_3.7 python=3.7 biobakery_workflows

First try:
biobakery_workflows vis --input run_test/wmgx_output/ --output run_test/vis/

Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/bin/vis.py”, line 123, in
args.introduction_text = “{#id .class width=675px height=505px}\n\n”+intro_methods
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/biobakery_workflows/utilities.py”, line 1170, in get_package_file
found_files=list(filter(lambda file: os.path.isfile(os.path.join(package_install_folder,file)),os.listdir(package_install_folder)))
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/biobakery_workflows/…/images’

As before, I did:
ln -s /home/theoscilicium/miniconda3/envs/biobakery_3.7/images /home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/images
Suggestion: fix conda install to avoid this path problem

Second try:
biobakery_workflows vis --input run_test/wmgx_output/ --output run_test/vis/

Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/anadama2/workflow.py”, line 481, in add_archive
output=subprocess.check_output([archive_software,“–help”])
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/subprocess.py”, line 411, in check_output
**kwargs).stdout
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/subprocess.py”, line 488, in run
with Popen(*popenargs, **kwargs) as process:
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/subprocess.py”, line 800, in init
restore_signals, start_new_session)
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/subprocess.py”, line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: ‘zip’: ‘zip’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/bin/vis.py”, line 214, in
remove_log=True)
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/anadama2/workflow.py”, line 483, in add_archive
raise EnvironmentError("Unable to archive document with software requested. Please install " +archive_software)
OSError: Unable to archive document with software requested. Please install zip

Problem solved by doing conda install zip. Suggestion: add zip as dependency

Third try:
biobakery_workflows vis --input run_test/wmgx_output/ --output run_test/vis/

(Nov 04 10:15:21) [1/3 - 33.33%] **Skipped ** Task 0: humann_rename_table_ecs
(Nov 04 10:15:21) [1/3 - 33.33%] **Ready ** Task 2: document
(Nov 04 10:15:21) [1/3 - 33.33%] **Started ** Task 2: document
(Nov 04 10:15:21) [2/3 - 66.67%] **Failed ** Task 2: document
(Nov 04 10:15:21) [3/3 - 100.00%] **Failed ** Task 4: archive
Run Finished
Task 2 failed
Name: document
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/anadama2/runners.py”, line 200, in _run_task_locally
action_func(task)
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/anadama2/document.py”, line 232, in create
from pweave import PwebPandocFormatter, Pweb, PwebProcessor
ImportError: cannot import name ‘PwebProcessor’ from ‘pweave’ (/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/pweave/init.py)

Task 4 failed
Name: archive
Original error:
Task failed because parent task `2’ failed
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/bin/vis.py”, line 217, in
workflow.go()
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/anadama2/workflow.py”, line 804, in go
self._handle_finished()
File “/home/theoscilicium/miniconda3/envs/biobakery_3.7/lib/python3.7/site-packages/anadama2/workflow.py”, line 836, in _handle_finished
raise RunFailed()
anadama2.workflow.RunFailed

So, as of 04/11/2022, the problem described in Sept. 2020 is still not resolved.

EDIT for more info:
return of conda list

packages in environment at /home/theoscilicium/miniconda3/envs/biobakery_3.7:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
alsa-lib 1.2.7.2 h166bdaf_0 conda-forge
anadama2 0.10.0 pyhb8aa349_0 biobakery
attrs 22.1.0 pyh71513ae_1 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.11.1 pyha770c72_0 conda-forge
biobakery_workflows 3.1 pyh0d95946_0 biobakery
biom-format 2.1.12 py37hda87dfa_1 conda-forge
biopython 1.79 py37h540881e_2 conda-forge
blast 2.13.0 hf3cf87c_0 bioconda
bleach 5.0.1 pyhd8ed1ab_0 conda-forge
boost-cpp 1.74.0 h75c5d50_8 conda-forge
bowtie2 2.5.0 py37hb24965f_0 bioconda
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
brotlipy 0.7.0 py37h540881e_1004 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2022.9.24 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
certifi 2022.9.24 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py37h43b0acd_1 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 py37h89c1867_0 conda-forge
cloudpickle 2.2.0 pyhd8ed1ab_0 conda-forge
cryptography 38.0.2 py37h5994e8b_1 conda-forge
curl 7.86.0 h2283fc2_1 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
debugpy 1.6.3 py37hd23a5d3_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
dendropy 4.5.2 pyh3252c3a_0 bioconda
diamond 2.0.15 hb97b32f_1 bioconda
entrez-direct 16.2 he881be0_1 bioconda
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
expat 2.5.0 h27087fc_0 conda-forge
fastqc 0.11.9 hdfd78af_1 bioconda
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.1 hc2a2eb6_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.38.0 py37h540881e_0 conda-forge
freetype 2.12.1 hca18f0e_0 conda-forge
future 0.18.2 py37h89c1867_5 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glpk 4.57 0 bioconda
graphite2 1.3.13 h58526e2_1001 conda-forge
h5py 3.7.0 nompi_py37hf1ce037_101 conda-forge
harfbuzz 5.3.0 h418a68e_0 conda-forge
hdf5 1.12.2 nompi_h4df4325_100 conda-forge
humann 3.6 pyhdf09c49_0 biobakery
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 4.11.4 py37h89c1867_0 conda-forge
importlib_resources 5.10.0 pyhd8ed1ab_0 conda-forge
ipykernel 6.16.2 pyh210e3f2_0 conda-forge
ipython 7.33.0 py37h89c1867_0 conda-forge
jedi 0.18.1 py37h89c1867_1 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jpeg 9e h166bdaf_2 conda-forge
jsonschema 4.17.0 pyhd8ed1ab_0 conda-forge
jupyter_client 7.4.4 pyhd8ed1ab_0 conda-forge
jupyter_core 4.11.1 py37h89c1867_0 conda-forge
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py37h7cecad7_0 conda-forge
kneaddata 0.12.0 pyhdfd78af_0 bioconda
krb5 1.19.3 h08a2579_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.39 hc81fddc_0 conda-forge
leveldb 0.193 py37h2f20fdb_0 biobakery
libblas 3.9.0 16_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_8 conda-forge
libbrotlidec 1.0.9 h166bdaf_8 conda-forge
libbrotlienc 1.0.9 h166bdaf_8 conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcups 2.3.3 h3e49a29_2 conda-forge
libcurl 7.86.0 h2283fc2_1 conda-forge
libdeflate 1.6 h516909a_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 7.2.0 h69d50b8_2 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.74.1 h7a41b64_0 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libidn2 2.3.4 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libnghttp2 1.47.0 hff17c54_1 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libpng 1.6.38 h753d276_0 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libsqlite 3.39.4 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libtiff 4.2.0 hf544144_3 conda-forge
libunistring 0.9.10 h7f98852_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.2.4 h166bdaf_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
markdown 3.4.1 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.1 py37h540881e_1 conda-forge
matplotlib-base 3.5.3 py37hf395dca_2 conda-forge
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
metaphlan 3.0 pyh5ca1d4c_1 bioconda
mistune 2.0.4 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.4 py37h7cecad7_0 conda-forge
munkres 1.0.7 py_1 bioconda
muscle 5.1 h9f5acd7_1 bioconda
nbclient 0.7.0 pyhd8ed1ab_0 conda-forge
nbconvert 7.2.3 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.2.3 pyhd8ed1ab_0 conda-forge
nbconvert-pandoc 7.2.3 pyhd8ed1ab_0 conda-forge
nbformat 5.7.0 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
networkx 2.7.1 pyhd8ed1ab_0 conda-forge
numpy 1.21.6 py37h976b520_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjdk 17.0.3 hea3dc9f_3 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 3.0.7 h166bdaf_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.5 py37he8f5f7f_0 conda-forge
pandoc 2.19.2 h32600fe_1 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pcre2 10.37 hc3806b6_1 conda-forge
perl 5.32.1 2_h7f98852_perl5 conda-forge
perl-archive-tar 2.40 pl5321hdfd78af_0 bioconda
perl-carp 1.38 pl5321hdfd78af_4 bioconda
perl-common-sense 3.75 pl5321hdfd78af_0 bioconda
perl-compress-raw-bzip2 2.201 pl5321h87f3376_1 bioconda
perl-compress-raw-zlib 2.105 pl5321h87f3376_0 bioconda
perl-encode 3.19 pl5321hec16e2b_1 bioconda
perl-exporter 5.72 pl5321hdfd78af_2 bioconda
perl-exporter-tiny 1.002002 pl5321hdfd78af_0 bioconda
perl-extutils-makemaker 6.66 0 bioconda
perl-io-compress 2.201 pl5321h87f3376_0 bioconda
perl-io-zlib 1.11 pl5321hdfd78af_0 bioconda
perl-json 4.10 pl5321hdfd78af_0 bioconda
perl-json-xs 2.34 pl5321h9f5acd7_5 bioconda
perl-list-moreutils 0.430 pl5321hdfd78af_0 bioconda
perl-list-moreutils-xs 0.430 pl5321hec16e2b_1 bioconda
perl-parent 0.236 pl5321hdfd78af_2 bioconda
perl-pathtools 3.75 pl5321hec16e2b_3 bioconda
perl-scalar-list-utils 1.62 pl5321hec16e2b_1 bioconda
perl-threaded 5.32.1 hdfd78af_1 bioconda
perl-types-serialiser 1.01 pl5321hdfd78af_0 bioconda
pexpect 4.8.0 py37hc8dfbb8_1 conda-forge
pickleshare 0.7.5 py37hc8dfbb8_1002 conda-forge
pillow 8.2.0 py37h4600e1f_1 conda-forge
pip 22.3 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.31 pyha770c72_0 conda-forge
psutil 5.9.3 py37h540881e_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pweave 0.30.3 py_1 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.13.0 pyhd8ed1ab_0 conda-forge
pyopenssl 22.1.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyrsistent 0.18.1 py37h540881e_1 conda-forge
pysam 0.16.0 py37ha9a96c6_0 bioconda
pysocks 1.7.1 py37h89c1867_5 conda-forge
python 3.7.12 hf930737_100_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.16.2 pyhd8ed1ab_0 conda-forge
python_abi 3.7 2_cp37m conda-forge
pytz 2022.6 pyhd8ed1ab_0 conda-forge
pyzmq 24.0.1 py37h0c0c2a8_0 conda-forge
raxml 8.2.12 hec16e2b_4 bioconda
readline 8.1.2 h0f457ee_0 conda-forge
requests 2.28.1 pyhd8ed1ab_1 conda-forge
samtools 0.1.19 h20b1175_10 bioconda
scipy 1.7.3 py37hf2a6cf1_0 conda-forge
setuptools 59.8.0 py37h89c1867_1 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge
sqlite 3.39.4 h4ff8645_0 conda-forge
tbb 2021.6.0 h924138e_1 conda-forge
tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tornado 6.2 py37h540881e_0 conda-forge
traitlets 5.5.0 pyhd8ed1ab_0 conda-forge
trf 4.09.1 hec16e2b_2 bioconda
trimmomatic 0.39 hdfd78af_2 bioconda
typing-extensions 4.4.0 hd8ed1ab_0 conda-forge
typing_extensions 4.4.0 pyha770c72_0 conda-forge
unicodedata2 14.0.0 py37h540881e_1 conda-forge
urllib3 1.26.11 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
wget 1.20.3 ha35d2d1_1 conda-forge
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
xorg-fixesproto 5.0 h7f98852_1002 conda-forge
xorg-inputproto 2.3.2 h7f98852_1002 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge
xorg-libxi 1.7.10 h7f98852_0 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-libxtst 1.2.3 h7f98852_1002 conda-forge
xorg-recordproto 1.14.2 h7f98852_1002 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zip 3.0 h7f98852_1 conda-forge
zipp 3.10.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zstd 1.5.2 h6239696_4 conda-forge

3

Update:

I am trying to systematically explore biobakery_workflow versions available through conda.
Version 3.0.0a4 build py37hb80360e_0 results in the same error as above:
conda create -c biobakery -n biobakery biobakery_workflows=3.0.0a4=py37hb80360e_0
Once the FileNotFoundError: [Errno 2] No such file or directory: 'zip': 'zip' error handled by manually installing zip (conda install zip), I encounter:

ImportError: cannot import name ‘PwebProcessor’ from ‘pweave’

4

With version 3.0.0a4 build pyh41f93f0_0:
conda create -c biobakery -n biobakery biobakery_workflows=3.0.0a4=pyh41f93f0_0
and within the conda environment: biobakery_workflows wmgx_vis --input run_test/output/ --output run_test/vis --project-name TEST
Returns:

Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a4/bin/wmgx_vis.py”, line 28, in
from anadama2 import Workflow
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a4/lib/python3.10/site-packages/anadama2/init.py”, line 83, in
from .workflow import Workflow
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a4/lib/python3.10/site-packages/anadama2/workflow.py”, line 24, in
from . import backends
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a4/lib/python3.10/site-packages/anadama2/backends.py”, line 7, in
import leveldb
ModuleNotFoundError: No module named ‘leveldb’

After fixing the dependency issue with conda install python-leveldb, fixing also
the zip dependency, running biobakery_workflows vis returns:

(Nov 04 12:24:40) [0/2 - 0.00%] **Ready ** Task 0: document
(Nov 04 12:24:40) [0/2 - 0.00%] **Started ** Task 0: document
(Nov 04 12:24:41) [1/2 - 50.00%] **Failed ** Task 0: document
(Nov 04 12:24:41) [2/2 - 100.00%] **Failed ** Task 6: archive
Run Finished
Task 0 failed
Name: document
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a4/lib/python3.10/site-packages/anadama2/runners.py”, line 200, in _run_task_locally
action_func(task)
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a4/lib/python3.10/site-packages/anadama2/document.py”, line 232, in create
from pweave import PwebPandocFormatter, Pweb, PwebProcessor
ImportError: cannot import name ‘PwebProcessor’ from ‘pweave’ (/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a4/lib/python3.10/site-packages/pweave/init.py)

So same problem as with build py37hb80360e_0.

5

With version 3.0.0a7 build pyh0b62126_0, exact same result as for version 3.0.0a4 build pyh41f93f0_0: need to conda install python-leveldb and zip, and then Pweave import error:

(Nov 04 14:20:34) [0/2 - 0.00%] **Ready ** Task 0: document
(Nov 04 14:20:34) [0/2 - 0.00%] **Started ** Task 0: document
(Nov 04 14:20:34) [1/2 - 50.00%] **Failed ** Task 0: document
(Nov 04 14:20:34) [2/2 - 100.00%] **Failed ** Task 6: archive
Run Finished
Task 0 failed
Name: document
Original error:
Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a7/lib/python3.10/site-packages/anadama2/runners.py”, line 200, in _run_task_locally
action_func(task)
File “/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a7/lib/python3.10/site-packages/anadama2/document.py”, line 232, in create
from pweave import PwebPandocFormatter, Pweb, PwebProcessor
ImportError: cannot import name ‘PwebProcessor’ from ‘pweave’ (/home/theoscilicium/miniconda3/envs/biobakery_3.0.0a7/lib/python3.10/site-packages/pweave/init.py)

6

With version 0.15.1 build py27h10287d5_0:
Some progress? There is no unresolved dependency issue and it works directly, only to fail with the following error:

ERROR: Unable to find file: run_test/output/metaphlan2/merged/metaphlan2_taxonomic_profiles.tsv

File description:
A tab-delimited file with samples as columns and relative abundance as rows.
This file contains the merged taxonomic profiles computed by MetaPhlAn2 for all samples.

Apparently, naming of output directories and files from biobakery_workflow wmgx changed at some point between biobakery_workflow versions 0.15 and 3.0.0a4, because I have folders and files named metaphlan and not metaphlan2.

I thus manually moved my files and folders to have metaphlan2 instead of metaphlan in names. Again encountering the zip dependency, and after fixing it:

(Nov 04 14:33:41) [0/2 - 0.00%] **Ready ** Task 0: document
(Nov 04 14:33:41) [0/2 - 0.00%] **Started ** Task 0: document
Exception:
[Errno 2] No such file or directory
Error messages will be included in output document
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/bin/wmgx_vis.py”, line 133, in
workflow.go()
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/anadama2/workflow.py”, line 804, in go
self._handle_finished()
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/anadama2/workflow.py”, line 836, in _handle_finished
raise RunFailed()
anadama2.workflow.RunFailed

Content of the anadama.log file:

2022-11-04 14:33:41,180 LoggerReporter started INFO: Beginning AnADAMA run with 5 tasks.
2022-11-04 14:33:41,181 LoggerReporter started INFO: Workflow description = A workflow for whole metagenome shotgun sequence visualization
2022-11-04 14:33:41,181 LoggerReporter started INFO: Workflow version = 0.1
2022-11-04 14:33:41,181 LoggerReporter started INFO: Workflow configuration options
2022-11-04 14:33:41,181 LoggerReporter started INFO: metadata_continuous =
2022-11-04 14:33:41,181 LoggerReporter started INFO: exclude_workflow_info = False
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid_time_max = None
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid_benchmark = off
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid_partition =
2022-11-04 14:33:41,181 LoggerReporter started INFO: log_level = INFO
2022-11-04 14:33:41,181 LoggerReporter started INFO: dry_run = False
2022-11-04 14:33:41,181 LoggerReporter started INFO: exclude_target = None
2022-11-04 14:33:41,181 LoggerReporter started INFO: until_task = None
2022-11-04 14:33:41,181 LoggerReporter started INFO: input_metadata = None
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid_scratch =
2022-11-04 14:33:41,181 LoggerReporter started INFO: input = run_test/output/
2022-11-04 14:33:41,181 LoggerReporter started INFO: config = None
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid_jobs = 0
2022-11-04 14:33:41,181 LoggerReporter started INFO: introduction_text = The data was run through the standard workflow for whole metagenome shotgun sequencing.
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid_submit_sleep = 5
2022-11-04 14:33:41,181 LoggerReporter started INFO: project_name = TEST
2022-11-04 14:33:41,181 LoggerReporter started INFO: jobs = 1
2022-11-04 14:33:41,181 LoggerReporter started INFO: format = pdf
2022-11-04 14:33:41,181 LoggerReporter started INFO: metadata_exclude =
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid = None
2022-11-04 14:33:41,181 LoggerReporter started INFO: metadata_categorical =
2022-11-04 14:33:41,181 LoggerReporter started INFO: target = None
2022-11-04 14:33:41,181 LoggerReporter started INFO: grid_environment = None
2022-11-04 14:33:41,182 LoggerReporter started INFO: grid_mem_max = None
2022-11-04 14:33:41,182 LoggerReporter started INFO: exclude_task = None
2022-11-04 14:33:41,182 LoggerReporter started INFO: quit_early = False
2022-11-04 14:33:41,182 LoggerReporter started INFO: grid_options = None
2022-11-04 14:33:41,182 LoggerReporter started INFO: output = run_test/vis_2.7
2022-11-04 14:33:41,182 LoggerReporter started INFO: skip_nothing = False
2022-11-04 14:33:41,202 LoggerReporter log_event INFO: task 0, document : ready and waiting for resources
2022-11-04 14:33:41,202 LoggerReporter log_event INFO: task 0, document : starting to run
2022-11-04 14:33:41,989 LoggerReporter task_failed ERROR: task 0, document : Failed! Error message : Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/anadama2/runners.py”, line 200, in _run_task_locally
action_func(task)
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/anadama2/document.py”, line 315, in create
doc.weave(shell=PwebProcessorSpaces)
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/pweave/pweb.py”, line 198, in weave
self.run(shell)
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/pweave/pweb.py”, line 149, in run
runner.run()
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/pweave/processors.py”, line 65, in run
self.executed = list(map(self._runcode, self.parsed))
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/pweave/processors.py”, line 131, in _runcode
chunk[‘content’] = self.loadinline(chunk[‘content’])
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/anadama2/document.py”, line 262, in loadinline
result = self.loadstring(code_str).lstrip().replace(“\n”,“”,1)
File “/home/theoscilicium/miniconda3/envs/biobakery_2.7/lib/python2.7/site-packages/pweave/processors.py”, line 296, in loadstring
exec(compiled, scope)
File “chunk”, line 1, in
NameError: name ‘caption’ is not defined

2022-11-04 14:33:41,989 LoggerReporter task_failed ERROR: task 3, archive : Failed! Error message : Task failed because parent task `0’ failed
2022-11-04 14:33:41,989 LoggerReporter finished ERROR: AnADAMA run finished with errors.

So, again the caption error

7

Summary of my trials:

  • biobakery_workflows wmgx_vis of the latest available (to date) Docker image fails with a ‘caption’ error. pip3 freeze informs that biobakery_workflow version is 3.0.0a6 in this image.

  • conda versions of biobakery_workflows fails for versions:

    • 3.1 (latest) installed with Python 3.6: fails with ‘caption’ error
    • 3.1 (latest) installed with Python 3.7: fails with ‘Pweave import’
    • 3.0.0a7 build pyh0b62126_0 installed with Python 2.7: fails with ‘caption’ error
    • 3.0.0a7 build pyh0b62126_0 installed with Python 3.6: fails with ‘caption’ error
    • 3.0.0a7 build pyh0b62126_0 installed with Python 3.10: fails with ‘Pweave import’
    • 3.0.0a4 build py37hb80360e_0 installed with Python 3.7: fails with ‘Pweave import’
    • 3.0.0a4 build pyh41f93f0_0 installed with Python 3.10: fails with ‘Pweave import’
    • 0.15.1 build py27h10287d5_0 installed with Python 2.7: fails with ‘caption’ error

So, when installed with Python >= 3.7, fails with Pweave import. According to this post, it is because Pweave version needs to be <=v0.25.

Anyway, the problem is at least 2 years old (!) as the latest Docker image is also affected.

I will try with older Docker images, but … well… not encouraging.

8

Update with tests of Docker images:

  • 3.0.0.a.7: fails with caption error
  • 3.0.0.a.6.metaphlanv3.0.7: fails with caption error
9

To the development team:

If you have a working version, could you please provide informations about the environment, such as pip freeze or conda list?

10

Update with tests of Docker images:

  • 3.0.0.a.7: fails with caption error
  • 3.0.0.a.6.metaphlanv3.0.7: fails with caption error
  • 3.0.0.a.6: fails with caption error
  • 3.0.0.alpha.5: fails with caption error
  • 0.15.1 fails with caption error
  • 0.13.2 fails with caption error (after renaming humann files and directories to humann2)
11

BIG NEWS!

I finally managed to get biobakery_workflows vis to run!

The solution that worked for me:

  1. Create conda env, but not installing biobakery_workflows
    mamba create -n biobakery_test python=3.7 pip zip gxx_linux-64 pandoc R
    (If you don’t know mamba, it is a drop-in replacement of conda that is much faster)

  2. Go inside the environment:
    conda activate biobakery_test

  3. Install biobakery_workflows and dependencies with pip:
    pip install numpy matplotlib humann scipy biobakery_workflows
    Warning: it is NOT recommended to mix pip and conda installs. However, installing packages with pip is OK as long as you do not install anything afterwards with conda (i.e: conda first, pip then), as per conda documentation.

  4. As observed previously (see first post), there is a path problem with images included for biobakery_workflow vis. Creating a symlink did the trick for me:
    ln -s /home/theoscilicium/miniconda3/envs/biobakery/images /home/theoscilicium/miniconda3/envs/biobakery/lib/python3.7/site-packages/images

  5. This is probably config-dependent and may vary in your setup, and from what I read cannot be easily deported to a conda env, but at this step I had to install additional packages for latex:
    sudo apt-get install texlive-latex-extra

  6. Run biobakery workflows!
    biobakery_workflows vis --input run_test/biobakery_results/ --output run_test/vis/ --project-name TEST

  7. Exit conda env:
    conda deactivate

Hope this can help other people! Included bellow are further information about the setup.

Best,
Theo

Output of conda list

packages in environment at /home/theoscilicium/miniconda3/envs/biobakery_test:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
_r-mutex 1.0.1 anacondar_1 conda-forge
anadama2 0.10.0 pypi_0 pypi
binutils_impl_linux-64 2.39 h6ceecb4_0 conda-forge
binutils_linux-64 2.39 h5fc0e48_11 conda-forge
biobakery-workflows 3.1 pypi_0 pypi
bwidget 1.9.14 ha770c72_1 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2022.9.24 ha878542_0 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
cloudpickle 2.2.0 pypi_0 pypi
curl 7.86.0 h2283fc2_1 conda-forge
cycler 0.11.0 pypi_0 pypi
expat 2.5.0 h27087fc_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.1 hc2a2eb6_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.38.0 pypi_0 pypi
freetype 2.12.1 hca18f0e_0 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
gcc_impl_linux-64 12.2.0 hcc96c02_19 conda-forge
gcc_linux-64 12.2.0 h4798a0e_11 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gfortran_impl_linux-64 12.2.0 h55be85b_19 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
gsl 2.7 he838d99_0 conda-forge
gxx_impl_linux-64 12.2.0 hcc96c02_19 conda-forge
gxx_linux-64 12.2.0 hb41e900_11 conda-forge
harfbuzz 5.3.0 h418a68e_0 conda-forge
humann 3.6 pypi_0 pypi
icu 70.1 h27087fc_0 conda-forge
importlib-metadata 5.0.0 pypi_0 pypi
jpeg 9e h166bdaf_2 conda-forge
kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.19.3 h08a2579_0 conda-forge
ld_impl_linux-64 2.39 hc81fddc_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
leveldb 0.201 pypi_0 pypi
libblas 3.9.0 16_linux64_openblas conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcurl 7.86.0 h2283fc2_1 conda-forge
libdeflate 1.14 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-devel_linux-64 12.2.0 h3b97bd3_19 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.74.1 h606061b_1 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libnghttp2 1.47.0 hff17c54_1 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libpng 1.6.38 h753d276_0 conda-forge
libsanitizer 12.2.0 h46fd767_19 conda-forge
libsqlite 3.39.4 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-devel_linux-64 12.2.0 h3b97bd3_19 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libtiff 4.4.0 h55922b4_4 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.2.4 h166bdaf_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.10.3 h7463322_0 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
make 4.3 hd18ef5c_1 conda-forge
markdown 3.4.1 pypi_0 pypi
matplotlib 3.5.3 pypi_0 pypi
ncurses 6.3 h27087fc_1 conda-forge
networkx 2.6.3 pypi_0 pypi
numpy 1.21.6 pypi_0 pypi
openssl 3.0.7 h166bdaf_0 conda-forge
packaging 21.3 pypi_0 pypi
pandoc 2.19.2 h32600fe_1 conda-forge
pango 1.50.11 h382ae3d_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
pillow 9.3.0 pypi_0 pypi
pip 22.3.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pweave 0.25 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
python 3.7.12 hf930737_100_cpython conda-forge
python-dateutil 2.8.2 pypi_0 pypi
r 4.2 r42hd8ed1ab_1006 conda-forge
r-base 4.2.2 hb87df5d_1 conda-forge
r-boot 1.3_28 r42hc72bb7e_1 conda-forge
r-class 7.3_20 r42h06615bd_1 conda-forge
r-cluster 2.1.4 r42h8da6f51_0 conda-forge
r-codetools 0.2_18 r42hc72bb7e_1 conda-forge
r-foreign 0.8_83 r42h06615bd_1 conda-forge
r-kernsmooth 2.23_20 r42hd009a43_1 conda-forge
r-lattice 0.20_45 r42h06615bd_1 conda-forge
r-mass 7.3_58.1 r42h06615bd_1 conda-forge
r-matrix 1.5_1 r42h5f7b363_0 conda-forge
r-mgcv 1.8_41 r42h5f7b363_0 conda-forge
r-nlme 3.1_160 r42h8da6f51_0 conda-forge
r-nnet 7.3_18 r42h06615bd_1 conda-forge
r-recommended 4.2 r42hd8ed1ab_1005 conda-forge
r-rpart 4.1.19 r42h06615bd_0 conda-forge
r-spatial 7.3_15 r42h06615bd_1 conda-forge
r-survival 3.4_0 r42h06615bd_1 conda-forge
readline 8.1.2 h0f457ee_0 conda-forge
scipy 1.7.3 pypi_0 pypi
sed 4.8 he412f7d_0 conda-forge
setuptools 65.5.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
sqlite 3.39.4 h4ff8645_0 conda-forge
sysroot_linux-64 2.12 he073ed8_15 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tktable 2.10 hb7b940f_3 conda-forge
typing-extensions 4.4.0 pypi_0 pypi
wheel 0.38.2 pyhd8ed1ab_0 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-libxt 1.2.1 h7f98852_2 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
zip 3.0 h7f98852_1 conda-forge
zipp 3.10.0 pypi_0 pypi
zlib 1.2.13 h166bdaf_4 conda-forge
zstd 1.5.2 h6239696_4 conda-forge

Output of pip freeze

anadama2==0.10.0
biobakery-workflows==3.1
cloudpickle==2.2.0
cycler==0.11.0
fonttools==4.38.0
humann==3.6
importlib-metadata==5.0.0
kiwisolver==1.4.4
leveldb==0.201
Markdown==3.4.1
matplotlib==3.5.3
networkx==2.6.3
numpy==1.21.6
packaging==21.3
Pillow==9.3.0
Pweave==0.25
pyparsing==3.0.9
python-dateutil==2.8.2
scipy==1.7.3
six==1.16.0
typing_extensions==4.4.0
zipp==3.10.0

1 Like
12

Just had the same problem with the image directory. The ln approach seems to work neat.
Thanks!

1 Like
13

A bit of a pain not to be able to edit posts…

At step 3. above, add hclust2 to the list of packages to install with pip if you want to have heatmaps included in the report:
pip install numpy matplotlib humann scipy biobakery_workflows hclust2

14

Thank you! Your post is incredibly helpful! I have followed all the steps you posted but still encountered this error:

2023-12-28 13:22:26,152 LoggerReporter task_failed ERROR: task 2, document : Failed! Error message : Error executing action 0. Original Exception:
Traceback (most recent call last):
File “/panfs/accrepfs.vampire/home/zhuw9/miniforge3/envs/biobakery_vis/lib/python3.7/site-packages/anadama2/runners.py”, line 200, in _run_task_locally
action_func(task)
File “/panfs/accrepfs.vampire/home/zhuw9/miniforge3/envs/biobakery_vis/lib/python3.7/site-packages/anadama2/document.py”, line 321, in create
sh(pandoc_command.format(intermediate_template, temp_report),log_command=True)()
File “/panfs/accrepfs.vampire/home/zhuw9/miniforge3/envs/biobakery_vis/lib/python3.7/site-packages/anadama2/helpers.py”, line 107, in actually_sh
ret = _sh(s, **kwargs)
File “/panfs/accrepfs.vampire/home/zhuw9/miniforge3/envs/biobakery_vis/lib/python3.7/site-packages/anadama2/util/init.py”, line 320, in sh
raise ShellException(proc.returncode, msg.format(cmd, ret[0].decode(‘utf-8’), ret[1].decode(‘utf-8’)))
anadama2.util.ShellException: [Errno 43] Command pandoc /panfs/accrepfs.vampire/data/zhu_group/10272023_Danyvid/Adapter_trimmed/Quality_trimmed/filter/clean_reads/Danyvid_vis1/tmpc10u64yp/wmgx_report.md -o /panfs/accrepfs.vampire/data/zhu_group/10272023_Danyvid/Adapter_trimmed/Quality_trimmed/filter/clean_reads/Danyvid_vis1/tmpc10u64yp/wmgx_report..pdf --variable=linkcolor:Blue --variable=toccolor:Blue --pdf-engine=pdflatex --standalone -V geometry:left=1in -V geometry:right=1in -V geometry:top=1in -V geometry:bottom=1in --toc' failed. Out: Err: Error producing PDF. ! LaTeX Error: File iftex.sty’ not found.

Type X to quit or to proceed,
or enter new name. (Default extension: sty)

Enter file name:
! Emergency stop.
<read *>

l.10 \ifPDFTeX

2023-12-28 13:22:26,152 LoggerReporter task_failed ERROR: task 4, archive : Failed! Error message : Task failed because parent task `2’ failed

Can you advise what is going on?

Thanks!

Best,

Wenhan

15

Dear Wenhan,

I am not in any way related to the team behind biobakery / biobakery workflows.

As of 2024; given the absence of answer from the development / maintenance team, the lack of official documentation and guidelines on how to solve these problems or just any feedback that these posts were helpful in any way, I would not recommend spending time trying to fix biobakery_workflows. I would generally not recommend using this biobakery_workflows, but instead use the individual tools that make up biobakery_workflows, for example using a widely used workflow manager such as Snakemake.
The time you will spend learning to use such a workflow manager and reproducing biobakery_workflows will be on the same order as the time you will spend if you choose to go down the rabbit hole of fixing the original biobakery_workflows, but you will have something that will be much easier to debug/use, you will have learnt how to use a new tool that is a standard in the bioinformatics community, and it will save you some headaches.

Just my opinion after 1+ year of struggling with biobakery.