Troubles with genomeassembler workflow

[BrunoBertolasi]

Hello to all,

I am new to nf and genomeassembler and I am having some difficult to understand errors and causes of process failures. I have installed and tested successfully the genomeassembler workflow. I then tried to use genomeassembler for a full genome assembly of my real data, but the process stopped after having completed only two tasks.

My PC configuration includes a 15 core (11th Gen Intel(R) Core(TM) i7-11700K @ 3.60GHz) with 128 Gb of RAM.
The starting file, containing the reads to assemble in format .gz, is 12.9 Gb.
I used this command:

nextflow run nf-core/genomeassembler \
 -profile singularity \
 -c /home/bruno/Desktop/test.config \
 --input /home/bruno/Desktop/samplesheet.csv \
 --outdir /home/bruno/Desktop/Acer_opalus_assembly \
 --ont true \
 --porechop true \
 --hifi false \
 --short_reads false \
 --polish_pilon false \
 --polish_medaka true \
 --lift_annotations false

The test.config file contains the line:
singularity.pullTimeout = "888h"

The workflow started correctly but, after about two hours, I got this message:

-[nf-core/genomeassembler] Pipeline completed with errors-
ERROR ~ Execution aborted due to an unexpected error

 -- Check script '.nextflow/assets/nf-core/genomeassembler/./workflows/../subworkflows/local/scaffolding/./ragtag/../../mapping/map_to_assembly/../../../../modules/nf-core/minimap2/align/main.nf' at line: 7 or see '.nextflow.log' file for more details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details

For the sake of precision I report the lines of the logfile in which an error appears:

Mar-03 17:49:55.033 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[id: 1; name: NFCORE_GENOMEASSEMBLER:GENOMEASSEMBLER:ONT:PREPARE_ONT:CHOP:PORECHOP (Acer_opalus_full); status: RUNNING; exit: -; error: -; workDir: /home/bruno/work/3b/5be87ce8f051b93db2498503c3284a]
Mar-03 17:53:32.712 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: NFCORE_GENOMEASSEMBLER:GENOMEASSEMBLER:ONT:PREPARE_ONT:CHOP:PORECHOP (Acer_opalus_full); status: COMPLETED; exit: 0; error: -; workDir: /home/bruno/work/3b/5be87ce8f051b93db2498503c3284a]
Mar-03 17:53:32.713 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Mar-03 17:53:32.728 [Actor Thread 1] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=https://depot.galaxyproject.org/singularity/nanoq:0.10.0--h031d066_2; path=/home/bruno/work/singularity/depot.galaxyproject.org-singularity-nanoq-0.10.0--h031d066_2.img
Mar-03 17:53:32.729 [Actor Thread 1] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=https://depot.galaxyproject.org/singularity/kmer-jellyfish:2.3.1--h4ac6f70_0; path=/home/bruno/work/singularity/depot.galaxyproject.org-singularity-kmer-jellyfish-2.3.1--h4ac6f70_0.img
Mar-03 17:53:32.730 [Actor Thread 3] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
 task: name=NFCORE_GENOMEASSEMBLER:GENOMEASSEMBLER:ASSEMBLE:MAP_TO_REF:ALIGN (1); work-dir=null
 error [nextflow.exception.ProcessUnrecoverableException]: Path value cannot be null
Mar-03 17:53:32.731 [Actor Thread 1] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/fa/fa1c1e961de38d24cf36c424a8f4a9920ddd07b63fdb4cfa51c9e3a593c3c979/data; path=/home/bruno/work/singularity/community-cr-prod.seqera.io-docker-registry-v2-blobs-sha256-fa-fa1c1e961de38d24cf36c424a8f4a9920ddd07b63fdb4cfa51c9e3a593c3c979-data.img
Mar-03 17:53:32.736 [TaskFinalizer-1] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=48; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Mar-03 17:53:32.742 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Mar-03 17:53:32.742 [Task submitter] INFO  nextflow.Session - [50/bdb9cd] Submitted process > NFCORE_GENOMEASSEMBLER:GENOMEASSEMBLER:ONT:PREPARE_ONT:RUN_NANOQ:NANOQ (Acer_opalus_full)
Mar-03 17:53:32.744 [Actor Thread 3] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error

Anyone has an idea about what happened? Is there any modification in the .config file that could solve the problem? I suspect the problem is something related to low resources, but I haven't found any exit code that indicates that.

Thanks in advance for any help!

Bruno

[nschan]

Hello Bruno,

it is a bit unclear to me which stage throws the error. The nextflow error seems to indicate that this is in the scaffolding/ragtag workflow, but from the log it seems to occur close to the start of nanoq, which would be much earlier in the pipeline. Could you clarify, or maybe attach the full .nextflow.log of the failing run?

Best
Niklas

[BrunoBertolasi]

Hello Niklas,
Thanks for your answer. Here you can find the full .nextflow.log
Best

Bruno

nextflow.log

[nschan]

Are you providing a reference in your samplesheet? If not, can you try with adding --use_ref false (this initialises as true).
Generally given that this pipeline has quite some parameters I would recommend creating a config that looks like below and save this as e.g. bruno.config:

params {
 input = "/home/bruno/Desktop/samplesheet.csv"
 outdir = "/home/bruno/Desktop/Acer_opalus_assembly"
 ont = true 
 porechop = true 
 hifi = false 
 short_reads = false 
 polish_pilon = false 
 polish_medaka = true 
 lift_annotations = false
 use_ref = false
}

and start your run using

nextflow run nf-core/genomeassembler \
 -profile singularity \
 -c /home/bruno/Desktop/test.config,/home/bruno/Desktop/bruno.config

[BrunoBertolasi]

Hello Niklas,
Finally I tried the run taking into the account your suggests. This is the content of the configuration file I used:

process {
  resourceLimits = [
    cpus: 16,
    memory: 125.GB,
    time: 888.h
  ]
}

params {
 input = "/home/bruno/Desktop/samplesheet.csv"
 outdir = "/home/bruno/Desktop/Acer_opalus_assembly"
 ont = true 
 porechop = true 
 hifi = false 
 short_reads = false 
 polish_pilon = false 
 polish_medaka = true 
 scaffold_links = true
 lift_annotations = false
 use_ref = false
}

As you can see I set the resource limits in order to optimize the use of my PC and avoid failure of the process. Despite this the pipeline failed and so I attach the nextflow.log.
I am afraid that the problem is related to insufficient power of the 16 processor. Is that correct? If this is the case is there any possibility to solve the problem? My genome is expected to be about 0,7 Gbp of size and the file .gz containing the reads to assemble is of 12,9 GB.
Thanks for your help.

Bruno

[nschan]

Hello @BrunoBertolasi,

the error you are receiving is a Segmentation fault from flye. I am not sure why Flye segfaults, I suggest you open an issue there to get feedback on your specific problem.

There are several issues there with similar errors, this one in particular seems to happen at a similar stage mikolmogorov/Flye#27 . They suggest changing the --min-overlap value for flye. You can do this using --flye_args (https://nf-co.re/genomeassembler/dev/parameters/#assembly-options) to pass additional arguments to flye.

[BrunoBertolasi]

Hello Niklas,
I tried to increase the --min-overlap value to 3000 incorporating this in the launch command:

nextflow run nf-core/genomeassembler \
-profile singularity \
-c /home/bruno/Desktop/test.config \
--flye_args = min-overlap 3000 \ 
-resume 19d6ab89-1c34-4849-97b1-6bfe76c0af61

By the way the process stopped complaining that "nf-core pipelines do not accept positional arguments The positional argument min-overlap has been detected." Could you explain me which is the right syntax to pass the additional --flye_args?

Furthermore I tried to resume the previous run but the process seems to start over even if I specified the session ID from which to start again. Any suggest?
Thanks for your time.

Bruno

[nschan]

Hi @BrunoBertolasi ,

you need to quote the arguments that should be passed to flye:

nextflow run nf-core/genomeassembler \
-profile singularity \
-c /home/bruno/Desktop/test.config \
--flye_args "--min-overlap 3000" 

Which process starts over? All of the pipeline, or only certain steps?

[BrunoBertolasi]

Hello Niklas,
I used the command you suggested:

nextflow run nf-core/genomeassembler \
-profile singularity \
-c /home/bruno/Desktop/test.config \
--flye_args "--min-overlap 3000" 
-resume 19d6ab89-1c34-4849-97b1-6bfe76c0af61 

The process started without problem, but stopped complaining "...Failed to pull singularity image...". He suggested me to: "Try and increase singularity.pullTimeout in the config (current is "37d")". By the way, in my configuration file, I have already set this parameter to 888.h (37 days): what else should I do?

pullTimeout: 888.h
singularity.pullTimeout = "888h"

process {
 resourceLimits = [
   cpus: 16,
   memory: 125.GB,
   time: 888.h
 ]
}

params {
input = "/home/bruno/Desktop/samplesheet.csv"
outdir = "/home/bruno/Desktop/Acer_opalus_assembly"
ont = true 
porechop = true 
hifi = false 
short_reads = false 
polish_pilon = false 
polish_medaka = true 
scaffold_links = true
lift_annotations = false
use_ref = false
}

I attach the log file nextflow.log for details.
Even if I used the option -resume, the program start over from the beginning of the pipeline...that sounds strange...
Thanks for your help.

Bruno

[nschan]

Hello @BrunoBertolasi ,

the jellyfish container problem is a bug, (see #72). I am very sorry for this, if you want to fix this, you need to edit the modules/local/jellyfish/stats/main.nf and modules/local/jellyfish/histo/main.nf in line 5 and replace mer-jellyfish with kmer-jellyfish. I will prepare a bugfix release this week that fixes this.

[nschan]

Hello @BrunoBertolasi I have now released version 1.0.1 which should solve the singularity problem, please let me know if you encounter any problems with the new pipeline version.

[BrunoBertolasi]

Hello Niklas,
Thanks for your quick reply and interest. I tried to update the workflow with the command:
nextflow pull nf-core/genomeassembler
but I received this message:
nf-core/genomeassembler contains uncommitted changes -- cannot pull from repository
What is the command to update the workflow to your latest version in order to get the changes?
Thanks.

Bruno

[d4straub]

If there are uncommitted changes then you need to delete the nextflow cache and pull again.

[BrunoBertolasi]

Do you mean to delete all the cache? Could you please specify the command? (nextflow clean...)
Thanks and sorry for my trivial questions ;-)

Bruno

[nschan]

The pipeline cache is usually in ~/.nextflow/assets and it is where nextflow stores the pipelines. If you made any changes (e.g. changed the jellyfish singularity image urls) in the cache without using git commit afterwards, nextflow cannot pull new pipeline versions. If you delete ~/.nextflow/assets/nf-core/genomeassembler nextflow will simply download the latest version when you try to run the pipeleine.

[BrunoBertolasi]

Hello Niklas and Daniel,
I was finally able to install the newest version of the pipeline and it seems overcame the jellyfish problems. Unluckily it stopped after awhile with a broken pipe error. I attach the log file for details nextflow.log. Any idea about what happened? Can I resume the run with better settings?
Thanks for your time.

Bruno

[d4straub]

Based on the log file I think the problem is

Caused by: java.lang.IllegalStateException: Failed to pull singularity image
  command: singularity pull  --name depot.galaxyproject.org-singularity-mer-jellyfish-2.3.1--h4ac6f70_0.img.pulling.1742209104190 https://depot.galaxyproject.org/singularity/mer-jellyfish:2.3.1--h4ac6f70_0 > /dev/null

you can attempt to manually download the image with singularity pull --name depot.galaxyproject.org-singularity-mer-jellyfish-2.3.1--h4ac6f70_0.img https://depot.galaxyproject.org/singularity/mer-jellyfish:2.3.1--h4ac6f70_0 into work/singularity or whereever you store those.

[nschan]

Hi,

that is still the error raised by the broken singularity image, which should have been fixed in 1.0.1. I am not sure if you are running the correct version, maybe you can try adding -r 1.0.1 to your nextflow command?
It also seems you are running from dev, but I think it should run from master since the release of 1.0.0? I am not sure what causes this.

[BrunoBertolasi]

Hi,
I am sure I have used the 1.0.1 version because I deleted the old genomeassembler folder before to download the newest version you linked me. What does it mean run form master?
Thanks again for your help!

Bruno

[nschan]

Can you try with:

nextflow run nf-core/genomeassembler \
-profile singularity \
-c /home/bruno/Desktop/test.config \
--flye_args "--min-overlap 3000" \
-r 1.0.1

[BrunoBertolasi]

Hello Niklas,
I re-run the pipeline with your last command and the process went significantly further and completed several new tasks but finally stopped at a certain point complaining a segmentation fault error...This is the last log file nextflow.log. How can I solve this?
Thanks for your time!

Bruno

[nschan]

Hello Bruno,

This is a problem with flye, and beyond what I can help with. I would suggest you ask the flye developers about this.