Cactus-minigraph: cactus-graphmap-split error #1837
Description
Hi
I run the job created by this code:
module load cactus/2.8.2 cactus-graphmap-split ./jobstore oryza.txt NIP.sv.gfa NIP.paf --outDir chroms --reference NIP --maxCores 2
The error:
`[2025-11-20T22:21:48+0800] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil
[2025-11-20T22:21:48+0800] [MainThread] [W] [toil.lib.humanize] Deprecated toil method. Please use "toil.lib.conversions.human2bytes()" instead."
[2025-11-20T22:21:48+0800] [MainThread] [W] [toil.lib.humanize] Deprecated toil method. Please use "toil.lib.conversions.human2bytes()" instead."
[2025-11-20T22:21:48+0800] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /public/home/software/opt/bio/software/cactus/2.8.2//bin/cactus-graphmap-split ./jobstore oryza.txt NIP.sv.gfa NIP.paf --outDir chroms --reference NIP --maxCores 2
[2025-11-20T22:21:48+0800] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: 2126683
[2025-11-20T22:21:48+0800] [MainThread] [I] [toil.statsAndLogging] Importing NIP.sv.gfa
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil.statsAndLogging] Importing NIP.paf
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil.statsAndLogging] Importing file:///public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/fasta/NIP-T2T.fa
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil.statsAndLogging] Importing file:///public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/fasta/9311_a2.fa
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil.statsAndLogging] Importing file:///public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/fasta/bl_a2.fa
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil.statsAndLogging] Importing file:///public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/fasta/dl_a2.fa
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil.statsAndLogging] Importing file:///public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/cactus/NIP.sv.gfa.fa
[2025-11-20T22:22:00+0800] [MainThread] [W] [toil.batchSystems.singleMachine] Not enough cores! User limited to 2 but we only have 1.
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host gpu02.
[2025-11-20T22:22:00+0800] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.
[2025-11-20T22:22:01+0800] [MainThread] [I] [toil.leader] Issued job 'graphmap_split_workflow' kind-graphmap_split_workflow/instance-yaz2qk1u v1 with job batch system ID: 1 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:22:03+0800] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run
[2025-11-20T22:22:13+0800] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-4_30vto_ v1 with job batch system ID: 2 and disk: 2.5 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:22:13+0800] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-x5ugdoyc v1 with job batch system ID: 3 and disk: 2.6 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:22:13+0800] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-6y6ga794 v1 with job batch system ID: 4 and disk: 2.6 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:22:13+0800] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-p723lbjv v1 with job batch system ID: 5 and disk: 3.3 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:22:13+0800] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-2x_rch8s v1 with job batch system ID: 6 and disk: 2.4 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:22:19+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:19.473334: Running the command: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/dccf/f407/tmpgx4zy31i/dl.fa dl -p"
[2025-11-20T22:22:22+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:22.613617: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/dccf/f407/tmpgx4zy31i/dl.fa dl -p" in 3.1082 seconds
[2025-11-20T22:22:22+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:22.683819: Running the command: "cactus_analyseAssembly dl.sanitized.fa"
[2025-11-20T22:22:25+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:25.358281: Successfully ran: "cactus_analyseAssembly dl.sanitized.fa" in 2.6326 seconds
[2025-11-20T22:22:26+0800] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 11-20-2025 22:22:26: Assembly stats for dl: Input-sample: dl.sanitized.fa Total-sequences: 12 Total-length: 383957143 Proportion-repeat-masked: 0.000002 ProportionNs: 0.000002 Total-Ns: 800 N50: 31257431 Median-sequence-length: 31257431 Max-sequence-length: 44089240 Min-sequence-length: 24324582
[2025-11-20T22:22:35+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:35.594194: Running the command: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/b22a/6741/tmpuri8ojf5/NIP.fa NIP -p"
[2025-11-20T22:22:38+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:38.852460: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/b22a/6741/tmpuri8ojf5/NIP.fa NIP -p" in 3.2135 seconds
[2025-11-20T22:22:38+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:38.914923: Running the command: "cactus_analyseAssembly NIP.sanitized.fa"
[2025-11-20T22:22:41+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:41.586314: Successfully ran: "cactus_analyseAssembly NIP.sanitized.fa" in 2.6415 seconds
[2025-11-20T22:22:42+0800] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 11-20-2025 22:22:42: Assembly stats for NIP: Input-sample: NIP.sanitized.fa Total-sequences: 12 Total-length: 385710679 Proportion-repeat-masked: 0.000209 ProportionNs: 0.000000 Total-Ns: 0 N50: 31111469 Median-sequence-length: 31111469 Max-sequence-length: 43929697 Min-sequence-length: 23931887
[2025-11-20T22:22:51+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:51.191165: Running the command: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/6b0a/f869/tmp2czxc1l3/9311.fa 9311 -p"
[2025-11-20T22:22:54+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:54.417922: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/6b0a/f869/tmp2czxc1l3/9311.fa 9311 -p" in 3.177 seconds
[2025-11-20T22:22:54+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:54.482462: Running the command: "cactus_analyseAssembly 9311.sanitized.fa"
[2025-11-20T22:22:57+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:22:57.194059: Successfully ran: "cactus_analyseAssembly 9311.sanitized.fa" in 2.6775 seconds
[2025-11-20T22:22:57+0800] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 11-20-2025 22:22:57: Assembly stats for 9311: Input-sample: 9311.sanitized.fa Total-sequences: 12 Total-length: 391333391 Proportion-repeat-masked: 0.000002 ProportionNs: 0.000002 Total-Ns: 900 N50: 32325236 Median-sequence-length: 32325236 Max-sequence-length: 44379266 Min-sequence-length: 24497711
[2025-11-20T22:23:09+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:09.092906: Running the command: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/18aa/106a/tmp8rwarac_/MINIGRAPH.fa MINIGRAPH -p"
[2025-11-20T22:23:13+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:13.210306: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/18aa/106a/tmp8rwarac_/MINIGRAPH.fa MINIGRAPH -p" in 4.0854 seconds
[2025-11-20T22:23:13+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:13.280241: Running the command: "cactus_analyseAssembly MINIGRAPH.sanitized.fa"
[2025-11-20T22:23:16+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:16.722785: Successfully ran: "cactus_analyseAssembly MINIGRAPH.sanitized.fa" in 3.4173 seconds
[2025-11-20T22:23:17+0800] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 11-20-2025 22:23:17: Assembly stats for MINIGRAPH: Input-sample: MINIGRAPH.sanitized.fa Total-sequences: 134467 Total-length: 489106044 Proportion-repeat-masked: 0.000000 ProportionNs: 0.000000 Total-Ns: 0 N50: 13329 Median-sequence-length: 537 Max-sequence-length: 4320062 Min-sequence-length: 1
[2025-11-20T22:23:30+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:30.205395: Running the command: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/2daf/cad8/tmp0vg4uo6r/bl.fa bl -p"
[2025-11-20T22:23:33+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:33.295078: Successfully ran: "cactus_sanitizeFastaHeaders /tmp/toilwf-92830bc327ca547aa72da353a57003c4/2daf/cad8/tmp0vg4uo6r/bl.fa bl -p" in 3.0247 seconds
[2025-11-20T22:23:33+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:33.370683: Running the command: "cactus_analyseAssembly bl.sanitized.fa"
[2025-11-20T22:23:35+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:35.992768: Successfully ran: "cactus_analyseAssembly bl.sanitized.fa" in 2.5716 seconds
[2025-11-20T22:23:36+0800] [Thread-4 ] [I] [toil.statsAndLogging] Got message from job at time 11-20-2025 22:23:36: Assembly stats for bl: Input-sample: bl.sanitized.fa Total-sequences: 12 Total-length: 374395146 Proportion-repeat-masked: 0.000003 ProportionNs: 0.000003 Total-Ns: 1100 N50: 30085505 Median-sequence-length: 30085505 Max-sequence-length: 43074264 Min-sequence-length: 22892172
[2025-11-20T22:23:36+0800] [MainThread] [I] [toil.leader] Issued job 'detect_ref_contigs' kind-detect_ref_contigs/instance-pbkzanzp v1 with job batch system ID: 7 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:23:50+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:50.508738: Running the command: "samtools faidx /tmp/toilwf-92830bc327ca547aa72da353a57003c4/77f1/aaeb/tmpl_21qom7/NIP.fa"
[2025-11-20T22:23:52+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:23:52.816397: Successfully ran: "samtools faidx /tmp/toilwf-92830bc327ca547aa72da353a57003c4/77f1/aaeb/tmpl_21qom7/NIP.fa" in 2.2685 seconds
[2025-11-20T22:23:52+0800] [MainThread] [I] [toil-rt] auto-detected --refContigs Chr01 Chr02 Chr03 Chr04 Chr05 Chr06 Chr07 Chr08 Chr09 Chr10 Chr11 Chr12
[2025-11-20T22:23:53+0800] [MainThread] [I] [toil.leader] Issued job 'filter_paf' kind-filter_paf/instance-x1esoowk v1 with job batch system ID: 8 and disk: 390.5 Mi, memory: 4.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:24:03+0800] [MainThread] [I] [toil-rt] Running PAF filter with minBlock=250000 minMAPQ=5 minIdentity=0.5
[2025-11-20T22:24:03+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:24:03.628484: Running the command: "gaffilter /tmp/toilwf-92830bc327ca547aa72da353a57003c4/4c37/5c5a/tmp1k6e3az7/mg.paf.filter -p -r 5.0 -m 0.0 -b 250000 -q 5 -i 0.5"
[2025-11-20T22:24:04+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:24:04.112378: Successfully ran: "gaffilter /tmp/toilwf-92830bc327ca547aa72da353a57003c4/4c37/5c5a/tmp1k6e3az7/mg.paf.filter -p -r 5.0 -m 0.0 -b 250000 -q 5 -i 0.5" in 0.446 seconds with job-memory 4.0 Gi
[2025-11-20T22:24:04+0800] [MainThread] [I] [toil.leader] Issued job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v1 with job batch system ID: 9 and disk: 2.5 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:24:11+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:24:11.569265: Running the command: "rgfa-split -p /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.paf -b /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split_ -Q 2 -a AMBIGUOUS -L /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split.log -n 0.75 -n 0.5 -n 0.25 -T 100000 -T 1000000 -g /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.gfa -G -r id=NIP| -A 5"
[2025-11-20T22:24:14+0800] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job toil_worker split_gfa file:/public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/cactus/jobstore kind-split_gfa/instance-hjtsg3ew.
[2025-11-20T22:24:14+0800] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'split_gfa' kind-split_gfa/instance-hjtsg3ew v1
Exit reason: None
[2025-11-20T22:24:14+0800] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'split_gfa' kind-split_gfa/instance-hjtsg3ew v2
[2025-11-20T22:24:14+0800] [MainThread] [W] [toil.leader] Log from job "kind-split_gfa/instance-hjtsg3ew" follows:
=========>
[2025-11-20T22:24:08+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2025-11-20T22:24:08+0800] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host gpu02.
[2025-11-20T22:24:08+0800] [MainThread] [I] [toil.worker] Working on job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v1
[2025-11-20T22:24:10+0800] [MainThread] [I] [toil.worker] Loaded body Job('split_gfa' kind-split_gfa/instance-hjtsg3ew v1) from description 'split_gfa' kind-split_gfa/instance-hjtsg3ew v1
[2025-11-20T22:24:11+0800] [MainThread] [I] [cactus.shared.common] Running the command ['rgfa-split', '-p', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.paf', '-b', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split', '-Q', '2', '-a', 'AMBIGUOUS', '-L', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split.log', '-n', '0.75', '-n', '0.5', '-n', '0.25', '-T', '100000', '-T', '1000000', '-g', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.gfa', '-G', '-r', 'id=NIP|', '-A', '5']
[2025-11-20T22:24:11+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:24:11.569265: Running the command: "rgfa-split -p /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.paf -b /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split_ -Q 2 -a AMBIGUOUS -L /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split.log -n 0.75 -n 0.5 -n 0.25 -T 100000 -T 1000000 -g /tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.gfa -G -r id=NIP| -A 5"
[2025-11-20T22:24:14+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
[2025-11-20T22:24:14+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-8061ce933b0e4bbc9f8beb1498e51c75/NIP.sv.gfa' to path '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.gfa'
[2025-11-20T22:24:14+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-filter_paf/instance-x1esoowk/file-d9f18687c3824c58b2f687aa7cc1c99d/mg.paf.filter.overlap' to path '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.paf'
Traceback (most recent call last):
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/worker.py", line 409, in workerScript
job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2845, in _runner
returnValues = self._run(jobGraph=None, fileStore=fileStore)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2761, in _run
return self.run(fileStore)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2990, in run
rValue = userFunction(*((self,) + tuple(self._args)), **self.kwargs)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/refmap/cactus_graphmap_split.py", line 417, in split_gfa
cactus_call(parameters=cmd, work_dir=work_dir, job_memory=job.memory)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/shared/common.py", line 912, in cactus_call
raise RuntimeError("{}Command {} signaled {}: {}".format(sigill_msg, call, signal.Signals(-process.returncode).name, out))
RuntimeError: Command ['rgfa-split', '-p', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.paf', '-b', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split', '-Q', '2', '-a', 'AMBIGUOUS', '-L', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/split.log', '-n', '0.75', '-n', '0.5', '-n', '0.25', '-T', '100000', '-T', '1000000', '-g', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/c77d/1ca2/tmpsckbciv4/mg.gfa', '-G', '-r', 'id=NIP|', '-A', '5'] signaled SIGABRT: stderr=terminate called after throwing an instance of 'std::invalid_argument'
what(): stol
[2025-11-20T22:24:14+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host gpu02
<=========
[2025-11-20T22:24:14+0800] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v2 with ID kind-split_gfa/instance-hjtsg3ew to 1
[2025-11-20T22:24:14+0800] [MainThread] [I] [toil.leader] Issued job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v3 with job batch system ID: 10 and disk: 2.5 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2025-11-20T22:24:21+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:24:21.800163: Running the command: "rgfa-split -p /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf -b /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split_ -Q 2 -a AMBIGUOUS -L /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split.log -n 0.75 -n 0.5 -n 0.25 -T 100000 -T 1000000 -g /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa -G -r id=NIP| -A 5"
[2025-11-20T22:24:24+0800] [Thread-1 ] [E] [toil.batchSystems.singleMachine] Got exit code 1 (indicating failure) from job toil_worker split_gfa file:/public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/cactus/jobstore kind-split_gfa/instance-hjtsg3ew.
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.leader] Job failed with exit value 1: 'split_gfa' kind-split_gfa/instance-hjtsg3ew v3
Exit reason: None
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.leader] The job seems to have left a log file, indicating failure: 'split_gfa' kind-split_gfa/instance-hjtsg3ew v5
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.leader] Log from job "kind-split_gfa/instance-hjtsg3ew" follows:
=========>
[2025-11-20T22:24:18+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2025-11-20T22:24:18+0800] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host gpu02.
[2025-11-20T22:24:18+0800] [MainThread] [I] [toil.worker] Working on job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v4
[2025-11-20T22:24:20+0800] [MainThread] [I] [toil.worker] Loaded body Job('split_gfa' kind-split_gfa/instance-hjtsg3ew v4) from description 'split_gfa' kind-split_gfa/instance-hjtsg3ew v4
[2025-11-20T22:24:21+0800] [MainThread] [I] [cactus.shared.common] Running the command ['rgfa-split', '-p', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf', '-b', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split', '-Q', '2', '-a', 'AMBIGUOUS', '-L', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split.log', '-n', '0.75', '-n', '0.5', '-n', '0.25', '-T', '100000', '-T', '1000000', '-g', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa', '-G', '-r', 'id=NIP|', '-A', '5']
[2025-11-20T22:24:21+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:24:21.800163: Running the command: "rgfa-split -p /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf -b /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split_ -Q 2 -a AMBIGUOUS -L /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split.log -n 0.75 -n 0.5 -n 0.25 -T 100000 -T 1000000 -g /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa -G -r id=NIP| -A 5"
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-8061ce933b0e4bbc9f8beb1498e51c75/NIP.sv.gfa' to path '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa'
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-filter_paf/instance-x1esoowk/file-d9f18687c3824c58b2f687aa7cc1c99d/mg.paf.filter.overlap' to path '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf'
Traceback (most recent call last):
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/worker.py", line 409, in workerScript
job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2845, in _runner
returnValues = self._run(jobGraph=None, fileStore=fileStore)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2761, in _run
return self.run(fileStore)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2990, in run
rValue = userFunction(*((self,) + tuple(self._args)), **self.kwargs)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/refmap/cactus_graphmap_split.py", line 417, in split_gfa
cactus_call(parameters=cmd, work_dir=work_dir, job_memory=job.memory)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/shared/common.py", line 912, in cactus_call
raise RuntimeError("{}Command {} signaled {}: {}".format(sigill_msg, call, signal.Signals(-process.returncode).name, out))
RuntimeError: Command ['rgfa-split', '-p', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf', '-b', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split', '-Q', '2', '-a', 'AMBIGUOUS', '-L', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split.log', '-n', '0.75', '-n', '0.5', '-n', '0.25', '-T', '100000', '-T', '1000000', '-g', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa', '-G', '-r', 'id=NIP|', '-A', '5'] signaled SIGABRT: stderr=terminate called after throwing an instance of 'std::invalid_argument'
what(): stol
[2025-11-20T22:24:24+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host gpu02
<=========
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.job] Due to failure we are reducing the remaining try count of job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v5 with ID kind-split_gfa/instance-hjtsg3ew to 0
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.leader] Job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v6 is completely failed
[2025-11-20T22:24:32+0800] [MainThread] [I] [toil.leader] Finished toil run with 3 failed jobs.
[2025-11-20T22:24:32+0800] [MainThread] [I] [toil.leader] Failed jobs at end of the run: 'sanitize_fasta_headers' kind-graphmap_split_workflow/instance-yaz2qk1u v6 'filter_paf' kind-filter_paf/instance-x1esoowk v2 'split_gfa' kind-split_gfa/instance-hjtsg3ew v6
Workflow Progress 100%|████████████████████████████████████████| 10/10 (2 failures) [02:30<00:00, 0.07 jobs/s]
[2025-11-20T22:24:32+0800] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2025-11-20T22:24:32+0800] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
Traceback (most recent call last):
File "/public/home/software/opt/bio/software/cactus/2.8.2//bin/cactus-graphmap-split", line 8, in
sys.exit(main())
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/refmap/cactus_graphmap_split.py", line 92, in main
cactus_graphmap_split(options)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/refmap/cactus_graphmap_split.py", line 170, in cactus_graphmap_split
wf_output = toil.start(Job.wrapJobFn(graphmap_split_workflow, options, config, input_seq_id_map, input_name_map,
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/common.py", line 915, in start
return self.runMainLoop(rootJobDescription)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/common.py", line 1391, in runMainLoop
return Leader(config=self.config,
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/leader.py", line 295, in run
raise FailedJobsException(self.jobStore, failed_jobs, exit_code=self.recommended_fail_exit_code)
toil.exceptions.FailedJobsException: The job store '/public/home/zjwang/de_novo_seq/3_parent/pan-genome/test/cactus/jobstore' contains 3 failed jobs: 'sanitize_fasta_headers' kind-graphmap_split_workflow/instance-yaz2qk1u v6, 'filter_paf' kind-filter_paf/instance-x1esoowk v2, 'split_gfa' kind-split_gfa/instance-hjtsg3ew v6
Log from job "'split_gfa' kind-split_gfa/instance-hjtsg3ew v6" follows:
=========>
[2025-11-20T22:24:18+0800] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2025-11-20T22:24:18+0800] [MainThread] [I] [toil] Running Toil version 6.1.0-3f9cba3766e52866ea80d0934498f8c8f3129c3f on host gpu02.
[2025-11-20T22:24:18+0800] [MainThread] [I] [toil.worker] Working on job 'split_gfa' kind-split_gfa/instance-hjtsg3ew v4
[2025-11-20T22:24:20+0800] [MainThread] [I] [toil.worker] Loaded body Job('split_gfa' kind-split_gfa/instance-hjtsg3ew v4) from description 'split_gfa' kind-split_gfa/instance-hjtsg3ew v4
[2025-11-20T22:24:21+0800] [MainThread] [I] [cactus.shared.common] Running the command ['rgfa-split', '-p', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf', '-b', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split', '-Q', '2', '-a', 'AMBIGUOUS', '-L', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split.log', '-n', '0.75', '-n', '0.5', '-n', '0.25', '-T', '100000', '-T', '1000000', '-g', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa', '-G', '-r', 'id=NIP|', '-A', '5']
[2025-11-20T22:24:21+0800] [MainThread] [I] [toil-rt] 2025-11-20 22:24:21.800163: Running the command: "rgfa-split -p /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf -b /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split -Q 2 -a AMBIGUOUS -L /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split.log -n 0.75 -n 0.5 -n 0.25 -T 100000 -T 1000000 -g /tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa -G -r id=NIP| -A 5"
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/no-job/file-8061ce933b0e4bbc9f8beb1498e51c75/NIP.sv.gfa' to path '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa'
[2025-11-20T22:24:24+0800] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-filter_paf/instance-x1esoowk/file-d9f18687c3824c58b2f687aa7cc1c99d/mg.paf.filter.overlap' to path '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf'
Traceback (most recent call last):
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/worker.py", line 409, in workerScript
job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2845, in _runner
returnValues = self._run(jobGraph=None, fileStore=fileStore)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2761, in _run
return self.run(fileStore)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/toil/job.py", line 2990, in run
rValue = userFunction(*((self,) + tuple(self._args)), **self.kwargs)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/refmap/cactus_graphmap_split.py", line 417, in split_gfa
cactus_call(parameters=cmd, work_dir=work_dir, job_memory=job.memory)
File "/public/home/software/opt/bio/software/cactus/2.8.2/lib/python3.8/site-packages/cactus/shared/common.py", line 912, in cactus_call
raise RuntimeError("{}Command {} signaled {}: {}".format(sigill_msg, call, signal.Signals(-process.returncode).name, out))
RuntimeError: Command ['rgfa-split', '-p', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.paf', '-b', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split', '-Q', '2', '-a', 'AMBIGUOUS', '-L', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/split.log', '-n', '0.75', '-n', '0.5', '-n', '0.25', '-T', '100000', '-T', '1000000', '-g', '/tmp/toilwf-92830bc327ca547aa72da353a57003c4/0c33/4846/tmpoc8ttwwh/mg.gfa', '-G', '-r', 'id=NIP|', '-A', '5'] signaled SIGABRT: stderr=terminate called after throwing an instance of 'std::invalid_argument'
what(): stol
[2025-11-20T22:24:24+0800] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host gpu02
<=========
`
Could you give me some suggestions