FastGA crash

[diego-rt]

Hello!

I've been trying the new fastga mode but I keep getting the following error when running on one specific locus. On another locus it behaves fine. Any idea why?

=========>
	[2025-12-28T20:24:35+0100] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
	[2025-12-28T20:24:35+0100] [MainThread] [I] [toil] Running Toil version 9.1.2-355b72ba0e425619c0367562caf2a337078dba65 on host n3501-005.
	[2025-12-28T20:24:35+0100] [MainThread] [I] [toil.worker] Working on job 'run_fastga' kind-run_fastga/instance-xci4cup5 v1
	[2025-12-28T20:24:41+0100] [MainThread] [I] [toil.worker] Loaded body Job('run_fastga' kind-run_fastga/instance-xci4cup5 v1) from description 'run_fastga' kind-run_fastga/instance-xci4cup5 v1
	[2025-12-28T20:24:41+0100] [MainThread] [I] [cactus.shared.common] Running the command ['FastGA', '-pafs', '-M', '-P./fgatmp', '-T8', 'Anc062_0.f.fa', 'Ornithorhynchus_anatinus_0.f.fa']
	[2025-12-28T20:24:41+0100] [MainThread] [I] [toil-rt] 2025-12-28 20:24:41.862186: Running the command: "FastGA -pafs -M -P./fgatmp -T8 Anc062_0.f.fa Ornithorhynchus_anatinus_0.f.fa"
	[2025-12-28T20:24:42+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
	[2025-12-28T20:24:42+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-make_chunked_alignments/0/instance-6mwdsqxr/cleanup/file-87169687bdc34aeea730a12940d41b1a/0.fa' to path '/tmp/toilwf-50c79cafa5e35242bfbea32ef89c8f1e/2032/job/tmpow9j6jyp/Ornithorhynchus_anatinus_0.fa'
	[2025-12-28T20:24:42+0100] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-make_chunked_alignments/0/instance-6mwdsqxr/cleanup/file-a38d3b86405b44539f4808b09d3c323d/0.fa' to path '/tmp/toilwf-50c79cafa5e35242bfbea32ef89c8f1e/2032/job/tmpow9j6jyp/Anc062_0.fa'
	[2025-12-28T20:24:42+0100] [MainThread] [C] [toil.worker] Worker crashed with traceback:
	Traceback (most recent call last):
	  File "/gpfs/data/fs72728/diego_rt/Projects/8_Transxolotl/2_TEmpura/bin/cactus_3.1.2/cactus-bin-v3.1.2/venv-cactus-v3.1.2/lib/python3.11/site-packages/toil/worker.py", line 595, in workerScript
	    job._runner(
	  File "/gpfs/data/fs72728/diego_rt/Projects/8_Transxolotl/2_TEmpura/bin/cactus_3.1.2/cactus-bin-v3.1.2/venv-cactus-v3.1.2/lib/python3.11/site-packages/toil/job.py", line 3254, in _runner
	    returnValues = self._run(jobGraph=None, fileStore=fileStore)
	                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/gpfs/data/fs72728/diego_rt/Projects/8_Transxolotl/2_TEmpura/bin/cactus_3.1.2/cactus-bin-v3.1.2/venv-cactus-v3.1.2/lib/python3.11/site-packages/toil/job.py", line 3137, in _run
	    return self.run(fileStore)
	           ^^^^^^^^^^^^^^^^^^^
	  File "/gpfs/data/fs72728/diego_rt/Projects/8_Transxolotl/2_TEmpura/bin/cactus_3.1.2/cactus-bin-v3.1.2/venv-cactus-v3.1.2/lib/python3.11/site-packages/toil/job.py", line 3452, in run
	    rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
	             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/gpfs/data/fs72728/diego_rt/Projects/8_Transxolotl/2_TEmpura/bin/cactus_3.1.2/cactus-bin-v3.1.2/venv-cactus-v3.1.2/lib/python3.11/site-packages/cactus/paf/local_alignment.py", line 169, in run_fastga
	    cactus_call(parameters=fastga_cmd, outfile=alignment_file, job_memory=job.memory, work_dir=work_dir)
	  File "/gpfs/data/fs72728/diego_rt/Projects/8_Transxolotl/2_TEmpura/bin/cactus_3.1.2/cactus-bin-v3.1.2/venv-cactus-v3.1.2/lib/python3.11/site-packages/cactus/shared/common.py", line 916, in cactus_call
	    raise RuntimeError("{}Command {} exited {}: {}".format(sigill_msg, call, process.returncode, out))
	RuntimeError: Command /usr/bin/time -f "CACTUS-LOGGED-MEMORY-IN-KB: %M" FastGA -pafs -M -P./fgatmp -T8 Anc062_0.f.fa Ornithorhynchus_anatinus_0.f.fa exited 1: stderr=
	FastGA: Call to GIXmake failed
	sh: -c: line 0: syntax error near unexpected token `('
	sh: -c: line 0: `GIXrm -fg ./Anc062_0.f(null)'
	sh: -c: line 0: syntax error near unexpected token `('
	sh: -c: line 0: `GIXrm -fg ./Ornithorhynchus_anatinus_0.f(null)'
	
	FastGA: Warning: Could not successfully remove .1gdb/.gix
	Command exited with non-zero status 1
	CACTUS-LOGGED-MEMORY-IN-KB: 141724
	
	
	[2025-12-28T20:24:42+0100] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host n3501-005
<=========

[glennhickey]

Which version of cactus are you using? This should be fixed in the latest one...

[diego-rt]

This was the latest one, 3.1.2

[glennhickey]

That's too bad. If you want to help debug this, you can try rerunning the failing command with --restart --cleanWorkDir never --workDir ./work (the workdir can be anything but you must make it before hand).

After it fails again, you should be able to find the two input files Anc062_0.f.fa Ornithorhynchus_anatinus_0.f.fa in the work directory. If there's something obviously crazy in Anc062_0.f.fa please let me know. If both inputs look okay, you can make an issue on the FastGA github (please mention it here in any case).

This error looks like one that's caused by contigs that have nothing but N's, which can happen in Anc genomes for whatever reason. But those should all be filtered out before getting sent to FastGA in that version of cactus. But maybe there's a corner case in this filtering that's not working...

[diego-rt]

Sure, happy to help debug! I think I had to delete this workdir because of storage issues, but I should be able to reproduce it later this week. Thanks a ton for the detailed feedback!