de-duplicating a large sam file

[morien]

I'm trying to de-duplicate a large sam file (it's ~200GB in size). This tool seems promising (although I'd like to know more about how it handles unpaired reads, which I would still like to use in my pipeline). Before I worry about that, I'd like to get it working.

Here's some system details:

OS: Linux 4.18.0-492.el8.x86_64
machine architecture: x86_64
compiler: gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-21)

The following command:

samblaster -r -i SC-13c-TCMP_S214_L001_R1_001.fastq.merged.sam -o SC-13c.rmdup.sam.gz

produces this output:

samblaster: Version 0.1.26
samblaster: Opening SC-13c-TCMP_S214_L001_R1_001.fastq.merged.sam for read.
samblaster: Opening SC-13c.rmdup.sam.gz for write.
samblaster: Loaded 439926763 header sequence entries.
samblaster: Unable to allocate signature set array.samblaster: Premature exit

The sam file is merged from many 10s of bam files, each of which is made from the same query sequence set mapped to a different reference set. There may be duplicate entries in the reference sets, so this is why I am trying to de-duplicate this merged file. This seems like a memory-related error, but I have over 900GB of free memory on this machine. Any help is greatly appreciated.

[GregoryFaust]

The output line immediately above the error reports that your input SAM file has almost 440 million @SQ lines in its header? Usual reference genomes, even ones comprised of contigs that are not fully assembled, do not have anywhere near that many sequences. If your merged SAM file really has that many (long) genome sequences in it, this is the immediate (proximate) cause of the error. However, it appears that samblaster could arguably do a better job of picking an initial hash table size than it currently does, at the possible expense of more hash table resize operations while processing the SAM file. The current code sets the initial hash table size assuming the SAM file has some combination of a large number of short sequences and/or a small number of long sequences, but not a large number of long sequences.

Does this help?

[morien]

This SAM is merged from many individual BAM files. Each of these is mapped to a different reference. Some of the references are genomes, but some are not. The references of different types (for example: refseq mammalian genomes and the NCBI NT database) may overlap in their sequence content. Thus the desire for deduplication. Databases may contain either long or short sequences. The NT database has quite a large number of small sequences, and my genome databases contain nearly all available genomes in refseq. I think this may indicate you are right about your suspected cause of the error.