Unexpected number of samples in summary output file cohorts.tsv

[koszulordie]

When running Intogen pipeline as a workflow in the Seqera launchpad, bbglabirb/ALP > intogen_cll_richter, the output file cohorts.tsv shows an inconsistent number of samples in the column SAMPLES.

I have checked that the parsing step BBGTOOLS:INTOGENPLUS:PARSE:GROUPBY (OpenVariant groupby) works as intended checking the ouput file of this process: /workspace/nobackup/work/intogen/intogen-richter/work/7e/c0c7be82455ae7407208bd61ceaa82/MASSONI.parsed.tsv.gz. When counting the unique samples with pandas, it gives the expected count of samples, 1069.

However, I have also checked that the step BBGTOOLS:INTOGENPLUS:PREPROCESS:VARIANTS_COUNT yields the wrong sample counts, as can be readily seen by executing the corresponding bash script:

variants=$(zcat MASSONI.parsed.tsv.gz | tail -n+2 | wc -l)
samples=$(zcat MASSONI.parsed.tsv.gz | tail -n+2 | cut -f1 | sort -u | wc -l)
echo "MASSONI	CLL	WGS	$variants	$samples" > MASSONI.counts

which yields the wrong count of 24 samples.

[FedericaBrando]

Hi @koszulordie ,

Thanks for noticing! I took a quick look at your files and at the code and this is a problem due to the fact that the process is expecting the first column to be SAMPLE. I checked your input parsed by openvariant and I see that the first column is CHROMOSOME, that is why the final output is 24.

For each FORMAT_* step, the order is not a problem, for this specific one although it is.

When building the yaml file that is fed to openvariant, the order should be as this:

intogen-plus/test/pipeline/input/cbioportal_prad_broad/metadata.yaml

Lines 5 to 17 in cf5f546

	columns:
	- SAMPLE
	- DONOR
	- CHROMOSOME
	- POSITION
	- REF
	- ALT
	- STRAND
	- ALT_TYPE
	- PLATFORM
	- GENOMEREF
	- CANCER
	annotation:

[koszulordie]

Thanks for the quick reply, @FedericaBrando. Absolutely, the script is cutting the first column and doing their unique counts there, got it, I did not pay attention when I opened the issue. This behavior seems artificial to me and error prone for future updates of the pipeline, thus I flagged it as a bug, but it might be rather some sort of unexpected, borderline invalid behavior. Thanks for appending the example.

[FedericaBrando]

Absolutely! In fact I think this could be handled in a most robust way by using awk for example, and retaining the column with SAMPLE in the header, rather than assuming the first one is the SAMPLE one!