OpenVariant Bug | Windows input files not parsed properly

[CarlosLopezElorduy]

Describe the issue

I encountered an issue where two different files with the same content had different behaviours when parsed with openvariant. One is successfully being parsed while the other one isn't. After some debugging, I saw that the difference is in the OS where the actual file was created.

• If the file was created in Linux, openvariant is succesful ✅
• If the file was created in Windows, openvariant doesn't parse properly the input ❌

This is caused by how line jumps are being handled between different Operating Systems.

OS	Line jump character
Linux	\n
Windows	\r\n

Steps to reproduce the bug

One of the input files needs to be created using a Windows machine, while the other should be created using Linux. The content should be identical.

• KRAS_linux.txt
• KRAS_windows.txt

Content of the TXT files

protein
KRAS:p.G12V
KRAS:p.G12C
KRAS:p.G12S
KRAS:p.G12F
KRAS:p.G12A
KRAS:p.G13D

Annotation file

pattern:
  - '.*\.txt$'
  - '.*\.tsv$'
  - '.*\.txt\.gz$'
  - '.*\.tsv\.gz$'

columns:
  - PROTEIN
  - TYPE

annotation:

  - type: internal
    field: GENE
    fieldSource:
      - 'gene'
      - 'Gene'
      - 'GENE'
      - 'Symbol'
      - 'SYMBOL'
      - 'GENE_SYMBOL'
  - type: internal
    field: PROTEIN
    fieldSource:
      - "protein"
      - "Protein"
      - "PROTEIN"
  - type: internal
    field: TRANSCRIPT
    fieldSource:
      - 'transcript'
      - 'TRANSCRIPT'
      - 'ENSEMBL_TRANSCRIPT'
      - 'ensembl_transcript'
      - 'MANE_SELECT'
      - 'mane_select'
      - 'MANE'
      - 'mane'
      - 'refseq_transcript'
      - 'REFSEQ_TRANSCRIPT'
      - 'refseq'
      - 'REFSEQ'
  - type: static
    field: "TYPE"
    value: "protein"

Python script(`openvariant_read.py`)

from openvariant import Annotation, Variant

dataset_file_linux = "KRAS_linux.txt"
dataset_file_windows = "KRAS_windows.txt"

annotation_file = "protein.yaml"

annotation = Annotation(annotation_path=annotation_file)
result_linux = Variant(path=dataset_file_linux, annotation=annotation)
result_windows = Variant(path=dataset_file_windows, annotation=annotation)

for n_line, line in enumerate(result_linux.read()):
    print(f'Line (linux) {n_line}: {line}')
    if n_line == 9:
        break

for n_line, line in enumerate(result_windows.read()):
    print(f'Line (windows) {n_line}: {line}')
    if n_line == 9:
        break

Error encountered or actual result

By running the previous script, this is the output that we obtain:

$ python openvariant_read.py
Line (linux) 0: {'PROTEIN': 'KRAS:p.G12V', 'TYPE': 'protein'}
Line (linux) 1: {'PROTEIN': 'KRAS:p.G12C', 'TYPE': 'protein'}
Line (linux) 2: {'PROTEIN': 'KRAS:p.G12S', 'TYPE': 'protein'}
Line (linux) 3: {'PROTEIN': 'KRAS:p.G12F', 'TYPE': 'protein'}
Line (linux) 4: {'PROTEIN': 'KRAS:p.G12A', 'TYPE': 'protein'}
Line (linux) 5: {'PROTEIN': 'KRAS:p.G13D', 'TYPE': 'protein'}
Line (windows) 0: {'PROTEIN': 'nan', 'TYPE': 'protein'}
Line (windows) 1: {'PROTEIN': 'nan', 'TYPE': 'protein'}
Line (windows) 2: {'PROTEIN': 'nan', 'TYPE': 'protein'}
Line (windows) 3: {'PROTEIN': 'nan', 'TYPE': 'protein'}
Line (windows) 4: {'PROTEIN': 'nan', 'TYPE': 'protein'}
Line (windows) 5: {'PROTEIN': 'nan', 'TYPE': 'protein'}

The protein changes in the linux input file are correctly parsed, while the windows input is returning "nan"

Expected result

The expected output should be that, independently of the OS where the input file was created, it should be parsed in the same way:

$ python openvariant_read.py
Line (linux) 0: {'PROTEIN': 'KRAS:p.G12V', 'TYPE': 'protein'}
Line (linux) 1: {'PROTEIN': 'KRAS:p.G12C', 'TYPE': 'protein'}
Line (linux) 2: {'PROTEIN': 'KRAS:p.G12S', 'TYPE': 'protein'}
Line (linux) 3: {'PROTEIN': 'KRAS:p.G12F', 'TYPE': 'protein'}
Line (linux) 4: {'PROTEIN': 'KRAS:p.G12A', 'TYPE': 'protein'}
Line (linux) 5: {'PROTEIN': 'KRAS:p.G13D', 'TYPE': 'protein'}
Line (windows) 0: {'PROTEIN': 'KRAS:p.G12V', 'TYPE': 'protein'}
Line (windows) 1: {'PROTEIN': 'KRAS:p.G12C', 'TYPE': 'protein'}
Line (windows) 2: {'PROTEIN': 'KRAS:p.G12S', 'TYPE': 'protein'}
Line (windows) 3: {'PROTEIN': 'KRAS:p.G12F', 'TYPE': 'protein'}
Line (windows) 4: {'PROTEIN': 'KRAS:p.G12A', 'TYPE': 'protein'}
Line (windows) 5: {'PROTEIN': 'KRAS:p.G13D', 'TYPE': 'protein'}

OpenVariant version

1.0.1

Python version

3.10.12

Installation method

pip install open-variant

Environment

Openvariant installed via pip in a conda environment. Ran using a regular Python script (shown in the "steps to reproduce the bug" section)

OS

Windows vs Linux

Other commentaries (optional)

No response

Contact details (optional)

No response

[FedericaBrando]

Hi Carlos, thanks for the report, can you upload both files for test purposes? Thanks

[CarlosLopezElorduy]

Sure, I have just edited the description (steps to reproduce the bug)

[FedericaBrando]

fixed in: #48

[CarlosLopezElorduy]

Apparently, there is another type of file we can't parse properly, also coming from windows. This time, the character used for the line break is \r.

$ od -c windows_vcf.vcf 
0000000   #   #   f   i   l   e   f   o   r   m   a   t   =   V   C   F
0000020   v   4   .   1   ;   ;   ;   ;   ;   ;   ;   ;   ;  \r   #   C
0000040   H   R   O   M   ;   P   O   S   ;   I   D   ;   R   E   F   ;
0000060   A   L   T   ;   Q   U   A   L   ;   F   I   L   T   E   R   ;
0000100   I   N   F   O   ;   F   O   R   M   A   T   ;   A   L   L   P
0000120   a   t   i   e   n   t  \r   c   h   r   1   6   ;   5   7   0
0000140   7   1   1   1   3   ;   1   ;   T   A   ;   C   G   ;   ;   ;
0000160   A   F   =   1   ;   ;  \r   c   h   r   1   0   ;   8   0   0
0000200   7   5   5   9   ;   1   ;   G   T   ;   C   C   ;   ;   ;   A
0000220   F   =   1   ;   ;  \r   c   h   r   5   ;   1   6   1   1   2
0000240   8   7   6   1   ;   1   ;   C   ;   G   ;   ;   ;   A   F   =
0000260   1   ;   ;  \r   c   h   r   2   ;   1   0   9   3   7   4   9
0000300   5   2   ;   1   ;   A   ;   G   ;   ;   ;   A   F   =   1   ;
0000320   ;  \r   c   h   r   3   ;   4   9   7   2   3   7   8   4   ;
0000340   1   ;   C   ;   A   ;   ;   ;   A   F   =   0   .   9   8   ;

The behaviour is the same in which the file is not properly parsed, as they are not detected as newlines by openvariant.

File used:

windows_vcf.vcf.zip

📌 @FedericaBrando