Transcription when trends change (v0.15)

[ewancook]

Bug:
v0.15 gives incorrect output when trends are added or removed in the middle of (Float) files.

Detail:
The parser works on the wrong assumption that trends would be added from the start of files or values would be backfilled with placeholders (e.g. 0). Trends can be added at any point, and the only indicators of this are in the (Float) file:

• Status of U (update)
• Markers of B and E (beginning and end)

The markers are usually shown once for each tag at the beginning and end of each file. If a trend is added or removed, E is assigned to each tag before the trend change, then B is assigned for each tag after the trend is added (see the example below). I.e. the current record ends and a new record begins, but all in the same file.

Proposed Fix:
SubclassedDBF will read n rows until B is not seen, and yield the required tag indices. This allows calculation of the initial number of tags. The subsequent required rows will be read (using row skipping) as before without the modular arithmetic parsing breaking.

If trends are added or removed (or the file ends), E will be present in every row. Therefore only the first required row in the set of n rows needs to be checked as below.

If more rows are present (trends were added or removed), the number of tags will be recalculated with the method above, and parsing will continue.

This fix has minimal speed reduction!

# checking one row in every set of required rows:
rows = itertools.islice(rows_since_first_set, num_required_rows)
first_row = next(rows)
yield first_row
yield from rows
# check if first_row["Marker"] == b'E' and recalculate tags if so

---

Beginning and end marker example:

TagIndex	Status	Marker
0		B
1		B
2		B
0
1
2
0		E
1		E
2		E
0	U	B
2	U	B
0
2
0		E
2		E

Proposed Fix: