New transaction is only created on `:61:` tag (statement) and not `:20:` tag (transaction reference number)

[aGitForEveryone]

In mt940/models.py lines 447-448, the comment states that new transactions are created from :20: and :61: tags. However, in the code it is actually only created on :61: tags (as the if-statement below only checks for Statement (:61:) tags.

I went through the git history and found that this was changed when going to v4.3.0, here is a link to the commit for this change. What was the reasoning to remove this? The comment was not updated then.

The reason for asking is that it gives issues when trying to do something with the balance information (:60F: and :62F:). I am working with ABN AMRO transactions which has the following structure

:20:
:25:
:28:
:60F:
(:61: + :86:) pair can occur 1-5 times
:62F:
-
:20:
:25:
:28:
:60F:
(:61: + :86:) pair can occur 1-5 times
:62F:
-

In this setup the transactions contain no info on the balance of the account before and after individual transactions. I am creating a post processor that loops through all the parsed transactions (so I have TypeOf[models.Transactions] and I loop over all the models.Transaction objects contained therein) and calculate the balance before and after each transaction.

Note: I set mt940.tags.BalanceBase.scope = mt940.models.Transaction (individual transaction)

In the current setup (without creating a new Transaction object when encountering tag :20:) this fails as the very first :60F: tag is skipped: when setting scope to Transaction, the check elif issubclass(tag.scope, Transaction) and self.transactions: will be only be True if at least 1 Transaction object has been created. At the start of the parsing however, self.transactions is still empty, so this check does not pass and the data is not added.

When parsing continues, the following happens: when the last :61: and :86: pair is encountered, the next tag will be :62F: and the parsing will continue to the next block where it will encounter a :60F:. Both these balances will be added to the last transaction of the first block. This is true because no new Transaction object was created (only happens when encountering :61: tags) and the Transaction scope tells the parser that the tag data should be added to the open Transaction object.

As a result, the balance info will not nicely match the closest transaction, which in my opinion should be :60F: to first transaction of block and :62F: to last transaction of block.

Resetting the logic to create a new Transaction object for both the :20: and :61: solves this issue and nicely matches the balance info to the closest transaction:

if isinstance(
    tag, 
    (mt940.tags.Statement, mt940.tags.TransactionReferenceNumber)
):

Is there any mt940 data from other banks that would not be possible to parse when updating the if-statement? Or can you otherwise remember why the :20: tag trigger was removed?

[wolph]

Wow, very detailed write-up, thank you! Sorry for the extremely slow response... I've been severely neglecting this repo.

The reason for that change was this issue: #26

The big issue with MT940 is that the format is not strictly defined between multiple banks... many banks have their own specific interpretations and fixing an issue for one bank breaks it for others.

I'm honestly not entirely sure what a good solution is for this case, you've clearly demonstrated that it breaks for ABN AMRO (which is the bank I mainly use so that's why it was initially working fine). But for some other banks this does not work :(
I can't say I've used this library much beyond the first few years so I never noticed that it was now not working as expected.

Perhaps some heuristics could work but I'm wary of making it smarter since it could introduce even more issues down the line.

[wichert]

Perhaps it would be useful to allow a dialect option to be passed in, similar to csv.reader for example?

[aGitForEveryone]

Thanks for the reply. Maybe the logic when to create a new transaction needs an update.

Perhaps the dialect option like @wichert proposes is a good option. When looking for information to parse MT940 files, I also saw that each bank has their own "standard", very annoying indeed.

Another solution could be to keep track of the block scope, and assign data of the block scope to each :61: tag transaction.

I will try to find some time soon to see I can come up with a solution to this.