database: Add basic database validation and error description

[german77]

Some databases are corrupted like #11. This PR adds an explanation for parser errors and checks for new types that aren't handled by this program.

The new error message will contain the ID of the corrupted entry a short explanation and a dump of the entry blob. This will be helpful for users reporting issues with corrupted entries. However since the entry couldn't be parsed correctly it can contain sensitive data encoded within the blob.

Corrupted Entry id 10004: Unsupported Field type
        '4D320100FF8801000400000031751001307510090033751008FFFFFFFF0100FE081427000032751021001000FE210450696E67'

[pmattia90]

Hello, now i can find this kind of description,

Reading database luglio25.db
Opened database successfully
Saving database luglio25.json
Invalid Entry expected 260936, found 0
Invalid Entry expected 261047, found 0
Invalid Entry expected 261049, found 0
Invalid Entry expected 261801, found 0
Invalid Entry expected 362679, found 0
Invalid Entry expected 580500, found 0
Invalid Entry expected 580506, found 0
Invalid Entry expected 580508, found 0
Invalid Entry expected 585445, found 0
Invalid Entry expected 856010, found 0
Invalid Entry expected 947641, found 0
Invalid Entry expected 947646, found 0
Invalid Entry expected 1183115, found 0
Invalid Entry expected 1301043, found 0
Invalid Entry expected 2134626, found 0
Invalid Entry expected 2160504, found 0
Invalid Entry expected 2162607, found 0
Invalid Entry expected 2253218, found 0

What i'm supposed to do? There are options to print details? If i search this entry and i delete them maybe i can fix the problem of the dude! I will try
Thanks

[german77]

I been working into adding ID validation for individual object references to produce outputs like.

Map 55910: Invalid notify id found 10275 in map BABA
Map 56071: Invalid notify id found 10272 in map KEKE
Map 56071: Invalid notify id found 10275 in map JIJI

However based on your log is quite interesting. No new data formats or types which is good. However your entries were parsed without errors but seem incomplete as the object id is zero. Give me some time to make the error a lot more verbose. I should be able to make it display all fields in plain text, hopefully this will allow us to identify what is wrong with these.

[german77]

@pmattia90 Give it a second try to this PR. I added a detailed description. If any error happens while the parser is running it will also display that error. The message should look as follows:

Invalid Entry expected 3516, found 0
Field data:  Format 0x39
        Category 0x10cb25, 0xa0: []
        Category 0x10cb20, 0x0: false
        Category 0x10cb23, 0x9: 0
        Category 0x10cb24, 0x9: 0
        Category 0xfe0001, 0x8: 3516
        Category 0x10cb22, 0x21: "Display Temperarure CPU"
        Category 0x10cb21, 0x21: "oid(\"iso.org.dod.internet.private.enterprises.mikrotik.mikrotikExperimentalModule.mtXRouterOs.mtxrHealth.11.0\")"
        Category 0xfe0010, 0x21: "Cpu_temp"

This is the same data but with text descriptions instead of hex values.

Invalid Entry expected 3516, found 0
Field data:  Format Function
        Category Function_ArgumentDescrs, StringArray: []
        Category Function_Builtin, Bool: false
        Category Function_MinArguments, Byte: 0
        Category Function_MaxArguments, Byte: 0
        Category SysId, Int: 3516
        Category Function_Descr, ShortString: "Display Temperarure CPU"
        Category Function_Code, ShortString: "oid(\"iso.org.dod.internet.private.enterprises.mikrotik.mikrotikExperimentalModule.mtXRouterOs.mtxrHealth.11.0\")"
        Category SysName, ShortString: "Cpu_temp"

[pmattia90]

Hello,
now this is what it reports, how can i translate to text descriptions?

`
Invalid Entry expected 260936, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 260937
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1704271933
Category 0x102ef0, 0x8: 268
Category 0x102ef1, 0x9: 0
Category 0x102ef2, 0x8: 62996
Category 0x102ef3, 0x9: 0
Category 0x102ef4, 0x8: 12475
Category 0xfe0001, 0x8: 260936
Category 0x102ee1, 0x8: 260934
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 261047, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 261048
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1715933030
Category 0x102ef0, 0x8: 270
Category 0x102ef1, 0x8: 2879
Category 0x102ef2, 0x8: 11455134
Category 0x102ef3, 0x8: 268951
Category 0x102ef4, 0x8: 12475
Category 0xfe0001, 0x8: 261047
Category 0x102ee1, 0x8: 261045
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 261049, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 261050
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1715933043
Category 0x102ef0, 0x8: 272
Category 0x102ef1, 0x8: 2879
Category 0x102ef2, 0x8: 11455150
Category 0x102ef3, 0x8: 268948
Category 0x102ef4, 0x8: 12475
Category 0xfe0001, 0x8: 261049
Category 0x102ee1, 0x8: 261045
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10174
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 261801, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 261802
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1732640061
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x8: 48504
Category 0x102ef2, 0x8: 23027034
Category 0x102ef3, 0x8: 5404077
Category 0x102ef4, 0x8: 12489
Category 0xfe0001, 0x8: 261801
Category 0x102ee1, 0x8: 261799
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 362679, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x1: true
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 362680
Category 0x102eed, 0x9: 3
Category 0x102eef, 0x8: 1723207577
Category 0x102ef0, 0x8: 270
Category 0x102ef1, 0x8: 3709085
Category 0x102ef2, 0x8: 12323278
Category 0x102ef3, 0x8: 3740342
Category 0x102ef4, 0x8: 109276
Category 0xfe0001, 0x8: 362679
Category 0x102ee1, 0x8: 362675
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 580500, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x8: 86400
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 580501
Category 0x102eed, 0x9: 1
Category 0x102eef, 0x8: 1736354761
Category 0x102ef0, 0x9: 50
Category 0x102ef1, 0x8: 629
Category 0x102ef2, 0x8: 15085397
Category 0x102ef3, 0x8: 50538
Category 0x102ef4, 0x9: 0
Category 0xfe0001, 0x8: 580500
Category 0x102ee1, 0x8: 580495
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10164
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: "Service"

Invalid Entry expected 580506, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x1: true
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 580507
Category 0x102eed, 0x9: 3
Category 0x102eef, 0x8: 1747638189
Category 0x102ef0, 0x8: 712
Category 0x102ef1, 0x8: 22101079
Category 0x102ef2, 0x8: 4318103
Category 0x102ef3, 0x8: 22101229
Category 0x102ef4, 0x8: 17225
Category 0xfe0001, 0x8: 580506
Category 0x102ee1, 0x8: 580505
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 580508, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x1: true
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 580509
Category 0x102eed, 0x9: 3
Category 0x102eef, 0x8: 1747638189
Category 0x102ef0, 0x8: 710
Category 0x102ef1, 0x8: 22101073
Category 0x102ef2, 0x8: 4318109
Category 0x102ef3, 0x8: 22101223
Category 0x102ef4, 0x8: 17226
Category 0xfe0001, 0x8: 580508
Category 0x102ee1, 0x8: 580505
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10164
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 585445, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x0: false
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x8: 86400
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 585446
Category 0x102eed, 0x9: 0
Category 0x102eef, 0x8: 1750687139
Category 0x102ef0, 0x8: 161854
Category 0x102ef1, 0x8: 690
Category 0x102ef2, 0x8: 27665595
Category 0x102ef3, 0x8: 615232
Category 0x102ef4, 0x9: 25
Category 0xfe0001, 0x8: 585445
Category 0x102ee1, 0x8: 579262
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10164
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: "Service"

Invalid Entry expected 856010, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x0: false
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x8: 86400
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 856009
Category 0x102eed, 0x9: 0
Category 0x102eef, 0x8: 1750687133
Category 0x102ef0, 0x8: 3807631
Category 0x102ef1, 0x8: 2910
Category 0x102ef2, 0x8: 29362453
Category 0x102ef3, 0x8: 53448
Category 0x102ef4, 0x9: 25
Category 0xfe0001, 0x8: 856010
Category 0x102ee1, 0x8: 856007
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10164
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: "Service"

Invalid Entry expected 947641, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 947640
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1738947522
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x8: 93357
Category 0x102ef2, 0x8: 32622844
Category 0x102ef3, 0x8: 794938
Category 0x102ef4, 0x8: 12490
Category 0xfe0001, 0x8: 947641
Category 0x102ee1, 0x8: 947638
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 947646, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 947645
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1738947523
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x8: 93357
Category 0x102ef2, 0x8: 32622845
Category 0x102ef3, 0x8: 794938
Category 0x102ef4, 0x8: 12489
Category 0xfe0001, 0x8: 947646
Category 0x102ee1, 0x8: 947643
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 1183115, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x1: true
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 1183114
Category 0x102eed, 0x9: 3
Category 0x102eef, 0x8: 1751275157
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x8: 3534207
Category 0x102ef2, 0x8: 761
Category 0x102ef3, 0x8: 3534207
Category 0x102ef4, 0x8: 260
Category 0xfe0001, 0x8: 1183115
Category 0x102ee1, 0x8: 1183112
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 1301043, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 1301044
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1750853246
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x8: 1910
Category 0x102ef2, 0x8: 2307392
Category 0x102ef3, 0x8: 133013
Category 0x102ef4, 0x8: 2800
Category 0xfe0001, 0x8: 1301043
Category 0x102ee1, 0x8: 1301037
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 2134626, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 2134625
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1751891777
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x8: 515782
Category 0x102ef2, 0x8: 2076228
Category 0x102ef3, 0x8: 1313498
Category 0x102ef4, 0x8: 377
Category 0xfe0001, 0x8: 2134626
Category 0x102ee1, 0x8: 2134623
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 2160504, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 2160503
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1751443487
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x9: 0
Category 0x102ef2, 0x8: 659532
Category 0x102ef3, 0x9: 0
Category 0x102ef4, 0x8: 2301
Category 0xfe0001, 0x8: 2160504
Category 0x102ee1, 0x8: 2160501
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 2162607, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 2162606
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1751447057
Category 0x102ef0, 0x8: 703
Category 0x102ef1, 0x9: 0
Category 0x102ef2, 0x8: 1415441
Category 0x102ef3, 0x9: 0
Category 0x102ef4, 0x8: 2419
Category 0xfe0001, 0x8: 2162607
Category 0x102ee1, 0x8: 2162604
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""

Invalid Entry expected 2253218, found 0
Field data: Format 0x11
Category 0x102ee9, 0x88: []
Category 0x102ee0, 0x1: true
Category 0x102eeb, 0x1: true
Category 0x102ee8, 0x0: false
Category 0x102eea, 0x0: false
Category 0x102ee4, 0x9: 0
Category 0x102ee5, 0x9: 0
Category 0x102ee6, 0x9: 0
Category 0x102ee7, 0x9: 0
Category 0x102eec, 0x8: 2253217
Category 0x102eed, 0x9: 2
Category 0x102eef, 0x8: 1751612224
Category 0x102ef0, 0x8: 720
Category 0x102ef1, 0x9: 0
Category 0x102ef2, 0x8: 1350408
Category 0x102ef3, 0x9: 0
Category 0x102ef4, 0x8: 4734
Category 0xfe0001, 0x8: 2253218
Category 0x102ee1, 0x8: 2253215
Category 0x102ee2, 0x8: -1
Category 0x102ee3, 0x8: 10160
Category 0x102eee, 0x10: 0
Category 0xfe0010, 0x21: ""
`

[german77]

The text descriptions are based on the enums at dude_field_id.h and dude_types.h. 0xfe0001 is the sysId which actually matches with the expected id. So this is parser error an not actually corruption.

All your entries are of type service. I will try to find the mistake in the parser and upload a fix.

[german77]

@pmattia90 Seems like I was trying to read a byte on an int field. I promoted a bunch of fields to int. Now it should output no errors. I also added the -i argument to run an integrity test on the data but that's still WIP.

[pmattia90]

I can confirm that now no errors are reported, So i can't understand which node or field creates problem in dude, correct?

[german77]

That's great news for me. Not so much for you. Correct we don't know which node or field is causing your dude to crash. Maybe after finishing the integrity test we will have more clues. For now we don't have the slightest idea.