Test KSML Schema Evolution support

[KasparMetsa]

• Test all these scenarios in the FULL Test Matrix below using AVRO & JSON Schema
  • Also for Protobuf when KSML fully supports Protobuf.
• Consider writing some integration tests, maybe not for all because it will be slow to build, but for most important ones in the "Recommended Minimal Test Matrix". Explore if possible to write unit tests for these scenarios, if possible, we should write unit tests for all scenarios.
• Work done so far:
  • Added IT test AvroSchemaEvolutionBackwardCompatibilityIT for scenario 1 in "Recommended Minimal Test Matrix".
  • Manually tested AVRO support for scenarios 1-5 in "Recommended Minimal Test Matrix".
• Also please note that we have at least four points where compatibility plays a part.
  • On consume, data is produced with Avro/Protobuf/JSON schema X and KSML might br configured to consume using an older or newer version of the schema.
  • Pre produce, the internal KSML schema and data might not match the specified Avro/PB/JSON schema
  • On produce, the schema configured by KSML might not match with the schema specified in the registry.
  • An operation function modifies the schema to a version that's not the same as specified for the next pipeline steps.

Recommended Minimal Test Matrix

For production confidence, test these scenarios on AVRO and JSON Schema:

Scenario	Backward	Forward	Status
1. Add optional field	Should PASS	N/A	✅ Tested + added IT
2. Remove optional field	N/A	Should PASS	✅ Tested
3. Add required field	Should FAIL	N/A	✅ Tested
4. Remove required field	N/A	Should FAIL	✅ Tested
5. Type promotion (int→long)	Should PASS	Should FAIL	✅ Tested, BUG: #417
6. Type incompatible (string→int)	Should FAIL	Should FAIL	❌ TODO

FULL Test Matrix

Comprehensive list of all schema evolution scenarios across all categories:

#	Category	Scenario	Backward	Forward	Status	Notes
1	Optional Fields	Add optional field with default	Should PASS	N/A	✅ Tested	Most common evolution
2	Optional Fields	Remove optional field	N/A	Should PASS	✅ Tested	Most common evolution
3	Required Fields	Add required field	Should FAIL	N/A	✅ Tested	Safety check
4	Required Fields	Remove required field	N/A	Should FAIL	✅ Tested	Safety check
5	Field Optionality	Make field optional → required	N/A	Should FAIL	❌ TODO	Breaking change
6	Field Optionality	Make field required → optional	Should PASS	N/A	❌ TODO	Safe change
7	Type Promotion	Change int → long	Should PASS	Should FAIL	✅ Tested, BUG: #417	AVRO allows promotion
8	Type Incompatible	Change string → int	Should FAIL	Should FAIL	❌ TODO	Incompatible types
9	Type Incompatible	Change int → string	Should FAIL	Should FAIL	❌ TODO	Incompatible types
10	Default Values	Change default value	N/A	N/A	❌ TODO	Semantic, not structural
11	Enum (AVRO)	Add enum value with default	Should PASS	N/A	❌ TODO	Common pattern
12	Enum (AVRO)	Remove enum value	N/A	Should FAIL	❌ TODO	Breaking change
13	Enum (AVRO)	Reorder enum values	Should PASS	Should PASS	❌ TODO	AVRO uses names
14	Nested Schema	Add optional field to nested record	Should PASS	N/A	❌ TODO	Same as top-level
15	Nested Schema	Change nested record type	Should FAIL	Should FAIL	❌ TODO	Low value test
16	Field Order	Reorder fields in schema	Should PASS	Should PASS	❌ TODO	AVRO is name-based
17	Union (AVRO)	Add type to union	Should PASS	N/A	❌ TODO	Advanced feature
18	Union (AVRO)	Remove type from union	N/A	Should FAIL	❌ TODO	Advanced feature

[KasparMetsa]

This is the error message currently when adding a required field, this is expected error, quite clear error message but not 100% clear

error_when_adding_required_field.txt