[Proposal]: Unify row entry normalizers in the adapters

[stloyd]

Describe the Proposal

Right now, we have a few row-level normalizers in adapters:

• csv,
• json,
• excel,
• parquet

All of them behave a bit differently, but most of them share a common path for resolving data from rows.

I propose reworking the CSV and JSON formats to match the approach introduced in the Excel adapter, where an unknown entry type throws an exception instead of falling back to an empty string (or enforcing a string value). I would skip reworking the Parquet normalizer (for now), as it's a binary format and it may require the current setup.

API Adjustments

No response

Are you intending to also work on proposed change?

Yes

Are you interested in sponsoring this change?

None

Integration & Dependencies

No response

[norberttech]

I propose reworking the CSV and JSON formats to match the approach introduced in the Excel adapter, where an unknown entry type throws an exception instead of falling back to an empty string (or enforcing a string value).

We should do it, no doubts there.

I would skip reworking the Parquet normalizer (for now), as it's a binary format and it may require the current setup.

And this is where things becomes tricky, Parquet unlike other formats, actually expects proper values, and then based on Entry Schema Definition it creates Parquet Schema Definition, which parquet library uses later to serialize into binary string.
So Parquet Normalizer is almost completely different than CSV/JSON (but its still a file format).

Similarly Doctrine Adapter, RowsNormalizer - it also passes values rather than strings (except for XML/XmlElement) and it's even more complicated in postgresql adapter, which needs the converter to pretty much convert entries into TypedValue so it can be passed to the client as typed parameter.

But I'm starting to see a pattern here, there are formats that expects entries to be normalized to strings, like CSV/Json/XML/Text/Excel/GoogleSheet/ChartJs/Logger
For all of those we might be able to create a single, fully configurable EntryNormalizer that could be reused across them.

For adapters like Doctrine/Parquet/PostgreSql/ElasticSearch/MeiliSearch we might need to take a different approach but I don't yet see how it could look.

---

Long story short, I think we can try to extract a common mechanism that would convert entries to string values like.

Flow\ETL\Row\Normalizer\EntryStringNormalizer with method normalize(Entry $entry) : string

That normalizer would need to be able to carry the configuration similar to what I mentioned in #2107 (comment)

That thing would become a single source of entry to string conversion and it should be used across all the places (including AsciiFormatters) to convert entry into a string.

Once This is in place we could remove toString() method from Entry as Entry should never got it in the first place. It's not for entry to decide how it should be represented as a string, because back to your point from EnumEntry, sometimes it would be name, other time value.

Would you like to try to design such a generic normalizer? The scope should be following:

• it should let to configure how we convert entries of given type or just specific entries
• it should be globally created on FlowContext but each loader should allow to use a custom one through withNormalizer(EntryStringNormalizer) : self and use it internally instead of grabbing it from context