Question: complete rewrite, again? #81
Description
Problem
Based on the #77 and #76 feature requests, and thus the subsequent need for #78 I think the only sensible thing is to back to the drawing board and rethink how to structure this SDK. Adding features to the project in its current state is extremely tedious and requires a lot of code-repetition.
Suggestion
Inspired by LLVM architecture, how about we structure everything in terms of frontend (managing reading input file and slicing it), intermediate representation (IR) of the data, and then target backend (takes the sliced IR and produces target output.
Question
What would suitable IR be? I guess on a byte-level makes sense. Wrap the byte buffer?
Do we still need the modularized crates? How can we enable others to use e.g. backends for targets that we have implemented while supplying their own frontend implementation? That way we could have written e.g. Parquet target backend and someone writes a custom frontend for their own strange file format that they want transformed to Parquet! 👍
Thoughts? @Ignalina