maroon-storage definition. v0.0.1 #5
Conversation
An attempt to define function of maroon-storage and raise some questions about internal parts. Also sketched a bit a configuration of storages(how crazy it looks?)
|
👍 |
| - A | ||
| - all updates are done or none | ||
| - C | ||
| - invariants are satisfied(across different repositories) |
There was a problem hiding this comment.
This is misleading imho — the "C" in our case has nothing to do with different repositories directly, the repositories are synced with in an eventually consistent way, decoupled from running ACID transactions.
| - since we'll have sequential read/writes - it's not relevant for migrator | ||
| - [!] fine only until we're "single-threaded" | ||
| - D | ||
| - set of data as durable as repository is durable(if some repository owns some subset of data - we're limited by this repository's durability) |
There was a problem hiding this comment.
Heh, I would argue the opposite: the "D" in our case is what the maroon execution framework guarantees — that ultimately all the changes will propagate to whatever repositories exist out there, but not necessarily immediately.
"D" is about not losing the data, and the maroon framework (imho) is quite allowed to (temporarily) be the source of truth for "hot" keys/values. Ideally, 99.9+% of the data is always durably stored in various repositories (including S3-grade ones, or even Iceberg-ed), but I'd refrain from making claims that our "D" implies prompt storage of all the data into the respective repositories.
| priority_migration( | ||
| # in that case data will be slowly copied from one storage to another | ||
| # TODO: we need to have a requirement here that transformation should cover all the fields and it should be checked | ||
| "eu-users-repository" ==> "eu-users-repository-new" |
There was a problem hiding this comment.
It this ==> a typo of => or an intentionally different operator? :-)
There was a problem hiding this comment.
intentionally different, it's not equal-or-greater
| # TODO: we need to have a requirement here that transformation should cover all the fields and it should be checked | ||
| "eu-users-repository" ==> "eu-users-repository-new" | ||
| ) | ||
| User.v1(country == "USA" => "us-users-repository"), |
There was a problem hiding this comment.
FTR, I personally love this, and am sure @AdamEther would love it too. Even though we didn't speak of this yet.
dimacurrentai
left a comment
dimacurrentai
left a comment
There was a problem hiding this comment.
I like this!
Shall we proceed with behavior-driven test descriptions?
In my mind it's taking shape. More as we speak next time!
3 participants
An attempt to define function of maroon-storage and raise some questions about internal parts.
Also sketched a bit a configuration of storages(how crazy it looks?)