DynamoDB Streams

[kiril-pirozenko-home24]

This is rough implementation of DynamoDB Streams API. The intent of this PR is to gather feedback.

TODO:

• Add streams support to following actions:
  • UpdateTable
  • DeleteTable
  • BatchWriteItems
• Improve streams support:
  • ListStreams does not filter streams by table name
  • Respect StreamViewType from StreamSpecification (currently they always work as if NEW_AND_OLD_IMAGES was provided)
  • Stream records have eventID equal to SequenceNumber
  • Shard iterator is valid indefinitely (should be valid only for 15min)
• Improve tests:
  • Add tests to check request validations
• Consider adding following functionality (optional):
  • Support for multiple shards per stream (DynamoDB does not specify maximum shard size, so it's perfectly fine to have only one per stream, but it might be useful to be able to emulate multiple shards)
  • Support for record trimming (DynamoDB stores items for 24h only, but I'm not sure whether it would be useful to emulate that)

Implements: #66

[mhart]

Wow, @kiril-pirozenko-home24 this is a huge undertaking – respect!

Before you go any further though – my original thoughts were that it would be a lot easier to just borrow a lot of the stream code from https://github.com/mhart/kinesalite – considering that Kinesis and DynamoDB Streams are very similar.

Is this something you considered at all?

[kiril-pirozenko-home24]

Thank you!

No, haven't checked kinesalite at all. Our first goal was to have basic working mock for our services.

Could you have a look at code/functionality/TODO posted above and add what's missing for this PR to be mergeable? Maybe a checklist of your requirements to accept a PR? Because, you know, other business priorities... need to balance all this stuff 🙂 It would be helpful to see a scope of this task.

In general I approve of code borrowing 🙂 How about using more modern JS features? Do you have any requirements for minimum Node version supported? What's your take on tools like Typescript?

[mcwhittemore]

Oh man. This is amazing!

From a usability point of view, it would be nice to be able to wire up triggers with something like dynalite.stream('table', function(){}). It would also be nice to know how many records are in the stream so you can wait for the streams to empty before continuing.

[jakubriedl]

Hi, I love this (and actually need this). Can I help you with finishing this?

[andrewkrug]

+1 on the request for stream support and triggers

[ZECTBynmo]

I just wanted to write to report that I've used @kiril-pirozenko-home24's fork and his streams implementation seems to work extremely well. I hope this PR gets accepted eventually

[ryanblock]

Revisiting this now (7 years later!), also seeing what went down over in #88, I'm wondering if we can't reignite this PR by pulling out the Kinesalite internals we want and using them here (with credit!), without adding it as a dependency? I'm open to ideas, would love to see this move forward again as well.