Update allowed maximum data age to 30 days

[samdozor]

Hello! I'm creating this PR mostly as an example. If you would like the integration to receive data that's older than 24 hours (perhaps, for data replays) - you'll have to update your ModuleRegistrationResponse as displayed here. I picked 30 days arbitrarily so feel free to edit as you wish.

If you do choose to make any change from 24 hours - mParticle can quickly re-import the integration to reflect.

[vbabenkoru]

Looks like this may hit 429s on our API side if we enable large replays.

[samdozor]

Agreed, but this is a bit orthogonal. Customers can overload the lambda with recent data too. Or they could send in a small amount of old data and not overload it.

A common case is a user opens an app, and then swipes it closed before the SDK has a chance to upload, and then comes back to the app a day (or more) later. Right now that data is not sent to iterable.

I'd recommend setting this value to whatever Iterable's backend can accept at the endpoint currently used by the lambda function, and then we separately figure a way to address volume spikes regardless of timing.

[dbmiller1128]

Hi @julianmclain Was this update ever merged? I still see "max_data_age_hours": 24, in the module registration response.

[julianmclain]

@dbmiller1128 We haven't merged it yet. I've been meaning to reach out about this actually. During an event replay, is it possible for the mParticle request payloads to be larger than 256k?

[dbmiller1128]

@julianmclain I don't think so

[julianmclain]

@dbmiller1128 Ok - if that's the case we should be able to enable it. Does mP have a data set we can use to test it?

[samdozor]

Data replays don't trigger any different batch size than normal (just a lot of batches/lambda invocations). mParticle takes the original data as it was received (and would have been otherwise sent to integrations at that time), and sends it as is.

[julianmclain]

@samdozor Ok thanks. For context, right now we're occasionally seeing event batches larger than 256k which is problematic. We need to redesign things in the firehose a bit to handle them properly (it'll happen but timeline tbd).

Does mP have a test data set we can import into our sandbox or otherwise replay somehow?

[dbmiller1128]

We do not have a data set to replay.

[julianmclain]

@dbmiller1128 Ok no problem