feat: add streaming support

[tysoekong]

Adds:

• Re-usable parser for Amazon VND stream events binary format ("chunk streaming")
• Execution branch in execute.lua to return the body stream reader to the caller, for use in the AWS Lambda plugin (streaming mode)
• Update SDK to v2.1353.0 to add missing Invoke Lambda with Streaming Response shapes

[github-actions]

Luacheck Report

27 tests    2 ✅  0s ⏱️
 1 suites   0 💤
 1 files    25 ❌

For more details on these failures, see this check.

Results for commit bec7666.

♻️ This comment has been updated with latest results.

[windmgc]

Just FYI, we might need to adjust the validation logic after upgrading SDK version.(I've tried before in #72, but didn't get back to this since then)

And a related ticket: https://konghq.atlassian.net/browse/KAG-3834

[tysoekong]

@windmgc Yeah I wondered why the tests randomly failing, I can look at that shortly if you want.

Do you know why string.buffer package is missing? I can't find its dependency in Luarocks, but I think it's something to do with ffi dependency?

I am totally prepared to re-test by hand the AWS vault, auth, and other stuff too btw in Kong(/EE) if you want me to...

[tysoekong]

@Tieske Yeah I missed up the release process completely, now the branch is stuck.

Ignore it, check this one instead.

[windmgc]

Do you know why string.buffer package is missing? I can't find its dependency in Luarocks, but I think it's something to do with ffi dependency?

String buffer does not exist in older LuaJIT versions, in the lua-resty-aws library's CI we test against OpenResty 1.17.8.2 which was released 4 years ago and may not have built-in string buffer in LuaJIT

[Tieske]

This is a tricky change. We have way too little test coverage over different AWS services to find potential issues.

wdyt @windmgc ?

[windmgc]

I'm afraid so... I do not have much confidence in upgrading it directly

[DiscreteTom]

Should this be if response.headers["Content-Type"] == "application/vnd.amazon.eventstream" then?

[DiscreteTom]

I fixed this locally in my test environment and applied other changes from f0ac80e , works like a charm in my lambda streaming case.

[tysoekong]

@DiscreteTom I have made changes to the header format in the stream handler, otherwise we are wasting CPU by scanning for "header_key == x"

Can you check your plugin code again?

Minor change, now headers are:

{
  body = "...",
  headers = {
    key = "value",
    key2 = "value2",
  },
}

[DiscreteTom]

No problem. I updated my code locally against this change and it is ok. Gonna push to my PR when this lib is released.

[DiscreteTom]

Any update on this?