Bug: S3Proxy does not support images.

[mhaley-miovision]

Hello,

I set up a very simple lambda to proxy requests from s3. It uses the aws serverless express library to setup a basic express app, and then I leverage this library to handle the proxy. It is a very small app, here is the basics of it:

const awsServerlessExpress = require('aws-serverless-express');
const express = require('express');
const S3Proxy = require('s3proxy');
const app = express();
const proxy = new S3Proxy({ bucket: config.bucket });
proxy.init();

app.route('/*').get(proxyToS3);

function proxyToS3(req, res) {
         proxy.get(req,res).on('error', () => res.end()).pipe(res);
}

const server = awsServerlessExpress.createServer(app);
exports.handler = (event, context) => awsServerlessExpress.proxy(server, event, context);

The problem is that lambda does not support streaming binary like data. So if I ever try to proxy an image it gets corrupted and the browser can never display it. To get around this I now do an s3 head request first to determine the type of object, and then updated my proxyToS3 function to look like

function proxyToS3(req, res) {
    <do s3 head request to get metadata>

    if (metadata.ContentType.match(/image/)) {
        const params = {
            Bucket: config.bucket,
            Key: req.url
        };

        s3.getSignedUrl('getObject', params, function (err, url) {
            if (err) {
                res.end();
            } else {
                res.redirect(url);
            }
        });
    } else {
         proxy.get(req,res).on('error', () => res.end()).pipe(res);
    }
}

What are people's thoughts about including a redirect functionality within the proxy that either we can call directly, or we can configure to tell it to redirect on certain object types instead of actually proxying?

Thanks,

Mark

[gmoon]

A couple questions:

1. Are you using API Gateway to front your Lambda?
2. Is redirect the preferred solution or just a workaround because serving images is not working?

[mhaley-miovision]

1.My Lambda is behind an ALB right now.
2. Redirect is my workaround.

[gmoon]

This is not an issue with S3Proxy, it is an issue with lambda/aws-serverless-express.

aws-serverless-express supports base64 encoding binary types (that you have to specify) using an array passed to createServer. It is not documented in the README but you can refer to this issue:

CodeGenieApp/serverless-express#99

I implemented an AWS Serverless Application Model (SAM) example in the branch above, in the examples/sam-app directory. It has a README that explains how I tested it.

This is using API Gateway, and not an ALB. I am not (yet) sure if ALB can accept a base64 encoded response from Lambda and do the right thing. API Gateways have some advantages over ALB but it comes with some tradeoffs (possibly cost depending on usage).

Please let me know if this is sufficient to close this ticket.

I can see redirecting to a signed S3 URL being advantageous in some use cases, since Lambda appears to need to load the binary object into memory to do the base64 encoding before passing it along. For large binaries, this is not ideal. If you would like to consider adding this as a feature (rather than a workaround), please open another issue.

Let me know your feedback and thanks for using this module.

[mhaley-miovision]

I see. It seems that what you have mentioned will not work with ALBs because I don't think there is a way to configure the ALB to transform the response like api gateway does.

I'll open another issue about the redirect option.

[gmoon]

It does appear that ALBs can handle base64 encoded binary data:

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#respond-to-load-balancer

Have you tried it with the aws-serverless-express base64 encoding?

[mhaley-miovision]

If that is the case, this is still an issue in the aws lambda express app is it not?

[gmoon]

Yes, it is, but I wanted to capture whether it was a viable solution with ALB or not.