Large ArrayBuffers (~20MB) passed through RPC crash Beaker

[kickscondor]

Hi, I've been working on a mixtape-sharing site that runs on Beaker (duxtape.kickscondor.com) and generally it works quite well. So, thank you!

There are issues, however, with large files that are added to the mixtape. I have tried to get familiar with Beaker's code before reporting this - but there is a lot going on! My stack trace seems to show that the crash happens in pauls-electron-rpc - at valueToIPCValue. It's an out of memory error.

Uploading large files through the Beaker library doesn't exhibit this problem: that page uses the internal importFromFilesystem call.

Options:

• Expose something similar to importFromFilesystem but which can work with the browser's File or Blob API. Could this be done securely?
• Stream data through RPC - I started looking at adding this, but it will take me some time to understand the RPC layer.

[kickscondor]

It seems that I am also experiencing this same crash when using the globalFetch API with large ~20MB files. Again, seems to be happening in valueToIPCValue. As I'm beginning to understand the RPC layer better, I think the problem is that a promise is returned from both of these methods - however, there are streams within the objects returned by these promises. These streams are being serialized into buffers, which loads the entire thing into memory.

I think valueToIPCValue needs to wrap promises and streams in RPC handlers - but I'm uncertain what the ramifications would be.

[pfrazee]

Yeah. Basically we need to be leveraging streams both internally and as available APIs.

[kickscondor]

pfrazee/pauls-electron-rpc#6