IPFS gateway hardening

Currently, the IPFS gateway (NGINX proxy) allows any content to be uploaded (with a 2MB max payload). If anything can be uploaded from anywhere, there are potential hazards, among them:

• spam content / listings
• inappropriate content
• illegal content

There are many ways to harden the gateway, for example:

• Formatted loglines, log handling, and some sort of log monitoring would give visibility into what is being uploaded through the gateway
• Validating uploaded data (adheres to schema, file formats for binary fields, etc)
• Authentication (ex. integration with ERC725, API keys to ensure uploads come from applications and services built by known developers, etc.)
• Spam filtering on the text content
• Image recognition for offensive content

[joshfraser]

Thanks for this @ambertch.

@DanielVF reminded me that we also need to add more redundancy to our IPFS gateway and have at least a second server that is constantly pinning our content. We're currently running a single gateway server and there's no guarantee that anyone else will ever pin our stuff without the appropriate incentive. @cuongdo can you own this?

[cuongdo]

@joshfraser Yes, I'll take on adding IPFS gateway redundancy.

@ambertch Thanks. I'll put in my comments later today.

[cuongdo]

@ambertch These are good ideas. I don't currently have access to the IPFS gateway, but these are some other ideas in order of my priority for them:

1. Adding a second IPFS gateway
2. Have IP-based ACLs for certain operations, such as pinning and unpinning. I'm not aware of a legitimate use for someone else pinning (outside of the auto-pinning that happens when objects are added) or unpinning stuff on a public gateway. Taking this further, there could be just a set of whitelisted operations. I'm a little worried that the various features of the IPFS gateway could make it a vector for attacks.
3. Basic monitoring for low disk space, etc.
4. Have a "shadow" IPFS server whose only job is to have pinned copies of IPFS objects that correspond to listings. No external access would be allowed. This ensures that there's always one IPFS server that has the content that's not exposed to external attacks.
5. Rate-limiting of external requests or some other form of DOS/DDOS protection

Of the ideas @ambertch suggested, I believe that adding better logging and validating uploaded data are the short-term priorities.

It sounds like besides application development, provisioning has to be done for the gateway server (configure logging, install agents for monitoring and alerting, writing the Nginx configs).

One option for provisioning is a system like ansible (python), which can deploy changes to all servers as well as provision/configure servers identically. Tags and variables in ansible allow for variations in deploys, ex. if the shadow server is mostly the same as the gateways but has a different setting to make it a read replica of sorts

The following issues track the above:

1. Adding a second IPFS gateway (Migrate to dual IPFS gateways origin-devops#5, Synchronize content between IPFS servers origin-devops#2)
2. Have IP-based ACLs for certain operations, such as pinning and unpinning (TODO)
3. Basic monitoring for low disk space, etc. (Basic monitoring for IPFS servers origin-devops#1)
4. Have a "shadow" IPFS server whose only job is to have pinned copies of IPFS objects that correspond to listings. No external access would be allowed. This ensures that there's always one IPFS server that has the content that's not exposed to external attacks.(Shadow IPFS server origin-devops#4)
5. Rate-limiting of external requests or some other form of DOS/DDOS protection (DDOS protection origin-devops#3)

For the setup and configuration of the servers, OriginProtocol/origin-devops#10

In addition, profiling of upload and download times as well as optimization of those: OriginProtocol/origin-devops#9