Implement zero downtime upgrades

[mperham]

Research possibilities as noted here: https://wutch.medium.com/zero-downtime-api-in-golang-d5b6a52cc0ed

[toddkazakov]

@mperham we're interested in using the enterprise version of Faktory for handling async workloads. Zero downtime upgrades are a must, because we have some long-running jobs (e.g. 30 minutes) and it's not be viable to halt accepting new jobs while existing ones finish.

I see that you're already working on Faktory 2. Are zero downtime upgrades on the roadmap?

[profsmallpine]

@mperham We have hit a point with our product that we will need no downtime deploys moving forward. We've allotted some time to look into this for our faktory/ECS setup. Have you had any ideas since you first proposed this in 2021? We are happy to do some work here to improve the deploy scenario. We may also be able to solve this on the client side. The goal would be able to avoid taking down our web server that is enqueuing job from webhooks that we receive.

[mperham]

Faktory's protocol is stateful, there's an authentication back-and-forth, so you can't just switch servers under the covers. The client libraries should try to reconnect if an error is raised though. You can try something like HAProxy but I believe it's simpler to run two servers and two clusters of workers in parallel. Restart your app processes to connect to the new server and let the old server and its workers tail off slowly. Does that make sense?

[mperham]

I'm talking about upgrading the Faktory server itself. You seem to be talking about app process deploys using a Faktory client library?

[profsmallpine]

Yes, I am also talking about the faktory server itself. ECS often has maintenance windows, and we have to recycle the ECS instance. We take the web app down (it's the only part of our infra that does enqueuing once no jobs are running). I am trying to solve for these maintenance windows and then any time we update the faktory server as well.

I had not considered running two faktory instances. That is an interesting idea.