Background Tasks Guide

Note

This wiki page is no longer maintained. The Background Tasks documentation has moved to the official docs site: https://rage-rb.dev/docs/deferred

---

Rage::Deferred is an in-process background job queue designed to offload long-running tasks from the request processing pipeline. This allows requests to be served to clients faster, without waiting for these tasks to complete.

All tasks pushed to Rage::Deferred are written to a write-ahead log. This log enables Rage to replay tasks if the server restarts or crashes. During a restart, Rage will also wait for up to 15 seconds to allow any currently in-progress tasks to complete.

Tasks are executed in fibers using Rage's fiber scheduler, so despite running in the same process, Rage::Deferred is much more efficient than thread-based background job processors.

The write-ahead log is disk-based by default, which means Rage::Deferred can be used without any initial setup. Future versions will include support for Redis and SQL-based logs.

Defining Tasks

To create a deferred task, define a class that includes the Rage::Deferred::Task module and implements the perform method:

class SayHello
  include Rage::Deferred::Task

  def perform(name:)
    Rage.logger.info "Hello, #{name}!"
  end
end

Enqueuing Tasks

To schedule a task for execution, use the enqueue method:

SayHello.enqueue(name: "World")

This call stores the serialized task in the write-ahead log and returns immediately. Once Rage has available capacity (e.g., after the request has finished or made a blocking call), the task will be executed. All logs produced within the task will be tagged with the ID of the request from which the task was enqueued.

Tasks can also be scheduled to execute later using the delay and delay_until options:

SayHello.enqueue(name: "World", delay: 10)
SayHello.enqueue(name: "World", delay_until: Time.now + 10)

Additionally, you can use Rage.deferred.wrap to push custom objects and classes that do not include the Rage::Deferred::Task module to the queue:

class SayHelloService
  def self.call(name:)
    Rage.logger.info "Hello, #{name}!"
  end
end

# execute immediately
SayHelloService.call(name: "World")

# push to the deferred queue
Rage::Deferred.wrap(SayHelloService).call(name: "World")

In the event of a failure, Rage will retry deferred tasks up to 5 times with exponential backoff.

Backpressure

In rare cases, the queue might grow uncontrollably. Normally, this isn't a concern; running in the same process, Rage naturally balances between processing incoming requests and deferred tasks. This makes scaling your application very straightforward: if response times increase, it means Rage is spending more time processing deferred tasks, and more servers should be added.

However, if each request generates a substantial number of deferred tasks, the backlog can grow faster than it is processed. To prevent this, Rage::Deferred can be configured to apply backpressure when the queue exceeds a specific number of tasks. Backpressure means that enqueuing new tasks will block until the queue size decreases. If the queue doesn't reduce enough within a specified interval (2 seconds by default), the Rage::Deferred::PushTimeout exception will be raised.

# enable backpressure
Rage.configure do
  config.deferred.backpressure = true
end

# the `enqueue` call can now block for up to 2 seconds
def create
  SayHello.enqueue(name: "World")
rescue Rage::Deferred::PushTimeout
  head 503
end

Use Cases

Thanks to executing tasks in fibers, Rage::Deferred is an ideal choice for I/O-heavy background tasks. Some potential use cases include:

• Communicating with slow or unreliable APIs in the background.
• Streaming updates to upstream systems that process data slower than it arrives.
• Sending email notifications.
• Performing data synchronization with external services.
• Generating reports.

Running in the same process also simplifies application monitoring. With Rage::Deferred, background tasks are an inherent part of the request workflow, eliminating the need to monitor and scale separate background processes. The default disk-based write-ahead log simplifies setup, allowing you to start using the queue immediately without the need to provision a separate database.

On the other hand, a separate background job processor might be more effective if your deferred tasks involve heavy computations. Running CPU-bound tasks in Rage::Deferred would mean your process has less time to handle incoming requests.

Rage::Deferred might also not be the best choice if you schedule a large number (at least over 10,000, see benchmarks) of tasks to run far in the future. Rage stores all deferred tasks in RAM, and such use case could lead to increased memory usage and processing times, as Rage would need to copy all delayed tasks to a new write-ahead log each time the log is rotated.

Benchmarks

The following benchmarks were performed on an AWS EC2 m5.large instance using Ruby 3.4.5 + YJIT.

Enqueuing 500,000 tasks

Source Code

# app/tasks/load_task.rb
class LoadTask
  include Rage::Deferred::Task

  @@count = 0

  def perform
    @@count += 1

    if @@count == 500_000
      Rage.logger.info "Tasks completed at #{Time.now.to_f}"
    end
  end
end

# app/controller/application_controller.rb
require "benchmark"

class ApplicationController
  def index
    time_to_enqueue = Benchmark.realtime { enqueue_tasks } * 1_000
    Rage.logger.info "Time to enqueue: #{time_to_enqueue}"
    Rage.logger.info "Enqueued tasks at #{Time.now.to_f}"
    head :ok
  end

  def enqueue_tasks
    500_000.times do
      LoadTask.enqueue
    end
  end
end

• Time to enqueue: 3.4 seconds
• Time to process: 6.25 seconds
• Throughput: 80,000 tasks/sec

Creating 10,000 tasks delayed by one hour

Source Code

# app/tasks/load_task.rb
class LoadTask
  include Rage::Deferred::Task
end

# app/controller/application_controller.rb
require "benchmark"

class ApplicationController
  def index
    time_to_enqueue = Benchmark.realtime { enqueue_tasks } * 1_000
    Rage.logger.info "Time to enqueue: #{time_to_enqueue}"
    head :ok
  end

  def enqueue_tasks
    10_000.times do
      LoadTask.enqueue(delay_until: Time.now + 3600)
    end
  end
end

• Time to enqueue: 526 ms
• RAM usage: 67 MB
• WAL size: 938 KB