Celery Redis Poll Backend

A specialized Redis backend for Celery that disables the default pub/sub mechanism for task result retrieval.

This enables polling based approach by simply calling ready() method on Celery's AsyncResult.

Why Polling Instead of Pub/Sub?

The default Celery Redis backend uses Redis pub/sub for real-time task result notifications. While pub/sub provides immediate updates, it can face challenges in certain scenarios:

• Deadlocks in highly concurrent/multi-threaded workloads due to single-threaded nature of Redis and Celery clients.
• Higher overhead with SUBSCRIBE channels.

This backend replaces SUBSCRIBE based backend with simple setter. This means that you need to check ready() == True when getting result.

Features

• Compatible with Existing Code: Drop-in replacement for the standard Redis backend
• Resource Efficient: Reduces Redis memory usage by eliminating pub/sub channels

Installation

pip install celery-redis-poll

After installation new backends are automatically registered with Celery:

• redis+poll
• rediss+poll
• redis+cluster_poll
• rediss+cluster_poll

Usage

Configure your Celery application to use the polling backend:

from celery import Celery

app = Celery('your_app',
             broker='redis://localhost:6379/0',
             backend='redis+poll://localhost:6379/0')

For clustered Redis, use redis+cluster_poll.

Requirements

• Python >= 3.7
• Celery >= 5.0.0
• Redis >= 4.5.0
• celery-redis-cluster >= 0.1.6

Development

For development, install extra dependencies:

pip install celery-redis-poll[dev]

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.