Celery with Upstash Redis

You can use Celery with Upstash Redis to build scalable and serverless task queues. Celery is a Python library that manages asynchronous task execution, while Upstash Redis acts as both the broker (queue) and the result backend.

Setup

Install Celery

To get started, install the necessary libraries using pip:

pip install "celery[redis]"

Database Setup

Create a Redis database using the Upstash Console. Export the UPSTASH_REDIS_HOST, UPSTASH_REDIS_PORT, and UPSTASH_REDIS_PASSWORD to your environment:

export UPSTASH_REDIS_HOST=<YOUR_HOST>
export UPSTASH_REDIS_PORT=<YOUR_PORT>
export UPSTASH_REDIS_PASSWORD=<YOUR_PASSWORD>

You can also use python-dotenv to load environment variables from a .env file:

.env

UPSTASH_REDIS_HOST=<YOUR_HOST>
UPSTASH_REDIS_PORT=<YOUR_PORT>
UPSTASH_REDIS_PASSWORD=<YOUR_PASSWORD

Example Application

Setting up Celery with Upstash Redis

tasks.py

import os
from celery import Celery
from dotenv import load_dotenv

load_dotenv()

# Configure Celery with Upstash Redis
UPSTASH_REDIS_HOST = os.getenv("UPSTASH_REDIS_HOST")
UPSTASH_REDIS_PORT = os.getenv("UPSTASH_REDIS_PORT")
UPSTASH_REDIS_PASSWORD = os.getenv("UPSTASH_REDIS_PASSWORD")

connection_link = f"rediss://:{UPSTASH_REDIS_PASSWORD}@{UPSTASH_REDIS_HOST}:{UPSTASH_REDIS_PORT}?ssl_cert_reqs=required"

celery_app = Celery("tasks", broker=connection_link, backend=connection_link)

@celery_app.task
def add(x, y):
    return x + y

Note that we should use the rediss:// protocol to connect to redis over TLS and set ssl_cert_reqs=required to enforce certificate validation.

Running the Worker

Start the Celery worker to execute tasks:

celery -A tasks worker --loglevel=info

Using the Task

You can now use the add task to perform background computations:

main.py

from tasks import add

result = add.delay(4, 6)
print(f"Task state: {result.state}")  # Outputs 'PENDING' initially

# Wait for the result
output = result.get(timeout=10)
print(f"Task result: {output}")  # Outputs '10'

Conclusion

To see a more detailed example of using Celery with Upstash Redis, check out the Job Processor with Celery example on our website.

For more details on Celery, visit the Celery Documentation. For Upstash Redis, check out the Upstash Redis Documentation.