A Celery-style task queue for Rust

selder

A Celery-style distributed task queue for Rust. Handlers are plain async functions with extractors (think Axum), middleware is Tower layers, and application state is typed.

Python makes you choose: prefork for low memory, threads or gevent for concurrency. Selder gives you both: one lean async process, thousands of concurrent tasks, and a single shared heap.

Read the docs GitHub

Memory

Prefork-class memory profile

One Rust process shares a single heap across every task, instead of forking a pool of full Python interpreters that each carry their own copy of everything.

Concurrency

Actual concurrency

Thousands of tasks make progress on one async runtime. No GIL, no pool size to tune, no monkey-patching: concurrency is the default, not a mode you opt into.

Handlers

Extractors, not decorators

Task handlers are ordinary async functions; arguments arrive through extractors like Json<T>, so handlers read like Axum route handlers and compose like normal Rust.

Middleware

Tower layers

Built-ins include .trace(), .logging(), .sentry() and .timeout(d); anything implementing Layer<BoxedService<S>> drops straight into the worker builder.

State

Typed application state

Pass any Clone + Send + Sync context (a database pool, an HTTP client, config) to the worker and access it from handlers. No global singletons.

Interop

Speaks Celery on the wire

Tasks travel as JSON over RabbitMQ (AMQP) and results land in Redis in Celery's format, so Python producers can send work to Rust workers and read the results back.

A quick taste

A complete program, no services required

This worker and client run against the in-memory broker and results backend, so the whole thing executes with nothing to install. Point the same code at AmqpBroker and RedisResultBackend when you are ready for real work.

Runnable examples 

use selder::backend::MemoryResultBackend;
use selder::broker::MemoryBroker;
use selder::client::Client;
use selder::error::TaskResult;
use selder::extract::json::Json;
use selder::worker::WorkerBuilder;
use serde::{Deserialize, Serialize};
use std::time::Duration;

#[derive(Deserialize, Serialize)]
struct AddArgs {
    a: i32,
    b: i32,
}

async fn add(Json(args): Json<AddArgs>) -> TaskResult<i32> {
    Ok(args.a + args.b)
}

#[tokio::main]
async fn main() {
    // The broker carries tasks from producers to workers.
    let broker = MemoryBroker::new();

    // A results backend is where the worker stores task outcomes so the
    // client can retrieve them later. Both sides need a handle to it.
    let results = MemoryResultBackend::new();

    // The worker runs the handlers. The second argument is your app state,
    // anything Clone + Send + Sync: a database pool, an HTTP client, config.
    // Here the task needs nothing, so it is the unit type.
    let worker = WorkerBuilder::new(broker.clone(), ())
        .default_queue("math")
        .route("add", add)
        .result_backend(results.clone())
        .trace()
        .build();
    tokio::spawn(async move {
        worker.run().await.unwrap();
    });

    // The client is the producing side.
    let client = Client::new(broker)
        .default_queue("math")
        .with_result_backend(results);

    let result = client.send("add", &AddArgs { a: 2, b: 3 }).await.unwrap();
    let sum: i32 = result.get(Duration::from_secs(5)).await.unwrap();
    println!("2 + 3 = {sum}");
}

The Python tradeoff

With Python you choose tradeoffs. With selder you have them all.

Celery makes you pick a pool, and every pool gives something up. Selder is built on async Rust, so the things pool choice trades against stop being traded.

Celery + prefork

Low memory, little concurrency

Processes are isolated and reasonably cheap, but each concurrent task needs its own full interpreter. Concurrency is capped by how many processes you can afford to keep resident.

Celery + threads

Shared memory, serialized by the GIL

Threads share one heap, but the GIL serializes Python bytecode. I/O-bound tasks overlap; anything CPU-bound still runs one at a time.

Celery + gevent

Green threads, monkey-patched

Cooperative concurrency works well until a C extension blocks the event loop, and every dependency has to survive the monkey-patching.

selder

Low memory and real concurrency

One lean async process: tasks are lightweight enough to run thousands concurrently while sharing a single heap. The tradeoff disappears because there is no pool to choose.

In practice

Define, wrap, send

A complete program runs with the in-memory broker and results backend and nothing to install. Swap in AmqpBroker and RedisResultBackend when you are ready for real work.

  1. Define a handler

    A plain async fn; arguments arrive through extractors like Json<AddArgs>, results return as TaskResult<T>.

  2. Wrap it in a worker

    Attach middleware with Tower layers (.trace(), .timeout(d)) and typed state with .context(). Point it at a broker.

  3. Send tasks from anywhere

    The client sends JSON over the broker; results land in the backend in Celery's format, readable from Python too.

Stop choosing tradeoffs.

Early days, but the core works: brokers, retries, timeouts, tracing, and the results backend. See the repository for runnable examples and the test suite.

Read the docs GitHub