ShipRust runs Rust background jobs through a Postgres-backed queue shared by the website and every generated project. Jobs are JSON payloads in a jobs table. Workers claim due rows with FOR UPDATE SKIP LOCKED, execute registered Rust handlers, and record success or schedule a retry. No Redis or Postgres extension is required.

The important part is the failure boundary. Saving a job inside your application transaction can make the data change and enqueue atomic. It cannot make a later email or third-party API call happen exactly once.

This walkthrough follows the code at revision 01a67e7, reviewed October 10, 2026. It explains the implementation, the supplied examples, and the guarantees you should preserve when adding your own tasks. Code excerpts belong to this ShipRust revision, which uses SQLx 0.9 and Tokio 1.53; they are not a standalone queue library.

In this article

Choose what belongs in a job
Commit application data and jobs together
Claim work without holding a transaction open
Define a Rust job handler
Retries, leases, and duplicate execution
Run and operate workers
Test the boundaries and know the trade-offs

Choose what belongs in a Rust background job

A welcome email can wait until after signup. A password check cannot: the HTTP response depends on its result. Moving both out of a request would obscure that distinction.

ShipRust uses four execution paths:

Work	Where it runs	Reason
Form validation and authentication decisions	Request handler	The response needs the answer.
Password hashing	Bounded CPU helper	Expensive work must not occupy async workers directly.
Welcome emails and purchase receipts	Durable job queue	Follow-up work should survive process restarts.
Cheap, idempotent periodic cleanup	Tokio interval	A missed cleanup tick can wait for the next one.

A tokio::spawn call schedules a future, but does not persist it. Tokio's spawn documentation explicitly says spawned tasks are not guaranteed to finish when the runtime shuts down. Our queue still uses Tokio to execute handlers; Postgres supplies the durable record of work remaining.

The distinction is visible in ShipRust's authentication flow. Sending a verification code stays in the synchronous signup path, with its delivery result and resend budgets. The generated project's welcome email is queued after successful verification, inside the transaction that creates the user.

Similarly, the site handles Stripe entitlement updates synchronously in the webhook transaction. Only the purchase receipt becomes a job. Sending the receipt does not block the webhook on the email provider.

Commit application data and jobs together

Consider a purchase that should grant access and send a receipt. Two separate writes leave a gap: the entitlement could commit, then the process could die before recording the receipt job.

ShipRust inserts the receipt job into the same Postgres transaction as fulfillment. Here is the enqueue call from app/src/payments.rs, inside that existing transaction:

shiprust_core::jobs::enqueue(
    &mut *tx,
    &crate::jobs::SendPurchaseReceipt { user_id: attempt.user_id },
    shiprust_core::jobs::Enqueue {
        dedupe_key: Some(attempt.user_id.to_string()),
        ..Default::default()
    },
)
.await?;

The surrounding code locks the account, checks whether this is its first fulfillment, updates the entitlement and checkout state, then commits with tx.commit().await?.

If the transaction rolls back, the job rolls back too. Other connections cannot claim the new job before commit. This is the useful guarantee: the database change and the durable intention to send a receipt succeed or fail together. Delivery happens later, outside that transaction.

The transaction ends before the handler starts. A failed attempt can return to the queue; exhausted or discarded work becomes dead.

The enqueue API also accepts a pool or connection. Use &pool for standalone work, but do not assume that it joins a transaction you opened elsewhere. Atomicity comes from passing that transaction's executor, &mut *tx.

What the dedupe key actually prevents

The migration defines this partial unique index:

CREATE UNIQUE INDEX jobs_dedupe
ON jobs (kind, dedupe_key)
WHERE dedupe_key IS NOT NULL
  AND state IN ('queued', 'running');

While a job with that kind and key is queued or running, another matching enqueue returns None. Finished and dead jobs release the key. Different job kinds can use the same key.

That is live enqueue deduplication, not permanent business-event deduplication. For purchase receipts, the separate first-fulfillment check prevents repeated webhook events from creating a fresh receipt after the previous job has finished. Neither mechanism makes an individual email attempt safe from every crash window.

How the Postgres queue claims work

The queue lives in crates/core/src/jobs/ in a generated project, under template/ in the ShipRust repository. The website imports this same core crate by path.

Each job row carries its kind, JSONB payload, state, earliest run time, attempt budget, and lease ownership. Partial indexes support due-job claims, expired leases, retention, and live deduplication.

Here is the claim statement from queue.rs, formatted for readability:

UPDATE jobs
SET state = 'running',
    locked_by = $1,
    locked_until = now() + make_interval(secs => $2),
    attempts = attempts + 1
WHERE id = (
    SELECT id
    FROM jobs
    WHERE state = 'queued'
      AND run_at <= now()
      AND kind = ANY($3)
    ORDER BY run_at
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
RETURNING id, kind, payload::text AS "payload!", attempts, max_attempts

$1 is the worker identity, $2 the lease duration in seconds, and $3 the registered job kinds. The "payload!" alias is SQLx's non-null override for the returned payload.

The subquery locks one eligible row. Another claimant skips a conflicting row lock instead of waiting on it. The outer update records ownership and increments attempts in the same statement. PostgreSQL documents queue-like tables as a use case for SKIP LOCKED, while warning that it does not provide a generally consistent view of the data.

Crucially, the worker does not hold the claim transaction open while calling an email provider. PostgreSQL's row locks end with the transaction. After the claim, the stored lease is the recovery mechanism.

ORDER BY run_at prefers earlier eligible work. It does not promise strict FIFO execution or completion: workers skip locked rows, run concurrently, and take different amounts of time. A future run_at is an earliest start time, not a delivery deadline.

Define and register a Rust job handler

A job is a serde payload implementing Job. The generated project's app/src/jobs.rs supplies SendWelcomeEmail; this shortened excerpt keeps its actual control flow and omits the message wording:

use crate::SharedState;
use app_core::jobs::{Context, Job, JobError};

#[derive(serde::Serialize, serde::Deserialize)]
pub struct SendWelcomeEmail {
    pub user_id: uuid::Uuid,
}

impl Job for SendWelcomeEmail {
    const KIND: &'static str = "send_welcome_email";
    const MAX_ATTEMPTS: u32 = 5;
    type State = SharedState;

    async fn run(self, state: SharedState, cx: Context) -> Result<(), JobError> {
        let Some(mail) = &state.email else { return Ok(()) };
        let Some(user) = sqlx::query!(
            "SELECT email FROM users WHERE id = $1", self.user_id
        )
        .fetch_optional(&cx.pool)
        .await? else { return Ok(()) };

        mail.send_message(&user.email, "Welcome", "Your account is ready.")
            .await?;
        Ok(())
    }
}

Register it in the existing jobs::registry() using Registry::new().register::<SendWelcomeEmail>(). Add further kinds to that registry. An unregistered kind is left queued rather than claimed by a worker that cannot run it.

The payload contains an identifier, not the recipient address or a secret. The handler looks up current account data. In the supplied example, a deleted user or absent mail configuration produces a successful no-op. Therefore, done means the handler returned success; it does not necessarily mean an email reached an inbox.

The default Job budget is ten attempts, including the first run, with a 60-second timeout. The welcome job overrides that budget to five; the site's purchase receipt uses eight.

Persisted payloads also constrain deployments. Keep KIND strings stable. Add fields with compatible defaults, such as #[serde(default)]; use a new kind for an incompatible contract and retain the old handler until its work is drained. Registry filtering helps mixed-version deployments, but cannot make incompatible payloads compatible.

Retries, leases, and the duplicate-delivery window

The worker has four stored states: queued, running, done, and dead. Handler results and lease recovery determine the next transition.

Event	Queue behavior	What it does not guarantee
Handler returns `Ok(())`	Mark the owned running row done	That a provider delivered the message to its final recipient
Retryable error or timeout	Requeue with backoff, or mark dead when attempts are exhausted	That the previous attempt had no external effect
`JobError::discard(...)` or invalid payload	Mark dead without further automatic attempts	That retrying unchanged input would help
Worker stops renewing its lease	A later sweep requeues it, or marks it dead if its budget is used	Recovery at an exact wall-clock deadline

Retry delays start with a one-second exponential base: one, two, four seconds, and so on. The base caps at 600 seconds, then the worker adds up to 10% jitter. The resulting delay can reach 660 seconds, so “a strict ten-minute maximum” would be inaccurate.

The default lease lasts 60 seconds and is refreshed every 20 seconds. If the process disappears, heartbeats stop. A worker's subsequent expiry sweep makes abandoned work eligible again, subject to the remaining attempt budget. Database availability, worker capacity, and sweep timing affect when it actually runs.

Timeouts drop the handler future. They cannot undo an API request the other system already accepted. Likewise, detecting lost lease ownership logs a warning; it does not cancel the old handler. Ownership checks protect bookkeeping against a different worker identity, not external side effects. The worker identity is reused across claims, so an earlier attempt and a reclaimed attempt in that same worker are not distinguished by a unique per-attempt token.

At-least-once processing is not exactly-once email

Imagine the provider accepts a receipt, then the process crashes before recording done. After lease recovery, the handler sends it again.

The current welcome and receipt examples do not supply a provider idempotency key. Duplicate emails remain possible. The queue's dedupe key does not close that gap because this is another attempt of the same job, not another enqueue.

For your own tasks, use a stable business-operation key at the destination when it supports idempotency, or make database-only effects repeat-safe with constraints and transactions. Merely checking a local “sent” flag before an external call still leaves a crash gap between the two systems. Choose the behavior deliberately for the effect you are performing.

At-least-once processing also does not mean eventual success: a finite retry budget can end in a dead job that needs investigation.

Run and operate workers

Every generated project includes the queue. It is not a generator checkbox. JOBS_WORKER selects the process role:

Setting	Process role
`true` (default)	HTTP server and worker together
`false`	HTTP server only; requires another worker to process jobs
`only`	Worker plus health endpoint

The worker command also selects worker-only mode. A separate container uses the same application image, not a second codebase; see the Docker deployment guide.

Default concurrency is two jobs per worker process. Idle polling is five seconds. Enqueueing also sends an in-process notification, followed by a short grace period for commit; other processes rely on polling. The notification is a latency optimization, not a durable cross-process signal or a guarantee of immediate execution.

JOBS_SHUTDOWN_SECONDS defaults to eight seconds. On shutdown the worker drains in-flight work within that budget; unfinished tasks are abandoned for lease recovery. Keep your container stop grace longer than the drain budget, and do not assume a timeout can preempt arbitrary blocking code.

Run these commands from the application's workspace with its database configuration loaded:

cargo run -- jobs stats
cargo run -- jobs retry <id>

Stats group rows by state and kind, including the oldest run_at. Investigate overdue work and dead jobs, not only total queue depth: intentionally delayed jobs may be healthy. Manual retry resets a dead job's attempts and queues it again. If a newer live job already occupies the same kind/dedupe key, the uniqueness constraint can reject that retry; inspect both before intervening.

Each attempt has a job.run tracing span with kind, ID, and attempt. Done rows become eligible for deletion after seven days; dead rows after 30 days. An hourly sweep applies retention. Export the evidence you need before relying on those rows as a long-term audit trail.

Test the boundaries and know the trade-offs

The repository's queue tests cover transactional rollback, live deduplication, retry exhaustion, delayed execution, timeouts, shutdown, and lease recovery. One test deliberately makes a lease stale while the original handler is still running and checks that the handler runs twice. That is a more useful demonstration of the delivery contract than a happy-path “one job, one email” test.

The purchase-receipt integration test uses a local mail double that first rejects delivery and then accepts a retry. It checks that failure path without sending real customer mail; it does not prove exactly-once delivery across a crash after provider acceptance.

For related concurrency-test techniques, PGSandbox’s guide to testing PostgreSQL transaction isolation shows coordinated connections and final-state assertions. It is a companion testing resource, not independent validation of ShipRust's queue.

Using Postgres avoids operating another broker, but queue traffic still consumes database connections, writes, and maintenance capacity. PostgreSQL's vacuuming guidance explains why updated and deleted rows require ongoing maintenance. This article makes no throughput claim. If your workload needs strict ordering, complex workflows, or isolation from application-database load, evaluate those requirements before adopting this design unchanged.

In a generated project, start with crates/core/src/jobs/mod.rs, queue.rs, and worker.rs; then read app/src/jobs.rs and app/content/docs/background-jobs.md. Those are also useful entry points for a coding agent: define the kind, register it, enqueue inside the right transaction, and test its repeat behavior.

Generate a ShipRust project, open the background-jobs guide, and replace the welcome-email example with a task your product needs. Decide what a repeated attempt should do before writing its happy path.