Skip to content
Backstop
Esc
↑↓navigate↵open⌘Jpreview
On this page

Concurrency & Scale Limits

What running in your own process means for the GIL, and where the concurrency ceiling actually is.

Backstop runs in your process, so it inherits Python’s execution model. This page documents where that helps, where it bites, and how to stay safe.

The GIL ceiling

Under CPython, all Backstop logic (budget reservation, circuit-breaker state transitions, retry/backoff) executes under the global interpreter lock. That is usually a non-issue — these are microsecond-scale in-memory operations and the real cost is waiting on the provider’s network I/O, which does release the GIL. But it means:

  • 10 / 20 / 50 concurrent wrap() sessions in one process is fine for typical request workloads — they spend nearly all of their time awaiting the model API, not holding the lock. The exact fraction is workload-dependent and is not something this repo measures, so treat it as a qualitative claim, not a number.
  • Hundreds of CPU-bound, synchronous sessions in one process will serialize and degrade tail latency. This is the same ceiling every in-process library (and every async framework) faces; it is not a Backstop bug.

Takeaway: if you need very high fan-out, run one process per agent / per tenant rather than one giant process. Backstop is designed for exactly this — each wrap() is a single line and is isolated.

Configurable ceiling

To keep the ceiling from being a surprise, set a soft cap. When live wrap() sessions exceed max_wrap_sessions, Backstop emits a UserWarning instead of silently degrading:

from openai import OpenAI
from backstop import Backstop, BackstopConfig

# Warn if more than 64 live sessions accumulate in this process.
config = BackstopConfig(max_wrap_sessions=64)
client = Backstop.wrap(OpenAI(), budget=50_000, config=config)

max_wrap_sessions=0 (the default) disables the check.

Shared budget across replicas

For multi-replica / multi-process deployments, don’t scale threads — scale processes and share one budget via Redis (Tier 1 / P1):

pip install "backstop-ai[redis]"
from openai import OpenAI
from backstop import Backstop, BackstopConfig

client = Backstop.wrap(
    OpenAI(),
    budget=1_000_000,
    config=BackstopConfig(shared_budget=True, redis_url="redis://localhost:6379"),
)

Redis operations use atomic Lua scripts, so N processes decrement the same cap without a race. This is the recommended path for “AI SaaS with runaway spend” and keeps the per-process GIL ceiling irrelevant to correctness (only to throughput).

Thread safety

  • In-memory budget + circuit state use threading.Lock, so they are safe to share across threads in one process.
  • The async transport uses asyncio locks; do not mix the sync and async clients on the same wrapped object.
  • BackstopConfig is immutable after construction (validated once). Reuse one instance across sessions.

When to use a proxy gateway instead

If you need per-tenant request routing, key vaulting, or multi-language services behind one endpoint, a proxy is a better fit. Backstop itself ships an optional one — backstop serve --target https://api.openai.com/v1, needing the fastapi extra — which runs the same policy engine in front of a real network hop. That is a genuinely different deployment from wrap(): it adds latency, a process to operate, and a place payloads transit through. Reach for it when wrapping every client construction site is impractical or the policy must be non-bypassable, and stay in-process otherwise.

Backstop’s wedge for single-language Python services is that you get the same controls with zero infra and a one-line drop-in. Pick a dedicated proxy when your topology is inherently multi-language or multi-team at the network edge.

Was this page helpful?