Connecting to platform-services

This page is for consumer projects — apps and dev containers that need to use the shared LLM runtime (and, eventually, the public edge proxy) provided by platform-services.

How this works

Consumer projects reach platform-services through the llm-gateway — a priority-aware async job queue that is the single entry point for both LLM (ollama) and document parsing (docling) work. The gateway API and wiring are identical regardless of which deployment you point at.

Two deployments, one of them is the default

Both run the same gateway code and expose the same API. What differs is which backends the gateway dispatches to, and that’s one env var on the platform side (LARGE_INSTANCE_IP) — not a different contract for you.

Concretely:

• Production consumer deployments run on the same Oracle host as platform-services and reach the gateway in-host.
• Local consumer development points at the OCI gateway via an SSH tunnel (covered in Pointing local dev at OCI below). This is the day-to-day dev path — the consumer code runs locally with all the usual editor/hot-reload niceties, but the actual LLM and docling work happens on real production-sized models in OCI.
• Running consumers against local platform-services is the fallback, and also the path for model evaluation. The wiring is identical to OCI (same compose, same env vars, just a different gateway URL), so flipping back and forth is one env var change on your side. The platform operator brings up local LLM/docling backends with make local-up and points the gateway at them — see Running the LLM backends on this machine in dev-setup-windows.md. Whether local output is comparable to OCI depends entirely on which profile is loaded: against small (8B-class), local dev validates wiring and not output quality; against candidate, comparing output quality is the whole point. Ask /diagnostics rather than assuming — and note that local timings are never comparable to OCI’s, since the hardware is different.

Connecting

Consumers reach platform-services through the gateway via the published host port. Your consumer doesn’t join the platform_default network — it stays decoupled from platform-services’ lifecycle and keeps starting/running when the platform is rebuilding or down.

services:
  app:
    extra_hosts:
      - "llm-gateway:host-gateway"

Behind the scenes, llm-gateway resolves via /etc/hosts to the host gateway IP; TCP connections then land on the published host port (11435).

URLs:

• llm-gateway → http://llm-gateway:11435 (private; via host-gateway + tunnel — see Pointing local dev at OCI below). If your machine also runs a local platform-services stack on :11435, the OCI tunnel binds :21435 instead to avoid the port collision — point your consumer at 21435 for OCI work. See Sharing localhost with a local platform-services for the full pattern. make tunnel from platform-services already does this; the trap is consumer URLs pinned to :11435 silently landing on the local stack instead of the tunnel.
• platform-docs → https://soggplatform.dedyn.io/ (public TLS edge — reachable from anywhere, no extra_hosts wiring or tunnel needed)

Why the two endpoints have different reachability stories: the gateway is a write-capable LLM dispatch API with no auth/rate-limit yet, so it’s deliberately kept off the public internet. Docs are read-only and safe to publish, so they’re served straight from the public TLS edge — no per-distro setup required.

Don’t hardcode these in app code — parameterize them via environment variables (LLM_GATEWAY_URL, PLATFORM_DOCS_URL). Same code then works under local dev, deployed prod, and the SSH-tunnel variant documented further down.

When platform-services is up, your consumer reaches it. When it’s down, the consumer still starts — gateway calls fail at the call site with a connection error, not at compose-up time. That’s the design intent.

What about direct ollama / docling access?

Not supported. ollama and docling are internal services of platform-services and have no consumer contract. The gateway owns all backend dispatch, including:

• Priority-aware queueing (interactive vs batch tier)
• Single global in-flight slot across both backends
• Job-state persistence across client disconnects and gateway restarts
• On-demand large-instance lifecycle: ollama and docling live on a separate OCI instance that the gateway spins up on the next job submission after it’s been idle-stopped, and stops again after LIFECYCLE_IDLE_TIMEOUT_SECONDS of no activity

If you were previously wired to ollama:11434 or docling:5005, see Migrating from direct backend access below.

What’s served

Notes:

• platform-docs is on a public TLS edge — browsers, external agents, and consumer containers all reach it the same way. No extra_hosts or tunnel needed. The same content is also served on http://platform-docs:5006/ to containers on the OCI host itself (in-host shortcut); local-dev consumers should ignore that and use the public URL.
• llm-gateway is not on the public edge — it’s a write-capable LLM dispatch API with no auth yet, so it stays private. Reach it from a consumer container via the host-gateway alias plus either a local platform-services stack or the SSH tunnel documented below.
• /models.json is proxied live from ollama’s /api/tags via platform-docs — hit this for model discovery rather than hardcoding model tags in your config.
• ollama’s :11434 and docling’s :5001 are not part of the consumer contract, and since 2026-08-19 they are not reachable from outside Docker on any deployment. See Direct backend access is gone below — if any of your code reads those ports, it stopped working on that date.

Calling the gateway

The gateway exposes a uniform async job API for both backends. Submit a job, get an ID back immediately, poll for the result. Jobs are expected to run roughly 5–15 minutes on the current hardware (longer once we cut over to 72B-class models) — long enough that holding an HTTP connection open across the work is the wrong default.

Submission shape

{
  "endpoint": "/api/chat",
  "payload": { "model": "qwen2.5:72b-instruct-q4_K_M", "messages": [...] },
  "priority": "interactive",
  "backend": "ollama",
  "timeout_seconds": 900,
  "callback_url": "https://hooks.your-app.example/gateway"
}

• endpoint — the upstream path to call on the chosen backend. For ollama: /api/chat, /api/generate, /api/embeddings, /v1/chat/completions, … For docling: /v1/convert/source/async.
• payload — the body you would have POSTed directly to the backend, verbatim (with caveats noted per-backend below).
• priority — optional. interactive or batch. Omitted → recorded as batch. See Choosing a priority for your job below. An unknown value (e.g. urgent) is rejected with 400 rather than silently falling back to batch.
• backend — optional. ollama or docling. Omitted → the gateway derives it from the endpoint path (/v1/convert/* → docling, anything else → ollama). If you supply backend explicitly and it contradicts the endpoint (e.g. endpoint=/api/chat + backend=docling), the submit is rejected with 400 — there is one unambiguous dispatch path per job, and the gateway loud-fails rather than silently routing to the wrong backend.
• timeout_seconds — optional. A positive integer: how long this job is worth running once it reaches the backend. Omitted → only the deployment-wide cap applies. It measures exactly the same interval the deployment cap does — see Timeouts: which clock measures what below, and read that section before choosing a number, because the interval is narrower than “time since I submitted”. A value above your backend’s cap is rejected with 400; the gateway will not silently clamp it.
• callback_url — optional. An https URL the gateway POSTs a signed notification to when the job reaches a terminal state (completed or failed), so you can be push-notified instead of polling. Omitted → no callback, behaviour unchanged. Requires per-caller setup (a shared secret + host allow-list registered with the operator). See Completion webhooks below for the full contract. A callback object ({"url": "..."}) is also accepted as a forward-compatible alternative to the flat field.

Choosing a priority for your job

Set priority on the submit body to one of:

• interactive — drained ahead of any queued batch work. For human-facing, latency-sensitive calls (editor assistant, chat UI).
• batch — drained when no interactive work is queued. For bulk ingestion, indexing, scheduled jobs. This is the default when priority is omitted.

Tier is per-job, not per-caller: a single consumer can mix interactive and batch submissions freely. There is no central caller→tier mapping; the gateway trusts whatever priority value the consumer declares (the trust model assumes a small number of fully-trusted consumers on the internal network).

Set the X-Caller-Id request header to a short, stable identifier for your consumer project (e.g. my-editor-assistant). This is recorded on every job and surfaces in /v1/jobs/{id} and the gateway log — useful for tracing your own traffic and for the operator when debugging — but it does not drive tier assignment. If the header is absent, the gateway falls back to the peer IP as the caller identifier; don’t rely on the fallback, it’s a backstop.

Preemption is not supported. An interactive job submitted while a batch job is in flight waits for that batch job to finish (can be 5–15 min for docling work, longer for big inference). Once the in-flight slot frees, the interactive job dispatches before any queued batch. If sub-batch-duration latency matters for your use case, raise it — the gateway design will need to change.

Single global in-flight slot

The gateway runs one job at a time globally across all backends. An ollama job in flight blocks a docling dispatch and vice versa; two queued jobs of any combination run sequentially.

Why one slot, not one per backend: the on-demand large instance is CPU-bound on both workloads (no GPU on the free-tier ARM shape), so running ollama and docling in parallel halves each one’s throughput. Serial dispatch with priority tiers gives each job full CPU and keeps the queue model simple. Rationale and the trade-off in full: plans/priority-queueing.md Queue shape section.

The queue_position returned on submission counts every queued job ahead of yours globally — 1 means “next to run,” regardless of which backend any of the queued jobs target.

One heavyweight backend at a time

Local deployment only, new 2026-08-18. On a RAM-tight dev box the platform keeps at most one heavyweight tenant in memory: an ollama model, or docling. When a job needs the one that isn’t loaded, the gateway evicts the resident one and brings up the one the job needs — then dispatches. The OCI deployment does not do this and is unchanged (96 GB fits both, so there is nothing to trade).

What changes for you: a docling job no longer fails just because docling is stopped.

before:  submit docling job → docling is stopped → job FAILED
                                                   "docling unreachable: …"

after:   submit docling job → docling is stopped → gateway starts it
                                                 → job waits ~15 s
                                                 → job runs normally

This is the fix for the 2026-08-18 outage in which eight jobs failed in 90 seconds because docling had been stopped between batches. Nobody has to announce a batch boundary any more, on either side.

A job waiting for a backend is running, not failed. While it waits, GET /v1/jobs/{id} reports status: running with phase set to evicting_backend (freeing the other backend’s RAM) or loading_backend (starting yours and waiting for its health check). Both are normal progress. As with every phase, treat them as a hint — don’t build control flow on them.

The wait does not consume your timeout_seconds. That field bounds the dispatch, and it starts when your job reaches the backend — same rule as the on-demand large’s boot. A job with timeout_seconds: 60 still gets its full 60 seconds after docling is up.

Nothing new can fail your job. If the platform can’t bring your backend up, the dispatch happens anyway and fails with the existing docling unreachable: … — the retry-safe class you already handle. There is no new error class and no new status to add to your matcher.

Cancelling during the wait. DELETE /v1/jobs/{id} on a job that is running but hasn’t reached its backend yet answers 409 with “job is running but not yet cancellable (dispatch has not started); retry in a moment”. This is pre-existing behaviour — the same answer you’d get during an OCI boot — and the window here is seconds, not minutes.

Send the docling half of a mixed batch first. A batch that alternates conversion and inference jobs pays a backend switch at every crossing; the same batch with all its conversions submitted first pays one. This is a recommendation, not a rule, and the platform does not reorder your jobs to achieve it — priority tier still outranks everything, so an interactive job from another consumer landing mid-batch can still force a switch. Worth doing anyway: it is free, and it is what a front-loaded pipeline does naturally.

Checking what’s resident. GET /diagnostics reports a residency block: enabled (false on OCI — everything else is then inert), resident (which backend the last dispatch arranged for), switches (how many crossings this gateway has paid) and last_error.

Job lifecycle

queued ─────► running ─────► completed
   │  ▲           │
   │  │           └────► failed (backend error, timeout, cancelled
   │  │                          via DELETE, gateway restart)
   │  └── waiting_for_budget ◄── (free-tier cap; drains at month reset)
   └────► failed (cancelled via DELETE, or platform-initiated bulk cancel)

waiting_for_budget appears when the deployment runs the free-tier fuse — enabled on the production deployment since 2026-07-24, so plan for it (see “Free-tier budget cap” below). It’s non-terminal — the job loops back to running on its own once budget frees up. Treat it like queued.

If the gateway operator clears the queue (incident recovery, stuck backend, etc.), every queued job transitions to status=failed with the recorded error set to a message the operator chose — by default cancelled by platform services, but operators may include incident context (e.g. large instance OOM, restart pending). Treat this as a normal job-level failure on the consumer side: surface error to the user, resubmit if your workflow needs to retry.

The sweep reason can never impersonate a class you gate on. It is free text, so it shares a field with the machine-readable prefixes below — but since 2026-09-07 the gateway rejects a reason that would read as one: the two retry-safe prefixes, the pre-2026-08-18 runner-death wordings, and the bare string cancelled. Without that guard a sweep could be labelled ollama crashed: … and every consumer following The two retry-safe classes would resubmit the work onto the queue an operator had just deliberately cleared. Incident context is unaffected and still lands verbatim.

cancelled_by says who asked for the cancel. error cannot: it records the fixed string cancelled whether you cancelled the job or an operator did, so those rows used to be byte-identical. On 2026-07-16 an operator loop cancelled 111 queued jobs belonging to a consumer who had no cancellation code at all, and settling that took until 2026-09-06 because nothing in the row said who did it. Since 2026-09-07 every cancel — per-job DELETE and the bulk sweep alike — records the canceller:

{"status": "failed", "error": "cancelled", "cancelled_by": "autostatement"}

The value is the canceller’s X-Caller-Id, or their IP when they sent none, which is what an operator working at a shell looks like. It is self-declared and is not an authentication claim — there is no auth layer here and anyone can send any caller id. It answers “was this us?”, which is the question that actually cost a consumer seven weeks; it does not answer “who is authorised”.

Treat cancelled as terminal regardless of who appears here — a cancel is deliberate whoever asked for it. Use cancelled_by to know whether to go looking at your own code.

The field is absent, never null. A job nobody cancelled omits it, and so does any row recorded before 2026-09-07 — those cannot answer the question, and the gateway will not invent an answer by backfilling one. "cancelled_by" in body is the check, not body["cancelled_by"] is None.

Don’t try to detect a sweep by timestamp equality. For rows older than cancelled_by, and for any failure that isn’t a cancel, you are still reading shape rather than a field — and this is the shape people reach for first. It is tempting, and it is wrong. Only the bulk endpoint writes one shared completed_at across every row; a loop of per-job DELETEs writes distinct ones — measured on that 2026-07-16 event at 2.84 cancels/s over 39 s, every timestamp different, median gap 0.268 s. An equality test would have missed the only real mass-cancellation either side has on record.

Use volume instead. N identical error strings inside a short window means something systemic is happening — an operator clearing your work, or a backend that is down — and backing off is the right response to both, so you do not need to tell them apart. autostatement runs five identical errors within 120 s as an override to terminal, which is a sound default. This is the brake that makes an attempt cap safe: a genuine backend crash arrives one job at a time as each is dispatched, while anything systemic arrives in a burst.

A timeout failure has two possible causes, with two different error strings: your own timeout_seconds expiring, or the deployment-wide dispatch cap expiring. They call for different responses. See Timeouts: which clock measures what.

An empty completion is a failure only when nothing says the model ran. ollama answers 200 with an empty completion during a model-load race, and storing that as your result would hand you an empty string indistinguishable from a real answer. So the gateway fails it — but narrowly, because several legitimate replies are also empty and failing those would throw away work you already paid for. It fails only when the completion is empty and done_reason is absent or load. The error reads ollama returned an empty completion for /api/chat (done_reason=…, eval_count=…) — nothing indicates the generation ran, and is worth retrying.

These stay successful:

• done_reason: "stop" or "length" — the model ran and stopped on its own terms. A reasoning model whose token budget ran out mid-thinking has empty content, a full thinking, and done_reason: "length"; you get the whole body. Retrying that would only reproduce it — raise num_predict instead.
• A model load or unload — /api/chat with no messages (or /api/generate with no prompt) is ollama’s way to preload a model, and the same with keep_alive: 0 unloads it. Empty is the answer you asked for.
• A tool call — empty content beside a populated tool_calls.

Only /api/chat and /api/generate are checked. A docling result arriving with no document object fails on the same principle.

Transient backend blips no longer lose a conversion. A docling status poll or result fetch that 5xxs, drops its connection, or returns a truncated body is retried (five consecutive failures by default) rather than failing your job — the conversion itself keeps running on the large throughout. A poll that hangs is not retried: that means docling is wedged rather than blipping, and it still fails as docling HTTP timeout.

While the conversion is still running your timeout_seconds bounds this as before — past your deadline a failing poll is reported as your bound firing, not retried. The one exception is after docling reports the conversion finished: collecting the result is retried with no deadline clamp, deliberately, because abandoning a finished conversion throws away work already paid for. If docling goes dark exactly then, your job can fail up to five GATEWAY_DOCLING_HTTP_TIMEOUT windows plus their sleeps — roughly 5 minutes on the defaults — after timeout_seconds expired. Size any client-side abandon deadline with that tail in mind.

A backend that is down right now fails your job with <backend> unreachable: <detail> — e.g. docling unreachable: Cannot connect to host docling:5001 ssl:default [Connect call failed …]. Match the prefix docling unreachable: (or ollama unreachable:), never the whole string: the detail is the raw connector error and is not stable. This is a platform condition, not a fault in your payload, and resubmitting is unconditionally safe — the job never reached the backend, so unlike a dispatch timeout there is no possible partial side effect upstream. It is most likely in the couple of minutes after the large boots, and while it lasts GET /diagnostics reports docling_ok: false.

A backend that died mid-call fails your job with <backend> crashed: <detail> — e.g. ollama crashed: HTTP 500: {"error":"model runner has unexpectedly stopped, this may be due to resource limitations or an internal error, check ollama server logs for details"}. Match the prefix ollama crashed:, never the whole string: the detail is the backend’s own words relayed verbatim and is not stable. The prefix is backend-scoped like <backend> unreachable: above, but only ollama produces it — no docling crash has ever been recorded on this platform.

New 2026-08-18, and your existing substring match keeps working. Until this landed the error was relayed with no stable class and you were told to match ollama’s wording directly. The class wraps the relayed text rather than replacing it, deliberately, so model runner has unexpectedly stopped is still inside the string. Match the prefix for anything recorded from 2026-08-18 on; keep the substring check for as long as you read rows recorded before it, which carry no prefix.

Correction to what this section said before. The second wording, llama-server process has terminated: signal: killed, was documented here as an older engine’s phrasing. It isn’t. Both wordings came off the same engine and mark where the runner died: the first when a request’s context allocation failed, the second when it died completing a cold model load. Both are current, and both classify.

It can also arrive as ollama crashed: stream error: …. The gateway streams the generation endpoints internally, so a runner that dies after ollama already returned 200 reaches you over a different transport. Same event, same class — which is the point of matching the prefix rather than the HTTP 500 inside it. Every crash observed so far struck before the first token and took the first form.

What it means: ollama’s inference subprocess died. Overwhelmingly this is resource exhaustion on the host — the kernel’s OOM killer taking the runner because it was the largest process, triggered by whatever else happened to allocate next. Retry is safe and usually succeeds. The job produced no output and had no upstream side effect, so resubmitting is as safe as <backend> unreachable:.

Two things not to conclude from it. It is not a signal about your payload: measured on the local stack 2026-08-17, three of 77 runs died this way, each 4–7 s after dispatch having generated zero tokens, and each was clean on retry — including the one carrying the largest prompt in the set, which was the obvious suspect and was not the cause. And a single occurrence is not grounds for marking a filing permanently failed; treating it as terminal is how three retryable crashes became three missing results.

The two retry-safe classes, as one check. crashed and unreachable are the errors the platform owns rather than your payload, and both are safe to resubmit unconditionally — neither leaves a partial side effect upstream:

RETRY_SAFE_PREFIXES = (
    "ollama crashed: ", "docling crashed: ",
    "ollama unreachable: ", "docling unreachable: ",
)
LEGACY_CRASH_SUBSTRINGS = (
    "model runner has unexpectedly stopped",
    "llama-server process has terminated",
)

def is_retryable(error: str) -> bool:
    return (error.startswith(RETRY_SAFE_PREFIXES)
            # Rows recorded before 2026-08-18 carry no crash prefix.
            or any(s in error for s in LEGACY_CRASH_SUBSTRINGS))

Everything else — your timeout_seconds, the deployment cap, an empty completion, a 4xx from either backend, and the two gateway-restart strings (Gateway restarts, above: retryable on ollama, back off first on docling, which is why they are not in the unconditional list) — needs a decision rather than a resubmit, because the remedies differ. A crash is also worth counting: resubmitting forever hides a box that has stopped being able to run the work at all, so cap the attempts and let the cap failing be the signal.

Pinning the retry contract

The block above is a copy. If we reword a prefix, nothing in your suite fails — your matcher simply stops recognising a class, genuine crashes reclassify as unrecognised, and they stop being retried. You would find out by noticing a block of infrastructure failures that should have been retries. A consumer raised exactly this on 2026-09-07.

GET /v1/error-contract closes it. It serves every class the gateway can record, derived from the same predicates that classify errors — so it cannot drift from the implementation — plus a version over the matching rules:

{
  "version": "008244ed38ed4c93",
  "classes": [
    {"id": "backend_crashed", "match": "prefix",
     "values": ["ollama crashed: ", "docling crashed: "], "retry": "safe"},
    {"id": "gateway_restart", "match": "exact",
     "values": ["gateway shutting down",
                "gateway restarted while job was running"],
     "retry": "conditional", "retry_when": {"backend": ["ollama"]}}
  ]
}

retry is one of four verdicts: safe (resubmit unconditionally), conditional (only when retry_when holds — read it), decide (a real answer about your job; retrying unchanged reproduces it), and never (deliberate; resubmitting fights whoever did it).

Assert the version in your own tests. Pin the string, compare it against a live fetch, and fail on mismatch — then a contract change surfaces as your build going red rather than as a bad week of retries. The version covers the matching rules only, so we can improve the prose without waking you. You need to reach the gateway to fetch it, so if you run through the tunnel this belongs wherever you already talk to us rather than in ordinary CI; it does not need to be fast or highly available.

Building your matcher from the response at runtime instead is supported and is strictly better — but pinning is what turns a silent reclassification into a failing test, so do that either way.

Apply this check at every stage that submits a job, not just the one where failures are common. A platform condition can land on any submission — a backend that is down affects whichever stage happens to need it, and that is rarely your busiest one. A pipeline whose retry covers, say, its labelling calls but not its document conversions will mark a transient platform condition as permanently failed the first time the other stage is the one that hits it, and the correct classification on our side never reaches the code that needed it. This is not hypothetical: a consumer lost eight filings to exactly that shape on 2026-08-18, and the errors they were handed were the retry-safe class throughout. If is_retryable lives in one place and every stage calls it, this cannot happen; if each stage matches its own strings, it will happen once per stage.

Free-tier budget cap. The free-tier fuse is enabled on the production deployment (since 2026-07-24; it remains opt-in and off by default in local/dev stacks). When the large has spent its monthly Always Free budget, a job needing it is not failed — it’s held in the non-terminal status waiting_for_budget, and dispatched automatically once the meter resets (UTC month rollover) or the operator raises the cap. Treat waiting_for_budget like queued: don’t error, don’t time out aggressively. If your workflow can’t wait for a possible month-boundary delay on budget-bound jobs, set a consumer-side deadline and DELETE /v1/jobs/{id} to cancel (cancellation works on held jobs just like queued ones). Jobs already running, or that don’t need the large, are unaffected.

timeout_seconds does not bound a budget hold. It is a dispatch bound: it starts counting when the job reaches the backend. A job with timeout_seconds: 60 can sit in waiting_for_budget for the rest of the month and only then spend its 60 seconds. The consumer-side deadline above is the lever for that wait, not this field.

Checking where the budget stands. If a job is sitting in waiting_for_budget, GET /queue’s lifecycle block tells you whether the wait is minutes or weeks:

• free_tier.binding_pct / free_tier.binding_metric — how much of the large’s monthly budget is spent, and which meter (OCPU-h or GB-h) is currently the constraint.
• free_tier.fuse_enabled / free_tier.stop_at_pct — whether the cap is active, and the percentage at which it trips (default 90).
• fuse.tripped / fuse.reason — whether it is holding jobs right now, and why.
• month_to_date.next_reset — the UTC timestamp at which the meter resets and held jobs drain.

If binding_pct is already above stop_at_pct, expect new large-backed jobs to park in waiting_for_budget until next_reset.

While status=queued, the response includes queue_position (1-indexed, global; 1 means “next to run,” counting every queued job ahead of yours regardless of backend). Once a job starts, the response includes started_at and queue_position is dropped. On completion, the verbatim backend response is returned under result.

timeout_seconds is echoed on every status, but only if you set it on submit — its absence means “no per-job bound”, not “the field is missing”. Don’t treat that as a bug.

While status=running, the response also includes a phase field describing what the gateway is currently doing with your job. The bundled dispatchers emit:

phase is a UX affordance — a consumer can render meaningfully different “loading model…” vs “generating…” states without guessing — but it’s not a control plane signal. Don’t write logic that depends on phase transitions firing in a specific order or at all (e.g. a very fast inference may transition straight to completed before the watcher even writes ollama_generating). Treat it as a hint, not a contract.

Correction (2026-09-06): ollama_generating reports residency, not activity. It is derived from polling ollama’s /api/ps, which answers “is this model in RAM”, and the production large is configured to keep the model resident indefinitely. So on a warm box the phase pins to ollama_generating from its first poll — about two seconds in — and stays there for the rest of the job whether or not a single token is ever produced. A job that is completely wedged reports exactly the same phase as one generating normally, for the whole hour.

The table above previously said “actively producing tokens”, and four failed jobs were diagnosed against that reading before it was caught. tokens_generated is the field that answers “is it actually producing” (below), and on a failed job output_tokens is (Reading a failure). Use phase to render a label; never to decide whether work is happening.

Watching a running job

For ollama generation jobs, a running response also reports how far the work has actually got:

This exists to answer one question that nothing else on this page can: is my job still working, or is it wedged? A generation that has been running 45 minutes is not by itself a problem — on a slow box it may be entirely normal — and before these fields the only way to tell the two apart was to go and look at the host. Now a rising tokens_generated says it is producing, and a last_token_at that has stopped moving says it is not — and either way you can act on it, because a running job can be cancelled (see Cancelling a running job).

tokens_per_second is the number to compare against a known-good rate for your deployment. It measures decode only: it starts at the first token, so it excludes the queue wait, the model load and the prefill that came before, all of which would otherwise drag it down for the first minute of every job and make a healthy job look sick. See Latency expectations for what is normal where.

How they behave, so you don’t read too much into them:

• These three appear only while status=running, and only once the first token has arrived. A job that is still loading the model, or queued, or has finished, or that sent tools (see below), has none of these fields. Their absence never means failure — check status and phase for that. A failed job reports the same work under different names, because the question there is a different one: see Reading a failure.
• tokens_per_second needs two tokens to have an interval to measure, so it lags tokens_generated by one report.
• Updated about once a second, not per token — every update is a write on the gateway’s job store. Polling faster than that returns the same numbers.
• tokens_generated is the gateway’s count, not ollama’s. It typically ends one below the eval_count on the finished result, because the terminating stop token arrives on a chunk that carries no text. Use it as a progress indicator; use the finished job’s eval_count when you need the authoritative number.
• This is a hint, like phase. Don’t build control flow on the exact values.

Nothing else changes. The gateway asks ollama for a token stream internally and reassembles it, but the API is still submit-and-poll, your result is still one complete body, and that body is identical to what a non-streamed call would have returned.

Tool calls

A payload carrying a non-empty tools array is dispatched non-streamed, because tool-call deltas are the one thing reassembly can get quietly wrong. Tool calling works normally and the response is passed through verbatim; the only difference is that those jobs report none of the progress fields above — which is what every job did before internal streaming existed.

Gateway restarts: queued jobs survive (state is persisted to SQLite). Any job that was actively running at restart time is marked failed, with one of two exact strings depending on how the gateway went down:

Both mean the same thing to you: the job did not finish, no result was stored, and the gateway is not going to produce one. They are two halves of one event, not two conditions.

Whether to resubmit depends on the backend, and the job body tells you which (backend):

• ollama — safe to resubmit, on the same reasoning as <backend> crashed:. Generation stopped when the process did, and inference mutates nothing upstream.
• docling — do not resubmit immediately. docling-serve exposes no cancel endpoint, so a conversion that was in flight is still running on the backend and will finish on its own. Resubmitting puts a second conversion on a machine that runs one job at a time. This is the same trap as a docling dispatch bound expiring; back off first.

Neither string carries a <backend> prefix, so they are matched whole rather than by prefix — and they are deliberately not in RETRY_SAFE_PREFIXES, because that list means “resubmit unconditionally” and the docling half of this is conditional.

Retention — a safety margin, not a home

Store every response on your side as soon as the job completes. The window exists so a collector that lags, crashes, or restarts mid-run can still pick up an answer it already paid for. It is margin for recovery, not storage.

A swept answer cannot be re-fetched, only re-run — the full inference again, on the single global slot, drawing the free-tier meter. See Whichever clock fires, don’t lose the job id under Timeouts for how often that gap actually opens.

The numbers. Completed and failed records are kept 72 hours after completed_at by default, then deleted by an hourly sweep. Both are env-tunable on the gateway (GATEWAY_JOB_TTL_SECONDS, default 259200; GATEWAY_CLEANUP_INTERVAL, default 3600).

Some callers have a longer window, granted per X-Caller-Id:

GET /queue reports what is actually loaded. Trust it over this table:

"storage": {
  "db_bytes": 41582592,
  "wal_bytes": 4157440,
  "retention": {
    "default_ttl_seconds": 259200,
    "overrides_by_caller": { "autostatement": 2592000 },
    "sweep_interval_seconds": 3600
  }
}

A longer window buys a bigger margin; it does not make the gateway a database. It carries no durability promise — jobs.sqlite is one file on one host, no backup, no replica, and an operator rebuilding the small may start it empty — and it does not widen the webhook window (see Completion webhooks).

Queued, running and waiting_for_budget jobs are never swept at any age — status decides, not the clock. The window starts when a job finishes.

After sweep, GET /v1/jobs/{id} returns HTTP 404 with body {"error": "job not found"} — indistinguishable from an ID that never existed. There is no expired state and no tombstone, so only your own record of what you submitted can tell a lost result from a bad id.

The completed_last_24h / failed_last_24h counters on /queue are independent of retention — always the last 24 h.

Direct backend access is gone

Changed 2026-08-19, and this is a breaking change if you were doing it. ollama’s :11434 and docling’s :5001 are no longer published on the host by a local platform-services stack. On OCI they were never reachable (the large is on a private VCN). So on every deployment, anything of yours that reads those ports directly stops working.

Say it the sharp way, because the polite way misled a consumer: this is not “you can stop avoiding those ports in your devcontainer config.” It is “any code path that fetches from them now fails.” That is a quieter class of breakage than a port clash — a clash is loud and immediate, whereas a lookup that silently returns nothing can keep a pipeline running while it records worse data. One consumer read model digests from :11434 for provenance and recorded a batch of unpinned bare tags before noticing.

Why. A host port on a developer laptop is a namespace nobody controls: VS Code’s forwarding claims ports and walks upward when one is taken, Docker’s allocator can hold a stale reservation, and a neighbouring compose project can bind first. It cost three separate incidents, including a healthy docling that was unreachable for twenty minutes because its binding had silently vanished. Internal traffic now stays on the container network, where none of that can reach it.

What to use instead — all of it was already the supported route:

The gateway (11435, or whatever GATEWAY_HOST_PORT says) and the docs vhost (5006) are unchanged and still published. Nothing about submitting or polling jobs changed.

Completion webhooks (optional)

Instead of holding a polling loop open to notice a job finished, you can register a callback_url on submit. When the job reaches a terminal state — completed or failed — the gateway makes one outbound POST to that URL. This lets an event-driven orchestrator resume the next pipeline stage without a live poller: if your poller dies (laptop sleeps, container restarts), the callback still fires and your handler picks up where it left off.

The webhook notifies, it does not ship the result. GET /v1/jobs/{id} remains the authoritative source of the result, for as long as your retention window keeps it. The callback is a best-effort convenience — if delivery fails after retries, nothing is lost; you can always fall back to polling.

A longer retention window does not widen the callback window. A restart re-sends only unfinished deliveries from the last 72 h (GATEWAY_WEBHOOK_RESUME_WINDOW_SECONDS), however long your records are kept — so you are never pushed an event for a job you finished three weeks ago.

Per-caller setup (required)

Callbacks need per-caller configuration that the operator holds server-side, keyed by your X-Caller-Id:

• a shared secret used to sign deliveries (so you can authenticate they came from the gateway);
• a host allow-list — the set of hosts the gateway is willing to call back for you.

Until that’s registered, a submit carrying a callback_url is rejected with 400. The onboarding handshake is:

• You send the operator: the exact X-Caller-Id you submit jobs with, and the exact callback host(s) you’ll use (e.g. hooks.your-app.example, or hooks.your-app.example:8443 for a non-default port).
• The operator sends you back: a secret string. Put it in an env var (e.g. GATEWAY_WEBHOOK_SECRET) — it never goes in the submit body or your repo; it’s only used to verify the signature on delivery.

A caller can also have a default callback URL configured, applied to every job that doesn’t set its own callback_url (a per-job callback_url always wins).

Your receiver must be reachable from the gateway, i.e. from the public internet. The gateway runs in OCI and dials out to your callback_url, and its SSRF guard refuses private/loopback/internal targets. So localhost and private addresses will not work as a callback, even for local development. For local dev, expose your receiver through a public tunnel (e.g. an ngrok-style HTTPS URL) and allow-list that host, or point the callback at a deployed receiver. This is independent of the inbound SSH tunnel you use to reach the gateway — that tunnel carries your requests in; the callback is a separate connection the gateway makes back out to you.

Payload

The body is a small, fixed envelope — enough to identify the job and fetch its result, never the result itself (docling markdown/tables can be large):

{
  "job_id": "0f5b…",
  "status": "completed",
  "backend": "docling",
  "tier": "batch",
  "caller_id": "autostatement",
  "submitted_at": "2026-07-16T09:25:01.124+00:00",
  "completed_at": "2026-07-16T09:31:44.881+00:00"
}

error is present (a string) only when status is failed. On receipt, GET /v1/jobs/{id} to fetch the actual result / full error.

Sent headers:

Verifying the signature

The signature authenticates the delivery and lets you reject replays. The scheme:

signing_input = X-Gateway-Timestamp + "." + <raw request body bytes>
signature     = HMAC_SHA256(shared_secret, signing_input)
header value  = "sha256=" + hex(signature)

Trust caveat. The signing key is selected by the job’s X-Caller-Id, which the gateway trusts without verifying (see What’s not here). On the current internal, single-consumer trust model that’s fine, but it means webhook authenticity is only as strong as X-Caller-Id spoofing resistance on the gateway’s network — a party able to submit jobs as your caller id can make the gateway send your endpoint a validly-signed notification. Before a second consumer joins, this must be backed by real caller authentication. Treat a verified signature as “came from the gateway,” not “was requested by me.”

Verify against the raw body bytes as received — do not re-serialize the parsed JSON, since any whitespace/key-order difference changes the hash. (The gateway signs a deterministic compact encoding with sorted keys; re-encoding it yourself is not guaranteed to reproduce it.)

import hashlib, hmac, time

SECRET = b"<your shared secret>"
MAX_SKEW_SECONDS = 300

def verify(request_body: bytes, headers) -> bool:
    ts = headers["X-Gateway-Timestamp"]
    if abs(time.time() - int(ts)) > MAX_SKEW_SECONDS:
        return False  # stale / replayed
    expected = "sha256=" + hmac.new(
        SECRET, ts.encode() + b"." + request_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, headers["X-Gateway-Signature"])

A minimal receiver

Putting it together — a complete endpoint that verifies, dedupes, responds fast, and fetches the result out of band. Shown in FastAPI; the shape is the same in any framework, but note the one framework- specific subtlety called out below.

import hashlib, hmac, json, os, time
from fastapi import FastAPI, Request, Response

SECRET = os.environ["GATEWAY_WEBHOOK_SECRET"].encode()  # from the operator
MAX_SKEW_SECONDS = 300
_seen: set[str] = set()          # replace with a durable store (DB/Redis) in prod

app = FastAPI()

@app.post("/gateway")
async def gateway_callback(request: Request):
    # (1) Read the RAW body for verification. Do NOT use the framework's
    #     parsed JSON here — re-serializing changes the bytes and breaks
    #     the HMAC. In FastAPI that means `await request.body()`, not a
    #     Pydantic model / `await request.json()`. (Flask: request.get_data();
    #     Express: express.raw() on this route, or the verify hook.)
    raw = await request.body()
    ts = request.headers.get("X-Gateway-Timestamp", "")
    sig = request.headers.get("X-Gateway-Signature", "")

    # (2) Reject stale/missing timestamps (replay), then verify signature.
    if not ts or abs(time.time() - int(ts)) > MAX_SKEW_SECONDS:
        return Response(status_code=401)
    expected = "sha256=" + hmac.new(SECRET, ts.encode() + b"." + raw,
                                    hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(raw)
    job_id = event["job_id"]

    # (3) Dedupe — delivery is at-least-once, a job may arrive more than once.
    if job_id in _seen:
        return Response(status_code=200)
    _seen.add(job_id)

    # (4) Respond 2xx FAST (the gateway uses a ~10s timeout). Do the real
    #     work — GET /v1/jobs/{id} for the result, then your next pipeline
    #     stage — asynchronously, not inline in this handler.
    enqueue_followup(job_id, event["status"])   # your own task queue / background job
    return Response(status_code=200)

The webhook body is notify-only, so step (4) still calls GET /v1/jobs/{id} to pull the actual result (or full error) — the callback tells you when, the GET tells you what.

Delivery semantics — at-least-once, dedupe on job_id

Delivery is at-least-once: the gateway retries with exponential backoff on transport errors and non-2xx responses, over a bounded number of attempts, then gives up and records the failure. There are no ordering guarantees, and a webhook may fire more than once for the same job (e.g. a retry after your handler already succeeded but its 2xx was lost). Your handler must be idempotent — dedupe on job_id. Respond 2xx promptly (the gateway uses a short outbound timeout, ~10 s); do the real work asynchronously on your side.

If delivery never succeeds within the retry window, fall back to polling — the job record is still there for your retention window.

Debugging — did my webhook fire?

GET /v1/jobs/{id} includes a callback sub-object once a callback is registered, so you can see delivery state without asking the operator to dig through logs:

{
  "status": "completed",
  "callback": {
    "url": "https://hooks.your-app.example/gateway",
    "status": "delivered",
    "attempts": 1,
    "last_status_code": 200,
    "last_error": null,
    "delivered_at": "2026-07-16T09:31:45.402+00:00"
  }
}

callback.status is one of:

Security rules you should know

The gateway treats the callback URL as untrusted and defends the outbound request:

• https only. http URLs are rejected at submit.
• Host must be allow-listed for your caller (above).
• Private / internal targets are refused. The host is resolved and the request is refused if it lands on a private, loopback, link-local, or otherwise non-public address (this includes the cloud metadata endpoint 169.254.169.254). The connection is pinned to the validated address, so a DNS change between submit and delivery can’t redirect it inward. Such a refusal shows up as callback.status = blocked and is not retried.

Point your callback_url at a public https endpoint you control.

Picking a model name

The production default is the 72B-class model on the OCI large host — qwen2.5:72b-instruct-q4_K_M. This is what consumers should target both in deployed prod and from dev (via the OCI tunnel). Qwen3 has no 72B size (its ladder jumps 32B → 235B), so the production target is Qwen2.5-72B; if you see a reference to “qwen3:72b” anywhere, it’s a docs error.

The 8B-class small profile provides one model:

• qwen3:8b-q4_K_M (stock registry tag) — control thinking per request with "think": false in the payload (forwarded verbatim), and the digest matches the public registry for provenance.

qwen3:8b-q4_K_M-nothink was removed on 2026-08-16. It never suppressed thinking — its template dropped the /no_think marker that does the work — and it is not established that it ever did. Measured on ollama 0.32.5: 7,636 characters of reasoning with no think field sent, zero with "think": false. Send "think": false against the stock tag instead.

It may still answer on a stack that already built it. Removal means the platform stops provisioning the tag, not that it vanishes from an ollama store that already has it — so a long-running local stack will keep serving it, badly, until someone runs ollama rm qwen3:8b-q4_K_M-nothink. Don’t read a successful call as evidence the tag is supported. Fresh stacks never get it.

Naming the old tag in your own historical records is fine — the platform doesn’t care, and it did not hold the removal open.

The profile exists for legacy compatibility and minimal-resource work — running ollama locally on a developer machine that can’t hold a 72B model, driving a regression suite where faster inference beats quality, or model-compare runs at batch priority. Do not depend on the small profile being available on the OCI large.

The candidate profile (qwen3.6:27b, gemma4:26b, qwen3.8:27b) is the default on a local stack and holds models under evaluation as possible replacements for the production 72B. It is a moving target by design — tags come and go as candidates are ruled in or out — so don’t pin a consumer to one of these names outside an evaluation run, and don’t expect them on the OCI large.

qwen3.8:27b is available on the local stack from 2026-08-26. It is a qwen3-era reasoning model, so Moving from a qwen2.5-era model to a qwen3-era one below applies to it unchanged — send "think": false. Two things are specific to it:

• It is local-only, but only because of the model profile now. The tag requires ollama 0.32.12 or newer. Until 2026-09-03 the OCI large ran 0.21.1, so on the large this was an engine that refused the tag outright. That is no longer the reason. The large now runs 0.32.15 too, and the tag would run there — it simply isn’t pulled, because the large loads the large profile and candidates aren’t in it. Asking for it on the large gets you a model-not-found, not a decode failure.
• 27b is the only size published. There is no smaller qwen3.8 to fall back to on a box that can’t hold it.

Adding it moved the local engine from 0.32.5 to 0.32.15, and two of those releases change behaviour you can observe whatever model you call. /v1/chat/completions streaming now matches OpenAI’s wire format — role on the first chunk only, finish_reason on its own chunk, usage in a separate chunk under stream_options.include_usage. And a truncated OpenAI-compat response now reports finish_reason: "length" instead of "tool_calls", which finally agrees with what Bounding output length tells you to check for.

Both changes now apply on the OCI large as well, which they did not before 2026-09-03. If you were keying off finish_reason == "tool_calls" to detect truncation against the large, that stops matching — switch to "length". Still read ollama.version from GET /diagnostics rather than assuming an engine: the two deployments are pinned independently and are free to diverge again.

A fourth candidate, qwen3.6:35b, is listed in the profile script but deliberately not pulled: it doesn’t fit the evaluation machine’s RAM.

Correction to what this section said before. It claimed a model that doesn’t fit “pages rather than failing”. That is wrong and it was the reasoning behind an unsound RAM budget. A model that doesn’t fit fails: on the evaluation box the runner was OOM-killed 4-7 seconds after dispatch having generated zero tokens, which reaches you as ollama crashed: and not as a slow job. Paging is the gentler outcome and not the one to plan against.

Moving from a qwen2.5-era model to a qwen3-era one

Reasoning tokens count against num_predict, and they will dominate it. Carrying your qwen2.5 settings across unchanged is enough to break extraction, and it breaks it in the most expensive way available — a long, apparently healthy generation that ends in unparseable output.

Measured by a consumer on 2026-08-16, one real financial-extraction prompt against qwen3.6:27b with num_predict: 4096:

Three things follow. "think": false in the request body is the only control that works — it is forwarded verbatim to ollama. A -nothink in a tag name is not equivalent — the platform’s own -nothink build measured 7,636 characters of reasoning against zero with "think": false, and has since been removed.

Omitting the field is not the same as sending false. This is the one that catches people, because “we don’t set think” reads like a neutral default and isn’t. With no think key in the body, the decision falls to the model, and a reasoning model decides on. If your client only adds the key when some THINK-ish config value is non-null, the unset case is thinking on — and paired with a num_predict of a few thousand that is exactly the 55.3-minute invalid-JSON row above. Send the field explicitly, or assert that you did.

And a truncated reasoning model looks like a normal answer: you get done_reason: "length" with the budget spent, so treat that as a hard error and never parse the result (see Bounding output length).

Hardcoding a model name in your consumer ties it to one deployment. Parameterize via env var, same way as the gateway URL:

# Production / dev-against-OCI (default — what every consumer
# should target unless they have a specific reason not to)
LLM_GATEWAY_URL=http://llm-gateway:11435  # or 21435 if tunneled
LLM_MODEL=qwen2.5:72b-instruct-q4_K_M

# Local platform-services only (legacy / minimal-resource tests).
# Send "think": false in the request body to disable thinking mode.
LLM_GATEWAY_URL=http://llm-gateway:11435
LLM_MODEL=qwen3:8b-q4_K_M

# Local platform-services, model evaluation (candidate profile)
LLM_GATEWAY_URL=http://llm-gateway:11435
LLM_MODEL=qwen3.6:27b

The model set actually loaded on each deployment lives in models/profiles/ (large.sh is the production target; small.sh is legacy). To discover what’s currently loaded on the gateway you’re pointing at, hit /diagnostics on the gateway — ollama.ps[].name lists what’s resident in RAM right now, ollama.tags lists everything on disk. The legacy https://soggplatform.dedyn.io/models.json proxy of /api/tags still works for public model discovery.

A model name does not fully identify what you’ll get. The deployments deliberately run different ollama versions, and the same tag can behave differently across them — badly enough that a model has decoded to gibberish on an older engine rather than failing outright. If you’re recording results you intend to compare or reproduce (benchmarks, evaluations, regression baselines), record what produced them alongside the output:

• ollama.version from /diagnostics — the engine
• ollama.tags.models[].digest — pins the exact weights behind a tag, which is why stock registry tags are preferred over locally built Modelfile variants: the digest matches the public registry

Read the digest from /diagnostics, not from ollama directly. This page used to describe the raw blocks as ops-debug-only two paragraphs below while recommending ollama.tags[].digest here — a contradiction that made it unclear whether a consumer could depend on it. Settling it in the direction the recommendation already pointed: ollama.tags on /diagnostics is a supported read for provenance and will not be removed without telling you. Its inner shape is ollama’s, not ours — we pass the response through — so read models[].digest defensively and treat an unexpected shape as “digest unavailable” rather than as an error.

Reaching ollama’s /api/tags directly is not a supported route and stopped working on local deployments on 2026-08-19 (see Direct backend access is gone below). A consumer doing that recorded unpinned bare tags for a batch and caught it only by luck.

Better: let the platform tell you what ran. Since 2026-08-20 every ollama job_metrics row records three model fields, and the distinction between them is the whole point:

Read them from GET /metrics/recent, matching on job_id.

They catch two different failures. model_resolved differing from model means a different model was served — which your own pre-flight check cannot catch, because it runs before you submit and never against what ran. A changed model_digest under an unchanged tag means the weights moved underneath the tag.

This closes a race you cannot close yourself. A digest you read at submit time describes the model on disk then, and your job dispatches later — the queue wait on this deployment reaches ~2 hours. Registry tags are mutable and the platform re-pulls them on demand, so the weights behind a name can change while your job waits. The gateway’s reading is taken after your job finishes, from the model still resident, so it describes the run rather than a guess made before it.

Both are null on rows written before 2026-08-20, and on any job whose probe missed — null never means “unchanged”, it means “not recorded”. A query that cares must exclude nulls rather than assume. The gateway will return no digest rather than a guessed one: if two models are resident and neither matches the name you asked for, you get null, because a wrong digest records confidently and cannot be caught afterwards.

The engine version is recorded on the same row (engine_version) for the same reason.

Context length is per deployment, and you probably shouldn’t pin num_ctx. The OCI large runs OLLAMA_CONTEXT_LENGTH=16384; the local stack runs 12288 (env-overridable). It stopped being uniform on 2026-08-17: on a shared dev box the context length is a memory setting as much as a capability one, because the KV cache and per-request compute buffers are sized from it, and a 16K window was allocating far more cache than any consumer’s traffic has ever used. The local stack also quantizes that cache (q8_0) where the large does not, so inference is not bit-identical across the two — record deployment beside engine_version if you compare outputs.

The gateway still forwards your options — num_ctx included — verbatim, with no cap or sanitization of its own; the only gateway-side limit is the 64 MB request body (GATEWAY_MAX_BODY_BYTES). So num_ctx in your payload overrides the deployment’s default entirely. This page previously called pinning it good self-documentation; that was wrong. Pinning it above a deployment’s default overrides tuning that exists for a reason — on the local stack, a consumer still sending num_ctx: 16384 gets exactly the memory footprint the 12288 default was chosen to avoid. Omit it and inherit the deployment’s value unless you need a specific window.

A prompt longer than the effective context is truncated by the engine, not rejected. No error, no flag on the result — the model answers from a clipped prompt and the answer looks normal. That is the one direction not to guess in, so if you set num_ctx at all, size it from a measurement rather than a hunch: input_tokens on GET /metrics/recent is your real prompt-token distribution, per job, and it outlives the job TTL.

What happens when the model isn’t loaded in RAM

Submitting a job whose model name is in available_models but not in loaded_models is legal and pays a one-time disk→RAM load cost — the gateway forwards the request to ollama, ollama loads the model from local disk (the bind-mounted weights volume), and then runs inference. The HTTP request just blocks while loading. From the consumer’s perspective the call takes longer than usual, and the phase field on /v1/jobs/{id} reports ollama_loading_model during the load window so you can render a meaningful UX state instead of “still loading…”. Once loaded the model stays resident and subsequent calls don’t pay the cost — indefinitely on the OCI large (OLLAMA_KEEP_ALIVE=-1), for 60 minutes of idle on the local stack (3600, finite because a dev box cycles through models that don’t co-fit). So a local consumer can pay the load again after a long pause; the large’s resident model only falls out when the instance stops.

Submitting a job whose model name is not in available_models at all is an operator-side configuration gap — the host this gateway points at hasn’t pulled the model. Ollama’s behavior in that case varies by version (recent releases auto-pull, older releases 404). Either way, don’t rely on auto-pull for a 40 GB production model — a silent 10–20 minute background download is the wrong default for an inference path. Pre-flight check by reading /diagnostics.available_models before submitting; if the model you need isn’t there, that’s an operator ask, not a consumer retry.

Latency expectations

These are calibration numbers for sizing consumer-side timeouts, not SLOs. They depend on the OCI large’s actual shape (currently 18 OCPU / 96 GB, CPU inference — no GPU) and the specific model loaded. Treat them as ballpark; measure against your own workload once you’ve got something in production.

They describe the OCI deployment only, and there is no equivalent table for the local one — deliberately. The local backends run on a developer machine that its operator also uses for everything else, so throughput there is a function of what else is running, not a property of the platform. Measured on one such box on 2026-08-09, across a single afternoon and with nothing misconfigured: decode throughput moved 8–15×, prefill 1.7–2.6×, model load 3–15×, and docling conversion ~1.9×, purely from concurrent use of the laptop. Even in the quiet band, repeating a byte-identical job gave ±39% run-to-run variance.

Two consequences for consumers targeting local, and where to look instead:

• Don’t build user-facing timing promises on it, and don’t treat any observed run as a floor.
• Don’t build a degradation detector or a retry-on-slow against these numbers. At ±39% noise you will generate false positives, and retries into a single-slot queue turn a delay into an outage.
• Do read the numbers off the box rather than off this page. GET /metrics/recent?deployment=local&model=<tag> returns prefill_tok_s, decode_tok_s and load_ms for jobs that actually ran there — see Checking recent throughput. That is the intended substitute for the table this section declines to print, and it is the better artefact: a measurement from your box beats an estimate from someone else’s, and it stays true when the model or the machine changes. A newly provisioned model has no rows until a job runs against it.

For the production model (qwen2.5:72b-instruct-q4_K_M) — numbers below are measured, not estimated, from the 2026-05-23 autostatement verify run on the OCI large host (then 20 OCPU / 140 GB, CPU inference). RAM size doesn’t affect inference latency, but the 2026-07-23 trim to 18 OCPU may add roughly ~10% to generation time if inference is compute-bound here — treat these as a mild under-estimate until re-measured at the new shape (the job_metrics table will show it):

Sizing a per-job timeout_seconds from this table: use the last row, not the one above it. The cold-load sits inside the dispatch bound, so a number chosen from “warm typical extraction” (~4 min) kills every first call after a cold start. Whether a given call is cold is not knowable at submit time — check /diagnostics.warm first if you want to bound warm and cold calls differently.

The cold-load cost is large enough that the gateway pre-warms the production model in the background after boot. When the operator sets LIFECYCLE_WARM_MODEL=qwen2.5:72b-instruct-q4_K_M on the gateway, the lifecycle controller kicks a warm probe (a single-token inference against that model) as a fire-and-forget background task immediately after backend health passes. From the consumer side this means:

• state=starting covers only OCI boot + the backend health probe (~70 s on the current OCI large), not the 20-min cold-load.
• After state flips to ready, the typical post-boot state is ready_cold for a few minutes (model not yet resident) and then ready_warm once the background probe completes the load.
• A consumer call that arrives during the warm window doesn’t wait for the probe to finish — it dispatches immediately, and either
  1. ollama already has the model resident and runs at warm speed, or (b) it races the probe and both wait on the same per-model load inside ollama. Either way, the cold-load cost is paid once per cold start and not duplicated.

If LIFECYCLE_WARM_MODEL is not set, no background probe is spawned and the first consumer job after each cold start triggers the load inside its own HTTP budget. Same wall-time, just different attribution.

GET /diagnostics surfaces the probe outcome under lifecycle.warm:

"lifecycle": {
  ...
  "warm": {
    "model": "qwen2.5:72b-instruct-q4_K_M",
    "last_attempt_at": "2026-05-25T10:14:23.117+00:00",
    "last_outcome": "success",
    "last_duration_seconds": 1187.4,
    "last_error": null,
    "model_loaded_at": "2026-05-25T10:14:23.117+00:00"
  }
}

Fields:

• last_outcome is one of success, timeout, transient_error, fatal_error, cancelled, error, or null (no probe run yet on the current boot). For consumer routing, treat anything other than success as “first call may pay cold-load” — the specific non-success value is operator-facing detail. Existing consumers that gate on last_outcome == "success" continue to work; the cancelled and error values are additive, not breaking.
• last_outcome=success plus model_loaded_at set is the strongest signal that the next call is warm. loaded_models on the same /diagnostics response is the live ground truth; model_loaded_at is a “when did we last positively confirm it” timestamp useful for predicting freshness over longer windows.
• A non-success outcome (probe timed out, hit a transient error, was cancelled by an unrelated lifecycle event, or raised an unexpected exception) doesn’t block READY — the controller stays ready and the first consumer call paying cold-load is the recovery path. Operator signal only.
• fatal_error indicates a misconfigured LIFECYCLE_WARM_MODEL (HTTP 4xx — most commonly 404 model not found). Consumer jobs using a different model name continue to work; consumer jobs using the same model name will see the same error. State stays READY in either case.
• cancelled means the probe was cancelled mid-flight by a reconcile-driven state transition (OCI reported the large stopped while the probe was still running). error means an unexpected exception escaped the probe — treat as a bug signal worth surfacing to the platform team.

Polling patience on cold starts

The gateway holds the ollama HTTP request server-side until ollama actually returns, up to min(timeout_seconds, GATEWAY_OLLAMA_TIMEOUT) — the deployment cap (default 60 min) unless you set a shorter per-job bound. A per-job bound has to cover load and generate. Cold-load is inside it: a 10-minute bound sized against “generation takes ~4 min” kills every cold first call, and on the local deployment there is no latency envelope to size against at all (below). Your consumer is polling /v1/jobs/{id}, not waiting on that HTTP call — each poll returns in milliseconds with status=running and the appropriate phase. phase=ollama_loading_model is now the normal signal that you’re paying cold-load (rather than the rare fallback it was when warm-on-boot blocked READY); phase=ollama_generating flips when ollama starts producing tokens. Your HTTP client’s per-request timeout only needs to cover one poll, not the whole job. The knob that matters for cold starts is how long your polling loop is willing to wait overall — for the 72B on the current OCI large, budget up to ~25 min for a cold first call (load + generate) and use the phase field to render meaningful state in the meantime.

For pre-flight smoke tests: a 5-token output against a trivial prompt ("Say pong.") completes in well under 30 seconds when state=ready_warm AND the requested model is in loaded_models.

If your verify runs produce additional measurements (especially on different output sizes or with different prompts), contribute them back — the table above is anchored on one filing’s worth of data plus the cold-load probe.

Timeouts: which clock measures what

The platform has three server-side clocks — two deployment caps and one you can set per job — plus at least two of your own. They measure different intervals, and sizing one from the other is the most expensive mistake on this page — it has produced both a consumer that abandoned jobs it should have waited for and a consumer that waited on jobs it should have abandoned.

Two more bounds apply to streamed ollama generation, and they measure progress rather than elapsed time — a silence, not a duration. They sit inside the caps above and are covered under Two errors that are not a clock running out: GATEWAY_OLLAMA_FIRST_TOKEN_TIMEOUT (1800 s, nothing arrives at all) and GATEWAY_OLLAMA_STALL_TIMEOUT (300 s, tokens stop arriving). Neither can be set per job and neither affects a job that keeps producing.

The third one is yours to set and the platform’s to enforce. It undercuts the cap for a single job — same clock, smaller number. See Bounding the job yourself below, but read the rest of this section first: the interval those caps measure is narrower than most people’s first guess, and timeout_seconds inherits it exactly.

None of the three includes queue wait. Worst case from your submit is queue_wait + min(timeout_seconds, cap), and queue wait is unbounded: there is one global in-flight slot, and a slow job legitimately holds it for the full cap. Two slow jobs ahead of you is two full caps of waiting before your own clock starts. Setting timeout_seconds does not bound that wait — it is the one sentence on this page most often misread as “my job dies N seconds after I submit it.”

The dispatch clock does not start where status=running appears. Those are two different moments:

• status=running and started_at are stamped when the worker claims your job off the queue — the first thing that happens to it.
• The dispatch cap starts later, at the backend HTTP call, after the gateway has parsed your payload, written the phase, and — on OCI — brought the on-demand large up.

That last step is what makes the gap deployment-dependent:

While that boot runs, your job reads status=running with phase=waiting_for_backend. That phase is the flag. A client clock anchored on the first running observation is aligned with the platform’s on local, and up to 30 minutes early on OCI — it can abandon a job the gateway was still perfectly willing to run.

Sizing your own clocks. You need two, because the platform splits the wait in two:

1. A run bound. Anchor it on the started_at field in the job response, not on your own first-observed-running timestamp. started_at is authoritative, immune to poll-interval skew, survives your poller restarting, and is reset to null if the job is ever demoted back to waiting_for_budget — so it self-corrects where a local variable would keep counting through a hold. Size it against the dispatch cap for your backend.
2. A queue bound, from submit. Size it against how long the work stays worth doing — not in multiples of the dispatch cap. The two are unrelated quantities, and “N in-flight timeouts, so something must be wrong” is invalid reasoning: N slow jobs ahead of you is normal, not a fault.

Bounding the job yourself: timeout_seconds

Your own run bound tells you when to stop waiting. It does not tell the platform anything, and the job keeps running — holding the single global in-flight slot, with everything behind it still queued. timeout_seconds is the lever that actually ends the work:

{
  "endpoint": "/api/chat",
  "payload": { "model": "qwen3:8b-q4_K_M", "messages": [...] },
  "timeout_seconds": 900
}

It is the same clock as the deployment cap, just a smaller number. Everything above about what that cap measures applies unchanged:

• It does not include queue wait. A job with timeout_seconds: 60 can sit queued for an hour first.
• It does not include the OCI instance boot. On OCI a bounded job can read status=running, phase=waiting_for_backend for up to 1800 s before its 60 s has even started. That is deliberate: timeout_seconds bounds the work, not the wait. You still need your own from-submit clock for the wait.
• For docling it spans the whole dispatch — submit, polling, and result fetch — matching GATEWAY_DOCLING_TIMEOUT.

Don’t invent a number — declare the one you already enforce. If your client gives up on a job after N seconds, send N. Picking a different value creates a third clock to reason about; sending the same one means both sides stop caring at the same moment.

Then check every leg. Wherever your client bound is lower than the backend’s cap, the difference is time the gateway holds the global slot for a job nobody is waiting for — and everything queued behind it waits out that gap too. Per backend:

The “lower” row is the 2026-08-09 incident’s shape, and it hides per-leg: a consumer whose ollama bound matched the 3600 s cap exactly still had a 300 s docling bound against the 1200 s docling cap — 900 s of orphaned slot on every slow conversion, on a leg nobody had thought to check. Run the comparison for each endpoint you call, not for your app as a whole.

If you reclaim, that same gap is where slow answers survive — so tightening has a cost, not only a benefit. The rule above assumes walking away is permanent. It isn’t, if you persisted the job id: an abandoned job keeps running, finishes, and its result waits in the store for your retention window, collectable at zero further inference cost. A timeout_seconds expiry is the opposite — the job is failed and there is no result to collect, ever. What you do get is partial_output: the text the gateway had assembled when the bound fired. That is salvage material, not an answer (see Cancelling a running job), so plan on an expiry costing you the job’s work.

Worked example, from the incident: a 2704 s job produced complete, correct output. Its consumer had stopped polling at 1800 s but nothing killed it, so they collected the full result 45 minutes later for free. Under a 900 s timeout_seconds that job dies with no usable answer and the work is paid for twice.

So the two rules cut different ways, and which one governs depends on your own recovery behaviour:

• You don’t persist job ids. Abandonment really is permanent. Declare timeout_seconds; the gap buys nobody anything.
• You do reclaim. Set the bound where re-running is genuinely cheaper than waiting — which on a slow backend is often not below the cap.

Cancelling is not the cheaper escape hatch. It is the same trade-off as a bound firing, with a human choosing the moment: DELETE on a running job destroys it and you get no result. It looks free only on a queued job, which has produced nothing yet. If you are weighing “let the user stop waiting” against “don’t burn the work”, the only option that preserves both is the one you already have: stop polling, keep the job id, collect the result later. Full detail in Cancelling a running job.

Validation is loud. 1 <= timeout_seconds <= your backend’s cap (3600 s ollama, 1200 s docling). Above the cap the submit is rejected with 400 naming the cap and its env var:

{"error": "'timeout_seconds'=5000 exceeds the ollama deployment cap GATEWAY_OLLAMA_TIMEOUT=3600s. Submit a value <= 3600, or ask the operator to raise the cap — the gateway will not silently clamp it."}

It is never silently clamped. A bound that quietly became a different number than the one you asked for would defeat the point of asking. Non-integers (including 900.0), zero, and negatives are rejected the same way. The value you set is echoed back on the accept response and on GET /v1/jobs/{id}, so you can confirm what the gateway recorded.

When it fires the job goes failed with an error naming your clock:

ollama dispatch exceeded the job's timeout_seconds=900

and when the platform’s own cap fires instead, naming its clock:

ollama dispatch exceeded the deployment cap GATEWAY_OLLAMA_TIMEOUT=3600s

Match on the substring timeout_seconds= to tell the two apart — they need different responses from you. Your bound firing means “this took longer than it was worth to me”, and raising your number is an option. The deployment cap firing means the work does not fit this deployment at all, and no number you can submit will change that — the lever is the operator’s, or the work has to get smaller.

At the boundary the two collapse into one. If you set timeout_seconds to exactly the cap, expiry is reported as the deployment cap, because that is the answer to the question the error exists to settle: there is no larger value you could have sent. To confirm what the gateway actually recorded for your job, read the echoed timeout_seconds field, not the error string.

The slot frees as soon as the bound fires, so the next queued job starts within seconds rather than waiting out the full deployment cap. That is the whole reason the field exists.

Two errors that are not a clock running out

New 2026-09-06. Both clocks above measure elapsed time and nothing else, so until now a job that streamed steadily for 59 minutes and one that never emitted a single byte failed with a byte-identical error at the same second. Four ollama jobs took the second path on 2026-09-01/02, burned four hours of backend time and produced nothing, and nothing in the record said which had happened.

Streamed ollama generation now also has two bounds on progress, inside the caps rather than replacing them, and each has its own error:

ollama produced no first token within GATEWAY_OLLAMA_FIRST_TOKEN_TIMEOUT=1800s — the backend accepted the request and then sent nothing at all

ollama stopped producing tokens for GATEWAY_OLLAMA_STALL_TIMEOUT=300s after 143 tokens — the stream stalled mid-generation

Match on the env var name — that is the stable part, the same convention the two clocks above use. Neither string contains dispatch exceeded, deployment cap or timeout_seconds=, so existing matching for the caps will not catch these and does not need changing.

They are deliberately separate errors because the remedies are opposite. A stall reported as a cap expiry would send you off raising timeout_seconds and asking the operator to raise GATEWAY_OLLAMA_TIMEOUT — when the job did not run out of time, it stopped. No amount of extra budget fixes a backend that produced nothing; it only buys a longer wait before you find out.

Two notes on scope. A payload carrying a non-empty tools array is dispatched non-streamed (Tool calls), and neither bound applies to it — there is no token stream to watch, so those jobs stay on the caps alone. And a slow generation is not a stall: the second bound measures the gap between tokens, so a job producing steadily at any rate runs to the cap as it always did.

Reading a failure

New 2026-09-06. A failed job now reports what it had actually got done, which was recorded all along and previously not returned:

output_tokens is the field to look at first, because it separates two failures that otherwise look identical:

• output_tokens: 0 — the job produced nothing. If the error is a cap expiry, the work never started rather than overrunning; a no-first-token error says the same thing explicitly.
• output_tokens in the thousands — the job was generating and ran out of time. Check it against your own num_predict before concluding the platform is at fault: at this deployment’s median throughput, num_predict: 4096 needs about 3,591 s of the 3,600 s cap, so a generation allowed to run to that length will hit the cap almost exactly, every time, with nothing wrong anywhere. Real extraction outputs on this platform run 182 tokens median.

output_tokens is the gateway’s own count, so it is typically one below the eval_count a completed job reports (the terminating stop token carries no text). It is absent — rather than zero — on failed tools jobs and on docling, neither of which feeds a token stream; a zero there would be an artefact of the transport rather than a measurement of the work.

Whichever clock fires, don’t lose the job id. A client-side give-up does not stop the job — it keeps running, finishes, and stores its result. Re-poll the persisted id before you declare the work failed; the record is retrievable for your full retention window. Every abandoned job we have seen in practice had a correct result sitting in the store that nothing on the consumer side ever collected.

A server-side expiry — your timeout_seconds or the deployment cap — is different, and differs again by backend:

• ollama: the gateway closes the connection, which aborts generation. The job is genuinely over and no result is stored. Resubmitting is the only way to get one.
• docling: the gateway stops waiting, but the conversion keeps running on the backend. docling-serve v1.16.1 exposes no cancel endpoint of any kind, so there is nothing for the gateway to call. Your job reads failed and the gateway holds no result — but the document is still converting, still occupying the backend, and will keep doing so until it finishes on its own. Resubmitting immediately therefore puts two conversions on a machine the platform runs one job at a time on. If a docling bound fires, give the box room before you retry.

Cancelling a running job

DELETE /v1/jobs/{id} works on a running job, not only a queued one. The dispatch is ended, the job goes failed with error: "cancelled", and the single global slot frees within seconds so the next queued job starts.

DELETE /v1/jobs/0f5b…
→ 200 {"id": "0f5b…", "status": "failed", "error": "cancelled"}

The response reports the job’s state once the cancel has settled, so in the ordinary case it already says failed. Three cases are worth knowing:

Cancelling a running job destroys it. This is the same trade-off as timeout_seconds expiring, with you choosing the moment rather than a clock. There is no result afterwards and resubmitting pays for the work again. Three distinct actions, only two of them cheap:

• Stop waiting — client-side, no API call. The job keeps running, finishes, and its result waits out the retention window. Costs nothing, frees nothing.
• Cancel — frees the slot immediately, destroys the work.
• Bound expiry — identical to cancel, automatic.

partial_output — salvage, not an answer

A job that ends early carries what the gateway had assembled at that moment, in its own field:

{
  "id": "0f5b…",
  "status": "failed",
  "error": "cancelled",
  "partial_output": "The three primary colours are red, blue"
}

It is never result, and that is deliberate. A truncated generation is not a shorter answer — a format-schema-constrained extraction cut mid-object is unparseable JSON (see Bounding output length), and prose cut mid-sentence can invert its own meaning. Treat it as evidence: what the model was producing, how far it got, whether it was on track. Anything that parses it as an answer is misusing it, which is exactly why it does not share a name with one.

Details worth knowing:

• Present only on failed ollama jobs that produced at least one token, for any early exit — cancel, your bound, or the deployment cap. Absent otherwise; its absence is not an error.
• Content only. Thinking output is not included.
• docling jobs never have it. There is nothing to salvage: the conversion is either complete or it has not returned anything.
• In the GET /v1/jobs listing it appears as partial_output_bytes (size only), the same treatment result gets.
• It rides the same retention as the rest of the job record.

What cancel does to the backend

• ollama: closing the connection aborts generation, so the work genuinely stops and the CPU frees.
• docling: slot-only. docling-serve v1.16.1 has no cancel endpoint, so the gateway stops waiting and nothing else — your conversion keeps running upstream and keeps occupying the backend until it finishes on its own. The gateway’s slot is free, but the machine is not. Don’t resubmit immediately or you will have two conversions racing on a box that runs one job at a time.

A rule worth stealing: only DELETE a queued job

One consumer’s adopted rule, and the cleanest answer we have seen to the three-actions problem above:

DELETE is issued only while the job is queued. A running job is never cancelled — it finishes, and the result is reclaimed later.

A queued job has computed nothing, so dropping it destroys nothing while still freeing your place in the queue. Their user-facing button is labelled “Stop waiting”, because that is what it does.

Note what did not fix this for them. They were advised — twice — to drive the button’s message off the response status rather than hardcoding it, and did. It was not enough: the semantics were what changed. The same button making the same call silently converted from free to destructive the day running-cancel deployed, with status-driven copy faithfully describing the new destructive behaviour in fluent prose. What fixed it was a rule about which jobs they are willing to DELETE at all. If your cancel path predates this feature, check that rule and not only the wording.

Bounding output length

Two ways to stop a generation running away, and they fail very differently.

num_predict truncates. Hitting it returns done_reason: "length" and whatever had been produced — which for a format-schema-constrained extraction is an unparseable object. It is only safe if you treat done_reason == "length" as a hard error and never parse that result.

On a reasoning model the budget is also what the thinking is drawn from, so a cap that was generous for a qwen2.5-era model can be consumed almost entirely before the answer starts — see Moving from a qwen2.5-era model to a qwen3-era one. Send "think": false rather than raising the cap to cover reasoning you don’t want.

A maxItems in your format schema terminates cleanly, and is strictly better where it applies. Verified against ollama 0.32.5 on 2026-08-11, independently by a consumer and by the platform: a schema capping an array at 2 items, given a prompt containing 5 obvious rows, returned exactly 2 items with done_reason: "stop" and valid, parseable JSON. The grammar closes the array at the cap rather than the decoder being cut off mid-object.

But a clean cap is silent data loss, which is worse for some workloads than a loud truncation. A document with more rows than the cap yields well-formed JSON, done_reason: "stop", and no error anywhere — quietly missing entries. So maxItems is a runaway fuse, not a size limit, and it is only safe paired with treating len(items) == maxItems as a failure worth investigating. A consumer extracting ledger rows declined to ship it as a default for exactly this reason.

Submitting an ollama job

import os, time, requests

GATEWAY = os.environ["LLM_GATEWAY_URL"]  # e.g. http://llm-gateway:11435
MODEL = os.environ["LLM_MODEL"]          # e.g. qwen2.5:72b-instruct-q4_K_M
HEADERS = {"X-Caller-Id": "my-editor-assistant"}

submit = requests.post(
    f"{GATEWAY}/v1/jobs",
    headers=HEADERS,
    json={
        "endpoint": "/api/chat",
        "payload": {
            "model": MODEL,
            "messages": [{"role": "user", "content": "Summarize ..."}],
        },
        "priority": "interactive",
    },
)
submit.raise_for_status()
job_id = submit.json()["id"]

while True:
    r = requests.get(f"{GATEWAY}/v1/jobs/{job_id}", headers=HEADERS)
    r.raise_for_status()
    job = r.json()
    if job["status"] == "completed":
        print(job["result"])
        break
    if job["status"] == "failed":
        raise RuntimeError(job["error"])
    time.sleep(5)

A 5-second poll cadence is fine for 5–15 minute jobs, and it costs the same whatever you submitted: since 2026-09-06 a status read no longer touches your payload, so polling a 64 MB docling submit is as cheap as polling a one-line prompt. Earlier advice to slow your polling on large submissions is retired. You receive the full response in result, never tokens-in-flight — the stream field in your payload is set by the gateway and whatever you put there is ignored, in either direction. It streams from ollama internally on generation endpoints and reassembles the result; that is invisible to you except for the progress fields described in Watching a running job above.

Submitting a docling job

Docling input goes inline in the payload as base64. The gateway does not accept multipart uploads — that simplifies the gateway and keeps the submission shape uniform between backends. For a 10 MB PDF this means ~13 MB of base64 text in the request body, well inside the gateway’s 64 MB body cap.

The gateway forwards your payload to docling’s /v1/convert/source/async, polls until docling reports the task terminal, and returns the docling result JSON verbatim in result.

Important — conversion options must be nested under options. docling-serve accepts conversion knobs (do_ocr, to_formats, do_table_structure, md_page_break_placeholder, etc.) under an options sub-object, not as top-level siblings of sources. The gateway is a verbatim passthrough; if you put options at the top level they reach docling but get silently ignored, and you’ll see defaults instead — most visibly: no \f page-break markers in markdown. Confirmed in production by an autostatement regression on 2026-05-23.

Images: default is dropped. The gateway interprets a top-level include_images: bool field on the docling payload (default false). When false the gateway both (a) sets docling’s image_export_mode=placeholder to suppress server-side rendering of images into the result, and (b) strips any image data that does land in the result before storing it. The strip nulls result.document.json_content.pages[<n>].image (the rendered page bitmaps — the main bloat source on real PDFs) and pictures[].image (semantic-object detections), and replaces inline base64 data URIs in md_content / html_content with placeholders. Set include_images: true if you actually need the image bytes (e.g. an upcoming vision-model interpreter); be aware that a single PDF page can produce 1–10 MB of base64 image data at the default 144 dpi and the gateway stores the full result for the job-TTL window. Background: plans/docling-image-handling.md.

import base64, os, time, requests

GATEWAY = os.environ["LLM_GATEWAY_URL"]
HEADERS = {"X-Caller-Id": "my-doc-ingest"}

with open("annual-report.pdf", "rb") as f:
    pdf_b64 = base64.b64encode(f.read()).decode("ascii")

submit = requests.post(
    f"{GATEWAY}/v1/jobs",
    headers=HEADERS,
    json={
        "endpoint": "/v1/convert/source/async",
        "backend": "docling",
        "priority": "batch",
        "payload": {
            "sources": [
                {
                    "kind": "file",
                    "base64_string": pdf_b64,
                    "filename": "annual-report.pdf",
                }
            ],
            # Gateway-level knob. Default is false (drop images).
            # Sibling of `sources` / `options`; the gateway extracts
            # it from the payload before forwarding to docling.
            # "include_images": False,
            # All docling conversion knobs go under `options`.
            # Putting them at the top level alongside `sources`
            # results in docling silently using defaults — see the
            # warning above this snippet.
            "options": {
                "to_formats": ["md", "json"],
                "do_ocr": True,
                "do_table_structure": True,
                "md_page_break_placeholder": "\f",
            },
        },
    },
)
submit.raise_for_status()
job_id = submit.json()["id"]

while True:
    r = requests.get(f"{GATEWAY}/v1/jobs/{job_id}", headers=HEADERS)
    r.raise_for_status()
    job = r.json()
    if job["status"] == "completed":
        result = job["result"]  # the verbatim docling result JSON
        break
    if job["status"] == "failed":
        raise RuntimeError(job["error"])
    time.sleep(5)

The docling result shape — markdown, JSON document tree, etc. — is whatever docling returns at /v1/result/{task_id}. The gateway is a pass-through for that body: it polls docling’s async task endpoints internally and stores the final result body, so you never see a task wrapper (task_id / task_status) — the same document.md_content your code would read from a synchronous /v1/convert/file call is at the same path. Skeleton of a completed job (default include_images: false):

{
  "id": "<job id>",
  "status": "completed",
  "backend": "docling",
  "result": {
    "document": {
      "md_content": "…markdown; inline images appear as data:image/png;base64,STRIPPED_BY_GATEWAY…",
      "html_content": "…same stripping rule…",
      "json_content": {
        "pages":    { "1": { "size": {}, "image": null, "page_no": 1 } },
        "pictures": [ { "image": null } ]
      }
    }
  }
}

Sibling keys of document are whatever your docling version returns — the gateway neither adds nor removes them. With include_images: true, image fields carry real data instead of placeholders/nulls. A 5-line regression test that asserts result["document"]["md_content"].count("\f") >= n_pages would catch a future shape regression like the 2026-05-23 incident in seconds.

Job size caps. Two timeouts apply, both server-side and configured on the gateway / docling services in docker-compose.yml:

• GATEWAY_DOCLING_TIMEOUT (default 1200 s) — how long the gateway will keep polling docling before giving up on a job and marking it failed. Includes submission + polling + result fetch.
• DOCLING_SERVE_MAX_DOCUMENT_TIMEOUT (900 s here) — docling’s own per-document processing budget. Weaker than it sounds, and worth knowing precisely, because it is the only backstop on a conversion the gateway has stopped waiting for:

  • It is cooperative, checked between page batches. A conversion wedged inside a single page or model stage never returns to that check and runs past it indefinitely.

  • It is per document, not per task — a submission with several sources can spend up to this long on each.

  • When it does fire, the document is not abandoned: docling truncates it and the task still reports task_status: success. So a conversion that hit the limit reaches you as a completed job with partial content, not as a failure.

    How to detect it. The gateway returns docling’s result body verbatim, so the signal is in your result:

    { "status": "partial_success", "errors": [ … ], "document": { … } }

    Compare result.status against "success". Note the wire value is the lowercase "partial_success" — PARTIAL_SUCCESS is the symbol name in docling’s source, not what arrives in JSON, so a consumer matching the upper-case form never matches. result.errors carries the detail. (Confirmed against the ConversionStatus enum docling-serve v1.16.1 publishes at GET /openapi.json.)

    Make this a hard check, not a log line, if truncation would be silent data loss for you — a statement missing its last pages still parses perfectly cleanly.

For ~150-page annual-report PDFs, both caps are comfortable. If you have larger documents, raise both before pushing them through.

GATEWAY_DOCLING_TIMEOUT is a per-dispatch cap with the same semantics as its ollama counterpart — it excludes queue wait, and it starts later than status=running. Undercut it for a single conversion with timeout_seconds on the submit body; a value above 1200 is rejected at submit. See Timeouts: which clock measures what before sizing either that or a client-side bound against it.

Checking platform load

GET /queue returns a snapshot without affecting the queue:

{
  "in_flight": 1,
  "queued": { "interactive": 0, "batch": 3 },
  "running_by_tier": { "interactive": 0, "batch": 1 },
  "waiting_for_budget": 0,
  "completed_last_24h": 47,
  "failed_last_24h": 2,
  "worker_alive": true,
  "storage": {
    "db_bytes": 41582592,
    "wal_bytes": 4157440,
    "retention": {
      "default_ttl_seconds": 259200,
      "overrides_by_caller": { "autostatement": 2592000 },
      "sweep_interval_seconds": 3600
    }
  },
  "timestamp": "2026-05-23T09:25:01.124+00:00"
}

storage is the job database on disk — operator-facing, and reported because nothing did before: in May 2026 it reached 6.8 GB on 453 rows (a backend returning images inline, stored verbatim) and the first symptom was the gateway appearing dead. It rises with what you have submitted over one retention window — the retention sub-object beside it says which window, which is the other half of reading this number. Nothing caps it, and nothing runs VACUUM, so it does not fall back after a sweep. Worth watching before an unusually large batch, or if you hold a longer window.

in_flight is 0 or 1 — there is a single global slot across all backends (see Single global in-flight slot above). Hit this before a non-urgent submission if you want to be a good citizen: if queued.batch is deep, you might choose to defer. worker_alive is true in normal operation. false means the one coroutine that runs jobs is gone and nothing will drain the queue until an operator restarts the gateway — your jobs will sit queued rather than fail, so this is the field that tells you a stall is the platform and not your submission. Worth reading before you go looking for a fault on your side. timestamp is the gateway’s wall-clock at snapshot time; use it to correlate samples across time without relying on the relative lifecycle.last_activity_age_seconds.

The response also includes a lifecycle block with the on-demand large-instance state and (when running) two uptime views:

{
  "lifecycle": {
    "state": "ready",
    "in_flight": 0,
    "last_activity_age_seconds": 12.4,
    "last_error": null,
    "session": {
      "started_at": "2026-05-23T08:12:01.034+00:00",
      "uptime_seconds": 4501.2
    },
    "month_to_date": {
      "wall_hours": 18.42,
      "ocpu_hours": 331.6,
      "gb_hours": 1768.3,
      "month_start": "2026-05-01T00:00:00+00:00",
      "next_reset": "2026-06-01T00:00:00+00:00"
    }
  }
}

session is null when the large is stopped. month_to_date is informational today — useful if you want to know roughly how much of the monthly OCI free-tier budget has been spent so far this UTC month. A future release will surface remaining-budget on each job-status response and add a dedicated /usage endpoint with the same data; until then /queue.lifecycle.month_to_date is the place to look.

Knowing when the platform is ready for your call

For decisions that need to happen before you submit — “is the backend even up?”, “is the model I want already in RAM, or am I about to pay a cold-load cost?” — GET /diagnostics gives you a single derived state field plus a few denormalized lists:

{
  "state": "ready_warm",
  "warm": true,
  "docling_ok": true,
  "loaded_models": ["qwen2.5:72b-instruct-q4_K_M"],
  "available_models": [
    "qwen2.5:72b-instruct-q4_K_M",
    "qwen3:8b-q4_K_M"
  ],
  "ollama": {
    "url": "http://ollama:11434",
    "version": "0.32.15",
    "ps":   { "models": [ { "name": "qwen2.5:72b-instruct-q4_K_M", "size_vram": 0, "...": "…" } ] },
    "tags": { "models": [ { "name": "qwen2.5:72b-instruct-q4_K_M", "digest": "…", "...": "…" } ] }
  },
  "docling": { "url": "http://docling:5001", "health": { "status_code": 200, "body": "…" } },
  "lifecycle": { ... state, in_flight, last_activity_age_seconds, last_error, warm — same as /queue's lifecycle EXCEPT /queue additionally embeds session / month_to_date / large_shape (the uptime-accounting block) ... },
  "timestamp": "2026-05-23T09:25:02.341+00:00"
}

Two shape rules on the ollama object worth knowing:

• ollama.version is always present — a string when ollama is reachable, null when it isn’t. If you observe the key itself missing, you are talking to a gateway built before 2026-08-04; that’s a deployment defect to report, not a documented state.
• version is a denormalized string, but ps and tags are raw probe bodies — on a probe failure they carry {"error": "…"} instead of their normal shape. So when version is null, the reason is directly readable in the sibling fields.

state is the single source of truth for “what’s the platform doing right now,” and is one of:

Changed 2026-08-13: a docling-only outage no longer shows as backend_unhealthy. It used to, which meant one dead docling blanket-told every consumer to stop submitting — for a backend most of them don’t use — while also blocking all ollama work at the controller for 30 minutes. docling now has its own top-level boolean, docling_ok: a live probe of docling’s /health taken while serving that request, never cached, so a docling that just recovered reads true on the very next call with no gateway restart. docling_ok: false does not mean “don’t submit” — it means docling jobs will fail fast with docling unreachable: … until it is back, while ollama work is unaffected. If you gate docling submissions on platform state, switch from state to docling_ok.

On the local deployment, docling_ok: false is even weaker than that. Since 2026-08-18 a stopped docling is started on demand for the job that needs it (see One heavyweight backend at a time), so a false reading there means “not up right now”, not “your job will fail”. Check residency.enabled in the same response if you need to tell the two apart — but the simpler answer is not to pre-gate at all: submit, and read the job.

Sleep and wake — the operational model in one paragraph. The on-demand large instance auto-stops itself after LIFECYCLE_IDLE_TIMEOUT_SECONDS (default 7200 s / 2 h) of no gateway dispatches, and auto-wakes on the next job submission — the gateway intercepts every job submit and, if the large is stopped, issues an OCI start before dispatching. Consumers don’t need to explicitly trigger wake. There is no “operator paused the platform, hold your submission” state in this API; that scenario is covered by disabled (controller off entirely) or failed (lifecycle gave up). For every other state a consumer can safely submit and let the gateway handle whatever’s needed; the phase field on /v1/jobs/{id} will then narrate boot → load → generation.

warm is the convenience boolean: lifecycle == ready AND at least one model in RAM. True implies your next ollama inference starts immediately — no boot, no load. Use warm for the simple “is this a fast path?” check; use state when you need the full picture.

loaded_models and available_models let you check whether the specific model you want is resident (fast next call) or just on disk (will pay ~1–3 min load). If the model you need isn’t in available_models at all, you’ve targeted the wrong host or the model hasn’t been refreshed there yet — that’s an operator issue, not something to retry around.

The raw ollama.ps, docling.health and lifecycle blocks are there for ops debugging. Each backend reports errors in-band, so a single broken backend doesn’t blind you to the other — useful when you’re triaging which side is having a bad day.

ollama.tags is the exception — it is a supported read, because this page tells you to take weight-level provenance from ollama.tags[].digest (see Picking a model name). Don’t infer from its neighbours here that it can vanish.

ollama.ps[].expires_at is not a liveness signal. Don’t read it as one. It is the last request boundary plus OLLAMA_KEEP_ALIVE — while a request is actually in flight the runner is pinned by refcount and the field is never refreshed. So on the local deployment any generation longer than the keep-alive window shows an expires_at in the past while the job is perfectly healthy; on OCI (keep-alive -1) it doesn’t behave that way at all. That window was 300 s when this was measured, which made a past expires_at routine on batch work; it is 3600 s since 2026-08-16, so the same trap now needs an hour-long generation to spring. Don’t code to either number — the point is that the field is a boundary stamp, not a heartbeat. Confirmed by sampling /api/ps every 3 s through a live generation: frozen at one value for every sample. A past expires_at tells you nothing about whether a job is progressing or wedged. For that, read the running job’s own tokens_generated and last_token_at (Watching a running job), with status + phase for what the gateway is doing around it.

Checking recent throughput

GET /metrics/recent returns the last finished jobs and what they actually achieved — newest first, one row each:

{
  "jobs": [
    {
      "job_id": "9f3c1e77-3f4a-4a1e-9a4a-8c2f1b7d0e55",
      "completed_at": "2026-08-09T15:47:03.882+00:00",
      "outcome": "completed",
      "error": null,
      "backend": "ollama",
      "model": "qwen3:8b-q4_K_M",
      "deployment": "local",
      "engine_version": "0.32.5",
      "caller_id": "regnskap",
      "tier": "batch",
      "input_bytes": 14203,
      "input_tokens": 3450,
      "output_tokens": 64,
      "input_pages": null,
      "output_bytes": 2841,
      "prefill_tok_s": 7.5,
      "decode_tok_s": 0.32,
      "load_ms": 96000,
      "queue_wait_ms": 0,
      "run_ms": 660000,
      "large_ocpu": 8,
      "large_gb": 24
    }
  ],
  "count": 1,
  "deployments": ["local"],
  "filter": {
    "backend": null, "deployment": null, "outcome": null,
    "model": null, "limit": 50
  },
  "timestamp": "2026-08-09T15:52:10.004+00:00"
}

This answers the question a single job can’t: is that rate normal here lately? A running job’s tokens_per_second tells you it is moving; this tells you whether it is moving at the speed this box normally manages. The row above is from the degraded window of 2026-08-09, where decode ran at 0.32 t/s against the 3.6–5.0 t/s the same work does on the same box on an ordinary day — visible in one request, where at the time it took an hour and a half of digging.

Filters, all optional and all absent by default:

How to read it without drawing the wrong conclusion:

• Compare like with like — same model, same deployment. A 72B and an 8B have nothing to say about each other, and the two deployments are different machines running different ollama pins. The deployments list shows every label present in the rows you got back; more than one means read it again with ?deployment=.
• outcome: "failed" rows ended early — cancelled, or out of time. They carry output_tokens (how far the work got before it was abandoned) but no rates, because there was no final response to read ollama’s timings from. Filter them out before comparing throughput, and read them for a different question: how often work is being thrown away, and how much of it.
• error says which kind of failure it was. outcome flattens four different problems into the word failed — you cancelled it, it hit your timeout_seconds, it hit the platform’s cap, or the backend errored. error carries the same string the job’s own record does, and outlives it: three days later the job is swept and this is the only place left that can tell a cancel from a timeout. null on completed jobs, and on rows written before this was recorded.
• No average, median or verdict is offered, deliberately. Run-to-run variance on byte-identical work measured ±39% on the local box, so a single summary figure would have looked healthy right through the incident this endpoint was built after. You set the threshold that matters to you; the platform reports what happened.
• Every row carries every field, null where a value doesn’t exist — never 0. A docling row has input_pages and null token rates; an ollama row the reverse. A cancelled job has a null decode rate because nobody measured one, and a 0 there would read as “it ran at no speed at all” to someone already hunting a problem.
• large_ocpu / large_gb are the machine that ran the model, as the gateway was configured to believe it. Correct on any deployment whose LARGE_INSTANCE_OCPU / LARGE_INSTANCE_GB are set to match reality — which for a local stack means setting them (see .env.example); left unset, a laptop stamps every row with the OCI machine’s shape.
• A row appears shortly after the job goes terminal, not at the same instant. The gateway writes the job’s final status, reads the engine version, then inserts the row. If you poll for completed and query this immediately, the row may not be there yet — retry rather than concluding it wasn’t recorded.
• decode_tok_s is ollama’s own number (eval_count ÷ eval_duration), where a running job’s tokens_per_second is the gateway counting tokens as they arrive. They should agree closely. The gateway’s is available while the job runs; ollama’s only once it has finished.
• Ignore prefill_tok_s in the thousands — that is a cache hit, not hardware. Resubmit a prompt whose prefix ollama still has in its KV cache and the prefill is skipped, so prompt_eval_duration comes back near zero and the arithmetic produces a nonsense rate. A real run of the same 2972-token prompt on the local box reported 24 t/s; the repeat two minutes later reported 5443 t/s. The second number describes cache reuse. Compare prefill only across first-runs, or don’t compare it at all — decode_tok_s is the honest health signal.
• These rows outlive the jobs they describe. The metrics table is not swept by job retention at all, so a job whose GET /v1/jobs/{id} already returns 404 still appears here. It is not a way to retrieve results — there is no result in these rows, by design.

Reference: end-to-end consumer pattern

The minimal examples earlier in this page elide a few things that matter in real consumers — pre-flight /diagnostics check, phase-aware UX, robust timeout, and explicit failure handling. Copy-paste the snippet below as a starting point rather than re-derive it. It assumes the model name and Choosing a priority for your job sections above have been read.

"""Minimal but production-shaped consumer of the platform gateway.

Reads LLM_GATEWAY_URL and LLM_MODEL from env (no defaults — failing
loudly beats defaulting to a wrong gateway, see #picking-a-model-name).
"""
import os
import time
from typing import Callable, Optional

import requests

GATEWAY = os.environ["LLM_GATEWAY_URL"]    # e.g. http://llm-gateway:11435
MODEL = os.environ["LLM_MODEL"]            # e.g. qwen2.5:72b-instruct-q4_K_M
CALLER = "my-consumer"                     # short stable identifier; logged
HEADERS = {"X-Caller-Id": CALLER}
PRIORITY = "batch"                         # "interactive" or "batch"; default batch


class GatewayError(RuntimeError):
    """Anything the gateway tells us went wrong. .job is the recorded
    row when the failure was a job-level error (so callers can read
    .job['error'], .job['phase'], etc.); None for pre-submit failures."""
    def __init__(self, message: str, job: Optional[dict] = None):
        super().__init__(message)
        self.job = job


def diagnostics() -> dict:
    r = requests.get(f"{GATEWAY}/diagnostics", timeout=5)
    r.raise_for_status()
    return r.json()


def preflight(model: str, backend: str = "ollama") -> dict:
    """Confirm the gateway will accept work for the given model.
    Raises GatewayError if the platform is in a non-submittable state
    or the model isn't available on the target host."""
    d = diagnostics()
    if d["state"] in ("failed", "backend_unhealthy"):
        # `backend_unhealthy` is ollama-specific since 2026-08-13.
        raise GatewayError(f"platform not ready: state={d['state']}")
    if (backend == "docling"
            and d["state"] not in ("stopped", "starting", "stopping")
            and not d["docling_ok"]):
        # docling health is its own field, not part of `state`. Skipped
        # whenever the large is asleep or in transition: docling lives
        # on the large, so nothing is reachable then, and submitting is
        # what wakes it — the job just queues until the boot finishes.
        raise GatewayError("docling is not responding; retry later")
    if model not in d["available_models"] and d["available_models"]:
        # Empty list means we couldn't reach ollama (likely instance
        # asleep); submission will trigger wake and we'll learn for
        # sure then. Only fail loudly when we have a definitive list.
        raise GatewayError(
            f"model {model!r} not on disk; available={d['available_models']}"
        )
    return d


def submit(backend: str, endpoint: str, payload: dict) -> str:
    r = requests.post(
        f"{GATEWAY}/v1/jobs",
        headers=HEADERS,
        json={
            "backend": backend,
            "endpoint": endpoint,
            "payload": payload,
            "priority": PRIORITY,
        },
        timeout=10,
    )
    r.raise_for_status()
    return r.json()["id"]


def wait_until_terminal(
    job_id: str,
    *,
    timeout_s: int = 7200,
    poll_interval_s: float = 5.0,
    on_phase_change: Optional[Callable[[Optional[str]], None]] = None,
) -> dict:
    """Poll /v1/jobs/{id} until completed or failed. Calls
    on_phase_change(phase) once per (deduped) transition — the right
    place to render UX state."""
    deadline = time.monotonic() + timeout_s
    last_phase: Optional[str] = "__init__"
    while time.monotonic() < deadline:
        r = requests.get(f"{GATEWAY}/v1/jobs/{job_id}",
                         headers=HEADERS, timeout=10)
        r.raise_for_status()
        job = r.json()
        phase = job.get("phase")
        if phase != last_phase:
            if on_phase_change:
                on_phase_change(phase)
            last_phase = phase
        if job["status"] == "completed":
            return job
        if job["status"] == "failed":
            raise GatewayError(f"job failed: {job.get('error')}", job=job)
        time.sleep(poll_interval_s)
    raise GatewayError(
        f"job {job_id} did not terminate within {timeout_s}s; "
        f"last phase was {last_phase!r}"
    )


# Example: chat completion
def chat(messages: list, timeout_s: int = 7200) -> str:
    preflight(MODEL)
    job_id = submit("ollama", "/api/chat", {"model": MODEL, "messages": messages})
    print(f"submitted {job_id}")
    job = wait_until_terminal(
        job_id,
        timeout_s=timeout_s,
        on_phase_change=lambda p: print(f"  phase={p}"),
    )
    return job["result"]["message"]["content"]


if __name__ == "__main__":
    print(chat([{"role": "user", "content": "Say pong."}]))

What this snippet encodes that the minimal examples don’t:

• Pre-flight /diagnostics with a defensive empty-list check. When the large is asleep, available_models is empty and we don’t fail loudly — submission will wake it and we’ll learn then. Only fail when we have a definitive list that excludes the requested model.
• Phase-change callback for UX state. Deduped against the previous phase so callers only see real transitions.
• Distinct error type that preserves the gateway’s recorded job row (including the final phase), so failure callers can read error.job["phase"] and render “failed during ollama_loading_model” instead of generic “failed.”
• Explicit timeout with the last-seen phase reported on expiry — far easier to diagnose than a bare TimeoutError.

Adapt the chat() example to your shape: docling jobs use submit("docling", "/v1/convert/source/async", {...}), embeddings use submit("ollama", "/api/embeddings", {...}). The submit/poll/phase plumbing stays identical.

Co-hosting a consumer on the small host

For a consumer that runs as its own compose project on the small host, next to the gateway — rather than on a laptop over the SSH tunnel. Everything in Calling the gateway still applies; this section is only the delta that co-location adds.

The compose contract

Your project is a sibling, not a member. Don’t join platform_default — stay decoupled so you keep starting when the platform is rebuilding (see Connecting).

name: your-project                     # own project name, not `platform`

services:
  worker:
    extra_hosts:
      - "llm-gateway:host-gateway"     # → the host's published :11435
    environment:
      - LLM_GATEWAY_URL=http://llm-gateway:11435
      - LLM_MODEL=qwen2.5:72b-instruct-q4_K_M
    volumes:
      - your_project_data:/data        # your own named volume
    restart: unless-stopped
    mem_limit: 512m                    # be explicit
    cpus: 0.5

volumes:
  your_project_data:

• Don’t publish 80, 443, 11435 or 5006 — all four are taken (public edge, gateway, docs vhost). Pick another port and let nginx proxy you if you need public exposure.
• Own named volume. Never write into llm_gateway_data.
• The small is 1 OCPU / 6 GB, shared with the gateway’s event loop and its SQLite writes. Politeness: poll no faster than every 5–10 s, no busy-wait loops, run a real WSGI/ASGI server (not Flask’s dev server), and set explicit mem_limit / cpus.

Runtime state comes from /queue, not from this page

This page is docs/consumer.md rendered by the docs vhost. It proves which doc SHA is deployed — never which environment variables the gateway is running with. The two drift: a feature can be documented here and switched off in /etc/environment, or the reverse. Don’t infer operational state from prose.

fuse_enabled reports the runtime constant itself. Note that FREE_TIER_FUSE_ENABLED defaults to false (gateway.py:82-84, docker-compose.yml:171) — it is on only if an operator set it on the host and force-recreated the container.

Pre-first-batch checklist

1. GET /queue — read free_tier.fuse_enabled, fuse.tripped, free_tier.binding_pct, month_to_date.next_reset, and budget-size the batch (below).
2. GET /diagnostics — state is not failed / backend_unhealthy; if the batch is docling work and the large is already awake (state not stopped / starting / stopping), docling_ok is true — it is not part of state, and it reads false whenever the large is asleep, which is not a fault; if the large is awake, available_models contains your pinned tag.
3. GET /status.json — record the live SHA next to your run so a later result diff can be attributed.

Budget sizing

On the current shapes — small 1 OCPU / 6 GB, large 18 OCPU / 96 GB — OCPU-h is the binding meter. (It was GB-h before the RAM cut; don’t hardcode either, read binding_metric.)

large budget ≈ (1500 OCPU-h − 1 OCPU × hours_in_month) ÷ 18 OCPU
             ≈ 42 wall-hours / month
usable       ≈ 42 × (stop_at_pct / 100)      ≈ 38 h at the default 90%
remaining    ≈ 42 × (1 − binding_pct / 100)

Re-derive if either shape changes — constants live in gateway.py:62-63 (LARGE_OCPU / LARGE_GB) and :87-88 (FREE_TIER_SMALL_OCPU / FREE_TIER_SMALL_GB), Makefile:737-740; the math is in budget.free_tier_status, budget.py:157-161.

A batch whose large-side wall-time exceeds remaining will not finish this month. It drains until the fuse trips, then parks the rest until next_reset (budget.py:104). Each cold start also spends ~20 min of the meter on the 72B disk→RAM load, so a batch spanning a month boundary pays that twice.

Job states when the fuse is tripped

Four behaviours that will break a naive poller:

1. running is not a latch. Parking is claim-then-demote: claim_next marks the job running, the fuse check runs, then set_waiting_for_budget demotes it with started_at cleared (JobStore.claim_next / .set_waiting_for_budget, gateway.py:783-824). A poll can land inside that window. Under a tripped fuse the queue’s head job legitimately cycles queued → running → waiting_for_budget about once a minute, indefinitely. Never infer “it started, a result is coming” from a single running observation.
2. Parking is rate-limited to one claim per FREE_TIER_BUDGET_CHECK_INTERVAL_SECONDS (default 60 s — BUDGET_CHECK_INTERVAL_SECONDS, gateway.py:94-96; the worker parks the job it claimed, then sleeps that long before claiming again, gateway.py:3373-3381) — and it re-claims the same head row every time. claim_next orders by tier then created_at and set_waiting_for_budget changes neither (JobStore.claim_next, gateway.py:783-810), so the head cycles indefinitely while everything behind it stays queued. Submit 100 jobs under a tripped fuse and /queue shows one waiting_for_budget and 99 queued, for as long as the fuse is up. Don’t assert prompt parking, and don’t read queued as “not yet noticed”.
3. Parked jobs are inert. They don’t count as activity for idle-stop — has_pending_or_running covers only queued and running (JobStore.has_pending_or_running, gateway.py:1186-1196). And a tripped fuse overrules the idle timer rather than queuing behind it: the idle tick converts its own “maybe stop” verdict into a fuse stop (LifecycleController._idle_tick, lifecycle.py:1534-1547), and the fuse-stop path commits without consulting that queued-work deferral at all — it waits only for an in-flight job to finish (lifecycle.py:1582-1593). So a queued pile can’t hold the large up while over budget. A pile parked under a tripped fuse burns nothing.
4. queue_position is absent while parked — computed only for status == "queued", None for anything else (JobStore.queue_position, gateway.py:757-781). Don’t drive progress UI from it.

DELETE /v1/jobs/{id} works on a held job (cancel_if_queued covers waiting_for_budget, gateway.py:977-991), so a consumer-side deadline → cancel is a valid escape. Held jobs are never swept — retention deletes only completed / failed rows (JobStore.cleanup, gateway.py:1050-1097) — so a month-long park loses nothing.

Model pre-flight when the large is asleep

Keep two signals separate:

• available_models (on disk) → correctness. Your fail-loud check.
• loaded_models / lifecycle.warm → latency only. A cold-but-available model is legal, just slower. Never gate correctness on these.

The trap: when the large is stopped — most of the time under an on-demand plus tripped-fuse regime — /diagnostics probes ollama, the probe fails, and available_models comes back [] (gateway.py:2117-2119). A naive “is my tag present?” check then fails closed and blocks every submission, indistinguishable from a genuinely missing model. Gate on state first:

Storage footprint on the small

Two things accumulate in the gateway’s SQLite on a 6 GB / 50 GB-boot host:

• Results are retained for your full window whether or not you’ve fetched them — collecting doesn’t free them server-side. Peak ≈ output bytes per job × jobs per day × retention days, so it scales with any override: autostatement’s 30 days is ten times the 72 h peak. Image-stripped docling JSON is far smaller than raw, but check the arithmetic against storage.db_bytes on /queue. Nothing runs VACUUM, so the file never shrinks back after a sweep — it only stops growing.
• job_metrics is deliberately never swept (the job_metrics table in the gateway’s SCHEMA) — one small row per finished job, kept past the jobs TTL on purpose so the CPU-scaling dataset survives, and readable via /metrics/recent long after the job record itself is gone. Rows are ~200 B, so thousands of jobs is single-digit MB.

Idempotency

There is no idempotency key on POST /v1/jobs. A gateway restart marks in-flight running jobs failed (“gateway restarted while job was running”) while queued jobs survive, so resubmission is a normal path — dedupe on your own job key; the gateway will not do it for you.

Pointing local dev at OCI (the default dev path)

For day-to-day consumer development, run your consumer locally and point it at the OCI gateway via an SSH tunnel. This gives you local hot-reload, editor tooling, and dev container niceties while the LLM and docling work runs on production-sized models in OCI.

Why this is the default rather than local platform-services:

• Model parity with prod. Your consumer’s prompts run against the same models prod will use. Behaviour you observe in dev is what consumers see in production.
• No 40+ GB model pulls on every new dev environment. Bootstrapping a new machine doesn’t have to wait on a giant download.
• Local machines can’t host the 72B models anyway. Local platform-services is sized for 8B models — useful for exercising wiring, not for evaluating output quality.

Reach for local platform-services instead when you have a concrete reason: working offline, intentionally testing the small models, or OCI is unavailable. Switching is one env var change.

Quickstart

Two tunnel patterns — pick by where your consumer runs:

• Inside a Docker dev container → Option B: tunnel from inside the container. The path that “just works” on Docker Desktop + WSL2 without host-side networking changes.
• Native on the WSL distro / Linux / macOS (no container) → Option A: tunnel from the host.

Either way, one-time bootstrap is the same: generate key → authorize on OCI → SSH alias → run tunnel. Errors: Troubleshooting.

How it works

Two viable topologies depending on where the consumer process runs. Same key, same OCI authorization, same SSH alias — only the location of the ssh -L and the consumer’s URL differ.

Option A — tunnel on the WSL host. ssh -L runs on the WSL distro; the consumer reaches it via host-gateway. Works cleanly for consumers running natively on the distro (no container). For consumer dev containers under Docker Desktop + WSL2, also requires WSL in mirrored networking mode — otherwise WSL-host-bound ports are invisible to containers, regardless of 0.0.0.0 bind.

consumer (in container or native) → host-gateway → ssh -L on WSL → OCI:localhost:11435

Option B — tunnel inside the dev container. ssh -L runs in the consumer container; the consumer hits its own loopback. Side- steps host-side networking entirely. Requires bind-mounting the WSL distro’s ~/.ssh into the container so the same key/config are available.

consumer (in container) → localhost → ssh -L in same container → OCI:localhost:11435

One SSH key per dev distro

Each WSL distro (or local machine) that wants the tunnel gets its own key. Don’t copy keys between distros — that defeats their isolation. In the consumer-project distro:

ssh-keygen -t ed25519 -C "<distro-name>-tunnel" -f ~/.ssh/oci_arm
cat ~/.ssh/oci_arm.pub

This is a separate key from the operator’s full-access key documented in ../README.md → SSH access. That key exists for managing the instance; this one is for tunnels only.

The key lives in the WSL distro’s ~/.ssh/ regardless of which tunnel pattern you use (see Run the tunnel). Under Option B the dev container bind-mounts that directory read-only and uses the same key — no copying, no separate identity. “One key per distro” means per WSL distro.

Authorize the key as tunnel-only

The public half goes onto OCI’s ~/.ssh/authorized_keys, prefixed with restrictions so the key cannot be used for anything except forwarding the gateway port:

command="echo tunnel-only access; exit 1",no-pty,no-agent-forwarding,no-X11-forwarding,no-user-rc,permitopen="localhost:11435" ssh-ed25519 AAAA... <distro-name>-tunnel

Why not the restrict umbrella keyword? OpenSSH’s restrict is the documented one-word equivalent of the four no-* options below. But on OpenSSH 9.6p1 (Ubuntu 24.04, current OCI image) we observed that restrict,permitopen=... parses correctly — sshd’s debug log shows the permitopen target listed — yet forwarding gets denied with administratively prohibited anyway. The expanded form (each restriction named individually) behaves correctly. Verified on 2026-05-19 during the autostatement onboarding; the pattern in this doc deliberately avoids restrict so future onboardings don’t repeat the debug session.

Where this command runs. Not from the consumer distro — that’s the distro we’re granting access, so it can’t authorize itself yet. The append runs from a machine that already has admin SSH to OCI — typically the operator’s platform-services WSL distro. One-liner from there:

echo 'command="echo tunnel-only access; exit 1",no-pty,no-agent-forwarding,no-X11-forwarding,no-user-rc,permitopen="localhost:11435" ssh-ed25519 AAAA... <distro-name>-tunnel' \
  | ssh oci-arm 'cat >> ~/.ssh/authorized_keys'

Verify it landed:

ssh oci-arm 'tail -1 ~/.ssh/authorized_keys'
ssh oci-arm 'tail -1 ~/.ssh/authorized_keys' | grep -oE 'ssh-ed25519 \S+ \S+$' | ssh-keygen -lf -

The grep -oE step strips the long options prefix (which contains spaces inside command="...") and isolates the bare <keytype> <keydata> <comment> so ssh-keygen -lf - can fingerprint it. Cross-check the printed fingerprint against the one the consumer distro printed in Generate the key (step 1). If they match, the key is intact through paste.

After this one-time bootstrap, the consumer distro talks to OCI directly forever — the operator distro is just the “trusted introducer” that vouches for the new key on day one.

What each option does:

• command="echo tunnel-only access; exit 1" — forces every non-tunnel session to print and exit, blocking ssh oci-arm 'cmd' from running arbitrary commands. Port forwarding uses a separate SSH protocol path and runs before the forced command, so forwards still work.
• no-pty — denies PTY allocation (no interactive shell).
• no-agent-forwarding — blocks SSH agent forwarding.
• no-X11-forwarding — blocks X11 forwarding.
• no-user-rc — blocks execution of ~/.ssh/rc at login.
• permitopen="localhost:11435" — permits forwarding to llm-gateway only.

If this key leaks, the worst an attacker can do is open forwards to the gateway port. They can’t get a shell, run commands, read files, forward agent or X11, or pivot to any other port.

SSH config alias

In the consumer-project distro’s ~/.ssh/config:

Host oci-arm
  HostName 79.76.60.187
  User ubuntu
  IdentityFile ~/.ssh/oci_arm
  IdentitiesOnly yes

Run the tunnel

Pick the option matching where your consumer runs. permitopen on OCI checks only the remote destination, so the same authorized_keys line works for either option.

Option A — from the WSL host

ssh -L 0.0.0.0:11435:localhost:11435 oci-arm -N

The 0.0.0.0 bind (not the default 127.0.0.1) is what lets a consumer dev container reach the port via host-gateway. On Docker Desktop + WSL2 this also requires the distro to be in mirrored networking mode — in Windows ~/.wslconfig:

[wsl2]
networkingMode=mirrored

then wsl --shutdown from PowerShell. Without mirroring, WSL-bound ports aren’t visible to containers no matter how the tunnel binds, and Option B is the right path.

Consumer-side wiring:

extra_hosts:
  - "llm-gateway:host-gateway"
environment:
  LLM_GATEWAY_URL: "http://llm-gateway:11435"

Option B — from inside the dev container

Bind-mount the WSL distro’s ~/.ssh into the container so the same key, config, and known_hosts from the bootstrap are available. In the consumer’s devcontainer.json, add to mounts:

"source=${localEnv:HOME}/.ssh,target=/home/vscode/.ssh,type=bind,readonly"

(${localEnv:HOME} resolves to the WSL distro’s $HOME. Adjust /home/vscode to your container user.) Rebuild the container. Then from a terminal inside it:

ssh -L 11435:localhost:11435 oci-arm -N

No 0.0.0.0 needed — the tunnel and the consumer share the container’s loopback. Consumer-side wiring:

environment:
  LLM_GATEWAY_URL: "http://localhost:11435"

No llm-gateway:host-gateway entry in extra_hosts — the URL points at the container’s own loopback.

Persistence

SSH tunnels die with sleep, network changes, or laptop suspend. Under Option A: wrap with autossh or a systemd --user unit. Under Option B: re-run in the container terminal, or add a postStartCommand that backgrounds it. Skip both until it actually annoys you.

Sharing localhost with a local platform-services

If you run the tunnel from the same machine that already has a local platform-services compose stack up, port 11435 is already claimed by the local stack. Bind the tunnel to an alternate local port instead — pick something far enough from the regular range to be unambiguous (the 21000 prefix is a useful convention):

ssh -L 21435:localhost:11435 oci-arm -N

This is the convention make tunnel in platform-services itself uses — it binds *:21435 precisely so the local stack on :11435 and the OCI tunnel can coexist.

The consumer chooses which to hit by switching its LLM_GATEWAY_URL env var between the two port numbers:

• http://llm-gateway:11435 → local platform-services
• http://llm-gateway:21435 → OCI via tunnel

The extra_hosts mapping is identical for both; only the port number discriminates. permitopen on the OCI side only checks the remote target (localhost:11435), never the local bind port — so the same authorized_keys line works regardless of which local port you pick.

Diagnosing a wrong-port hit. If your consumer points at :11435 and you have a local stack running, the call lands on the local stack silently — there’s no error, just data from the wrong place. The reliable tell in a GET /queue response is lifecycle.state:

• "disabled" → you are on a local gateway. LIFECYCLE_ENABLED defaults to false off OCI, so a local gateway reports disabled and never manages an instance.
• Any of stopped / starting / ready / … → you are on OCI, where the lifecycle controller is enabled.

Don’t try to tell them apart by response shape (missing timestamp, in_flight_by_backend instead of a scalar in_flight, absent lifecycle block). Those only ever indicated an out-of-date gateway, and a local stack built from the current repo is byte-identical to OCI’s — same code, same API. Shape tells you the version, not the location.

If you’re on :11435 and meant to reach OCI, switch to :21435 (assuming make tunnel is up) and re-check.

How granular the local↔︎OCI switch is depends on how your consumer reads the env var. If it’s baked in at container start (e.g. fixed in the compose file), switching means a container restart. If it’s read per process invocation (common Python pattern: os.environ.get(...) inside the entry point), you can flip per command — and even run one process against OCI and another against local simultaneously from the same container:

LLM_GATEWAY_URL=http://llm-gateway:21435 \
  python -m scripts.coverage_probe_pass2 --full

Your devcontainer can quietly take the gateway’s port

Port politeness on a shared dev machine is the same idea as Co-hosting a consumer on the small host above, with one extra hazard that has nothing to do with compose: your editor forwards ports too, and it picks the host port silently.

VS Code auto-forwards ports from a devcontainer to the host. When the host port it wants is already taken, it doesn’t warn or fail — it takes the next free port. Which is how a consumer asking for a perfectly reasonable port ends up holding the gateway’s.

Observed 2026-08-12, and worth reading as the shape rather than the specific numbers: a consumer’s devcontainer forwarded container port 11434 (ollama). Host 11434 was already published by the local docker-compose.local.yml ollama, so VS Code took 11435 — the gateway’s port. The platform stack was then unreachable on the dev machine for as long as that devcontainer stayed open.

The symptom does not look like a port conflict. Connections hang and time out having received zero bytes, rather than being refused. A refusal would have said “nothing is listening”; a hang says “something accepted and never answered”, which reads like a wedged gateway. Every probe behaves this way — curl, make queue, make status.

Two further tells that point away from the real cause, so know them in advance:

• docker compose ps shows the gateway with a bare 8080/tcp and no host mapping — while docker inspect <container> --format '{{json .HostConfig.PortBindings}}' still shows 11435 was requested. Compose asked for the publish, Docker couldn’t take the port, and nothing raised an error. --force-recreate reproduces it exactly, which makes it look like a broken compose file.
• Other containers’ ports work fine, because nothing collided with them. “Publishing works, just not for this one service” is the fingerprint.

Check it in two places. In VS Code, the PORTS panel — and read the Forwarded Address column, not the Port column: the row is labelled with the container port, so the one holding 11435 may be listed under 11434. On Windows:

Get-NetTCPConnection -LocalPort 11435 -ErrorAction SilentlyContinue |
  Group-Object State | Select-Object Name,Count

A Listen entry owned by Code.exe is the confirmation.

Fix it by right-clicking the offending row → Stop Forwarding Port. That releases the listener, but sockets the extension already leaked stay in CloseWait until that editor window exits (1150 of them in the observed case) — harmless, since they cannot block a new bind, but don’t read them as “still broken”.

Stop it recurring in the consumer’s own .devcontainer/devcontainer.json:

"portsAttributes": {
  "11434": { "onAutoForward": "ignore" }
}

The general rule: a consumer devcontainer should not auto-forward any port the platform uses — 11435 (gateway), 5006 (docs vhost), 21435 (the OCI tunnel). Forwarding a port near them is enough to cause this, because the next-free-port rule walks upward into the range.

Shorter than it used to be, as of 2026-08-19. 11434 (ollama) and 5001 (docling) are no longer on that list because the local backends publish no host ports at all — they talk to the gateway over the container network, where nothing on Windows can reach them. This followed a second incident of the same kind: docling ran healthy for 20 minutes while the gateway could not reach it, because its host port had silently stopped being published. Keeping those two out of your portsAttributes costs nothing, but they can no longer collide with anything.

If the port does not come back, move the gateway instead. Docker can stay unable to publish a port after one failed attempt — observed 2026-08-12 surviving a daemon restart and a machine reboot, with the port provably free on the host and no container claiming it. Rather than keep hunting, set GATEWAY_HOST_PORT on that machine (see .env.example) and redeploy; the container port is unchanged, so a consumer only edits the port in its URL:

LLM_GATEWAY_URL=http://llm-gateway:31435

This is a dev-machine escape hatch, not a contract change — the default is still 11435, which is what OCI serves and what every example on this page assumes. Pick a replacement well away from 11434: the failure is a walk upward from a taken port, so 11436 is the worst available choice.

Trade-offs

• Latency. ~10–30 ms per request from Northern Europe to Stockholm. Negligible for 5–15 minute jobs where one extra round trip on submission is invisible against the work itself.
• Bandwidth on docling uploads. PDFs are base64’d in the request body and travel over the tunnel. A 5 MB PDF (~7 MB base64) on a 20 Mbps upstream is ~3 seconds before docling starts. Fine for “interpret one or a few documents at a time”; notable for batch ingestion.
• Shared compute. The deployed runtime has a single global in-flight slot across all backends. Heavy local dev usage queues behind (or in front of) prod traffic. The gateway’s priority tiers are how you cooperate — submit dev jobs with priority: batch so they don’t preempt interactive prod work.
• Tunnel lifecycle. SSH tunnels die with sleep, network changes, or laptop suspend. Either restart by hand or wrap in autossh.

When not to reach for the tunnel

• For a consumer deployed alongside platform-services on the OCI host. Use the regular extra_hosts: ["llm-gateway:host-gateway"] wiring directly — same Docker daemon, no tunnel needed.
• When the consumer needs to keep working offline. The tunnel needs the remote host to be reachable; if you need to work on a plane, fall back to local platform-services with the small models (set LLM_GATEWAY_URL to http://llm-gateway:11435 and LLM_MODEL to an 8B tag — output quality will differ from prod, but the wiring exercises end-to-end).
• When OCI is unavailable (capacity error on the on-demand large instance, region outage). Same fallback as above.

Troubleshooting

Migrating from direct backend access

If your consumer was wired to call ollama or docling directly, the migration is:

1. Add the gateway to extra_hosts (if it wasn’t already): "llm-gateway:host-gateway". Drop the old ollama: and docling: entries from the same list.
2. Replace direct backend calls with gateway job submissions. For ollama: POST /v1/jobs with endpoint: "/api/chat" (or whichever ollama path) and your old payload as payload. Then poll GET /v1/jobs/{id} until terminal. Worked example above.
3. For docling: switch from multipart upload to JSON+base64 submission via the gateway. Worked example above. The result shape is unchanged — the gateway returns docling’s /v1/result body verbatim under result.
4. If you were on the deprecated pattern A (joining platform_default), drop the networks: block — pattern A is gone. The extra_hosts: wiring above is the only supported shape.

The streaming-tokens code path (if you had one) doesn’t carry over — the gateway never streams to you. It streams from ollama internally, but the API stays submit-and-poll and hands you one complete body. For 5–15 minute generations nobody is watching tokens form anyway, and Watching a running job covers the part people actually wanted (is it moving, how fast). If you genuinely need tokens as they are produced, that’s a feature request, not a migration step.

What’s not here

• Authentication — the platform network is trusted; anything on it (or anything reaching the published gateway port) can use the gateway. No auth at the API layer.
• Rate limiting / per-caller quotas — not implemented. The gateway enforces priority and serialization but does not cap how much any one caller can submit.
• Authentication on the gateway — the X-Caller-Id header is trusted, not verified. Suitable for an internal network of cooperating consumers; not suitable for untrusted clients. (Note: this is about inbound auth. The outbound completion webhook is authenticated the other way — the gateway signs its callback to you with a per-caller HMAC secret so you can verify it. See Completion webhooks.)
• Streaming partial output from the gateway — clients see status changes via polling, not tokens-in-flight.
• Direct access to ollama or docling — internal to platform-services, see What about direct ollama / docling access? above.
• Tutorials for the ollama or docling APIs — see their upstream docs. (Exception: the docling submission shape used by the gateway is covered in Submitting a docling job above.)