Connecting to platform-services

This page is for consumer projects — apps and dev containers that need to use the shared LLM runtime (and, eventually, the public edge proxy) provided by platform-services.

How this works

Consumer projects reach platform-services through the llm-gateway — a priority-aware async job queue that is the single entry point for both LLM (ollama) and document parsing (docling) work. The gateway API and wiring are identical regardless of which deployment you point at.

Two deployments, one of them is the default

Both run the same gateway code and expose the same API. What differs is which backends the gateway dispatches to, and that’s one env var on the platform side (LARGE_INSTANCE_IP) — not a different contract for you.

Concretely:

• Consumers co-hosted on the OCI small run on the same Oracle host as platform-services and reach the gateway in-host.
• Local consumer development points at the OCI gateway via an SSH tunnel (covered in Pointing local dev at OCI below). This is the day-to-day dev path — the consumer code runs locally with all the usual editor/hot-reload niceties, but the actual LLM and docling work happens on the platform’s machines.
• Running consumers against local platform-services is the fallback, and also the path for model evaluation. The wiring is identical to OCI (same compose, same env vars, just a different gateway URL), so flipping back and forth is one env var change on your side. The platform operator brings up local LLM/docling backends with make local-up and points the gateway at them — see Running the LLM backends on this machine in dev-setup-windows.md. Whether local output is comparable to OCI depends entirely on which profile is loaded: against small (8B-class), local dev validates wiring and not output quality; against candidate, comparing output quality is the whole point. Ask /diagnostics rather than assuming — and note that local timings are never comparable to OCI’s, since the hardware is different.

Connecting

Consumers reach platform-services through the gateway via the published host port. Your consumer doesn’t join the platform_default network — it stays decoupled from platform-services’ lifecycle and keeps starting/running when the platform is rebuilding or down.

services:
  app:
    extra_hosts:
      - "llm-gateway:host-gateway"

Behind the scenes, llm-gateway resolves via /etc/hosts to the host gateway IP; TCP connections then land on the published host port (11435).

llm-gateway is an /etc/hosts alias, not a Docker-network name. Using it does not put you on platform_default: extra_hosts points the name at the Docker host, and the gateway is reached through its published port like any other host service. If your runtime can’t set extra_hosts — a VS Code devcontainer, for instance — address the host directly instead: http://host.docker.internal:<port>, :21435 for the OCI tunnel.

Don’t assume the local port is 11435. That is only the default — the local stack publishes whatever GATEWAY_HOST_PORT says, and it gets moved whenever something else on the machine has taken 11435 (one dev box runs it on 31435). Ask the operator rather than guessing, and confirm which stack answered before trusting it — a single /diagnostics response settles it.

Read state for this, not residency. state: "disabled" means the lifecycle controller is off, which is true of every non-OCI deployment and never true of OCI — OCI always reports a real lifecycle state (stopped, ready_warm, …). That is the signal.

residency.enabled: true corroborates local but is opt-in, so false proves nothing on its own: a local stack without the policy configured reports state: "disabled" with residency.enabled: false. Don’t pair the two as though both always hold. If they do disagree — residency.enabled: true beside a real lifecycle state — treat the answer as unknown and say so, rather than trusting whichever field you looked at first; that combination shouldn’t occur, and something is wrong with either the deployment or your read of it. And prefer the boolean’s shape over the string’s spelling when you code against this: state is prose we may reword, residency.enabled is a boolean that can’t drift.

That tells you local-vs-OCI, not which local box. Every local stack looks the same in those fields, and so does its deployment label — they name the shape of a setup, not a machine. When more than one machine can answer you, read host_id (see Which machine answered below).

URLs:

• llm-gateway → http://llm-gateway:11435 (private; via host-gateway + tunnel — see Pointing local dev at OCI below). 11435 is the default, not a fixed fact — a local stack publishes whatever GATEWAY_HOST_PORT says (see above). If your machine also runs a local platform-services stack on :11435, the OCI tunnel binds :21435 instead to avoid the port collision — point your consumer at 21435 for OCI work. See Sharing localhost with a local platform-services for the full pattern. platform-services’ own platform-tunnel already does this; the trap is consumer URLs pinned to :11435 silently landing on the local stack instead of the tunnel.
• platform-docs → https://soggplatform.dedyn.io/ (public TLS edge — reachable from anywhere, no extra_hosts wiring or tunnel needed)

Why the two endpoints have different reachability stories: the gateway is a write-capable LLM dispatch API with no auth/rate-limit yet, so it’s deliberately kept off the public internet. Docs are read-only and safe to publish, so they’re served straight from the public TLS edge — no per-distro setup required.

Don’t hardcode these in app code — parameterize them via environment variables (LLM_GATEWAY_URL, PLATFORM_DOCS_URL). Same code then works whether you run on the OCI host, on a local stack, or over the SSH tunnel documented further down.

When platform-services is up, your consumer reaches it. When it’s down, the consumer still starts — gateway calls fail at the call site with a connection error, not at compose-up time. That’s the design intent.

What about direct ollama / docling access?

Not supported. ollama and docling are internal services of platform-services and have no consumer contract. The gateway owns all backend dispatch, including:

• Priority-aware queueing (interactive vs batch tier)
• Single global in-flight slot across both backends
• Job-state persistence across client disconnects and gateway restarts
• On-demand large instance: since 2026-10-15 a worker machine for qwen3.8:27b, which the gateway starts when that model’s work has waited 10 minutes with every other machine busy, or 1 minute with none up, and stops again after LIFECYCLE_IDLE_TIMEOUT_SECONDS without a job

If you were previously wired to ollama:11434 or docling:5001, see Migrating from direct backend access below.

What’s served

Notes:

• platform-docs is on a public TLS edge — browsers, external agents, and consumer containers all reach it the same way. No extra_hosts or tunnel needed. The same content is also served on http://platform-docs:5006/ to containers on the OCI host itself (in-host shortcut); local-dev consumers should ignore that and use the public URL.
• llm-gateway is not on the public edge — it’s a write-capable LLM dispatch API with no auth yet, so it stays private. Reach it from a consumer container via the host-gateway alias plus either a local platform-services stack or the SSH tunnel documented below.
• /models.json is proxied live from ollama’s /api/tags via platform-docs — hit this for model discovery rather than hardcoding model tags in your config.
• ollama’s :11434 and docling’s :5001 are not part of the consumer contract, and since 2026-08-19 they are not reachable from outside Docker on any deployment. See Direct backend access is gone below — if any of your code reads those ports, it stopped working on that date.

Calling the gateway

The gateway exposes a uniform async job API for both backends. Submit a job, get an ID back immediately, poll for the result. Jobs are expected to run roughly 5–15 minutes on the current hardware (longer once we cut over to 72B-class models) — long enough that holding an HTTP connection open across the work is the wrong default.

Authentication

Send your caller token on every call: Authorization: Bearer <token>. The operator issues one per caller name, with separate names for prod and dev (regnskap, regnskap-dev), so a dev token cannot read prod jobs. The token’s name is your caller id. It is what caller_id and cancelled_by record, and what retention overrides and webhooks are keyed on. Leave X-Caller-Id out, or send the same name: a different one is refused.

GET /diagnostics → caller_auth says what this gateway enforces:

Read caller_auth before sending anything private, and send nothing unless it says required.

A 401 is configuration, so never retry it: no token under required, an unknown or revoked token, or an X-Caller-Id naming someone else (under either setting). It is not an error-contract class: tell it by its status. Keep the token in a file or a secret store, never on a command line or in a repo. If it is lost or leaked, the operator revokes it and issues a new one.

What a token does not hide:

• Root on the small can read payloads and results in SQLite until retention deletes them, 72 h by default. You can ask for a shorter retention for your caller (Retention).
• A docling job runs on a home worker machine, which receives its payload. A pool_group keeps your jobs on the machines the owner puts in that group (Worker machines).

Submission shape

{
  "endpoint": "/api/chat",
  "payload": { "model": "qwen3.8:27b", "messages": [...] },
  "priority": "interactive",
  "backend": "ollama",
  "timeout_seconds": 900,
  "callback_url": "https://hooks.your-app.example/gateway"
}

• endpoint — the upstream path to call, exactly one of: /api/chat, /api/generate, /api/embeddings, /api/show and /v1/chat/completions (ollama); /v1/convert/source/async (docling). Anything else — a near-miss spelling, or ollama’s administration such as /api/pull — is refused with 403 and no job is created; the body’s error starts endpoint not allowed: and allowed_endpoints lists this gateway’s kinds. A job a home worker machine runs can also fail with either class: model not pinned here: for a model the machine does not serve, endpoint not allowed: for a conversion of anything but attached files. Both are classes in /v1/error-contract, verdict never. Need another kind? Ask; it is a config line and a line here.
• payload — the body you would have POSTed directly to the backend, verbatim (with caveats noted per-backend below).
• priority — optional. interactive or batch. Omitted → recorded as batch. See Choosing a priority for your job below. An unknown value (e.g. urgent) is rejected with 400 rather than silently falling back to batch.
• backend — optional. ollama or docling. Omitted → the gateway derives it from the endpoint path (/v1/convert/* → docling, anything else → ollama). If you supply backend explicitly and it contradicts the endpoint (e.g. endpoint=/api/chat + backend=docling), the submit is rejected with 400 — there is one unambiguous dispatch path per job, and the gateway loud-fails rather than silently routing to the wrong backend.
• timeout_seconds — optional. A positive integer: how long this job is worth running once it reaches the backend. Omitted → only the deployment-wide cap applies. It measures exactly the same interval the deployment cap does — see Timeouts: which clock measures what below, and read that section before choosing a number, because the interval is narrower than “time since I submitted”. A value above your backend’s cap is rejected with 400; the gateway will not silently clamp it.
• callback_url — optional. An https URL the gateway POSTs a signed notification to when the job reaches a terminal state (completed or failed), so you can be push-notified instead of polling. Omitted → no callback, behaviour unchanged. Requires per-caller setup (a shared secret + host allow-list registered with the operator). See Completion webhooks below for the full contract. A callback object ({"url": "..."}) is also accepted as a forward-compatible alternative to the flat field.
• pool_group — optional. Which worker machines may run the job (Worker machines). Omitted or null → default, as every job was before groups. A job naming another group runs only on that group’s machines — never on the large unless the operator put it in the group — and waits queued while none is up. Groups are set up by the operator; ask for one. A name is lowercase letters, digits, _ and -, at most 30 characters, else 400. A group with no enabled machine listing the job’s model (or docling) is refused with 403 and no job is created; the error starts pool group not served:, verdict never. A named group is echoed as pool_group on the accept response and every status. Check the echo on first use: a gateway from before groups ignores the field and runs the job in default.

Choosing a priority for your job

Set priority on the submit body to one of:

• interactive — drained ahead of any queued batch work. For human-facing, latency-sensitive calls (editor assistant, chat UI).
• batch — drained when no interactive work is queued. For bulk ingestion, indexing, scheduled jobs. This is the default when priority is omitted.

Tier is per-job, not per-caller: a single consumer can mix interactive and batch submissions freely. There is no central caller→tier mapping; the gateway trusts whatever priority value the consumer declares (the trust model assumes a small number of fully-trusted consumers on the internal network).

Your caller id is your token’s name (Authentication). Without a token, set the X-Caller-Id request header to a short, stable identifier for your consumer project (e.g. my-editor-assistant). This is recorded on every job and surfaces in /v1/jobs/{id} and the gateway log — useful for tracing your own traffic and for the operator when debugging — but it does not drive tier assignment. If the header is absent, the gateway falls back to the peer IP as the caller identifier; don’t rely on the fallback, it’s a backstop.

Preemption is not supported. An interactive job submitted while a batch job is in flight waits for that batch job to finish (can be 5–15 min for docling work, longer for big inference). Once the in-flight slot frees, the interactive job dispatches before any queued batch — on the worker pool, to the first machine that finishes, whatever model it holds. If sub-batch-duration latency matters for your use case, raise it — the gateway design will need to change.

Single global in-flight slot

The large runs one job at a time across all backends. An ollama job in flight blocks a docling dispatch and vice versa; two queued jobs of any combination run sequentially. Worker machines are the exception: each runs one job of its own, in parallel (next section).

Why one slot, not one per backend: the on-demand large instance is CPU-bound on both workloads (no GPU on the free-tier ARM shape), so running ollama and docling in parallel halves each one’s throughput. Serial dispatch with priority tiers gives each job full CPU and keeps the queue model simple. Rationale and the trade-off in full: plans/priority-queueing.md Queue shape section.

The queue_position returned on submission counts every queued job ahead of yours that the large will run — 1 means “next to run,” regardless of which backend any of the queued jobs target. With no worker machine enabled that is every queued job.

Worker machines

Some models may also be served by worker machines: home computers the gateway hands jobs to, each running one job at a time, in parallel with the large and with each other. A job goes to the machines in its pool group — default unless it names one (pool_group, Submission shape) — when an enabled one lists its model, exact tag match or docling for a conversion; a default job goes to the large otherwise. Who submitted it plays no part. Nothing about submitting changes. What does:

• A machine-bound job waits for a machine. While no machine serving its model is live it stays queued. machines on /queue shows each one’s pool_group and which are live, stopped, silent, offline, paused, refused or disabled; for a default conversion, docling_ok on /diagnostics says whether one is.
• The OCI large is one of the machines for qwen3.8:27b since 2026-10-15 and qwen3:8b-q4_K_M since 2026-11-01, last in line. It starts when such a job has waited 10 minutes with every home machine serving it busy, or 1 minute with none of them up, so that job’s wait includes the large’s boot and, on its first job after a wake, about 8 minutes to load the 27B. It reads a prompt about as fast as a home machine.
• It can go back to queued if its machine goes silent for about two minutes, loses the job, or is paused for its owner’s use (Job lifecycle). The second time a machine loses it after it had started there — or the fifth time for any reason but a pause — it fails with machine lost: …, which is retry-safe.
• machine on a running, completed or failed job names the machine that ran it; it is absent when the large did. That job’s /metrics/recent row carries the machine as host_id and the commit its gateway was built from as gateway_commit — -dirty when it had uncommitted changes, as a developer’s machine in the pool may.
• queue_position of a machine-bound job counts the queued jobs for its model in its group ahead of it — an estimate, because among jobs of one priority a machine takes those for the model it already holds first.
• Cancelling one that is running records cancelled at once, with no partial_output.

One heavyweight backend at a time

Local deployment only, new 2026-08-18. On a RAM-tight dev box the platform keeps at most one heavyweight tenant in memory: an ollama model, or docling. When a job needs the one that isn’t loaded, the gateway evicts the resident one and brings up the one the job needs — then dispatches. The OCI deployment does not do this and is unchanged (96 GB fits both, so there is nothing to trade).

What changes for you: a docling job no longer fails just because docling is stopped.

before:  submit docling job → docling is stopped → job FAILED
                                                   "docling unreachable: …"

after:   submit docling job → docling is stopped → gateway starts it
                                                 → job waits ~15 s
                                                 → job runs normally

This is the fix for the 2026-08-18 outage in which eight jobs failed in 90 seconds because docling had been stopped between batches. Nobody has to announce a batch boundary any more, on either side.

A job waiting for a backend is running, not failed. While it waits, GET /v1/jobs/{id} reports status: running with phase set to evicting_backend (freeing the other backend’s RAM) or loading_backend (starting yours and waiting for its health check). Both are normal progress. As with every phase, treat them as a hint — don’t build control flow on them.

The wait does not consume your timeout_seconds. That field bounds the dispatch, and it starts when your job reaches the backend — same rule as the on-demand large’s boot. A job with timeout_seconds: 60 still gets its full 60 seconds after docling is up.

Nothing new can fail your job. If the platform can’t bring your backend up, the dispatch happens anyway and fails with the existing docling unreachable: … — the retry-safe class you already handle. There is no new error class and no new status to add to your matcher.

Cancelling during the wait. DELETE /v1/jobs/{id} on a job that is running but hasn’t reached its backend yet answers 409 with “job is running but not yet cancellable (dispatch has not started); retry in a moment”. This is pre-existing behaviour — the same answer you’d get during an OCI boot — and the window here is seconds, not minutes.

Send the docling half of a mixed batch first. A batch that alternates conversion and inference jobs pays a backend switch at every crossing; the same batch with all its conversions submitted first pays one. This is a recommendation, not a rule, and the platform does not reorder your jobs to achieve it — priority tier still outranks everything, so an interactive job from another consumer landing mid-batch can still force a switch. Worth doing anyway: it is free, and it is what a front-loaded pipeline does naturally.

Checking what’s resident. GET /diagnostics reports a residency block: enabled (false on OCI — everything else is then inert), resident (which backend the last dispatch arranged for), switches (how many crossings this gateway has paid) and last_error.

Job lifecycle

queued ─────► running ─────► completed
   │  ▲  ▲        │  │
   │  │  │        │  └────► failed (backend error, timeout, cancelled
   │  │  │        │                 via DELETE, gateway restart, machine lost)
   │  │  │        └─ back to queued (its worker machine lost it)
   │  │  └── waiting_for_large ◄── (the large is locked; see below)
   │  └── waiting_for_budget ◄── (free-tier cap; drains at month reset)
   └────► failed (cancelled via DELETE, or platform-initiated bulk cancel)

waiting_for_budget appears when the deployment runs the free-tier fuse — enabled on the OCI deployment since 2026-07-24, so plan for it (see “Free-tier budget cap” below). It’s non-terminal — the job loops back to running on its own once budget frees up. Treat it like queued.

waiting_for_large appears while the operator has locked the large: the gateway will not start it, so a job that needs it — one no worker machine serves — is held, non-terminal, until the lock is lifted, or a machine serving its model is enabled and it waits queued for that. Treat it like waiting_for_budget: DELETE works, timeout_seconds does not bound the wait. Both holds carry waiting_reason, a sentence saying why (absent on jobs held before 2026-10-04). Whether the large is locked right now is lifecycle.lock.locked on /queue and /diagnostics; state still reads stopped while it is, so check the lock before expecting a cold start.

A running job can go back to queued when it runs on a worker machine (see Worker machines) and that machine goes silent, loses the job or is paused. It keeps its place, reads moves: 1, loses started_at, and runs again when a machine takes it. Never read running as final.

If the gateway operator clears the queue (incident recovery, stuck backend, etc.), every queued job transitions to status=failed with the recorded error set to a message the operator chose — by default cancelled by platform services, but operators may include incident context (e.g. large instance OOM, restart pending). Treat this as a normal job-level failure on the consumer side: surface error to the user, resubmit if your workflow needs to retry.

The sweep reason can never impersonate a class you gate on. It is free text, so it shares a field with the machine-readable prefixes below — but since 2026-09-07 the gateway rejects a reason that would read as one: the two retry-safe prefixes, the pre-2026-08-18 runner-death wordings, and the bare string cancelled. Without that guard a sweep could be labelled ollama crashed: … and every consumer following The retry-safe classes would resubmit the work onto the queue an operator had just deliberately cleared. Incident context is unaffected and still lands verbatim.

cancelled_by says who asked for the cancel. error cannot: it records the fixed string cancelled whether you cancelled the job or an operator did, so those rows used to be byte-identical. On 2026-07-16 an operator loop cancelled 111 queued jobs belonging to a consumer who had no cancellation code at all, and settling that took until 2026-09-06 because nothing in the row said who did it. Since 2026-09-07 every cancel — per-job DELETE and the bulk sweep alike — records the canceller:

{"status": "failed", "error": "cancelled", "cancelled_by": "autostatement"}

The value is the canceller’s caller id: their token’s name, else their X-Caller-Id, else their IP, which is what an operator working at a shell looks like. Without a token it is self-declared, and anyone can send any caller id (Authentication). It answers “was this us?”, which is the question that actually cost a consumer seven weeks; it does not answer “who is authorised”.

Treat cancelled as terminal regardless of who appears here — a cancel is deliberate whoever asked for it. Use cancelled_by to know whether to go looking at your own code.

The field is absent, never null. A job nobody cancelled omits it, and so does any row recorded before 2026-09-07 — those cannot answer the question, and the gateway will not invent an answer by backfilling one. "cancelled_by" in body is the check, not body["cancelled_by"] is None.

Don’t try to detect a sweep by timestamp equality. For rows older than cancelled_by, and for any failure that isn’t a cancel, you are still reading shape rather than a field — and this is the shape people reach for first. It is tempting, and it is wrong. Only the bulk endpoint writes one shared completed_at across every row; a loop of per-job DELETEs writes distinct ones — measured on that 2026-07-16 event at 2.84 cancels/s over 39 s, every timestamp different, median gap 0.268 s. An equality test would have missed the only real mass-cancellation either side has on record.

Use volume instead. N identical error strings inside a short window means something systemic is happening — an operator clearing your work, or a backend that is down — and backing off is the right response to both, so you do not need to tell them apart. autostatement runs five identical errors within 120 s as an override to terminal, which is a sound default. This is the brake that makes an attempt cap safe: a genuine backend crash arrives one job at a time as each is dispatched, while anything systemic arrives in a burst.

A timeout failure has two possible causes, with two different error strings: your own timeout_seconds expiring, or the deployment-wide dispatch cap expiring. They call for different responses. See Timeouts: which clock measures what.

An empty completion is a failure only when nothing says the model ran. ollama answers 200 with an empty completion during a model-load race, and storing that as your result would hand you an empty string indistinguishable from a real answer. So the gateway fails it — but narrowly, because several legitimate replies are also empty and failing those would throw away work you already paid for. It fails only when the completion is empty and done_reason is absent or load. The error reads ollama returned an empty completion for /api/chat (done_reason=…, eval_count=…) — nothing indicates the generation ran, and is worth retrying.

These stay successful:

• done_reason: "stop" or "length" — the model ran and stopped on its own terms. A reasoning model whose token budget ran out mid-thinking has empty content, a full thinking, and done_reason: "length"; you get the whole body. Retrying that would only reproduce it — raise num_predict instead.
• A model load or unload — /api/chat with no messages (or /api/generate with no prompt) is ollama’s way to preload a model, and the same with keep_alive: 0 unloads it. Empty is the answer you asked for.
• A tool call — empty content beside a populated tool_calls.

Only /api/chat and /api/generate are checked. A docling result arriving with no document object fails on the same principle.

Transient backend blips no longer lose a conversion. A docling status poll or result fetch that 5xxs, drops its connection, or returns a truncated body is retried (five consecutive failures by default) rather than failing your job — the conversion itself keeps running on the large throughout. A poll that hangs is not retried: that means docling is wedged rather than blipping, and it still fails as docling HTTP timeout.

While the conversion is still running your timeout_seconds bounds this as before — past your deadline a failing poll is reported as your bound firing, not retried. The one exception is after docling reports the conversion finished: collecting the result is retried with no deadline clamp, deliberately, because abandoning a finished conversion throws away work already paid for. If docling goes dark exactly then, your job can fail up to five GATEWAY_DOCLING_HTTP_TIMEOUT windows plus their sleeps — roughly 5 minutes on the defaults — after timeout_seconds expired. Size any client-side abandon deadline with that tail in mind.

A backend that is down right now fails your job with <backend> unreachable: <detail> — e.g. docling unreachable: Cannot connect to host docling:5001 ssl:default [Connect call failed …]. Match the prefix docling unreachable: (or ollama unreachable:), never the whole string: the detail is the raw connector error and is not stable. This is a platform condition, not a fault in your payload, and resubmitting is unconditionally safe — the job never reached the backend, so unlike a dispatch timeout there is no possible partial side effect upstream. It is most likely in the couple of minutes after the large boots, and while it lasts GET /diagnostics reports docling_ok: false.

A backend that died mid-call fails your job with <backend> crashed: <detail> — e.g. ollama crashed: HTTP 500: {"error":"model runner has unexpectedly stopped, this may be due to resource limitations or an internal error, check ollama server logs for details"}. Match the prefix ollama crashed:, never the whole string: the detail is the backend’s own words relayed verbatim and is not stable. The prefix is backend-scoped like <backend> unreachable: above, but only ollama produces it — no docling crash has ever been recorded on this platform.

New 2026-08-18, and your existing substring match keeps working. Until this landed the error was relayed with no stable class and you were told to match ollama’s wording directly. The class wraps the relayed text rather than replacing it, deliberately, so model runner has unexpectedly stopped is still inside the string. Match the prefix for anything recorded from 2026-08-18 on; keep the substring check for as long as you read rows recorded before it, which carry no prefix.

Correction to what this section said before. The second wording, llama-server process has terminated: signal: killed, was documented here as an older engine’s phrasing. It isn’t. Both wordings came off the same engine and mark where the runner died: the first when a request’s context allocation failed, the second when it died completing a cold model load. Both are current, and both classify.

It can also arrive as ollama crashed: stream error: …. The gateway streams the generation endpoints internally, so a runner that dies after ollama already returned 200 reaches you over a different transport. Same event, same class — which is the point of matching the prefix rather than the HTTP 500 inside it. Every crash observed so far struck before the first token and took the first form.

What it means: ollama’s inference subprocess died. Overwhelmingly this is resource exhaustion on the host — the kernel’s OOM killer taking the runner because it was the largest process, triggered by whatever else happened to allocate next. Retry is safe and usually succeeds. The job produced no output and had no upstream side effect, so resubmitting is as safe as <backend> unreachable:.

Two things not to conclude from it. It is not a signal about your payload: measured on the local stack 2026-08-17, three of 77 runs died this way, each 4–7 s after dispatch having generated zero tokens, and each was clean on retry — including the one carrying the largest prompt in the set, which was the obvious suspect and was not the cause. And a single occurrence is not grounds for marking a filing permanently failed; treating it as terminal is how three retryable crashes became three missing results.

machine lost: … fails a job that worker machines lost twice after it had started on them, or five times in all, not counting a machine pausing — earlier losses only send it back to the queue. The gateway discards anything a lost machine sends afterwards, so resubmitting is safe; the detail names the machine and is not stable.

The retry-safe classes, as one check. crashed, unreachable and machine lost are the errors the platform owns rather than your payload, and all are safe to resubmit unconditionally — none leaves a partial side effect upstream:

RETRY_SAFE_PREFIXES = (
    "ollama crashed: ", "docling crashed: ",
    "ollama unreachable: ", "docling unreachable: ",
    "machine lost: ",
)
LEGACY_CRASH_SUBSTRINGS = (
    "model runner has unexpectedly stopped",
    "llama-server process has terminated",
)

def is_retryable(error: str) -> bool:
    return (error.startswith(RETRY_SAFE_PREFIXES)
            # Rows recorded before 2026-08-18 carry no crash prefix.
            or any(s in error for s in LEGACY_CRASH_SUBSTRINGS))

Everything else — your timeout_seconds, the deployment cap, an empty completion, a 4xx from either backend, and the two gateway-restart strings (Gateway restarts, above: retryable on ollama, back off first on docling, which is why they are not in the unconditional list) — needs a decision rather than a resubmit, because the remedies differ. A crash is also worth counting: resubmitting forever hides a box that has stopped being able to run the work at all, so cap the attempts and let the cap failing be the signal.

Pinning the retry contract

The block above is a copy. If we reword a prefix, nothing in your suite fails — your matcher simply stops recognising a class, genuine crashes reclassify as unrecognised, and they stop being retried. You would find out by noticing a block of infrastructure failures that should have been retries. A consumer raised exactly this on 2026-09-07.

GET /v1/error-contract closes it. It serves every class the gateway can record, derived from the same predicates that classify errors — so it cannot drift from the implementation — plus a version over the matching rules. Abridged to two of the fourteen classes; the version is the real one, covering all fourteen:

{
  "version": "b83f40ead6db0805",
  "classes": [
    {"id": "backend_crashed", "match": "prefix",
     "values": ["ollama crashed: ", "docling crashed: "], "retry": "safe"},
    {"id": "gateway_restart", "match": "exact",
     "values": ["gateway shutting down",
                "gateway restarted while job was running"],
     "retry": "conditional", "retry_when": {"backend": ["ollama"]}}
  ]
}

retry is one of four verdicts: safe (resubmit unconditionally), conditional (only when retry_when holds — read it), decide (a real answer about your job; retrying unchanged reproduces it), and never (deliberate; resubmitting fights whoever did it).

Assert the version in your own tests. Pin the string, compare it against a live fetch, and fail on mismatch — then a contract change surfaces as your build going red rather than as a bad week of retries. The version covers the matching rules only, so we can improve the prose without waking you. You need to reach the gateway to fetch it, so if you run through the tunnel this belongs wherever you already talk to us rather than in ordinary CI; it does not need to be fast or highly available.

Building your matcher from the response at runtime instead is supported and is strictly better — but pinning is what turns a silent reclassification into a failing test, so do that either way.

Apply this check at every stage that submits a job, not just the one where failures are common. A platform condition can land on any submission — a backend that is down affects whichever stage happens to need it, and that is rarely your busiest one. A pipeline whose retry covers, say, its labelling calls but not its document conversions will mark a transient platform condition as permanently failed the first time the other stage is the one that hits it, and the correct classification on our side never reaches the code that needed it. This is not hypothetical: a consumer lost eight filings to exactly that shape on 2026-08-18, and the errors they were handed were the retry-safe class throughout. If is_retryable lives in one place and every stage calls it, this cannot happen; if each stage matches its own strings, it will happen once per stage.

Free-tier budget cap. The free-tier fuse is enabled on the OCI deployment (since 2026-07-24; it remains opt-in and off by default in local/dev stacks). When the large has spent its monthly Always Free budget, a job needing it is not failed — it’s held in the non-terminal status waiting_for_budget, and dispatched automatically once the meter resets (UTC month rollover) or the operator raises the cap. Treat waiting_for_budget like queued: don’t error, don’t time out aggressively. If your workflow can’t wait for a possible month-boundary delay on budget-bound jobs, set a consumer-side deadline and DELETE /v1/jobs/{id} to cancel (cancellation works on held jobs just like queued ones). Jobs already running, or that don’t need the large, are unaffected.

timeout_seconds does not bound a budget hold. It is a dispatch bound: it starts counting when the job reaches the backend. A job with timeout_seconds: 60 can sit in waiting_for_budget for the rest of the month and only then spend its 60 seconds. The consumer-side deadline above is the lever for that wait, not this field.

Checking where the budget stands. If a job is sitting in waiting_for_budget, GET /queue’s lifecycle block tells you whether the wait is minutes or weeks:

• free_tier.binding_pct / free_tier.binding_metric — how much of the large’s monthly budget is spent, and which meter (OCPU-h or GB-h) is currently the constraint.
• free_tier.fuse_enabled / free_tier.stop_at_pct — whether the cap is active, and the percentage at which it trips (default 90).
• fuse.tripped / fuse.reason — whether it is holding jobs right now, and why.
• month_to_date.next_reset — the UTC timestamp at which the meter resets and held jobs drain.

If binding_pct is already above stop_at_pct, expect new large-backed jobs to park in waiting_for_budget until next_reset.

While status=queued, the response includes queue_position (1-indexed, global; 1 means “next to run,” counting every queued job ahead of yours regardless of backend). Once a job starts, the response includes started_at and queue_position is dropped. On completion, the verbatim backend response is returned under result.

timeout_seconds is echoed on every status, but only if you set it on submit — its absence means “no per-job bound”, not “the field is missing”. Don’t treat that as a bug.

While status=running, the response also includes a phase field describing what the gateway is currently doing with your job. The bundled dispatchers emit:

phase is a UX affordance — a consumer can render meaningfully different “loading model…” vs “generating…” states without guessing — but it’s not a control plane signal. Don’t write logic that depends on phase transitions firing in a specific order or at all (e.g. a very fast inference may transition straight to completed before the watcher even writes ollama_generating). Treat it as a hint, not a contract.

Correction (2026-09-06): ollama_generating reports residency, not activity. It is derived from polling ollama’s /api/ps, which answers “is this model in RAM”, and the OCI large is configured to keep the model resident indefinitely. So on a warm box the phase pins to ollama_generating from its first poll — about two seconds in — and stays there for the rest of the job whether or not a single token is ever produced. A job that is completely wedged reports exactly the same phase as one generating normally, for the whole hour.

The table above previously said “actively producing tokens”, and four failed jobs were diagnosed against that reading before it was caught. tokens_generated is the field that answers “is it actually producing” (below), and on a failed job output_tokens is (Reading a failure). Use phase to render a label; never to decide whether work is happening.

Watching a running job

For ollama generation jobs, a running response also reports how far the work has actually got:

This exists to answer one question that nothing else on this page can: is my job still working, or is it wedged? A generation that has been running 45 minutes is not by itself a problem — on a slow box it may be entirely normal — and before these fields the only way to tell the two apart was to go and look at the host. Now a rising tokens_generated says it is producing, and a last_token_at that has stopped moving says it is not — and either way you can act on it, because a running job can be cancelled (see Cancelling a running job).

tokens_per_second is the number to compare against a known-good rate for your deployment. It measures decode only: it starts at the first token, so it excludes the queue wait, the model load and the prefill that came before, all of which would otherwise drag it down for the first minute of every job and make a healthy job look sick. See Latency expectations for what is normal where.

How they behave, so you don’t read too much into them:

• These three appear only while status=running, and only once the first token has arrived. A job that is still loading the model, or queued, or has finished, or that sent tools (see below), has none of these fields. Their absence never means failure — check status and phase for that. A failed job reports the same work under different names, because the question there is a different one: see Reading a failure.
• tokens_per_second needs two tokens to have an interval to measure, so it lags tokens_generated by one report.
• Updated about once a second, not per token — every update is a write on the gateway’s job store. Polling faster than that returns the same numbers.
• tokens_generated is the gateway’s count, not ollama’s. It typically ends one below the eval_count on the finished result, because the terminating stop token arrives on a chunk that carries no text. Use it as a progress indicator; use the finished job’s eval_count when you need the authoritative number.
• This is a hint, like phase. Don’t build control flow on the exact values.

Nothing else changes. The gateway asks ollama for a token stream internally and reassembles it, but the API is still submit-and-poll, your result is still one complete body, and that body is identical to what a non-streamed call would have returned.

Tool calls

A payload carrying a non-empty tools array is dispatched non-streamed, because tool-call deltas are the one thing reassembly can get quietly wrong. Tool calling works normally and the response is passed through verbatim; the only difference is that those jobs report none of the progress fields above — which is what every job did before internal streaming existed.

Gateway restarts: queued jobs survive (state is persisted to SQLite), and so does a job running on a worker machine. Any other job that was actively running at restart time is marked failed, with one of two exact strings depending on how the gateway went down:

Both mean the same thing to you: the job did not finish, no result was stored, and the gateway is not going to produce one. They are two halves of one event, not two conditions.

Whether to resubmit depends on the backend, and the job body tells you which (backend):

• ollama — safe to resubmit, on the same reasoning as <backend> crashed:. Generation stopped when the process did, and inference mutates nothing upstream.
• docling — do not resubmit immediately. docling-serve exposes no cancel endpoint, so a conversion that was in flight is still running on the backend and will finish on its own. Resubmitting puts a second conversion on a machine that runs one job at a time. This is the same trap as a docling dispatch bound expiring; back off first.

Neither string carries a <backend> prefix, so they are matched whole rather than by prefix — and they are deliberately not in RETRY_SAFE_PREFIXES, because that list means “resubmit unconditionally” and the docling half of this is conditional.

Retention — a safety margin, not a home

Store every response on your side as soon as the job completes. The window exists so a collector that lags, crashes, or restarts mid-run can still pick up an answer it already paid for. It is margin for recovery, not storage.

A swept answer cannot be re-fetched, only re-run — the full inference again, on the single global slot, drawing the free-tier meter. See Whichever clock fires, don’t lose the job id under Timeouts for how often that gap actually opens.

The numbers. Completed and failed records are kept 72 hours after completed_at by default, then deleted by an hourly sweep. Both are env-tunable on the gateway (GATEWAY_JOB_TTL_SECONDS, default 259200; GATEWAY_CLEANUP_INTERVAL, default 3600).

Some callers have a longer window, granted per caller id:

GET /queue reports what is actually loaded. Trust it over this table:

"storage": {
  "db_bytes": 41582592,
  "wal_bytes": 4157440,
  "retention": {
    "default_ttl_seconds": 259200,
    "overrides_by_caller": { "autostatement": 2592000 },
    "sweep_interval_seconds": 3600
  }
}

A longer window buys a bigger margin; it does not make the gateway a database. It carries no durability promise — jobs.sqlite is one file on one host, no backup, no replica, and an operator rebuilding the small may start it empty — and it does not widen the webhook window (see Completion webhooks).

Queued, running and waiting_for_budget jobs are never swept at any age — status decides, not the clock. The window starts when a job finishes.

After sweep, GET /v1/jobs/{id} returns HTTP 404 with body {"error": "job not found"} — indistinguishable from an ID that never existed. There is no expired state and no tombstone, so only your own record of what you submitted can tell a lost result from a bad id.

The completed_last_24h / failed_last_24h counters on /queue are independent of retention — always the last 24 h.

Direct backend access is gone

Changed 2026-08-19, and this is a breaking change if you were doing it. ollama’s :11434 and docling’s :5001 are no longer published on the host by a local platform-services stack. On OCI they were never reachable (the large is on a private VCN). So on every deployment, anything of yours that reads those ports directly stops working.

Say it the sharp way, because the polite way misled a consumer: this is not “you can stop avoiding those ports in your devcontainer config.” It is “any code path that fetches from them now fails.” That is a quieter class of breakage than a port clash — a clash is loud and immediate, whereas a lookup that silently returns nothing can keep a pipeline running while it records worse data. One consumer read model digests from :11434 for provenance and recorded a batch of unpinned bare tags before noticing.

Why. A host port on a developer laptop is a namespace nobody controls: VS Code’s forwarding claims ports and walks upward when one is taken, Docker’s allocator can hold a stale reservation, and a neighbouring compose project can bind first. It cost three separate incidents, including a healthy docling that was unreachable for twenty minutes because its binding had silently vanished. Internal traffic now stays on the container network, where none of that can reach it.

What to use instead — all of it was already the supported route:

The gateway (11435, or whatever GATEWAY_HOST_PORT says) and the docs vhost (5006) are unchanged and still published. Nothing about submitting or polling jobs changed.

Completion webhooks (optional)

Instead of holding a polling loop open to notice a job finished, you can register a callback_url on submit. When the job reaches a terminal state — completed or failed — the gateway makes one outbound POST to that URL. This lets an event-driven orchestrator resume the next pipeline stage without a live poller: if your poller dies (laptop sleeps, container restarts), the callback still fires and your handler picks up where it left off.

The webhook notifies, it does not ship the result. GET /v1/jobs/{id} remains the authoritative source of the result, for as long as your retention window keeps it. The callback is a best-effort convenience — if delivery fails after retries, nothing is lost; you can always fall back to polling.

A longer retention window does not widen the callback window. A restart re-sends only unfinished deliveries from the last 72 h (GATEWAY_WEBHOOK_RESUME_WINDOW_SECONDS), however long your records are kept — so you are never pushed an event for a job you finished three weeks ago.

Per-caller setup (required)

Callbacks need per-caller configuration that the operator holds server-side, keyed by your caller id (Authentication):

• a shared secret used to sign deliveries (so you can authenticate they came from the gateway);
• a host allow-list — the set of hosts the gateway is willing to call back for you.

Until that’s registered, a submit carrying a callback_url is rejected with 400. The onboarding handshake is:

• You send the operator: the exact caller id you submit jobs with, and the exact callback host(s) you’ll use (e.g. hooks.your-app.example, or hooks.your-app.example:8443 for a non-default port).
• The operator sends you back: a secret string. Put it in an env var (e.g. GATEWAY_WEBHOOK_SECRET) — it never goes in the submit body or your repo; it’s only used to verify the signature on delivery.

A caller can also have a default callback URL configured, applied to every job that doesn’t set its own callback_url (a per-job callback_url always wins).

Your receiver must be reachable from the gateway, i.e. from the public internet. The gateway runs in OCI and dials out to your callback_url, and its SSRF guard refuses private/loopback/internal targets. So localhost and private addresses will not work as a callback, even for local development. For local dev, expose your receiver through a public tunnel (e.g. an ngrok-style HTTPS URL) and allow-list that host, or point the callback at a deployed receiver. This is independent of the inbound SSH tunnel you use to reach the gateway — that tunnel carries your requests in; the callback is a separate connection the gateway makes back out to you.

Payload

The body is a small, fixed envelope — enough to identify the job and fetch its result, never the result itself (docling markdown/tables can be large):

{
  "job_id": "0f5b…",
  "status": "completed",
  "backend": "docling",
  "tier": "batch",
  "caller_id": "autostatement",
  "submitted_at": "2026-07-16T09:25:01.124+00:00",
  "completed_at": "2026-07-16T09:31:44.881+00:00"
}

error is present (a string) only when status is failed. On receipt, GET /v1/jobs/{id} to fetch the actual result / full error.

Sent headers:

Verifying the signature

The signature authenticates the delivery and lets you reject replays. The scheme:

signing_input = X-Gateway-Timestamp + "." + <raw request body bytes>
signature     = HMAC_SHA256(shared_secret, signing_input)
header value  = "sha256=" + hex(signature)

Trust caveat. The signing key is selected by the job’s caller id. Under caller_auth: required only a holder of your token can submit as you. Under off the id is self-declared, so anyone able to submit jobs can make the gateway send your endpoint a validly-signed notification (Authentication). Either way, treat a verified signature as “came from the gateway,” not “was requested by me.”

Verify against the raw body bytes as received — do not re-serialize the parsed JSON, since any whitespace/key-order difference changes the hash. (The gateway signs a deterministic compact encoding with sorted keys; re-encoding it yourself is not guaranteed to reproduce it.)

import hashlib, hmac, time

SECRET = b"<your shared secret>"
MAX_SKEW_SECONDS = 300

def verify(request_body: bytes, headers) -> bool:
    ts = headers["X-Gateway-Timestamp"]
    if abs(time.time() - int(ts)) > MAX_SKEW_SECONDS:
        return False  # stale / replayed
    expected = "sha256=" + hmac.new(
        SECRET, ts.encode() + b"." + request_body, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, headers["X-Gateway-Signature"])

A minimal receiver

Putting it together — a complete endpoint that verifies, dedupes, responds fast, and fetches the result out of band. Shown in FastAPI; the shape is the same in any framework, but note the one framework- specific subtlety called out below.

import hashlib, hmac, json, os, time
from fastapi import FastAPI, Request, Response

SECRET = os.environ["GATEWAY_WEBHOOK_SECRET"].encode()  # from the operator
MAX_SKEW_SECONDS = 300
_seen: set[str] = set()          # replace with a durable store (DB/Redis) for real use

app = FastAPI()

@app.post("/gateway")
async def gateway_callback(request: Request):
    # (1) Read the RAW body for verification. Do NOT use the framework's
    #     parsed JSON here — re-serializing changes the bytes and breaks
    #     the HMAC. In FastAPI that means `await request.body()`, not a
    #     Pydantic model / `await request.json()`. (Flask: request.get_data();
    #     Express: express.raw() on this route, or the verify hook.)
    raw = await request.body()
    ts = request.headers.get("X-Gateway-Timestamp", "")
    sig = request.headers.get("X-Gateway-Signature", "")

    # (2) Reject stale/missing timestamps (replay), then verify signature.
    if not ts or abs(time.time() - int(ts)) > MAX_SKEW_SECONDS:
        return Response(status_code=401)
    expected = "sha256=" + hmac.new(SECRET, ts.encode() + b"." + raw,
                                    hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return Response(status_code=401)

    event = json.loads(raw)
    job_id = event["job_id"]

    # (3) Dedupe — delivery is at-least-once, a job may arrive more than once.
    if job_id in _seen:
        return Response(status_code=200)
    _seen.add(job_id)

    # (4) Respond 2xx FAST (the gateway uses a ~10s timeout). Do the real
    #     work — GET /v1/jobs/{id} for the result, then your next pipeline
    #     stage — asynchronously, not inline in this handler.
    enqueue_followup(job_id, event["status"])   # your own task queue / background job
    return Response(status_code=200)

The webhook body is notify-only, so step (4) still calls GET /v1/jobs/{id} to pull the actual result (or full error) — the callback tells you when, the GET tells you what.

Delivery semantics — at-least-once, dedupe on job_id

Delivery is at-least-once: the gateway retries with exponential backoff on transport errors and non-2xx responses, over a bounded number of attempts, then gives up and records the failure. There are no ordering guarantees, and a webhook may fire more than once for the same job (e.g. a retry after your handler already succeeded but its 2xx was lost). Your handler must be idempotent — dedupe on job_id. Respond 2xx promptly (the gateway uses a short outbound timeout, ~10 s); do the real work asynchronously on your side.

If delivery never succeeds within the retry window, fall back to polling — the job record is still there for your retention window.

Debugging — did my webhook fire?

GET /v1/jobs/{id} includes a callback sub-object once a callback is registered, so you can see delivery state without asking the operator to dig through logs:

{
  "status": "completed",
  "callback": {
    "url": "https://hooks.your-app.example/gateway",
    "status": "delivered",
    "attempts": 1,
    "last_status_code": 200,
    "last_error": null,
    "delivered_at": "2026-07-16T09:31:45.402+00:00"
  }
}

callback.status is one of:

Security rules you should know

The gateway treats the callback URL as untrusted and defends the outbound request:

• https only. http URLs are rejected at submit.
• Host must be allow-listed for your caller (above).
• Private / internal targets are refused. The host is resolved and the request is refused if it lands on a private, loopback, link-local, or otherwise non-public address (this includes the cloud metadata endpoint 169.254.169.254). The connection is pinned to the validated address, so a DNS change between submit and delivery can’t redirect it inward. Such a refusal shows up as callback.status = blocked and is not retried.

Point your callback_url at a public https endpoint you control.

Picking a model name

The 72B is retired (2026-10-15). qwen2.5:72b-instruct-q4_K_M is on no OCI machine any more; a job for it waits waiting_for_large. On OCI, target a model the worker machines serve (/queue → machines): qwen3.8:27b runs on the home machines and on the OCI large.

Who picks the model: you do, per request. The model field in your job payload names the tag and the gateway loads it on demand. There is no bring-up step to request and no operator hand-off. The operator only decides what is on disk — a local stack’s MODELS_PROFILE picks which tags get provisioned (small → the 8B, candidate → the evaluation set, the default), and a worker machine holds the models the hub lists for it. LLM_MODEL is a convention of your config, never a platform input: the gateway does not read it, your code reads it and puts the value in the payload.

Check what a given host holds before you target it — GET /diagnostics → available_models for the live host, or https://soggplatform.dedyn.io/models.json for the published list. A tag that isn’t listed still “works”: ollama pulls it on first use, which on a 17–40 GB tag looks like a hang for many minutes. Treat an absent tag as a question for the operator, not something to submit blind.

The 8B-class small profile provides one model:

• qwen3:8b-q4_K_M (stock registry tag) — control thinking per request with "think": false in the payload (forwarded verbatim), and the digest matches the public registry for provenance.

qwen3:8b-q4_K_M-nothink was removed on 2026-08-16. It never suppressed thinking — its template dropped the /no_think marker that does the work — and it is not established that it ever did. Measured on ollama 0.32.5: 7,636 characters of reasoning with no think field sent, zero with "think": false. Send "think": false against the stock tag instead.

It may still answer on a stack that already built it. Removal means the platform stops provisioning the tag, not that it vanishes from an ollama store that already has it — so a long-running local stack will keep serving it, badly, until someone runs ollama rm qwen3:8b-q4_K_M-nothink. Don’t read a successful call as evidence the tag is supported. Fresh stacks never get it.

Naming the old tag in your own historical records is fine — the platform doesn’t care, and it did not hold the removal open.

The profile exists for legacy compatibility and minimal-resource work — running ollama locally on a developer machine that can’t hold a 72B model, driving a regression suite where faster inference beats quality, or model-compare runs at batch priority. Do not depend on the small profile being available on the OCI large.

The candidate profile (qwen3.6:27b, gemma4:26b, qwen3.8:27b) is the default on a local stack. The evaluation that chose the 72B’s replacement is over: qwen3.8:27b is the extraction model (settled 2026-11-01), and the only one of the three on OCI. The profile is a moving target by design, so don’t pin a consumer to the other two.

qwen3.8:27b is available on the local stack from 2026-08-26. It is a qwen3-era reasoning model, so Moving from a qwen2.5-era model to a qwen3-era one below applies to it unchanged — send "think": false. Two things are specific to it:

• It runs on OCI’s worker machines, the large among them from 2026-10-15 (Worker machines), each on the same pinned weights. The tag requires ollama 0.32.12 or newer.
• 27b is the only size published. There is no smaller qwen3.8 to fall back to on a box that can’t hold it.

Adding it moved the local engine from 0.32.5 to 0.32.15, and two of those releases change behaviour you can observe whatever model you call. (The local engine moved again, to 0.34.2, on 2026-09-22.) /v1/chat/completions streaming now matches OpenAI’s wire format — role on the first chunk only, finish_reason on its own chunk, usage in a separate chunk under stream_options.include_usage. And a truncated OpenAI-compat response now reports finish_reason: "length" instead of "tool_calls", which finally agrees with what Bounding output length tells you to check for.

Both changes now apply on the OCI large as well, which they did not before 2026-09-03. If you were keying off finish_reason == "tool_calls" to detect truncation against the large, that stops matching — switch to "length". Still read ollama.version from GET /diagnostics rather than assuming an engine: the two deployments are pinned independently and are free to diverge again.

A fourth candidate, qwen3.6:35b, is listed in the profile script but deliberately not pulled: it doesn’t fit the evaluation machine’s RAM.

Correction to what this section said before. It claimed a model that doesn’t fit “pages rather than failing”. That is wrong and it was the reasoning behind an unsound RAM budget. A model that doesn’t fit fails: on the evaluation box the runner was OOM-killed 4-7 seconds after dispatch having generated zero tokens, which reaches you as ollama crashed: and not as a slow job. Paging is the gentler outcome and not the one to plan against.

Moving from a qwen2.5-era model to a qwen3-era one

Reasoning tokens count against num_predict, and they will dominate it. Carrying your qwen2.5 settings across unchanged is enough to break extraction, and it breaks it in the most expensive way available — a long, apparently healthy generation that ends in unparseable output.

Measured by a consumer on 2026-08-16, one real financial-extraction prompt against qwen3.6:27b with num_predict: 4096:

Three things follow. "think": false in the request body is the only control that works — it is forwarded verbatim to ollama. A -nothink in a tag name is not equivalent — the platform’s own -nothink build measured 7,636 characters of reasoning against zero with "think": false, and has since been removed.

Omitting the field is not the same as sending false. This is the one that catches people, because “we don’t set think” reads like a neutral default and isn’t. With no think key in the body, the decision falls to the model, and a reasoning model decides on. If your client only adds the key when some THINK-ish config value is non-null, the unset case is thinking on — and paired with a num_predict of a few thousand that is exactly the 55.3-minute invalid-JSON row above. Send the field explicitly, or assert that you did.

And a truncated reasoning model looks like a normal answer: you get done_reason: "length" with the budget spent, so treat that as a hard error and never parse the result (see Bounding output length).

Hardcoding a model name in your consumer ties it to one deployment. Parameterize via env var, same way as the gateway URL:

# OCI, direct or tunneled (default — what every consumer
# should target unless they have a specific reason not to)
LLM_GATEWAY_URL=http://llm-gateway:11435  # or 21435 if tunneled
LLM_MODEL=qwen3.8:27b   # served by the worker machines (/queue → machines)

# Local platform-services only (legacy / minimal-resource tests).
# Send "think": false in the request body to disable thinking mode.
LLM_GATEWAY_URL=http://llm-gateway:11435
LLM_MODEL=qwen3:8b-q4_K_M

# Local platform-services, model evaluation (candidate profile)
LLM_GATEWAY_URL=http://llm-gateway:11435
LLM_MODEL=qwen3.6:27b

The model set actually loaded on each deployment lives in models/profiles/ for a local stack (small.sh is legacy); a worker machine serves what /queue → machines lists for it. To discover what’s currently loaded on the gateway you’re pointing at, hit /diagnostics on the gateway — ollama.ps[].name lists what’s resident in RAM right now, ollama.tags lists everything on disk. The legacy https://soggplatform.dedyn.io/models.json proxy of /api/tags still works for public model discovery.

A model name does not fully identify what you’ll get. The deployments deliberately run different ollama versions, and the same tag can behave differently across them — badly enough that a model has decoded to gibberish on an older engine rather than failing outright. If you’re recording results you intend to compare or reproduce (benchmarks, evaluations, regression baselines), record what produced them alongside the output:

• ollama.version from /diagnostics — the engine
• ollama.tags.models[].digest — pins the exact weights behind a tag, which is why stock registry tags are preferred over locally built Modelfile variants: the digest matches the public registry
• host_id and deployment from /diagnostics — the machine, and the setup label that machine stamps on its own metrics rows

Read the digest from /diagnostics, not from ollama directly. This page used to describe the raw blocks as ops-debug-only two paragraphs below while recommending ollama.tags[].digest here — a contradiction that made it unclear whether a consumer could depend on it. Settling it in the direction the recommendation already pointed: ollama.tags on /diagnostics is a supported read for provenance and will not be removed without telling you. Its inner shape is ollama’s, not ours — we pass the response through — so read models[].digest defensively and treat an unexpected shape as “digest unavailable” rather than as an error.

Reaching ollama’s /api/tags directly is not a supported route and stopped working on local deployments on 2026-08-19 (see Direct backend access is gone below). A consumer doing that recorded unpinned bare tags for a batch and caught it only by luck.

Better: let the platform tell you what ran. Since 2026-08-20 every ollama job_metrics row records three model fields, and the distinction between them is the whole point:

Read them from GET /metrics/recent?job_id=<id>.

They catch two different failures. model_resolved differing from model means a different model was served — which your own pre-flight check cannot catch, because it runs before you submit and never against what ran. A changed model_digest under an unchanged tag means the weights moved underneath the tag.

This closes a race you cannot close yourself. A digest you read at submit time describes the model on disk then, and your job dispatches later — the queue wait on this deployment reaches ~2 hours. Registry tags are mutable and the platform re-pulls them on demand, so the weights behind a name can change while your job waits. The gateway’s reading is taken after your job finishes, from the model still resident, so it describes the run rather than a guess made before it.

Both are null on rows written before 2026-08-20, and on any job whose probe missed — null never means “unchanged”, it means “not recorded”. A query that cares must exclude nulls rather than assume. The gateway will return no digest rather than a guessed one: if two models are resident and neither matches the name you asked for, you get null, because a wrong digest records confidently and cannot be caught afterwards.

The engine version is recorded on the same row (engine_version) for the same reason.

Read the context window live — never copy it into your config. GET /diagnostics → context_window is two numbers, and you need both:

"context_window": {
  "tokens": 12288, "clip_to": 6146, "clip_to_basis": "measured",
  "clip_to_measured_at": "2026-09-22T21:38:11Z",
  "live": true, "observed_model": "qwen3.8:27b",
  "observed_at": "2026-09-22T23:27:52Z", "source": "ollama /api/ps"
}

tokens is the window. Size up to it. A prompt up to this many tokens is evaluated whole — the server honours the number it reports (jobs of 7,988–8,806 tokens evaluated whole on 12288, September 2026). Measured 2026-09-24 on the 27B: a ~12,200-token chat prompt was evaluated whole, a 12,456-token one was cut. So the edge is the window itself, give or take the ±0.5% of that estimate — keep ~100 tokens of margin, not more. Your num_predict plays no part: an 8,918-token prompt sent with num_predict: 4096 evaluated whole.

clip_to is what a longer prompt is cut down to. Not trimmed to fit: ollama keeps the first 4 tokens and then half of the rest, so a 12,289-token prompt on a 12288 window is evaluated as 6,146 — and the instruction at the top does not survive. clip_to_basis is measured once this gateway has seen a clip, estimated (4 + (tokens − 4) / 2) before then. Sizing under clip_to instead of tokens costs double the jobs for nothing; this page said to do exactly that for one day, 2026-09-22, and was wrong.

Both numbers survive a gateway restart, so an idle deployment still answers. live: false means nothing is resident and tokens is the last window this gateway saw, stamped with when — a dated reading, not a promise about now. null means it has never seen one.

On a gateway that hands jobs to worker machines the block covers them too: tokens is the smaller of this gateway’s own window and the one it holds every machine to, so a prompt sized under it is evaluated whole wherever it runs. source says which one it is; a machine’s clip_to is always estimated.

Two nearby fields are not the window either, and both are far larger. ollama.tags[].details.context_length, and /api/show’s <arch>.context_length, report the model’s trained maximum — 262144 for qwen3.8:27b against a real window of 12288. Either one will tell you a 200k-token prompt fits. They are raw upstream passthrough inside an ops block; context_window is the contract.

It is not a constant because the window is a memory setting as much as a capability one — the KV cache and compute buffers are sized from it — so it differs per deployment and gets retuned. The local stack also quantizes that cache (q8_0) where the OCI large does not, so inference is not bit-identical across the two; record deployment beside engine_version if you compare outputs.

How long a prompt that fits may take to start: first_token_bound_seconds, beside context_window on /diagnostics. Read it live the same way; never copy it into your config. seconds is how long the gateway waits for your first token before failing the job (Two errors that are not a clock running out), sized so any prompt up to tokens can finish prefilling. With basis: "derived" it comes from the slowest prefill among model’s recent and largest jobs on this deployment, plus a load allowance, and the inputs it used are listed with it. With basis: "flat" it is 1800 s, because there is no rate yet. It is never below 1800 s, and it is already capped at GATEWAY_OLLAMA_TIMEOUT, the cap on the whole call; capped_by_deployment_timeout: true says the cap set it. The 300 s bound between tokens is unchanged.

Don’t pin num_ctx. The gateway forwards your options verbatim with no cap of its own (the only gateway-side limit is the 64 MB body, GATEWAY_MAX_BODY_BYTES), so num_ctx overrides the deployment’s default entirely — including the tuning that default exists for. Omit it and inherit.

A clipped prompt is flagged, or refused if you ask. By default ollama cuts an over-long prompt rather than refusing it: no error, and an answer that reads as normal. Send "truncate": false at the top level of the payload and it refuses instead: the job fails with prompt exceeds context: and the engine’s own words, which currently include your prompt’s full token count. Verdict decide: the same prompt is refused again. Leave shift out; it is applied when the model loads, not per request. Completed ollama jobs carry:

/api/generate gets the exact basis: its response carries the full tokenization, including the part that was discarded. /api/chat carries no equivalent, so without truncate: false its basis is window. A cut lands on the clip size (clip_to), never below half the window, so a count from half the window to 8 above clip_to is reported truncated, which also flags a prompt that merely ends in that range. Biased that way deliberately: a false “check this” costs you less than a missed clip.

All five fields are absent when the gateway could not tell — a docling job, a failed job, a body with no token counts, or any job finished before 2026-09-22. Absent never means “not truncated”; read it as unknown. For sizing prompts against reality rather than a limit, input_tokens on GET /metrics/recent is your own prompt-token distribution per job, and it outlives the job TTL.

What happens when the model isn’t loaded in RAM

Submitting a job whose model name is in available_models but not in loaded_models is legal and pays a one-time disk→RAM load cost — the gateway forwards the request to ollama, ollama loads the model from local disk (the bind-mounted weights volume), and then runs inference. The HTTP request just blocks while loading. From the consumer’s perspective the call takes longer than usual, and the phase field on /v1/jobs/{id} reports ollama_loading_model during the load window so you can render a meaningful UX state instead of “still loading…”. Once loaded the model stays resident and subsequent calls don’t pay the cost — indefinitely on the OCI large (OLLAMA_KEEP_ALIVE=-1), for 60 minutes of idle on the local stack (3600, finite because a dev box cycles through models that don’t co-fit). So a local consumer can pay the load again after a long pause; the large’s resident model only falls out when the instance stops.

Submitting a job whose model name is not in available_models at all is an operator-side configuration gap — the host this gateway points at hasn’t pulled the model. Ollama’s behavior in that case varies by version (recent releases auto-pull, older releases 404). Either way, don’t rely on auto-pull for a 40 GB model — a silent 10–20 minute background download is the wrong default for an inference path. Pre-flight check by reading /diagnostics.available_models before submitting; if the model you need isn’t there, that’s an operator ask, not a consumer retry.

managed_models tells you which kind of gap it is. It lists every tag the platform’s own model profiles pull, union across all of them. It is a local read rather than a backend probe, so it stays correct exactly when you need it most — while ollama is unreachable or mid-pull. Read it against available_models:

The middle row can’t tell you which of its two cases you’re in, and that’s deliberate. managed_models is the union across every profile, and the gateway does not know which profile its own ollama host was built with. So a tag on its way to this host and a tag that lives on a different host entirely look the same — on a local stack, the OCI large’s qwen2.5:72b-instruct-q4_K_M sits in this row permanently. Only the operator can say which, and that’s why the row says ask.

The row is new as of 2026-09-10. Before it, a consumer that found its tag absent had nothing to read that said whether the platform had the model at all: on 2026-09-09 one checked during a pull, found its tag missing, and had to ask the operator whether it could be pulled on that machine, while the tag was already on its way. With the tag visible in managed_models, the question changes from “does this platform have it?” to “is it coming to this host?”.

An empty managed_models means “not knowable from here” — the gateway could not read its profiles mount — never “nothing is managed”.

Which machine answered

deployment (local, oci-large) names the shape of a setup. It does not name a machine, and two dev boxes both report local — so a .env copied from one to the other labels the second one’s runs as the first’s, with nothing in any response to contradict it. You would find out from a query weeks later, if at all.

host_id is the field that separates them. It is stable across restarts and rebuilds, and distinct per machine; it is not secret and not authenticated, so treat it as a label, not a credential. Assert it the way you assert /v1/error-contract’s version hash — pin the value you expect for the gateway you think you’re talking to, and fail the run when it changes:

d = requests.get(f"{GATEWAY}/diagnostics", timeout=5).json()
if d.get("host_id") != EXPECTED_HOST_ID:
    raise RuntimeError(
        f"gateway is {d.get('host_id')!r}, expected {EXPECTED_HOST_ID!r}")

Two values you may see:

• A name — laptop-lenovo, desktop, oci-small. The operator set GATEWAY_HOST_ID. This is the normal case and the one to expect.
• host- followed by hex — nobody set the variable, so the gateway minted an id into its own data volume on first boot and has reused it ever since. Still stable, still distinct per machine, just not readable. Fine to pin; worth asking the operator to name the box.

null means the gateway could not record one (a read-only data volume). Treat it as unknown provenance rather than as a match — the alternative we rejected, an id regenerated each boot, would have failed your assertion as though the host had changed and sent you looking at the wrong thing.

Every job_metrics row written since 2026-10-04 carries the same host_id (see Checking recent throughput); older rows read null.

Latency expectations

These are calibration numbers for sizing consumer-side timeouts, not SLOs. They depend on the machine’s shape (the OCI large: 5 OCPU / 30 GB since 2026-10-15, CPU inference — no GPU) and the specific model loaded. Treat them as ballpark; measure against your own workload once you have a real batch running.

They describe the OCI deployment only, and there is no equivalent table for the local one — deliberately. The local backends run on a developer machine that its operator also uses for everything else, so throughput there is a function of what else is running, not a property of the platform. Measured on one such box on 2026-08-09, across a single afternoon and with nothing misconfigured: decode throughput moved 8–15×, prefill 1.7–2.6×, model load 3–15×, and docling conversion ~1.9×, purely from concurrent use of the laptop. Even in the quiet band, repeating a byte-identical job gave ±39% run-to-run variance.

Two consequences for consumers targeting local, and where to look instead:

• Don’t build user-facing timing promises on it, and don’t treat any observed run as a floor.
• Don’t build a degradation detector or a retry-on-slow against these numbers. At ±39% noise you will generate false positives, and retries into a single-slot queue turn a delay into an outage.
• Do read the numbers off the box rather than off this page. GET /metrics/recent?deployment=local&model=<tag> returns prefill_tok_s, decode_tok_s and load_ms for jobs that actually ran there — see Checking recent throughput. That is the intended substitute for the table this section declines to print, and it is the better artefact: a measurement from your box beats an estimate from someone else’s, and it stays true when the model or the machine changes. A newly provisioned model has no rows until a job runs against it.

The 72B is retired (2026-10-15), and with it the warm-up below: what follows is its history. For the 72B model (qwen2.5:72b-instruct-q4_K_M) — numbers below are measured, not estimated, from the 2026-05-23 autostatement verify run on the OCI large host (then 20 OCPU / 140 GB, CPU inference). RAM size doesn’t affect inference latency, but the 2026-07-23 trim to 18 OCPU may add roughly ~10% to generation time if inference is compute-bound here — treat these as a mild under-estimate until re-measured at the new shape (the job_metrics table will show it):

Sizing a per-job timeout_seconds from this table: use the last row, not the one above it. The cold-load sits inside the dispatch bound, so a number chosen from “warm typical extraction” (~4 min) kills every first call after a cold start. Whether a given call is cold is not knowable at submit time — check /diagnostics.warm first if you want to bound warm and cold calls differently.

The cold-load cost is large enough that the gateway pre-warms that model in the background after boot. When the operator sets LIFECYCLE_WARM_MODEL=qwen2.5:72b-instruct-q4_K_M on the gateway, the lifecycle controller kicks a warm probe (a single-token inference against that model) as a fire-and-forget background task immediately after backend health passes. From the consumer side this means:

• state=starting covers only OCI boot + the backend health probe (~70 s on the current OCI large), not the 20-min cold-load.
• After state flips to ready, the typical post-boot state is ready_cold for a few minutes (model not yet resident) and then ready_warm once the background probe completes the load.
• A consumer call that arrives during the warm window doesn’t wait for the probe to finish — it dispatches immediately, and either
  1. ollama already has the model resident and runs at warm speed, or (b) it races the probe and both wait on the same per-model load inside ollama. Either way, the cold-load cost is paid once per cold start and not duplicated.

If LIFECYCLE_WARM_MODEL is not set, no background probe is spawned and the first consumer job after each cold start triggers the load inside its own HTTP budget. Same wall-time, just different attribution.

GET /diagnostics surfaces the probe outcome under lifecycle.warm:

"lifecycle": {
  ...
  "warm": {
    "model": "qwen2.5:72b-instruct-q4_K_M",
    "last_attempt_at": "2026-05-25T10:14:23.117+00:00",
    "last_outcome": "success",
    "last_duration_seconds": 1187.4,
    "last_error": null,
    "model_loaded_at": "2026-05-25T10:14:23.117+00:00"
  }
}

Fields:

• last_outcome is one of success, timeout, transient_error, fatal_error, cancelled, error, or null (no probe run yet on the current boot). For consumer routing, treat anything other than success as “first call may pay cold-load” — the specific non-success value is operator-facing detail. Existing consumers that gate on last_outcome == "success" continue to work; the cancelled and error values are additive, not breaking.
• last_outcome=success plus model_loaded_at set is the strongest signal that the next call is warm. loaded_models on the same /diagnostics response is the live ground truth; model_loaded_at is a “when did we last positively confirm it” timestamp useful for predicting freshness over longer windows.
• A non-success outcome (probe timed out, hit a transient error, was cancelled by an unrelated lifecycle event, or raised an unexpected exception) doesn’t block READY — the controller stays ready and the first consumer call paying cold-load is the recovery path. Operator signal only.
• fatal_error indicates a misconfigured LIFECYCLE_WARM_MODEL (HTTP 4xx — most commonly 404 model not found). Consumer jobs using a different model name continue to work; consumer jobs using the same model name will see the same error. State stays READY in either case.
• cancelled means the probe was cancelled mid-flight by a reconcile-driven state transition (OCI reported the large stopped while the probe was still running). error means an unexpected exception escaped the probe — treat as a bug signal worth surfacing to the platform team.

Polling patience on cold starts

The gateway holds the ollama HTTP request server-side until ollama actually returns, up to min(timeout_seconds, GATEWAY_OLLAMA_TIMEOUT) — the deployment cap (default 60 min) unless you set a shorter per-job bound. A per-job bound has to cover load and generate. Cold-load is inside it: a 10-minute bound sized against “generation takes ~4 min” kills every cold first call, and on the local deployment there is no latency envelope to size against at all (below). Your consumer is polling /v1/jobs/{id}, not waiting on that HTTP call — each poll returns in milliseconds with status=running and the appropriate phase. phase=ollama_loading_model is now the normal signal that you’re paying cold-load (rather than the rare fallback it was when warm-on-boot blocked READY); phase=ollama_generating flips when ollama starts producing tokens. Your HTTP client’s per-request timeout only needs to cover one poll, not the whole job. The knob that matters for cold starts is how long your polling loop is willing to wait overall — for the 72B on the current OCI large, budget up to ~25 min for a cold first call (load + generate) and use the phase field to render meaningful state in the meantime.

For pre-flight smoke tests: a 5-token output against a trivial prompt ("Say pong.") completes in well under 30 seconds when state=ready_warm AND the requested model is in loaded_models.

If your verify runs produce additional measurements (especially on different output sizes or with different prompts), contribute them back — the table above is anchored on one filing’s worth of data plus the cold-load probe.

Timeouts: which clock measures what

The platform has three server-side clocks — two deployment caps and one you can set per job — plus at least two of your own. They measure different intervals, and sizing one from the other is the most expensive mistake on this page — it has produced both a consumer that abandoned jobs it should have waited for and a consumer that waited on jobs it should have abandoned.

Two more bounds apply to streamed ollama generation, and they measure progress rather than elapsed time — a silence, not a duration. They sit inside the caps above and are covered under Two errors that are not a clock running out: GATEWAY_OLLAMA_FIRST_TOKEN_TIMEOUT (1800 s, nothing arrives at all) and GATEWAY_OLLAMA_STALL_TIMEOUT (300 s, tokens stop arriving). Neither can be set per job and neither affects a job that keeps producing.

For docling, a clock firing ends the conversion as well as the wait: the job fails with nothing in it, never a document cut short. See Job size caps under Submitting a docling job.

The third one is yours to set and the platform’s to enforce. It undercuts the cap for a single job — same clock, smaller number. See Bounding the job yourself below, but read the rest of this section first: the interval those caps measure is narrower than most people’s first guess, and timeout_seconds inherits it exactly.

None of the three includes queue wait. Worst case from your submit is queue_wait + min(timeout_seconds, cap), and queue wait is unbounded: there is one global in-flight slot, and a slow job legitimately holds it for the full cap. Two slow jobs ahead of you is two full caps of waiting before your own clock starts. Setting timeout_seconds does not bound that wait — it is the one sentence on this page most often misread as “my job dies N seconds after I submit it.”

The dispatch clock does not start where status=running appears. Those are two different moments:

• status=running and started_at are stamped when the worker claims your job off the queue — the first thing that happens to it.
• The dispatch cap starts later, at the backend HTTP call, after the gateway has parsed your payload, written the phase, and — on OCI — brought the on-demand large up.

That last step is what makes the gap deployment-dependent:

While that boot runs, your job reads status=running with phase=waiting_for_backend. That phase is the flag. A client clock anchored on the first running observation is aligned with the platform’s on local, and up to 30 minutes early on OCI — it can abandon a job the gateway was still perfectly willing to run.

Sizing your own clocks. You need two, because the platform splits the wait in two:

1. A run bound. Anchor it on the started_at field in the job response, not on your own first-observed-running timestamp. started_at is authoritative, immune to poll-interval skew, survives your poller restarting, and is reset to null if the job is ever demoted back to waiting_for_budget — so it self-corrects where a local variable would keep counting through a hold. Size it against the dispatch cap for your backend.
2. A queue bound, from submit. Size it against how long the work stays worth doing — not in multiples of the dispatch cap. The two are unrelated quantities, and “N in-flight timeouts, so something must be wrong” is invalid reasoning: N slow jobs ahead of you is normal, not a fault.

Bounding the job yourself: timeout_seconds

Your own run bound tells you when to stop waiting. It does not tell the platform anything, and the job keeps running — holding the single global in-flight slot, with everything behind it still queued. timeout_seconds is the lever that actually ends the work:

{
  "endpoint": "/api/chat",
  "payload": { "model": "qwen3:8b-q4_K_M", "messages": [...] },
  "timeout_seconds": 900
}

It is the same clock as the deployment cap, just a smaller number. Everything above about what that cap measures applies unchanged:

• It does not include queue wait. A job with timeout_seconds: 60 can sit queued for an hour first.
• It does not include the OCI instance boot. On OCI a bounded job can read status=running, phase=waiting_for_backend for up to 1800 s before its 60 s has even started. That is deliberate: timeout_seconds bounds the work, not the wait. You still need your own from-submit clock for the wait.
• For docling it spans the whole dispatch — submit, polling, and result fetch — matching GATEWAY_DOCLING_TIMEOUT.

Don’t invent a number — declare the one you already enforce. If your client gives up on a job after N seconds, send N. Picking a different value creates a third clock to reason about; sending the same one means both sides stop caring at the same moment.

Then check every leg. Wherever your client bound is lower than the backend’s cap, the difference is time the gateway holds the global slot for a job nobody is waiting for — and everything queued behind it waits out that gap too. Per backend:

The “lower” row is the 2026-08-09 incident’s shape, and it hides per-leg: a consumer whose ollama bound matched the 3600 s cap exactly still had a 300 s docling bound against the 1200 s docling cap — 900 s of orphaned slot on every slow conversion, on a leg nobody had thought to check. Run the comparison for each endpoint you call, not for your app as a whole.

If you reclaim, that same gap is where slow answers survive — so tightening has a cost, not only a benefit. The rule above assumes walking away is permanent. It isn’t, if you persisted the job id: an abandoned job keeps running, finishes, and its result waits in the store for your retention window, collectable at zero further inference cost. A timeout_seconds expiry is the opposite — the job is failed and there is no result to collect, ever. What you do get is partial_output: the text the gateway had assembled when the bound fired. That is salvage material, not an answer (see Cancelling a running job), so plan on an expiry costing you the job’s work.

Worked example, from the incident: a 2704 s job produced complete, correct output. Its consumer had stopped polling at 1800 s but nothing killed it, so they collected the full result 45 minutes later for free. Under a 900 s timeout_seconds that job dies with no usable answer and the work is paid for twice.

So the two rules cut different ways, and which one governs depends on your own recovery behaviour:

• You don’t persist job ids. Abandonment really is permanent. Declare timeout_seconds; the gap buys nobody anything.
• You do reclaim. Set the bound where re-running is genuinely cheaper than waiting — which on a slow backend is often not below the cap.

Cancelling is not the cheaper escape hatch. It is the same trade-off as a bound firing, with a human choosing the moment: DELETE on a running job destroys it and you get no result. It looks free only on a queued job, which has produced nothing yet. If you are weighing “let the user stop waiting” against “don’t burn the work”, the only option that preserves both is the one you already have: stop polling, keep the job id, collect the result later. Full detail in Cancelling a running job.

Validation is loud. 1 <= timeout_seconds <= your backend’s cap (3600 s ollama, 3900 s docling; live on /diagnostics → timeout_seconds). Above the cap the submit is rejected with 400 naming the cap and its env var:

{"error": "'timeout_seconds'=5000 exceeds the ollama deployment cap GATEWAY_OLLAMA_TIMEOUT=3600s. Submit a value <= 3600, or ask the operator to raise the cap — the gateway will not silently clamp it."}

It is never silently clamped. A bound that quietly became a different number than the one you asked for would defeat the point of asking. Non-integers (including 900.0), zero, and negatives are rejected the same way. The value you set is echoed back on the accept response and on GET /v1/jobs/{id}, so you can confirm what the gateway recorded.

When it fires the job goes failed with an error naming your clock:

ollama dispatch exceeded the job's timeout_seconds=900

and when the platform’s own cap fires instead, naming its clock:

ollama dispatch exceeded the deployment cap GATEWAY_OLLAMA_TIMEOUT=3600s

The backend’s name leads the string and its own cap is named, so a docling job that outruns the deployment cap reads:

docling dispatch exceeded the deployment cap GATEWAY_DOCLING_TIMEOUT=3900s

Match on the substring timeout_seconds= to tell the two apart — they need different responses from you. Your bound firing means “this took longer than it was worth to me”, and raising your number is an option. The deployment cap firing means the work does not fit this deployment at all, and no number you can submit will change that — the lever is the operator’s, or the work has to get smaller.

At the boundary the two collapse into one. If you set timeout_seconds to exactly the cap, expiry is reported as the deployment cap, because that is the answer to the question the error exists to settle: there is no larger value you could have sent. To confirm what the gateway actually recorded for your job, read the echoed timeout_seconds field, not the error string.

The slot frees as soon as the bound fires, so the next queued job starts within seconds rather than waiting out the full deployment cap. That is the whole reason the field exists.

Two errors that are not a clock running out

New 2026-09-06. Both clocks above measure elapsed time and nothing else, so until now a job that streamed steadily for 59 minutes and one that never emitted a single byte failed with a byte-identical error at the same second. Four ollama jobs took the second path on 2026-09-01/02, burned four hours of backend time and produced nothing, and nothing in the record said which had happened.

Streamed ollama generation now also has two bounds on progress, inside the caps rather than replacing them, and each has its own error:

ollama produced no first token within GATEWAY_OLLAMA_FIRST_TOKEN_TIMEOUT=1800s — the backend accepted the request and then sent nothing at all

ollama stopped producing tokens for GATEWAY_OLLAMA_STALL_TIMEOUT=300s after 143 tokens — the stream stalled mid-generation

Match on the env var name — that is the stable part, the same convention the two clocks above use. Neither string contains dispatch exceeded, deployment cap or timeout_seconds=, so existing matching for the caps will not catch these and does not need changing.

They are deliberately separate errors because the remedies are opposite. A stall reported as a cap expiry would send you off raising timeout_seconds and asking the operator to raise GATEWAY_OLLAMA_TIMEOUT — when the job did not run out of time, it stopped. No amount of extra budget fixes a backend that produced nothing; it only buys a longer wait before you find out.

Two notes on scope. A payload carrying a non-empty tools array is dispatched non-streamed (Tool calls), and neither bound applies to it — there is no token stream to watch, so those jobs stay on the caps alone. And a slow generation is not a stall: the second bound measures the gap between tokens, so a job producing steadily at any rate runs to the cap as it always did.

Reading a failure

New 2026-09-06. A failed job now reports what it had actually got done, which was recorded all along and previously not returned:

output_tokens is the field to look at first, because it separates two failures that otherwise look identical:

• output_tokens: 0 — the job produced nothing. If the error is a cap expiry, the work never started rather than overrunning; a no-first-token error says the same thing explicitly.
• output_tokens in the thousands — the job was generating and ran out of time. Check it against your own num_predict before concluding the platform is at fault: at this deployment’s median throughput, num_predict: 4096 needs about 3,591 s of the 3,600 s cap, so a generation allowed to run to that length will hit the cap almost exactly, every time, with nothing wrong anywhere. Real extraction outputs on this platform run 182 tokens median.

output_tokens is the gateway’s own count, so it is typically one below the eval_count a completed job reports (the terminating stop token carries no text). It is absent — rather than zero — on failed tools jobs and on docling, neither of which feeds a token stream; a zero there would be an artefact of the transport rather than a measurement of the work.

Whichever clock fires, don’t lose the job id. A client-side give-up does not stop the job — it keeps running, finishes, and stores its result. Re-poll the persisted id before you declare the work failed; the record is retrievable for your full retention window. Every abandoned job we have seen in practice had a correct result sitting in the store that nothing on the consumer side ever collected.

A server-side expiry — your timeout_seconds or the deployment cap — is different, and differs again by backend:

• ollama: the gateway closes the connection, which aborts generation. The job is genuinely over and no result is stored. Resubmitting is the only way to get one.
• docling: the gateway stops waiting, but the conversion keeps running on the backend. docling-serve v1.34.0 exposes no cancel endpoint of any kind, so there is nothing for the gateway to call. Your job reads failed and the gateway holds no result — but the document is still converting, still occupying the backend, and will keep doing so until it finishes on its own. Resubmitting immediately therefore puts two conversions on a machine the platform runs one job at a time on. If a docling bound fires, give the box room before you retry.

Cancelling a running job

DELETE /v1/jobs/{id} works on a running job, not only a queued one. The dispatch is ended, the job goes failed with error: "cancelled", and the single global slot frees within seconds so the next queued job starts.

DELETE /v1/jobs/0f5b…
→ 200 {"id": "0f5b…", "status": "failed", "error": "cancelled"}

The response reports the job’s state once the cancel has settled, so in the ordinary case it already says failed. Three cases are worth knowing:

Cancelling a running job destroys it. This is the same trade-off as timeout_seconds expiring, with you choosing the moment rather than a clock. There is no result afterwards and resubmitting pays for the work again. Three distinct actions, only two of them cheap:

• Stop waiting — client-side, no API call. The job keeps running, finishes, and its result waits out the retention window. Costs nothing, frees nothing.
• Cancel — frees the slot immediately, destroys the work.
• Bound expiry — identical to cancel, automatic.

partial_output — salvage, not an answer

A job that ends early carries what the gateway had assembled at that moment, in its own field:

{
  "id": "0f5b…",
  "status": "failed",
  "error": "cancelled",
  "partial_output": "The three primary colours are red, blue"
}

It is never result, and that is deliberate. A truncated generation is not a shorter answer — a format-schema-constrained extraction cut mid-object is unparseable JSON (see Bounding output length), and prose cut mid-sentence can invert its own meaning. Treat it as evidence: what the model was producing, how far it got, whether it was on track. Anything that parses it as an answer is misusing it, which is exactly why it does not share a name with one.

Details worth knowing:

• Present only on failed ollama jobs that produced at least one token, for any early exit — cancel, your bound, or the deployment cap. Absent otherwise; its absence is not an error.
• Content only. Thinking output is not included.
• docling jobs never have it. There is nothing to salvage: the conversion is either complete or it has not returned anything.
• In the GET /v1/jobs listing it appears as partial_output_bytes (size only), the same treatment result gets.
• It rides the same retention as the rest of the job record.

What cancel does to the backend

• ollama: closing the connection aborts generation, so the work genuinely stops and the CPU frees.
• docling: docling-serve v1.34.0 has no cancel endpoint, so on the worker machines the gateway stops docling, which ends the conversion; the next one starts it again. On the OCI large it cannot: the conversion runs on up to 300 s past the docling cap, so don’t resubmit there at once or two conversions share the box.

A rule worth stealing: only DELETE a queued job

One consumer’s adopted rule, and the cleanest answer we have seen to the three-actions problem above:

DELETE is issued only while the job is queued. A running job is never cancelled — it finishes, and the result is reclaimed later.

A queued job has computed nothing, so dropping it destroys nothing while still freeing your place in the queue. Their user-facing button is labelled “Stop waiting”, because that is what it does.

Note what did not fix this for them. They were advised — twice — to drive the button’s message off the response status rather than hardcoding it, and did. It was not enough: the semantics were what changed. The same button making the same call silently converted from free to destructive the day running-cancel deployed, with status-driven copy faithfully describing the new destructive behaviour in fluent prose. What fixed it was a rule about which jobs they are willing to DELETE at all. If your cancel path predates this feature, check that rule and not only the wording.

Bounding output length

Two ways to stop a generation running away, and they fail very differently.

num_predict truncates. Hitting it returns done_reason: "length" and whatever had been produced — which for a format-schema-constrained extraction is an unparseable object. It is only safe if you treat done_reason == "length" as a hard error and never parse that result.

On a reasoning model the budget is also what the thinking is drawn from, so a cap that was generous for a qwen2.5-era model can be consumed almost entirely before the answer starts — see Moving from a qwen2.5-era model to a qwen3-era one. Send "think": false rather than raising the cap to cover reasoning you don’t want.

A maxItems in your format schema terminates cleanly, and is strictly better where it applies. Verified against ollama 0.32.5 on 2026-08-11, independently by a consumer and by the platform: a schema capping an array at 2 items, given a prompt containing 5 obvious rows, returned exactly 2 items with done_reason: "stop" and valid, parseable JSON. The grammar closes the array at the cap rather than the decoder being cut off mid-object.

But a clean cap is silent data loss, which is worse for some workloads than a loud truncation. A document with more rows than the cap yields well-formed JSON, done_reason: "stop", and no error anywhere — quietly missing entries. So maxItems is a runaway fuse, not a size limit, and it is only safe paired with treating len(items) == maxItems as a failure worth investigating. A consumer extracting ledger rows declined to ship it as a default for exactly this reason.

Submitting an ollama job

import os, time, requests

GATEWAY = os.environ["LLM_GATEWAY_URL"]  # e.g. http://llm-gateway:11435
MODEL = os.environ["LLM_MODEL"]          # e.g. qwen3.8:27b
HEADERS = {"X-Caller-Id": "my-editor-assistant"}

submit = requests.post(
    f"{GATEWAY}/v1/jobs",
    headers=HEADERS,
    json={
        "endpoint": "/api/chat",
        "payload": {
            "model": MODEL,
            "messages": [{"role": "user", "content": "Summarize ..."}],
        },
        "priority": "interactive",
    },
)
submit.raise_for_status()
job_id = submit.json()["id"]

while True:
    r = requests.get(f"{GATEWAY}/v1/jobs/{job_id}", headers=HEADERS)
    r.raise_for_status()
    job = r.json()
    if job["status"] == "completed":
        print(job["result"])
        break
    if job["status"] == "failed":
        raise RuntimeError(job["error"])
    time.sleep(5)

A 5-second poll cadence is fine for 5–15 minute jobs, and it costs the same whatever you submitted: since 2026-09-06 a status read no longer touches your payload, so polling a 64 MB docling submit is as cheap as polling a one-line prompt. Earlier advice to slow your polling on large submissions is retired. You receive the full response in result, never tokens-in-flight — the stream field in your payload is set by the gateway and whatever you put there is ignored, in either direction. It streams from ollama internally on generation endpoints and reassembles the result; that is invisible to you except for the progress fields described in Watching a running job above.

Submitting a docling job

Docling input goes inline in the payload as base64. The gateway does not accept multipart uploads — that simplifies the gateway and keeps the submission shape uniform between backends. For a 10 MB PDF this means ~13 MB of base64 text in the request body, well inside the gateway’s 64 MB body cap.

The gateway forwards your payload to docling’s /v1/convert/source/async, polls until docling reports the task terminal, and returns the docling result JSON verbatim in result.

Record which docling converted each document. Verbatim means a docling bump can change your inputs even when the API does not, and the deployments are bumped separately, without notice. Each job’s row on GET /metrics/recent?job_id=<id> carries what ran it:

• engine_version and engine_packages: the docling-serve release, and the docling packages inside it (docling, docling-core, …). A new value is the converter changing.
• docling_options: the options docling was sent — yours, plus the document_timeout the gateway adds and, unless include_images, its image_export_mode. An option you leave out takes docling’s default.
• docling_manifest: what docling’s container is — its libraries (packages, and the torch and onnxruntime builds), threads and device (accelerator, environment), server_settings, request_defaults (those defaults), its model files (models: each one’s SHA-256, and the Hugging Face commit where one was recorded) and its cpu. Only on a read by job_id; every docling row, listings included, carries the manifest’s SHA-256 as docling_manifest_sha256. One hash means one setup.

Stamp every conversion with these. The docling_* fields are null on rows written before they were recorded, and the manifest ones from a docling not yet redeployed with it. When the same bytes convert differently, compare engine_packages, then docling_options, then the two manifests, models and packages before cpu. Manifests that differ only in cpu (and torch.cpu_capability, which follows it) leave the CPU and how docling batched the pages, which follows the machine’s load and is recorded nowhere; whether either changes output is untested. /diagnostics → docling.version is the release answering right now.

Important — conversion options must be nested under options. docling-serve accepts conversion knobs (do_ocr, to_formats, do_table_structure, md_page_break_placeholder, etc.) under an options sub-object, not as top-level siblings of sources. The gateway is a verbatim passthrough; if you put options at the top level they reach docling but get silently ignored, and you’ll see defaults instead — most visibly: no \f page-break markers in markdown. Confirmed on the OCI large by an autostatement regression on 2026-05-23.

Images: default is dropped. The gateway interprets a top-level include_images: bool field on the docling payload (default false). When false the gateway both (a) sets docling’s image_export_mode=placeholder to suppress server-side rendering of images into the result, and (b) strips any image data that does land in the result before storing it. The strip nulls result.document.json_content.pages[<n>].image (the rendered page bitmaps — the main bloat source on real PDFs) and pictures[].image (semantic-object detections), and replaces inline base64 data URIs in md_content / html_content with placeholders. Set include_images: true if you actually need the image bytes (e.g. an upcoming vision-model interpreter); be aware that a single PDF page can produce 1–10 MB of base64 image data at the default 144 dpi and the gateway stores the full result for the job-TTL window. Background: plans/docling-image-handling.md.

import base64, os, time, requests

GATEWAY = os.environ["LLM_GATEWAY_URL"]
HEADERS = {"X-Caller-Id": "my-doc-ingest"}

with open("annual-report.pdf", "rb") as f:
    pdf_b64 = base64.b64encode(f.read()).decode("ascii")

submit = requests.post(
    f"{GATEWAY}/v1/jobs",
    headers=HEADERS,
    json={
        "endpoint": "/v1/convert/source/async",
        "backend": "docling",
        "priority": "batch",
        "payload": {
            "sources": [
                {
                    "kind": "file",
                    "base64_string": pdf_b64,
                    "filename": "annual-report.pdf",
                }
            ],
            # Gateway-level knob. Default is false (drop images).
            # Sibling of `sources` / `options`; the gateway extracts
            # it from the payload before forwarding to docling.
            # "include_images": False,
            # All docling conversion knobs go under `options`.
            # Putting them at the top level alongside `sources`
            # results in docling silently using defaults — see the
            # warning above this snippet.
            "options": {
                "to_formats": ["md", "json"],
                "do_ocr": True,
                "do_table_structure": True,
                "md_page_break_placeholder": "\f",
            },
        },
        # How long this conversion may run, sized from its page count —
        # see *Job size caps* below.
        "timeout_seconds": 1800,
    },
)
submit.raise_for_status()
job_id = submit.json()["id"]

while True:
    r = requests.get(f"{GATEWAY}/v1/jobs/{job_id}", headers=HEADERS)
    r.raise_for_status()
    job = r.json()
    if job["status"] == "completed":
        result = job["result"]  # the verbatim docling result JSON
        break
    if job["status"] == "failed":
        raise RuntimeError(job["error"])
    time.sleep(5)

The docling result shape — markdown, JSON document tree, etc. — is whatever docling returns at /v1/result/{task_id}. The gateway is a pass-through for that body: it polls docling’s async task endpoints internally and stores the final result body, so you never see a task wrapper (task_id / task_status) — the same document.md_content your code would read from a synchronous /v1/convert/file call is at the same path. Skeleton of a completed job (default include_images: false):

{
  "id": "<job id>",
  "status": "completed",
  "backend": "docling",
  "result": {
    "document": {
      "md_content": "…markdown; inline images appear as data:image/png;base64,STRIPPED_BY_GATEWAY…",
      "html_content": "…same stripping rule…",
      "json_content": {
        "pages":    { "1": { "size": {}, "image": null, "page_no": 1 } },
        "pictures": [ { "image": null } ]
      }
    }
  }
}

Sibling keys of document are whatever your docling version returns — the gateway neither adds nor removes them. With include_images: true, image fields carry real data instead of placeholders/nulls. A 5-line regression test that asserts result["document"]["md_content"].count("\f") >= n_pages would catch a future shape regression like the 2026-05-23 incident in seconds.

Job size caps. One limit, which you lower per job:

• GATEWAY_DOCLING_TIMEOUT (default 3900 s) — how long the gateway will keep polling docling before giving up on a job and marking it failed. Includes submission + polling + result fetch.

• timeout_seconds on the submit — set your own, sized from the page count and your rate (run_ms / input_pages on GET /metrics/recent): autostatement measured 8–9 s a page on the worker machines in October 2026. A value per document is fine. Read the caps live from GET /diagnostics:

  "timeout_seconds": {"docling": {"default_seconds": 3900, "max_seconds": 3900}, …}

  • When either fires, the job fails with nothing in it — never a document cut short. On the worker machines the gateway also stops the conversion, so the machine is free for the next job.

  • Don’t send options.document_timeout: it is refused with 400. The gateway gives every conversion the same one, 300 s past the cap, so docling never cuts a document short itself and keeps its models loaded between conversions.

  • result.status can still be partial_success: docling could not convert some pages. The job completes, with those pages missing.

    How to detect it. The gateway returns docling’s result body verbatim, so the signal is in your result:

    { "status": "partial_success", "errors": [ … ], "document": { … } }

    Compare result.status against "success". Note the wire value is the lowercase "partial_success" — PARTIAL_SUCCESS is the symbol name in docling’s source, not what arrives in JSON, so a consumer matching the upper-case form never matches. result.errors carries the detail. (Confirmed against the ConversionStatus enum docling-serve v1.34.0 publishes at GET /openapi.json.)

    Make this a hard check, not a log line, if truncation would be silent data loss for you — a statement missing its last pages still parses perfectly cleanly.

GATEWAY_DOCLING_TIMEOUT is a per-dispatch cap with the same semantics as its ollama counterpart — it excludes queue wait, and it starts later than status=running. Undercut it for a single conversion with timeout_seconds on the submit body; a value above 3900 is rejected at submit. See Timeouts: which clock measures what before sizing either that or a client-side bound against it.

Checking platform load

GET /queue returns a snapshot without affecting the queue:

{
  "in_flight": 1,
  "queued": { "interactive": 0, "batch": 3 },
  "running_by_tier": { "interactive": 0, "batch": 1 },
  "waiting_for_budget": 0,
  "waiting_for_large": 0,
  "completed_last_24h": 47,
  "failed_last_24h": 2,
  "worker_alive": true,
  "storage": {
    "db_bytes": 41582592,
    "wal_bytes": 4157440,
    "retention": {
      "default_ttl_seconds": 259200,
      "overrides_by_caller": { "autostatement": 2592000 },
      "sweep_interval_seconds": 3600
    }
  },
  "timestamp": "2026-05-23T09:25:01.124+00:00"
}

storage is the job database on disk — operator-facing, and reported because nothing did before: in May 2026 it reached 6.8 GB on 453 rows (a backend returning images inline, stored verbatim) and the first symptom was the gateway appearing dead. It rises with what you have submitted over one retention window — the retention sub-object beside it says which window, which is the other half of reading this number. Nothing caps it, and nothing runs VACUUM, so it does not fall back after a sweep. Worth watching before an unusually large batch, or if you hold a longer window.

in_flight counts running jobs: at most one on the large (see Single global in-flight slot above), plus one per busy worker machine. machines lists those machines — each one’s state (live; stopped, its agent stopped and said so, since stopped_at; silent, gone without saying; offline, silent over an hour; paused for its owner’s use, why as paused_reason, since paused_since; refused; disabled), the models it serves, the gateway_commit its gateway was built from as of last_checkin_at (-dirty with uncommitted changes, as a developer’s machine may be; null before its first check-in), what it has loaded (null unless it is checking in), its restarts of the last 30 days as it last reported them (clean, crashed; null before its first check-in), and the job it holds since job_started_at — with live and enabled counts. pins is what the gateway requires of every machine, one set of values for all, not a reading from each: a machine running anything else is refused and takes no work. For the window, read context_window. Hit this before a non-urgent submission if you want to be a good citizen: if queued.batch is deep, you might choose to defer. worker_alive is true in normal operation. false means the one coroutine that runs jobs is gone and nothing will drain the queue until an operator restarts the gateway — your jobs will sit queued rather than fail, so this is the field that tells you a stall is the platform and not your submission. Worth reading before you go looking for a fault on your side. timestamp is the gateway’s wall-clock at snapshot time; use it to correlate samples across time without relying on the relative lifecycle.last_activity_age_seconds.

The response also includes a lifecycle block with the on-demand large-instance state and (when running) two uptime views:

{
  "lifecycle": {
    "state": "ready",
    "in_flight": 0,
    "last_activity_age_seconds": 12.4,
    "last_error": null,
    "session": {
      "started_at": "2026-05-23T08:12:01.034+00:00",
      "uptime_seconds": 4501.2
    },
    "month_to_date": {
      "wall_hours": 18.42,
      "ocpu_hours": 331.6,
      "gb_hours": 1768.3,
      "month_start": "2026-05-01T00:00:00+00:00",
      "next_reset": "2026-06-01T00:00:00+00:00"
    }
  }
}

session is null when the large is stopped. month_to_date is informational today — useful if you want to know roughly how much of the monthly OCI free-tier budget has been spent so far this UTC month. A future release will surface remaining-budget on each job-status response and add a dedicated /usage endpoint with the same data; until then /queue.lifecycle.month_to_date is the place to look.

Knowing when the platform is ready for your call

For decisions that need to happen before you submit — “is the backend even up?”, “is the model I want already in RAM, or am I about to pay a cold-load cost?” — GET /diagnostics gives you a single derived state field plus a few denormalized lists:

{
  "state": "ready_warm",
  "warm": true,
  "docling_ok": true,
  "loaded_models": ["qwen3.8:27b"],
  "available_models": [
    "qwen3.8:27b",
    "qwen3:8b-q4_K_M"
  ],
  "managed_models": [
    "qwen3.8:27b",
    "qwen3:8b-q4_K_M"
  ],
  "ollama": {
    "url": "http://ollama:11434",
    "version": "0.34.2",
    "source": "this gateway's ollama",
    "ps":   { "models": [ { "name": "qwen3.8:27b", "size_vram": 0, "...": "…" } ] },
    "tags": { "models": [ { "name": "qwen3.8:27b", "digest": "…", "...": "…" } ] }
  },
  "docling": {
    "url": "http://docling:5001",
    "version": "1.34.0",
    "packages": { "docling-serve": "1.34.0", "docling": "2.128.0", "docling-core": "2.97.0", "...": "…" },
    "health": { "status_code": 200, "body": "…" },
    "source": "this gateway's docling"
  },
  "lifecycle": { ... state, in_flight, last_activity_age_seconds, last_error, warm — same as /queue's lifecycle EXCEPT /queue additionally embeds session / month_to_date / large_shape (the uptime-accounting block) ... },
  "timestamp": "2026-05-23T09:25:02.341+00:00"
}

Two shape rules on the ollama object worth knowing:

• ollama.version is always present — a string when ollama is reachable, null when it isn’t; where worker machines take ollama work, the release every machine is held to (machines.pins.ollama), and source says which. If you observe the key itself missing, you are talking to a gateway built before 2026-08-04; that’s a deployment defect to report, not a documented state.
• version is a denormalized string, but ps and tags are raw probe bodies — on a probe failure they carry {"error": "…"} instead of their normal shape. So when version is null, the reason is directly readable in the sibling fields.

docling.version and docling.packages follow the same rule (absent keys mean a gateway built before 2026-09-27): the docling-serve release and the docling packages inside it, or null when docling doesn’t answer — routine on a local stack, which stops docling while an ollama model holds the memory. So stamp conversions from each job’s metrics row, not from here. Where worker machines take conversions, version is the release every machine is held to (machines.pins.docling) and packages is null; source says which docling the block describes.

Backend bumps are not announced. Each engine reports its version here and on every job’s GET /metrics/recent row (engine_version, plus engine_packages for docling). A change there is your notice.

gateway_commit is the commit the gateway’s image was built from, with -dirty appended if it had uncommitted changes (null if it was built without one), and machines is the same block as on /queue.

state is the single source of truth for “what’s the platform doing right now,” and is one of:

Changed 2026-08-13: a docling-only outage no longer shows as backend_unhealthy. It used to, which meant one dead docling blanket-told every consumer to stop submitting — for a backend most of them don’t use — while also blocking all ollama work at the controller for 30 minutes. docling now has its own top-level boolean, docling_ok: a live probe of docling’s /health taken while serving that request, never cached, so a docling that just recovered reads true on the very next call with no gateway restart. docling_ok: false does not mean “don’t submit” — it means docling jobs will fail fast with docling unreachable: … until it is back, while ollama work is unaffected. If you gate docling submissions on platform state, switch from state to docling_ok.

On the local deployment, docling_ok: false is even weaker than that. Since 2026-08-18 a stopped docling is started on demand for the job that needs it (see One heavyweight backend at a time), so a false reading there means “not up right now”, not “your job will fail”. Check residency.enabled in the same response if you need to tell the two apart — but the simpler answer is not to pre-gate at all: submit, and read the job.

Where worker machines take ollama work — an enabled default machine lists an ollama model — a live one decides state: ready_warm while one is idle holding a model, ready_cold while one is idle, else busy (warming if none holds one yet). loaded_models and available_models add what machines hold and list. With none live, state follows the large; the large as a machine reads starting until the hub accepts it. Before 2026-10-29 the hub read backend_unhealthy while its machines ran jobs.

Where worker machines take conversions — an enabled machine lists docling in machines — none runs on the gateway’s own docling, so docling_ok is true while a live machine lists docling, and false means a conversion waits queued for one rather than failing. docling.health stays the gateway’s own probe.

Sleep and wake — the operational model in one paragraph. The on-demand large instance auto-stops itself after LIFECYCLE_IDLE_TIMEOUT_SECONDS (default 1800 s / 30 min) without a job, and auto-wakes when it is needed — unlocked, on the next job submission (the gateway issues an OCI start before dispatching); as a worker machine, when its model’s work has waited 10 min with every other machine busy, or 1 min with none up. Consumers don’t need to explicitly trigger wake. There is no “operator paused the platform, hold your submission” state in this API; that scenario is covered by disabled (controller off entirely) or failed (lifecycle gave up). For every other state a consumer can safely submit and let the gateway handle whatever’s needed; the phase field on /v1/jobs/{id} will then narrate boot → load → generation.

warm is the convenience boolean: state is ready_warm or busy, so a model is in RAM. True implies your next ollama inference starts immediately — no boot, no load. Use warm for the simple “is this a fast path?” check; use state when you need the full picture.

loaded_models and available_models let you check whether the specific model you want is resident (fast next call) or just on disk (will pay ~1–3 min load). If the model you need isn’t in available_models at all, you’ve targeted the wrong host or the model hasn’t been refreshed there yet — that’s an operator issue, not something to retry around.

The raw ollama.ps, docling.health and lifecycle blocks are there for ops debugging. Each backend reports errors in-band, so a single broken backend doesn’t blind you to the other — useful when you’re triaging which side is having a bad day.

ollama.tags is the exception — it is a supported read, because this page tells you to take weight-level provenance from ollama.tags[].digest (see Picking a model name). Don’t infer from its neighbours here that it can vanish.

ollama.ps[].expires_at is not a liveness signal. Don’t read it as one. It is the last request boundary plus OLLAMA_KEEP_ALIVE — while a request is actually in flight the runner is pinned by refcount and the field is never refreshed. So on the local deployment any generation longer than the keep-alive window shows an expires_at in the past while the job is perfectly healthy; on OCI (keep-alive -1) it doesn’t behave that way at all. That window was 300 s when this was measured, which made a past expires_at routine on batch work; it is 3600 s since 2026-08-16, so the same trap now needs an hour-long generation to spring. Don’t code to either number — the point is that the field is a boundary stamp, not a heartbeat. Confirmed by sampling /api/ps every 3 s through a live generation: frozen at one value for every sample. A past expires_at tells you nothing about whether a job is progressing or wedged. For that, read the running job’s own tokens_generated and last_token_at (Watching a running job), with status + phase for what the gateway is doing around it.

Checking recent throughput

GET /metrics/recent returns the last finished jobs and what they actually achieved — newest first, one row each:

{
  "jobs": [
    {
      "job_id": "9f3c1e77-3f4a-4a1e-9a4a-8c2f1b7d0e55",
      "completed_at": "2026-08-09T15:47:03.882+00:00",
      "outcome": "completed",
      "error": null,
      "backend": "ollama",
      "model": "qwen3:8b-q4_K_M",
      "deployment": "local",
      "host_id": "laptop-lenovo",
      "engine_version": "0.32.5",
      "caller_id": "regnskap",
      "tier": "batch",
      "input_bytes": 14203,
      "input_tokens": 3450,
      "output_tokens": 64,
      "input_pages": null,
      "output_bytes": 2841,
      "prefill_tok_s": 7.5,
      "decode_tok_s": 0.32,
      "load_ms": 96000,
      "queue_wait_ms": 0,
      "run_ms": 660000,
      "large_ocpu": 8,
      "large_gb": 24
    }
  ],
  "count": 1,
  "deployments": ["local"],
  "filter": {
    "backend": null, "deployment": null, "host_id": null,
    "outcome": null, "model": null, "job_id": null, "limit": 50
  },
  "timestamp": "2026-08-09T15:52:10.004+00:00"
}

This answers the question a single job can’t: is that rate normal here lately? A running job’s tokens_per_second tells you it is moving; this tells you whether it is moving at the speed this box normally manages. The row above is from the degraded window of 2026-08-09, where decode ran at 0.32 t/s against the 3.6–5.0 t/s the same work does on the same box on an ordinary day — visible in one request, where at the time it took an hour and a half of digging.

Filters, all optional and all absent by default:

How to read it without drawing the wrong conclusion:

• Compare like with like — same model, same deployment. A 72B and an 8B have nothing to say about each other, and the two deployments are different machines running different ollama pins. The deployments list shows every label present in the rows you got back; more than one means read it again with ?deployment=. deployment names a setup’s shape and repeats across machines (every local box says local); host_id names the machine whose gateway ran the job — see Which machine answered. null on rows written before 2026-10-04. gateway_commit names the code that gateway ran (null before 2026-10-07).
• outcome: "failed" rows ended early — cancelled, or out of time. They carry output_tokens (how far the work got before it was abandoned) but no rates, because there was no final response to read ollama’s timings from. Filter them out before comparing throughput, and read them for a different question: how often work is being thrown away, and how much of it.
• error says which kind of failure it was. outcome flattens four different problems into the word failed — you cancelled it, it hit your timeout_seconds, it hit the platform’s cap, or the backend errored. error carries the same string the job’s own record does, and outlives it: three days later the job is swept and this is the only place left that can tell a cancel from a timeout. null on completed jobs, and on rows written before this was recorded.
• No average, median or verdict is offered, deliberately. Run-to-run variance on byte-identical work measured ±39% on the local box, so a single summary figure would have looked healthy right through the incident this endpoint was built after. You set the threshold that matters to you; the platform reports what happened.
• Every row carries every field, null where a value doesn’t exist — never 0. A docling row has input_pages, engine_packages, the docling_* fields and null token rates; an ollama row the reverse. A cancelled job has a null decode rate because nobody measured one, and a 0 there would read as “it ran at no speed at all” to someone already hunting a problem.
• large_ocpu / large_gb are the machine that ran the model, as the gateway was configured to believe it. Correct on any deployment whose LARGE_INSTANCE_OCPU / LARGE_INSTANCE_GB are set to match reality — which for a local stack means setting them (see .env.example); left unset, a laptop stamps every row with the OCI machine’s shape.
• A row appears shortly after the job goes terminal, not at the same instant. The gateway writes the job’s final status, reads the engine version, then inserts the row. If you poll for completed and query this immediately, the row may not be there yet — retry rather than concluding it wasn’t recorded.
• decode_tok_s is ollama’s own number (eval_count ÷ eval_duration), where a running job’s tokens_per_second is the gateway counting tokens as they arrive. They should agree closely. The gateway’s is available while the job runs; ollama’s only once it has finished.
• Ignore prefill_tok_s in the thousands — that is a cache hit, not hardware. Resubmit a prompt whose prefix ollama still has in its KV cache and the prefill is skipped, so prompt_eval_duration comes back near zero and the arithmetic produces a nonsense rate. A real run of the same 2972-token prompt on the local box reported 24 t/s; the repeat two minutes later reported 5443 t/s. The second number describes cache reuse. Compare prefill only across first-runs, or don’t compare it at all — decode_tok_s is the honest health signal.
• These rows outlive the jobs they describe. The metrics table is not swept by job retention at all, so a job whose GET /v1/jobs/{id} already returns 404 still appears here. It is not a way to retrieve results — there is no result in these rows, by design.

Reference: end-to-end consumer pattern

The minimal examples earlier in this page elide a few things that matter in real consumers — pre-flight /diagnostics check, phase-aware UX, robust timeout, and explicit failure handling. Copy-paste the snippet below as a starting point rather than re-derive it. It assumes the model name and Choosing a priority for your job sections above have been read.

"""Minimal but properly-shaped consumer of the platform gateway.

Reads LLM_GATEWAY_URL and LLM_MODEL from env (no defaults — failing
loudly beats defaulting to a wrong gateway, see #picking-a-model-name).
"""
import os
import time
from typing import Callable, Optional

import requests

GATEWAY = os.environ["LLM_GATEWAY_URL"]    # e.g. http://llm-gateway:11435
MODEL = os.environ["LLM_MODEL"]            # e.g. qwen3.8:27b
CALLER = "my-consumer"                     # short stable identifier; logged
# Your caller token's file, when you have one (#authentication); it names you.
TOKEN_FILE = os.environ.get("LLM_GATEWAY_TOKEN_FILE")
HEADERS = ({"Authorization": f"Bearer {open(TOKEN_FILE).read().strip()}"}
           if TOKEN_FILE else {"X-Caller-Id": CALLER})
PRIORITY = "batch"                         # "interactive" or "batch"; default batch


class GatewayError(RuntimeError):
    """Anything the gateway tells us went wrong. .job is the recorded
    row when the failure was a job-level error (so callers can read
    .job['error'], .job['phase'], etc.); None for pre-submit failures."""
    def __init__(self, message: str, job: Optional[dict] = None):
        super().__init__(message)
        self.job = job


def diagnostics() -> dict:
    r = requests.get(f"{GATEWAY}/diagnostics", timeout=5)
    r.raise_for_status()
    return r.json()


def preflight(model: str, backend: str = "ollama",
              expect_host: str | None = None) -> dict:
    """Confirm the gateway will accept work for the given model.
    Raises GatewayError if the platform is in a non-submittable state
    or the model isn't available on the target host."""
    d = diagnostics()
    if expect_host is not None and d.get("host_id") != expect_host:
        # Which *machine* answered. Worth checking whenever more than one
        # box can serve you: `deployment` is the same string on all of
        # them, so a misrouted URL or a copied .env is otherwise silent
        # and lands in whatever provenance you record.
        raise GatewayError(
            f"gateway is {d.get('host_id')!r}, expected {expect_host!r}")
    if d["state"] in ("failed", "backend_unhealthy"):
        # `backend_unhealthy` is ollama-specific since 2026-08-13.
        raise GatewayError(f"platform not ready: state={d['state']}")
    if (backend == "docling"
            and d["state"] not in ("stopped", "starting", "stopping")
            and not d["docling_ok"]):
        # docling health is its own field, not part of `state`. Skipped
        # whenever the large is asleep or in transition: docling lives
        # on the large, so nothing is reachable then, and submitting is
        # what wakes it — the job just queues until the boot finishes.
        raise GatewayError("docling is not responding; retry later")
    if model not in d["available_models"] and d["available_models"]:
        # Empty list means we couldn't reach ollama (likely instance
        # asleep); submission will trigger wake and we'll learn for
        # sure then. Only fail loudly when we have a definitive list.
        raise GatewayError(
            f"model {model!r} not on disk; available={d['available_models']}"
        )
    return d


def submit(backend: str, endpoint: str, payload: dict) -> str:
    r = requests.post(
        f"{GATEWAY}/v1/jobs",
        headers=HEADERS,
        json={
            "backend": backend,
            "endpoint": endpoint,
            "payload": payload,
            "priority": PRIORITY,
        },
        timeout=10,
    )
    r.raise_for_status()
    return r.json()["id"]


def wait_until_terminal(
    job_id: str,
    *,
    timeout_s: int = 7200,
    poll_interval_s: float = 5.0,
    on_phase_change: Optional[Callable[[Optional[str]], None]] = None,
) -> dict:
    """Poll /v1/jobs/{id} until completed or failed. Calls
    on_phase_change(phase) once per (deduped) transition — the right
    place to render UX state."""
    deadline = time.monotonic() + timeout_s
    last_phase: Optional[str] = "__init__"
    while time.monotonic() < deadline:
        r = requests.get(f"{GATEWAY}/v1/jobs/{job_id}",
                         headers=HEADERS, timeout=10)
        r.raise_for_status()
        job = r.json()
        phase = job.get("phase")
        if phase != last_phase:
            if on_phase_change:
                on_phase_change(phase)
            last_phase = phase
        if job["status"] == "completed":
            return job
        if job["status"] == "failed":
            raise GatewayError(f"job failed: {job.get('error')}", job=job)
        time.sleep(poll_interval_s)
    raise GatewayError(
        f"job {job_id} did not terminate within {timeout_s}s; "
        f"last phase was {last_phase!r}"
    )


# Example: chat completion
def chat(messages: list, timeout_s: int = 7200) -> str:
    preflight(MODEL)
    job_id = submit("ollama", "/api/chat", {"model": MODEL, "messages": messages})
    print(f"submitted {job_id}")
    job = wait_until_terminal(
        job_id,
        timeout_s=timeout_s,
        on_phase_change=lambda p: print(f"  phase={p}"),
    )
    return job["result"]["message"]["content"]


if __name__ == "__main__":
    print(chat([{"role": "user", "content": "Say pong."}]))

What this snippet encodes that the minimal examples don’t:

• Pre-flight /diagnostics with a defensive empty-list check. When the large is asleep, available_models is empty and we don’t fail loudly — submission will wake it and we’ll learn then. Only fail when we have a definitive list that excludes the requested model.
• Phase-change callback for UX state. Deduped against the previous phase so callers only see real transitions.
• Distinct error type that preserves the gateway’s recorded job row (including the final phase), so failure callers can read error.job["phase"] and render “failed during ollama_loading_model” instead of generic “failed.”
• Explicit timeout with the last-seen phase reported on expiry — far easier to diagnose than a bare TimeoutError.

Adapt the chat() example to your shape: docling jobs use submit("docling", "/v1/convert/source/async", {...}), embeddings use submit("ollama", "/api/embeddings", {...}). The submit/poll/phase plumbing stays identical.

Co-hosting a consumer on the small host

For a consumer that runs as its own compose project on the small host, next to the gateway — rather than on a laptop over the SSH tunnel. Everything in Calling the gateway still applies; this section is only the delta that co-location adds.

The compose contract

Your project is a sibling, not a member. Don’t join platform_default — stay decoupled so you keep starting when the platform is rebuilding (see Connecting).

name: your-project                     # own project name, not `platform`

services:
  worker:
    extra_hosts:
      - "llm-gateway:host-gateway"     # → the host's published :11435
    environment:
      - LLM_GATEWAY_URL=http://llm-gateway:11435
      - LLM_MODEL=qwen3.8:27b
    volumes:
      - your_project_data:/data        # your own named volume
    restart: unless-stopped
    mem_limit: 512m                    # be explicit
    cpus: 0.5

volumes:
  your_project_data:

• Don’t publish 80, 443, 11435 or 5006 — all four are taken (public edge, gateway, docs vhost); 12435 is autostatement’s sidecar door, on loopback. Pick another port and let nginx proxy you if you need public exposure. A door of your own that agents reach over the tunnel binds 127.0.0.1:<port>:<port> and gets a second permitopen on your key line (Authorize the key as tunnel-only).
• Own named volume. Never write into llm_gateway_data.
• Your token as a file. Mount it as a Compose secrets: file, not an environment: value: docker inspect shows environment values to anyone who can run docker on the small. Every process on the small can reach the gateway, so send private documents only while caller_auth is required (Authentication).
• The small is 1 OCPU / 6 GB, shared with the gateway’s event loop and its SQLite writes. Politeness: poll no faster than every 5–10 s, no busy-wait loops, run a real WSGI/ASGI server (not Flask’s dev server), and set explicit mem_limit / cpus.

Runtime state comes from /queue, not from this page

This page is docs/consumer.md rendered by the docs vhost. It proves which doc SHA is deployed — never which environment variables the gateway is running with. The two drift: a feature can be documented here and switched off in /etc/environment, or the reverse. Don’t infer operational state from prose.

fuse_enabled reports the runtime constant itself. Note that FREE_TIER_FUSE_ENABLED defaults to false (gateway.py:82-84, docker-compose.yml:171) — it is on only if an operator set it on the host and force-recreated the container.

Pre-first-batch checklist

1. GET /queue — read free_tier.fuse_enabled, fuse.tripped, free_tier.binding_pct, month_to_date.next_reset, and budget-size the batch (below).
2. GET /diagnostics — state is not failed / backend_unhealthy; if the batch is docling work and worker machines take it or the large is already awake (docling.source reads worker machines: …, or state is not stopped / starting / stopping), docling_ok is true — it is not part of state, and without machines it reads false whenever the large is asleep, which is not a fault; if the large is awake, available_models contains your pinned tag.
3. GET /status.json — record the live SHA next to your run so a later result diff can be attributed.
4. GET /diagnostics — host_id is the machine you meant, if more than one can answer you. Record it beside the SHA: it is the only field that distinguishes two boxes sharing a deployment label.

Budget sizing

On the current shapes — small 1 OCPU / 6 GB, large 18 OCPU / 96 GB — OCPU-h is the binding meter. (It was GB-h before the RAM cut; don’t hardcode either, read binding_metric.)

large budget ≈ (1500 OCPU-h − 1 OCPU × hours_in_month) ÷ 18 OCPU
             ≈ 42 wall-hours / month
usable       ≈ 42 × (stop_at_pct / 100)      ≈ 38 h at the default 90%
remaining    ≈ 42 × (1 − binding_pct / 100)

Re-derive if either shape changes — constants live in gateway.py:62-63 (LARGE_OCPU / LARGE_GB) and :87-88 (FREE_TIER_SMALL_OCPU / FREE_TIER_SMALL_GB), Makefile:737-740; the math is in budget.free_tier_status, budget.py:157-161.

A batch whose large-side wall-time exceeds remaining will not finish this month. It drains until the fuse trips, then parks the rest until next_reset (budget.py:104). Each cold start also spends ~20 min of the meter on the 72B disk→RAM load, so a batch spanning a month boundary pays that twice.

Job states when the fuse is tripped

Four behaviours that will break a naive poller:

1. running is not a latch. Parking is claim-then-demote: claim_next marks the job running, the fuse check runs, then set_waiting_for_budget demotes it with started_at cleared (JobStore.claim_next / .set_waiting_for_budget, gateway.py:783-824). A poll can land inside that window. Under a tripped fuse the queue’s head job legitimately cycles queued → running → waiting_for_budget about once a minute, indefinitely. Never infer “it started, a result is coming” from a single running observation.
2. Parking is rate-limited to one claim per FREE_TIER_BUDGET_CHECK_INTERVAL_SECONDS (default 60 s — BUDGET_CHECK_INTERVAL_SECONDS, gateway.py:94-96; the worker parks the job it claimed, then sleeps that long before claiming again, gateway.py:3373-3381) — and it re-claims the same head row every time. claim_next orders by tier then created_at and set_waiting_for_budget changes neither (JobStore.claim_next, gateway.py:783-810), so the head cycles indefinitely while everything behind it stays queued. Submit 100 jobs under a tripped fuse and /queue shows one waiting_for_budget and 99 queued, for as long as the fuse is up. Don’t assert prompt parking, and don’t read queued as “not yet noticed”.
3. Parked jobs are inert. They don’t count as activity for idle-stop — has_pending_or_running covers only queued and running (JobStore.has_pending_or_running, gateway.py:1186-1196). And a tripped fuse overrules the idle timer rather than queuing behind it: the idle tick converts its own “maybe stop” verdict into a fuse stop (LifecycleController._idle_tick, lifecycle.py:1534-1547), and the fuse-stop path commits without consulting that queued-work deferral at all — it waits only for an in-flight job to finish (lifecycle.py:1582-1593). So a queued pile can’t hold the large up while over budget. A pile parked under a tripped fuse burns nothing.
4. queue_position is absent while parked — computed only for status == "queued", None for anything else (JobStore.queue_position, gateway.py:757-781). Don’t drive progress UI from it.

DELETE /v1/jobs/{id} works on a held job (cancel_if_queued covers waiting_for_budget, gateway.py:977-991), so a consumer-side deadline → cancel is a valid escape. Held jobs are never swept — retention deletes only completed / failed rows (JobStore.cleanup, gateway.py:1050-1097) — so a month-long park loses nothing.

Model pre-flight when the large is asleep

Keep two signals separate:

• available_models (on disk) → correctness. Your fail-loud check.
• loaded_models / lifecycle.warm → latency only. A cold-but-available model is legal, just slower. Never gate correctness on these.

The trap: when the large is stopped — most of the time under an on-demand plus tripped-fuse regime — /diagnostics probes ollama, the probe fails, and available_models comes back [] (gateway.py:2117-2119). A naive “is my tag present?” check then fails closed and blocks every submission, indistinguishable from a genuinely missing model. Gate on state first:

Storage footprint on the small

Two things accumulate in the gateway’s SQLite on a 6 GB / 50 GB-boot host:

• Results are retained for your full window whether or not you’ve fetched them — collecting doesn’t free them server-side. Peak ≈ output bytes per job × jobs per day × retention days, so it scales with any override: autostatement’s 30 days is ten times the 72 h peak. Image-stripped docling JSON is far smaller than raw, but check the arithmetic against storage.db_bytes on /queue. Nothing runs VACUUM, so the file never shrinks back after a sweep — it only stops growing.
• job_metrics is deliberately never swept (the job_metrics table in the gateway’s SCHEMA) — one small row per finished job, kept past the jobs TTL on purpose so the CPU-scaling dataset survives, and readable via /metrics/recent long after the job record itself is gone. Rows are ~200 B, so thousands of jobs is single-digit MB.

Idempotency

There is no idempotency key on POST /v1/jobs. A gateway restart marks in-flight running jobs failed (“gateway restarted while job was running”) while queued jobs survive, so resubmission is a normal path — dedupe on your own job key; the gateway will not do it for you.

Pointing local dev at OCI (the default dev path)

For day-to-day consumer development, run your consumer locally and point it at the OCI gateway via an SSH tunnel. This gives you local hot-reload, editor tooling, and dev container niceties while the LLM and docling work runs on the 72B-class models in OCI.

Why this is the default rather than local platform-services:

• Model parity. Your consumer’s prompts run against the same models every other path into the OCI large uses, so behaviour you observe from a dev machine is what you get running anywhere else.
• No 40+ GB model pulls on every new dev environment. Bootstrapping a new machine doesn’t have to wait on a giant download.
• Local machines can’t host the 72B models anyway. Local platform-services is sized for 8B models — useful for exercising wiring, not for evaluating output quality.

Reach for local platform-services instead when you have a concrete reason: working offline, intentionally testing the small models, or OCI is unavailable. Switching is one env var change.

Quickstart

Two tunnel patterns — pick by where your consumer runs:

• Inside a Docker dev container → Option B: tunnel from inside the container. The path that “just works” on Docker Desktop + WSL2 without host-side networking changes.
• Native on the WSL distro / Linux / macOS (no container) → Option A: tunnel from the host.

Either way, one-time bootstrap is the same: generate key → authorize on OCI → SSH alias → run tunnel. Errors: Troubleshooting.

How it works

Two viable topologies depending on where the consumer process runs. Same key, same OCI authorization, same SSH alias — only the location of the ssh -L and the consumer’s URL differ.

Option A — tunnel on the WSL host. ssh -L runs on the WSL distro; the consumer reaches it via host-gateway. Works cleanly for consumers running natively on the distro (no container). For consumer dev containers under Docker Desktop + WSL2, also requires WSL in mirrored networking mode — otherwise WSL-host-bound ports are invisible to containers, regardless of 0.0.0.0 bind.

consumer (in container or native) → host-gateway → ssh -L on WSL → OCI:localhost:11435

Option B — tunnel inside the dev container. ssh -L runs in the consumer container; the consumer hits its own loopback. Side- steps host-side networking entirely. Requires mounting the tunnel folder (below) into the container — that folder only, never the distro’s ~/.ssh.

consumer (in container) → localhost → ssh -L in same container → OCI:localhost:11435

One SSH key per dev distro

Each WSL distro (or local machine) that wants the tunnel gets its own key, in its own folder. Don’t copy keys between distros — that defeats their isolation. In the consumer-project distro:

mkdir -m 700 -p ~/.ssh-oci-tunnel
ssh-keygen -t ed25519 -C "<distro-name>-tunnel" -f ~/.ssh-oci-tunnel/oci_tunnel
cat ~/.ssh-oci-tunnel/oci_tunnel.pub

This is a separate key from the operator’s full-access key documented in ../README.md → SSH access. That key exists for managing the instance; this one is for tunnels only. A distro that also deploys to OCI keeps its deploy key elsewhere, with a passphrase.

The folder holds everything the tunnel needs — key, SSH config, known_hosts — and nothing else, so Option B can mount it into the dev container without exposing anything in your ~/.ssh. “One key per distro” means per WSL distro.

Authorize the key as tunnel-only

Tunnel keys log in to OCI as platform-tunnel, never ubuntu. That user has no shell and no groups, and OCI’s sshd lets it do one thing: forward a local port to a listed loopback port — the gateway at localhost:11435, or a co-hosted consumer’s own door — no reverse forwards, no Unix-socket forwards. (Tunnel keys used to go into ubuntu’s file, where ubuntu’s docker group made a socket forward to /var/run/docker.sock root on the host.)

The public half goes into OCI’s /etc/ssh/platform-tunnel/authorized_keys, with per-key restrictions as a second layer:

command="echo tunnel-only access; exit 1",no-pty,no-agent-forwarding,no-X11-forwarding,no-user-rc,permitopen="localhost:11435" ssh-ed25519 AAAA... <distro-name>-tunnel

Why not the restrict umbrella keyword? restrict also turns off port forwarding, so restrict,permitopen=... refuses the tunnel with administratively prohibited — documented OpenSSH behaviour, not a 9.6p1 bug; restrict,port-forwarding,permitopen=... would work. The expanded form stays because it is the one verified live (2026-05-19).

Where this command runs. Not from the consumer distro — that’s the distro we’re granting access, so it can’t authorize itself yet. The append runs from a machine that already has admin SSH to OCI — typically the operator’s platform-services WSL distro. One-liner from there:

echo 'command="echo tunnel-only access; exit 1",no-pty,no-agent-forwarding,no-X11-forwarding,no-user-rc,permitopen="localhost:11435" ssh-ed25519 AAAA... <distro-name>-tunnel' \
  | ssh oci-arm 'sudo tee -a /etc/ssh/platform-tunnel/authorized_keys >/dev/null'

Verify it landed:

ssh oci-arm 'tail -1 /etc/ssh/platform-tunnel/authorized_keys'
ssh oci-arm 'tail -1 /etc/ssh/platform-tunnel/authorized_keys' | grep -oE 'ssh-ed25519 \S+ \S+$' | ssh-keygen -lf -

The grep -oE step strips the long options prefix (which contains spaces inside command="...") and isolates the bare <keytype> <keydata> <comment> so ssh-keygen -lf - can fingerprint it. Cross-check the printed fingerprint against the one the consumer distro printed in Generate the key (step 1). If they match, the key is intact through paste.

After this one-time bootstrap, the consumer distro talks to OCI directly forever — the operator distro is just the “trusted introducer” that vouches for the new key on day one.

What each option does (OCI’s sshd enforces the same for platform-tunnel; these are the second layer):

• command="echo tunnel-only access; exit 1" — forces every non-tunnel session to print and exit, blocking ssh … 'cmd' from running arbitrary commands. Port forwarding uses a separate SSH protocol path and runs before the forced command, so forwards still work.
• no-pty — denies PTY allocation (no interactive shell).
• no-agent-forwarding — blocks SSH agent forwarding.
• no-X11-forwarding — blocks X11 forwarding.
• no-user-rc — blocks execution of ~/.ssh/rc at login.
• permitopen="localhost:11435" — permits forwarding to llm-gateway only. A co-hosted consumer’s key adds one more permitopen="localhost:<port>" for its own door; sshd must list that port too (scripts/ssh-tunnel-user.sh).

If this key leaks, the holder reaches the gateway port and nothing else on the host: no shell, no commands, no files, no other port. The gateway itself checks no identity, though, so that means every gateway endpoint — submit, read any job, cancel all — until the key’s line is removed from that file.

SSH config alias

In the tunnel folder, as ~/.ssh-oci-tunnel/config:

Host oci-tunnel
  HostName 79.76.60.187
  User platform-tunnel
  IdentityFile ~/.ssh-oci-tunnel/oci_tunnel
  IdentitiesOnly yes
  UserKnownHostsFile ~/.ssh-oci-tunnel/known_hosts

Every tunnel command passes -F ~/.ssh-oci-tunnel/config, so ssh reads this file and nothing from ~/.ssh. Run the first tunnel from the WSL distro: it asks you to confirm OCI’s host key and records it in the folder. (Older setups tunnelled through a Host oci-arm alias in ~/.ssh/config that logged in as ubuntu; that login no longer takes tunnel keys.)

Run the tunnel

Pick the option matching where your consumer runs. permitopen on OCI checks only the remote destination, so the same authorized_keys line works for either option.

Option A — from the WSL host

ssh -F ~/.ssh-oci-tunnel/config -L 0.0.0.0:11435:localhost:11435 oci-tunnel -N

The 0.0.0.0 bind (not the default 127.0.0.1) is what lets a consumer dev container reach the port via host-gateway. On Docker Desktop + WSL2 this also requires the distro to be in mirrored networking mode — in Windows ~/.wslconfig:

[wsl2]
networkingMode=mirrored

then wsl --shutdown from PowerShell. Without mirroring, WSL-bound ports aren’t visible to containers no matter how the tunnel binds, and Option B is the right path.

Consumer-side wiring:

extra_hosts:
  - "llm-gateway:host-gateway"
environment:
  LLM_GATEWAY_URL: "http://llm-gateway:11435"

Option B — from inside the dev container

Changed 2026-10-01: Option B no longer mounts the distro’s whole ~/.ssh. That mount handed the container every key in the folder, and — with the read-only flag found not applied on one consumer — the power to rewrite ~/.ssh/config, whose lines can run commands the next time you use ssh on the host. If your setup relies on the old mount, don’t rework it on your own: raise it with the operator, for a discussion with an agent in the platform-services repo.

Mount the tunnel folder, and only it, read-only at the same path in the container. This needs a compose-based devcontainer: readonly in a devcontainer.json mounts entry isn’t honoured. In the dev service’s volumes: in .devcontainer/docker-compose.yml:

      - ${HOME}/.ssh-oci-tunnel:/home/vscode/.ssh-oci-tunnel:ro

(${HOME} is the WSL distro’s home. Adjust /home/vscode to your container user, who needs the WSL user’s uid: ssh refuses a key it doesn’t own.) Rebuild the container, then confirm from a terminal inside it that the mount really is read-only:

touch ~/.ssh-oci-tunnel/x 2>/dev/null && echo "WRITABLE: delete x, fix the mount first" || echo "read-only: ok"

Then:

ssh -F ~/.ssh-oci-tunnel/config -L 11435:localhost:11435 oci-tunnel -N

No 0.0.0.0 needed — the tunnel and the consumer share the container’s loopback. Consumer-side wiring:

environment:
  LLM_GATEWAY_URL: "http://localhost:11435"

No llm-gateway:host-gateway entry in extra_hosts — the URL points at the container’s own loopback.

Persistence

SSH tunnels die with sleep, network changes, or laptop suspend. Under Option A: wrap with autossh or a systemd --user unit. Under Option B: re-run in the container terminal, or add a postStartCommand that backgrounds it. Skip both until it actually annoys you.

Sharing localhost with a local platform-services

If you run the tunnel from the same machine that already has a local platform-services compose stack up, port 11435 is already claimed by the local stack. Bind the tunnel to an alternate local port instead — pick something far enough from the regular range to be unambiguous (the 21000 prefix is a useful convention):

ssh -F ~/.ssh-oci-tunnel/config -L 21435:localhost:11435 oci-tunnel -N

This is the convention platform-services’ own platform-tunnel uses — it binds *:21435 precisely so the local stack on :11435 and the OCI tunnel can coexist.

The consumer chooses which to hit by switching its LLM_GATEWAY_URL env var between the two port numbers:

• http://llm-gateway:11435 → local platform-services — but only if GATEWAY_HOST_PORT is unset there; verify rather than assume
• http://llm-gateway:21435 → OCI via tunnel

The extra_hosts mapping is identical for both; only the port number discriminates. permitopen on the OCI side only checks the remote target (localhost:11435), never the local bind port — so the same authorized_keys line works regardless of which local port you pick.

Diagnosing a wrong-port hit. If your consumer points at :11435 and you have a local stack running, the call lands on the local stack silently — there’s no error, just data from the wrong place. The reliable tell in a GET /queue response is lifecycle.state:

• "disabled" → you are on a local gateway. LIFECYCLE_ENABLED defaults to false off OCI, so a local gateway reports disabled and never manages an instance.
• Any of stopped / starting / ready / … → you are on OCI, where the lifecycle controller is enabled.

Don’t try to tell them apart by response shape (missing timestamp, in_flight_by_backend instead of a scalar in_flight, absent lifecycle block). Those only ever indicated an out-of-date gateway, and a local stack built from the current repo is byte-identical to OCI’s — same code, same API. Shape tells you the version, not the location.

If you’re on :11435 and meant to reach OCI, switch to :21435 (assuming the tunnel is up) and re-check.

How granular the local↔︎OCI switch is depends on how your consumer reads the env var. If it’s baked in at container start (e.g. fixed in the compose file), switching means a container restart. If it’s read per process invocation (common Python pattern: os.environ.get(...) inside the entry point), you can flip per command — and even run one process against OCI and another against local simultaneously from the same container:

LLM_GATEWAY_URL=http://llm-gateway:21435 \
  python -m scripts.coverage_probe_pass2 --full

Your devcontainer can quietly take the gateway’s port

Port politeness on a shared dev machine is the same idea as Co-hosting a consumer on the small host above, with one extra hazard that has nothing to do with compose: your editor forwards ports too, and it picks the host port silently.

VS Code auto-forwards ports from a devcontainer to the host. When the host port it wants is already taken, it doesn’t warn or fail — it takes the next free port. Which is how a consumer asking for a perfectly reasonable port ends up holding the gateway’s.

Observed 2026-08-12, and worth reading as the shape rather than the specific numbers: a consumer’s devcontainer forwarded container port 11434 (ollama). Host 11434 was already published by the local docker-compose.local.yml ollama, so VS Code took 11435 — the gateway’s port. The platform stack was then unreachable on the dev machine for as long as that devcontainer stayed open.

The symptom does not look like a port conflict. Connections hang and time out having received zero bytes, rather than being refused. A refusal would have said “nothing is listening”; a hang says “something accepted and never answered”, which reads like a wedged gateway. Every probe behaves this way — curl, make queue, make status.

Two further tells that point away from the real cause, so know them in advance:

• docker compose ps shows the gateway with a bare 8080/tcp and no host mapping — while docker inspect <container> --format '{{json .HostConfig.PortBindings}}' still shows 11435 was requested. Compose asked for the publish, Docker couldn’t take the port, and nothing raised an error. --force-recreate reproduces it exactly, which makes it look like a broken compose file.
• Other containers’ ports work fine, because nothing collided with them. “Publishing works, just not for this one service” is the fingerprint.

Check it in two places. In VS Code, the PORTS panel — and read the Forwarded Address column, not the Port column: the row is labelled with the container port, so the one holding 11435 may be listed under 11434. On Windows:

Get-NetTCPConnection -LocalPort 11435 -ErrorAction SilentlyContinue |
  Group-Object State | Select-Object Name,Count

A Listen entry owned by Code.exe is the confirmation.

Fix it by right-clicking the offending row → Stop Forwarding Port. That releases the listener, but sockets the extension already leaked stay in CloseWait until that editor window exits (1150 of them in the observed case) — harmless, since they cannot block a new bind, but don’t read them as “still broken”.

Stop it recurring in the consumer’s own .devcontainer/devcontainer.json:

"portsAttributes": {
  "11434": { "onAutoForward": "ignore" }
}

The general rule: a consumer devcontainer should not auto-forward any port the platform uses — 11435 (gateway), 5006 (docs vhost), 21435 (the OCI tunnel). Forwarding a port near them is enough to cause this, because the next-free-port rule walks upward into the range.

Shorter than it used to be, as of 2026-08-19. 11434 (ollama) and 5001 (docling) are no longer on that list because the local backends publish no host ports at all — they talk to the gateway over the container network, where nothing on Windows can reach them. This followed a second incident of the same kind: docling ran healthy for 20 minutes while the gateway could not reach it, because its host port had silently stopped being published. Keeping those two out of your portsAttributes costs nothing, but they can no longer collide with anything.

If the port does not come back, move the gateway instead. Docker can stay unable to publish a port after one failed attempt — observed 2026-08-12 surviving a daemon restart and a machine reboot, with the port provably free on the host and no container claiming it. Rather than keep hunting, set GATEWAY_HOST_PORT on that machine (see .env.example) and redeploy; the container port is unchanged, so a consumer only edits the port in its URL:

LLM_GATEWAY_URL=http://llm-gateway:31435

This is a dev-machine escape hatch, not a contract change — the default is still 11435, which is what OCI serves and what every example on this page assumes. Pick a replacement well away from 11434: the failure is a walk upward from a taken port, so 11436 is the worst available choice.

Trade-offs

• Latency. ~10–30 ms per request from Northern Europe to Stockholm. Negligible for 5–15 minute jobs where one extra round trip on submission is invisible against the work itself.
• Bandwidth on docling uploads. PDFs are base64’d in the request body and travel over the tunnel. A 5 MB PDF (~7 MB base64) on a 20 Mbps upstream is ~3 seconds before docling starts. Fine for “interpret one or a few documents at a time”; notable for batch ingestion.
• Shared compute. The deployed runtime has a single global in-flight slot across all backends. Heavy local dev usage queues behind (or in front of) everyone else’s. The gateway’s priority tiers are how you cooperate — submit exploratory jobs with priority: batch so they don’t preempt interactive work.
• Tunnel lifecycle. SSH tunnels die with sleep, network changes, or laptop suspend. Either restart by hand or wrap in autossh.

When not to reach for the tunnel

• For a consumer deployed alongside platform-services on the OCI host. Use the regular extra_hosts: ["llm-gateway:host-gateway"] wiring directly — same Docker daemon, no tunnel needed.
• When the consumer needs to keep working offline. The tunnel needs the remote host to be reachable; if you need to work on a plane, fall back to local platform-services with the small models (set LLM_GATEWAY_URL to http://llm-gateway:11435 and LLM_MODEL to an 8B tag — output quality will differ from the 72B, but the wiring exercises end-to-end).
• When OCI is unavailable (capacity error on the on-demand large instance, region outage). Same fallback as above.

Troubleshooting

Migrating from direct backend access

If your consumer was wired to call ollama or docling directly, the migration is:

1. Add the gateway to extra_hosts (if it wasn’t already): "llm-gateway:host-gateway". Drop the old ollama: and docling: entries from the same list.
2. Replace direct backend calls with gateway job submissions. For ollama: POST /v1/jobs with endpoint: "/api/chat" (or whichever ollama path) and your old payload as payload. Then poll GET /v1/jobs/{id} until terminal. Worked example above.
3. For docling: switch from multipart upload to JSON+base64 submission via the gateway. Worked example above. The result shape is unchanged — the gateway returns docling’s /v1/result body verbatim under result.
4. If you were on the deprecated pattern A (joining platform_default), drop the networks: block — pattern A is gone. The extra_hosts: wiring above is the only supported shape.

The streaming-tokens code path (if you had one) doesn’t carry over — the gateway never streams to you. It streams from ollama internally, but the API stays submit-and-poll and hands you one complete body. For 5–15 minute generations nobody is watching tokens form anyway, and Watching a running job covers the part people actually wanted (is it moving, how fast). If you genuinely need tokens as they are produced, that’s a feature request, not a migration step.

What’s not here

• Users and roles. A caller token names a consumer, not a person, and the only role is admin (Authentication). Under caller_auth: off, the default on local stacks, anything that reaches the gateway port can use it, and X-Caller-Id is trusted, not verified. (This is about inbound auth. The outbound completion webhook is authenticated the other way — the gateway signs its callback to you with a per-caller HMAC secret so you can verify it. See Completion webhooks.)
• Rate limiting / per-caller quotas — not implemented. The gateway enforces priority and serialization but does not cap how much any one caller can submit.
• Streaming partial output from the gateway — clients see status changes via polling, not tokens-in-flight.
• Direct access to ollama or docling — internal to platform-services, see What about direct ollama / docling access? above.
• Tutorials for the ollama or docling APIs — see their upstream docs. (Exception: the docling submission shape used by the gateway is covered in Submitting a docling job above.)