Documentation

API documentation

Every call needs a key from an approved account — see Accounts. Three AI API versions are live: v3 is current, v1 and v2 keep working. mPAKT Compute (boxes that run your code on volunteers' devices) is a separate service at version 0, beta. The same reference as plain text is at /llms.txt.

Why v3Loading a model GraphsOperationsArchitectures SessionsCompiled graphs Server-side decodingPython Endpoint reference

Why v3 exists

v2 could express a matrix multiply but not a transformer. Running two decoder layers of a real model meant pulling activations back to your machine between every pair of steps it could offload, and getting the weights out there meant uploading them one HTTP request at a time into a coordinator-side quota. v3 fixes both.

Problem in v2What v3 does
Weights uploaded from your machine, base64 in JSON, against a 128 MB per-key quotaYou name a model. The coordinator streams it from the model host straight to volunteers. Nothing is uploaded and nothing is stored, so there is no quota to hit.
bf16 weights widened to f32 before upload, doubling the bytesbf16 and f16 travel as-is and are widened on the volunteer's device.
No rmsnorm, rope, grouped-query attention or swiglu, so a layer needed 3+ round tripsAll of them are operations. A whole decoder layer is one request.
Nothing persisted between calls, so a KV cache had to be resent every tokenSessions keep the cache resident on one worker and update it in place.

Loading a model

Look before you load. This reads only the tensor index — one range request, no weight data:

curl "https://YOUR-HOST/api/v3/models/inspect?source=hf:Qwen/Qwen3-0.6B" \
  -H "x-api-key: YOUR_KEY"

# { "count": 311, "totalBytes": 1191000000, "dtypes": { "bf16": 311 },
#   "tensors": [ { "name": "model.layers.0.self_attn.q_proj.weight",
#                  "shape": [1024, 2048], "dtype": "bf16", "bytes": 4194304 }, ... ] }

Then load the parts you want. include and exclude take glob patterns; names takes an exact list.

curl -X POST "https://YOUR-HOST/api/v3/models" \
  -H "x-api-key: YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"source":"hf:Qwen/Qwen3-0.6B",
       "include":["model.layers.26.*","model.layers.27.*"]}'

# { "modelId": "m_...", "tensors": [ { "name": "...", "tensorId": "t_..." }, ... ],
#   "bytesTransferred": 71300000, "throughputMBs": 24.6, "storedOnCoordinator": 0 }

Sources. hf:Org/Model, hf:Org/Model@revision/file.safetensors, or any https:// URL to a safetensors file whose host honours range requests.

Sharded models work. Most models past a couple of gigabytes ship as an index plus numbered shards rather than one file. If model.safetensors is absent, the loader reads model.safetensors.index.json beside it, takes the shard names from weight_map, and range-requests each one. You can also name the index outright. Shard names are never guessed — the usual model-00001-of-000NN convention is not universal, and real repos are zero-indexed — so weight_map is the only source of truth. inspect reports sharded and shards.

Large tensors are streamed in pieces. An embedding table is vocab x hidden and is the largest tensor in essentially every model — 134 MB in a 350M model, 525 MB in a 2B one. Tensors are fetched and forwarded in 16 MB pieces, so coordinator memory is bounded by the piece size rather than by the model. There is no per-tensor ceiling here; a tensor does still have to fit on one worker, since it cannot span them.

Preflight without transferring anything. GET /api/versions carries a limits block, and /api/v3/status reports device.maxTensorBytes — the largest single tensor any one connected worker could hold. Between those and inspect you can decide whether a load will succeed before starting it.

Weights stay packed. A bf16 model occupies its published size on a volunteer, not twice it — the operations that consume weights read the packed form in place. limits.widenFactor reports 1 for every dtype when the whole pool does this, and 2 for the half-precision ones when a worker in the pool predates it, since a tensor could land on any of them.

Every byte figure here is resident bytes — what a tensor occupies on a worker, not its size in the checkpoint. limits.byteUnits says so explicitly. With packed weights the two coincide for bf16; without, resident is twice stored. Compare device.maxTensorBytes against the resident figure, or you will accept a model that cannot be placed and find out only after transferring it.

Quantizing on load. A model that is too big for the pool can be held in a smaller form. Add "quantize" to the load:

{"source":"hf:HuggingFaceTB/SmolLM2-135M-Instruct", "quantize":"q8"}

# { ..., "quantize": "q8", "residentBytes": 143000000, "bytesTransferred": 270000000, ... }
quantize
"none"As published (bf16, f16 or f32, kept packed). The default, unless the operator has chosen another default for the grid.
"q8"Matrices held as 8-bit integers with a 16-bit scale per block of 32 weights: 8.5 bits per weight against 16 for bf16.
"q4"The same with 4-bit integers and a scale and offset per block: 5 bits per weight. The embedding table, the output head and the value and down projections stay at q8, where 4 bits hurts most.
"auto"The least aggressive of none, q8 and q4 whose resident size fits the pool right now.

Nothing changes on the wire: the published bytes travel as before and each volunteer converts them as they arrive, so the results do not depend on where the conversion ran. Norms, biases, small tensors and anything whose rows are not a multiple of 32 wide are left as published. The ops that read weights (matmul, linear, moe, embedding) use the quantized form directly. The load response reports quantize and residentBytes. A volunteer whose program predates this keeps the published dtype, so a request is never answered wrongly, only less approximately; capacity is planned at the quantized size only when every connected volunteer can do it. This costs accuracy. On SmolLM2-135M-Instruct, over 49 teacher-forced positions against bf16, q8 agreed with the top token on 48 (KL divergence 9e-4; weights 270 to 143 MB) and q4 on 43 (KL 0.07; 110 MB). Larger models lose less, but test yours; the chat page uses q8 or q4 only when a model does not otherwise fit, and says which.

Capacity is checked first. If the selection does not fit in the volunteers currently connected, the request is refused with the numbers, before anything is transferred:

{ "error": "model needs 1420 MB but only 890 MB is free across 6 worker(s)
             — load fewer layers or wait for more volunteers" }

Durability. Weights are not stored on the coordinator, so if a volunteer holding one disappears it is re-fetched from the same URL and byte range and pushed to someone else. The model host is the backing store. Your graphs do not notice.

What the volunteers run on

/api/v3/status reports backends — a count of what the pool is made of:

Backend
webgpuWGSL compute shaders. The usual case, and the fastest for most work.
npuA neural processing unit through WebNN — Intel AI Boost, Qualcomm Hexagon, the Apple Neural Engine. Silicon that is otherwise idle, so it does not compete with the browser's rendering and draws less power for the same arithmetic.
npu+webgpuBoth present. The worker times the first few matmuls on each and keeps the faster; an NPU is not automatically quicker for a given shape.
cpuPlain JavaScript kernels. Always available, and the fallback for anything a device declines.

You do not choose. The scheduler ranks workers on measured end-to-end time, so a backend earns work by being fast rather than by being named.

TPUs are not reachable from a browser — Cloud TPUs have no client-side API and a Coral Edge TPU needs a native driver — so they are not offered.

Graphs

Same shape as v2, with inputs that can name something already in the network instead of carrying data.

Input formMeaning
{ name, shape, data | dataB64 }Uploaded with this call. Use for activations.
{ name, model, weight }A weight from a loaded model, by its name in the file. Nothing transfers.
{ name, session }A tensor carried over from a previous step in this session.
{ name, tensorId }A tensor uploaded with the v2 tensor endpoint.

Add "encoding":"base64" beside graph to get outputs back as base64 f32 rather than a JSON number array — about a quarter the bytes, which matters for logits.

Operations

Every worker ships all of these. Nothing is compiled or downloaded per job. Shapes are row-major f32; ops that speak of rows and columns fold every leading dimension into rows, so [batch, seq, channels] behaves as [batch*seq, channels]. GET /api/ops describes each one, with its parameters, as JSON.

Parameters marked feature are optional capabilities. A step using one is routed only to volunteers that implement it, and a graph that no connected volunteer can run is refused with the missing op or feature named — never computed without it.

OpInputsParamsMeaning
matmula, btransBRow-major product. transB reads the right operand as [out, in] — the layout checkpoints store weights in — so no transposed copy is needed and no transpose steps are spent. It is also the faster arrangement: both operands are then walked contiguously. A weight is read as it is held on the volunteer (f32, bf16, f16, or the 8- and 4-bit forms below) without being widened to f32 first; a product with a few rows, such as a decode step, has a kernel of its own.
linearx, w, biastransBx·w + bias, fused. transB as for matmul.
rmsnorm v3x, gammaeps; features weightOffset, groupSizeRMS normalisation over the last dimension. weightOffset: 1 multiplies by 1 + gamma (Gemma). groupSize normalises each group of channels separately — per-head q/k norm (Qwen3, Gemma 3) without a reshape; gamma then has groupSize values.
rope v3xheadDim, offset, base; features rotaryDim, interleaved, scaling (invFreq, attentionFactor)Rotary embedding. offset is the absolute position of the first row. Half-split layout by default (Llama, Qwen, Mistral, NeoX); interleaved pairs adjacent channels (GPT-J, Cohere, GLM). rotaryDim rotates only the first channels of each head. scaling takes config.json's rope_scaling as-is — llama3, yarn, longrope, linear, dynamic — computed as transformers computes it.
attention v3q, k, v, bias?heads, headDim, kvHeads, vHeadDim, causal, posOffset; features scale, softcap, window, alibi, biasFused scaled dot-product attention with grouped-query support; the score matrix stays on the worker, and only the keys a row can see are read. Computed in one pass with a running softmax; for a few query rows (a decode step) the keys are also split across the device and combined, with the same result as one pass. vHeadDim for value widths that differ (MLA). window is sliding-window attention, masked exactly as HF does. softcap and scale cover Gemma 2; alibi: true BLOOM, alibiMax: 8 MPT. A fourth input [T,S] or [heads,T,S] is added to the scores (T5 position bias, custom masks — use -1e9 to mask).
swiglu v3gate, upsilu(gate) * up.
geglu v3gate, upgelu(gate) * up — the Gemma feed-forward.
moe v3x, logits, bias?, experts…topK, scoring, normalize, scale, nGroup, topKGroup, groupTopK, selectionBias, layout, activationA sparse mixture-of-experts feed-forward as one op: routes every token to its top-k experts, runs only those, sums the weighted results. Expert weights as stored, either one gate/up/down per expert (layout: "separate", as on the Hub) or 3-D "fused"/"stacked" tensors. Covers Mixtral, Qwen-MoE, OLMoE and DeepSeek-V2/V3 routing — see Architectures.
embedding v3ids, tableRow lookup; ids are integers stored as f32.
short_conv v3x, w_in, w_conv, w_out, stateA whole gated short-conv block as one step. Weights passed as stored [out, in]. state is advanced in place. See below.
conv1d v3x, weight, stateDepthwise causal 1-D convolution. x [T,C], weight [C,1,K], state [K-1,C] holding the positions immediately before x. Pure — see below for carrying the state.
argmax v3aIndex of the maximum over the last axis. This is how a token is chosen without sending the logits home — four bytes instead of 600 KB. Ties resolve to the lowest index.
reduce_max v3aMaximum over the last axis, one per row.
sample v3atemperature, topK, topP, seed, stepDraws a token on the device. temperature: 0 is exactly argmax. The RNG is counter-based on (seed, step), so a generation replays exactly and the worker holds no sampler state.
cache_write v3cache, valuesoffsetWrites rows into a cache in place and returns it. The only op with a side effect.
zeros v3—shapeAllocates zeros in the network, so a cache costs no upload.
concat / slice v3variesaxis, start, lengthWork on the leading axis or the last (feature) one; axis: -1 means the last. Middle axes are not supported.
reshape v3ashapeSame bytes, new shape.
add / sub / mula, bElementwise; b may be the same shape, a row vector, or a scalar.
scaleafactorMultiply by a constant.
relu / silu / tanh / sigmoidaActivations.
geluaGELU, tanh approximation — HF gelu_new and gelu_pytorch_tanh (GPT-2, Phi-2, Gemma).
gelu_exact v3aGELU with erf — HF gelu (BERT, Falcon, GPT-NeoX). Not interchangeable with gelu.
gelu_quick v3ax·sigmoid(1.702x) — HF quick_gelu (CLIP).
softcap v3acapcap·tanh(a/cap) — Gemma 2 final-logit softcapping.
softmaxaOver the last dimension, max-subtracted.
layernormx, gamma, betafeature epsOver the last dimension. eps defaults to 1e-5 (BERT uses 1e-12).
transposea2-D transpose.
causal_maskaSets the strict upper triangle of each square block to -inf.
reduce_sum / reduce_meanaOver every element.

A whole decoder layer, one request

{"graph":{
  "inputs":[
    {"name":"x","shape":[6,1024],"dataB64":"..."},
    {"name":"ln","model":"m_...","weight":"model.layers.26.input_layernorm.weight"},
    {"name":"wq","model":"m_...","weight":"model.layers.26.self_attn.q_proj.weight"},
    {"name":"wk","model":"m_...","weight":"model.layers.26.self_attn.k_proj.weight"},
    {"name":"wv","model":"m_...","weight":"model.layers.26.self_attn.v_proj.weight"},
    {"name":"wo","model":"m_...","weight":"model.layers.26.self_attn.o_proj.weight"}],
  "steps":[
    {"op":"rmsnorm",  "in":["x","ln"],      "out":"h",  "params":{"eps":1e-6}},
    {"op":"matmul",   "in":["h","wq"],      "out":"q0"},
    {"op":"matmul",   "in":["h","wk"],      "out":"k0"},
    {"op":"matmul",   "in":["h","wv"],      "out":"v"},
    {"op":"rope",     "in":["q0"],          "out":"q", "params":{"headDim":128,"offset":0}},
    {"op":"rope",     "in":["k0"],          "out":"k", "params":{"headDim":128,"offset":0}},
    {"op":"attention","in":["q","k","v"],   "out":"a",
       "params":{"heads":16,"headDim":128,"kvHeads":8,"causal":true}},
    {"op":"matmul",   "in":["a","wo"],      "out":"ao"},
    {"op":"add",      "in":["x","ao"],      "out":"y"}],
  "outputs":["y"]}}

Architectures

Most differences between transformer families are a parameter, not a different graph. Map config.json onto them rather than approximating — several of these give plausible wrong numbers, not errors, when left out.

config.json / modelParameter
rope_scaling / rope_parametersrope scaling, the object as-is. Llama 3.1, 3.2 and 3.3 use llama3 and are wrong without it even at short context. For longrope include max_position_embeddings unless attention_factor is set.
partial_rotary_factor, rotary_dimrope rotaryDim (GPT-NeoX/Pythia, Phi-2, StableLM, GPT-J).
GPT-J, CodeGen, Cohere, GLMrope interleaved: true.
sliding_windowattention window, on the layers that slide (Mistral v0.1, Gemma 2/3, Cohere 2).
attn_logit_softcapping · final_logit_softcappingattention softcap · a softcap op on the logits (Gemma 2).
query_pre_attn_scalar · attention_multiplierattention scale = value-0.5 (Gemma) · the multiplier (Granite).
ALiBiattention alibi: true (BLOOM, Baichuan-13B) or alibiMax: 8 (MPT).
q_norm / k_norm per headrmsnorm groupSize: head_dim (Qwen3, Gemma 3).
Gemma norms and embeddingsrmsnorm weightOffset: 1; scale the embeddings by √hidden_size.
hidden_actgelu → gelu_exact; gelu_new, gelu_pytorch_tanh → gelu; quick_gelu → gelu_quick; Gemma's MLP → geglu.
layer_norm_epslayernorm eps.
Mixtralmoe with scoring: "softmax", normalize: true.
Qwen2 / Qwen3-MoE, OLMoEmoe softmax, normalize = norm_topk_prob. Qwen's shared expert is an ordinary MLP scaled by sigmoid of its gate.
DeepSeek-V2 / V3moe with scoring: "sigmoid" (V3), selectionBias = e_score_correction_bias, nGroup/topKGroup, normalize, scale = routed_scaling_factor. MLA attention uses vHeadDim and feature-axis slices.

Tensors stored as f64, int64 or bool — buffers such as position_ids and masks that many checkpoints carry — are converted to f32 as they stream, so loading a whole checkpoint no longer fails over them.

Still refused: quantised checkpoints (GPTQ, AWQ, bitsandbytes, int8, fp8) — there is no dequantisation path, and guessing one would give plausible wrong numbers. The error says so and names the tensors.

Sessions and the KV cache

A session is a named tensor namespace pinned to one worker. Tensors named in retain stay there under a name of your choosing instead of being returned, which is what makes a decode step one round trip: the cache never travels.

POST /api/v3/sessions        {"modelId":"m_..."}  -> { "session": { "sessionId": "s_..." } }
GET  /api/v3/sessions/<id>                        -> resident tensors, bytes, step count
DELETE /api/v3/sessions/<id>                      -> releases everything it holds

Allocate the cache once, in the network:

{"graph":{"session":"s_...","inputs":[],
  "steps":[{"op":"zeros","in":[],"out":"kc","params":{"shape":[2048,1024]}},
           {"op":"zeros","in":[],"out":"vc","params":{"shape":[2048,1024]}}],
  "retain":{"kcache":"kc","vcache":"vc"}}}

Then each token writes into it at its position and attends over it. cache_write mutates in place, so the cache is one allocation no matter how long the context grows:

{"graph":{"session":"s_...",
  "inputs":[{"name":"x","shape":[1,1024],"dataB64":"..."},
            {"name":"kc","session":"kcache"},
            {"name":"vc","session":"vcache"},
            {"name":"wk","model":"m_...","weight":"...k_proj.weight"}],
  "steps":[
    {"op":"matmul",     "in":["x","wk"],   "out":"k0"},
    {"op":"rope",       "in":["k0"],       "out":"kr", "params":{"headDim":128,"offset":41}},
    {"op":"cache_write","in":["kc","kr"],  "out":"kc2","params":{"offset":41}},
    {"op":"attention",  "in":["q","kc2","vc2"], "out":"a",
       "params":{"heads":16,"headDim":128,"kvHeads":8,"causal":true,"posOffset":41}}],
  "outputs":["a"],
  "retain":{"kcache":"kc2","vcache":"vc2"}}}

You do not need retain for a cache. cache_write mutates the tensor the session already holds, so the session still points at it afterwards — listing all 56 cache tensors in retain on every call achieves nothing. Use retain only for a value that is genuinely a new tensor you want to keep.

Several sessions in one graph

Decoding several independent conversations on the same model can share one pass: the weights are read once for all of them. A graph may list more than one session, all pinned to the same worker, and an input may name which one it reads:

{"graph":{"session":"s_a", "sessions":["s_a","s_b","s_c"],
  "inputs":[{"name":"kc_a","session":"kcache","of":0},
            {"name":"kc_b","session":"kcache","of":1}, ...],
  ...}}

session is the primary one (of counts from 0 along sessions; an input without of reads the primary), at most 64 are listed, and a session that lives on a different worker is refused. The graph runs there and cache_write updates each cache in place, so each conversation's tokens are those it would have had alone. The chat page does this for conversations that happen to decode at the same time.

Surviving a lost worker

A session's cache is mutated in place, so unlike an ordinary tensor it cannot be rebuilt from lineage. By default, losing the worker holding it means the next call fails and you must re-prefill. Checkpointing avoids that:

POST /api/v3/sessions/<id>/checkpoint   -> { "tensors": 56, "bytes": 469762048 }
POST /api/v3/sessions/<id>/restore      -> puts it back on another volunteer

With a checkpoint in place, a disconnect is handled automatically — the coordinator restores onto another volunteer and the next call proceeds. Worth doing once after prefilling a long prompt.

Checkpointing is opt-in, and it is the one place the coordinator deliberately holds bulk data. A 28-layer KV cache at 2048 context is several hundred megabytes; holding that for every session by default would make the coordinator the data plane again. The durable budget is checked before anything is fetched, and a failed checkpoint never destroys a good one. Model weights need none of this — those are re-fetched from the model host.

Short-conv blocks

conv1d is the depthwise causal convolution used by hybrid conv/attention stacks — LFM2, and Mamba- or Hyena-style gated convs. It is deliberately pure: the state it reads is not modified. Carrying the state forward uses ops that already exist, in the same graph, so a decode step is still one round trip:

{"op":"conv1d",     "in":["x","w","st"],  "out":"y"},
{"op":"slice",      "in":["st"],          "out":"tail","params":{"start":1,"length":K-2}},
{"op":"concat",     "in":["tail","x"],    "out":"next"},
{"op":"cache_write","in":["st","next"],   "out":"st2", "params":{"offset":0}}

Prefilling T ≥ K-1 rows at once needs only slice(x, T-K+1, K-1) for the new state. Two extra ops rather than a second in-place mechanism to get right — and cache_write is the one already carrying every KV cache.

A fused projection splits with the ops you have. slice accepts a model weight directly, so an LFM2 conv.in_proj [3C, C] or a Phi-3 qkv_proj becomes three parts on axis 0, each transposed, with no new ops:

{"op":"slice",    "in":["Win"], "out":"Wb","params":{"start":0,   "length":C}},
{"op":"slice",    "in":["Win"], "out":"Wg","params":{"start":C,   "length":C}},
{"op":"slice",    "in":["Win"], "out":"Wx","params":{"start":2*C, "length":C}},
{"op":"transpose","in":["Wb"],  "out":"Wbt"},
{"op":"matmul",   "in":["x","Wbt"], "out":"B"}

Partial rotary embeddings

Because slice and concat work on the feature axis, a head can be cut apart, partially rotated and put back — which is what DeepSeek-style low-rank attention needs, since it ropes only the last 64 of each 192-wide query head:

{"op":"reshape","in":["q"],  "out":"qf","params":{"shape":[T*heads,192]}},
{"op":"slice",  "in":["qf"], "out":"qn","params":{"start":0,  "length":128,"axis":1}},
{"op":"slice",  "in":["qf"], "out":"qp","params":{"start":128,"length":64, "axis":1}},
{"op":"rope",   "in":["qp"], "out":"qr","params":{"headDim":64,"offset":41}},
{"op":"concat", "in":["qn","qr"],"out":"qc","params":{"axis":1}},
{"op":"reshape","in":["qc"], "out":"qq","params":{"shape":[T,heads*192]}}

Six ops, one graph, one round trip. Pair it with vHeadDim on attention when the value width differs from the query/key width, and the whole MLA block is expressible with no operation specific to it.

The fused block

short_conv does the whole thing in one step — nine ops down to one, which in a 16-layer model with ten conv layers is eighty fewer steps for the worker to dispatch:

{"op":"short_conv","in":["x","w_in","w_conv","w_out","state"],"out":"y"}
InputShape
x[T, C]the layer input
w_in[3C, C]in_proj as stored; transposed on the device
w_conv[C, 1, K]as stored, no reshape needed
w_out[C, C]out_proj as stored
state[K-1, C]the last K-1 rows of Bx; advanced in place

The computation, in order:

BCx  = in_proj(x)      # [T, 3C], chunked in order B, C, x
Bx   = B * x           # gate BEFORE the convolution
conv = conv1d(Bx)      # depthwise causal, over the carried state
y    = C * conv        # gate AFTER  the convolution
out  = out_proj(y)
Two things here are easy to get backwards and produce a plausible output rather than an error if you do. The two gates sit on opposite sides of the convolution — B multiplies the input, C multiplies the output. And the tensor that is convolved and cached is Bx, the gated third chunk — not the layer input. That is why this is one op with the order baked in rather than a recipe for each caller to reassemble.

Because state is written in place, a session carrying it needs no retain clause, and it stays one allocation however long the sequence runs.

Compiled graphs

A decode step for a 28-layer model is around 650 operations. Sent inline that is roughly 78 KB of JSON per token to move four bytes of payload — re-serialised, re-uploaded, re-parsed and re-validated every time. Compile it once instead:

POST /api/v3/graphs
{"label":"decode","graph":{
   "inputs":[
     {"name":"ids","shape":[1],"bind":"ids"},           # contents vary per call
     {"name":"wq","model":"m_...","weight":"...q_proj.weight"}],
   "steps":[
     {"op":"rope","in":["q0"],"out":"q",
        "params":{"headDim":128,"offset":{"$bind":"pos"}}},   # value varies per call
     ...],
   "outputs":["tok"]}}

# { "graphId":"gt_...", "steps":653, "bindings":["ids","pos"],
#   "bodyBytes":79417, "perCallBytes":88, "savedPerCall":79329 }

Then each call carries only what changed:

POST /api/v3/graph
{"graphId":"gt_...","session":"s_...","bind":{"ids":[4097],"pos":41}}

Two kinds of hole can be left in a template. An input with "bind" keeps a fixed shape and takes new contents per call. A parameter written {"$bind":"pos"} takes a number per call, optionally with "add" or "mul" applied. Everything else — weights, session tensors, the step list — is fixed at compile time, because that is the part that never changes between tokens.

Because every shape is known at compile time, a template that compiles will not fail shape validation later. Validation happens once per template rather than once per token.

Server-side decoding

This is the largest single win available. Generating 64 tokens by calling the graph endpoint 64 times is 64 crossings of the open internet, and the latency of each dwarfs the arithmetic it carries. POST /api/v3/generate runs the loop next to the worker and streams the tokens back over one request.

POST /api/v3/generate
{"graphId":"gt_...","sessionId":"s_...",
 "ids":[9707],            # the last prompt token
 "startPos":41,           # where the prompt ended
 "maxNewTokens":64,
 "stopIds":[151645],
 "temperature":0.8,"topK":40,"seed":7}

event: token
data: {"token":3838,"index":0,"pos":41,"ms":38}

event: token
data: {"token":11,"index":1,"pos":42,"ms":36}
...
event: done
data: {"ok":true,"tokens":[...],"stopped":"stop token",
       "elapsedMs":2410,"tokensPerSecond":26.6,"roundTrips":1,"workerLoop":true,"nextPos":105}

The compiled graph must bind both ids and pos, and must end in a single token id from argmax or sample. Sampling parameters passed here override whatever the template baked in, so one compiled graph serves greedy and sampled decoding alike; step is supplied automatically so a seeded run is reproducible.

Prefill is your own single call — run the prompt through the graph once, then start generating from the last token with startPos set to the prompt length. Pass "stream": false to get the finished list in one JSON body instead of an event stream.

With a session, the loop can run on the volunteer itself. When the session's worker holds everything the graph reads and supports it (the downloadable volunteer program does), every token after the first is produced there and streamed back, with no round trip through the grid per token — typically several times faster. workerLoop in the result says whether that happened. The tokens, the cache writes and nextPos are exactly what the ordinary path produces; pass "workerLoop": false to insist on the ordinary path. It applies to graphs whose only bindings are ids, pos, step and sampling parameters, whose inputs are model weights or session tensors, and that retain nothing.

Speculative decoding

Each pass can also carry a few drafted tokens, checked in the same pass. The drafts are not from a second model: they are the continuation of the most recent earlier occurrence, in the context so far, of the last one to four tokens. The pass is the decode step over several positions, each sampled exactly as the one-token step would sample it (same seed, step + row); the longest agreeing prefix plus the first disagreeing row is kept. A rejected draft costs nothing but the rows it took, so greedy and seeded output are the tokens plain decoding gives, only sooner when the text repeats (code, quotations, structured output). How many tokens to draft is chosen per pass from how often drafts have been right and how much longer a longer pass takes.

To use it with /api/v3/generate, compile verify graphs yourself: the decode step over k+1 positions, ids bound with shape [k+1], every position sampled with {"$bind":"step","add":row} in its sample step, ending in the k+1 token ids. Then:

{"graphId":"gt_...", "sessionId":"s_...", "ids":[...prompt tokens...], "startPos":41, "maxNewTokens":64,
 "speculative":{"graphIds":["gt_verify4","gt_verify2"], "context":[...token ids so far...]}}

event: done
data: {..., "speculation":{"verifies":14,"drafted":47,"accepted":31,"produced":45,"plain":9}}

Up to 16 verify graphs; context defaults to ids; "always": true drafts on every pass instead of letting the scheduler back off. Each verify graph must bind ids (two or more positions) and pos. speculation in the result counts verify passes, drafted and accepted tokens, tokens produced and plain steps. When the session's worker supports the worker-side loop, it drafts and verifies itself, with no round trip in between. The chat page does all of this by itself and is where it is most useful; "speculative": false in a chat message turns it off.

Small inputs (a step's token ids, a stage's activations) are sent together with the work that reads them instead of ahead of it, which removes a round trip per step on volunteers that keep messages in order.

Python

clients/mpakt_grid.py is dependency-free and builds the graph for you. Account covers sign-up and keys.

from mpakt_grid import Account, VirtualGPU, Graph

acct = Account("https://YOUR-HOST")
acct.login("alice", "a long password")       # once the account is approved
gpu = VirtualGPU("https://YOUR-HOST", key=acct.create_key("script")["key"])
gpu.wait_for_workers(2)

model = gpu.load_model("hf:Qwen/Qwen3-0.6B",
                       include=["model.layers.26.*", "model.layers.27.*"])
print(f"{model.bytes_transferred/1e6:.0f} MB into the network, 0 bytes uploaded")

MAXLEN, KVW = 2048, 1024
with gpu.session(model, label="chat") as s:
    g = Graph()
    kc, vc = g.zeros([MAXLEN, KVW]), g.zeros([MAXLEN, KVW])
    s.run(g, retain={"kcache": kc, "vcache": vc})

    for pos, hidden in enumerate(stream_of_hidden_states):
        g = Graph()
        x  = g.input("x", [1, 1024], hidden)
        kc = g.session_tensor("kcache", [MAXLEN, KVW])
        vc = g.session_tensor("vcache", [MAXLEN, KVW])
        w  = {k: g.weight(model, f"model.layers.26.{n}")
              for k, n in LAYER_WEIGHTS.items()}
        y, kc, vc = g.decoder_layer(x, w, heads=16, head_dim=128, kv_heads=8,
                                    pos_offset=pos, kcache=kc, vcache=vc)
        out = s.run(g, outputs=[y], retain={"kcache": kc, "vcache": vc})
        hidden = out[y.ref]          # one round trip for the whole layer

That is still one request per token. A real generation loop compiles the graph and lets the server run the loop:

from mpakt_grid import VirtualGPU, Graph, bind

g   = Graph()
ids = g.bound_input("ids", [1], "ids")
h   = g.embedding(ids, g.weight(model, "model.embed_tokens.weight"))
for L in range(n_layers):
    h, kc, vc = g.decoder_layer(h, weights[L], heads=16, head_dim=128, kv_heads=8,
                                pos_offset=bind("pos"),
                                kcache=g.session_tensor(f"k{L}", [2048, 1024]),
                                vcache=g.session_tensor(f"v{L}", [2048, 1024]))
tok = g.sample(g.matmul(h, lm_head), temperature=0.8, top_k=40, seed=7)

decode = gpu.compile(g, outputs=[tok], label="decode")
print(f"{decode.body_bytes} bytes -> {decode.per_call_bytes} per call")

with gpu.session(model) as s:
    for token in s.generate(decode, [last_prompt_token],
                            max_new_tokens=64, start_pos=len(prompt),
                            stop_ids=[eos], temperature=0.8, top_k=40, seed=7):
        print(tokenizer.decode([token]), end="", flush=True)

The client keeps one connection open across calls. Opening a fresh one per request costs a TCP and, over HTTPS, a TLS handshake each time — around 110 ms per call through a CDN, which for anything token-by-token is most of the wall clock.

Endpoint reference

EndpointPurpose
GET /api/versionsWhat versions exist and what each supports. No key required.
GET /api/v3/statusWorkers, backends, ops, and free bytes — check before loading.
GET /api/v3/models/inspect?source=…Tensor index only; loads nothing.
POST /api/v3/modelsStream a model into the network.
GET /api/v3/modelsYour loaded models and network capacity.
DELETE /api/v3/models/:idRelease a model everywhere.
POST /api/v3/sessionsOpen a session.
GET /api/v3/sessions/:idIts resident tensors and step count.
POST /api/v3/sessions/:id/checkpointCopy its tensors here so a lost worker is survivable.
POST /api/v3/sessions/:id/restorePut a checkpointed session back on a volunteer.
DELETE /api/v3/sessions/:idClose it and free what it holds.
POST /api/v3/graphRun a graph — inline, or a compiled one with graphId and bind.
POST /api/v3/graphsCompile a graph; returns a graphId.
GET /api/v3/graphsYour compiled graphs and their bindings.
DELETE /api/v3/graphs/:idDelete one.
POST /api/v3/generateDecode on the server; streams tokens over SSE.
GET /api/v3/randomTrue random integers: min, max, count (up to 10000). Default range 84 – 2147483646, the one generation seeds come from.

Limits: 4 models, 16 sessions and 32 compiled graphs per key. Graphs time out after 5 minutes; a generation is capped at 4096 new tokens.

All versions share one key. Keys idle for over 30 days are deactivated — request or revoke one here.