Layer-Aligned Key-Value (KV) Cache Streaming over Media over QUIC Transport (MOQT)

1.1. Motivation

Inference systems increasingly treat the KV cache as state that is decoupled from the engine that computed it: a prefill instance or cache server produces it, and decode instances, other replicas, or other sites consume it. The same prefix (for example a long system prompt or a shared document) is often needed by many consumers. Point-to-point transfer from the producer to each consumer wastes backbone capacity and loads the producer.¶

MOQT [I-D.ietf-moq-transport] already solves the analogous problem for media: an original publisher sends objects once to a relay tree, relays cache them, and many subscribers receive them with per-object priorities, without the relays understanding the payload. A KV cache has a natural hierarchy that fits the MOQT object model: a cache belongs to a prefix, consists of key and value tensors, is computed layer by layer, and is divided into blocks of tokens.¶

1.2. Summary of the Profile

Track Namespace  ("kvcs-1", scope_id, compat_id, anchor_key, pub_id)
 |
 +-- Track "manifest"        Group m: manifest version m
 |
 +-- Track "k.raw.s0"        key tensors, codec "raw", segment 0
 |    +-- Group 0            layer 0
 |    |    +-- Object 0      tokens of block 0, layer 0
 |    |    +-- Object 1      tokens of block 1, layer 0
 |    |    +-- ...
 |    +-- Group 1            layer 1
 |    +-- ...
 |    +-- Group L-1          layer L-1
 |
 +-- Track "v.raw.s0"        value tensors, same structure
 +-- Track "kv.<codec>.s0"   jointly encoded K and V (optional)
 +-- Track "k.raw.s1" ...    segment 1 (appended tokens)

The design choices are:¶

• Namespace from identity, not from text. The namespace carries the keyed identifiers of the KV Cache Storage Protocol (KVSP) [I-D.li-kvcache-efficient-storage], so relays and observers learn neither the model name nor the prompt, and two different prompts share a namespace only with negligible probability (Section 3.1).¶
• Component tracks. Key, value, and jointly encoded tensors are separate tracks, so a consumer can subscribe to what it needs and a publisher can offer several encodings (Section 3.2).¶
• Layer-aligned groups. A Group is one layer. Production proceeds layer by layer, so Group IDs increase over time as MOQT expects, and a consumer can compute layer l while layer l+1 is still arriving (Section 3.3).¶
• Block objects. An Object is the tensor of one token block in one layer, the smallest unit that is cached by relays and that can be selected with object filters (Section 3.4).¶
• Self-describing payloads. Every Object carries its own descriptor and digest, so a subscriber that joins mid-track can use every Object it receives (Section 3.5).¶

1.3. Relationship to Other Work

[I-D.shi-moq-kvcache] also describes KV cache transmission over MOQT. The present document differs in several respects that affect interoperability and security: (1) it does not place the model name or prompt in the namespace but keyed identifiers, which avoids disclosure and keeps different prompts in different tracks; (2) it aligns Groups with layers rather than with token chunks, so that consumers can pipeline computation across layers and Group IDs increase with production time; (3) it separates key, value, and jointly encoded tensors into tracks and identifies codecs through a registry instead of fixed compression levels; and (4) it defines the object payload, a manifest, integrity checks, and retrieval procedures for both live and completed caches.¶

KVCS depends on KVSP for the Compatibility ID, Scope ID, Block Keys, and the codec registry. Object payload protection uses [I-D.ietf-moq-secure-objects] when relays are not trusted with KV cache contents.¶

3.1. Track Namespace

A KVCS Track Namespace consists of exactly five Track Namespace Fields:¶

Fields 1 to 4 are raw octets, not their hexadecimal encodings. The Publication ID is chosen at random by the Original Publisher for each production of the cache.¶

Fields 0 to 3 identify content: any consumer that knows the token prefix, the scope key, and the Compatibility Descriptor can compute them. Field 4 is needed because MOQT requires that an Object have identical bytes regardless of where it is retrieved (Section 2.1 of [I-D.ietf-moq-transport]), while two independent computations of the same prefix are not guaranteed to be bit-identical on all hardware. A consumer that knows fields 0 to 3 discovers the available publications through namespace prefix subscription (Section 5.1).¶

Namespace prefix matching gives useful aggregation points: the prefix ("kvcs-1", Scope ID) covers all publications of a scope and is the natural unit of authorization; the prefix ("kvcs-1", Scope ID, Compatibility ID, Anchor Key) covers all publications of one token prefix.¶

3.2. Track Names

Track Names are ASCII strings with the following syntax, in ABNF [RFC5234]:¶

track-name = "manifest" / "manifest.cose" / data-track
data-track = component "." codec-name "." "s" segment
component  = "k" / "v" / "kv"
codec-name = 1*32( %x61-7A / DIGIT / "-" )   ; KVSP codec name
segment    = "0" / ( %x31-39 *DIGIT )        ; no leading zeros

codec-name is the Name of a codec in the "KVSP Codecs" registry [I-D.li-kvcache-efficient-storage]. For example, "k.raw.s0" carries the uncompressed key tensors of segment 0. A publisher MAY offer the same segment in several encodings, for example "k.raw.s0" and "v.raw.s0" together with a "kv" track using a registered compressing codec; each data track is complete on its own. A consumer selects either the pair of "k" and "v" tracks of one codec or one "kv" track.¶

3.3. Groups

The Group ID of a data track is the layer index, from 0 to L-1, where L is the number of layers in the Compatibility Descriptor. The Original Publisher MUST publish the Objects of Group g before those of Group g+1 in the same track. Producers that compute the cache in several passes over the token sequence (for example chunked prefill, where each chunk passes through all layers) MUST publish each pass as a separate segment, so that Group IDs within every track increase with time as recommended by Section 2.3.1 of [I-D.ietf-moq-transport].¶

The Objects of a Group are independently usable: each carries the tensor of one block for that layer. A Group is a natural join point because a consumer needs every block of a layer before computing attention for that layer.¶

The Original Publisher sends all Objects of a Group in a single Subgroup with Subgroup ID 0, in ascending Object ID order. After the last Object of Group g it sends an Object with status End of Group. After the last Group of a track it sends an Object with status End of Track.¶

3.4. Objects

The Object ID is the index of the block within the segment, starting at 0. The absolute block index is the first block index of the segment, given in the manifest, plus the Object ID. Each Object carries one KVCS payload (Section 3.5). Objects are never modified; a publisher that discovers an error in a publication stops it and starts a new publication with a new Publication ID.¶

The Original Publisher SHOULD set the Publisher Priority of the Objects in Group g to floor(128 * g / L), so that, under contention, earlier layers, which a consumer needs first, are sent first. The manifest track uses Publisher Priority 0.¶

3.5. Object Payload

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Magic ("KVO1" = 0x4B564F31)                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Version    |   Component   |             Flags             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                             Layer                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Block Index                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Token Start (64 bits)                     |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Token Count                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             Codec             |           Reserved            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Data Length                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Data Digest (SHA-256, 32 octets)             |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Data (variable)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Magic:

0x4B564F31.¶

Version:

1.¶

Component:

1 for "k", 2 for "v", 3 for "kv".¶

Flags:

Reserved; set to zero and ignored on receipt.¶

Layer:

The layer index; equal to the Group ID.¶

Block Index:

The index of the block within the segment; equal to the Object ID.¶

Token Start:

The absolute position in the context of the first token of the block.¶

Token Count:

The number of tokens; equal to the block size of the Compatibility Descriptor.¶

Codec:

A value from the KVSP Codecs registry.¶

Data Length:

The length of Data in octets.¶

Data Digest:

SHA-256 [RFC6234] of Data.¶

Data:

The tensor of the block for this layer. For codec "raw" and component "k" or "v", this is the K or V section of the KVSP container for one layer; for component "kv" the K section is followed by the V section.¶

For codec "raw", the SHA-256 of the Data of a "k" Object followed by the Data of the "v" Object at the same Location, or the Data Digest of the corresponding "kv" Object, equals the KVSP layer digest of that block and layer, so that the two documents protect the same content.¶

The header is 68 octets. An End Subscriber MUST verify that Magic and Version are as specified, that Layer equals the Group ID, that Block Index equals the Object ID, that the header and Data Length account for the whole payload, and that the Data Digest matches. An Object that fails any check MUST be discarded and treated as missing.¶

3.6. Manifest Track

The manifest track describes the publication. Each version of the manifest is Object 0 of a new Group, with Group IDs increasing by one per version. A publication has exactly one manifest track: "manifest", whose payload is a JSON [RFC8259] object, or "manifest.cose", whose payload is a COSE_Sign1 structure [RFC9052] with that JSON object as its payload. The JSON object is:¶

{
  "kvcs": 1,
  "compat": { ...KVSP Compatibility Descriptor... },
  "layers": 32,
  "block_tokens": 16,
  "deterministic": false,
  "segments": [
    { "segment": 0, "first_block": 0, "blocks": 64,
      "block_keys_sha256": "<hex32>",
      "tracks": ["k.raw.s0", "v.raw.s0"],
      "complete": true },
    { "segment": 1, "first_block": 64, "blocks": 3,
      "block_keys_sha256": "<hex32>",
      "tracks": ["k.raw.s1", "v.raw.s1"],
      "complete": false }
  ]
}

"block_keys_sha256" is the SHA-256 of the concatenated raw Block Keys of the segment, which lets a consumer that has computed the keys confirm that the segment covers its tokens. A segment entry SHOULD also carry "object_digests": for each data track, an array with one element per completed Group, in Group order, each element being the array of the hex Data Digests of the Group's Objects in Object order. During production, the publisher SHOULD publish a new manifest version at least every eight completed layers, so that a consumer can verify layers while later ones are still produced. Because every Object has its own listed digest, consumers can also verify partial retrievals (a range of layers or of blocks).¶

With "manifest.cose", the publisher signs each version with its producer key, using the COSE algorithm ES256 or EdDSA [RFC9053], and places the key identifier in the "kid" header parameter. Producer keys are distributed to the members of the scope together with the scope key, by means outside this document. "deterministic" is true only if the publisher guarantees that any publication of the same content fields produces bit-identical Objects. A new manifest version is published whenever a segment is added or becomes complete. The manifest MUST be consistent with the Compatibility ID in the namespace: a consumer MUST compute the Compatibility ID of "compat" and discard the publication if it differs.¶

5.1. Discovering Publications

A consumer computes the Compatibility ID, Scope ID, and Block Keys of its prompt with KVSP. For the longest prefix it wants to reuse, it uses the Block Key of that prefix's last block as the Anchor Key and sends SUBSCRIBE_NAMESPACE for the prefix ("kvcs-1", Scope ID, Compatibility ID, Anchor Key). Each NAMESPACE response identifies one publication. Alternatively, a KVSP Index Service lookup can return replica records whose holder URI is an MOQT URI identifying the full namespace. If no publication exists for the full prefix, the consumer can try shorter prefixes, because the Anchor Key of a shorter prefix is an earlier Block Key in the same chain. A publication whose prefix is longer than the consumer's needs cannot be found this way, because the consumer cannot compute its Anchor Key; the consumer finds such publications through a KVSP Index Service lookup of its own Block Keys, whose replica records name the publication.¶

5.2. Retrieval during Production

To receive a cache that is still being produced, and to receive every Object exactly once, the consumer subscribes to each needed data track with:¶

• a LOCATION_FILTER of type Next Object (0x05); and¶
• a FILL_PARAMETERS parameter containing a LOCATION_FILTER of type Absolute Start (0x02) with StartGroup 0 and StartObject 0, which retrieves everything published before the subscription on a fill fetch stream.¶

This combination is the exactly-once pattern described in Section 3.4 of [I-D.ietf-moq-transport]. The consumer also subscribes to the manifest track: first to "manifest.cose", and, only if the publisher reports that this track does not exist and the consumer does not require producer authenticity, to "manifest". It uses a LOCATION_FILTER of type Next Object (0x05) and a FILL_PARAMETERS parameter containing a LOCATION_FILTER of type Relative Start (0x01) with StartGroup 1, which delivers the current manifest on the fill fetch stream and later versions on the subscription. A data track is complete for the consumer when it has received End of Track, or PUBLISH_DONE indicating that the track ended, or when the fill fetch stream has ended with a FIN and every Group listed in a manifest that marks the segment complete has been received. Object Status values are not carried on fetch streams, so the last condition is the one that applies to a consumer that subscribes after production has finished.¶

5.3. Retrieval of Completed Caches

To retrieve layers a through b of a completed segment, the consumer sends FETCH for each needed data track with a LOCATION_FILTER of type Absolute Start, Group End (0x03) with StartGroup a, StartObject 0, and EndGroupDelta b-a, and with GROUP_ORDER Ascending. The publisher or relay delivers Groups in ascending order, so the consumer can start computing layer a while later layers arrive.¶

A consumer that needs only the first n blocks of a segment, because its prompt diverges after them, adds an OBJECTID_FILTER with the range 0 to n-1 when the peer's MAX_FILTER_RANGES setup option is non-zero; otherwise it discards the excess Objects.¶

5.4. Verification and Recovery

The consumer verifies every Object as described in Section 3.5 and every manifest as described in Section 3.6. A layer is usable when all Objects that the consumer needs in the corresponding Group, in all selected tracks, have been received and verified. The consumer MUST NOT combine Objects of different publications in one segment unless the manifests of both publications state "deterministic": true.¶

If Objects are missing (for example, reported as an unknown range in a fetch stream, or the stream is reset), the consumer retries with another relay or publication, retrieves the blocks with KVSP over HTTP, or recomputes the affected tokens. It does not wait indefinitely: it SHOULD bound recovery by the time that recomputation would take.¶