Graph deltas¶
A graph delta is one JSON object per Kafka record, under the contract ankka.graph-delta.v1. It says
what one element of a graph — a node or an edge — looks like now, or that it is gone. The
Neo4j merge sink reads deltas and merges them into a graph; a streamlet in any
language writes them, usually by mapping a service's events.
In the Python SDK an outlet of deltas is a GraphDeltaOutlet, which owns the contract and builds each
record with its key; see Python SDK:
deltas = GraphDeltaOutlet("deltas")
In any other language, the contract is {"format": "json", "schema_name": "ankka.graph-delta.v1"}
with the fingerprint Base64(SHA-256("ankka.graph-delta.v1")), and the writer sets each record's key
itself, by the rule in The record key.
The three kinds¶
A node merge carries the node's labels and properties:
{"kind": "node", "id": "cart:cart-1", "version": 1790627790360,
"labels": ["Cart"], "properties": {"cartId": "cart-1"}}
An edge merge carries its type, its endpoints and its properties. Its direction is from → to:
{"kind": "edge", "id": "checked-out:cart-1:1790627790360", "version": 1790627790360,
"type": "CHECKED_OUT", "from": "cart:cart-1", "to": "checkout:cart-1:1790627790360",
"properties": {}}
A tombstone marks an element deleted. An edge tombstone names the edge's type and endpoints, so the edge is found without scanning the graph:
{"kind": "tombstone", "element": "node", "id": "cart:cart-1", "version": 1790627800000}
{"kind": "tombstone", "element": "edge", "id": "checked-out:cart-1:1790627790360", "version": 1790627800000,
"type": "CHECKED_OUT", "from": "cart:cart-1", "to": "checkout:cart-1:1790627790360"}
The record key¶
A delta's Kafka record key is its element key: the element's kind and its id, as UTF-8 text.
| Delta | Key |
|---|---|
a node merge for id X |
node:X |
a tombstone with "element": "node" for id X |
node:X |
an edge merge for id X |
edge:X |
a tombstone with "element": "edge" for id X |
edge:X |
The id follows the first colon verbatim, so an id that contains colons is unambiguous: the node
cart:cart-1 has the key node:cart:cart-1. Two records have the same key exactly when they describe
the same element. Nodes and edges are separate id spaces, so a node and an edge that share an id have
different keys.
The key is a requirement, not advice. A delta topic is compacted by default (see Topics and Kafka clusters), and a compacted topic keeps the last record per key: a delta under any other key would replace, or be replaced by, another element's record. So the sink refuses a delta whose key is missing or is not its own element key, on every delta topic, compacted or not. The batch fails, nothing of it is written, and the message gives the key the delta should have had:
neo4j merge failed for inlet 'in' partition 1: offset 42: key 'cart-1' is not this delta's element key 'node:cart:cart-1'
neo4j merge failed for inlet 'in' partition 1: offset 43: no key; this delta's element key is 'edge:checked-out:cart-1:1790627790360'
protocol/fixtures/graph-deltas/keys.json
lists deltas with their keys, including an id with colons and one with non-ASCII characters; the sink
and the Python SDK are both tested against it.
deltas.json
beside it lists deltas with every kind of property value, each with its key and the kind the sink
reads each property as; a writer in another repository can test itself against both files.
Delete markers¶
A record with a key and no value is a delete marker: Kafka's way of removing a key from a compacted topic. It is not a delta. The sink applies nothing for it, counts it, and acknowledges it with its batch; a record with an empty value is read the same way. A marker never stalls a partition.
The platform writes no delete markers. A tombstone stays in the topic as its element's last record, so the topic is bounded by the elements ever created and a graph rebuilt from it has deletions marked. A writer that wants a tombstoned element's record gone from the topic sends the marker itself, under the element's key, after the tombstone. The graph that already holds the element is unchanged; once the broker has compacted the key away, a graph rebuilt from the topic no longer has it.
A marker written for an element that was never tombstoned is a mistake to avoid: the element is still live in the graph that was built as the records arrived, live in a rebuild until the broker compacts its key, and absent from every rebuild after that.
Fields¶
| Field | Type | On | Rule |
|---|---|---|---|
kind |
node, edge or tombstone |
all | anything else fails the batch |
id |
non-empty string | all | global; nodes and edges are separate id spaces |
version |
whole number ≥ 0, within 64 bits | all | rises with the source entity's own history |
labels |
array of identifiers | node | may be empty or absent |
type |
identifier | edge, edge tombstone | one type per edge |
from, to |
non-empty strings | edge, edge tombstone | node ids |
properties |
object | node, edge | may be empty or absent |
element |
node or edge |
tombstone | which id space the tombstone marks |
An identifier matches [A-Za-z_][A-Za-z0-9_]*. A property value is a string, a number, a boolean, or
a non-empty array of values of one of those kinds. A whole number becomes a 64-bit integer in the
graph and any other number a float. Top-level fields the contract does not name are ignored.
Rules a writer keeps¶
- State, not change. A node or edge delta carries the element's whole state. To remove a property, send the state without it; nothing means "add to" or "increment".
- One writer per element, and its version rises.
versionis the source entity's sequence number, or any integer that rises with that entity's history, such as an event time in milliseconds when one event per millisecond is guaranteed. Two source entities never write the same element id. - Ids are global and stable. Prefix them by kind of thing (
cart:,checkout:) so ids from different entities cannot collide. - Key the record by its element key,
node:<id>oredge:<id>. All deltas for one element then land on one partition and are applied in order, and compaction keeps the element's latest. An edge and its endpoints are usually on different partitions, and that is fine: the sink creates a placeholder for an endpoint it has not seen yet. - Property values are plain. No
null, no nested objects, no mixed arrays. The keysid,_versionand_deletedbelong to the sink. - A tombstone marks; it does not remove. The element stays, marked, with its labels and properties cleared, so a straggling older delta cannot bring it back. A tombstone's key is the key of the element it marks.
- A new contract version is a new schema name. A change a reader of
v1could not accept isankka.graph-delta.v2.
What the sink does with a delta¶
The sink applies a delta only when its version is greater than the version the graph holds for that
element; an equal or lower version is counted as stale and changes nothing. Within one batch it keeps
one delta per element — the highest version, the first on a tie. So a delta applied twice, or an old
one arriving after a newer one, leaves the graph as it was.
Validation¶
A record that breaks the contract fails its whole batch: the partition stalls, nothing in the batch is written, and the sink's log names the record's offset and the problem. Nothing is skipped; the fix is in the streamlet that wrote the record.
| Check | Message |
|---|---|
| the value is a JSON object | offset N: not a JSON object |
kind is present |
offset N: kind missing |
kind is known |
offset N: unknown kind 'x' |
id is a non-empty string |
offset N: id missing or empty |
version is a whole number ≥ 0 within 64 bits |
offset N: version is not a non-negative integer |
labels is an array of identifiers |
offset N: labels must be an array of identifiers |
an edge has type, from and to |
offset N: edge needs type, from and to |
| a tombstone names its element | offset N: tombstone needs element 'node' or 'edge' |
an edge tombstone has type, from and to |
offset N: tombstone of an edge needs type, from and to |
properties is an object |
offset N: properties must be an object |
| a property is a scalar or an array of one scalar kind | offset N: property 'p' is not a scalar or array of scalars |
| a property key is not the sink's | offset N: property 'p' is reserved |
| the record's key is the delta's element key | offset N: key 'k' is not this delta's element key 'node:x' |
| the record has a key | offset N: no key; this delta's element key is 'node:x' |
A record with no value, or an empty one, is not checked at all: it is a delete marker.
For a writer built before the key rule¶
The contract's name is unchanged, so a blueprint whose writer keys its deltas some other way — by the
bare id, or by its input's key — still passes flow verify. The sink refuses that writer's first
delta at run time, the partition stalls, and the message gives the key expected. A delta topic that
already holds records keyed the earlier way is refused at the first of them when read from the start.
To bring a pipeline across:
- Scale the writer and the sink to zero. Set
replicas = 0for both in a--conffile, generate and apply. - Delete the delta topic. The platform never alters or deletes a topic; it recreates a managed one on the next reconcile, compacted.
- Deploy the writer that keys its deltas
node:<id>andedge:<id>— in Python, one that declares aGraphDeltaOutlet. - Reset the pipeline:
flow reset <pipeline>. - Scale both back up. The writer re-emits its history under the new keys, and the sink finds what the graph already holds stale.