Observe a pipeline¶
A pipeline reports at three levels: the AnkkaFlow's status says whether the whole pipeline runs, the
events on it say what the operator did and why, and each pod's sidecar says what one streamlet is doing
through its log and its metrics.
The phase¶
kubectl -n shop get aflow
kubectl -n shop get aflow cart -o wide -w
NAME PIPELINE PHASE DETAIL AGE
cart cart Ready 4m
| phase | meaning |
|---|---|
Pending |
a streamlet is still rolling out |
Ready |
every streamlet has all its pods ready and every topic exists |
Degraded |
an unmanaged topic is missing or Kafka is unreachable, or a streamlet has fewer ready pods than desired |
Failed |
the operator refused the resource and applied nothing; DETAIL lists why |
DETAIL joins the reasons with ;, such as topic 'shop.cart-events.v1' does not exist or
router: 1 of 3 ready. The full status, per streamlet and per topic:
kubectl -n shop get aflow cart -o jsonpath='{.status}' | jq .
A streamlet's pod is ready when its sidecar is: a conversation with the process is running and every inlet is subscribed to a topic that exists. The process container has no probes of its own.
Events¶
The operator records what it did on the AnkkaFlow:
kubectl -n shop get events --field-selector involvedObject.kind=AnkkaFlow
TopicCreated, StreamletRolled, StreamletRemoved and ResetOffsets are Normal. Every Warning names
something to act on; each is described in Troubleshooting. Stalled partitions are
recorded by the sidecar on its own pod:
kubectl -n shop get events --field-selector reason=PartitionStalled
Logs¶
Every pod has two containers: sidecar and process.
kubectl -n shop logs deploy/flow-cart-router -c sidecar
kubectl -n shop logs deploy/flow-cart-router -c process
The sidecar logs discovery, each conversation, partitions assigned and revoked, every stream failure with its cause, and every reconnect. When the sidecar refuses to start because the process does not match the deployed descriptor, the problems are in both logs: the sidecar sends them to the process before it exits. No record value is ever logged.
Lag and throughput¶
Each sidecar serves Prometheus metrics on port 2050. Every Kafka client's id is
<pipeline>.<streamlet>.<port>, so lag is attributed to one streamlet's inlet:
kubectl -n shop port-forward deploy/flow-cart-router 2050 &
curl -s localhost:2050/metrics | grep records_lag
kafka_consumer_consumer_fetch_manager_metrics_records_lag{client_id="cart.router.in",partition="0",topic="shop_cart-events_v1"} 0.0
Kafka writes topic names in these labels with dots replaced by underscores. The metrics worth watching:
| metric | watch for |
|---|---|
kafka_consumer_consumer_fetch_manager_metrics_records_lag |
lag per inlet partition that grows and does not fall |
kafka_consumer_consumer_fetch_manager_metrics_records_consumed_rate |
an inlet that stops reading |
kafka_producer_producer_metrics_record_send_rate |
emits per outlet |
ankka_flow_sidecar_stalled_seconds |
a partition that has not committed for a long time |
ankka_flow_sidecar_in_flight |
a batch that stays with the process |
Every metric is listed in the sidecar reference.
Scraping with Prometheus¶
Streamlet pods carry prometheus.io/scrape: "true" and prometheus.io/port: "2050", and the port is
named metrics. A Prometheus that discovers pods by those annotations scrapes every sidecar without
further configuration. With the Prometheus Operator, a PodMonitor selecting
app.kubernetes.io/managed-by: ankka-flow on port metrics does the same:
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: ankka-flow
namespace: shop
spec:
selector:
matchLabels:
app.kubernetes.io/managed-by: ankka-flow
podMetricsEndpoints:
- port: metrics
Pods also carry flow.ankka.thinkmorestupidless.com/pipeline and
flow.ankka.thinkmorestupidless.com/streamlet labels, which a relabelling rule can turn into series
labels.
Stalled partitions¶
A batch the process fails is redelivered from the last commit, indefinitely; nothing is skipped. A batch
that fails every time therefore stalls its partition. It shows three ways: that partition's lag grows,
ankka_flow_sidecar_stalled_seconds rises, and after FLOW_STALL_WARNING_AFTER (five minutes by default)
the sidecar records one PartitionStalled Warning on its pod, naming the inlet, the partition and the
last error. The fix is in the streamlet's code: skip the record by acknowledging without emitting, or
correct what makes it fail. See Delivery and failure.
The stall is measured from the first attempt at the batch, across the sidecar's reconnects, so the warning appears even though every failure tears the stream down and starts it again.
A built-in stage¶
A built-in streamlet's pod, such as the Neo4j merge sink, exports every inlet metric, plus four counters of its own per inlet partition:
| Metric | Meaning |
|---|---|
ankka_flow_stage_deltas_written_total |
deltas applied to the graph |
ankka_flow_stage_deltas_stale_total |
deltas found stale: folded away within a batch, or older than what the graph holds |
ankka_flow_stage_batches_failed_total |
batches whose transaction failed or whose records could not be read |
ankka_flow_stage_delete_markers_total |
records with no value, which remove a key from a compacted topic, passed over |
A rising written count with lag near zero is a healthy sink. A replay from the start, or a burst of
redelivery, shows as stale deltas rather than written ones: nothing in the graph changed. Failed
batches beside growing lag mean the database is refusing or not answering, or a delta is unreadable or
wrongly keyed; the sidecar's log and the PartitionStalled note carry the reason. Delete markers are
counted on their own and are never a failure. During a
rebuild from the delta topic the written count rises until the lag is zero.
A sink whose credentials may not create the uniqueness constraint it needs records one
ConstraintNotCreated Warning on its pod each time it opens, and keeps going. Merges then scan the
graph instead of looking elements up, so throughput falls as the graph grows. Create the constraint
with an account that may:
CREATE CONSTRAINT element_id IF NOT EXISTS FOR (n:Element) REQUIRE n.id IS UNIQUE