Blueprint¶

A blueprint is a HOCON file, conventionally blueprint.conf, that names the streamlets a pipeline uses and the topics that connect their ports. flow verify and flow generate read it together with a directory of streamlet descriptors.

blueprint {
  name = cart
  streamlets {
    router = cart-router
    sink   = sink
  }
  topics {
    cart-events {
      managed    = false
      topic.name = "shop.cart-events.v1"
      cluster    = shop
      consumers  = [router.in]
      consumer-config { auto.offset.reset = earliest }
    }
    valid-carts {
      producers  = [router.valid]
      consumers  = [sink.in]
      partitions = 6
      replicas   = 1
      topic { retention.ms = 86400000 }
    }
    review-carts {
      producers = [router.review]
    }
  }
}

`blueprint.name`¶

Optional. The pipeline id when flow generate is given no --pipeline; without either, the pipeline id is the blueprint file's name up to its first dot. A pipeline id is 1 to 40 characters of a-z, 0-9 and -, not starting or ending with -. It prefixes managed topics' Kafka names, consumer groups, client ids and the names of everything the operator creates.

`blueprint.streamlets`¶

Required, and must name at least one streamlet. Each key is a streamlet's name in this pipeline; each value is the name of the descriptor it runs, the streamlet.name in one of the descriptors given with --descriptors.

streamlets {
  router    = cart-router
  router-eu = cart-router     # the same descriptor, a second streamlet
}

A streamlet name is a DNS label: at most 63 characters of a-z, 0-9 and -, not starting or ending with -. Two streamlets may not share a name.

`blueprint.topics`¶

Each key is a topic id. Each value is an object with these keys, all optional:

Key	Type	Meaning
`producers`	list of port paths	outlets that write to the topic
`consumers`	list of port paths	inlets that read from the topic
`managed`	boolean, default `true`	`false`: the topic belongs to something else and is only read
`topic.name`	string	the Kafka topic name; defaults to `<pipeline>.<id>` for a managed topic and `<id>` for an unmanaged one
`cluster`	string	the Kafka cluster, a `kafka-cluster-<name>` Secret beside the operator
`bootstrap.servers`	string	brokers, instead of or over the cluster's
`partitions`	integer	partitions of a managed topic when the operator creates it
`replicas`	integer	replication factor of a managed topic when the operator creates it
`topic`	object	Kafka topic configuration for a managed topic, such as `retention.ms`
`connection-config`	object	Kafka client properties for every client of the topic, such as `security.protocol`
`producer-config`	object	Kafka producer properties for the topic's producers
`consumer-config`	object	Kafka consumer properties for the topic's consumers, and the batch limits

A port path is <streamlet>.<port>: router.valid is the valid port of the streamlet named router in blueprint.streamlets.

Nested objects are flattened into Kafka property names, so topic { retention.ms = 86400000 } and topic.retention.ms = 86400000 are the same setting. How settings left out here are filled from a Kafka cluster is on Topics and Kafka clusters.

A topic with no producers and no consumers is ignored: it does not reach the resource and no Kafka topic is created for it.

Batch limits¶

The sidecar sends an inlet's records to the process in batches: whatever arrived on one partition while that partition's previous batch was in flight, capped by two limits set under the topic's consumer-config:

consumer-config {
  flow.batch.max-records = 100      # the default
  flow.batch.max-bytes   = 1 MiB    # the default
}

The flow. keys are the platform's and never reach Kafka; every other consumer-config key is passed to the consumer.

Deploy-time overrides¶

Everything under blueprint.topics.<id> except producers, consumers, managed and topic.name can be overridden at deploy time under flow.topics.<id> in a --conf file, and a streamlet's replicas and parameters under flow.streamlets.<name>. Configure at deploy time describes them.

Verification rules¶

flow verify and flow generate report every problem in one pass and refuse when there is any. The blueprint is refused when:

the file is not valid HOCON, has no blueprint.streamlets section, or names no streamlets;
it has a blueprint.connections section, an older format: connect ports through blueprint.topics;
the descriptor directory holds no descriptors, a descriptor is invalid, or two descriptors declare the same streamlet;
a streamlet name is not a DNS label, or two streamlets share a name;
a streamlet names a descriptor that no descriptor declares;
a port path is not of the form <streamlet>.<port>;
a port path names a streamlet or port that does not exist (the message suggests the ports the streamlet has);
a producer is an inlet, or a consumer is an outlet;
a port is bound to more than one topic;
an inlet is connected to no topic;
two ports on one topic have different contracts (the message names both ports and both contracts);
a port's contract has a format other than json;
a topic's Kafka name has characters other than a-z, A-Z, 0-9, ., _ and -, or is longer than 249 characters;
a cluster name is not a DNS label;
an unmanaged topic has producers, or names neither bootstrap.servers nor cluster;
a parameter has no default in its descriptor and no value in --conf, or a value that is not its type;
a --conf file names a topic or streamlet the blueprint does not, or a parameter the streamlet does not declare, or gives a replica count that is negative or not a number.

flow generate also refuses when a streamlet has no image or the pipeline id is not valid.

An outlet connected to no topic is allowed: flow verify prints it as a note and carries on.

note: Outlet router.review is not connected.