Placement Contract

Bloodraven uses node labels and taints to control where MySQL pods run and to evict application workloads during failover. This page describes the labeling contract cluster administrators must satisfy.

Why node labeling matters

When the operator performs a failover, it taints nodes at the old active site with NoExecute to force Kubernetes to evict application pods that do not tolerate the taint. Taints are scoped per failover group, so a failover in one group does not disrupt pods belonging to another group.

Required node labels

Each site declares the exact nodes it controls with spec.sites[].taintNodeSelector:

spec:
  sites:
    - name: iad
      zone: us-east-1a
      taintNodeSelector:
        shipstream.io/failover-group.orders: "true"
        shipstream.io/site.orders: iad

The selector is required. Bloodraven does not infer taint targets from the failover group name or site name.

Apply matching labels to every node that should receive the site's taint:

kubectl label node node-iad-1 \
  shipstream.io/failover-group.orders=true \
  shipstream.io/site.orders=iad

kubectl label node node-pdx-1 \
  shipstream.io/failover-group.orders=true \
  shipstream.io/site.orders=pdx

You can label multiple nodes per site. The operator taints or untaints every node matching the selector for that site.

Taint behavior

When a site loses its primary role, the operator applies a per-group taint to all nodes selected by that site's taintNodeSelector:

shipstream.io/db-readonly-<group>=true:NoExecute

For example, for a failover group named orders:

shipstream.io/db-readonly-orders=true:NoExecute

When a site becomes the active primary, the operator removes this taint from its selected nodes.

Effect on pods

The NoExecute effect means:

• Pods that do not tolerate the taint are immediately evicted
• Pods that do tolerate the taint continue running

This gives you two categories of workloads:

Shared-node support

Because taints and selectors are scoped per failover group, multiple groups can share the same physical nodes. Put one label pair per failover group on each shared node:

kubectl label node node-iad-1 \
  shipstream.io/failover-group.orders=true \
  shipstream.io/site.orders=iad \
  shipstream.io/failover-group.inventory=true \
  shipstream.io/site.inventory=iad

Then configure each group with its own selector:

# orders MysqlFailoverGroup
spec:
  sites:
    - name: iad
      taintNodeSelector:
        shipstream.io/failover-group.orders: "true"
        shipstream.io/site.orders: iad

# inventory MysqlFailoverGroup
spec:
  sites:
    - name: iad
      taintNodeSelector:
        shipstream.io/failover-group.inventory: "true"
        shipstream.io/site.inventory: iad

A failover in orders applies shipstream.io/db-readonly-orders=true:NoExecute only. Pods for inventory are unaffected if they tolerate other groups' taints.

Cross-group tolerations

On shared nodes, application pods must tolerate taint keys from other failover groups but not their own. This ensures they are only evicted by their own group's failover:

# Pod for the "orders" group on nodes shared with "inventory"
spec:
  tolerations:
    - key: shipstream.io/db-readonly-inventory
      operator: Exists
      effect: NoExecute
    # Do not tolerate shipstream.io/db-readonly-orders.

A mutating admission webhook or Helm template can automate generating these tolerations across groups.

Scheduling MySQL pods

The operator schedules MySQL pods using the site's zone field via topology.kubernetes.io/zone. The taintNodeSelector controls application-node tainting; it is not used as the MySQL pod scheduler selector.

Operator-managed pods

Every pod the operator creates for a group tolerates that group's own shipstream.io/db-readonly-<group> taint (operator: Exists, any effect). This covers MySQL, Dragonfly, backup, backup verification, restore (bootstrap and in-place), backup cleanup, and the backup/verification schedule trigger pods. A failover never evicts them, and they can schedule onto the read-only site:

• Backups dump from the replica by default, over its Service, so the read-only side is a natural place for them. A PVC-backed profile on node-local storage can also pin the Job to a node on that side.
• Verification loads the dump into a throwaway mysqld inside its own pod and never touches the group's MySQL. Running it on the read-only side keeps the load off the primary's nodes.
• Restore Jobs load into the target site over its Service. Killing one mid-load would leave a half-dropped or half-loaded schema, so it is left to finish or fail by itself.

Only MySQL and Dragonfly pods are pinned to a site's zone. The Jobs carry no nodeSelector or affinity because they reach MySQL through Services, and a node-bound backup PVC decides placement anyway. Operator pods do not tolerate other groups' taints or the legacy shipstream.io/db-readonly key. On nodes shared between groups, a failover in another group can still evict them.