Placement Contract

Bloodraven uses node labels and taints to control where MySQL pods run and to evict application workloads during failover. This page describes the labeling contract cluster administrators must satisfy.

Why node labeling matters

When the operator performs a failover, it taints nodes at the old active site with NoExecute to force Kubernetes to evict application pods that do not tolerate the taint. Taints are scoped per failover group, so a failover in one group does not disrupt pods belonging to another group. The taint key carries the group's namespace and name, so two groups of the same name in different namespaces never share a taint, even on nodes both select.

The selector is not limited to any namespace: the operator taints every node in the cluster that matches it. Treat the right to create or edit a MysqlFailoverGroup as the right to taint nodes (see Security Model).

Required node labels

Each site declares the exact nodes it controls with spec.sites[].taintNodeSelector:

spec:
  sites:
    - name: iad
      serverID: 101
      zone: us-east-1a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: iad

The selector is required. Bloodraven does not infer taint targets from the failover group name or site name.

Apply matching labels to every node that should receive the site's taint:

kubectl label node node-iad-1 \
  bloodraven.shipstream.io/failover-group.orders=true \
  bloodraven.shipstream.io/site.orders=iad

kubectl label node node-pdx-1 \
  bloodraven.shipstream.io/failover-group.orders=true \
  bloodraven.shipstream.io/site.orders=pdx

You can label multiple nodes per site. The operator taints or untaints every node matching the selector for that site.

Taint behavior

When a site loses its primary role, the operator applies a per-group taint to all nodes selected by that site's taintNodeSelector:

<namespace>.db-readonly.bloodraven.shipstream.io/<group>=true:NoExecute

For example, for a failover group named orders in namespace default:

default.db-readonly.bloodraven.shipstream.io/orders=true:NoExecute

When a site becomes the active primary, the operator removes this taint from its selected nodes.

Effect on pods

The NoExecute effect means:

• Pods that do not tolerate the taint are immediately evicted
• Pods that do tolerate the taint continue running

This gives you two categories of workloads:

Shared-node support

Because taints and selectors are scoped per failover group, multiple groups can share the same physical nodes. Put one label pair per failover group on each shared node:

kubectl label node node-iad-1 \
  bloodraven.shipstream.io/failover-group.orders=true \
  bloodraven.shipstream.io/site.orders=iad \
  bloodraven.shipstream.io/failover-group.inventory=true \
  bloodraven.shipstream.io/site.inventory=iad

Then configure each group with its own selector:

# orders MysqlFailoverGroup
spec:
  sites:
    - name: iad
      serverID: 101
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: iad

# inventory MysqlFailoverGroup
spec:
  sites:
    - name: iad
      serverID: 101
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.inventory: "true"
        bloodraven.shipstream.io/site.inventory: iad

A failover in orders applies default.db-readonly.bloodraven.shipstream.io/orders=true:NoExecute only. Pods for inventory are unaffected if they tolerate other groups' taints.

Cross-group tolerations

On shared nodes, application pods must tolerate taint keys from other failover groups but not their own. This ensures they are only evicted by their own group's failover:

# Pod for the "orders" group on nodes shared with "inventory"
spec:
  tolerations:
    - key: default.db-readonly.bloodraven.shipstream.io/inventory
      operator: Exists
      effect: NoExecute
    # Do not tolerate default.db-readonly.bloodraven.shipstream.io/orders.

A mutating admission webhook or Helm template can automate generating these tolerations across groups.

Scheduling MySQL pods

The operator schedules MySQL pods using the site's zone field via topology.kubernetes.io/zone. The taintNodeSelector controls application-node tainting; it is not used as the MySQL pod scheduler selector.

Operator-managed pods

Every pod the operator creates for a group tolerates that group's own <namespace>.db-readonly.bloodraven.shipstream.io/<group> taint (operator: Exists, any effect). This covers MySQL, Dragonfly, backup, backup verification, restore (bootstrap and in-place), backup cleanup, and the backup/verification schedule trigger pods. A failover never evicts them, and they can schedule onto the read-only site:

• Backups dump from the replica by default, over its Service, so the read-only side is a natural place for them. A PVC-backed profile on node-local storage can also pin the Job to a node on that side.
• Verification loads the dump into a throwaway mysqld inside its own pod and never touches the group's MySQL. Running it on the read-only side keeps the load off the primary's nodes.
• Restore Jobs load into the target site over its Service. Killing one mid-load would leave a half-dropped or half-loaded schema, so it is left to finish or fail by itself.

Only MySQL and Dragonfly pods are pinned to a site's zone. The Jobs carry no nodeSelector or affinity because they reach MySQL through Services, and a node-bound backup PVC decides placement anyway. Operator pods do not tolerate other groups' taints. On nodes shared between groups, a failover in another group can still evict them.