Upgrading without an incident

There are three things you will upgrade, they are upgraded by three different mechanisms, and only one of them is automatic. MySQL moves by an ordered rollout the operator drives. The operator moves by helm upgrade. The CRDs move by neither, and that is the one that catches people.

Everything else in this course has been about failure arriving unannounced. This is the opposite problem: you are the one causing the disturbance, on a Tuesday, on purpose, and the only question is whether anybody notices.

Upgrading MySQL: the standby goes first

Change spec.image and apply. What follows is not a Deployment rollout — the operator deliberately takes that away from Kubernetes and runs it itself.

The mechanism is the spec hash from the last topic. Under updateStrategy.type: OrderedUpdate (the default) the reconciler leaves existing site Deployments untouched, so the desired hash and the live Deployment annotation diverge. The runner notices that drift on its next pass and hands the rollout to the ordered updater, which does this:

Two things about that sequence are worth being able to defend.

The order is MySQL’s requirement, not a preference. A replica may run a newer MySQL than its source; a source may not run a newer MySQL than its replica. Upgrading the standby first keeps the newer version on the replica side of replication for the whole window, which is the supported direction. Upgrading the primary first would put an older replica behind a newer source — unsupported, and the failure is not always immediate or obvious.

Your primary moves, and it does not move back. Fail-back is current-state-driven, not identity-driven: nothing in the operator remembers that iad “should” be primary. After an image bump your active site is whichever one used to be the standby. If your application has any site affinity — a warm standby per site, a taint toleration, a latency assumption — that is a change you scheduled without noticing. Run the upgrade in the direction you want to end up.

The refusals, and the one that will surprise you

The updater refuses to start at all if the standby is not genuinely a standby:

precondition: standby <site> is writable; refusing to start ordered update
precondition: standby <site> is not replicating

Both run before the updater takes its lock, so a refused attempt leaves nothing behind and you can simply fix the standby and re-apply.

Then the interesting one, which fires mid-rollout. WaitReplica is not just a timeout — it watches for a specific shape and aborts on it:

standby is writable but replication is not running; aborting ordered update

That is the restart hazard made explicit. A MySQL pod that restarted used to come up writable for a few seconds before anything fenced it — you met that in Unit 5 as the old source of split brains. Every mysqld now starts with --read-only=ON, so a standby restarting onto the new spec comes back read-only and does not trip this; the check stays as the backstop for a standby that is writable anyway, because something cleared that flag. During an ordered update cross-site recovery is suppressed, so nothing is going to start replication for you. A standby that comes back writable with no working replication is not a slow standby; it is a stuck one, and waiting the full five minutes before saying so would leave the group in the rollout for no reason. So the updater counts writable observations and gives up early.

The counter is deliberately not a strict streak — a probe error leaves it alone rather than resetting it — because an alternating pattern of dial errors and “writable, no source” reads is exactly what a stale connection pool produces, and a strict streak would let that mask a genuinely broken standby until the outer deadline.

One more nicety worth knowing when you are reading the logs: a standby pod restart preserves replication metadata, but the operator runs mysqld with --skip-replica-start, so the threads come back stopped with SourceHost still populated. The updater owns that window, recognises the shape and issues START REPLICA itself rather than waiting for something else to.

Recreate is a decision about write availability

spec.updateStrategy.type takes OrderedUpdate or Recreate, and the difference is not cosmetic. Under Recreate the runner clears the drift list entirely and the reconciler patches every site Deployment in one pass — so their pod restarts may overlap, and a group with two sites can have both of them down at the same moment. That is the total-loss window OrderedUpdate exists to prevent.

Use Recreate only when you can afford to lose the primary and the standby simultaneously, and never for a MySQL major-version bump: if one pod’s in-place upgrade stalls, you have no healthy primary and no rollback path.

What an image bump does to your dashboards

Say this out loud before you run one, because otherwise your on-call will say it at 02:00.

The Failover phase performs a real promotion, through the same code path as an emergency failover, and its completion callback stamps the durable failover record and increments bloodraven_failovers_total. And it is not cooldown-gated — failoverCooldown guards the automatic path only, and nothing in the update controller consults it.

So a routine image bump produces: a BloodravenFailoverOccurred alert, a moved activeSite, a fresh lastFailover stamp that will suppress the next automatic failover for the whole cooldown window, and a DNSEndpoint flip. None of it is a fault. All of it looks exactly like one.

Two mitigations, and you want both. Alert on the ordered-update log lines — ordered update: updating standby, ordered update: failing over to updated standby, ordered update complete — so a failover with a rollout around it is visibly different from a failover without one. And watch status.updatePhase, which is non-empty for exactly the duration of a rollout and is the cheapest possible “is this us?” check:

kubectl -n bloodraven-playground get mysqlfailovergroup playground \
  -o jsonpath='{.status.updatePhase}{"\n"}'

Upgrading the operator, and the CRDs it does not bring with it

The operator is one Deployment with replicaCount: 1 and leader election. Upgrading it is helm upgrade, and the blast radius is genuinely small — from Unit 5 you know the operator is not on the request path, so a restart costs failover cover for a few seconds and costs your data plane nothing.

The CRDs are the trap, and it is a Helm one rather than a Bloodraven one:

Helm installs CRDs from a chart’s crds/ directory on first install, and never upgrades them.

So helm upgrade moves the operator binary and silently leaves the CRD schema at whatever version you first installed. A new operator against an old CRD is the worst shape available: the fields the new version wants are pruned by the API server exactly as preferSite was in Unit 5 — admitted, dropped, no error anywhere — and you get an operator behaving as though you never configured the thing you configured.

Apply the CRDs explicitly, do it first, and do it server-side. The MysqlFailoverGroup CRD is larger than the 256 KiB kubectl.kubernetes.io/last-applied-configuration annotation that a plain kubectl apply -f writes, so client-side apply fails with metadata.annotations: Too long. kubectl apply --server-side --force-conflicts keeps no such annotation. If Argo CD owns your CRDs, the same thing is its ServerSideApply=true sync option.

Step 4 catches people out because it is not a step you take: it is a consequence of step 3. The sidecar ships as its own image, and a group that leaves spec.sidecar.image unset stores no image at all — it runs the one the operator release ships with, which the chart hands the operator from its sidecar.image.repository and sidecar.image.tag values. So the helm upgrade that moves the operator also changes every such group’s pod spec, and moving the sidecar is a pod restart on every site — the same ordered update as an image bump, with the same failover. Plan the chart upgrade as you would plan that rollout. A group that pins spec.sidecar.image moves only when you change the pin, and a pin from another release gets a SidecarVersionSkew Warning. Bloodraven tolerates a one-minor skew between operator and sidecar in either direction while pods roll, because both sides of the HTTP surface between them are additive-only. Beyond one minor is untested.

The sidecar and MySQL share a pod, so a sidecar move and a spec.image bump go through the same ordered update; there is no separate sidecar sequencing to arrange.

The one-way doors

Three things you cannot undo, collected in one place because each is discovered late.

MySQL does not support downgrade of data. Once a datadir has been opened by a newer MySQL, an older mysqld may refuse it outright or corrupt it. MysqlBackup.status.mysqlImage records the image tag that produced each dump precisely so you can check before restoring; restore onto the same major version that produced the backup, never onto an older one.

A steady-state per-site version split is not supported. spec.image is one field per group and SiteSpec has no override. Edit a site’s Deployment out of band to pin a different tag and the next reconcile reverts it. Transient skew during a rollout is fine and expected; permanent skew is not a configuration, it is a fight with the reconciler.

There is no version admission check. Setting spec.image to something Bloodraven does not support produces no CRD validation error. It surfaces as MySQL pods failing, which is a much worse place to find out. The supported baseline, and what to run to re-check it, is in the version appendix, row C1.

Where this leaves you

You can roll a MySQL image change and name the phase you are in from status.updatePhase. You can say why the standby goes first, what the updater refuses to start on, and which mid-rollout shape makes it abort early rather than wait. You can predict the alert your dashboards will show and tell it apart from a real failover. And you can upgrade the operator without letting Helm leave your CRDs behind.

That is day 2 complete. What is left is the part you take away from the screen.

Mark as read