brprep — the day-0 pre-flight and the change plan

Goal

Write a shell pre-flight that reads a MysqlFailoverGroup manifest and reports what the API server will reject, what it will silently admit, and what an ordered update of it would actually do — so the first time anyone sees your production group's problems is before it is applied, not during an incident.

Unit 7 project — Day 0 and day 2. No Python, no cluster: bash and jq against JSON fixtures.

Goal

Write a shell pre-flight that reads a MysqlFailoverGroup manifest and reports what the API server will reject, what it will silently admit, and what an ordered update of it would actually do — so the first time anyone sees your production group’s problems is before it is applied, not during an incident.

How this works

Topic 1 of this unit drew the line that matters on day 0: admission validates the object, and nothing validates the object against the cluster. A manifest that sets both credential modes is refused in milliseconds. A manifest whose taintNodeSelector names labels no node carries is accepted, applied, and silently does nothing for the rest of its life.

brprep makes that line executable. It reads a manifest as JSON and prints three kinds of line:

Line	Meaning	Effect on the exit code
`REJECT <rule>`	the API server refuses this object — a CEL rule or a schema bound	exit `1`
`WARN <finding>`	admission accepts it, and it will hurt later	none — exit stays `0`
the change plan	what an ordered update to `--target-image` would do	none

That split is the point of the project. A tool that treats every finding as an error is a tool people turn off.

./starter/brprep.sh tests/fixtures/good.json --target-image mysql:9.8

Everything runs against the eight manifests in tests/fixtures/. No cluster is needed to finish the code. If you have a live group, kubectl get mysqlfailovergroup <name> -o json produces exactly the input shape, and so does yq -o json < manifest.yaml.

Your tasks

Open starter/brprep.sh. Three functions are stubbed, marked TODO A, TODO B and TODO C. Run the grader first, before you change anything:

./tests/run.sh

It reports nine failing fixtures. Read the diff for good first — that is the shape everything else is measured against.

TODO A — check_admission. Print one REJECT <rule> line per admission rule the manifest violates, in the order the RULE_* constants are declared. A clean manifest prints nothing at all. The nine rules are documented in the script — seven CEL rules on the CRD plus the MinItems/MaxItems bound on spec.sites, all of them refusals at kubectl apply — and two are worth flagging here.

The role default is load-bearing. spec.sites[].role is optional and defaults to primary-candidate, so a site with no role key counts toward the two-candidate minimum and is required to carry lbIP and taintNodeSelector. Reading a missing role as “not a candidate” fails half the fixtures.

The reader exemption is the discriminator. taintNodeSelector and lbIP are required unless the role is read-only — because a reader is never promoted and never tainted, so it needs neither. tests/fixtures/reader-lbip.json has a legitimate reader without them and a primary-candidate missing both. A check that demands them unconditionally fails it; a check that never demands them fails too.

TODO B — check_silent. Print one WARN <finding> line per day-0 mistake nothing validates, using the WARN_* templates verbatim. Six findings, documented in the script:

a floating image tag — mysql:9 and mysql:latest float, mysql:9.7 does not
a backup profile on storage.type: PVC, named
a site whose requests do not equal its limits, named — no Guaranteed QoS, so the kubelet may evict your primary first
readOnlyMaxLagSeconds above maxLagSeconds
encryptionAtRest enabled with no backup profile at all
updateStrategy: Recreate

None of these may change the exit code.

TODO C — plan_upgrade. Turn --target-image into a written plan: which standby is upgraded first and why, that a real promotion follows, which site ends up active, and the three observable consequences — a bloodraven_failovers_total increment, a lastFailover stamp that consumes the anti-flap budget, and a DNS flip. Print no change when the target already matches spec.image.

A plan that says “the pods will restart” is not a plan. The whole reason this belongs in a change record is that a routine image bump moves your primary and fires a failover alert, and somebody is going to be woken up by it.

What the scaffolding is for

You do not have to write any of this:

argument parsing, the MysqlFailoverGroup kind check, and the exit-code plumbing
q <jq-filter>, a one-line reader over the manifest — use it for every field access; grep over JSON is how a checker starts lying to you
secs <duration>, Go-duration to whole seconds, so the spec.sidecar interval rules are arithmetic rather than string comparison. It returns 0 for an empty or unparsable value, which is never a valid setting and therefore always reads as “violates the minimum”
the exact RULE_* and WARN_* strings. The grader matches these tokens, so do not reword them
the report assembly and the verdict: line

Expected output

A clean manifest with a plan:

$ ./starter/brprep.sh tests/fixtures/good.json --target-image mysql:9.8
brprep: ledger-db/ledger
plan: mysql:9.7 -> mysql:9.8
1. upgrade standby pdx first (a replica may run a newer MySQL than its source; a source may not run a newer MySQL than its replica)
2. promote pdx through the ordinary nine-step sequence
3. upgrade the former active iad, now a standby
active site after this rollout: pdx
expect: bloodraven_failovers_total increments
expect: lastFailover is stamped and consumes the anti-flap cooldown
expect: the DNSEndpoint A record flips
verdict: APPLYABLE (0 finding(s))
$ echo $?
0

And a manifest that will apply cleanly and then disappoint you:

$ ./starter/brprep.sh tests/fixtures/silently-wrong.json
brprep: ledger-db/ledger
WARN spec.image is a floating tag; pin an immutable one or a restart can drift you onto an unsupported MySQL
WARN backup profile nightly uses storage.type PVC; a backup sharing a failure domain with the data is an assumption, not a backup
WARN site iad sets resources.requests != resources.limits; without Guaranteed QoS the kubelet may evict this MySQL first
WARN replication.readOnlyMaxLagSeconds (300) is above maxLagSeconds (30); the reader endpoint is now looser than the group threshold
WARN updateStrategy Recreate patches every site Deployment in one pass; both sites can restart at once
verdict: APPLYABLE (5 finding(s))
$ echo $?
0

Five findings, zero rejections, exit 0. That manifest is a perfectly valid MysqlFailoverGroup and a bad idea, and no cluster anywhere will tell you so.

Then the cluster half, if you have one:

kubectl -n bloodraven-playground get mysqlfailovergroup playground -o json > playground-live.json
./starter/brprep.sh playground-live.json --target-image mysql:9.8

The playground group will show findings — it sets failoverCooldown: 30s and maxLagSeconds: 30 precisely so experiments finish while you are watching. Read them and decide which ones you would carry into production and which are playground-only. That decision is the Unit 6 go-live gate, arrived at from the other direction.

Rules

bash and jq only. No Python, no kubectl, no network, no yq at grading time.
Read every field through q / jq. A regex over JSON works on the fixtures and fails on the first real manifest with different key ordering or a multi-line value.
Keep set -euo pipefail, and keep the function names — the grader calls the script, but a structural check looks for them.
Findings never change the exit code. Rejections always do.
Output must be stable and diffable: no timestamps, no absolute paths, no colour.
Do not edit anything under tests/. The fixtures and expectations are the grader’s copy; changing them changes the question rather than the answer.

Steps

Run the pre-flight before changing anything
./tests/run.sh drives starter/brprep.sh over eight fixtures. It runs as given and gets every one of them wrong in the same way: check_admission and check_silent return nothing, and plan_upgrade prints a placeholder. Read the expected output beside the actual for fixtures/good.json first — that is the shape everything else is measured against.
Done when: ./tests/run.sh exits non-zero and reports 9 failing fixtures.
Implement check_admission (TODO A)
Emit one REJECT <rule> line per CEL rule the manifest violates, in the fixed order given in the script. The nine rules are listed in the brief; each maps to a jq expression over the manifest. A manifest that violates none produces no output at all.
Done when: ./tests/run.sh reports admission matching for all nine cases, including both-credentials.json, one-candidate.json, reader-lbip.json and lease-too-short.json.
Implement check_silent (TODO B)
Emit one WARN <finding> line per day-0 mistake that admission does not catch: a floating image tag, a PVC-only backup profile, requests not equal to limits on a MySQL container, a readOnlyMaxLagSeconds above maxLagSeconds, a group with encryptionAtRest and no verified backup profile, and updateStrategy: Recreate. These are findings, not errors — they never change the exit code on their own.
Done when: ./tests/run.sh reports silent matching for all nine cases, and fixtures/silently-wrong.json produces exactly five WARN lines.
Implement plan_upgrade (TODO C)
Given --target-image, print the ordered-update plan: the standby site that goes first, the promotion that follows, the site that ends up active, and the three observable consequences — a bloodraven_failovers_total increment, a lastFailover stamp that consumes the anti-flap budget, and a DNS flip. Print no change when the target image already matches spec.image.
Done when: ./tests/run.sh exits 0 with ALL 9 FIXTURES MATCH.
Run it against a manifest you wrote
Write a MysqlFailoverGroup for a group of your own — a real one if you have it, the ledger shape from topic 1 of this unit otherwise — convert it with yq -o json or by hand, and run brprep over it. Fix what it finds. Keep the output; it is the first page of a change record.
Done when: starter/brprep.sh <your-manifest>.json runs and reports zero REJECT lines.

How it is graded

Criterion	What earns it	Weight
Admission rules reproduced faithfully	All nine admission rules are checked, each against the manifest rather than against a hard-coded fixture name, and each emits the exact `REJECT <rule>` token in the fixed order. A manifest violating several produces several lines. `reader-lbip.json` is the discriminator: `lbIP` and `taintNodeSelector` are required unless the role is `read-only`, so a check that demands them unconditionally fails it, and a check that never demands them fails `good.json`'s mutation.	35
Silent findings separated from rejections	`check_silent` reports the six documented findings as `WARN` and never as `REJECT`, and never changes the exit code by itself. The floating-tag check catches `mysql:9` and `mysql:latest` but not `mysql:9.7`. The QoS check compares requests to limits for both CPU and memory rather than checking that limits merely exist.	30
The change plan is specific and correct	`plan_upgrade` names the standby that is upgraded first, states that a real promotion follows, names the site that ends up active, and lists the three observable consequences. It prints `no change` when the target already matches. A plan that says 'the pods will restart' without naming the promotion or the moved primary does not earn this.	25
Shell discipline	`set -euo pipefail` retained, `jq` used for every field read rather than `grep`/`sed` over JSON, no `eval`, and the script exits 0 with findings but non-zero with rejections. Output is stable and diffable — no timestamps, no absolute paths.	10
Total		100

Test cases

Test	Checks	Expected	Weight
good_manifest_is_clean	A production-shaped manifest with a reader produces no REJECT and no WARN. Catches a shortcut: a checker that emits findings unconditionally.	`PASS`	15
admission_rules_all_nine	Each fixture violates one rule and must produce exactly that one REJECT line. Catches a shortcut: reporting every rule whenever any rule fails.	`PASS`	30
reader_exemption	Adversarial: the reader legitimately has no `lbIP` and no `taintNodeSelector`, while a `primary-candidate` in the same manifest is missing both. Exactly one REJECT. Catches both a check that demands them unconditionally and one that never demands them.	`PASS`	20
silent_findings_do_not_reject	Five WARN lines, zero REJECT lines, exit code 0. Catches a shortcut: treating findings as errors so the fixture 'fails' for the wrong reason.	`PASS`	20
change_plan_names_the_promotion	The plan for `good.json` at a new image names the standby first, the promotion, the resulting active site and the three consequences; the plan at the current image prints `no change`.	`PASS`	15

Starter files

1 files

brprep.sh

                    #!/usr/bin/env bash
# brprep — day-0 pre-flight for a MysqlFailoverGroup manifest.
#
#   ./brprep.sh <manifest.json> [--target-image <tag>]
#
# Reads a MysqlFailoverGroup as JSON (yq -o json < manifest.yaml, or kubectl
# get -o json) and reports three things:
#
#   REJECT <rule>   the API server will refuse this object — a CEL rule on the
#                   CRD, reproduced here so you find out before you apply
#   WARN <finding>  admission will happily accept this, and it will hurt later
#   the change plan for an ordered update to --target-image
#
# Exit codes:  0 clean or warnings only · 1 one or more REJECTs · 2 bad input
#
# Standard tools only: bash and jq. No cluster, no kubectl, no network.
set -euo pipefail

# ---------------------------------------------------------------- input

MANIFEST=""
TARGET_IMAGE=""
while [ $# -gt 0 ]; do
  case "$1" in
    --target-image) TARGET_IMAGE="${2:-}"; shift 2 ;;
    -h|--help) sed -n '2,14p' "$0"; exit 0 ;;
    -*) echo "brprep: unknown flag $1" >&2; exit 2 ;;
    *) MANIFEST="$1"; shift ;;
  esac
done

[ -n "$MANIFEST" ] || { echo "usage: brprep.sh <manifest.json> [--target-image <tag>]" >&2; exit 2; }
[ -r "$MANIFEST" ] || { echo "brprep: cannot read $MANIFEST" >&2; exit 2; }
command -v jq >/dev/null 2>&1 || { echo "brprep: jq is required" >&2; exit 2; }

jq -e '.kind == "MysqlFailoverGroup"' "$MANIFEST" >/dev/null 2>&1 \
  || { echo "brprep: $MANIFEST is not a MysqlFailoverGroup" >&2; exit 2; }

# q <jq-filter> — read one value out of the manifest.
q() { jq -r "$1" "$MANIFEST"; }

# ---------------------------------------------------------------- scaffolding
#
# Everything below this line up to TODO A is written for you.

# secs <duration> — Go duration ("20s", "5m", "1h30m") to whole seconds.
# Prints 0 for an empty or unparsable value, which is never a valid setting
# and therefore always reads as "violates the minimum".
secs() {
  local d="${1:-}" total=0 n unit
  [ -n "$d" ] || { echo 0; return; }
  while [[ "$d" =~ ^([0-9]+)(h|m|s|ms)(.*)$ ]]; do
    n="${BASH_REMATCH[1]}"; unit="${BASH_REMATCH[2]}"; d="${BASH_REMATCH[3]}"
    case "$unit" in
      h) total=$(( total + n * 3600 )) ;;
      m) total=$(( total + n * 60 )) ;;
      s) total=$(( total + n )) ;;
      ms) : ;;   # sub-second precision is never load-bearing for these fields
    esac
  done
  echo "$total"
}

# The nine admission rules, in the fixed order brprep reports them. Seven are
# CEL rules on the CRD; the site-count one is a MinItems/MaxItems constraint on
# the schema. Both are refusals at `kubectl apply`, which is all brprep claims.
# The grader matches these tokens exactly, so do not reword them.
RULE_CREDENTIALS="exactly one of secretName or credentials must be set"
RULE_SITE_COUNT="spec.sites must contain between 2 and 16 entries"
RULE_SITE_NAMES="spec.sites[].name must be unique"
RULE_TWO_CANDIDATES="spec.sites must contain at least two sites with role primary-candidate"
RULE_PRIORITIES="splitBrainPolicy.sitePriorities entries must match the names of sites with role primary-candidate"
RULE_SITE_FIELDS="taintNodeSelector and lbIP are required unless role is read-only"
RULE_PEER_INTERVAL="sidecar.peerCheckInterval must be at least 1s"
RULE_LEASE_MIN="sidecar.leaseTimeout must be at least 3s"
RULE_LEASE_RATIO="sidecar.leaseTimeout must be at least 3x sidecar.peerCheckInterval"

# The six silent findings, in the fixed order brprep reports them.
WARN_FLOATING_TAG="spec.image is a floating tag; pin an immutable one or a restart can drift you onto an unsupported MySQL"
WARN_PVC_BACKUP="backup profile %s uses storage.type PVC; a backup sharing a failure domain with the data is an assumption, not a backup"
WARN_QOS="site %s sets resources.requests != resources.limits; without Guaranteed QoS the kubelet may evict this MySQL first"
WARN_READER_GATE="replication.readOnlyMaxLagSeconds (%s) is above maxLagSeconds (%s); the reader endpoint is now looser than the group threshold"
WARN_UNVERIFIED_ENCRYPTION="encryptionAtRest is enabled but no backup profile is configured; an encrypted group with no verified restore path is one keyring away from unrecoverable"
WARN_RECREATE="updateStrategy Recreate patches every site Deployment in one pass; both sites can restart at once"

# ---------------------------------------------------------------- TODO A
#
# check_admission — print one `REJECT <rule>` line per CEL rule this manifest
# violates, in the order the RULE_* constants are declared above. A manifest
# that violates none prints nothing at all.
#
# The nine rules, and what each one actually checks:
#
#   RULE_CREDENTIALS    exactly one of .spec.secretName and .spec.credentials
#                       is present and non-empty. Both, or neither, is a
#                       rejection.
#   RULE_SITE_COUNT     2 <= (.spec.sites | length) <= 16.
#   RULE_SITE_NAMES     every .spec.sites[].name is distinct.
#   RULE_TWO_CANDIDATES at least two sites have role "primary-candidate".
#                       Remember the CRD default: an omitted role *is*
#                       primary-candidate, so count a missing role as one.
#   RULE_PRIORITIES     every entry of .spec.splitBrainPolicy.sitePriorities
#                       names a site whose effective role is
#                       primary-candidate. An absent list is fine.
#   RULE_SITE_FIELDS    every site whose effective role is NOT "read-only" has
#                       both .lbIP and .taintNodeSelector. A read-only site
#                       needs neither — it is never promoted and never tainted.
#                       This is the one the fixtures try hardest to break.
#   RULE_PEER_INTERVAL  secs(.spec.sidecar.peerCheckInterval) >= 1, when set.
#   RULE_LEASE_MIN      secs(.spec.sidecar.leaseTimeout) >= 3, when set.
#   RULE_LEASE_RATIO    secs(leaseTimeout) >= 3 * secs(peerCheckInterval),
#                       when both are set.
#
# Return 0 always; the caller counts the lines.
check_admission() {
  : # TODO A — replace this
}

# ---------------------------------------------------------------- TODO B
#
# check_silent — print one `WARN <finding>` line per day-0 mistake that
# admission accepts. Use the WARN_* templates above verbatim, filling the
# printf placeholders. Order is the order they are declared.
#
#   WARN_FLOATING_TAG          .spec.image has no tag at all, or a tag of
#                              "latest", or a tag with no dot in it
#                              ("mysql:9" floats, "mysql:9.7" does not).
#   WARN_PVC_BACKUP            any .spec.backup.profiles[] with
#                              .storage.type == "PVC". One line per profile,
#                              naming it.
#   WARN_QOS                   any site where requests.cpu != limits.cpu or
#                              requests.memory != limits.memory. Compare the
#                              strings; a site that omits resources entirely
#                              is not a finding here.
#   WARN_READER_GATE           .spec.replication.readOnlyMaxLagSeconds is set
#                              and strictly greater than maxLagSeconds.
#   WARN_UNVERIFIED_ENCRYPTION .spec.encryptionAtRest.enabled is true and
#                              .spec.backup.profiles is absent or empty.
#   WARN_RECREATE              .spec.updateStrategy == "Recreate".
#
# These are findings, never errors: they must not change the exit code.
check_silent() {
  : # TODO B — replace this
}

# ---------------------------------------------------------------- TODO C
#
# plan_upgrade — given $TARGET_IMAGE, print the ordered-update plan.
#
# When $TARGET_IMAGE is empty, print nothing. When it equals .spec.image,
# print exactly:
#
#     no change
#
# Otherwise print, in this order and one per line:
#
#     plan: mysql:9.7 -> mysql:9.8
#     1. upgrade standby <site> first (a replica may run a newer MySQL than
#        its source; a source may not run a newer MySQL than its replica)
#     2. promote <site> through the ordinary nine-step sequence
#     3. upgrade the former active <site>, now a standby
#     active site after this rollout: <site>
#     expect: bloodraven_failovers_total increments
#     expect: lastFailover is stamped and consumes the anti-flap cooldown
#     expect: the DNSEndpoint A record flips
#
# The standby is the *second* site whose effective role is primary-candidate,
# in `spec.sites` order; the former active is the first. This manifest carries
# no status, so declaration order is the only signal available — say so in
# your own runbook, and read `status.activeSite` when you have a live group.
plan_upgrade() {
  : # TODO C — replace this
}

# ---------------------------------------------------------------- report

echo "brprep: $(q '.metadata.namespace // "default"')/$(q '.metadata.name')"

rejects="$(check_admission || true)"
warns="$(check_silent || true)"

[ -n "$rejects" ] && printf '%s\n' "$rejects"
[ -n "$warns" ] && printf '%s\n' "$warns"

plan="$(plan_upgrade || true)"
[ -n "$plan" ] && printf '%s\n' "$plan"

if [ -n "$rejects" ]; then
  echo "verdict: NOT APPLYABLE ($(printf '%s\n' "$rejects" | wc -l | tr -d ' ') rejection(s))"
  exit 1
fi
echo "verdict: APPLYABLE ($( [ -n "$warns" ] && printf '%s\n' "$warns" | wc -l | tr -d ' ' || echo 0 ) finding(s))"
exit 0

Projects are not auto-graded here. The rubric and the test cases above are the grading contract — run them yourself on your own machine.