Day 0 and day 2
Quick check: can you build a group from an empty namespace, design a certificate every client can verify against, and roll a MySQL upgrade without turning it into an incident?
Passing score: 70%. Feedback appears as you go, and only your first answer to each question counts.
Show answer
Answer: The per-site Deployments, PVCs and ConfigMaps, eight Services for a three-site group, a PodDisruptionBudget, the DNSEndpoint, and the init-users ConfigMap that creates the MySQL users
The line between yours and the operator's is the first thing to get right on day 0. Option 2 is a common hope and would be a security problem: you supply the Secrets, always. Option 3 is the specific trap this unit warns about — the operator only reads the Secret spec.tls.secretName names and never creates the Certificate; forget it and the pods sit in ContainerCreating mounting a Secret that does not exist. Option 4 misattributes both the PVC (the operator reconciles it per site) and the DNSEndpoint (the operator writes the object; external-dns turns it into a record). (objective 1)
The line between yours and the operator's is the first thing to get right on day 0. Option 2 is a common hope and would be a security problem: you supply the Secrets, always. Option 3 is the specific trap this unit warns about — the operator only reads the Secret spec.tls.secretName names and never creates the Certificate; forget it and the pods sit in ContainerCreating mounting a Secret that does not exist. Option 4 misattributes both the PVC (the operator reconciles it per site) and the DNSEndpoint (the operator writes the object; external-dns turns it into a record). (objective 1)
Show answer
Answer: The operator reconciles the MySQL user on its next poll and the new password takes effect within seconds
The operator watches every Secret spec.credentials names. A changed role Secret changes its credential hash, so on the next reconcile it connects to the primary and runs CREATE USER IF NOT EXISTS / ALTER USER … IDENTIFIED BY with the new bytes, and replication carries the change to the other sites. The spec hash includes the role Secrets too, so the pods also roll. Option 1 describes the first-boot script, which is not the only path: it is what still applies to MYSQL_ROOT_PASSWORD and a dedicated replication user's password. Option 2 is the misreading the restart encourages. Option 4 invents an immutability rule. (objective 2)
The operator watches every Secret spec.credentials names. A changed role Secret changes its credential hash, so on the next reconcile it connects to the primary and runs CREATE USER IF NOT EXISTS / ALTER USER … IDENTIFIED BY with the new bytes, and replication carries the change to the other sites. The spec hash includes the role Secrets too, so the pods also roll. Option 1 describes the first-boot script, which is not the only path: it is what still applies to MYSQL_ROOT_PASSWORD and a dedicated replication user's password. Option 2 is the misreading the restart encourages. Option 4 invents an immutability rule. (objective 2)
Show answer
Answer: False
Nothing validates a manifest against the cluster it refers to. CEL rules check the object — uniqueness, mutual exclusion, required-unless-read-only, the interval relationships on spec.sidecar — and they are immediate and cheap. A taintNodeSelector matching nothing is admitted, and then fails completely and silently: the NoExecute taint is applied to no node, so the eviction half of your failover strategy does not exist and no status field, event or log line says so. It is worse than the missing StorageClass in the same class of error, because that one at least leaves you a Pending PVC to notice. (objective 3)
Nothing validates a manifest against the cluster it refers to. CEL rules check the object — uniqueness, mutual exclusion, required-unless-read-only, the interval relationships on spec.sidecar — and they are immediate and cheap. A taintNodeSelector matching nothing is admitted, and then fails completely and silently: the NoExecute taint is applied to no node, so the eviction half of your failover strategy does not exist and no status field, event or log line says so. It is worse than the missing StorageClass in the same class of error, because that one at least leaves you a Pending PVC to notice. (objective 3)
Show answer
Answer: Working self-fencing and a working startup safety net on every site — both still run but cannot reach MySQL, and they are the two mechanisms that hold correctness when the operator cannot be reached
The sidecar verifies its own MySQL against its site's Service name because it dials loopback, which appears in no certificate. Without that SAN it cannot query MySQL and /health returns 503, so the pod is never ready and drops out of the Service endpoints — that is the immediate outage. Nothing restarts, because liveness is /peer/ping; the FencingMonitor and the startup safety net keep running, but neither can set super_read_only on a MySQL it cannot query. Unit 5's whole argument was that an operator which cannot reach a site cannot fence it, and the sites you most need fenced are exactly the ones you cannot reach; a certificate mistake has just removed the answer to that. Option 3 is wrong on the data path — replication is MySQL to MySQL. Option 4 inverts the division of labour. (objectives 4, 6)
The sidecar verifies its own MySQL against its site's Service name because it dials loopback, which appears in no certificate. Without that SAN it cannot query MySQL and /health returns 503, so the pod is never ready and drops out of the Service endpoints — that is the immediate outage. Nothing restarts, because liveness is /peer/ping; the FencingMonitor and the startup safety net keep running, but neither can set super_read_only on a MySQL it cannot query. Unit 5's whole argument was that an operator which cannot reach a site cannot fence it, and the sites you most need fenced are exactly the ones you cannot reach; a certificate mistake has just removed the answer to that. Option 3 is wrong on the data path — replication is MySQL to MySQL. Option 4 inverts the division of labour. (objectives 4, 6)
Show answer
Answer: With TLS it gains SOURCE_SSL=1; without TLS it gains GET_SOURCE_PUBLIC_KEY=1, and without either the IO thread exits asynchronously after a clean START REPLICA
The failure this prevents is the memorable part: with neither clause, START REPLICA returns cleanly and the IO thread exits afterwards, leaving a site permanently not replicating with nothing wrong at the point of the command. That is the shape you would create by running the statement by hand. Option 2 is the assumption the code comment exists to correct. Option 3 invents a conflict — SOURCE_AUTO_POSITION=1 is always used. Option 4 misplaces the setting; spec.mysqlConf renders my.cnf and does not reach a replication channel. (objective 6)
The failure this prevents is the memorable part: with neither clause, START REPLICA returns cleanly and the IO thread exits afterwards, leaving a site permanently not replicating with nothing wrong at the point of the command. That is the shape you would create by running the statement by hand. Option 2 is the assumption the code comment exists to correct. Option 3 invents a conflict — SOURCE_AUTO_POSITION=1 is always used. Option 4 misplaces the setting; spec.mysqlConf renders my.cnf and does not reach a replication channel. (objective 6)
Show answer
Answer: activeSite moves to the standby and does not move back, bloodraven_failovers_total increments, BloodravenFailoverOccurred fires, and the fresh lastFailover suppresses the next automatic failover for the whole cooldown window
Every clause of option 1 is a consequence someone meets at 02:00 having not been told. The Failover phase is a real promotion through the ordinary sequence, and it is not cooldown-gated — but it records a failover, so it consumes your anti-flap budget for a genuine failure that follows. Option 2 describes Recreate, which is exactly the window OrderedUpdate exists to avoid. Option 3 is the exclusion people assume exists. Option 4 is the fail-back misconception from Unit 3: promotion is current-state-driven, and nothing remembers which site 'should' be primary. (objectives 7, 8)
Every clause of option 1 is a consequence someone meets at 02:00 having not been told. The Failover phase is a real promotion through the ordinary sequence, and it is not cooldown-gated — but it records a failover, so it consumes your anti-flap budget for a genuine failure that follows. Option 2 describes Recreate, which is exactly the window OrderedUpdate exists to avoid. Option 3 is the exclusion people assume exists. Option 4 is the fail-back misconception from Unit 3: promotion is current-state-driven, and nothing remembers which site 'should' be primary. (objectives 7, 8)
Show answer
Answer: False
It is a fail-fast abort, not a timeout — the outer deadline is five minutes and this fires long before it. mysqld boots read_only=ON, so a plain restart no longer produces this; a standby that is writable anyway, with cross-site recovery deliberately suppressed during an ordered update, has nothing going to start replication for it. A site in that shape is stuck rather than slow, and continuing to wait would tell you nothing you do not already know. The counter behind it is deliberately not a strict streak: a probe error leaves it alone, because alternating dial errors and 'writable, no source' reads are exactly what a stale connection pool produces and a strict streak would let that mask the fault. (objective 8)
It is a fail-fast abort, not a timeout — the outer deadline is five minutes and this fires long before it. mysqld boots read_only=ON, so a plain restart no longer produces this; a standby that is writable anyway, with cross-site recovery deliberately suppressed during an ordered update, has nothing going to start replication for it. A site in that shape is stuck rather than slow, and continuing to wait would tell you nothing you do not already know. The counter behind it is deliberately not a strict streak: a probe error leaves it alone, because alternating dial errors and 'writable, no source' reads are exactly what a stale connection pool produces and a strict streak would let that mask the fault. (objective 8)
Show answer
Answer: CRDs by kubectl apply --server-side first, then helm upgrade the operator, whose sidecar image then rolls every group without a spec.sidecar.image pin — because Helm never upgrades CRDs from crds/, and a new operator against an old schema has its new fields silently pruned
Helm installs CRDs from crds/ on first install and never upgrades them — on upgrade or on rollback. Skip the explicit apply and you get the silent-pruning failure from Unit 5 by a different route: the object is admitted, the unknown field is dropped, and nothing errors at apply, in an event, or in the log. Option 3 is the belief that causes it. Option 2 invents a hook. Option 4 gets the sidecar in the wrong place: a sidecar move triggers an ordered update, so it belongs last, after the control plane is on the new release, which is what leaving spec.sidecar.image unset gives you. (objective 9)
Helm installs CRDs from crds/ on first install and never upgrades them — on upgrade or on rollback. Skip the explicit apply and you get the silent-pruning failure from Unit 5 by a different route: the object is admitted, the unknown field is dropped, and nothing errors at apply, in an event, or in the log. Option 3 is the belief that causes it. Option 2 invents a hook. Option 4 gets the sidecar in the wrong place: a sidecar move triggers an ordered update, so it belongs last, after the control plane is on the new release, which is what leaving spec.sidecar.image unset gives you. (objective 9)
Show answer
Answer: Delete it and alert on bloodraven_failovers_total instead — there is no Failover reason; the failover row of the matrix emits SiteUnhealthy, and the five reasons that reach status are Healthy, SiteUnhealthy, SplitBrain, NoPrimary, TotalLoss
A rule matching a reason string the operator never emits does not fire rarely. It sits green forever, through every promotion. Option 2 is a real trap rather than an obvious wrong answer: reason=SiteUnhealthy is what the failover row emits, but it is also what a live primary with an unreachable peer emits and what a writable non-promotable site awaiting fencing emits, so as a failover alert it is far too broad. The counter is the right signal for 'a failover happened'; the reason strings are for topology shape. (objectives 11, 12)
A rule matching a reason string the operator never emits does not fire rarely. It sits green forever, through every promotion. Option 2 is a real trap rather than an obvious wrong answer: reason=SiteUnhealthy is what the failover row emits, but it is also what a live primary with an unreachable peer emits and what a writable non-promotable site awaiting fencing emits, so as a failover alert it is far too broad. The counter is the right signal for 'a failover happened'; the reason strings are for topology shape. (objectives 11, 12)
Show answer
Answer:
One: the reference card, because the first thirty seconds of an incident should be spent reading rather than recalling — it holds the shipped defaults beside the playground overrides, the five condition reasons, the nine promotion steps with which four are fatal, the four Service kinds and their selectors, and the metric label sets that are not uniform. Two: the version appendix, because everything with a date on it — issue states, upstream pins, licence status, 'the published page says X' — is exactly what a course cannot keep true, and it carries the command that re-checks each one, so a stale memory costs a lookup rather than a wrong answer in a meeting. Three: the alert-to-runbook-to-first-command map from Unit 6, because an alert without a next step is a page with nowhere to go, and it is where the one alert nobody ships — application write failures after a successful failover — is owned by a named human. If I could add a fourth it would be the honest statement about the group itself: it promotes unattended in about 12 seconds, switches over on purpose at RPO 0 by construction, reports its exact lost-transaction count, and will not save them from a DNS record nobody flipped.
A full-credit answer shows: A strong answer names the reference card (and at least two of: defaults-versus-overrides, the five reasons, fatality per step, Service selectors, label sets), the version appendix (and why — dated facts rot, and it carries re-check commands), and the alert-to-runbook map (and that the application-side write alert is unowned by default). Credit any third choice that is defensible and explained. Do not credit a list of three artefacts with no statement of what each is for.
The question is a check on whether the course produced judgement rather than recall. All three artefacts exist because a fact is only useful at the moment it is needed, in the form it is needed in — one screen, dated, with a next command attached. (objectives 10, 12)
Sample answer
One: the reference card, because the first thirty seconds of an incident should be spent reading rather than recalling — it holds the shipped defaults beside the playground overrides, the five condition reasons, the nine promotion steps with which four are fatal, the four Service kinds and their selectors, and the metric label sets that are not uniform. Two: the version appendix, because everything with a date on it — issue states, upstream pins, licence status, 'the published page says X' — is exactly what a course cannot keep true, and it carries the command that re-checks each one, so a stale memory costs a lookup rather than a wrong answer in a meeting. Three: the alert-to-runbook-to-first-command map from Unit 6, because an alert without a next step is a page with nowhere to go, and it is where the one alert nobody ships — application write failures after a successful failover — is owned by a named human. If I could add a fourth it would be the honest statement about the group itself: it promotes unattended in about 12 seconds, switches over on purpose at RPO 0 by construction, reports its exact lost-transaction count, and will not save them from a DNS record nobody flipped.
A full-credit answer shows
A strong answer names the reference card (and at least two of: defaults-versus-overrides, the five reasons, fatality per step, Service selectors, label sets), the version appendix (and why — dated facts rot, and it carries re-check commands), and the alert-to-runbook map (and that the application-side write alert is unowned by default). Credit any third choice that is defensible and explained. Do not credit a list of three artefacts with no statement of what each is for.
The question is a check on whether the course produced judgement rather than recall. All three artefacts exist because a fact is only useful at the moment it is needed, in the form it is needed in — one screen, dated, with a next command attached. (objectives 10, 12)