Emergency failover, end to end
You hold a site down and watch the whole sequence run — fence, kill, drain, promote, confirm, record, flip. You time it against a stopwatch rather than against the documentation, then prove exactly what the outage cost you in transactions, and decide what to do with the old primary when it comes back carrying writes nobody else has. It is a textbook-perfect promotion: pdx is writable in about twelve seconds and the counter application carries on as if nothing happened. That is the problem. It is still reading from iad, the site you just demoted.
Topics
Unit test
Quick check: can you name which failover steps abort on error and which only warn, say why the same promotion takes 12 seconds one day and 36 the next, and read two GTID sets to decide whether a returning primary rejoins or blocks?
Project
The post-failover audit report
(Optional — see the brief for the shorter jq route and what skipping it costs you.) Build a tool that turns a post-failover status into an incident record — when it happened, from which site to which, the promotion GTID, the divergent set and its transaction count, and a verdict of RPO 0 or N transactions lost — so that the number you report after an outage is measured rather than assumed.