The poll loop and per-site state

playground is healthy. iad is writable. pdx and reader are read-only. The counter is clicking. This is the only calm moment you will get to ask: what is the operator doing in the two seconds between one status read and the next?

The loop, the probe, the four states

spec.pollInterval defaults to 2s, and the operator hard-defaults to 2 s again in Go when the field’s pointer is nil — so an omitted pollInterval is 2 s, not zero and not “never”. Every tick, the loop does the same four things: probe every site, apply the debounce counters, compute each site’s state, then look for transitions.

The probe is one statement. SELECT @@read_only. Not a ping, not a replication check, not a SELECT 1 — one server variable, read over the connection the operator already holds. Every site is probed in parallel, each under its own context.WithTimeout(ctx, 5*time.Second). Two consequences follow, and you want both in your head: a dark site cannot cost the cycle more than 5 s, and a slow site does not delay its healthy peers’ probes — but the cycle itself does not finish until the slowest probe returns.

That single boolean, plus whether the query returned an error at all, is the entire input to the state machine. There are four states, and they are exactly these:

The artifact: the three knobs, and the function they feed

Nothing in playground changed since you stood it up. What changed is that three lines of that spec are now load-bearing rather than boilerplate:

# playground — unchanged since you stood it up; these three lines are the subject now
spec:
  pollInterval: 2s        # <-- the tick
  failureThreshold: 3     # <-- consecutive failures before "Unreachable"
  recoveryThreshold: 2    # <-- consecutive writable answers before "Writable"

They land in one function, and reading it is faster than reading any prose about it:

func (tm *TopologyManager) computeState(site *siteTracker, readOnly bool, err error) state.SiteState {
	if err != nil {
		site.recoveryCount = 0
		site.failCount++
		if site.failCount >= tm.cfg.FailureThreshold {
			return state.StateUnreachable
		}
		return site.state // not enough failures yet, keep current state
	}

	// Successful poll.
	site.failCount = 0

	if readOnly {
		site.recoveryCount = 0
		return state.StateReadOnly
	}

	// read_only=0 (writable)
	if site.state != state.StateWritable {
		site.recoveryCount++
		if site.recoveryCount >= tm.cfg.RecoveryThreshold {
			return state.StateWritable
		}
		return site.state // not enough recoveries yet
	}

	return state.StateWritable
}

Both counters reset on the opposite outcome, so “three failures” means three consecutive failures. One good answer in the middle puts failCount back to zero and you start again.

The asymmetry, and the number people get wrong

Now look at what that function believes instantly and what it makes prove itself. A read_only=1 answer becomes ReadOnly on a single poll — no counter, no waiting — and it zeroes recoveryCount on the way through. A read_only=0 answer from a site that is not already writable becomes Writable only after recoveryThreshold consecutive successes. A connection error becomes Unreachable only after failureThreshold consecutive failures.

One logic, two directions. Read-only is cheap to be wrong about; writable is expensive.

Which gives you the derivation everyone reaches for and half of everyone gets wrong. Detection delay for a dead site is pollInterval × failureThreshold = 2 s × 3 = 6 s. Three probes, one every 2 s, and the third one returns Unreachable. recoveryThreshold is not a term in that sum. It never was. It gates the opposite transition — the way back to Writable — and adding it to get 10 s is the single most common wrong answer about this operator. The 5 s probe ceiling is not a term either; it bounds one probe, it does not pace the loop.

The backoff, and what it costs

A site that stays down is not probed at full rate forever. Once its failCount climbs past failureThreshold, the wait before its next probe doubles per extra failure — interval := base * time.Duration(1<<uint(backoffFails)) — with the exponent capped at maxPollBackoffExponent = 4 and a 30 s hard cap on top.

Walk it at defaults. failCount 3 gives no backoff; 4 → 2 s × 2 = 4 s; 5 → 8 s; 6 → 16 s; 7 → 2 s × 2⁴ = 32 s, which the 30 s cap clips to 30 s. So roughly half a minute into a site being down, it is probed every 30 s. On the polls in between it is skipped, and it keeps the state it had.

The backoff is per site, and that is the fact to carry. The loop itself still ticks every pollInterval and probes every other site on every tick. pdx has been down for five minutes and is probed every 30 s; iad is still probed every 2 s. Now iad dies. Detection needs three consecutive failures, 2 s apart: 6 s, exactly as if nothing else were wrong. A second fault, at the moment you have no spare site left, is detected as fast as the first.

What the backoff costs is the way back. When pdx returns, its next probe can be up to 30 s away, so the operator can take that long to notice — to rejoin it, or to see that it came back writable. The sidecar’s fencing rules cover a returning old primary in that window, not the operator’s poll. The skip also saves the loop real time: every poll waits for its slowest probe, so a site that hangs rather than refusing costs each poll its full 5 s ceiling — but only until it has been declared Unreachable and its probes start backing off.

Put the pair in your runbook: 6 s to detect any site’s failure, up to 30 s to notice a long-down site has come back. A single number is the wrong shape for this answer. If an older runbook says a second fault takes up to 90 s, it describes a shared backoff the operator no longer uses — see the version appendix, row A2.

One bypass

Last thing. A writable observation on a non-promotable site — anything whose role is not PrimaryCandidate, so reader in playground — skips recoveryThreshold entirely and records Writable on the first poll. The comment in the code is the whole argument: a writable non-promotable site is an immediate safety fact, and authority invalidation must not be debounced behind the normal recovery threshold. Being slow to believe a new primary is prudence. Being slow to notice a reader taking writes is not.

You can now take any sequence of probe results for one site and say which state it lands in and after how many seconds, including under backoff. What you cannot yet say is what the operator does with three states at once — because iad unreachable while pdx and reader are read-only is not three independent facts, it is one row in a table. That table is next.

Mark as read