Deployment API

The optional /deploy/v1 API coordinates application deployments with a MySQL failover group. It provides a group-wide migration mutex and renewable holds on planned disruption. It does not run migrations, grant MySQL credentials, or delay emergency failover.

Transport and chart values

The API is disabled by default and requires leader election (leaderElection.enabled=true, the chart default). It shares the existing keyring escrow TLS listener, configured by BLOODRAVEN_ESCROW_TLS_ADDR (default :8443). Plain HTTP on :8082 never serves /deploy/v1, and there is no plaintext fallback. Non-leaders return 503 {"error":"not_leader"}.

auxiliary:
  service:
    enabled: true
    name: bloodraven
  escrowTLS:
    enabled: true
    port: 8443
    existingSecret: bloodraven-auxiliary-tls
  deployAPI:
    enabled: true
    audience: bloodraven-deploy
    networkPolicy:
      enabled: true
      additionalIngress: []

The existing Secret must be in the operator release namespace and contain tls.crt and tls.key. The certificate SAN must cover the auxiliary Service FQDN. Deployment clients must trust its issuer and validate the hostname; encrypted MySQL sidecars must also trust it through their group TLS configuration. Enabling the TLS listener does not require enabling MySQL encryption at rest.

For a release in namespace bloodraven, the example publishes https://bloodraven.bloodraven.svc.cluster.local:8443/deploy/v1.

NetworkPolicy boundaries

The built-in TLS ingress rule requires both a pod label bloodraven.shipstream.io/deploy-client: "true" and a namespace label shipstream.io/mysql-client: "true". The two selectors are in one peer entry, so they are an AND, not an OR. With the default port, this admits TCP 8443.

The same listener serves /keyring/escrow. A separate peer admits pods labelled app.kubernetes.io/name: mysql and app.kubernetes.io/managed-by: bloodraven from all namespaces, preserving operator-managed sidecar escrow pushes. NetworkPolicy cannot distinguish URL paths: those sidecars can reach the deployment listener but still need deployment ServiceAccount authorization to use it. Deployment clients similarly cannot use escrow without its per-site credential.

Selecting operator pods isolates all their ingress, not only TLS. This template intentionally leaves pod ports 8080 (metrics), 8081 (probes), and 8082 (auxiliary status, self-fencing, PITR retention, WebSocket) reachable from any source. It changes TLS reachability without silently breaking existing flows. It is not a complete operator ingress lockdown. To restrict those three ports, replace this policy with a site-specific policy that retains the required sources; adding a restrictive policy cannot narrow an existing allow rule. Policies are additive, and any other broad allow policy can also defeat this template's TLS restriction. No egress policy is created.

Custom escrow clients with different labels can be admitted with additional rules, for example:

auxiliary:
  deployAPI:
    networkPolicy:
      additionalIngress:
        - from:
            - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: database-system
              podSelector:
                matchLabels:
                  app.kubernetes.io/name: escrow-client
          ports:
            - protocol: TCP
              port: 8443

Additional rules use Kubernetes NetworkPolicy syntax and do not inherit the TLS port. Match any port override explicitly. Protect namespace and pod label assignment through RBAC; labels are a reachability control, not authentication. Verify access with a CNI that enforces NetworkPolicy, including a denied client missing either deployment label and an allowed sidecar escrow push.

Authentication and authorization

Every endpoint requires Authorization: Bearer <projected ServiceAccount token>. The operator uses authentication.k8s.io/v1 TokenReview with the configured audience and accepts the authenticated identity system:serviceaccount:<namespace>:<name>. The default audience is bloodraven-deploy, not the Kubernetes API's default audience. Clients must reread rotating projected tokens rather than cache them for the Pod lifetime.

The calling namespace and ServiceAccount pair must be declared in MysqlDatabase.spec.deploymentClients[], on a database whose spec.groupRef.name matches the path's {group}. The database and group are in the same namespace. The MysqlDatabase resource name, not spec.databaseName, is the lease's instance identity.

This example is a spec fragment for an existing MysqlDatabase named orders-app in the group's namespace:

spec:
  groupRef:
    name: orders
  deploymentClients:
    - namespace: application
      serviceAccount: application-deploy

deploymentClients is optional and defaults to no callers. Each entry requires namespace and serviceAccount; the list is keyed by namespace, with one declared ServiceAccount per namespace. Authority to edit a database's deployment clients is authority to grant access to its group's deployment coordination. A deployment caller does not need generic permission to patch MysqlFailoverGroup or read Secrets.

A Pod's token projection can use:

volumes:
  - name: deployment-identity
    projected:
      sources:
        - serviceAccountToken:
            path: token
            audience: bloodraven-deploy
            expirationSeconds: 3600

Mount that volume read-only into the deployment client. The operator ClusterRole grants create on tokenreviews.authentication.k8s.io and get/list/watch/create/update/patch/delete on leases.coordination.k8s.io. The latter covers durable deployment coordination as well as leader election.

Unauthenticated requests return 401. An authenticated but unauthorized caller receives 403 with {"error":"forbidden"}, without revealing whether the group exists. A lease token is an additional ownership credential for renewal and release, never a replacement for ServiceAccount authentication.

Successful TokenReviews are cached for 15 seconds by token hash, with at most 1,024 entries. Authorization uses uncached database resources on every request. If the caller is authorized for identically named groups in multiple namespaces and the request cannot identify one group through its instance, it returns 403 forbidden. Use distinct deployment ServiceAccounts for those groups.

Group observation

GET /deploy/v1/groups/{group} returns 200 with the authorized group's state:

{
  "group": "orders",
  "namespace": "bloodraven",
  "activeSite": "iad",
  "topologyGeneration": 17,
  "health": {
    "ready": true,
    "degradedReason": "",
    "replicationRunning": true,
    "replicationLagSeconds": 0.4,
    "recoveryPending": false
  },
  "operations": {
    "plannedFailover": {"phase": "", "reason": ""},
    "updatePhase": "",
    "restoreInPlace": "",
    "dragonflyUpgrade": ""
  },
  "stable": true,
  "unstableReason": "",
  "leases": []
}

stable=false when an operation phase is active, the group is Degraded, SplitBrain, or NoPrimary, recovery is pending, or replication is not running. Lag alone does not set stable=false; the caller must enforce its own lag threshold. A healthy GET is an observation, not permission to continue after losing a lease.

During an unconfirmed topology transition or an operation reservation not yet persisted to status, GET and renewal can return 423 unstable rather than expose stale authority. A client must not treat that response as a successful renewal or extend its locally recorded expiry. This transient response never permits migration work to continue beyond the last confirmed deadline. Resume observation only within that deadline; 409 revoked remains the terminal ownership-loss response after revocation is persisted.

Grant or re-grant a lease

POST /deploy/v1/groups/{group}/leases accepts JSON:

{"kind":"migration","operationId":"11111111-1111-4111-8111-111111111111","instance":"orders-app","ttlSeconds":30}

Only one migration lease can be held per group. The model permits multiple failover-hold leases, but this implementation grants a hold only to the current migration holder. Acquire migration first, then failover-hold with the same instance and operation ID.

expiresAt is a timestamp; topologyGeneration is an integer. token is a random secret returned only in the grant response. Durable storage holds only its hash, so the operator cannot retrieve and return the original token after a lost response or restart. A same-operation POST therefore returns a new token and invalidates the previous one. Serialize re-grants with renewal and atomically replace the client's stored token. Do not use POST to bypass a revocation or automatically resume uncertain DDL.

Renew a lease

PUT /deploy/v1/groups/{group}/leases/{kind}/{operationId} accepts {"token":"<lease-token>","ttlSeconds":30}. The path identifies the existing lease; TTL has the same [5, 120] clamp.

Authentication and authorization checks still apply to every renew request.

Release a lease

DELETE /deploy/v1/groups/{group}/leases/{kind}/{operationId} requires the ServiceAccount bearer token and X-Lease-Token: <lease-token>. Success returns 204 without a body. Release is idempotent: an absent lease also returns 204. An existing lease still requires the matching ownership token; ServiceAccount authorization is not bypassed for an absent lease.

Durability and fencing

Deployment leases are persisted as coordination.k8s.io/Lease objects in the API server, separate from the operator's leader-election lease. Restarting the operator does not release or reset them. Expiry is enforced using the operator clock: now > expiresAt ends the lease's effect and a subsequent renew returns 404. Keep operator node clocks synchronized.

The stored spec.leaseDurationSeconds is a whole-second projection with a minimum of 1, including for expired, released, and revoked records, because Kubernetes rejects zero durations. The deployment record's exact expiresAt, state, and topology generation remain authoritative. The minimum adds no grace period, does not reactivate terminal records, and does not delay release or revocation. Terminal holds no longer block planned admission once their state is persisted; normal safety checks still apply.

Per-operation revocation tombstones remain until the group is deleted, so a surviving old client receives 409 even after another operation acquires the migration mutex. These resources contain token hashes, never raw tokens. Account for their storage growth when sizing and backing up the Kubernetes API server.

MysqlFailoverGroup.status.topologyGeneration is persisted and rehydrated after restart. It increments on every authoritative active-site change, including planned promotion, emergency failover, ordered-update failover, restore-in-place promotion, and bootstrap completion. A generation change revokes all group leases with topology_changed; it is not a pod restart counter or the CR's metadata.generation.

An unexpired failover-hold defers planned failover with phase Deferred, reason DeploymentHold, and retryAfter equal to the earliest hold expiry. The message identifies the blocking operationId and instance. Revalidation after expiry proceeds through normal safety checks. Ordered update and restore-in-place also defer behind a hold. A migration lease alone does not hold planned operations. Emergency failover, returning-old-primary fencing, and primary reassertion never consult deployment leases.

Administrators can explicitly revoke all group deployment leases using bloodraven.shipstream.io/revoke-deployment-leases; renewals report operator_revoked. See deployment coordination operations for the approved intervention procedure. Never delete the leader-election Lease or edit topology generation to clear a deployment hold.

Clients renew both leases every ttl/3. A 404 or 409 renewal response, or any generation differing from the grant generation, means lost authority: stop the deployment heartbeat and do not reconnect to a new primary and continue migrations. Before DDL, perform only the application's fenced abort/cleanup if still authorized. At or after DDL, stop further database writes and require manual schema verification. Release the hold on every failure path; release the migration mutex after the deployment's completion or failure state has been safely recorded. A TTL is not an exactly-once DDL guarantee.

Metrics, events, and logs

Unauthorized grants that cannot resolve an authorized group use group="" to prevent arbitrary request paths from creating unbounded metric series.

Revocation emits Kubernetes Event DeploymentLeaseRevoked on the MysqlFailoverGroup. Planned holds are visible through PlannedFailoverDeferred, Deferred/DeploymentHold status, and the deferral counter. Correlate events and the deployment's own DDL boundary before deciding whether retry is safe.

Every request emits one structured log with msg="deploy api" and fields handler, group, instance, namespace, operationId, status, duration_ms. duration_ms is the contract's explicit exception to camelCase; retain its spelling. Tokens, Authorization headers, and lease token hashes are not log fields. See the log schema contract.