App Integration

This page explains how applications connect to a MySQL failover group, optional Dragonfly cache/session endpoint, and handle failover events.

Application checklist

• Use mysql-<group>-primary or the external active-site DNS name for writes.
• If spec.dragonfly.enabled=true, use <group>-dragonfly for Redis-compatible cache/session traffic.
• Keep DNS and connection-pool lifetimes short enough for your failover target.
• Retry connection establishment and failed transactions that are safe to retry.
• Configure Redis/Dragonfly clients to reconnect after socket drops; planned failover may intentionally close old-master client connections.
• Do not pin writes to a site-local Service unless you are implementing warm-standby logic deliberately.
• Alert on application write failures, connection pool exhaustion, and repeated read-only errors.

DNS and pool guidance by runtime

Connection examples

Go DSN shape:

orders_app:password@tcp(mysql-orders-primary.orders.svc.cluster.local:3306)/orders?parseTime=true&timeout=5s&readTimeout=10s&writeTimeout=10s

JDBC:

jdbc:mysql://mysql-orders-primary.orders.svc.cluster.local:3306/orders?connectTimeout=5000&socketTimeout=10000

Node.js mysql2:

import mysql from 'mysql2/promise';

const pool = mysql.createPool({
  host: 'mysql-orders-primary.orders.svc.cluster.local',
  user: process.env.MYSQL_USER,
  password: process.env.MYSQL_PASSWORD,
  database: 'orders',
  waitForConnections: true,
  connectionLimit: 10,
  connectTimeout: 5000,
});

Rails:

production:
  adapter: mysql2
  host: mysql-orders-primary.orders.svc.cluster.local
  username: <%= ENV.fetch("MYSQL_USER") %>
  password: <%= ENV.fetch("MYSQL_PASSWORD") %>
  database: orders
  reconnect: true

Connection endpoints

For a MysqlFailoverGroup named orders in namespace default, the operator creates these Services:

Client-facing site Services expose only the named MySQL port. The sidecar is not externally exposed and never receives a NodePort. Internal Services are always ClusterIP, set publishNotReadyAddresses: true, and do not select on the reader's healthy label. This preserves the operator's route for probes, replication repair, clone, backup, and sidecar peer traffic even while the reader pod is not ready or its application endpoint has been shed.

When spec.tls is enabled, operator Go clients dial the internal Service but continue verifying the existing client-facing site hostname as TLS ServerName; existing certificates do not need to add the -internal name. Strict hostname verification configured inside MySQL-native replication or clone is separate and must trust the internal Service hostname.

When spec.dragonfly.enabled=true, Bloodraven also owns the cache/session endpoints for the same failover group:

Do not deploy separate tenant-chart Dragonfly CRs for a failover group where Bloodraven has spec.dragonfly.enabled=true. Bloodraven is the owner of the StatefulSets, Services, replication role labels, active Service selector, and planned/emergency promotion status. Tenant charts should either omit their Dragonfly templates or gate them behind a disabled-by-default value, then point application settings at the Bloodraven-created orders-dragonfly Service.

Recommended app-facing environment variables:

- name: REDIS_HOST
  value: orders-dragonfly.default.svc.cluster.local
- name: REDIS_PORT
  value: "6379"
- name: REDIS_PASSWORD
  valueFrom:
    secretKeyRef:
      name: tenant-dragonfly
      key: password

Redis/Dragonfly clients must reconnect after connection drops. During planned failover Bloodraven may remove the old master from the active Service and issue CLIENT KILL TYPE NORMAL so clients reconnect to the newly promoted site instead of holding stale sockets.

Snapshot-restore upgrade

By default (spec.dragonfly.upgradeStrategy: Rolling) a change of spec.dragonfly.image rolls the Dragonfly StatefulSets like any other template change: replicas first, then the active site after a promotion, so cache contents survive only through replication. For image upgrades that can tolerate a short cache/session outage but should keep the active site's data, set upgradeStrategy: SnapshotRestore and configure native snapshot restore:

spec:
  dragonfly:
    image: docker.dragonflydb.io/dragonflydb/dragonfly:<target>
    upgradeStrategy: SnapshotRestore
    snapshot:
      dir: s3://tenant-dragonfly/orders/prod
      serviceAccountName: dragonfly-backup
      credentialsSecretName: dragonfly-s3
      s3EndpointURL: http://rustfs.default.svc.cluster.local:9000

Bloodraven passes snapshot.dir to Dragonfly as --dir, turns snapshot.s3EndpointURL into Dragonfly's --s3_endpoint (the host and port) and --s3_use_https (false for http://, true for https://; omit the URL for AWS S3), projects snapshot.credentialsSecretName as AWS credential environment variables when set, and assigns snapshot.serviceAccountName to the Dragonfly pods so cloud IAM systems such as EKS IRSA can grant bucket access. Treat this as session-continuity insurance for planned maintenance, not as the durable backup for application state. Admission refuses SnapshotRestore without snapshot.dir.

The request is the spec itself: edit spec.dragonfly.image (in Git, if you use GitOps). There is no annotation, and the operator never writes the spec. Bloodraven compares spec.dragonfly.image with the image each site's live StatefulSet runs; when one differs it starts a run, which saves a snapshot on the MySQL active site, restarts that site on the new image restoring from the snapshot, then moves the other sites and re-attaches them as replicas. A site whose StatefulSet does not exist yet has no live image, so a new group never starts a run. Under SnapshotRestore the ordinary render never moves an image: only the run does. The render still rolls out every other change first (the snapshot block's --dir and S3 arguments, the credential environment, the ServiceAccount), each site keeping its running image, and a run starts only once every site runs those settings: the snapshot is saved by the running pod, and the run changes nothing but the image. So an edit that changes the image and adds or fixes the snapshot block in one commit first rolls the snapshot settings, then upgrades.

Track the run in status.dragonfly.upgrade (phase, sourceImage, targetImage, specDigest, times, reason, message); status.dragonfly.image shows the image of the active Dragonfly site's StatefulSet. During SavingSnapshot, UpdatingActive, and WaitingForActiveRestore, Bloodraven intentionally removes the active Dragonfly endpoint so clients see a planned cache outage instead of writing to a half-restored pod. A run interrupted by an operator restart continues from its recorded phase; a run whose active site already runs the new image takes no second snapshot and does not restart the active: it starts at WaitingForActiveRestore, so no replica is touched before the active has loaded.

A run waits, without moving any site, while the sites do not yet run spec.dragonfly's settings other than the image (above), while status.activeSite is empty, while a planned failover run or automation hold owns the sites, while the Dragonfly master is not on the MySQL active site, or while any site's Dragonfly StatefulSet has a selector the operator does not render (refused as WorkloadSelectorMismatch, Degraded=True with a Warning Event, as the ordinary render refuses it); a Normal DragonflyUpgradeDeferred Event names the cause, and the run starts once it clears. A run already in flight checks every site's StatefulSet, uncached, before each step. A refused one ends the run phase: Failed with reason: WorkloadSelectorMismatch before it saves a snapshot, deletes a pod or changes an image, so a refused StatefulSet is never touched. There is one exception: once the active site's image has moved (late UpdatingActive and WaitingForActiveRestore), ending the run gives the active's traffic back, so it waits until the active has loaded its snapshot. A refused replica is recorded at once (Degraded=True), and the run ends when it reaches ReattachingReplicas, the step that touches it, after the active restored. A refused active is recorded at once too; the run keeps the active's traffic off, holds, and ends once the active answers INFO persistence as loaded (deleting the refused StatefulSet instead re-creates it on the target image and the run continues). Either way a loading Dragonfly never gets traffic back. Fix the refusal by deleting the refused StatefulSet (kubectl delete statefulset <group>-dragonfly-<site>), one site at a time; the operator re-creates it with the rendered selector (see Labels). Then retry a failed run as below.

A StatefulSet deleted while a run is in flight does not stall the run. The run re-creates it on the target image and continues where its own steps would leave the site on that image anyway: a replica in any phase, and the active between the saved snapshot and its restore (UpdatingActive, WaitingForActiveRestore), where the new pod restores from the snapshot as the restarted one would. An active deleted before its snapshot was saved (Pending, SavingSnapshot) or after it restored (ReattachingReplicas) has lost the data the run moves. The run then ends phase: Failed with reason: ActiveWorkloadDeleted, and the ordinary render re-creates the StatefulSet.

A run moves the sites for the spec.dragonfly it started with (its specDigest). Changing spec.dragonfly during a run (reverting or moving the image, switching upgradeStrategy to Rolling, editing the snapshot block or any other field) ends it phase: Failed with reason: SpecChanged, and the operator then decides again for the new spec: under Rolling the remaining sites roll the ordinary way. If the active site's image had already moved, the run first keeps the active's traffic off until it has loaded its snapshot. Disabling or removing spec.dragonfly ends the run with reason: DragonflyDisabled, and the Dragonfly workloads are removed. A run that has not finished an hour after it started (an active that never restores, a replica that never syncs) ends with reason: TimedOut, which gives the group's reconcile back. Whatever ends a run, an active site the run restarted that does not yet answer INFO persistence as loaded keeps its traffic off: the run's message says so, and the pod is marked bloodraven.shipstream.io/dragonfly-load-hold, and the operator's label sweep, which re-probes it every couple of seconds, gives the site its traffic (and the master role) back only once it answers as loaded, probing it with the port and password its pod was started with. While the mark is there the sweep also takes back traffic another writer (a promotion restoring its source) gave the pod. The Failed record stays while that hold lasts.

Each upgrade step takes the group's Dragonfly gate, the same one Dragonfly promotions hold, so it never relabels, re-points or restarts Dragonfly in the middle of a promotion. If MySQL fails over, or a promotion that was already running when the upgrade started moves the Dragonfly master away from the site the upgrade retained, the upgrade is abandoned with phase: Failed and reason: ActiveSiteChanged, and the routing labels are left to that promotion.

Retrying a failed run. A run that ends Failed (a failed SAVE, an abandoned run) is not restarted on its own: each start sheds the active site's cache traffic, so the operator waits for you, and emits the DragonflyUpgradeFailed Warning once. The record stays, with the specDigest of the spec.dragonfly it ran against. Any change to spec.dragonfly (a new image, a fixed snapshot block, a strategy change) starts a new run. Before retrying, check which image each site's StatefulSet runs:

kubectl get statefulset -n <ns> -o custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image' | grep -- '-dragonfly-'

• Every site still runs the source image (the run failed before any site moved, for example a failed SAVE). If the fix was outside the spec (the contents of the credentials Secret, the bucket), retry in two steps: set spec.dragonfly.image back to the image the sites run, which ends the drift and removes the Failed record, then set it forward again.
• Some sites already run the target image (for example ActiveWorkloadDeleted after the active restored, or a run abandoned after the active moved). Do not set the image back: with mixed images that is not a no-op but a new run toward the source image, which downgrades the sites that moved (if MySQL has since moved to a site on the source image, the run skips the active and downgrades the upgraded peers). Keep the target image. If the cause is in spec.dragonfly (the snapshot block), fix it there: that change re-arms the run toward the same target. If it was outside the spec, switch upgradeStrategy to Rolling to finish the remaining sites the ordinary way.

To give up on the snapshot path, switch upgradeStrategy to Rolling: the pending change then rolls the ordinary way.

Default connection pattern

For workloads that should move with the active database site:

# Writes
mysql-orders-primary.<namespace>.svc.cluster.local:3306

# Reads
mysql-orders-replicas.<namespace>.svc.cluster.local:3306

The -primary Service selector is updated by the operator whenever a failover occurs. Use this for workers, cron jobs, runners, and other write-dependent workloads that Bloodraven should evict and reschedule to the active site.

For warm-standby web pods that intentionally keep one deployment per site, use the site-local pattern instead.

Failover strategies

Bloodraven supports two strategies for handling application failover. Choose the one that fits your deployment model.

Strategy 1: Taint-based failover

Best for applications that run on dedicated or shared nodes and should be fully migrated when the database fails over.

How it works:

1. The operator taints nodes at the old active site with <namespace>.db-readonly.bloodraven.shipstream.io/<group>=true:NoExecute, where <namespace> and <group> are the MysqlFailoverGroup's namespace and name (for orders in default: default.db-readonly.bloodraven.shipstream.io/orders). The namespace is part of the key so that two groups of the same name in different namespaces never share a taint or each other's tolerations
2. Kubernetes evicts pods that do not tolerate this taint
3. Those pods are rescheduled to the new active site's nodes (which have the taint removed)

Application requirements:

• Deploy on nodes selected by spec.sites[].taintNodeSelector, such as bloodraven.shipstream.io/failover-group.orders=true and bloodraven.shipstream.io/site.orders=iad
• Do not add a toleration for <namespace>.db-readonly.bloodraven.shipstream.io/<name>:NoExecute (your own group's taint)
• On shared nodes, add tolerations for other groups' taints (see Placement Contract)
• Write tolerations against the namespace-scoped key. A toleration or taint for the key of earlier releases, shipstream.io/db-readonly-<group>, does not match it. Taints an earlier operator left on nodes can be removed with kubectl taint nodes --all shipstream.io/db-readonly-<group>-
• Use mysql-orders-primary for writes (the Service follows the active site)

Example pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: order-processor
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: bloodraven.shipstream.io/failover-group
                operator: In
                values: ["orders"]
  containers:
    - name: app
      image: myapp:latest
      env:
        - name: MYSQL_HOST
          value: mysql-orders-primary.default.svc.cluster.local
        - name: MYSQL_PORT
          value: "3306"
  # No toleration for default.db-readonly.bloodraven.shipstream.io/orders -- pod will be evicted on failover
  # On shared nodes, add tolerations for OTHER groups' taints:
  # tolerations:
  #   - key: default.db-readonly.bloodraven.shipstream.io/inventory
  #     operator: Exists
  #     effect: NoExecute

Strategy 2: Service-based failover

Best for applications that are deployed independently per site and only need to follow the database.

How it works:

1. Each site's application deployment connects to mysql-orders-primary for writes
2. On failover, the -primary Service selector shifts to the new active site
3. Applications at both sites see the endpoint change through normal Kubernetes Service discovery

Application requirements:

• Connect to mysql-orders-primary for writes
• Optionally connect to mysql-orders-replicas for reads
• Handle brief connection errors during failover (seconds, not minutes)

Strategy 3: Site-local warm standby

Best for web/API pods that run in every site and should serve a maintenance page when their local MySQL is not writable. This avoids cross-site app-to-database traffic and lets each site's frontend reflect the state of the local database instance.

How it works:

1. Each site deployment connects to its own site Service: mysql-orders-iad in iad, mysql-orders-pdx in pdx.
2. The app checks @@global.read_only or handles MySQL error 1290.
3. If local MySQL is read-only, the app serves maintenance / read-only mode instead of sending writes across the WAN.
4. User-facing DNS or traffic steering moves users to the active site; Bloodraven still updates the failover group's external DNS record for clients that use spec.dns.hostname directly.

Application requirements:

• Run one deployment per site, pinned by node affinity or your GitOps generator.
• Set MYSQL_HOST to the site-local Service, not mysql-orders-primary.
• Do not tolerate your own group's read-only taint (<namespace>.db-readonly.bloodraven.shipstream.io/<group>) for workers that must move; web pods that intentionally stay warm in both sites may tolerate it and rely on the local read-only check.
• Keep database writes behind retry/maintenance handling. A local read-only response is expected during normal failover.

Example per-site deployment fragment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-web-iad
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: bloodraven.shipstream.io/site
                    operator: In
                    values: [iad]
      tolerations:
        - key: default.db-readonly.bloodraven.shipstream.io/orders
          operator: Exists
          effect: NoExecute
      containers:
        - name: web
          image: example/orders-web:latest
          env:
            - name: MYSQL_HOST
              value: mysql-orders-iad.default.svc.cluster.local
            - name: MYSQL_PORT
              value: "3306"

For an Argo CD ApplicationSet or Helm chart, make site a parameter and derive both the node selector and MYSQL_HOST from it:

env:
  - name: MYSQL_HOST
    value: mysql-orders-{{ .Values.site }}.default.svc.cluster.local

Use the moving mysql-orders-primary Service for background workers in the same application if those workers should only run where writes are accepted.

Handling failover in application code

During a failover, there is a brief window (typically under 10 seconds) where:

1. The old primary becomes read-only
2. The new primary is being promoted
3. DNS and Service endpoints are being updated

Applications should:

• Retry failed write operations with backoff
• Expect read-only errors (Error 1290: The MySQL server is running with the --read-only option) and retry after a short delay
• Not cache DNS indefinitely -- respect TTLs so that DNS changes propagated by external-dns take effect

External access via DNS

For applications outside the Kubernetes cluster, the operator manages a DNSEndpoint CR whose A record points spec.dns.hostname to the active site's load balancer IP (spec.sites[].loadBalancerIP). external-dns watches this CR and syncs it to your configured DNS provider. When a failover occurs, the operator updates the DNSEndpoint to point to the new active site. Changing spec.dns.hostname or spec.dns.ttlSeconds on a live group rewrites that same object on the next poll — the previous name is replaced, not left alongside the new one.

Applications should CNAME their own DNS names to the failover group's hostname (i.e. spec.dns.hostname). For example, if the failover group has dns.hostname: orders.az.example.com, an application might create:

orders.myapp.example.com  CNAME  orders.az.example.com

External applications should:

• CNAME to the group's spec.dns.hostname (not hardcoded IPs)
• Respect DNS TTLs
• Implement connection retry logic

Read replicas

The mysql-orders-replicas Service selects pods with:

• bloodraven.shipstream.io/role: replica
• bloodraven.shipstream.io/healthy: "true"

A replica is considered healthy when:

• It is reachable
• It is replicating (replication: Healthy or Lagging in its status.sites[] entry)
• Its replication lag is within spec.replication.maxLagSeconds

If no healthy replicas exist, the Service will have no endpoints. Applications reading from the replicas Service should handle this by falling back to the primary or surfacing an error.

Site-specific reader pools

A site with role: ReadOnly is a non-promotable follower intended for a local application read pool. Connect to its client-facing site Service, for example mysql-orders-reader.default.svc.cluster.local:3306. Never send writes to it: readers are held in super_read_only, are excluded from promotion and active DNS, and do not participate in application node taints.

The reader Service publishes an endpoint only when the latest debounced topology snapshot confirms all of the following:

• MySQL is read-only.
• Both replication I/O and SQL threads are healthy.
• sourceConvergenceState is Converged and the canonical Source_Host directly names the uniquely confirmed active primary.
• Replication lag is known and is less than or equal to spec.replication.readOnlyMaxLagSeconds. When omitted, this inherits maxLagSeconds; explicit zero means only zero lag is accepted.

If any check fails, Kubernetes removes the reader from the client Service at the normal topology poll cadence. Existing connections are not an operator-managed proxy and are not forcibly killed; applications must handle connection errors and retry. Direct pod access and the internal Service remain available to the operator for diagnosis and recovery. Reader failures remain visible in site status and metrics but do not set the failover group's shared Ready=False or Degraded=True conditions.