Credentials And TLS

Bloodraven manages one MySQL user per credential role. spec.credentials is required, and operatorSecretName is its one required role.

Credential roles

Replication account

MYSQL_REPLICATION_USER and MYSQL_REPLICATION_PASSWORD name the account that replicas, clones and old-primary recovery use. Set both or neither: with neither, the operator's username and password replicate. When they name a dedicated account, the first-boot init script creates it on an empty datadir with REPLICATION SLAVE, REPLICATION CLIENT, BACKUP_ADMIN, CLONE_ADMIN. Clones copy it to the other sites. Two cases need manual action:

• On a datadir that already exists, the init script does not run. Create the account on every site with those grants.
• The operator does not rotate the account's password. To rotate it, change it in MySQL and in the Secret together.

The operator refuses a Secret that sets only one of MYSQL_REPLICATION_USER and MYSQL_REPLICATION_PASSWORD (an empty value counts as unset), one whose MYSQL_REPLICATION_USER is the operator's username but whose MYSQL_REPLICATION_PASSWORD differs from password, and any empty username or password (Event SecretMissingKey).

Create Secrets

Backup and restore Jobs use MySQL credential contents exactly, including leading or trailing whitespace and line endings. When using --from-file, include only the intended credential bytes: Kubernetes does not add or remove a trailing newline.

kubectl create namespace orders

kubectl create secret generic orders-mysql-operator -n orders \
  --from-literal=username=bloodraven \
  --from-literal=password='replace-with-random-operator-password' \
  --from-literal=MYSQL_ROOT_PASSWORD='replace-with-random-root-password' \
  --from-literal=MYSQL_REPLICATION_USER=replicator \
  --from-literal=MYSQL_REPLICATION_PASSWORD='replace-with-random-replication-password'

kubectl create secret generic orders-mysql-app -n orders \
  --from-literal=username=orders_app \
  --from-literal=password='replace-with-random-app-password'

kubectl create secret generic orders-mysql-readonly -n orders \
  --from-literal=username=orders_readonly \
  --from-literal=password='replace-with-random-readonly-password'

kubectl create secret generic orders-mysql-monitor -n orders \
  --from-literal=username=orders_monitor \
  --from-literal=password='replace-with-random-monitor-password'

kubectl create secret generic orders-mysql-backup -n orders \
  --from-literal=username=orders_backup \
  --from-literal=password='replace-with-random-backup-password'

cert-manager TLS Secret

spec.tls has one field, secretName, a Secret with ca.crt, tls.crt, and tls.key. The operator does not issue certificates and has no certificates permission; it only reads the Secret. With cert-manager, create a Certificate in the same namespace as the failover group:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: orders-mysql-tls
  namespace: orders
spec:
  secretName: orders-mysql-tls
  duration: 2160h
  renewBefore: 360h
  issuerRef:
    name: mysql-ca
    kind: ClusterIssuer
  commonName: orders.az.example.com
  dnsNames:
    - orders.az.example.com
    - mysql-orders-primary.orders.svc.cluster.local
    - mysql-orders-replicas.orders.svc.cluster.local
    - mysql-orders-iad.orders.svc.cluster.local
    - mysql-orders-pdx.orders.svc.cluster.local

Adopting spec.tls, changing spec.tls.secretName, or rotating the TLS Secret bytes restarts the topology manager so the operator's own MySQL checkers pick up the new material. MySQL pods roll independently via the spec-hash drift path.

The operator uses verified TLS for its MySQL connections when spec.tls.secretName is set. Include ca.crt in the Secret and make sure the certificate covers the primary Service plus every per-site Service (mysql-<group>-<site>.<namespace>.svc.cluster.local) so failover probes, promotion, bootstrap/reclone, and credential reconciliation can validate the server identity.

The per-site Service SAN is required for a second reason: spec.tls also makes the operator set require_secure_transport=ON, so each sidecar's MySQL connection is TLS as well. The sidecar connects over loopback, which appears in no certificate, so it verifies the server against its own site's Service name. Without that SAN the sidecar cannot query MySQL at all: /health returns 503, so the pod never becomes ready, and the self-fencing monitor and the super_read_only safety net keep running but cannot fence anything.

Replication channels between sites use SOURCE_SSL=1 with SOURCE_SSL_CA set to the mounted ca.crt, so a replica accepts only a source certificate that chains to that CA. Put the root CA of the server certificate's chain in ca.crt; an intermediate alone may not verify under MySQL's OpenSSL client, which backup and restore Jobs already rely on as well. Host names are not verified on replication channels, and clone transfers are encrypted without certificate verification.

Backup and restore Jobs get the same Secret mounted at /etc/mysql/tls and connect with ssl-mode=VERIFY_CA against its ca.crt — both the mysqlsh session and the mysqlbinlog | mysql PITR replay. A TLS-enabled Job fails before connecting if the CA path is empty, missing, unreadable, or does not contain usable CA certificates; it never downgrades to unverified TLS.

VERIFY_CA checks the certificate chain, not the host name: the Jobs dial the -internal per-site Service, which the SAN list above does not cover. Replication channels have the same property. Any certificate issued by the CA in ca.crt is therefore accepted as the MySQL server, so issue the MySQL certificate from a CA dedicated to the failover group (for example a namespaced CA Issuer) rather than a ClusterIssuer that also signs certificates for other workloads. Without spec.tls, Jobs use the client default ssl-mode=PREFERRED, which is not authenticated.

Verification Jobs are different. They load the backup into an ephemeral mysqld that listens on loopback without a certificate, so that local connection stays plaintext and does not use the group's TLS material.

Manual TLS Secret

kubectl create secret generic orders-mysql-tls -n orders \
  --from-file=ca.crt=./ca.crt \
  --from-file=tls.crt=./tls.crt \
  --from-file=tls.key=./tls.key

Failover group example

apiVersion: shipstream.io/v1alpha1
kind: MysqlFailoverGroup
metadata:
  name: orders
  namespace: orders
spec:
  image: mysql:9.7
  credentials:
    operatorSecretName: orders-mysql-operator
    appSecretName: orders-mysql-app
    readOnlySecretName: orders-mysql-readonly
    monitorSecretName: orders-mysql-monitor
    backupSecretName: orders-mysql-backup
  tls:
    secretName: orders-mysql-tls
  dns:
    hostname: orders.az.example.com
    ttlSeconds: 60
  sites:
    - name: iad
      serverID: 101
      zone: us-east-1a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: iad
      loadBalancerIP: 10.0.1.1
      storage:
        storageClassName: fast-ssd
        size: 100Gi
    - name: pdx
      serverID: 102
      zone: us-west-2a
      taintNodeSelector:
        bloodraven.shipstream.io/failover-group.orders: "true"
        bloodraven.shipstream.io/site.orders: pdx
      loadBalancerIP: 10.0.2.1
      storage:
        storageClassName: fast-ssd
        size: 100Gi

Client connection examples

mysql --host=orders.az.example.com \
  --user=orders_app \
  --password \
  --ssl-mode=VERIFY_IDENTITY \
  --ssl-ca=/etc/mysql/ca.crt

JDBC:

jdbc:mysql://orders.az.example.com:3306/orders?sslMode=VERIFY_IDENTITY&serverSslCert=/etc/mysql/ca.crt

Go go-sql-driver/mysql should register a TLS config with the CA and use tls=<name> in the DSN. See App Integration for application reconnect guidance.

Rotation expectations

• Updating a referenced Secret is the supported rotation path.
• Replacing a Secret by delete-and-recreate (Argo CD Replace, External Secrets recreation, kubectl replace) is safe. While the operator or TLS Secret cannot be read, the running topology manager keeps its existing connections and logs connection material unreadable; keeping the running topology manager. Once the Secret is back, it restarts only if the bytes changed. A group that has no running manager retries every 30s until the Secret can be read.
• Rotate one role at a time and watch operator Events.
• Keep old app credentials valid until application pods have restarted or connection pools have reconnected.
• Treat the TLS private key and backup encryption passphrases as recovery material.
• Renewing the certificate under the same CA needs no extra step.
• Changing the CA (a new CA key or a different issuer) takes three steps, because replication channels verify the source's certificate against the mounted ca.crt. A replica that has rolled onto a new CA alone stops replicating from a source that still serves a certificate from the old CA. First, publish a ca.crt containing both the old and the new CA and let every pod roll. Then switch tls.crt and tls.key to the new CA. Remove the old CA from ca.crt only after every pod is serving the new certificate.

Related pages

• Security Model
• Production Hardening
• CRD Reference