MSE907 failover study
How it works
On this pageHow a failover unfoldsArchitecture and technologyWalkthrough: build, run, analyse
Guide
How a failover unfolds
Both conditions run on the same platform with two roles, not two clouds: a primary role that serves payments and a standby role that waits. A single direction switch assigns the roles to AKS and EKS, so the same code runs in both directions.
A reconciliation-aware orchestrator
- Detect. The controller on the standby side declares the primary failed after three consecutive failed probes (hysteresis), so a single blip does not trigger a failover.
- Lease. It takes a lease so only one controller can act.
- Fence. It fences the failed primary so that, if it comes back, two sites cannot both accept payments.
- Promote and switch. It promotes the CloudNativePG standby to primary and switches the DNS record (TTL 60 s). A second Debezium connector starts following the new primary.
- Reconcile. It reads both change logs in Kafka, finds acknowledged payments the new primary is missing and replays them with an idempotency key, then re-checks when the failed site returns.
B backup and restore baseline
- Observe. The controller runs in
observemode: it watches and records, it does not act. - Back up. Velero (Kopia file-system backup) keeps the database volume backed up to the standby cloud’s own storage. The backup is at most 120 seconds old at the fault.
- Restore. A driver script restores the latest backup on the standby side and starts the application.
- Switch DNS. The same driver points the DNS record at the restored application.
- Nothing reconciles. Payments made after the backup are gone. The Kafka change log is used only afterwards, to count how many of them could have been recovered.
Why client time is longer than platform time
The platform can have a writable primary and a switched DNS record while the client still fails. The client resolver holds the old answer until its 60 second TTL runs out, and retries have their own backoff. The study therefore reports the platform phases next to the client’s numbers. In A’s AKS-to-EKS pod-kill runs, DNS switched at about 10 s, yet the client first succeeded at about 31 s.
First success is not the same as steady
After the first successful payment a client can fail again. In B the restored standby fails again under the backlog of retries. In A’s EKS-to-AKS partition runs the client succeeds at about 77 s and keeps failing until about 110 s. For that reason the study reports two recovery times:
- Client RTO: first success after the fault.
- Steady RTO: first success after the last relapse failure.
Fixed parameters
| Parameter | Value | Why |
|---|---|---|
| Load | 5 payments per second, one load level | Same offered load for every run; synthetic data only. |
| Detection | Three consecutive failed probes | Hysteresis against false positives. |
| DNS TTL | 60 s | Bounds how long a client can hold the old answer. |
| Backup age at the fault (B) | At most 120 s | A constant design parameter. The loss windows measured in the 20 B runs are 123 to 151 s, because the fault arrives some time after the last backup. |
| Idempotency key | Per payment | Makes replay safe, so duplicates are designed out. Duplicates are 0 in every run by design. |
| Pairs | Run i of A matched with run i of B | Same fault, direction and load. The order within a pair alternates. |
Guide
Architecture and technology
Select any box for what it is. On a phone, swipe the diagram sideways.
| From | To | What travels | How |
|---|---|---|---|
| Load generator | Cloudflare DNS | The client looks up bank.nirmalchathura.dev and gets the address of the active site. | DNS |
| Load generator | Bank app (AKS) | Payments sent to the active site’s load balancer. | HTTPS |
| Bank app (AKS) | PostgreSQL (AKS) | The payment and its receipt are written in one transaction. | PostgreSQL |
| cert-manager (AKS) | Bank app (AKS) | The HTTPS certificate for the bank name is issued and renewed. | ACME |
| PostgreSQL (AKS) | WireGuard (AKS) | The primary database streams its write-ahead log to the standby. | PostgreSQL streaming |
| WireGuard (AKS) | WireGuard (EKS) | Across the WireGuard tunnel to the other cloud. | WireGuard, UDP 51820 |
| WireGuard (EKS) | PostgreSQL (EKS) | The standby applies the stream, so it follows the primary. | PostgreSQL streaming (TLS) |
| PostgreSQL (AKS) | WireGuard (AKS) | Logical decoding of the primary’s changes. | PostgreSQL logical replication |
| WireGuard (AKS) | WireGuard (EKS) | Debezium reads the primary through the same tunnel. | WireGuard |
| WireGuard (EKS) | Debezium (EKS) | The changes arrive at Debezium on the standby side. | TCP 5432 via the gateway service |
| Debezium (EKS) | Kafka (EKS) | Each change is published to Kafka: a second record of every payment. | Kafka |
Roles swap with the direction. Kafka, Debezium and the controller run on the standby side only. Velero storage is used by condition B only, and both stores are in Sydney.
Layout in one paragraph
AKS runs in Azure New Zealand North and EKS runs in AWS ap-southeast-6 (New Zealand). A WireGuard tunnel joins the two networks so the standby can replicate from the primary and the controller can probe it. Cloudflare holds the DNS record that the failover switches, and also hosts two pages outside both clouds: customers.nirmalchathura.dev (a Worker with a Durable Object, always up) and bank.nirmalchathura.dev (up only while the infrastructure runs).
| Component | Version or detail | Role in the study |
|---|---|---|
| AKS | Kubernetes 1.36, New Zealand North | One of the two managed clusters; primary or standby by direction. |
| EKS | Kubernetes 1.31, ap-southeast-6 | The other cluster. Two managed node groups: default and streaming. |
| Terraform | Run from GitHub Actions with OIDC sign-in | Creates clusters, networks, node pools and the Velero storage; also destroys them after the runs. |
| CloudNativePG | PostgreSQL cluster with a live standby | Replication and promotion for condition A. |
| Strimzi and Kafka | Strimzi 0.45.0 (version override on Kubernetes 1.33 and later) | The change log. Only the Kafka, Debezium, controller and reconciler run on the standby side. |
| Debezium | Postgres CDC connectors; a second connector follows the new primary | Records every acknowledged write, a second record of each change. |
| Velero | Server 1.18.4, Azure and AWS plugins v1.14.4, Kopia file-system backup | Condition B backup and restore. Two backup locations on both clusters: Azure Blob (Australia East) and S3 (ap-southeast-2). |
| Chaos Mesh | Pod-kill and network partition | The injected faults, identical for A and B. |
| WireGuard | Cross-cloud tunnel | Private path between the two networks. |
| Cloudflare | DNS, Worker, Durable Object (SQLite) | DNS cutover record; customers and bank pages outside both clouds. |
| Prometheus-format summary | Exported with every run | Per-run metrics alongside result.json. |
| Python | Controller, reconciler, load generator, analysis | All custom logic; the analysis has unit tests for the statistics. |
Where condition B keeps its backups, and why
Each run’s backups go to the standby cloud’s own storage: an S3 bucket when the standby is EKS, an Azure container when the standby is AKS. That makes a restore equally local in both directions. Velero’s Kopia volume backup could not use a bucket in the AWS New Zealand region, so both stores are in Sydney; using the same distance on both sides keeps the baseline fair. Condition A keeps its data in New Zealand. “Reset experiment” swaps the read-write and read-only access modes of the two backup locations when the direction changes, so no reinstall is needed.
A change made during the study. The first batch kept every backup in one Azure container, so a restore on EKS read from another cloud while one on AKS read its own. After the first 20 runs this was changed to the standby’s own storage and all 40 runs were repeated on the final infrastructure. The results on this site are from that final run only.
Guide
Walkthrough: build, run, analyse
Everything is run from the repository’s GitHub Actions tab with “Run workflow”. The workflows are numbered in build order. Nothing is run by hand on a laptop, so run 40 uses exactly the configuration of run 1.
- 01 Terraform. Plan, apply or destroy. Creates the AKS and EKS clusters, networks, node pools (default and streaming on both) and the Velero storage: an Azure storage account and container in Australia East, and an S3 bucket in ap-southeast-2 with an IAM user and policy. Terraform runs from GitHub Actions with OIDC sign-in.
- 00 Build everything. Runs the build steps for one direction in order:
02 Bootstrap platform,03 Deploy cloud-demo(the bank application),04 Deploy experiment platform(Strimzi Kafka, Debezium, CloudNativePG),05 Install Chaos Mesh,06 Deploy failover controller,07aand07b(certificate and HTTPS),08 Velero. - 09 Check platform and 10 Check reconciliation. Confirm clusters, databases, replication, Kafka, connectors, controller mode and DNS, and that reconciliation works, before a run.
- Reset experiment. Returns the system to a known starting state for a direction and a controller mode (
autofor A,observefor B). It deletes the database secrets (after a direction switch a stale TLS secret blocked the database), swaps the Velero backup-location access modes by direction, and a Velero install clears all backup-repository records so a replaced bucket is re-initialised. - Run experiment. One run: a platform check and a smoke payment, the load generator at 5 payments per second, fault injection with Chaos Mesh, the failover (controller for A, restore driver for B), then the load log is closed. The run saves
result.jsonand a Prometheus-format summary as an artifact namedrun-<id>. A run that fails is repeated under the same id; trial runs are never repeated or counted. - Batch workflows. “Run all batches” runs four chunks of 10 runs, one chunk per fault and direction. Inside a chunk the pairs alternate: reset for A, run A-01, reset for B, run B-01, and so on, with the order within a pair alternating.
- Analyse runs. Builds the statistics from the saved results, pooled and by direction:
runs.csv,summary.csv,incidents.csvand a comparison report. They are the files downloadable on this site and the numbers on the dashboard are recomputed fromruns.csv. - Destroy. After the last chunk is saved the clusters are destroyed, because Azure and AWS charge by the hour.
The run record
Each of the 40 runs contributes one row with these fields. The same rows are in the explorer below.
| Field | Meaning |
|---|---|
run_id | Identifier such as A-podkill-03 or B-partition-07. A or B, the fault, then the pair number. Pairs 1 to 5 are AKS to EKS and 6 to 10 are EKS to AKS. A-podkill-06-rerun is the repeat of the failed A-podkill-06. |
condition, failure_type, direction, pair | The design factors. Run i of A is matched with run i of B. |
rto_s | Client RTO: seconds from the fault to the first successful payment the client sees. Empty when no recovery was recorded. |
rto_steady_s | Steady RTO: seconds from the fault to the first success after the last relapse failure. |
phase_writable_s | Platform view: seconds until the new primary database accepts writes. |
phase_dns_switched_s | Platform view: seconds until the DNS record points at the new primary. |
acked | Payments the client was told succeeded. |
lost | Acknowledged payments missing from the final database. |
rpo_writes, rpo_s | Payments lost at recovery, and the loss window in seconds. |
duplicated | Acknowledged payments that appear more than once. Zero in every run, by design (idempotency key). |
delayed_but_recovered | Payments that were missing at recovery but present at the end. |
clean | Acknowledged payments that are present exactly once. Zero together with acked > 0 means the run was not classified. |
relapse_failed_attempts | Failed payment attempts after the first recovery. |
failed_never_acked | Attempts that never received an acknowledgement. |
Downloads: runs.csv, summary-pooled.csv, incidents.csv, comparison-pooled.md, comparison-aks-to-eks.md, comparison-eks-to-aks.md, artifacts.tsv. runs.csv and artifacts.tsv include the repeat of A-podkill-06. The summary, incident and comparison files were produced before that repeat, so their pod-kill figures use 9 valid pairs; the site and runs.csv use 10.