# MSE907 failover study: full text of https://mseproject.nirmalchathura.dev

This file is the text of the whole site, generated at build time. Source: Unagallage Nirmal Chathura Unagalle, MSE907 capstone, Yoobee (Master of Software Engineering). Data: 40 measured runs of two recovery strategies for a stateful PostgreSQL workload across Azure AKS and AWS EKS. All data is synthetic.

MSE907 capstone · Yoobee · 40 measured runs · 20 H3 runs

# What happens to every payment when a cloud fails
Two ways to recover a PostgreSQL database across Azure AKS and AWS EKS, tested with the same injected faults in both directions: a live standby with a reconciling controller (A) against backup and restore (B). Drag the timeline to watch one failover, then explore every run.
Fault

Direction

Replay

A · Serving payments
0 acknowledged payments lost
The standby already holds every acknowledged payment, so promotion loses none. Median over 5 runs: new primary writable at 9.1 s, DNS switched at 9.5 s, client first succeeds at 31.1 s.

B · Serving payments
317 payments only in the primary since the last backup
The last backup is 123 s before the fault and the load runs at 5 payments per second, so everything since then exists only on the failed primary. The restore is writable at 145.3 s and the client first succeeds at 181.8 s, then relapses until 240.8 s. Median over 5 runs.

Every figure in this timeline is the median of the measured runs for the selected cell (five matched pairs). The shaded stretch is the time the platform is ready but the client is still failing, which the cached DNS answer (60 s TTL) and retries explain.

Overview

## The question and the short answer
When one cloud fails, how many acknowledged payments does a stateful workload lose, and how long does a customer wait? This study measures it on two managed Kubernetes services, Azure AKS and AWS EKS, with identical injected failures.
Two recovery strategies run the same bank-payment application. Condition A is the proposed orchestrator: a live PostgreSQL standby, a controller that detects the failure, fences the failed site, promotes the standby and switches DNS, and a reconciler that reads a change log to find and replay payments the standby is missing. Condition B is the common practice: a Velero backup at most 120 seconds old, restored on the other cloud. Velero is only the baseline here. It is not the mechanism being proposed.
Each strategy faces two faults (a pod-kill and a network partition) in both directions (AKS to EKS and EKS to AKS), with five matched pairs per cell: 2 × 2 × 2 × 5 = 40 runs.

20 of 20B pairs where B lost payments. A lost a payment in 0 of 20. No run duplicated a payment. A-podkill-06 is left out because its fault had no effect.
630 / 709Median acknowledged payments lost per B run, pod-kill / partition. A lost 0.
31 vs 182 sMedian client RTO under pod-kill, A vs B (10 pairs, exact p = 0.0020).
76 vs 170 sMedian client RTO under partition, A vs B (10 pairs, exact p = 0.0020).

### Research questions and hypotheses

- RQ1, data loss. By how much does A reduce the share of runs with a lost or duplicated acknowledged write? H1: A’s incident rate is lower.
- RQ2, time. What is the RTO and RPO trade-off? H2: median RTO and RPO are lower for A.
- RQ3, return. Can writes lost during failover be found and recovered once the failed site returns? H3: every acknowledged write that is in the change log can be restored.
- RQ4, direction. Do the answers to RQ1 and RQ2 hold when the direction is swapped? H4: the sign of the difference does not depend on direction.

### The four gaps the study fills

- G1, write-level accounting. No study reports whether each acknowledged write was still alive at the end.
- G2, client-observed recovery time. Recovery is reported from the server side, not what a client sees through DNS caching and retries.
- G3, direction and return. Failover is studied, but not reconciliation when the failed site comes back.
- G4, governance of the second copy. Logs, standbys and backups are copies of data, and who governs them is not discussed.
What is and is not shown. H1, H2 and H4 are tested by the 40 runs below. H3 (replay of lost writes) is tested separately by ten matched pairs in a new fault mode that creates a real gap: replay recovered every acknowledged payment in 10 of 10 pairs, and the replay-off control lost them (see “Findings and caveats”). Condition B is deliberately naive, and the author built both conditions; Section “Findings and caveats” lists the limits.

NextHow it works (https://mseproject.nirmalchathura.dev/guide)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)

MSE907 failover study

# How it works
How a failover unfolds, the architecture and technology, and the build, run and analyse walkthrough.
On this pageHow a failover unfolds (#mechanics)Architecture and technology (#architecture)Walkthrough: build, run, analyse (#walkthrough)
Guide

## How a failover unfolds
Both conditions run on the same platform with two roles, not two clouds: a primary role that serves payments and a standby role that waits. A single direction switch assigns the roles to AKS and EKS, so the same code runs in both directions.

### A reconciliation-aware orchestrator

- Detect. The controller on the standby side declares the primary failed after three consecutive failed probes (hysteresis), so a single blip does not trigger a failover.
- Lease. It takes a lease so only one controller can act.
- Fence. It fences the failed primary so that, if it comes back, two sites cannot both accept payments.
- Promote and switch. It promotes the CloudNativePG standby to primary and switches the DNS record (TTL 60 s). A second Debezium connector starts following the new primary.
- Reconcile. It reads both change logs in Kafka, finds acknowledged payments the new primary is missing and replays them with an idempotency key, then re-checks when the failed site returns.

### B backup and restore baseline

- Observe. The controller runs in observe mode: it watches and records, it does not act.
- Back up. Velero (Kopia file-system backup) keeps the database volume backed up to the standby cloud’s own storage. The backup is at most 120 seconds old at the fault.
- Restore. A driver script restores the latest backup on the standby side and starts the application.
- Switch DNS. The same driver points the DNS record at the restored application.
- Nothing reconciles. Payments made after the backup are gone. The Kafka change log is used only afterwards, to count how many of them could have been recovered.

### Why client time is longer than platform time
The platform can have a writable primary and a switched DNS record while the client still fails. The client resolver holds the old answer until its 60 second TTL runs out, and retries have their own backoff. The study therefore reports the platform phases next to the client’s numbers. In A’s AKS-to-EKS pod-kill runs, DNS switched at about 10 s, yet the client first succeeded at about 31 s.

### First success is not the same as steady
After the first successful payment a client can fail again. In B the restored standby fails again under the backlog of retries. In A’s EKS-to-AKS partition runs the client succeeds at about 77 s and keeps failing until about 110 s. For that reason the study reports two recovery times:

- Client RTO: first success after the fault.
- Steady RTO: first success after the last relapse failure.

### Fixed parameters

Parameter | Value | Why
Load | 5 payments per second, one load level | Same offered load for every run; synthetic data only.
Detection | Three consecutive failed probes | Hysteresis against false positives.
DNS TTL | 60 s | Bounds how long a client can hold the old answer.
Backup age at the fault (B) | At most 120 s | A constant design parameter. The loss windows measured in the 20 B runs are 123 to 151 s, because the fault arrives some time after the last backup.
Idempotency key | Per payment | Makes replay safe, so duplicates are designed out. Duplicates are 0 in every run by design.
Pairs | Run i of A matched with run i of B | Same fault, direction and load. The order within a pair alternates.

Guide

## Architecture and technology
Direction of the run

Show

Animation

Select any box for what it is. On a phone, swipe the diagram sideways.

From | To | What travels | How
 Load generator | Cloudflare DNS | The client looks up bank.nirmalchathura.dev and gets the address of the active site. | DNS
 Load generator | Bank app (AKS) | Payments sent to the active site’s load balancer. | HTTPS
 Bank app (AKS) | PostgreSQL (AKS) | The payment and its receipt are written in one transaction. | PostgreSQL
 cert-manager (AKS) | Bank app (AKS) | The HTTPS certificate for the bank name is issued and renewed. | ACME
 PostgreSQL (AKS) | WireGuard (AKS) | The primary database streams its write-ahead log to the standby. | PostgreSQL streaming
 WireGuard (AKS) | WireGuard (EKS) | Across the WireGuard tunnel to the other cloud. | WireGuard, UDP 51820
 WireGuard (EKS) | PostgreSQL (EKS) | The standby applies the stream, so it follows the primary. | PostgreSQL streaming (TLS)
 PostgreSQL (AKS) | WireGuard (AKS) | Logical decoding of the primary’s changes. | PostgreSQL logical replication
 WireGuard (AKS) | WireGuard (EKS) | Debezium reads the primary through the same tunnel. | WireGuard
 WireGuard (EKS) | Debezium (EKS) | The changes arrive at Debezium on the standby side. | TCP 5432 via the gateway service
 Debezium (EKS) | Kafka (EKS) | Each change is published to Kafka: a second record of every payment. | Kafka

Roles swap with the direction. Kafka, Debezium and the controller run on the standby side only. Velero storage is used by condition B only, and both stores are in Sydney.

### Layout in one paragraph
AKS runs in Azure New Zealand North and EKS runs in AWS ap-southeast-6 (New Zealand). A WireGuard tunnel joins the two networks so the standby can replicate from the primary and the controller can probe it. Cloudflare holds the DNS record that the failover switches, and also hosts two pages outside both clouds: customers.nirmalchathura.dev (https://customers.nirmalchathura.dev) (a Worker with a Durable Object, always up) and bank.nirmalchathura.dev (https://bank.nirmalchathura.dev) (up only while the infrastructure runs).

Component | Version or detail | Role in the study
AKS | Kubernetes 1.36, New Zealand North | One of the two managed clusters; primary or standby by direction.
EKS | Kubernetes 1.31, ap-southeast-6 | The other cluster. Two managed node groups: default and streaming.
Terraform | Run from GitHub Actions with OIDC sign-in | Creates clusters, networks, node pools and the Velero storage; also destroys them after the runs.
CloudNativePG | PostgreSQL cluster with a live standby | Replication and promotion for condition A.
Strimzi and Kafka | Strimzi 0.45.0 (version override on Kubernetes 1.33 and later) | The change log. Only the Kafka, Debezium, controller and reconciler run on the standby side.
Debezium | Postgres CDC connectors; a second connector follows the new primary | Records every acknowledged write, a second record of each change.
Velero | Server 1.18.4, Azure and AWS plugins v1.14.4, Kopia file-system backup | Condition B backup and restore. Two backup locations on both clusters: Azure Blob (Australia East) and S3 (ap-southeast-2).
Chaos Mesh | Pod-kill and network partition | The injected faults, identical for A and B.
WireGuard | Cross-cloud tunnel | Private path between the two networks.
Cloudflare | DNS, Worker, Durable Object (SQLite) | DNS cutover record; customers and bank pages outside both clouds.
Prometheus-format summary | Exported with every run | Per-run metrics alongside result.json.
Python | Controller, reconciler, load generator, analysis | All custom logic; the analysis has unit tests for the statistics.

### Where condition B keeps its backups, and why
Each run’s backups go to the standby cloud’s own storage: an S3 bucket when the standby is EKS, an Azure container when the standby is AKS. That makes a restore equally local in both directions. Velero’s Kopia volume backup could not use a bucket in the AWS New Zealand region, so both stores are in Sydney; using the same distance on both sides keeps the baseline fair. Condition A keeps its data in New Zealand. “Reset experiment” swaps the read-write and read-only access modes of the two backup locations when the direction changes, so no reinstall is needed.
A change made during the study. The first batch kept every backup in one Azure container, so a restore on EKS read from another cloud while one on AKS read its own. After the first 20 runs this was changed to the standby’s own storage and all 40 runs were repeated on the final infrastructure. The results on this site are from that final run only.

Guide

## Walkthrough: build, run, analyse
Everything is run from the repository’s GitHub Actions tab with “Run workflow”. The workflows are numbered in build order. Nothing is run by hand on a laptop, so run 40 uses exactly the configuration of run 1.

- 01 Terraform. Plan, apply or destroy. Creates the AKS and EKS clusters, networks, node pools (default and streaming on both) and the Velero storage: an Azure storage account and container in Australia East, and an S3 bucket in ap-southeast-2 with an IAM user and policy. Terraform runs from GitHub Actions with OIDC sign-in.
- 00 Build everything. Runs the build steps for one direction in order: 02 Bootstrap platform, 03 Deploy cloud-demo (the bank application), 04 Deploy experiment platform (Strimzi Kafka, Debezium, CloudNativePG), 05 Install Chaos Mesh, 06 Deploy failover controller, 07a and 07b (certificate and HTTPS), 08 Velero.
- 09 Check platform and 10 Check reconciliation. Confirm clusters, databases, replication, Kafka, connectors, controller mode and DNS, and that reconciliation works, before a run.
- Reset experiment. Returns the system to a known starting state for a direction and a controller mode (auto for A, observe for B). It deletes the database secrets (after a direction switch a stale TLS secret blocked the database), swaps the Velero backup-location access modes by direction, and a Velero install clears all backup-repository records so a replaced bucket is re-initialised.
- Run experiment. One run: a platform check and a smoke payment, the load generator at 5 payments per second, fault injection with Chaos Mesh, the failover (controller for A, restore driver for B), then the load log is closed. The run saves result.json and a Prometheus-format summary as an artifact named run-<id>. A run that fails is repeated under the same id; trial runs are never repeated or counted.
- Batch workflows. “Run all batches” runs four chunks of 10 runs, one chunk per fault and direction. Inside a chunk the pairs alternate: reset for A, run A-01, reset for B, run B-01, and so on, with the order within a pair alternating.
- Analyse runs. Builds the statistics from the saved results, pooled and by direction: runs.csv, summary.csv, incidents.csv and a comparison report. They are the files downloadable on this site and the numbers on the dashboard are recomputed from runs.csv.
- Destroy. After the last chunk is saved the clusters are destroyed, because Azure and AWS charge by the hour.

### The run record
Each of the 40 runs contributes one row with these fields. The same rows are in the explorer below.

Field | Meaning
run_id | Identifier such as A-podkill-03 or B-partition-07. A or B, the fault, then the pair number. Pairs 1 to 5 are AKS to EKS and 6 to 10 are EKS to AKS. A-podkill-06-rerun is the repeat of the failed A-podkill-06.
condition, failure_type, direction, pair | The design factors. Run i of A is matched with run i of B.
rto_s | Client RTO: seconds from the fault to the first successful payment the client sees. Empty when no recovery was recorded.
rto_steady_s | Steady RTO: seconds from the fault to the first success after the last relapse failure.
phase_writable_s | Platform view: seconds until the new primary database accepts writes.
phase_dns_switched_s | Platform view: seconds until the DNS record points at the new primary.
acked | Payments the client was told succeeded.
lost | Acknowledged payments missing from the final database.
rpo_writes, rpo_s | Payments lost at recovery, and the loss window in seconds.
duplicated | Acknowledged payments that appear more than once. Zero in every run, by design (idempotency key).
delayed_but_recovered | Payments that were missing at recovery but present at the end.
clean | Acknowledged payments that are present exactly once. Zero together with acked > 0 means the run was not classified.
relapse_failed_attempts | Failed payment attempts after the first recovery.
failed_never_acked | Attempts that never received an acknowledgement.

Downloads: runs.csv (https://mseproject.nirmalchathura.dev/data/runs.csv), summary-pooled.csv (https://mseproject.nirmalchathura.dev/data/summary-pooled.csv), incidents.csv (https://mseproject.nirmalchathura.dev/data/incidents.csv), comparison-pooled.md (https://mseproject.nirmalchathura.dev/data/comparison-pooled.md), comparison-aks-to-eks.md (https://mseproject.nirmalchathura.dev/data/comparison-aks-to-eks.md), comparison-eks-to-aks.md (https://mseproject.nirmalchathura.dev/data/comparison-eks-to-aks.md), artifacts.tsv (https://mseproject.nirmalchathura.dev/data/artifacts.tsv). runs.csv and artifacts.tsv include the repeat of A-podkill-06. The summary, incident and comparison files were produced before that repeat, so their pod-kill figures use 9 valid pairs; the site and runs.csv use 10.

PreviousThe failover, replayed (https://mseproject.nirmalchathura.dev/)NextMethod and statistics (https://mseproject.nirmalchathura.dev/method)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)

MSE907 failover study

# Method and statistics
Design, fault injection, measurement and the tests used.
Guide

## Method and statistics
The study follows design science (Hevner et al., 2004): an artefact is built for a known problem and tested in a controlled experiment (Wohlin et al., 2012), in the spirit of chaos engineering (Basiri et al., 2016). The strategy (A or B) is the independent variable. Recovery times and the fate of each payment are the dependent variables. Interviews or surveys would not suit, because the questions are about measured behaviour.

### Design and control

- Factors. Condition (A, B) × fault (pod-kill, partition) × direction (AKS to EKS, EKS to AKS) × 5 pairs = 40 runs, which gives 10 pairs per fault.
- Matching. Run i of A is paired with run i of B, with the same load, seed and fault, the same load generator and the same instrumentation. The platform is reset before every run.
- Data. The load generator logs every attempt and acknowledgement. The Kafka change log is a second record. The controller timestamps each step. One result.json per run classifies each payment as clean, lost, delayed but recovered, or duplicated.

### Metrics

- RTO. Client RTO is the time to the first success. Steady RTO is the time until success is sustained. The platform phases (new primary writable, DNS switched) are reported beside them.
- RPO. Acknowledged payments lost, and the length of the loss window in seconds.
- Incident rate. The share of runs with at least one lost or duplicated acknowledged payment.

### Tests

- Wilcoxon signed-rank (Wilcoxon, 1945), two-sided, exact, on matched pairs, zero differences dropped. Used for RTO, RPO and the platform phases.
- Rank-biserial correlation (Kerby, 2014) as the effect size: −1 means A is lower in every pair.
- Hodges–Lehmann estimate (Hodges & Lehmann, 1963): the median of the Walsh averages of the paired differences, a robust size of the shift.
- McNemar’s exact test (McNemar, 1947) for incident rates, with Clopper–Pearson 95% intervals.
A limit built into the design. With n pairs the smallest exact two-sided p is 2 / 2n: 0.0625 for 5 pairs (one cell), 0.0039 for 9 pairs and 0.0020 for 10 pairs. Single cells can never be “significant” at 0.05 when each has only five pairs, which is why the tests pool the two directions to reach 10 pairs per fault.

### When a run counts
A run is valid when the fault was injected, the controller or the restore finished, the load log is closed and result.json holds every metric. Failed runs are repeated under the same id, so no run is dropped for looking unfavourable. All 40 runs are reported, including any where A is worse.

### Threats to validity

- Power. Ten pairs per fault is low. Effect sizes are reported alongside p-values.
- One load level and synthetic data. 5 payments per second; no real customer data.
- Naive baseline. Condition B is deliberately plain Velero restore. It represents common practice and the evidence that restores often fail (Jin et al., 2024), but a tuned backup scheme could do better.
- Builder bias. The author built both conditions and has an interest in A winning. Matched pairs, alternating order, identical instrumentation and a fixed analysis limit that.
- DNS caching. Client RTO includes up to 60 seconds of DNS cache wait, so platform times are shown beside it.

PreviousHow it works (https://mseproject.nirmalchathura.dev/guide)NextResults (https://mseproject.nirmalchathura.dev/results)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)

MSE907 failover study

# Results
The analysis dashboard, every run, and the findings with their caveats.
On this pageAnalysis dashboard (#dashboard)All 40 runs (#explorer)Findings and caveats (#findings)
Results

## Analysis dashboard
Every number here is computed in your browser from the 40 run records. Choose a fault and a direction, then pick a metric. The statistics match the analysis workflow: exact two-sided Wilcoxon signed-rank on matched pairs.

Fault

Direction

10 of 10B runs with a lost or duplicated payment (95% CI 69 to 100%)
0 of 10A runs with a lost or duplicated payment (95% CI 0 to 31%)
31 / 182 sMedian client RTO, A / B (10 pairs)
630Median payments lost per B run (A: 0)

### One metric, run by run
 Seconds from the fault to the first payment the client sees succeed.

 Each line is one matched pair (same pair number, fault and direction). Solid lines are AKS to EKS, dashed lines are EKS to AKS. 10 pairs plotted.

 Dots are runs. The black bar is the median and the tinted box is the interquartile range.

Client RTO: A median 31.1 (IQR 30.9 to 31.2), B median 181.7 (IQR 181.5 to 182.7). Exact Wilcoxon p = 0.0020 on 10 pairs; the smallest p this many pairs can give is 0.0020.

### All metrics for this selection

Metric | Pairs | A median | B median | Median diff (A − B) | Hodges–Lehmann | Rank-biserial r | W | p (exact)
Client RTO (s) | 10 | 31.1 | 181.7 | -150.9 | -151.1 | -1.00 | 0.0 | 0.0020
Steady RTO (s) | 10 | 37.9 | 240.9 | -203.6 | -203.6 | -1.00 | 0.0 | 0.0020
Platform: new primary writable (s) | 10 | 11.4 | 142.9 | -131.9 | -131.9 | -1.00 | 0.0 | 0.0020
Platform: DNS switched (s) | 10 | 11.9 | 143.5 | -132.0 | -131.9 | -1.00 | 0.0 | 0.0020
Acknowledged payments lost (at the end) | 10 | 0.0 | 630.0 | -630.0 | -632.5 | -1.00 | 0.0 | 0.0020
RPO: payments lost at recovery | 10 | 0.0 | 630.0 | -630.0 | -632.5 | -1.00 | 0.0 | 0.0020
RPO window (s) | 10 | 0.0 | 125.8 | -125.8 | -126.2 | -1.00 | 0.0 | 0.0020
Failed attempts after first recovery | 10 | 7.0 | 2,357.0 | -2,351.0 | -2,360.0 | -1.00 | 0.0 | 0.0020
Attempts never acknowledged | 10 | 0.0 | 454.5 | -454.5 | -456.5 | -1.00 | 0.0 | 0.0020

Two-sided Wilcoxon signed-rank test on matched pairs, zero differences dropped, p computed exactly. Differences are A − B, so negative means A is lower. Rank-biserial r of −1 means A was lower in every pair. A pair is left out of a metric when either run has no value for it.

### Incident rate

Outcome | A runs | A rate (95% CI) | B runs | B rate (95% CI) | A relative to B
Lost | 0 of 10 | 0% (0 to 31) | 10 of 10 | 100% (69 to 100) | 100% fewer
Duplicated | 0 of 10 | 0% (0 to 31) | 0 of 10 | 0% (0 to 31) | –
Delayed but recovered | 0 of 10 | 0% (0 to 31) | 0 of 10 | 0% (0 to 31) | –
Incident (lost or duplicated) | 0 of 10 | 0% (0 to 31) | 10 of 10 | 100% (69 to 100) | 100% fewer

Clopper–Pearson intervals. Exact McNemar test on the 10 discordant pairs (only A: 0, only B: 10): p = 0.0020. Payments lost or duplicated per 1,000 acknowledged: A 0.00 (0 of 10,500), B 281.00 (6,334 of 22,541).

### The four cells side by side

Fault | Direction | A client RTO | B client RTO | A steady | B steady | A lost | B lost | B loss window (s)
Pod-kill | AKS to EKS | 31.1 | 181.8 | 39.2 | 240.8 | 0 | 619 | 123.5
Pod-kill | EKS to AKS | 30.8 | 181.7 | 31.4 | 243.0 | 0 | 645 | 128.8
Partition | AKS to EKS | 22.6 | 172.7 | 22.6 | 230.7 | 0 | 682 | 136.1
Partition | EKS to AKS | 77.6 | 166.4 | 110.1 | 228.6 | 0 | 746 | 148.9

Medians over five runs per condition (four for A in EKS to AKS pod-kill, where one run has no recovery time). With only five pairs, a single cell can never reach an exact p below 0.0625; the tests above pool the directions to reach 10 pairs per fault.

Results

## All 40 runs
 Condition

Download runs.csv (https://mseproject.nirmalchathura.dev/data/runs.csv)

 | | | | | | | | | | | | |
A-podkill-01 | A | pod-kill | aks-to-eks | 1 | 25 Oct, 15:53 | 31.3 | 38.0 | 9.9 | 10.3 | 0 | 0 | 1,050 | 5
B-podkill-01 | B | pod-kill | aks-to-eks | 1 | 25 Oct, 16:19 | 182.4 | 244.3 | 138.8 | 139.3 | 619 | 123.5 | 2,210 | 2,412
B-podkill-02 | B | pod-kill | aks-to-eks | 2 | 25 Oct, 16:43 | 181.8 | 241.1 | 166.6 | 167.0 | 621 | 123.9 | 2,255 | 2,258
A-podkill-02 | A | pod-kill | aks-to-eks | 2 | 25 Oct, 17:02 | 31.2 | 42.5 | 9.7 | 10.3 | 0 | 0 | 1,050 | 28
A-podkill-03 | A | pod-kill | aks-to-eks | 3 | 25 Oct, 17:19 | 31.1 | 38.7 | 8.9 | 9.4 | 0 | 0 | 1,050 | 12
B-podkill-03 | B | pod-kill | aks-to-eks | 3 | 25 Oct, 17:44 | 181.7 | 240.8 | 154.5 | 154.8 | 621 | 123.9 | 2,246 | 2,373
B-podkill-04 | B | pod-kill | aks-to-eks | 4 | 25 Oct, 18:10 | 182.8 | 239.3 | 145.3 | 145.7 | 618 | 123.3 | 2,231 | 2,226
A-podkill-04 | A | pod-kill | aks-to-eks | 4 | 25 Oct, 18:27 | 31.1 | 41.0 | 9.1 | 9.5 | 0 | 0 | 1,050 | 9
A-podkill-05 | A | pod-kill | aks-to-eks | 5 | 25 Oct, 18:44 | 31.1 | 39.2 | 8.5 | 9.0 | 0 | 0 | 1,050 | 14
B-podkill-05 | B | pod-kill | aks-to-eks | 5 | 25 Oct, 19:08 | 181.3 | 238.7 | 142.0 | 142.4 | 617 | 123.1 | 2,248 | 2,239
A-partition-01 | A | partition | aks-to-eks | 1 | 25 Oct, 19:27 | 21.8 | 21.8 | 12.8 | 13.3 | 0 | 0 | 1,050 | 0
B-partition-01 | B | partition | aks-to-eks | 1 | 25 Oct, 19:53 | 172.7 | 230.7 | 158.9 | 159.4 | 682 | 136.1 | 2,323 | 2,248
B-partition-02 | B | partition | aks-to-eks | 2 | 25 Oct, 20:16 | 172.2 | 231.4 | 157.7 | 158.1 | 667 | 133.1 | 2,294 | 2,256
A-partition-02 | A | partition | aks-to-eks | 2 | 25 Oct, 20:35 | 23.7 | 23.7 | 11.9 | 12.3 | 0 | 0 | 1,050 | 0
A-partition-03 | A | partition | aks-to-eks | 3 | 25 Oct, 20:52 | 22.2 | 22.2 | 12.2 | 12.7 | 0 | 0 | 1,050 | 0
B-partition-03 | B | partition | aks-to-eks | 3 | 25 Oct, 21:16 | 173.8 | 231.5 | 143.7 | 144.0 | 667 | 133.2 | 2,284 | 2,279
B-partition-04 | B | partition | aks-to-eks | 4 | 25 Oct, 21:41 | 172.7 | 230.2 | 166.6 | 167.5 | 685 | 136.8 | 2,313 | 2,250
A-partition-04 | A | partition | aks-to-eks | 4 | 25 Oct, 21:59 | 75.6 | 111.3 | 11.2 | 11.7 | 0 | 0 | 1,050 | 1,424
A-partition-05 | A | partition | aks-to-eks | 5 | 25 Oct, 22:17 | 22.6 | 22.6 | 8.9 | 9.4 | 0 | 0 | 1,050 | 0
B-partition-05 | B | partition | aks-to-eks | 5 | 25 Oct, 22:41 | 172.2 | 226.9 | 167.0 | 167.4 | 684 | 136.5 | 2,309 | 2,111
B-podkill-06 | B | pod-kill | eks-to-aks | 6 | 25 Oct, 23:27 | 181.5 | 237.9 | 143.9 | 144.6 | 654 | 130.6 | 2,273 | 2,341
B-podkill-07 | B | pod-kill | eks-to-aks | 7 | 25 Oct, 23:50 | 184.2 | 245.0 | 133.2 | 133.6 | 645 | 128.8 | 2,232 | 2,404
A-podkill-07 | A | pod-kill | eks-to-aks | 7 | 26 Oct, 00:08 | 30.8 | 36.4 | 14.8 | 15.2 | 0 | 0 | 1,050 | 5
A-podkill-08 | A | pod-kill | eks-to-aks | 8 | 26 Oct, 00:25 | 30.8 | 30.8 | 13.0 | 13.4 | 0 | 0 | 1,050 | 0
B-podkill-08 | B | pod-kill | eks-to-aks | 8 | 26 Oct, 00:49 | 147.7 | 205.1 | 131.1 | 131.5 | 644 | 128.5 | 2,424 | 2,313
B-podkill-09 | B | pod-kill | eks-to-aks | 9 | 26 Oct, 01:14 | 183.7 | 250.3 | 126.2 | 126.6 | 639 | 127.6 | 2,177 | 3,022
A-podkill-09 | A | pod-kill | eks-to-aks | 9 | 26 Oct, 01:31 | 31.4 | 31.4 | 15.9 | 16.4 | 0 | 0 | 1,050 | 0
A-podkill-10 | A | pod-kill | eks-to-aks | 10 | 26 Oct, 01:48 | 31.2 | 37.9 | 14.3 | 14.7 | 0 | 0 | 1,050 | 12
B-podkill-10 | B | pod-kill | eks-to-aks | 10 | 26 Oct, 02:13 | 181.7 | 243.0 | 156.8 | 157.3 | 656 | 130.9 | 2,245 | 2,600
A-partition-06 | A | partition | eks-to-aks | 6 | 26 Oct, 02:32 | 78.0 | 108.6 | 17.9 | 18.3 | 0 | 0 | 1,050 | 1,093
B-partition-06 | B | partition | eks-to-aks | 6 | 26 Oct, 02:57 | 166.1 | 228.6 | 147.2 | 147.8 | 751 | 149.8 | 2,360 | 2,572
B-partition-07 | B | partition | eks-to-aks | 7 | 26 Oct, 03:21 | 166.4 | 229.1 | 160.2 | 160.6 | 746 | 148.8 | 2,349 | 2,717
A-partition-07 | A | partition | eks-to-aks | 7 | 26 Oct, 03:39 | 77.6 | 112.3 | 17.8 | 18.4 | 0 | 0 | 1,050 | 1,398
A-partition-08 | A | partition | eks-to-aks | 8 | 26 Oct, 03:56 | 76.7 | 110.1 | 19.3 | 19.7 | 0 | 0 | 1,050 | 1,384
B-partition-08 | B | partition | eks-to-aks | 8 | 26 Oct, 04:20 | 165.5 | 230.4 | 146.8 | 147.3 | 751 | 149.9 | 2,319 | 2,844
B-partition-09 | B | partition | eks-to-aks | 9 | 26 Oct, 04:44 | 166.5 | 225.7 | 148.9 | 149.6 | 746 | 148.9 | 2,321 | 2,838
A-partition-09 | A | partition | eks-to-aks | 9 | 26 Oct, 05:02 | 76.9 | 105.6 | 19.5 | 20.0 | 0 | 0 | 1,050 | 985
A-partition-10 | A | partition | eks-to-aks | 10 | 26 Oct, 05:20 | 77.6 | 112.7 | 18.4 | 18.9 | 0 | 0 | 1,050 | 1,406
B-partition-10 | B | partition | eks-to-aks | 10 | 26 Oct, 05:45 | 167.9 | 223.5 | 148.5 | 149.0 | 732 | 146.2 | 2,365 | 2,237
A-podkill-06-rerun | A | pod-kill | eks-to-aks | 6 | 26 Oct, 20:08 | 25.0 | 25.0 | 13.6 | 14.0 | 0 | 0 | 1,050 | 0

40 of 40 runs. Click a row for the remaining fields. Times are New Zealand time.

Results

## Findings and caveats

### What the 40 runs support

- H1, incident rate. A lost no acknowledged payment in any run. B lost between 617 and 751 payments in every run. Nothing was duplicated anywhere, as designed.
- H2, time. A’s median client RTO is lower than B’s under both faults. The gap is large for a pod-kill (about 31 s against 182 s) and smaller for a partition (pooled medians 76 s against 170 s). The pooled partition median for A sits between two clusters: about 22 s in AKS to EKS and about 77 s in EKS to AKS (see the first caveat below).
- H4, direction. In all four cells A is better than B on payment loss and on client RTO. The sign does not depend on direction. The size of A’s advantage does.

### What they do not support yet

- H3, replay. The 40 runs cannot show it: no condition A run ever had a gap to replay. It is tested separately by the ten-pair H3 test below, which does create a real gap. Every lost B payment was still in the change log, so reconciliation could have recovered them, but the 40 runs themselves do not demonstrate that.
- What the 40 runs do and do not show about the reconciler. They show that condition A preserved every acknowledged payment under the tested faults. They do not show that the reconciliation mechanism is the reason, because live replication had already copied each payment, so there was nothing for the reconciler to recover (missing_before was 0). The result compares the whole approach, replication plus orchestration, against backup and restore.
- Per-direction significance. Each cell has five pairs, so its exact p cannot be below 0.0625.

### The H3 test: replay of lost writes
Status: ten matched pairs done. All ten pairs ran on 26 October 2026 on the same AKS and EKS platform as the 40 runs, in four chunks of one workflow, all in the AKS-to-EKS direction. With replay on, every pair passed: 10 of 10, exact McNemar p = 0.0020, 95% Clopper–Pearson interval 69% to 100%. The two earlier trial runs are shown in the diagram and are not part of the ten pairs.

- Warm up as in every run. Then cut replication to the standby for 15 seconds with two Chaos Mesh partitions scoped to the standby database pod, so that the change-log connector still reads the primary: one stops the standby pod sending to the tunnel gateway, the other stops the gateway sending the write-ahead log to the standby pod.
- During those 15 seconds about 75 payments are acknowledged on the primary only. They are also in the Kafka change log. A lookup through the standby’s own bank app records how many are missing there (the gap).
- Kill the primary pod, as in the pod-kill runs. The controller detects it, moves the lease, fences the old primary, promotes the standby and switches DNS. The standby is promoted without the gap payments.
- The reconciler reads the change log and replays the missing payments through the bank app with their original payment ids (the idempotency key). The failed site then returns and the reconciler runs again.
Each pair is two runs with the same load: H3-A-NN with replay on, and H3-control-NN with replay off, where the reconciler only classifies and the same payments should stay lost. Pass criteria for a replay run: a gap above 0, no acknowledged payment missing at the end, no duplicate, and at least the gap recovered by replay. Ten pairs allow an exact McNemar p of 0.0020 and a 95% lower bound of 69% on the pass rate. The test is a condition A run in a new fault mode, so it is a separate analysis and does not change the 40-run results. How the controller behaves when the standby is cut off at promotion time is already seen in the partition runs, where replication was cut; in both trial runs the cut held until the standby was promoted.

#### One run, step by step

Reconciler

Animation

1. Normal running. The primary in AKS takes payments. Its writes stream to the standby in EKS, and Debezium records each change in Kafka.

The numbers are from two trial runs (trial-h3-01 with replay on, trial-h3-02 with replay off), done before the ten pairs. They are not part of the pair statistics; the ten pairs are in the table below.

From | To | What travels | How
 Load generator | Cloudflare DNS | The client looks up the bank name. | DNS
 Load generator | Bank app (AKS) | Payments go to the primary (AKS). | HTTPS
 Bank app (AKS) | PostgreSQL (AKS) | Each payment and its receipt are written to the database. | PostgreSQL
 PostgreSQL (AKS) | WireGuard (AKS) | The primary streams its write-ahead log. | PostgreSQL streaming
 WireGuard (AKS) | WireGuard (EKS) | Through the WireGuard tunnel to the other cloud. | WireGuard
 WireGuard (EKS) | PostgreSQL (EKS) | The standby applies it and so follows the primary. | PostgreSQL streaming
 PostgreSQL (AKS) | WireGuard (AKS) | Debezium reads the primary’s changes. | logical decoding
 WireGuard (AKS) | WireGuard (EKS) | Through the same tunnel and gateway. | WireGuard
 WireGuard (EKS) | Debezium (EKS) | They reach Debezium on the standby side. | TCP 5432 via the gateway service
 Debezium (EKS) | Kafka (EKS) | Each change is stored in Kafka: a second record of every payment. | Kafka

In the trial with replay on, the gap at the fault was 75 payments; the reconciler replayed 98 (the gap plus payments acknowledged before the pod died), and no payment was lost or duplicated. With replay off the gap was 77 and 102 acknowledged payments stayed lost, although all of them were still in Kafka. These two runs are a demonstration, not a result: the ten pairs below are what the statistics use.

### The ten pairs
Each row is one matched pair: the replay-on run and the replay-off control with the same seed and load, AKS to EKS. “Gap” is the number of acknowledged payments missing on the standby just before the primary was killed. “Recovered” is how many the reconciler wrote back (replay on); “lost” is how many acknowledged payments are missing at the end.

Pair | Gap (on) | Recovered (on) | Lost (on) | Duplicated (on) | RTO on (s) | Gap (control) | Lost (control) | RTO control (s)
1 | 76 | 99 | 0 | 0 | 61.6 | 77 | 107 | 57.3
2 | 76 | 98 | 0 | 0 | 62.2 | 76 | 105 | 56.9
3 | 75 | 104 | 0 | 0 | 59.8 | 76 | 97 | 61.4
4 | 75 | 97 | 0 | 0 | 62.7 | 76 | 104 | 58.2
5 | 76 | 101 | 0 | 0 | 60.4 | 77 | 107 | 56.0
6 | 75 | 102 | 0 | 0 | 59.8 | 75 | 100 | 59.7
7 | 76 | 107 | 0 | 0 | 57.1 | 76 | 106 | 58.4
8 | 77 | 101 | 0 | 0 | 62.4 | 76 | 100 | 61.9
9 | 77 | 105 | 0 | 0 | 59.7 | 76 | 97 | 61.2
10 | 77 | 103 | 0 | 0 | 51.9 | 76 | 107 | 57.2

Result. With replay on, every acknowledged payment was present at the end in all ten runs, none was duplicated, and the reconciler recovered at least the gap each time (1,017 payments recovered against a combined gap of 760). With replay off, the same kind of payments stayed lost in all ten runs (1,030 lost against a combined gap of 761), although all of them were still in the Kafka change log. In every pair the replay run recovered and the control lost; there was no pair where the control did better, so the exact McNemar test gives p = 0.0020, the smallest p that ten pairs allow. The pass rate for replay is 10 of 10, with a 95% Clopper–Pearson interval of 69% to 100%. H3 is supported for this fault and direction.
Why recovered and lost exceed the gap. The gap is measured shortly before the primary is killed. Payments acknowledged between that measurement and the pod dying are also missing on the new primary, so the reconciler replays them too, and the control loses them too. That is why 97 to 107 payments are recovered (or lost) against a gap of 75 to 77.
What this does not show. It is one fault (a 15 s replication cut followed by a pod-kill) in one direction (AKS to EKS), with a replication gap that the experiment creates on purpose; it does not show how often such gaps occur in real use. Recovery time was about 60 s in the replay runs (median 60.1 s) and about 58 s in the controls (median 58.3 s). RTO is not what H3 tests, and this difference was not tested. It is longer than the 31 s of the 40 pod-kill runs; the cause was not investigated. Results were produced by the author's own harness and the ten pairs are not blinded.

### Two things in the data to read carefully
1. A’s EKS-to-AKS partition recovery is slow in every run. The client first succeeds at 76.7 to 78.0 s in all five runs, against about 22 s in AKS to EKS. The platform had switched DNS at 18 to 20 s, so the client waited about 58 s more, which is close to the full 60 s DNS TTL. That fits a cached answer. It has not been tested. It lowers A’s advantage in that cell (77.6 s against 166.4 s) but does not remove it. In AKS to EKS one partition run also relapsed (75.6 s, steady 111 s).

2. One A run in EKS-to-AKS pod-kill (A-podkill-06) had to be repeated. In the first attempt (25 October) the fault did not take effect: the controller stayed Healthy at lease epoch 1 with no failover, all 1,050 payments succeeded first time, and reconciliation never ran, so the run is not a failover measurement and was dropped. The result folder also held stale reconciliation and partition files from 24 October, and no fault log, so the exact reason the kill did not fire is not proven. The run was repeated on 26 October as A-podkill-06-rerun (EKS to AKS, pod-kill, condition A, pair 6, same load seed 1006) and replaces it in all figures. In the repeat the controller failed over (lease epoch 2, writable at 13.6 s, DNS switched at 14.0 s), the client first succeeded at 25.0 s, and no payment was lost or duplicated. Two limits to keep in mind. The repeat ran on a rebuilt copy of the platform, made from the same code and configuration on 26 October, so it is not from the same deployment as the other 39 runs. And it was run because the first attempt failed, so it is not an independent draw; the failure was in the fault injection, not in the outcome. Its recovery time (25.0 s) is about 6 s lower than the other nine condition A pod-kill runs (30.8 to 31.4 s); the client switched to the new address at 24.3 s, and the cause of the difference was not investigated. Pod-kill RTO therefore uses all 10 pairs. The original run’s zip is kept for reference.

### Other observations

- B’s steady RTO exceeds its first-success RTO by about 58 to 62 s in the cell medians: the restored standby fails again under the retry backlog.
- B’s EKS-to-AKS pod-kill run 8 recovered faster than the rest (147.7 s against about 182 s).
- The platform phases show the same pattern as the clients: A’s new primary is writable at 8.5 to 19.5 s, B’s at 126 to 167 s.

PreviousMethod and statistics (https://mseproject.nirmalchathura.dev/method)NextRun artifacts (zips) (https://mseproject.nirmalchathura.dev/artifacts)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)

MSE907 failover study

# Run artifacts
The raw evidence for every measured run, downloadable.
Results

## Run artifacts
Each measured run produced one artifact in GitHub Actions: a zip with the load generator’s attempt log, the ledger check, the change events and the evidence collected from both clusters. All 40 are listed here, one download each. The all-runs zip also holds the failed first attempt of A-podkill-06. Zips stored with the site download directly; the rest are fetched from GitHub when you click.
The zips are the raw evidence behind every figure on the dashboard. The run table’s RTO, loss and window numbers come from each zip’s result.json.

Download all measured runs (one zip) (https://mseproject.nirmalchathura.dev/api/all) Built when you click, so it can take 10 to 30 seconds to start. Contains the 40 run zips and a README.

Run | Fault | Direction | Started (NZ) | RTO (s) | Result | Size | Download
A-partition-01 | partition | AKS to EKS | 25 Oct, 19:27 | 21.8 | No loss | | run-A-partition-01.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-01)
A-partition-02 | partition | AKS to EKS | 25 Oct, 20:35 | 23.7 | No loss | | run-A-partition-02.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-02)
A-partition-03 | partition | AKS to EKS | 25 Oct, 20:52 | 22.2 | No loss | | run-A-partition-03.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-03)
A-partition-04 | partition | AKS to EKS | 25 Oct, 21:59 | 75.6 | No loss | | run-A-partition-04.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-04)
A-partition-05 | partition | AKS to EKS | 25 Oct, 22:17 | 22.6 | No loss | | run-A-partition-05.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-05)
A-partition-06 | partition | EKS to AKS | 26 Oct, 02:32 | 78.0 | No loss | | run-A-partition-06.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-06)
A-partition-07 | partition | EKS to AKS | 26 Oct, 03:39 | 77.6 | No loss | | run-A-partition-07.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-07)
A-partition-08 | partition | EKS to AKS | 26 Oct, 03:56 | 76.7 | No loss | | run-A-partition-08.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-08)
A-partition-09 | partition | EKS to AKS | 26 Oct, 05:02 | 76.9 | No loss | | run-A-partition-09.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-09)
A-partition-10 | partition | EKS to AKS | 26 Oct, 05:20 | 77.6 | No loss | | run-A-partition-10.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-partition-10)
A-podkill-01 | pod-kill | AKS to EKS | 25 Oct, 15:53 | 31.3 | No loss | | run-A-podkill-01.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-01)
A-podkill-02 | pod-kill | AKS to EKS | 25 Oct, 17:02 | 31.2 | No loss | | run-A-podkill-02.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-02)
A-podkill-03 | pod-kill | AKS to EKS | 25 Oct, 17:19 | 31.1 | No loss | | run-A-podkill-03.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-03)
A-podkill-04 | pod-kill | AKS to EKS | 25 Oct, 18:27 | 31.1 | No loss | | run-A-podkill-04.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-04)
A-podkill-05 | pod-kill | AKS to EKS | 25 Oct, 18:44 | 31.1 | No loss | | run-A-podkill-05.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-05)
A-podkill-06-rerun | pod-kill | EKS to AKS | 26 Oct, 20:08 | 25.0 | No loss | | run-A-podkill-06-rerun.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-06-rerun)
A-podkill-07 | pod-kill | EKS to AKS | 26 Oct, 00:08 | 30.8 | No loss | | run-A-podkill-07.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-07)
A-podkill-08 | pod-kill | EKS to AKS | 26 Oct, 00:25 | 30.8 | No loss | | run-A-podkill-08.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-08)
A-podkill-09 | pod-kill | EKS to AKS | 26 Oct, 01:31 | 31.4 | No loss | | run-A-podkill-09.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-09)
A-podkill-10 | pod-kill | EKS to AKS | 26 Oct, 01:48 | 31.2 | No loss | | run-A-podkill-10.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=A-podkill-10)
B-partition-01 | partition | AKS to EKS | 25 Oct, 19:53 | 172.7 | 682 lost | | run-B-partition-01.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-01)
B-partition-02 | partition | AKS to EKS | 25 Oct, 20:16 | 172.2 | 667 lost | | run-B-partition-02.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-02)
B-partition-03 | partition | AKS to EKS | 25 Oct, 21:16 | 173.8 | 667 lost | | run-B-partition-03.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-03)
B-partition-04 | partition | AKS to EKS | 25 Oct, 21:41 | 172.7 | 685 lost | | run-B-partition-04.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-04)
B-partition-05 | partition | AKS to EKS | 25 Oct, 22:41 | 172.2 | 684 lost | | run-B-partition-05.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-05)
B-partition-06 | partition | EKS to AKS | 26 Oct, 02:57 | 166.1 | 751 lost | | run-B-partition-06.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-06)
B-partition-07 | partition | EKS to AKS | 26 Oct, 03:21 | 166.4 | 746 lost | | run-B-partition-07.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-07)
B-partition-08 | partition | EKS to AKS | 26 Oct, 04:20 | 165.5 | 751 lost | | run-B-partition-08.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-08)
B-partition-09 | partition | EKS to AKS | 26 Oct, 04:44 | 166.5 | 746 lost | | run-B-partition-09.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-09)
B-partition-10 | partition | EKS to AKS | 26 Oct, 05:45 | 167.9 | 732 lost | | run-B-partition-10.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-partition-10)
B-podkill-01 | pod-kill | AKS to EKS | 25 Oct, 16:19 | 182.4 | 619 lost | | run-B-podkill-01.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-01)
B-podkill-02 | pod-kill | AKS to EKS | 25 Oct, 16:43 | 181.8 | 621 lost | | run-B-podkill-02.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-02)
B-podkill-03 | pod-kill | AKS to EKS | 25 Oct, 17:44 | 181.7 | 621 lost | | run-B-podkill-03.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-03)
B-podkill-04 | pod-kill | AKS to EKS | 25 Oct, 18:10 | 182.8 | 618 lost | | run-B-podkill-04.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-04)
B-podkill-05 | pod-kill | AKS to EKS | 25 Oct, 19:08 | 181.3 | 617 lost | | run-B-podkill-05.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-05)
B-podkill-06 | pod-kill | EKS to AKS | 25 Oct, 23:27 | 181.5 | 654 lost | | run-B-podkill-06.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-06)
B-podkill-07 | pod-kill | EKS to AKS | 25 Oct, 23:50 | 184.2 | 645 lost | | run-B-podkill-07.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-07)
B-podkill-08 | pod-kill | EKS to AKS | 26 Oct, 00:49 | 147.7 | 644 lost | | run-B-podkill-08.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-08)
B-podkill-09 | pod-kill | EKS to AKS | 26 Oct, 01:14 | 183.7 | 639 lost | | run-B-podkill-09.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-09)
B-podkill-10 | pod-kill | EKS to AKS | 26 Oct, 02:13 | 181.7 | 656 lost | | run-B-podkill-10.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=B-podkill-10)

Hover a download link to see its SHA-256. GitHub artifact ids for each run (the 40 runs, the repeat and the H3 runs) are in artifacts.tsv (https://mseproject.nirmalchathura.dev/data/artifacts.tsv); the original files live in the private repository’s Actions tab (artifacts are kept for the repository’s retention period, 90 days by default, so a copy stored with the site is the permanent record).

### The H3 runs
The ten matched pairs of the H3 test (replay of lost writes) are saved the same way: one artifact per run. H3-A-NN is the run with replay on and H3-control-NN the same pair with replay off. All twenty ran AKS to EKS on 25 and 26 October 2026. They are not part of the 40 measured runs, and the zips have the same layout (see the list below).

Download the 20 H3 runs (one zip) (https://mseproject.nirmalchathura.dev/api/all-h3)

Run | Pair | Replay | Gap | Recovered | Lost | Duplicated | RTO (s) | Download
H3-A-01 | 1 | on | 76 | 99 | 0 | 0 | 61.6 | run-H3-A-01.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-01)
H3-A-02 | 2 | on | 76 | 98 | 0 | 0 | 62.2 | run-H3-A-02.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-02)
H3-A-03 | 3 | on | 75 | 104 | 0 | 0 | 59.8 | run-H3-A-03.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-03)
H3-A-04 | 4 | on | 75 | 97 | 0 | 0 | 62.7 | run-H3-A-04.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-04)
H3-A-05 | 5 | on | 76 | 101 | 0 | 0 | 60.4 | run-H3-A-05.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-05)
H3-A-06 | 6 | on | 75 | 102 | 0 | 0 | 59.8 | run-H3-A-06.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-06)
H3-A-07 | 7 | on | 76 | 107 | 0 | 0 | 57.1 | run-H3-A-07.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-07)
H3-A-08 | 8 | on | 77 | 101 | 0 | 0 | 62.4 | run-H3-A-08.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-08)
H3-A-09 | 9 | on | 77 | 105 | 0 | 0 | 59.7 | run-H3-A-09.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-09)
H3-A-10 | 10 | on | 77 | 103 | 0 | 0 | 51.9 | run-H3-A-10.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-A-10)
H3-control-01 | 1 | off | 77 | 0 | 107 | 0 | 57.3 | run-H3-control-01.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-01)
H3-control-02 | 2 | off | 76 | 0 | 105 | 0 | 56.9 | run-H3-control-02.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-02)
H3-control-03 | 3 | off | 76 | 0 | 97 | 0 | 61.4 | run-H3-control-03.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-03)
H3-control-04 | 4 | off | 76 | 0 | 104 | 0 | 58.2 | run-H3-control-04.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-04)
H3-control-05 | 5 | off | 77 | 0 | 107 | 0 | 56.0 | run-H3-control-05.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-05)
H3-control-06 | 6 | off | 75 | 0 | 100 | 0 | 59.7 | run-H3-control-06.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-06)
H3-control-07 | 7 | off | 76 | 0 | 106 | 0 | 58.4 | run-H3-control-07.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-07)
H3-control-08 | 8 | off | 76 | 0 | 100 | 0 | 61.9 | run-H3-control-08.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-08)
H3-control-09 | 9 | off | 76 | 0 | 97 | 0 | 61.2 | run-H3-control-09.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-09)
H3-control-10 | 10 | off | 76 | 0 | 107 | 0 | 57.2 | run-H3-control-10.zip (https://mseproject.nirmalchathura.dev/api/artifact?run=H3-control-10)

Gap = acknowledged payments missing on the standby just before the primary was killed. Recovered = payments the reconciler wrote back (replay on). Lost = acknowledged payments missing at the end. Values are from each run’s result.json, as summarised by the H3 analysis.

### What is inside a zip

File | What it holds
run.json, result.json | The run record: condition, fault, direction, seed, fault time, and the measured result (RTO, RPO, outcome counts).
attempts.jsonl | Every payment attempt the load generator made: send and end time, HTTP status, error, which site answered and the address it reached.
ledger.jsonl, writes.csv | Each payment id the client was told succeeded, matched against what the database holds.
cdc.jsonl | Debezium change events captured from Kafka (the change log the reconciler replays).
loadgen.json, loadgen.out | Load generator settings (5 payments/s, 210 s, retry window 120 s, seed) and totals.
controller-status.json | The FailoverPolicy status at the end of the run: active site, lease epoch, failover history, probe streaks.
reconciliation.json | The reconciler’s pass over the change log: which payments were clean, delayed, lost or duplicated.
eks-*.log, eks-db-diagnostics.txt | Evidence collected from the clusters during the run: app readiness, database endpoints, pod events.
connect-log.txt, connectors.txt | Debezium Kafka Connect log and connector state.
run-summary.prom | The same result as Prometheus gauges.

The list is from run A-podkill-06-rerun. The exact file set can differ by condition and fault (condition B has no controller), so check a zip before relying on a file being present.
A-podkill-06 and its repeat. The first attempt (25 October, run-A-podkill-06) was a failed fault injection and is not in the figures; its folder also held stale files from an earlier partition run on 24 October. The figures use the repeat, run-A-podkill-06-rerun (26 October).

PreviousResults (https://mseproject.nirmalchathura.dev/results)NextEthics and Māori data sovereignty (https://mseproject.nirmalchathura.dev/ethics)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)

MSE907 failover study

# Ethics and Māori data sovereignty
Data, ethics and sovereignty considerations.
Context

## Ethics and Māori data sovereignty
Māori data is a taonga, and Māori data sovereignty is the principle that data about Māori should be subject to Māori governance (Te Mana Raraunga, 2018). The CARE principles say something similar internationally (Carroll et al., 2020; Kukutai & Taylor, 2016). Automatic failover puts such data at risk because it moves and copies data without a person deciding. Failover also creates extra copies: a standby, a Kafka log, backups. Each copy needs the protection and retention rules of the original, because a deletion request is not met while a copy remains.
This is treated as a design requirement, not a compliance claim. The experiment uses synthetic data only, so no consent is needed and no real data are at risk. Both clusters are in New Zealand. The exceptions are the baseline’s backups in Sydney and the synthetic payments stored on Cloudflare’s global network for the customers page.

Area | In place now | Needed for real use
Location (rangatiratanga) | Both clusters in New Zealand; exceptions: baseline backups (Sydney), Cloudflare dashboard. | An allowedRegions check before promotion; a New Zealand backup store the tool supports.
Protection (kaitiakitanga) | TLS, WireGuard, fencing, encrypted keys and Terraform state. | Encryption at rest and customer-managed keys.
Second copy (whakapapa) | Logs and backups are known to be copies. | Retention and deletion rules; wipe on request.
Access (manaakitanga) | Synthetic data; dashboard on Cloudflare’s global network. | Role-based access and audit; limit the dashboard to counters.
Governance | None in code, by design. | A Māori data governance process and adviser before real use.

### Other ethical points

- Privacy. A real deployment would fall under the Privacy Act 2020, including breach notification where serious harm is likely.
- Bias and honesty. The baseline could look like a straw man, so the choice is anchored in independent evidence and the report states what B could have recovered. All 40 runs are reported.
- Legal. Faults are injected only into the author’s own clusters. Components are open source under their own licences. A financial-sector deployment would face further obligations this prototype does not address.
- AI assistance. An AI coding assistant was used. The author takes responsibility for every claim.

PreviousRun artifacts (zips) (https://mseproject.nirmalchathura.dev/artifacts)NextExecution, cost and decisions (https://mseproject.nirmalchathura.dev/execution)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)

MSE907 failover study

# Execution, cost and decisions
How the runs were executed, what they cost, the problems met, and the decisions and status.
On this pageExecution, cost and problems (#operations)Decisions and status (#decisions)
Context

## Execution, cost and what went wrong

### Execution log
The final 40 runs ran as four chunks of 10 on 25 and 26 October 2026 (NZ time). Start times below are taken from the run artifacts. One run, A-podkill-06, was repeated on the evening of 26 October after the first attempt turned out to be a failed fault injection (see “Findings and caveats”); the repeat is counted in chunk 3 and its start time is not shown in the table.

Chunk | Direction | Fault | Runs | First run started (NZ) | Last run started (NZ)
1 | AKS to EKS | Pod-kill | 10 | 25 Oct, 15:53 | 25 Oct, 19:08
2 | AKS to EKS | Partition | 10 | 25 Oct, 19:27 | 25 Oct, 22:41
3 | EKS to AKS | Pod-kill | 10 | 25 Oct, 23:27 | 26 Oct, 02:13
4 | EKS to AKS | Partition | 10 | 26 Oct, 02:32 | 26 Oct, 05:45

Total time. The 40 measured runs took about 14 h 11 min of wall-clock time: four chunks of about 3 h 30 min each, plus a short analysis job after every chunk, run back to back in one GitHub Actions workflow.
GitHub shows a total of 15 h 1 min 4 s for the workflow. That figure also contains 49 min 31 s for the last job, which destroys both clusters. Destroying the clusters is clean-up, not part of the measurement, so it is subtracted here (15 h 1 min 4 s − 49 min 31 s = 14 h 11 min 33 s).

### Cost and constraints

- Azure. Everything ran inside the free trial. The 40 runs took place from 25 October into the early hours of 26 October, and the 20 H3 runs on 26 October, and the project was finished before the trial ended. The account was upgraded to pay-as-you-go, but the free sign-up credit still covered the charges. October charges to 26 October were NZ$33.40 (forecast NZ$37.96 for the month), and the amount due was NZ$0.00. The largest items were the virtual machines, the Premium SSD disks and the public IP addresses.
- AWS. US$78.84 month to date on 26 October, forecast US$105.99 for the month if left running; last month was US$1.99. By service: Elastic Kubernetes Service US$48.19, EC2 compute US$12.33, EC2 other US$11.31 (items such as the NAT gateway, disks and data transfer), load balancing US$4.19, VPC US$1.88, other services US$0.94. The cluster, NAT gateway and disks were destroyed after the runs, so the forecast overstates the final bill. All of it was covered by AWS free credits.
- GitHub Actions. US$42.58 gross metered usage for October against US$18.00 of included usage, so about US$24.58 was metered beyond the included amount. All of it was the MSE907 repository. Minutes ran out in the middle of the first batch, so a spending budget was raised to finish. GitHub showed no payment due at the time of writing.
- Cloudflare. The free Durable Object plan allows 100,000 rows written a day. The customers page exceeded it and returned errors, so the account moved to the paid Workers plan: one US$5.00 invoice, paid on 25 October 2026.
- Pattern. The infrastructure is created from code, used, and destroyed when idle, so cost follows the runs and every run starts from the same state.

### Problems met and what changed

Problem | Cause | What changed
Client recovers at 31 s though DNS switched at 9 s | Cached DNS answer (60 s TTL) and retries. | Report platform phases beside client RTO.
Success, then failure again | Relapse under a retry backlog. | Report first-success RTO and steady RTO.
Database blocked after switching direction | Stale TLS secret. | Reset deletes the database secrets.
Strimzi would not run on a new cluster | Strimzi 0.45.0 does not support Kubernetes 1.33 and later. | Version override.
Backups unfair between directions | One Azure container served both; EKS restores read across clouds. | Backups go to the standby cloud’s own storage; all 40 runs repeated.
Velero could not use the New Zealand bucket | Kopia volume backup does not work with a bucket in ap-southeast-6. | Both stores in Sydney, so distance is equal in both directions.
“Repository not initialized” | Velero kept a backup repository marked Ready after its storage was replaced. | Install deletes all backup-repository records.
Terraform replaced a storage account | Fixed account name taken; old bucket in state. | Delete-then-create step and state removal in the workflow.
Customers page returned 500 | Durable Object free-tier row limit. | Paid Workers plan.
Actions minutes exhausted mid-batch | Four chunks of about 3.5 hours each. | Spending budget raised.
Destroy stalled on an AKS node pool | Pool deletion ran for over 40 minutes; cause not confirmed. | Resource group deleted directly.

Context

## Decisions, status and timeline

Decision | Choice | Reason
D1 Client routing | DNS only, no PgBouncer. | A cloud outage takes the application down with it, so this is a site-level decision. PgBouncer would help for a database-only failure but adds cross-cloud latency. The client DNS cache is reported as a result. Supervisor confirmation pending.
D2 Duplicates | Zero by design. | The idempotency key prevents duplicates, so the study measures loss.
D3 Condition B | Stays naive. | It represents what the evidence says is fragile, and the proposal defined it that way.
D4 Data sovereignty | A design requirement; synthetic data only; no compliance claim. | Both clusters are in New Zealand and the second copy is treated as governed data.
Direction | One switch maps roles to clouds. | The second direction changed only the role-mapping script.

### Software engineering principles applied

- Separation of concerns (Parnas, 1972): controller, reconciler, load generator, fault injection and analysis are separate modules, so the direction swap changed one script.
- Design for failure (Nygard, 2018; Desai, 2026; Sun et al., 2022): hysteresis, lease, fencing, idempotency keys and reconciliation guard against a controller acting on a stale view.
- Infrastructure as code (Humble & Farley, 2010): run 40 is configured identically to run 1.
- Iterative development (Beck et al., 2001): trials showed the need for the reverse direction and for the Sydney backups.
- Testing: unit tests for the statistics, a fake batch harness and a pre-flight check that the controller matches the repository.

### Status

Deliverable | Status
Clusters in New Zealand regions, tunnel, DNS, HTTPS | Complete both directions
Standby, CDC pipeline, second connector | Complete
Failover controller (hysteresis, lease, fence, promote, DNS) | Complete modes auto and observe
Reconciler and failed-site return (RQ3) | Complete replay recovered every acknowledged payment in 10 of 10 H3 pairs (AKS to EKS)
Condition B baseline (Velero, driver) | Complete backups on the standby cloud
Fault injection, result files, Prometheus summary | Complete
Analysis and batch workflows | Complete pooled and by direction
40 measured runs | Complete 25 to 26 October 2026
Analysis dashboard (this site) | Live
Supervisor review; protocol, traceability, technical record | In progress 3 to 5 November
Final report: results, discussion, limitations, future work | Due 8 November

PreviousEthics and Māori data sovereignty (https://mseproject.nirmalchathura.dev/ethics)NextReferences and about (https://mseproject.nirmalchathura.dev/about)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)

MSE907 failover study

# References and about
Sources cited, and who made this.
On this pageReferences (#references)About (#about)
Context

## References

- Abadi, D. J. (2012). Consistency tradeoffs in modern distributed database system design. IEEE Computer, 45(2), 37–42. doi.org/10.1109/MC.2012.33 (https://doi.org/10.1109/MC.2012.33)
- Alquraan, A., et al. (2018). An analysis of network-partitioning failures in cloud systems. OSDI ’18, 51–68. www.usenix.org/conference/osdi18/presentation/alquraan (https://www.usenix.org/conference/osdi18/presentation/alquraan)
- Andreakis, A., & Papapanagiotou, I. (2020). DBLog: A watermark-based change-data-capture framework. arXiv. arxiv.org/abs/2010.12597 (https://arxiv.org/abs/2010.12597)
- Basiri, A., et al. (2016). Chaos engineering. IEEE Software, 33(3), 35–41. doi.org/10.1109/MS.2016.60 (https://doi.org/10.1109/MS.2016.60)
- Beck, K., et al. (2001). Manifesto for agile software development. agilemanifesto.org (https://agilemanifesto.org)
- Bhavsar, S., et al. (2023). Kubernetes cluster disaster recovery using AWS. ICCUBEA 2023. IEEE. doi.org/10.1109/ICCUBEA58933.2023.10391973 (https://doi.org/10.1109/ICCUBEA58933.2023.10391973)
- Brahmandam, L. M. K. (2026). A decision framework for multi-cloud microservice deployment across AWS and GCP. IJETCSIT, 7(1), 365–373. doi.org/10.63282/3050-9246.IJETCSIT-V7I1P152 (https://doi.org/10.63282/3050-9246.IJETCSIT-V7I1P152)
- Carroll, S. R., et al. (2020). The CARE principles for Indigenous data governance. Data Science Journal, 19(1), Article 43. doi.org/10.5334/dsj-2020-043 (https://doi.org/10.5334/dsj-2020-043)
- Chaos Mesh. (n.d.). Documentation. chaos-mesh.org/docs/ (https://chaos-mesh.org/docs/)
- CloudNativePG. (n.d.). Documentation. cloudnative-pg.io/documentation/ (https://cloudnative-pg.io/documentation/)
- Corbett, J. C., et al. (2013). Spanner: Google’s globally distributed database. ACM TOCS, 31(3), Article 8. doi.org/10.1145/2491245 (https://doi.org/10.1145/2491245)
- Debezium. (n.d.). Documentation. debezium.io/documentation/ (https://debezium.io/documentation/)
- Desai, J. B. (2026). Reliable event-driven processing in distributed systems. JISEM, 11(2s), 1449–1461. doi.org/10.52783/jisem.v11i2s.14608 (https://doi.org/10.52783/jisem.v11i2s.14608)
- Gilbert, S., & Lynch, N. (2002). Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51–59. doi.org/10.1145/564585.564601 (https://doi.org/10.1145/564585.564601)
- Gunawi, H. S., et al. (2016). Why does the cloud stop computing? SoCC ’16, 1–16. doi.org/10.1145/2987550.2987583 (https://doi.org/10.1145/2987550.2987583)
- Hevner, A. R., et al. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105. doi.org/10.2307/25148625 (https://doi.org/10.2307/25148625)
- Hodges, J. L., & Lehmann, E. L. (1963). Estimates of location based on rank tests. Annals of Mathematical Statistics, 34(2), 598–611. doi.org/10.1214/aoms/1177704172 (https://doi.org/10.1214/aoms/1177704172)
- Humble, J., & Farley, D. (2010). Continuous delivery. Addison-Wesley.
- Jin, R., et al. (2024). Baking disaster-proof Kubernetes applications with efficient recipes. ICPE ’24 Companion. research.spec.org/icpe_proceedings/2024/companion/p174.pdf (https://research.spec.org/icpe_proceedings/2024/companion/p174.pdf)
- Kerby, D. S. (2014). The simple difference formula. Comprehensive Psychology, 3, Article 11.IT.3.1. doi.org/10.2466/11.IT.3.1 (https://doi.org/10.2466/11.IT.3.1)
- Kim, J.-B., et al. (2024). Design and implementation of an automated disaster-recovery system for a Kubernetes cluster using LSTM. Applied Sciences, 14(9), 3914. doi.org/10.3390/app14093914 (https://doi.org/10.3390/app14093914)
- Kleppmann, M. (2017). Designing data-intensive applications. O’Reilly Media.
- Kukutai, T., & Taylor, J. (Eds.). (2016). Indigenous data sovereignty: Toward an agenda. ANU Press. doi.org/10.22459/CAEPR38.11.2016 (https://doi.org/10.22459/CAEPR38.11.2016)
- Madupati, B. (2023). Kubernetes for multi-cloud and hybrid cloud. JSAER, 10(6), 290–297. doi.org/10.5281/zenodo.14050147 (https://doi.org/10.5281/zenodo.14050147)
- McNemar, Q. (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153–157. doi.org/10.1007/BF02295996 (https://doi.org/10.1007/BF02295996)
- Nygard, M. T. (2018). Release it! (2nd ed.). Pragmatic Bookshelf.
- Parnas, D. L. (1972). On the criteria to be used in decomposing systems into modules. Communications of the ACM, 15(12), 1053–1058. doi.org/10.1145/361598.361623 (https://doi.org/10.1145/361598.361623)
- Shi, J., et al. (2008). Study on log-based change data capture and handling mechanism in real-time data warehouse. CSSE 2008, 478–481. doi.org/10.1109/CSSE.2008.926 (https://doi.org/10.1109/CSSE.2008.926)
- Strimzi. (n.d.). Documentation. strimzi.io/docs/operators/latest/ (https://strimzi.io/docs/operators/latest/)
- Strimzi project. (n.d.). Issue 11456. GitHub. github.com/strimzi/strimzi-kafka-operator/issues/11456 (https://github.com/strimzi/strimzi-kafka-operator/issues/11456)
- Sun, X., et al. (2022). Automatic reliability testing for cluster management controllers. OSDI ’22, 143–159. www.usenix.org/conference/osdi22/presentation/sun (https://www.usenix.org/conference/osdi22/presentation/sun)
- Te Mana Raraunga. (2018). Principles of Māori data sovereignty. www.temanararaunga.maori.nz/principles-of-maori-data-sovereignty (https://www.temanararaunga.maori.nz/principles-of-maori-data-sovereignty)
- Velero. (n.d.). Documentation. velero.io/docs/ (https://velero.io/docs/)
- Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83. doi.org/10.2307/3001968 (https://doi.org/10.2307/3001968)
- Wohlin, C., et al. (2012). Experimentation in software engineering. Springer.

Context

## About this project
Student: Unagallage Nirmal Chathura Unagalle (ID 270766362), Master of Software Engineering, Yoobee College of Creative Innovation. Course: MSE907 capstone. Supervisor: Usman Ghani.
Project: Reconciliation-Aware Failover Orchestrator for Stateful Kubernetes Workloads across Heterogeneous Cloud Providers (AKS ↔ EKS).

- Code: github.com/Nirmal-C/MSE907 (https://github.com/Nirmal-C/MSE907). The repository is private. Request collaborator access from GitHub user Nirmal-C.
- Portfolio: nirmalchathura.dev (https://nirmalchathura.dev), the author’s portfolio website.
- Customers page: customers.nirmalchathura.dev (https://customers.nirmalchathura.dev) is always up and shows payments as customers see them.
- Bank page: bank.nirmalchathura.dev (https://bank.nirmalchathura.dev) is up only while the infrastructure runs. It is destroyed when not in use to save cost, so the figures in the report show it instead.
All data are synthetic. No compliance claim is made. The dashboard and the 40-run results are computed in your browser from the 40 run records in runs.csv. The 20 H3 runs (the replay test) are a separate experiment, so they are not in that file; they are in h3-runs.csv and the table in the H3 section, and their raw artifacts are on the Run artifacts page.

PreviousExecution, cost and decisions (https://mseproject.nirmalchathura.dev/execution)
© 2026 Unagallage Nirmal Chathura Unagalle. All rights reserved. MSE907 capstone, Yoobee (270766362). Synthetic data only; no regulatory compliance is claimed. Portfolio: nirmalchathura.dev (https://nirmalchathura.dev)