Results

Every number here is computed in your browser from the 40 run records. Choose a fault and a direction, then pick a metric. The statistics match the analysis workflow: exact two-sided Wilcoxon signed-rank on matched pairs.

What the 40 runs support

• H1, incident rate. A lost no acknowledged payment in any run. B lost between 617 and 751 payments in every run. Nothing was duplicated anywhere, as designed.
• H2, time. A’s median client RTO is lower than B’s under both faults. The gap is large for a pod-kill (about 31 s against 182 s) and smaller for a partition (pooled medians 76 s against 170 s). The pooled partition median for A sits between two clusters: about 22 s in AKS to EKS and about 77 s in EKS to AKS (see the first caveat below).
• H4, direction. In all four cells A is better than B on payment loss and on client RTO. The sign does not depend on direction. The size of A’s advantage does.

What they do not support yet

• H3, replay. The 40 runs cannot show it: no condition A run ever had a gap to replay. It is tested separately by the ten-pair H3 test below, which does create a real gap. Every lost B payment was still in the change log, so reconciliation could have recovered them, but the 40 runs themselves do not demonstrate that.
• What the 40 runs do and do not show about the reconciler. They show that condition A preserved every acknowledged payment under the tested faults. They do not show that the reconciliation mechanism is the reason, because live replication had already copied each payment, so there was nothing for the reconciler to recover (missing_before was 0). The result compares the whole approach, replication plus orchestration, against backup and restore.
• Per-direction significance. Each cell has five pairs, so its exact p cannot be below 0.0625.

The H3 test: replay of lost writes

1. Warm up as in every run. Then cut replication to the standby for 15 seconds with two Chaos Mesh partitions scoped to the standby database pod, so that the change-log connector still reads the primary: one stops the standby pod sending to the tunnel gateway, the other stops the gateway sending the write-ahead log to the standby pod.
2. During those 15 seconds about 75 payments are acknowledged on the primary only. They are also in the Kafka change log. A lookup through the standby’s own bank app records how many are missing there (the gap).
3. Kill the primary pod, as in the pod-kill runs. The controller detects it, moves the lease, fences the old primary, promotes the standby and switches DNS. The standby is promoted without the gap payments.
4. The reconciler reads the change log and replays the missing payments through the bank app with their original payment ids (the idempotency key). The failed site then returns and the reconciler runs again.

Each pair is two runs with the same load: H3-A-NN with replay on, and H3-control-NN with replay off, where the reconciler only classifies and the same payments should stay lost. Pass criteria for a replay run: a gap above 0, no acknowledged payment missing at the end, no duplicate, and at least the gap recovered by replay. Ten pairs allow an exact McNemar p of 0.0020 and a 95% lower bound of 69% on the pass rate. The test is a condition A run in a new fault mode, so it is a separate analysis and does not change the 40-run results. How the controller behaves when the standby is cut off at promotion time is already seen in the partition runs, where replication was cut; in both trial runs the cut held until the standby was promoted.

One run, step by step

In the trial with replay on, the gap at the fault was 75 payments; the reconciler replayed 98 (the gap plus payments acknowledged before the pod died), and no payment was lost or duplicated. With replay off the gap was 77 and 102 acknowledged payments stayed lost, although all of them were still in Kafka. These two runs are a demonstration, not a result: the ten pairs below are what the statistics use.

The ten pairs

Each row is one matched pair: the replay-on run and the replay-off control with the same seed and load, AKS to EKS. “Gap” is the number of acknowledged payments missing on the standby just before the primary was killed. “Recovered” is how many the reconciler wrote back (replay on); “lost” is how many acknowledged payments are missing at the end.

Result. With replay on, every acknowledged payment was present at the end in all ten runs, none was duplicated, and the reconciler recovered at least the gap each time (1,017 payments recovered against a combined gap of 760). With replay off, the same kind of payments stayed lost in all ten runs (1,030 lost against a combined gap of 761), although all of them were still in the Kafka change log. In every pair the replay run recovered and the control lost; there was no pair where the control did better, so the exact McNemar test gives p = 0.0020, the smallest p that ten pairs allow. The pass rate for replay is 10 of 10, with a 95% Clopper–Pearson interval of 69% to 100%. H3 is supported for this fault and direction.

Why recovered and lost exceed the gap. The gap is measured shortly before the primary is killed. Payments acknowledged between that measurement and the pod dying are also missing on the new primary, so the reconciler replays them too, and the control loses them too. That is why 97 to 107 payments are recovered (or lost) against a gap of 75 to 77.

What this does not show. It is one fault (a 15 s replication cut followed by a pod-kill) in one direction (AKS to EKS), with a replication gap that the experiment creates on purpose; it does not show how often such gaps occur in real use. Recovery time was about 60 s in the replay runs (median 60.1 s) and about 58 s in the controls (median 58.3 s). RTO is not what H3 tests, and this difference was not tested. It is longer than the 31 s of the 40 pod-kill runs; the cause was not investigated. Results were produced by the author's own harness and the ten pairs are not blinded.

Two things in the data to read carefully

Other observations

• B’s steady RTO exceeds its first-success RTO by about 58 to 62 s in the cell medians: the restored standby fails again under the retry backlog.
• B’s EKS-to-AKS pod-kill run 8 recovered faster than the rest (147.7 s against about 182 s).
• The platform phases show the same pattern as the clients: A’s new primary is writable at 8.5 to 19.5 s, B’s at 126 to 167 s.