The study follows design science (Hevner et al., 2004): an artefact is built for a known problem and tested in a controlled experiment (Wohlin et al., 2012), in the spirit of chaos engineering (Basiri et al., 2016). The strategy (A or B) is the independent variable. Recovery times and the fate of each payment are the dependent variables. Interviews or surveys would not suit, because the questions are about measured behaviour.

Design and control

Factors. Condition (A, B) × fault (pod-kill, partition) × direction (AKS to EKS, EKS to AKS) × 5 pairs = 40 runs, which gives 10 pairs per fault.
Matching. Run i of A is paired with run i of B, with the same load, seed and fault, the same load generator and the same instrumentation. The platform is reset before every run.
Data. The load generator logs every attempt and acknowledgement. The Kafka change log is a second record. The controller timestamps each step. One result.json per run classifies each payment as clean, lost, delayed but recovered, or duplicated.

Metrics

RTO. Client RTO is the time to the first success. Steady RTO is the time until success is sustained. The platform phases (new primary writable, DNS switched) are reported beside them.
RPO. Acknowledged payments lost, and the length of the loss window in seconds.
Incident rate. The share of runs with at least one lost or duplicated acknowledged payment.

Tests

Wilcoxon signed-rank (Wilcoxon, 1945), two-sided, exact, on matched pairs, zero differences dropped. Used for RTO, RPO and the platform phases.
Rank-biserial correlation (Kerby, 2014) as the effect size: −1 means A is lower in every pair.
Hodges–Lehmann estimate (Hodges & Lehmann, 1963): the median of the Walsh averages of the paired differences, a robust size of the shift.
McNemar’s exact test (McNemar, 1947) for incident rates, with Clopper–Pearson 95% intervals.

A limit built into the design. With n pairs the smallest exact two-sided p is 2 / 2ⁿ: 0.0625 for 5 pairs (one cell), 0.0039 for 9 pairs and 0.0020 for 10 pairs. Single cells can never be “significant” at 0.05 when each has only five pairs, which is why the tests pool the two directions to reach 10 pairs per fault.

When a run counts

A run is valid when the fault was injected, the controller or the restore finished, the load log is closed and result.json holds every metric. Failed runs are repeated under the same id, so no run is dropped for looking unfavourable. All 40 runs are reported, including any where A is worse.

Threats to validity

Power. Ten pairs per fault is low. Effect sizes are reported alongside p-values.
One load level and synthetic data. 5 payments per second; no real customer data.
Naive baseline. Condition B is deliberately plain Velero restore. It represents common practice and the evidence that restores often fail (Jin et al., 2024), but a tuned backup scheme could do better.
Builder bias. The author built both conditions and has an interest in A winning. Matched pairs, alternating order, identical instrumentation and a fixed analysis limit that.
DNS caching. Client RTO includes up to 60 seconds of DNS cache wait, so platform times are shown beside it.

Method and statistics