Execution, cost and decisions

Execution log

The final 40 runs ran as four chunks of 10 on 25 and 26 October 2026 (NZ time). Start times below are taken from the run artifacts. One run, A-podkill-06, was repeated on the evening of 26 October after the first attempt turned out to be a failed fault injection (see “Findings and caveats”); the repeat is counted in chunk 3 and its start time is not shown in the table.

Total time. The 40 measured runs took about 14 h 11 min of wall-clock time: four chunks of about 3 h 30 min each, plus a short analysis job after every chunk, run back to back in one GitHub Actions workflow.

GitHub shows a total of 15 h 1 min 4 s for the workflow. That figure also contains 49 min 31 s for the last job, which destroys both clusters. Destroying the clusters is clean-up, not part of the measurement, so it is subtracted here (15 h 1 min 4 s − 49 min 31 s = 14 h 11 min 33 s).

Cost and constraints

• Azure. Everything ran inside the free trial. The 40 runs took place from 25 October into the early hours of 26 October, and the 20 H3 runs on 26 October, and the project was finished before the trial ended. The account was upgraded to pay-as-you-go, but the free sign-up credit still covered the charges. October charges to 26 October were NZ$33.40 (forecast NZ$37.96 for the month), and the amount due was NZ$0.00. The largest items were the virtual machines, the Premium SSD disks and the public IP addresses.
• AWS. US$78.84 month to date on 26 October, forecast US$105.99 for the month if left running; last month was US$1.99. By service: Elastic Kubernetes Service US$48.19, EC2 compute US$12.33, EC2 other US$11.31 (items such as the NAT gateway, disks and data transfer), load balancing US$4.19, VPC US$1.88, other services US$0.94. The cluster, NAT gateway and disks were destroyed after the runs, so the forecast overstates the final bill. All of it was covered by AWS free credits.
• GitHub Actions. US$42.58 gross metered usage for October against US$18.00 of included usage, so about US$24.58 was metered beyond the included amount. All of it was the MSE907 repository. Minutes ran out in the middle of the first batch, so a spending budget was raised to finish. GitHub showed no payment due at the time of writing.
• Cloudflare. The free Durable Object plan allows 100,000 rows written a day. The customers page exceeded it and returned errors, so the account moved to the paid Workers plan: one US$5.00 invoice, paid on 25 October 2026.
• Pattern. The infrastructure is created from code, used, and destroyed when idle, so cost follows the runs and every run starts from the same state.

Problems met and what changed

Software engineering principles applied

• Separation of concerns (Parnas, 1972): controller, reconciler, load generator, fault injection and analysis are separate modules, so the direction swap changed one script.
• Design for failure (Nygard, 2018; Desai, 2026; Sun et al., 2022): hysteresis, lease, fencing, idempotency keys and reconciliation guard against a controller acting on a stale view.
• Infrastructure as code (Humble & Farley, 2010): run 40 is configured identically to run 1.
• Iterative development (Beck et al., 2001): trials showed the need for the reverse direction and for the Sydney backups.
• Testing: unit tests for the statistics, a fake batch harness and a pre-flight check that the controller matches the repository.

Status