Final word on testing

Next: Property (starting)

In two previous posts we’ve discussed how to test software and physical infrastructure in cases where chaos engineering and in the last post how to handle testing when this is simply not possible (real-time services(.

Outside of these two categories there are more variants where we cannot use either option. Let’s explore how to test critical infrastructure where you cannot mess from the outside in any manner whatsoever.

How to test in physical world?

In the physical world there are many domains where one cannot perform chaos engineering. How to test in those environments?

These tests are today fairly common today and are called randomised trials. For example, to know how good some anesthetic is, it would be unethical to run a test where someone is left without, but perfectly fine to run a test where two alternative drugs are compared against each other – current best practice and the new one under testing. Which works better and are the differences based on some characteristics of the patients?

Testing two types of concretes for bridges is also possible if both are known to work but it is unclear which one is better on the long term in specific conditions (i.e., you monitor the wear over decades to see which mix requires repair earlier).

Testing is sometimes possible also in health. For example, 2013 Finnish team of surgeons showed that knee operation for meniscal tear in middle aged patients was no more effective than sham surgery. The operation – meniscectomy – is performed millions of times every year. All chirurgical operations carry a risk to the patient and the cost to society for ineffective operation is big. The last example is from the book ‘Randomistas: How Radical Researchers Are Changing Our World’ by Andrew Leigh.