Friday, January 1, 2016

Testing Complex Software manually is impossible

Suppose that we have a program that we managed to split into multiple pure functions.

Each pure function has one a switch statemenet with 5 possible cases, thus 5 different paths of code execution.
Let us assume that the output of one function is the input of another. In other words, our main program is the composition of those functions.

Let us assume that the total number of functions is n.

How many unit tests do we need to make if n=6?
number of functions * number of code execution paths = 6 * 5 = 30

How many tests do we need to make to test the global behavior of our program if n=6?

5^6 = 15624

One may say then that we should only do unit tests and avoid global tests. This is wrong.
Testing allows us to check whether the behavior of a function matches our intuition and our expectations. Does it have the results we designed it to give?

Thus the global behavior might indeed be determined by the behavior of all the functions, but can that tell us anything about the behavior of the whole system? Can it tell us if the global function does what it is supposed to do from our design?


So, the only way to test the global behavior of programs is to use fuzzers that check all possible paths of our code.

But fuzzing doesn't mean giving random data as input. That would not work. One needs to create data that will trigger new paths of execution. A good article on fuzzing can be found here.