How to Test Software in 2022
I’ve been developing software for ~ 30 years. Here are my current ideas on software testing given that experience.
Prompts and Constraints.
If you’ve read any of my recent articles these terms are clear. They are how I’m currently viewing the entire software world, and indeed much of the world of ‘creative intelligence’ in general.
Prompt: A small natural language (English, etc..) body of text that hints at the desired outcome.
Constraint: A domain specific set of rules that set boundaries around the problem.
It is important that constraints are automatable. They should be so easy to use they are almost invisible. Like gravity. Always there, to the point that the one creating a solution is focused on the problem itself not on wrestling with the boundaries of the problem. Constraints are accepted and internalized by the problem-solver, and are helps to narrow and focus the problem domain.
Prompts describe the problem and solution. It is important that they are human readable. There may be specialized wording ( jargon) used, but it should be kept to a minimum. If jargon is used, it should be used consistently so as not to create confusion. The goal of a good prompt is clear, concise communication.
What is a Test?
In the context of software development tests take on several roles:
- Constraints as described above
- Confidence building (building trust in the solution)
- Domain knowledge (understanding the problem)
I’ll argue that writing software without tests is impossible. I can say this as I have a broad view of what a ‘test’ is.
One can open up a text editor and type random characters, but this is unlikely to yield a good result.
The first ‘test’ one often encounters is the compiler/evaluator of the program. It will simply not do anything remotely useful unless it is satisfied.
From there one goes on to more and more sophisticated testing. Anything from type systems to linters to unit/integration/functional tests, etc…
Importantly, tests are composable. That is to say they are independent, and one can add and mix and match them without limit. One can always add a new test on the existing pile.
Trust
Note that I added ‘confidence building’ as a key aspect of a test. As one deals with creating software, one comes to realize that one is mainly dealing with managing complexity over time.
It is one thing to say one has a solution to a problem that worked once, it is another to say that one has a reliable solution to a problem that is likely to work over time.
As one builds up a testing environment, one also builds in trust that the written software does what is intended, and can that it be extended without fear.
A good testing environment becomes critically important as the project size grows. Eventually it becomes impossible to add additional complexity (code) to a system unless one has trust in one’s tests. Regressions will come to dominate otherwise.
In particular these tests MUST be automated so that they become constraints to the project as a whole. This is what gives one the ability to solve new problems (add features) within the context of an existing project’s ‘software ecosystem’.
Tools/Frameworks/Paradigms
Over time many testing tools and frameworks and paradigms have been developed.
There is no one ‘right’ tool/framework/paradigm for testing, and there doesn’t need to be.
Testing is composable. This means one has the freedom to pick and choose, add and remove, as needs dictate.
It is important to be smart and pick good testing tools/frameworks/paradigms, as this in many ways will determine the outcome one gets.
Software moves fast. It is important to ride the current waves, while also paying attention to overall larger trends.
Adopt and try out a bit of everything and see what eventually works. Don’t be afraid to try new things, but also lean on experience to know what has proved useful in the past.
Bureaucracy
I want to make a special note that ‘too much of a good thing’ is real.
It is seductively easy to fall into the trap of a testing environment becoming poisonous to the very project it is trying to serve.
As the testing environment becomes heavier and heavier, with more and more tests added to the mix, there is a real danger that the testing environment becomes too overbearing. Progress itself becomes impossible, and the project hardens, even fossilizes, to the point that it can no longer adapt quickly, eventually it can no longer adapt at all.
Like most things in life, it is a balance. One is fighting complexity and there are dangers on both the ‘too much’ and ‘too little’ side of the question of ‘how much order to the chaos do I bring?’.
Bravery is the only solution to this I am aware of. One has to tear down the very foundations of the world to make it better. I’ll note that for good reasons, this is a dangerous activity more likely to bring ruin than delight.
IMHO it is better to fight this fight in small chunks vs all at once. Recognize it is a hard problem and continual struggle.
In other words, it is better to always be on continual guard for this problem, and be quick to tear down impediments to progress as they become recognized, before aggressive ‘revolutionary’ solutions are needed which are a roll of the dice, more likely to kill the project than save it.
Tests as a Repository of Domain Knowledge
This is more of an ‘advanced’ use of testing environments. One I feel is often overlooked, especially on smaller projects.
The reality of software development is that individuals come and go as the project continues.
The project itself is its own entity. In business we have come to understand there is such a thing as a ‘corporate person’
Unlike a ‘coproarate person’ a project doesn’t have legal rights (at least not as of the time of writing :) ) but I feel it is the correct lens from which to view any sufficiently large project.
After a while a successful project really does ‘take on a life of its own’, and I mean this in a non-derogatory, non-ironic way.
Tests can be used to store knowledge about the ‘business domain’ that is stable and lasts over time.
One paradigm that attempts to incorporate this view is Behavior Driven Development (BDD). I’m not sure I’d recommend this particular approach/paradigm on the whole, but I do think one can use it ‘lightly’ and steal from it the ‘good parts’.
In particular I think that BDD leads one to writing tests that expose the ‘prompts’ in clear, understandable natural language, that govern the project, paired with a ‘constraint’ that can be used as an automated test.
As new people are brought into the project, there is a natural place for them to go to understand the project quickly. For instance, jargon and business rules can be defined ‘by example’ for those who can read code.
BDD falls into ‘Test Driven Development’ TDD. I think one can become ‘too religious’ on this topic and can lead to the nightmares of bureaucracy I described above.
However, I do think the concept of pairing the natural language prompts with the code (constraint) in a way that both can be ‘linked’ together has great value (if it can be achieved) to strengthen the ‘bones’ of the project, and insure it stands the test of time.
Summary
Testing is a critical area of software development. One is going to create a ‘testing environment’ whether one realizes it or not.
It is better to be conscious and deliberate (awake) when making the choices of a testing environment, as this environment can determine the fate of the project (whether one is aware of it or not).
There are no silver bullets. It will take time and effort.
Prompts and Constraints are a good lens from which to view the problem of testing. Write clear, simple natural language for describing the ‘business’ side. Use fast, simple, effective testing tools/frameworks to constrain and therefore aid development. Use both prompts and constraints linked together to gain confidence in the project, and ensure it has a long life.
Originally published at https://github.com/matthewjosephtaylor.
