The goals of automated testing
Testing is more than creating a safety net. Let’s talk about its multiple facets.
In an agile mindset, the team is responsible for the delivered software. In this scenario, QA is ensured by the whole team. Automation is a core tenet of DevOps culture, meaning developers should write automated tests. Manually testing everything does not scale and it’s not compatible with continuous delivery. QA-specialized people should be doing other types of work like exploratory testing, usability testing, acceptance testing, etc.
Automating everything — from build to tests, deployment and infrastructure — is your only way forward. The Practical Test Pyramid
According to the xUnit Test Patterns book, there are multiple goals of test automation:
Tests as specification
Tests are like the floor plan of a house (although in software development we can iterate); they define the system’s behavior, or in other words, what is expected: as soon as tests go green, they lock the implementation to that contract (design by contract). That’s how precious and precise a test is, given that it enforces a specific implementation (the how) to faithfully respect that spec (the what).
I know this sounds ridiculous; but consider. If somehow all your production code got deleted, but you had a backup of your tests, then you’d be able to recreate the production system with a little work. Test First
Tests are a self-verifying executable specification since you can run them and they will automatically report the outcome. In RSpec and BDD in general, tests are called specs for that reason. Each test should execute only one action on the test subject. Also, that action may have multiple side effects, but each test should assert only one.
TDD (test-driven development) takes the concept of specification further by actually driving the implementation (only implement what the test says and no more; if there’s no test, you can’t implement or refactor anything; feel free to refactor as long as all the tests still pass).
TDD forces you to separately think about the what (API) and the how (implementation), so you dedicate more brainpower to each (without TDD, you think about both simultaneously). Even before knowing about TDD, I used comments to specify my intentions — what the implementation was supposed to do. Tests should target the test subjects as seen from the outside consumers, making them more likely to be black-box. TDD is sometimes referred to as test-driven design — it guides the design since the test is the very first client of the implementation.
When we write a test, we imagine the perfect interface for our operation. We are telling ourselves a story about how the operation will look from the outside. Test-Driven Development: By Example
Tests as a feedback mechanism
Using a REPL, print-based debugging, regular debugging, and point&click testing are ephemeral and cumbersome testing strategies. Automated tests provide determinism and repeatability. This is a cornerstone to providing near real-time automated feedback.
When using TDD, you have multiple small and immediate feedback cycles. This encourages you to move steadily in small steps. For TDD to be successful, the feedback must be quick, trustable, and deterministic. This is also true in regression testing, where you want to know immediately if you broke some test (even better, you want to know the line of the failed assertion).
The very act of thinking through various scenarios in enough detail to turn them into tests helps us identify those areas where the requirements are ambiguous or self-contradictory. Such analysis improves the quality of the specification, which improves the quality of the software so specified. xUnit Test Patterns
Tests as documentation
When building a new feature or when a bug is caught, you’re supposed to create tests (we do tests by creating examples). This enforces the idea that tests capture the behaviors of a system. Tests have enormous descriptive power. Unlike code comments, readme files, or ADRs, tests are living documentation.
📝 I don’t recommend code comments in general. They get quickly outdated and nobody cares anymore. Additionally, they’re an excuse to make bad code with good explanations. You can’t do the same with tests.
Whenever I look at new code, I start with the tests, especially the higher-level ones. This allows me to understand what the code does. Like with an instructions manual, both tell me “as a user, what are the features and how can I use them?”. I don’t need to disassemble the washing machine nor do I need to walk through the implementation code to understand its features.
Repelling bugs isn’t the only thing the tests can do for us. They can also show the test reader how the code is supposed to work. Black box component tests are — in effect — describing the requirements of that of software component. […] Without automated tests, we would need to pore over the SUT code trying to answer the question, “What should be the result if …?” xUnit Test Patterns
[…] Improved readability of tests brings benefits far beyond just catching problems. Tests are often the most accurate documentation about a system. New people joining teams can quickly get up to speed by reading the tests. People can use tests for discussions about potential changes and improvements with stakeholders. Fifty Quick Ideas To Improve Your Tests
Tests are also a learning tool: writing high-level tests facilitates discussions about the system and the domain among teammates.
Remember that tests are what and the implementation is how. When a test fails, the first thing you do is to understand it. This means you can start by understanding what is failing by simply looking at the failing test. The implementation can change as long as the specs are respected.
The fact that tests are documentation has some consequences on the patterns and anti-patterns that arise in their source code; for example:
- Make tests more “DAMP”; less “DRY”;
- For high-level tests, consider creating methods resembling a DSL;
- Follow the Arrange, Act, Assert pattern.
Tests as a safety net
Feeling safe is the strongest reason to write automated tests. Ideally, the system should be resilient and protected from mistakes. I’d argue that having fun while coding is a side-effect of feeling safe doing it. You can’t enjoy coding if you are always afraid.
I’ve seen bugs arising because of a refactor without a high-level test of the feature before. There are anecdotes of code being merged only after being reviewed by the team leader (the bus factor is ridiculously high here). In other cases, developers get their code peer-reviewed by multiple people to reduce the fear of changes. This is like driving a car without a seat belt with everyone continuously on the lookout for dangers. Tests are like insurance: you don’t like paying it, but if something happens, you’re happy you had it in the first place. Tests protect from inadvertent breaking changes (regression testing).
“The more tests the better” is a fallacy; the safety net doesn’t imply that you should test everything in every way; tests don’t prove that the software works well in all conditions since they are only examples (leaving out property-based testing). This is about risk management: finding the sweet spot between the minimum set of tests and the maximum level of confidence. At the end of the day, what matters is how confident you feel in the system.
I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence. (Kent Beck)
You need to be careful with some anti-patterns that can hinder the safety net. For example, slow tests or randomly failing tests can be disabled by developers, thereby defeating the sole purpose of testing as a safety net.
Tests as defect locators
If you push some changes and a bug happens in the CI/CD pipeline, you don’t want to spend an hour trying to understand what was wrong while the build is red and nothing goes to production. Ideally, you should be able to quickly identify the culprit. You should be able to run tests locally and get quick feedback. High-level tests are not ideal for that because they’re slow, have more code (they might even need DSLs), and target a wide scope of implementation code. This is why we also need low-level tests. It’s also a good counterargument to “I only need customer tests”.
If our unit tests are fairly small (i.e., we test only a single behavior in each one), we should be able to pinpoint the bug quickly based on which test fails. This specificity is one of the major advantages that unit tests enjoy over customer tests. xUnit Test Patterns
Tests as a refactoring tool
Tests are there to help you with the constant process of refactoring. They are not supposed to be in the way when evolving the architecture. As Martin Fowler puts it:
Whenever I do refactoring, the first step is always the same. I need to ensure I have a solid set of tests for that section of code. The tests are essential because even though I will follow refactorings structured to avoid most of the opportunities for introducing bugs, I’m still human and still make mistakes. Refactoring
Conclusion
Whenever I have to create a new test, I think about where to place it in the testing pyramid. Then, I ask if that placement fulfills the goals:
- Tests as specification: Does each test have only one action and assert one behavior? Do tests help to drive the implementation?
- Tests as feedback mechanism: Do tests provide quick, trustable, and repeatable feedback? Do they fail when you change the implementation?
- Tests as documentation: Can the tests tell you what the system under test is capable of? Do they document the behaviors?
- Tests as a safety net: Does your CI/CD pipeline prevent the deployment of failing builds? Do you run the tests before pushing?
- Tests as defect locators: Can you quickly pinpoint the cause when a test fails?
- Tests as a refactoring tool: How confident do you feel when refactoring? Do you feel supported?
