Unit Testing: Values and Principles
What’s a unit test? A function test? Can it interact with the outside? That’s meaningless if we don’t start with why we do testing in the first place.
Why do we write tests? I heard sentences like “because we need to have coverage”, “if you have a public class it needs a test”, and “because it’s part of our methodology”. None of that addresses why we test things… and we end up with a mechanical and blind approach to testing. Trying to precisely define a ‘unit test’ or providing a set of guardrails does more harm than good. People start working to appease the definitions rather than the original goals (when a measure becomes a target, it ceases to be a good measure). Instead of just another definition, I propose we start with why we do testing. Identifying reasons can provide direction and purpose.
Values and principles
Values and practices are an ocean apart. Values are universal. Ideally, my values as I work are exactly the same as my values in the rest of my life. Practices, however, are intensely situated. […] Bridging the gap between values and practices are principles. Principles are domain-specific guidelines for life. Extreme Programming Explained
Automated testing is just a practice and as such it should be guided by principles and values. Values represent our mental compass and what drives our principles. Values are ultimately why we do things.
There are only two ways to influence human behavior: you can manipulate it or you can inspire it. Start with Why
By uncovering the values and principles of testing, we can end up with a better definition of the practice. It provides direction and purpose — when in doubt, we can always ask “Why do we write automated tests?”. We end up with a better testing strategy. Questions like “What kind of test should I do?”, “Is it worth testing this?”, “Should we have contract testing?” all concerns our values and principles.
What are the values behind unit testing? It depends on the context, but usually, it’s about quality, maintainability, collaboration, and feedback. We could ask why and explore them further; we would possibly talk about the joy of crafting solutions and making people’s lives better, but that’s not the goal here. What about principles? Here’s a typical list:
- Automation: tests should be automated so that they can be run repeatedly and easily, locally and in the CI/CD pipeline. Running tests by point and click is too tedious and error-prone.
- Determinism: the same scenarios should always yield the same outcome. Testing units in isolation from the rest of the system can help a lot.
- Quick tests: the test suite should not be so slow that people start bypassing it. The economy of time should be always in our minds.
- Testable code: write code that is easy to test, which involves adhering to certain design principles and best practices. TDD is key here because tests are the first clients.
- Early and frequent feedback: write early in the development process and frequently run to detect defects and bugs as early as possible.
- Refactor with confidence: make changes to the code without fear of breaking existing functionality.
- Refactor smoothly: make changes to the code without tests being a burden. You shouldn’t spend most of your time updating tests or they lose most of their value. Not depending on implementation details is key here.
- Integrate and deploy frequently: automated testing is an essential enabler of CI/CD, which emphasizes frequent and automated testing, integration, and delivery of changes.
- Documentation: tests provide clear examples of how to use the code in different scenarios. Tests serve as living documentation as they’re always up to date in opposition to code comments and documents.
These principles are closely related to the automated testing goals. It’s all about building a safety net and achieving a good measure of maintainability and evolvability.
It should be clear by now that the team’s definition of ‘unit test’ should come up as a consequence of the team’s principles. Each team may come up with their answers, depending on the context. Each team may assign different weights to each principle. The testing strategy should also be derived from the principles. It’s important to have these conversations, or each team member will be guided by different values and principles.
The “unit is a function” myth
I often see codebases that rely on low-level unit tests to test everything; there are just a few high-level tests for happy cases. This emerges from the myth that unit tests have to be low-level, such as function tests. It’s probably also because these tests are easier to write.
Unit tests are not function tests or they would be called function tests instead (the same applies to classes). According to the Cambridge Dictionary, a unit is a single thing or a separate part of something larger. In software, a unit just means an isolatable and cohesive part; not the tiniest piece of software.
A function is a low-level detail that you should be able to refactor freely, without making constant updates to tests. Classes and even software layers are implementation details. Surely, some components may need to be tested in isolation (e.g. because they’re reusable or have an intrinsic complexity) but that is the exception, not the rule.
Testing all components directly greatly increases the exposure to changes. Tests should fail due to domain reasons, not technical reasons. For example:
- A test should fail not because you changed a CSS class name but instead, because a button is not visible (your users don’t care about test/HTML identifiers; these are technical and volatile details).
- A test should fail because a user was created but it’s not able to log in; not because a mock was not called.
Variable unit size
Tests can target units of varying sizes but generally, bigger is better. The smaller ones are called solitary; at the other end, there are sociable. These can interact with databases or fake services under the same umbrella. Although the distinction is a bit irrelevant to me, you can call integration tests the ones that involve other systems with their own identity. These definitions can become handy for practical communication purposes but should come up at last, after alignment on the values and principles.
But not all unit testers use solitary unit tests. Indeed when xunit testing began in the 90’s we made no attempt to go solitary unless communicating with the collaborators was awkward (such as a remote credit card verification system). Unit Test (Martin Fowler)
If you do many micro/solitary tests, you’re too coupled to implementation details. Tests become about technology rather than the domain. You’re constantly dragged by the weight of thousands of tiny tests. Refactoring becomes painful. These tests barely document code; they’re just creating a false sense of safety through an elusive high test coverage.
Don’t test technical things like utilities — that’s an antipattern (even software layers can be seen as an implementation detail which is why mocks are to be avoided). Units are subsets of the domain (i.e. the problem the software solves) rather than technical aspects. Your tests should revolve around units. Naming your test files after the unit they exercise (e.g. TestProfilePage.kt, test_upgrade_user.py, EnterEconomyMode.test.ts) can help to foster this mindset.
📝 You should organize/name everything (packages, folders, files, test titles, etc.) after the domain rather than the technical side of it. Make your codebase revolve around the domain, not the technology.
Observable behavior
The unit is the system under test (or test subject). It’s a discrete (isolatable) part that provides value on its own by exhibiting a set of behaviors. You can isolate UI components, whole systems, or anything in between but it must exhibit discernible behavior. Unit tests should target that observable behavior regardless of the way they’re implemented. It’s all about the usage standpoint.
If you think about a unit managing a state, a behavior can be obtaining a part of that state or a way to manipulate it. This is known as the query-command separation and it applies to any level of unit size: from a function to a system. Any observable behavior of a unit materializes as a query or as a command:
- Queries return a result and do not change the observable state of the system (are free of side effects) (e.g. a function to sum two numbers, an endpoint to list orders, or a screen to check the bank balance).
- Commands change the state of a system but do not return a value (e.g. a function to emit an event, an endpoint to store a profile, or a button to save a document).
Unit testing is about the isolation of behaviors. It’s about the capture and exercise of specific behaviors of a system in a predictable way, from the outside-in. That can be achieved through its interfaces (i.e. entry points).
- While testing a website, the unit is a page; the actions are clicks and keystrokes.
- In a web API, the unit is each endpoint; the actions are the interactions.
- In end-to-end testing of a web app, the unit is a user journey (although no one calls these tests ‘unit tests’, the same principles apply). On higher levels, the observable behaviors match the apps’ use cases as they describe functionalities in usage terms.
In the spectrum of testing levels, what varies is the size of the unit. Sometimes, the unit can be a lower-level concept (solitary unit tests) if it has a stable interface, it’s potentially reusable, and has its own rich behavior that shouldn't leak outside:
- If you’re testing a rich UI like a dashboard, the unit can be the individual UI panel.
- In a UI component library, the unit is the component (e.g. auto-complete dropdown, date range picker, filterable table).
- An API gateway adapter may be the unit under test as you may want to test auto-retrying or caching.
Surely, the entry points may be functions but that doesn't make them necessarily the test subjects. They’re solely used as interfaces for the actual behaviors, the real test subject. Many will be tested indirectly. Therefore, always test as a user (an interface user), from the outside, knowing as little as possible about the implementation. Testing only using interfaces is a great way to ensure the decoupling of tests from implementation (e.g. don’t share constants).
The more your tests resemble the way your software is used, the more confidence they can give you. (Kent Dodds)
Testing strategy
I haven’t strictly defined what a unit test is because I don’t like the distinction between unit tests and other types of tests. In my view, there’s no unit testing; there’s only automated testing which displays a spectrum of abstraction levels that all work toward the testing goals.
It follows from the principles and the elastic nature of the unit size that not everything must be tested the same way. For example, you may have lower-level testing when you want to do lots of combinations and variations. You may have higher-level testing (like smoke testing) for things that are on the critical path. There’s no universal testing strategy. The ideal depends on the nature of the system under test; it’s the one that best respects your principles.
Additionally, if all tests were fast and a good source of documentation and pinpointing abilities, nothing would impede us from testing everything as a user. Trade-offs should happen strategically rather than by decree. The practical testing pyramid is just a side-effect of those tradeoffs rather than an ultimate goal.
Let me leave you with some additional advice that can help you shape your testing strategy:
- Don’t focus too much on guardrails like “it can’t contact the network”, “every class needs a test”, or “it must be under 50 milliseconds”. Instead, focus on why we do testing in the first place — when in doubt, revisit your principles. My mnemonic is “safety net, docs, ease of refactoring, pinpointing ability”.
- Don’t waste time trying to distinguish unit testing from integration testing, and end-to-end testing, … The line that divides them is so thin and arbitrary that I prefer to ignore it and just be guided by the principles while doing tests, regardless of the size of the unit. Even though the team can come up with their definitions, that should come in last, as a consequence of the principles.
- Tests should never feel like a burden. Feeling annoyed because you have to update tests or because they’re slow or brittle are clear signs that the testing strategy is flawed. You should feel tests are a blessing or you’re doing something wrong. Tests should always pay their cost.
- Tests are not a goal. They’re just a supporting mechanism. They exist to assist us in delivering software with quality. We need to find ways to get the most out of them while doing the least of them.
- Beware of the fallacy “Tests are good; therefore, the more tests, the better”. Per test, you should do an informal cost/benefit analysis, asking if it’s worth it while being guided by the agreed-upon principles. Those will also help to define the size of the unit under test and if the test is worth writing in the first place. I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence. (Kent Beck)
- Trying to come up with a testing strategy from the start to follow from there on is a waterfall mindset. We uncover it only by trial and error. We keep adjusting it as we find we’re deviating from the principles. Ten or twenty years from now we’ll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order. (Kent Beck)
Conclusion
Many people enjoy strict definitions. It’s all about feeling in control of the things around us. However, trying to force fit things into boxes can be wasteful because they can’t always be organized so nicely. It can also be harmful because people end up writing tests for their own sake forgetting about the original goals.
Tools, techniques, and practices should never be upfront decided; they should come as a consequence of values and principles which is why I propose you start there. For inspiration purposes, I still recommend reading some definitions. There are some desirable universal properties that tests should have: the test desiderata.
Writting tests just to appease formal definitions may bring consistency and high coverage but it doesn’t ensure a good safety net or code that it’s easy and enjoyable to refactor. For example, tests can become a burden because they’re slow or need constant updates. We may be all well-intentioned, but if our values and principles are not aligned, we are all optimizing for different goals; thus doing a different practice.
Besides, I’m tired of all the energy-draining discussions around the difference between ‘unit testing’ and ‘integration testing’. I propose we redirect those efforts toward defining values and principles. Whenever I don't know what to do, I revisit them (as a pair or as a team) and that guides most of my decisions. This allows for more flexibility in each case. There’s no hard rule. I may decide on a different type of test per case, depending on the confidence I need to feel, the costs of the test, how decoupled they need to be from the implementation, how they’re documenting the code, etc.
