A vertical testing strategy
One of the goals of automated testing is to support you while refactoring, which is a continuous task. Tests should not be in the way. Let’s see how.
📝 I’ll use the Arrange-Act-Assert terms (the same as Given-When-Then), which is the way tests should be split.
A flawed strategy
I often see codebases that rely on low-level unit tests to test everything; there are just a few high-level tests for happy cases. I think this derives from the myth that unit tests have to be low-level. It’s probably also because these tests are easier to write. However, unit tests are not strictly function or class tests.
A unit test is a disputed and misunderstood concept. Many think it targets a function, but that’s a low-level detail that you should be able to change freely (if the unit was always a function or a class they would have called it a function test or class test). Even classes, components, and layers are low-level details. The unit is the test subject which is a fluid concept. Unit tests target observable behavior regardless of the way they’re implemented:
Due to the unit test myth, many test suites end up with:
- Less documentary power: Tests play a central role in documenting the codebase. I have a hard time getting the gist of the app’s functionalities when all I see are technical tests.
- Weak safety net: If you test all parts separately, you’re only proving that each part works separately. You can achieve high coverage by testing low-level components but that’s a deceptive and risky strategy. I lack confidence when changing code.
- Waste: If you do lots of lower-level component testing, you also have to prove that things work when the components are integrated; you end up with a significant overlap (only varying the entry point). Repetition is usually categorized as waste. You have more tests to update when things change.
- Excessive maintenance work: Low-level unit tests know too much about structure. You end up too locked to implementation details rather than intents. Maintaining the code is more painful.
Testing all components directly greatly increases your exposure to changes. Tests should fail due to business reasons, not technical reasons. The users couldn’t care less about HTML identifiers as these are technical and volatile details. Tests should fail not because you changed a CSS class name but instead because a button is not visible. Why do we maintain tests that are so reliant on such petty details? Technical tests are brittle tests and consume too much of our time.
Changing internal APIs (e.g. adding a parameter) or code structure (e.g. splitting a component) should not have a substantial blast radius in the tests. Who cares if you use 2 or 10 components to implement a feature? We can reduce our exposure to the app’s internals by relying only on its APIs and by making them more cohesive:
… but that doesn’t seem enough. Your app keeps changing and it feels like you’re always updating tests, namely, maintaining dozens of mocks (in its worse version, patches). Sometimes, it feels like most of the effort is spent updating tests. The idea is to couple tests to behavior, not to structure. Change is a constant so we better learn to embrace it.
Vertical tests approach
How well is your testing strategy serving the goals of automated testing — quick feedback, documentation, safety net, defect locators, and refactoring tool? There are multiple ways to serve those goals so I’ll try to present mine. I propose that we create functional tests by default: tests that exercise vertical slices of your app as much as reasonable. They’re not necessarily full-stack tests — they can provide test doubles for the boundaries.
Most of your tests should target higher-level entry points. Vertical tests target functionality; horizontal tests target technical details. Vertical tests bring realism, document your code, and make refactoring work like a charm. Horizontal tests are in constant need of care, as refactoring is supposed to be an ongoing activity.
A system is never the sum of its parts; it’s the product of their interaction. (Russell Ackoff)
I believe that the distinction between unit and integration is not relevant. I do tests that give me confidence regardless of how “tall” they are. I may have some tests that are “taller” than others (e.g. involving a real database) or tests that only target the domain. In a nutshell, I recommend “taller” tests by default, cascading down if there’s a reason, but always targeting a vertical slice.
⚠️ I only recommend this strategy when applied to a single deployable unit (e.g. a server-side rendered app, a web API, a single-page app). For cross-boundaries testing, we need integration or pact testing.
A notable advantage of a vertical testing approach is its reduced reliance on your application’s underlying architecture. Vertical testing places less emphasis on the intricacies of your implementation, such as the number of layers and components, and instead highlights the user value. In essence, you should question the extent to which your internal design choices (such as the clean architecture) should influence your testing strategy.
Painless refactoring
Vertical tests have minimal coupling with the implementation and they document the code as it’s supposed to be used. After you lock the behavior of a unit with a passing test, you can freely and safely refactor it. For example, you can replace an input box with a date picker, split a component into pieces, tune up an algorithm as long as the report is generated, start without any layers, and later move on to the hexagonal architecture. All that is irrelevant as long as you keep the behavior of the unit intact.
Tests should not be a hurdle. Always remember the main reason to write tests: to keep us safe when refactoring or adding features. Tests should not impose implementation A or B; tests should solely ensure that the behavior/value is kept. I’d prefer a poor implementation with good tests than the other way around because I can refactor it in baby steps while still complying with the spec (i.e. tests).
Trade-offs
In the extreme case of vertical testing, all tests are done from the outside, “as a user” — this tests the app’s wiring, booting, interaction, business logic, and data storage most realistically. If these tests are fast, document code, help to locate defects and support refactoring, I don’t see any issue. If not, that’s not an invitation to create hundreds of solitary unit tests (black-or-white fallacy). Integration tests are a scam only if you fail to compromise strategically. This discussion should be part of your testing strategy.
The practical testing pyramid, the testing trophy, and the honeycomb describe typical trade-offs (they might be all the same given the misconceptions around unit tests).
⚠️ The “just sharing” principle: Don’t force anything just because it worked elsewhere. Scrum, the Spotify model, the microservices approach, and the testing pyramid are just learning tools and inspiration.
The ideal testing strategy and compromises depend on the nature of what you’re testing — the ideal is what better serves the automated testing goals. If it’s a CRUD app, you can get away with full-stack tests; if it’s about highly complex business logic, you can do some full-stack tests (ideally, the happy paths and error cases) and do lots of variations hitting the business domain directly (parameterized testing can help). Sometimes, you may even need to do some solitary unit tests. Don’t try to guess the exact shape from day one; run experiments and reflect on it.
Ten or twenty years from now we’ll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order. (Kent Beck)
Here’s my strategy for a domain-centric app (e.g. hexagonal):
Should we avoid mocks?
Mocks are often abused, especially when they target the intermediate layers of your app, and even more so when you mock what you don’t own. They’re too tempting but they know too much about internal APIs, which is why they drag us down when refactoring (lots of mocks to update). To make it worse, with time, you’re likely setting up mocks for things that can’t happen anymore in real life, but the tests are still green…
State verification
The way to reduce the reliance on mocks is by verifying the impact on the system (state verification). This way, we test what the code intends to do; not how it does it. We rely on the system to test the system (dogfooding). In fact, in this case, we rely on stubs, not mocks.
Imagine you wanted to develop a set from scratch with TDD. Here’s a possible initial approach:
1. Create a size test that relies on add
2. Implement add without direct tests
3. Implement size
4. Create a test for contains
5. Implement it
Notice that add was indirectly tested. As a result, size and contain can fail when there is an issue with add. I believe this trade-off is worth it as we know zero about the implementation — no mocks were used.
The idea now is to apply state verification to our apps. For example, if you transferred money, surely there’s a way to check that (i.e. view balance). If a money transfer is not verifiable, why are you building it in the first place? You shouldn’t need to develop new code for the sake of asserting; use an existing feature (e.g. view balance) to assert the transfer money command (and you might not even need to test it directly).
Begin with the end in mind
In the vertical testing approach (and in sociable unit testing), we can generally avoid mocks (we use stubs for the boundaries). This closely resembles the classicist/state-based/black-box TDD (learn more). However, there’s a twist: with vertical testing, I prefer an outside-in approach rather than an inside-out. It’s difficult for me to be lean if I develop from the inside out, as I have to guess how things are going to be used; I prefer to develop software from the outside in, by being guided by actual needs. I try to encode those needs in the tests, from top to bottom.
Let’s say I’m creating an endpoint to get a user profile. I’ll start by asserting a 200. I hardcode it in the implementation — this forces me to wire the endpoint and create a handler. I assert the actual user profile’s JSON and then hardcode it in the handler. I do the same in the remaining layers, one by one. The final step is the proper implementation. Now that I’m in a green state, I can safely refactor. This allows several rounds of feedback. It is motivating to see green tests frequently. It helps me drive the implementation in baby steps, caring about each part separately. It helps me to specify the goals first, which means I begin with the end in mind.
You may argue that you need to develop other tests in between, to help you drive the implementation with TDD. That’s fine. You can always delete them later after they’ve done their part.
Finally, any tests that were motivated solely by the development process should be considered for deletion. Working Effectively with Unit Tests
📝 TDD helps you drive the implementation and forces you to make wise API choices through examples. If it’s difficult to test, it’s difficult to use. TDD is a design strategy; not a testing strategy.
Till the boundaries
I used to resort to mocks a lot and I thought a mocking library was essential. But I also do lots of refactoring. Often, it seemed that I was working to satisfy tests and not the other way around. Nowadays, I stub the boundaries of my app (the secondary adapters, in the hexagonal architecture) and I barely use a library. In many cases, even that is not required, since I can use an in-memory or a real database for testing purposes (a fake). Regarding web service dependencies, I can stub the external replies by simulating external web services with localhost services. This way, the tests can exercise the app down to its boundaries.
What about pinpointing bugs?
Lower-level unit tests may provide better abilities at pinpointing defects, but at what price? Since change is constant, there’s a high effort to maintain them, which quickly dilutes that advantage.
We didn’t find it difficult to track down the actual fault, even if it caused neighboring tests to fail. So we felt allowing our tests to be sociable didn’t lead to problems in practice. Unit Test (Martin Fowler)
With vertical tests, we’re targeting more layers; pinpointing bugs can get more challenging, at first sight. However, vertical tests target atomic slices of value — the use cases. When there’s a bug, you just need to tell which layer. Hopefully, you have no more than 3 layers; otherwise, you may have bigger problems to solve.
📝 What if our implementation respected a use-case approach? Each use case does one thing and shares no code with others.
Won’t the tests be slower?
Performance concerns and flakiness are among the top reasons for developers skipping or deleting tests. The main goal of tests is feeling safe, so that’s bad news. To prevent that, we need to look for root causes and fix them.
Various antipatterns can cause your test suite to take too long (e.g. sleeps instead of waits, depending on external unreliable services), which is out of scope here. One thing is certain: when it comes to performance, the impact of architectural decisions and coding practices is much more significant than the programming language or database being used (and even the host and OS).
I personally prefer this [functional tests] approach for tests, as each test touches the whole system, and you don’t risk making mistakes while manually specifying expected behavior (mocking). Javalin’s test suite starts and stops more than 400 Javalin instances, and running all the tests takes about ten seconds total. Testing Javalin Applications
I just locally ran a test suite of a backend app (with domain logic serviced through a web API made with FastAPI). Around 440 tests run locally in under a minute and most of them use the vertical testing approach going through the web API, business logic, persistence with a real database, and back. With big frameworks, you may need to trade off a little more, but the general idea remains.
Had I done mostly solitary unit testing, surely the suite could run even faster, but I think the trade-off for realism is worth it. I like to sleep well at night.
Won’t the tests be long and verbose?
With vertical tests, the app’s outer boundaries (e.g. a web page, a web API) are the favored entry points. Consequently, many tests will be long and filled with low-level details such as calling web APIs and interacting with GUIs. This hurts readability and maintenance. The solution is to isolate those calls:
📝 Use appropriate testing patterns to keep the tests small (e.g. each test should isolate a behavior/effect/consequence).
After my team converted a bunch of low-level tests that manipulated the database to high-level tests that talked through APIs, the number of lines of code decreased significantly.
Conclusion
I’ve used this strategy in my last two companies. It provides me with a high level of psychological safety besides making refactoring pleasurable. Surely, there’s a trade-off in speed, but the alternative is worse in my opinion.
I feel that tests are working for me rather than the other way around. I noticed that I update tests mostly for business reasons, and rarely for technical reasons. This is a good sign because tests were always about behavior and never about code structure.
📝 The presented strategy bears some resemblance with ATDD and BDD, so it might be worth reading more about them.
