Automated testing patterns

Testing patterns serve the goals of automated testing: specification, documentation, support for refactoring, and help in locating defects.

Creating patterns to describe something creates nomenclature. Once you have a name for something, it’s easier to recognize when you see it again. The Productive Programmer

I’ll present the testing patterns that I apply in every project since they’ve proven very beneficial to the team concerning the goals of automated testing. I’ll skip the well-known patterns like “don’t mock what you don’t own”, “tests should be independent”, “tests should run fast”, and “tests should run in the pipeline”. This is a very incomplete list as I’ll focus only on the most impactful patterns.

“One SUT action per test”: focus on a single SUT ability per test;
“Arrange, Act, Assert”: always divide tests into three parts;
“Split tests by behavior”: don’t assert distinct behaviors in a single test;
“Resort to object generators”: use test data generators to highlight only what matters in the test;
“Create a SUT client”: if SUT calls are verbose, extract them to a reusable place;
“Assert with the SUT”: avoid under-the-hood assertions like mocks and spies;
“Arrange with the SUT”: avoid under-the-hood setups.

All the patterns apply regardless of the test abstraction level; from low-level unit tests to system tests. They are valid regardless of the programming language. Most patterns only show their glory when applying test-driven development.

📝 The system under test (SUT), also known as the test subject, is whatever you’re testing. It can be a function, a component, a set of components, a UI, a whole app, or a whole system, … You interact with it through its interface, such as function invocations, HTTP calls, GUI interactions, etc. A test double is an umbrella term for dummies, fakes, stubs, spies, and mocks.

One SUT action per test

A test should isolate one scenario. If you have to think too much about a test title, you may be testing multiple scenarios. These are multiple tests disguised as one. Avoid “telling of a story” with multiple steps. Even end-to-end tests are supposed to isolate a single meaningful user task. Don’t anchor tests on others only because the system is in a certain state (known as “The Free Ride” or the “Piggyback” anti-pattern).

❓Why? Tests are also specifications and defect locators so it’s a good idea that each test only has one reason to fail. Long tests drain the energy of future readers. Keeping tests short and focused eases refactoring.

❓How? Split the test into independent tests and don’t worry if you have to repeat some actions to put the system in the desired state. The pattern “Create a SUT client” may help. Name your test after a single clear action per test to ensure that you only exercise a specific SUT ability (e.g. “download a file when clicking Export”).

Arrange, Act, Assert

Any test, regardless of its abstraction level, should display exactly three phases: Arrange, Act, and Assert (also known as “Setup, Exercise, Verify”, “Given; When; Then” or a Four-phase test):

Arrange: initializes the SUT, test doubles, and fixtures;
Act: executes the SUT action;
Assert: verifies test expectations.

This is a simple and powerful pattern due to what it promotes but it’s too underrated as I rarely see it in codebases. It sets the stage for all the other patterns since it defines a structure and terminology so don’t move forward before understanding it.

❓Why? Improved documentation because each part is quickly identified by the reader. Tests reveal intent.

❓How? Follow the “One SUT action per test” and “Split tests by behavior” patterns. As a hard guideline, each test should only have two empty new lines. The Arrange and the Assert should bear the minimum required for the test case. The Act should be a single-liner. There should exist some decoupling between Arrange, Act, and Assert.

Split tests by behavior

You might still be testing multiple things if you follow the “One action per test” pattern. For example, a user registration test could check email sending, actual user creation, HTTP output, and generated logs. However, each of those behaviors deserves a dedicated test. A behavior is a reaction to a SUT action — an output or one of its side effects. Think of this pattern as a way to separately describe the different facets of a SUT action. If you need to say “and” in the test title, you’re likely asserting more than one behavior.

The “Single expectation test” (a guideline from the RSpec community) defends one assert per test. It’s a good rule of thumb, but I think it’s too extreme: it’s fine to have multiple asserts as long as they all relate to the same behavior.

❓Why? If you’re asserting multiple behaviors in a test, you’re providing your test with multiple reasons to fail. When things fail, you want to quickly know why — to see one or a few assertions that relate to the same consequence.

❓How? Before applying this pattern make sure you're following “One action per test”. Then, split tests that assert multiple behaviors into multiple tests. It’s OK to have repeated Arrange, Act parts (if you end up with long and verbose code, the “Create a SUT client” pattern may help). However, beware of repeated assertions between tests, as each test should focus on a single behavior. The test title should reference the behavior as a verb (e.g. “download a file when clicking Export”).

Resort to object generators

By default, everything that a test needs should be in place, but quickly comes a time when there are too many lines, especially in high-level tests. Object generators come to the rescue — they are generators of sample objects (e.g. domain, database) to be used in the fixture setup (in Arrange).

def create_order(
    status: Status = Status.CLOSED,
    po_number: str = "xyz123",
    destination: Location = Store("there")
): 
   return Order(
      po_number = po_number,
      status = status,
      destination = destination,
   )

def test_x():
   # override only what matters for this test:
   order = create_order(status=Status.OPEN)
   ...

Externalizing object creation might not be needed in simple tests. However, since you want to keep tests short and focused, it quickly pays off.

📝 This is an exception to the rule to avoid shared global utilities since multiple tests might need domain or database objects.

❓Why? A test is supposed to isolate and document a scenario — a test reader should only have to read what’s relevant for that test and no more. Object factories hide data that is not relevant (i.e. you could randomize and the test would still pass) therefore highlighting what matters for the test scenario. Also, since object creation is isolated, refactorings can be less painful. Fewer lines of code mean better maintenance.

❓How? Object generators can be immutable copyable objects. Alternatively, they can be functions, which can rely on the factory method pattern or the builder pattern. What matters is that these functions generate objects built with dummy data (which ultimately can be randomized) and each test will pass them only what it’s relevant to it.

⚠️ Don’t do side-effects in testing helpers like storing objects in the database. Factories should just return objects. Side effects should be explicit in the test.

Create a SUT client

Think about the verbosity of making HTTP calls or resorting to HTML page selectors in the tests. What if those were just normal function calls? The test would read almost like prose. The solution is to segregate them all in a place — you can call it a SUT client; i.e. a set of reusable functions to interact with the SUT mimicking the way real clients use it.

I’ve described this pattern in terms of HTML, but the same pattern applies equally well to any UI technology. I’ve seen this pattern used effectively to hide the details of a Java swing UI and I’ve no doubt it’s been widely used with just about every other UI framework out there too. Page Object (Martin Fowler)

❓Why? It reduces boilerplate code and hides low-level details by respecting the “Same level abstraction” principle (so you can focus on the what; not the how). It significantly reduces the amount of test code. It contributes to testing as documentation as it shows how clients should use the SUT and it talks to you in domain language. It focuses tests on possible scenarios rather than impossible ones (e.g. when manipulating the database or mocking, you can create impossible scenarios). It makes refactoring easier because the interactions with the SUT are segregated.

❓How?

Decoupling tests from the implementation interfaces

Changing interfaces (e.g. API web handlers, queue event handlers, web pages) is quite common but it can impact many…

lsoares.medium.com

Assert with the SUT

SUT actions can be queries — return data without any side-effect or commands— side effects that mutate the system. A query is easy to test: just assert its output. The heated discussion starts when testing commands. One solution is to analyze its side-effect on a test double (indirect); the other is to ask the SUT about its new state (direct).

Let’s say you’re implementing your set data type. How do you test the remove method? I guess you won’t be checking the set’s private details and just resort to contains. Now, you just need to apply this idea to higher-level testing. For example, to test a user creation through a REST API, you’d assert using its details resource (or any other client visible mechanism). Here’s a realistic example where the method list users is the SUT client function used in the Assert phase — no need to populate the database or set up a mock:

In short, asserting a command should be done against the same API as the command itself. As a consequence, the tests become more vertical/full-stack rather than targeting each component individually and passing mocks. The testing surface becomes smaller. Also, some SUT methods don’t even require dedicated tests.

📝 This pattern goes in hand with the Detroit school of TDD (state-based testing).

📝 How long do you spend refactoring tests? Roughly calculate the return on investment and that will make you rethink the types of tests you do.

❓Why? Tests become less fragile. More realistic testing — testing as a SUT client — thus more confidence in the tests. Very low coupling with implementation details (less mocking) hence easier refactoring. Also, tests become more about business and less about technology.

❓How? Follow the patterns “One action per test”, “Split tests by behavior”, and “Create a SUT client” to keep the tests concise. Don’t assert with test doubles (use them only for the boundaries of the SUT). Instead, assert with the SUT API but make sure you don’t create implementation code solely for the sake of tests.

The direct approach of asserting has less coupling with implementation details.

Arrange with the SUT

For consistency and symmetry with “Assert with the SUT”, you should rely on the SUT’s API during the Arrange phase. For example, to test a user deletion, you’d start by calling the SUT’s method to create a user rather than going to the database and inserting rows.

def test_delete_user():
   App().start()
   client = ApiClient()
   client.create_user(...)

   client.delete_user(...)

   assert client.get_user(...) is None

Here’s a realistic example where create user is the SUT client function used in the Arrange phase:

⚠️ In this pattern we’re trading pinpointing abilities for realism. A test may fail due to unrelated reasons (e.g. the test “deleting a user” can fail due to a bug in “create a user”).

❓Why? The same as “Assert with the SUT”.

❓How? The same as “Assert with the SUT”. Also, since we’re using the SUT itself to arrange, there’s no way we jump over important business rules.

The direct approach of arranging has less coupling with implementation details.

Conclusion

The presented set of patterns makes your tests achieve the automated testing goals but also allows you to follow the good practices of BDD and ATDD.

These ideas didn’t qualify as patterns but they complement them. I try to keep them in the back of my mind whenever I write tests:

Remember to ask, for each test, how it is serving each of the automated testing goals.
Don’t make me think. A test should be self-contained and self-explanatory. Don’t make it depend on anything other than itself. The only typical exceptions are the patterns “Resort to object generators” and “Create a SUT client”.
A test should not contain more code than its scenario needs. After the scenario is properly isolated, remove everything that doesn’t make the test go red.
A test barely needs variables. Resort to the “Evident data” pattern (TDD by Example, Kent Beck, Chapter 25).