Rethinking Unit Test Assertions

Well written automated tests always act as a good bug report when they fail, but few developers spend time to think about what information a good bug report needs.

There are 5 questions every unit test must answer. I’ve described them in detail before, so we’ll just skim them this time:

1. What is the unit under test (module, function, class, whatever)?
2. What should it do? (Prose description)
3. What was the actual output?
4. What was the expected output?
5. How do you reproduce the failure?

A lot of test frameworks allow you to ignore one or more of these questions, and that leads to bug reports that aren’t very useful.

Let’s take a look at this example using a fictional testing framework that supplies the commonly supplied pass() and fail() assertions:

describe('addEntity()', async ({ pass, fail }) => {
  const myEntity  = { id: 'baz', foo: 'bar' };

  try {
    const response = await addEntity(myEntity);
    const storedEntity = await getEntity(response.id);
    pass('should add the new entity');
  } catch(err) {
    fail('failed to add and read entity', { myEntity, error });
  }
});

We’re on the right track here, but we’re missing some information. Let’s try to answer the 5 questions using the data available in this test:

1. What is the unit under test? addEntity()
2. What should it do? 'should add the new entity'
3. What was the actual output? Oops. We don’t know. We didn’t supply this data to the testing framework.
4. What was the expected output? Again, we don’t know. We’re not testing a return value here. Instead, we’re assuming that if it doesn’t throw, everything worked as expected — but what if it didn’t? We should be testing the resulting value if the function returns a value or resolving promise.
5. How do you reproduce the failure? We can see this a little bit in the test setup, but we could be more explicit about this. For example, it would be nice to have a prose description of the input that you’re feeding in to give us a better understanding of the intent of the test case.

I’d score this 2.5 out of 5. Fail. This test is not doing its job. It is clearly not answering the 5 questions every unit test must answer.

The problem with most test frameworks is that they’re so busy making it easy for you to take shortcuts with their “convenient” assertions that they forget that the biggest value of a test is realized when the test fails.

At the failure stage, the convenience of writing the test matters a lot less than how easy it is to figure out what went wrong when we read the test.

In “5 Questions Every Unit Test Must Answer”, I wrote:

“equal() is my favorite assertion. If the only available assertion in every test suite was equal(), almost every test suite in the world would be better for it.”

In the years since I wrote that, I doubled down on that belief. While testing frameworks got busy adding even more “convenient” assertions, I wrote a thin wrapper around Tape that only exposed a deep equality assertion. In other words, I took the already minimal Tape library, and removed features to make the testing experience better.

I called the wrapper library “RITEway” after the RITE Way testing principles. Tests should be:

• Readable
• Isolated (for unit tests) or Integrated (for functional and integration tests, test should be isolated and components/modules should be integrated)
• Thorough, and
• Explicit

RITEway forces you to write Readable, Isolated, and Explicit tests, because that’s the only way you can use the API. It also makes it easier to be thorough by making test assertions so simple that you’ll want to write more of them.

Here’s the signature for RITEway’s assert():

assert({
  given: Any,
  should: String,
  actual: Any,
  expected: Any
}) => Void

The assertion must be in a describe() block which takes a label for the unit under test as the first parameter. A complete test looks like this:

describe('sum()', async assert => {
  assert({
    given: 'no arguments',
    should: 'return 0',
    actual: sum(),
    expected: 0
  });
});

Which produces the following:

TAP version 13
# sum()
ok 1 Given no arguments: should return 0

Let’s take another look at our 2.5 star test from above and see if we can improve our score:

describe('addEntity()', async assert => {
  const myEntity  = { id: 'baz', foo: 'bar' };
  const given =  'an entity';
  const should = 'read the same entity from the api';

  try {
    const response = await addEntity(myEntity);
    const storedEntity = await getEntity(response.id);

    assert({
      given,
      should,
      actual: storedEntity,
      expected: myEntity
    });
  } catch(error) {
    assert({
      given,
      should,
      actual: error,
      expected: myEntity
    });
  }
});

1. What is the unit under test? addEntity()
2. What should it do? 'given an entity: should read the same entity from the api'
3. What was the actual output? { id: 'baz', foo: 'bar' }
4. What was the expected output? { id: 'baz', foo: 'bar' }
5. How do you reproduce the failure? Now the instructions to reproduce the test are more explicitly spelled out in the message: The given and should descriptions are supplied.

Nice! Now we’re passing the testing test.

Is a Deep Equality Assertion Really Enough?

I have been using RITEway on an almost-daily basis across several large production projects for almost a year and a half. It has evolved a little. We’ve made the interface even simpler than it originally was, but I’ve never wanted another assertion in all that time, and our test suites are the simplest, most readable test suites I have ever seen in my entire career.

I think it’s time to share this innovation with the rest of the world. If you want to get started with RITEway:

npm install --save-dev riteway

It’s going to change the way you think about testing software.

In short:

Simple tests are better tests.

P.S. I’ve been using the term “unit tests” throughout this article, but that’s just because it’s easier to type than “automated software tests” or “unit tests and functional tests and integration tests”, but everything I’ve said about unit tests in this article applies to every automated software test I can think of. I like these tests much better than Cucumber/Gherkin for functional tests, too.

Next Steps

TDD Day is an online recorded webinar deep dive on test driven development, different kinds of tests and the roles they play, how to write more testable software, and how TDD made me a better developer, and how it can do the same for you. It’s a great master class to help you or your team reach the next level of TDD practice, featuring 5 hours of video content and interactive quizzes to test your memory.

More video lessons on test driven development are available for members of EricElliottJS.com. If you’re not a member, sign up today.

Eric Elliott is the author of the books, “Composing Software” and “Programming JavaScript Applications”. As co-founder of EricElliottJS.com and DevAnywhere.io, he teaches developers essential software development skills. He builds and advises development teams for crypto projects, and has contributed to software experiences for Adobe Systems, Zumba Fitness, The Wall Street Journal, ESPN, BBC, and top recording artists including Usher, Frank Ocean, Metallica, and many more.

He enjoys a remote lifestyle with the most beautiful woman in the world.