How Much Automated Testing Should Be Done in Software Development?

Introduction

In the world of software development, testing is often viewed as a time-consuming and resource-intensive process that slows down the development cycle. Some developers and probably most product owners hate it. However, the truth is that testing is a crucial part of programming, as it ensures that the software works as intended. In this article, we’ll explore why developers, product owners, and decision-makers need to embrace it and how they find the right amount of automated tests for their needs.

The article starts with the disadvantages and advantages of automated testing and then focuses on the right amount of testing. In the end, you will have a clearer understanding of which criteria is important to consider to decide the amount of testing you should do in your private or professional project.

Disadvantages of automated testing

1. It takes time (time = $)

One of the main disadvantages of automated testing is that it takes more time upfront to set up and maintain automated tests. Automated tests require planning, design, implementation, and maintenance, which can (in the short term) take longer than simply running manual tests. Also, automated tests may require specialized skills or knowledge that may be limited within the development team.

2. Infrastructure costs

If you run your automated tests only on your own computer you normally do not pay anything for it (apart from your computer). To include automated tests in your deployment pipeline, which is quite reasonable, you normally have to pay if you exceed a certain limit. These infrastructure costs may be a barrier to entry for some organizations, especially those with limited resources.

Advantages of automated testing

1. Fewer bugs

One of the biggest advantages of automated testing is that it can help catch bugs earlier in the development cycle. Automated testing can quickly and easily run hundreds or thousands of tests, which can detect bugs and errors that might go unnoticed with manual testing. This is often the case if a new feature changes another part of the app accidentally which has not been checked again during manual testing. By catching and fixing these bugs early on, developers can avoid costly and time-consuming bug-fixing later in the development cycle.

Important to mention here is that this pays only off if further new features and changes are planned for the software. If you already programmed a perfect app that will never be changed again, you do not need to add any tests afterward. You only avoid bugs if you change parts of your software which normally is the case.

2. Better code

Automated testing encourages developers to write better code. Since automated tests require code to be testable, developers are encouraged to write more modular, decoupled, and maintainable code. This makes it easier to update the codebase and lessens the likelihood of introducing bugs or errors when making changes. Code quality does not pay off directly but in the mid-term and long-term in the form of development speed (and beauty).

Dave Thomas tells in the famous book “The Pragmatic Programmer” (Amazon affiliate link, 2020) that he no longer writes tests. He has been coding for over 45 years and was writing automated tests for 30 of them but when he stopped writing tests it did not significantly downgrade his code. In his opinion, the main benefit of testing is thinking about the test and its impact on the code. Therefore, less experienced developers should write automated tests but experienced developers can try to omit writing tests. Nowadays, he only writes tests as a way to communicate with other developers.

3. Communication with other developers

Automated testing can help facilitate communication between developers, as it provides a shared understanding of the codebase. They can help to understand how different parts of the code work together and how changes in one part of the codebase can affect other parts. This can lead to more collaborative and effective development processes.

4. Less manual testing

Automated testing can reduce the amount of manual testing needed, which can save time and resources. Since automated tests can be run quickly and easily, developers can run them as often as needed, without the need for manual intervention.

5. Faster release cycles

Because of the reasons mentioned automated testing can help speed up the release cycle of software by catching bugs and errors earlier in the development cycle. This allows developers to fix issues quickly and efficiently, without the need for time-consuming and costly bug-fixing later on. Faster release cycles also mean that developers can release new features and updates more frequently, which can lead to higher user satisfaction and increased revenue.

How much-automated testing should you do?

There is no rigid perfect solution for the amount of automated tests. The “perfect” quantity always depends on your individual case. To speak in an economical way you should do testing if its benefits are greater than its costs. It is easy to quantify the costs of automated testing but it is hard to estimate the benefits of it. The following figure illustrates which factors are important to consider to find the right amount of testing for you. The benefit of testing in € is on the y-axis against the ratio between the time of writing tests and writing features on the x-axis. If you would do one hour of testing for every two hours of feature programming you would get a ratio of 0.5 (1h/2h).

I assume that the benefits of testing have asymptotic utility. This means that the benefit e.g. of increased code quality is less valuable the more I have of it (decreasing marginal utility) but always positive. In contrast, developer costs are linear because they do not change with an increasing amount. The figure is just an example and looks different for every individual case.

So what does this mean for your project? Important factors for decision-making are the time horizon (automated testing can be seen as an investment), the cost of having an error in your software, the desired code quality, and the current knowledge of the developers.

Following I will describe three different scenarios where three extremely different testing strategies make sense.

You should think about not using automated tests at all if...

you have a small project (<3 months) and the code will not be used or changed again.
you do not continue working on the project and therefore the future quality (=speed) is not important.
the costs of mistakes in your code are close to zero. You program e.g. a private app that no one uses.
you do not like testing and do not want to improve your testing skills.
nobody working on the project knows anything about automated testing and you do not have the budget to acquire the knowledge.

2. You should think about 100% code coverage (and even more) if...

the costs of making a mistake are extremely high. This could be the case if you work e.g. for a bank.

3. You should do something in between if...

the costs of making a mistake are greater than zero but not that high that you would read about it in the newspapers.
you want to ensure better code quality (= speed).
the project should exist for more than a month.
you want to add new features in the future.

Conclusion

The amount of automated tests that brings you the highest benefit always depends always on the current circumstances of your project. The article named criteria with which you can make a decision about the right testing strategy such as the time horizon, the costs of bugs, the desired future code quality (= future speed), and the developers’ knowledge of automated tests. Every team should address these points to find a testing strategy with which you catch the most important bugs and at the same time even speed up your development because of improved code quality.

Thanks for reading the article! Do you have questions or see another important (dis-)advantage of automated testing? Please let me know in the comments. Any further feedback is highly appreciated!

If you liked the article clap (50x), highlight, comment, and share it. Not only but especially technical articles got disadvantaged by the new Medium Partner Program incentives. If you want to support your favorite (technical) writers on Medium, remember to interact with the articles. You find more information about this here: The New Medium Partner Program is Bad for Quality Writing!

Also, check out my other articles:

Further information

Martin Fowler about the TestPyramid