Removing and simplifying tests is part of development too

The evolution of apps QA at Azimo

This is the fourth in a series of blog posts in which we outline our multiple years’ experience with our Android app testing at Azimo. Most of the principles, goals, and achievements also apply to our iOS app.

Table of content

• The evolution of apps QA, first days, and unit testing
• QA engineers, functional and UI testing
• Custom tooling for testing stack management
• (This post) — Removing and subtracting tests is part of development too
• QA testing in the cloud

Azimo product development is driven by mission teams - independent units with their goals, dedicated Product Managers, and engineering teams. In 2019 it was true for backend engineers. Apps' development was still managed in a centralized way.

With a 1-week release train and almost fully automated QA (but also better code modularisation and improved development processes - which is material for separate blog posts), we were ready to take the next step. We decided to split apps teams and do development from many mission teams in parallel. We thought that decentralization would allow us to focus on multiple projects simultaneously and move responsibility for apps from a single to many product managers. Today we know, the final results of this decision are great (especially in scalability), but we needed to face some new challenges back then.

Reducing the number of tests

Years ago, we took the approach to write tests where it's the easiest to write them. It worked well when Azimo was a monolithic system, and testing our backend solutions was quite challenging. Many of our UI tests covered platform functionalities because apps were the most accessible places to build test automation. But in 2019, the hard-to-test backend wasn't the case for us anymore. At the same time, having >400 functional/UI tests in our apps, where many of them took more than 1 minute to run, became a big room for improvements. We decided to change our approach to testing and make sure that we write tests where it's the right place to write them. We got many tests taking minutes on the apps side, while for backend QA it was no more than a couple of seconds to run.

Speeding up functional tests

Many of our functional and UI tests require a test account with the prefilled data. In the past, when we didn't have a convenient way for building internal tools, data was filled by the test "manually". Some tests clicked through the app to log in, e.g., create a transaction, and in the end, do all assertions. Some tests spent 90% of their time just preconfiguring testing criteria.

With a new backend infrastructure, it became possible to build QA utils microservice, which does the initial configuration before the test runs.

Improving tests flakiness

In the worst moment, our average failure rate was terrible — about 20%. When tests are taking hours, even with 32GB-RAM Macbook, there is always something that can hit a test suit unexpectedly. Chrome eating half of your memory, WiFi going down for a second, ADB glitches.

Significant reduction of tests and testing time improved our flakiness by some percentage points. With some extra effort, by the end of 2019, our flakiness was less than 10%.

Side note — we’ve also written a separate blog post about what flakiness is: What is flakiness and how we deal with it.

Software development cycle has two high-level phases closely correlated to the complexity and technical debt. Our testing tech stack is no different here.

In the first phase, you do rapid development and feel that every day of work brings new value. But over time, things get more complex. Growth slows down. It starts the second phase, where you spend more and more time dealing with technical debt. In this phase, it’s crucial to notice not only development opportunities. Very often, more value can be generated by removing or simplifying things. And it was the key lesson in our case.

Side note: There is an interesting Nature’s article on how hard it is for humans to notice opportunities in removing features.

By removing tens of tests and simplifying hundreds of steps in testing scenarios, we accomplished another critical step in our software development process:

• We reduced the testing time from 3 hours to less than 2 hours.
• We reduced test flakiness from 20% to less than 10%.
• We removed more than a hundred tests and simplified the remaining ones significantly. Many of the tests got a better place — from UI tests taking minutes to execute to API / backend tests doing the same in seconds.
• With fewer tests, the quality of our app didn’t go down. We kept a crash-free ratio above 99.5% and had no major (non-crashing) issues.
• We were able to start distributed software development by moving our engineers from mobile development silo to independent, fully staffed mission teams.

But this wasn’t over for us. Our appetite for faster and more effective testing grew even bigger. In the next blog post, we will cover our most recent steps in improving our testing tech stack. Stay tuned!

Towards financial services available to all

We’re working throughout the company to create faster, cheaper, and more available financial services all over the world, and here are some of the techniques that we’re utilizing. There’s still a long way ahead of us, and if you’d like to be part of that journey, check out our careers page.