Using Cucumber JVM With Gradle and JUnit

Some time ago we were tasked with implementing an end to end UI automation testing framework to be used for an anonymous company’s native iOS and Android apps. We did a PoC of tools in the market and after some time we settled upon using Appium.

The constraints which were put upon us was that we needed to:

• Use a language which was mobile developer-friendly
• Ensure that the framework could execute tests for both iOS and Android.
• Use BDD to make this appealing to the Agile Delivery Managers (ADMs) for their story tracking, and manual QA’s who wanted to learn UI automation.

With the above in mind we settled upon using Cucumber JVM with the Appium Java client, and Gradle as the build management system. We chose Gradle because this is the default in the Android world hence we were appealing to one half of our mobile devs. Little did we know about the challenges which we would face as a result of that Gradle choice.

When looking at the Cucumber JVM documentation we found a wealth of information about how we can use JUnit runners to execute our tests, however; it was focused on Maven.

What about Gradle? The only official documentation which we found revolved around how we can use the Command Line Interface (CLI) via a Gradle task to execute the Cucumber JVM directly. (https://cucumber.io/docs/tools/java/#gradle) This is very limited in scope and didn’t scale for several things which we had our eye on.

Given that the majority of our team had predominantly come from a Maven background this was quite a challenge. It lasted a good few weeks and consisted of a lot of hair pulling, browsing Stack Overflow and drinking a lot of coffee or in my case tea (I don’t like the taste of coffee).

We started by first understanding how Gradle tasks work and the different types of tasks which you can have. We then combined this with the documentation which Cucumber has for Maven and JUnit and this is what we came up with. It may not be the most optimal solution, hence we welcome any feedback as to how this can be improved.

1. We created a Cucumber Runner Class

We were using the fairly common src/main and src/test Java project layout so this class lived in its own package within src/test. It was aptly called MobileTestRunner and looked like this:

package testRunners;

import io.cucumber.junit.Cucumber;
import io.cucumber.junit.CucumberOptions;
import org.junit.runner.RunWith;

@RunWith(Cucumber.class)
@CucumberOptions(
        features = "src/test/resources/features/",
        glue = "stepDefinitions",
        strict = true,
        plugin = {"pretty",
                "html:target/cucumberReports/mobile/report.html",
                "json:target/cucumberReports/mobile/report.json"
        })
public class MobileTestRunner {
}

We were able to run this directly in the IDE and this would execute all features which existed in our src/test/resources/features folder and provide the relevant Cucumber reports. This was nice but what about executing only a small chunk of features, or executing this from the CLI? (the end goal was to have Continuous Integration (CI) executions after all)

2. We created a boilerplate Test task in our Gradle build script

Out of the box Gradle will look to run all Test Classes when this is invoked. If we therefore had multiple runners in the future we would have some weird and wonderful things going on. To overcome this we used the wonder that is Regex to target the single class in question, which is our MobileTestRunner

task regressionTests(type: Test) {
    include '**/MobileTestRunner*'
}

3. We altered the behaviour of the Gradle test tasks in our build script to get the verbose information of our scenario executions

Those who use Cucumber know the value of the verbose output which you receive when executing scenarios in the feature files. Gradlesuppresses this information for test tasks out of the box, so a bit of Googling magic later we stumbled across this:

tasks.withType(Test) {
    dependsOn cleanTest
    testLogging {
        showStandardStreams = true
    }
    systemProperties System.properties
}

We added the systemProperties line so that we could use Gradle’s systemProperty argument to override the runner’s default Cucumber options at the task execution level.

4. We added values to change the Cucumber options in the task which was created in Step 2

Given the mobile test runner needs to be targeted for each task, we placed the regex in a variable at the top of our build script. This way we could alter it with a one line change in the future if we needed to.

We also wanted the ability to target certain feature file sub folders as opposed to the whole directory when executing scenarios. We therefore created a variable to the parent directory which we could append to at the task level.

def testRunnerRegex = '**/MobileTestRunner*'
def mobileFeatureFilesPath = 'src/test/resources/features/'

With the above variables in place it was now just a case of altering the task which was created in Step 2, and adding in any more which we needed. We used Gradle’s systemProperty argument so that we could override the Cucumber options in the MobileTestRunner in order to execute what we wanted.

For example:

task androidLoginTests(type: Test) {
    include testRunnerRegex
    systemProperty("cucumber.features", mobileFeatureFilesPath + "login")
    systemProperty("cucumber.filter.tags", "@androidLogin")
}

Would run our Android login tests which lived across the feature files in the login subdirectory (src/test/resources/features/login)

Whereas this:

task androidSmokeTests(type: Test) {
    include testRunnerRegex
    systemProperty("cucumber.filter.tags", "@androidsmoke")
}

Would run our Android smoke tests which lived across the feature files in all folders at the location src/test/resources/features

5. We grouped the tasks and added in a description to make CLI executions a bit more readable

We found that when executing things through the CLI as is often the case when you’re testing out something to be added in CI, it was quite difficult to know what each task was doing, especially if it wasn’t correctly named. To that end we used Gradle’s very nice groups and description offering to make it a little easier on the eye when invoking gradle tasks

The group description was stored as a variable at the top of the file, to make it easier to update in the future if need be.

def appiumTesting="Appium UI Testing"

task androidLoginTests(type: Test, group: appiumTesting) {
    description = "Execute login tests for Android"
    include testRunnerRegex
    systemProperty("cucumber.features", mobileFeatureFilesPath + "login")
    systemProperty("cucumber.filter.tags", "@androidLogin")
}

With all of the above in place we were now in a position to ensure that our Appium server was running with a connected device/emulator. From here we could then run whichever Gradle task we liked in order to run the cluster of tests which we needed to.

I hope that you have found this article useful, and that it saves you a lot of research, trial and error if you are in a similar situation :)

My thanks to my peer Laxmikant Somni who was there with me pulling that hair and drinking that coffee to conquer this.