Testing success: BDD is not (just) test automation

I’ve seen several projects using Gherkin to define their automated tests, without a previous collaborative effort to define the examples/scenarios between the business stakeholders, development team and testers.

We’ll see in this article that the test automation is a final part of the BDD process, a good side-effect, but not the goal or way of working in BDD.

What is BDD?

BDD is a framework to improve collaboration between business teams and technical teams​.

💬 BDD is a methodology for developing software through continuous example-based communication between developers, QAs and BAs

The BDD process moves through three phases: discovery, formulation, and automation.

Discovery phase

🔍 In the discovery phase The Product Owner or Product Manager creates acceptance criteria (business rules) in a free business format, for writing a story or feature.

Acceptance criteria are a set of rules or conditions that the system must satisfy to be accepted by a user, customer or other stakeholders

e.g: “The user should receive feedback indicating the strength of the password entered”, “Only strong passwords should be accepted”.

💬 The discovery process is collaborative; team members also discover and contribute additional criteria and talk about concrete examples of the new functionality to explore, discover and agree on the details of what’s expected to be done.

It doesn’t matter if some of the rules lack examples. As long as the acceptance criteria are clear enough to easily derive the relevant scenarios, you can keep them as acceptance criteria without examples until the next phase.

Maybe you’re naming this phase as Requirement/​Discovery workshop​, Product backlog refinement​ or Backlog grooming, they all are the same.

There are various discovery techniques, and one that I particularly favor is Example Mapping.

It’s important to note that each team member is offering a unique perspective:

1. Customer-centric stakeholders understand customer and business needs and the relative desirability and viability of a new requirement.
2. Development-centric stakeholders understand the solution space and technological feasibility.
3. Test-centric stakeholders consider the exceptions, edge cases, and boundary conditions for the new behavior.

🔔 In neither this phase nor the next is it necessary to delve into implementation details. However, you can include them as questions or comments (if you’re using JIRA, avoid putting this information in the description and instead place it in the comments).

Various parts of the system need to work together to fulfill the software requirements or user stories. The technical details of how each component functions are outlined in a Software Architecture Document, which falls outside the scope of the BDD process.

Formulation phase

At the formulation phase we document the examples found at discovery phase in a way that can be automated, creating scenarios using Gherkin language.

Given [context or pre-conditions]
When [event or action]
Then [outcome or assertion]

Gherkin has become the prevailing method for specifying scenarios during the formulation phase. However, if you have an interest in alternative approaches, it was once commonplace to use tools like Fitnesse

Gherkin styles

There are two styles of writing in Gherkin:

a) Declarative. Is specifying WHAT needs to be accomplished → Mostly understandable when read by non-developers. b) Imperative. Is essentially to spell out in as much detail as necessary HOW to accomplish something.

Gherkin purpose is to define what needs to be accomplished, and not explaining technical details. Despite of this fact I’ve seen a lot of people using the imperative style. If you see imperative style it’s a clue that has been defined from testing, and not as a collaborative effort.

“One of the most important rules when writing scenarios is to prefer declarative business rules over imperative test-script style sequences of steps” (John Ferguson)

I would try to give some tips on how to use correctly Gherkin in a forthcoming article.

Automation (and implementation) phase

Note by the author: I’ve included “and implementation” to underscore that we are developing the functionality at this stage. In BDD, this phase is referred to as “automation” exclusively.

🧑🏻‍🔧 Finally, we implement the behaviour described by each documented example (scenario), starting with an automated test — using a BDD tool — to guide the development of the code. This is related with first-test approach or TDD. If your team is not well or yet prepared for that, you can develop and test at the same time.

BDD tools turn the scenarios defined at formulation phase into executable tests before the related behavior has been implemented in the application itself.

The majority of BDD test tools are either derived or variations of:

• Specflow, used for .NET development
• Cucumber, mainly used for Java and NodeJS
• Behave for other technologies

All these tools are for test automation. If you’re going to test some of the scenarios manually for some reason, you’re not going to need it.

Can BDD (Gherkin) used for test automation only?

If we don’t do the conversations at discovery and formulation phases, we can produce as many scenarios as we want, automate and run them continuously, but we are not getting much value and … we are not doing BDD.

🔔 In some cases BDD scenarios are used to replace automation tests, but they’ve been not written for developing some behaviour.

I’ve heard from some teams that BDD testing tool, work seamlessly for their testing needs, without adhering to the three phases we discussed earlier. I believe that this decision is often influenced by trends rather than a thorough comparison of BDD tool usage and other tools like xUnit, for instance.

The BDD tools come with certain pitfalls and challenges that you should know.

Test development time increases

There is an extra layer (the Gherkin feature file) to connect the sentence to the code, and one or more runner files. Creating and maintaining the feature files and scenarios requires an overhead investment of time and effort.

You should know that each step (Given/When/Then) needs to be implemented as a separated method. Every method has to pick up data or leave data in a “pre-arranged” location. This, in Computer Science, is called a side effect (bad practice).

The reuse of step definitions within other step definitions, and the composition of higher level steps become too difficult to maintain and get out of sync. You can’t extend a step definition class from a parent class.

Did you know that Cucumber doesn’t verify if your step definition truly corresponds to Given/When/Then? It focuses solely on the text within the step. Consequently, you can use Given in your feature file, while implementing it with When in the step definition.

Gherkin syntax is prone to be misused and complex

Sometimes we’re saying the tests are more readable and useful with Gherkin, but mostly they are not valuable (with the business rule is enough). I’ve seen many examples like this one:

Given I have a previous instrument created
When I update the data
Then the data is updated

What is the value of this? Also conflicts arise about the style of Gherkin. Some testers use imperative, some others use declarative. What is the correct way if we’re using BDD tools only for automation, not for BDD?

We’re using badly Given and When. People sometimes don’t really understand the differences between them. We need to check them in the pull requests.

Parameterisation and data tables are not well prepared for data-intensive apps

BDD tools have parameterisation and data tables built, but it adds extra work and it’s difficult for data-intensive apps. Some of the pitfalls are:

• If we use “big” data-tables (more than 7–8 columns) BDD Automation tools are cumbersome. Writing and formatting such extensive information in plain text can be time-consuming due to excessive indentation, screen space limitations, and the need to scroll!
• We define all the setup conditions in the Given scenario statements, but this can be complex (a lot of steps as And… And ….).
• It has not direct support for loading data files (CSV, …) (like JUnit 5 does, for example)

BDD Automation tools struggle with reuse

Problems start once you need to reuse step definitions across many features and you want to keep code DRY.

The same step can be reused in different features, but needs to be defined only in one of them. If we remove step def implementation of this feature, we can affect the other features.

It’s also easy to have the same step but defined with a different text in different features.

BDD Automation tools struggle with parallelization

Cucumber/SpecFlow do parallelization in a sub-optimal manner. They parallelize at the feature file level. If we want to run 50 tests in parallel we need 50 feature files.

BDD tools integrations with testing frameworks

The BDD tool needs to be seamlessly linked, from step definitions into a specific testing framework’s implementation, enabling interaction with the Software Under Test (SUT). For example, this involves calling the Cypress or Playwright API from a test step definition to interact with a web application.

However, integrating BDD with certain tools can pose challenges. We’ve encountered difficulties numerous times while searching for and implementing solutions to use BDD in tools like Cypress. We use Cypress for End-to-End (E2E) UI testing, focusing on testing user journeys and workflows. But the question arises: should we define a user journey with Gherkin? Personally, I don’t believe it’s the ideal approach.

Benefits of BDD tools

If you’re following the correct path, starting with BDD from the discovery and formulation phases before converting examples into automated scenarios, you can get several advantages. While it’s true that tests become more complex and development efforts increase, the benefits you gain in return include:

1. BDD increases and improves collaboration. It enables everyone involved in the project to easily engage in the product development cycle. And by using plain language, all are able to write behavior scenarios.
2. Living documentation. There’s a connection between documentation and tests. As a result, when tests fail because of an incorrect specification, we can update executable specifications in the documentation. In this way, the documentation always is up to date.
3. Traceability. Traceability between tests and requirements is built in from the start. This provides simple coverage reporting at the requirements level.

The last two advantages are not limited to BDD tools; we can achieve the same results using tools like Allure.

@Feature("FEATURE-KEY")
class FilebasedShipmentPasswordProviderTest {  
    @Test
    @Story("...")
    @Issue("...")
    @TmsLink("...")
    @DisplayName("Verify that when the secret file is unavailable an exception is thrown")   
    void should_ThrowException_When_SecretFileIsNotAvailable(){
        setup();
        codeTriggers();
        assertThat(...);
    }
}

Don’t use Gherkin language for technical specifications

It’s always important to make a scenario readable by a stakeholder. However, for an endpoint (REST API, Async API, …) , the stakeholder is going to be another programmer writing a client for the interface.

In such cases, the question arises: why should we employ BDD or Gherkin syntax, which isn’t a language tailored for technical professionals?

Technicians typically communicate through different means, such as diagrams, code, and interface contracts like OpenAPI or AsyncAPI, rather than using the Gherkin language.

Conclusion

I’ve frequently observed some projects using BDD primarily as a testing tool rather than as a collaborative one. I want to clarify that I don’t consider BDD to be inherently flawed; rather, it should be used wisely, when the phases of formulation, discovery, and automation are followed in this specific order.

👏 Enjoyed this article? Show your appreciation by giving it a round of 1 to 50 claps! Your applause serves as a powerful motivator for me to continue producing valuable content for your benefit. Thank you for your support!

In crafting this article, I have relied on various web references for accurate and up-to-date information. I extend my appreciation to the sources and authors whose contributions enrich this content:

• Why Cucumber and the Gherkin Language should not be used for Software Testing
• Cucumber is not a testing framework
• When shouldn’t you use Cucumber?
• Why Bother With Cucumber Testing?
• Is BDD Automation Actually Killing Your Project?
• Why would someone use Cucumber?
• A Case Against Cucumber
• Difficulties with BDD framework
• BDD is — BDD is not
• SAFE Behaviour-Driven Development
• Cucumber Behaviour-Driven Development
• Acceptance Criteria vs. Scenarios
• Scenario-Oriented vs. Rules-Oriented Acceptance Criteria
• BDD Pros and Cons
• Declarative vs Imperative Gherkin Scenarios for Cucumber
• Acceptance tests in practice behavior driven development