Cucumber is not a testing tool. I know this.
One of the big noises I make about Cucumber is that it is not a testing tool, it is a tool to help collaboration, discussion of requirements & generate living documentation.
How then did I fall into the trap of trying to use Cucumber as an automated testing tool?
In my organisation, we use Cucumber feature files to capture the acceptance criteria as scenarios with the intention of being able to automate the execution of those scenarios.
One of my big problems with this is that we have ALL the acceptance criteria written as scenarios to be executed in Cucumber.
The whole team knows that we do not need a Cucumber scenario for each acceptance criteria as it is possible for us to check the code is adhering to the business logic lower in the stack, say with Jasmine or Junit.
With this fact in mind I started looking at ways we could Cucumber scenarios to reflect the results of those checks performed lower down the stack.
The problem I perceived was that the Business stakeholders would only want to see progress of the automated execution in one place - in our case Cucumber. I didn’t want the PO to have to dig into Jasmine & Junit reports.
I started looking for hooks to represent Jasmine & Junit results in Cucumber reports, related to the acceptance criteria.
I wanted the Cucumber reports to show % complete so that PO would have an idea how we, the dev team, were progressing. It was going to be glorious.
It’s a Trap!
Would anyone care to count how many traps I fell into with this thinking? Lets work through those traps I’ve identified now…
Trap 1 – Cucumber can be used to demonstrate progress of development through the number of passing scenarios.
Trap 2 – Counting (scenarios) test cases. “We have 10 scenarios, 5 are passing, we are 50% of the way through this feature”
Trap 3 – Passing Cucumber scenarios are an indication that the software is working as expected.
Trap 4 – (Our) Business stakeholders are interested in automated test results.
Insight
It was whilst talking through my dilemma with Chris Parsons that I realised the error of my ways.
Cucumber was never meant to be a tool for reporting testing progress & I know this.
One anti-pattern we had adopted with the use of Cucumber was to use JIRA to understand & refine acceptance criteria in the 3 amigos session & then Testers would copy those acceptance criteria into a feature file before creating the scenarios.
The scenarios that had been written by the Testers were then “reviewed” by the Business if they had the time or inclination to read through a very long feature file.
I have put a fair amount of effort into the team opening a text editor in the 3 amigos & creating the happy path scenarios there & then. At least we’re now collaborating in a feature file & the Business & Programmers are involved with the scenario creation. This really helps with team buy-in & understanding as to what we’re aiming to achieve.
Throughout the development cycle, the Business are invited to mini demo’s where some of the acceptance criteria have been met. They understand its not the finished product, but at least they get the chance to say “that’s not what I want” which then enables us to determine what they want next.
It’s through this close collaboration that trust is built between the development team & the Business stakeholders.
The Business are not interested in our automated test results. They get so much more value from knowing that we all have a better idea about what the right thing is, seeing it being developed & having continuous feedback & input.
The automated test results are useful for the development team as change detectors; they help us identify where the software is not meeting the original spec & have a conversation with the Business about what to do about it.
New (Correct?) Way of Thinking
Used correctly, I believe Cucumber is great at doing what it was intended to do - be a single source of truth. From the business perspective, Cucumber helps to gather requirements/specifications during product discovery to create a set of living documentation. From a development perspective, we can continually check that the software is adhering to the specifications.
My craving for test results is of no interest to the Business, but it is invaluable to the development team. This information we have collected & presented elsewhere in the system, which the Business can access if they really want to.
Through close collaboration during the entire development lifecycle, trust is established between the development team & the Business which goes along way to ensuring we build & deliver the right thing
—UPDATE—
There has been some good discussion around Cucumber & how it has been (mis)used since it was first released.
“The Cucumber that [nearly] ate itself” (Skills Matter SkillsCast)
Kickstart Academy Podcast (includes discussion around the wider use of BDD)