This post outlines some examples of how I’m currently using our current incarnation of the test matrix
My development team practice Behaviour Driven Development (BDD) & to aid close collaboration we use Cucumber (this post assumes prior knowledge of the BDD cycle. Also the pros & cons of Cucumber won’t be discussed here).
The application is built in Java, HTML, JavaScript & PHP.
Consequently there is testing at all levels of the application we can be related to the matrix.
As a development team, including the Business stakeholders we start in the TL quad writing a suite of scenarios in the Given|When|Then (Gherkin) format which when executed using Cucumber will satisfy the (Acceptance) requirements.
These test scenarios are written before the production code & the Business are most definitely in them.
These scenarios will fail when executed as there is no production code for them to be executed against yet.
Testers & Programmers pair up on writing the step definitions (glue code) which translate the human readable scenarios into commands that the Ruby code can execute.
There will come a point when writing the step definitions some production code will need to be written.
At this point we (literally) drop into the TDD cycle which lives in the BL quad (TDD itself being born of code craftsmanship are writing failing unit & integration tests which
The unit & integration tests are written before the production code & Business generally has little interest in the activities going on at this stage. These tests are ultimately for the good of the application, but for now they’re here to help the Programmers.
Now the Programmers can write some production code in order to make the unit & integration tests pass (go green).
The tests demonstrate that the business logic for the code which has just been written has been fulfilled.
Remember, at this point we have only written enough production code to make the unit & integrations tests pass which will hopefully result in the Cucumber scenarios passing as well. We have now proven (to some degree) that the application can work as the Business asked for.
These automated tests & scenarios remain in place as change detectors. They can be executed whenever in order to ascertain if any business logic has changed after writing more production code.
But what about all the edge cases? What about the non-functional requirements?
Now we move into the right hand column where we can explore the newly written application.
In the TR quad, we start our exploratory testing. Artefacts include test charters, session reports & potentially more scenarios.
The information this exploratory testing provides should be of great interest to the Business & Programmers alike.
There will be times when our exploring takes us into the backend of the system (I’m thinking working with APIs or the DB here), consequently we’ll drop into the BR quad. We may find lots of interesting things in here but I doubt the Business would be interested in hearing about them.
It may be necessary demonstrate how a bug in the API (for example) presents itself on the front end, which would require moving back up to the TR quad.
There will also be those edge cases which can be executed at the unit & integration level. Again, the Business generally won’t be interested in these tests.
We’re demonstrating in the right column that the application will work under different circumstances.
Once we have demonstrated the application behaves in different states & flows, the tests, or a subset of them, shift back to the left column where their role becomes that of change detectors - are further changes we’re making to the production code having an undesired impact elsewhere in the existing code.
So that’s my example of how we might flow through each quadrant of the matrix throughout the development of a feature.
Next posts:
Post 6: Wrap Up
Previous posts:
Post 1: Intro
Post 2: Rectangles
Post 3: Columns