Batch Size Reduction and Software Testing - Duncan Nisbet

I’ve been studying Don Reinertsen‘s work on product development flow for some time. Whilst I talk about the ideas from a testing perspective, I’ve never actually attempted to get my thoughts on to paper.

I’m attending Don’s workshop on flow in Cambridge at the end of September, so I’m using this post to record my current thoughts so I can use them for reflection after the course.

These notes still need a lot of refining…

3-mini-burgers3

TL;DR

Testing provides information on the state of the product for stakeholders to make decisions. Ideally, this feedback needs to be delivered in a timely manner so that stakeholders can make those decisions at the appropriate time.

Staged delivery cycles, such as Waterfall, have long / delayed feedback loops which results in decisions often being made after they are most effective.

Agile development practices aim to shorten feedback cycles between each stage of the development cycle.

How have they managed that? Predominantly through batch size reduction - where “batch size” relates to the size of a work product or process. For example:

Requirements have been taken out of sprawling documents & written on the back of 3×5 index cards
Development teams discuss the requirements on the index cards & agree what they can deliver in set time period
Frequent demo’s of the software being developed - even when it is incomplete
Programmers write tests for their code before writing the code
Deploying code more frequently than twice a year

So how is software testing impacted by batch size reduction?

Batch size reduction & software testing

Software testing is often seen as the bottleneck in the development process, with Testers feeling the brunt of the glares & furrowed brows of those who want to get the software delivered.

Guess what, those Testers (by & large) also want to see the sofware delivered as well but the implemented process in traditional software development means that they don’t get to see the software until late in the process.

In this context, it’s the process that is hindering the progress of development, not the testing specifically & certainly not the Testers themselves.

The Agile principles (page 2 of the Manifesto - go check them out) are focused around establishing a smooth flow & frequent opportunities for feedback. The frameworks, like Scrum, strive to help teams implement the underlying agile principles.

It is reduced batch sizes that facilitate this smooth flow & fast feedback offered by the Agile principles.

Don calls out 10 benefits of reduced batch sizes - here I attempt to map those benefits to software testing:

Don’s 10 benefits of batch size reduction

B1 Batch Size Queuing Principle: Reducing batch size reduces cycle time

Testing fewer/smaller changes in the code base means that the testing activities can be performed faster many/large changes.

Not only because of testing the change, but also the associated tasks, such as bug triage, generating reports & building release notes

B2 The Batch Size Variability Principle: Reducing batch sizes reduces variability in flow

How many times as a Tester have you sat waiting for a build or release candidate to make it into a test environment?

Bigger, more complex release candidates are hard to test & debug so tend to take longer getting through the development pipeline and as such the time taken to progress large release candidates is more of an unknown than smaller releases

B3 The Batch Size Feedback Principle: Reducing batch sizes accelerates feedback

This is a biggie for me as a Tester - I want to be able to provide the appropriate feedback at the time it will be most effective.

With large / complex release candidates, it’s going to take me longer to test & as such longer to provide feedback.

This may be OK in some contexts, but in others, delayed feedback loops can be detrimental (see Cost of Delay, also discussed in a future post)

B4 The Batch Size Risk Principle: Reducing batch size reduces risk

It is far easier for us mortal humans to understand the risks associated with 1 line of code change than that of a large / complex release candidate.

If testing is seen as risk mitigation, I would far rather test several little changes through the pipe (of which I can understand the majority of the risks) than I would test all those little changes in one release candidate (the interactions between which I might not fully understand)

Also, if we change 1 line of code & we discover a bug in the Production environment after it is deployed, it is far easier for us to debug the problem than if we release many lines of code across different modules.

B5 The Batch Size Overhead Principle: Reducing batch size reduces overhead

This is a real brainbender, as you should imagine with smaller releases comes more deployments.

Fortunately, deployment processes tend to be repetitive & therefore far easier to reduce the “cost” of (see transaction costs below)

Don uses the example of bugs in this principle; largely releases are likely to take longer & have more open bugs that need to be triaged for not only priority & severity but also duplication.

Search through 300 bugs to see bug 301 has been raised before will take longer than search through 30 bugs to see if bug 31 has been raised before.

Also, if a release takes weeks to be tested & we have provide a bugs status report for each week for the release, then we are creating more test reports.

B6 The Batch Size Efficiency Principle: Large batches reduce efficiency

Large batches are great for improving the efficiency of the individual Programmer or Tester as they get long periods of uninterrupted time to do their work.

However, from Systems Thinking we understand that this local efficiency reduces overall efficiency of the development process, largely due to the lack of connectivity & feedback between the sub-systems.

B7 The Psychology Principle of Batch Size: Large batches inherently lower motivation & urgency

People like getting things done. This closure gives them a release of Dopamine which is a very addictive chemical. For example, when you win a hand of poker, or your horse wins the race. (Simon Sinek “Leaders Eat Last“)

Its the same for software development - we get a lot more Dopamine when complete tasks more frequently. We then crave more of that chemical which motivates us to get more work completed.

Completing work is easier when the size of the work to be completed is small.

Also, in Agile development teams, it is very hard to hide for any prolonged period of time. You take on a task & you are expected provide feedback on that task, usually within a couple of days.

B8 The Batch Size Slippage Principle: Large batches cause exponential cost & schedule growth

That large release comes into test & it’s buggy. It takes ages to debug, fix & retest.

The other Programmers are not twiddling their thumbs. Nope, they’re writing more code on top of the buggy code which will later need to be unpicked when the fixes for the related bugs are causing merge conflicts.

These bugs prevent the release being deployed, so the Programmers are unable to commit their (seemingly) unrelated changes for a different feature until the release has gone.

So they keep their unreviewed changes on a branch & move onto the next change…

These branches need to merged into the Master branch at some point, likely causing significant merging / integration pain.

And so the batches get larger & larger…

B9 The Batch Size Death Spiral Principle: Large batches lead to even larger batches

And then the loss aversion kicks in - the large batch becomes the project that must not fail & all manner of money gets thrown at it.

More people get added to the project in order to ensure that the software gets delivered on time (which thanks to Brooks & his book The Mythical Man Month we know is a myth) which means the batch just gets larger & any problems already in the codebase & process get exacerbated, leading to larger batches still.

B10 The Least Common Denominator Principle of Batch Size: The entire batch is limited by it’s worse element

You have a release that is comprised of 20 stories. You find a bug in one of those stories, but the other 19 are ok.

You cannot release the 19 stories until the bug in the 20th is fixed.

This means you have 19 stories sitting there, waiting to be deployed, not delivering the company any value yet the company has invested a lot to get those stories to this point (inventory).

Wouldn’t it better to release the stories in smaller batches so that you can start getting the value out of them?

Benefits of Small Batch Testing

Don even specifically calls out the benefits of reduced batch sizes for software testing (Chapter 5, p.120) & provides this useful diagram:

(figure 5-3)

What’s stopping me from reducing all the things?

The enemy of reducing batch size is transaction cost.

Sometimes, it “costs” too much to shift a small piece of work through the development cycle, for example:

Key stakeholders are not available for demo’s of the software adhoc - they only visit the office 2 days a week
Regression testing takes a team of 4 people 2 working days - too much time for 1 change to be delivered
Release notes take days to be signed off & processed through the ticketing system
Software can only be released out of hours

So how do we reduce the batch size of testing?

We don’t.

We break it down into smaller components & spread it across the whole development cycle, striving for continuous testing alongside the corresponding requirements gathering or programming practices.

This sounds pretty similar to the V-Model doesn’t it? Well there’s a major difference - each stage of the model is contained within the feature being developed which should be over in days as opposed to weeks or months.

Here is an example of a diagram I used with a client as a reference for sharing my test approach:

See how the focus of testing (for the entire team) shifts from a unit level through to a system integration level as the software progresses through the development pipeline - we are continuously testing throughout development.

Continuous testing isn’t a new idea, just an existing phrase rebadged. For example, Toyota are famous for continuously checking the quality of their vehicles as they progress along the production line. Also, Gilb has being using the phrase “Continuous Inspection” since 1988.

Ideas for how you can test continuously

Chris Matts calls when information is discovered (in the development lifecycle) “information arrival“. He has a great diagram in his European Testing Conference keynote “We don’t need Testers! What we really need is Testers!” (from 12 minutes).

I’ll be discussing the diagram in more detail in later post, but the general idea is that testing can & should discover information throughout the development lifecycle, not just in a phase at the end.

The following are a set of ideas for continuously delivering your feedback of the state of the software:

Get involved with requirements gathering - understand the problem, challenge the propopsed solution
Discuss testability & generate a test approach with the whole team before any code is written so everyone knows how the software will be tested & who is doing what level of testing.
Get access to the production & test code the Programmers are writing so you can start testing on your machine & provide feedback before the code is checked in to the repository.
Pair with or take a handover from Programmers to understand the pain points in the software they have written - this will help generate a raft of test ideas & focus. Programmers will start to understand how you approach test execution.
Let decision makers (e.g. Product Owners) play with the software on your machine so they get to say the fabled words “that’s not want I want” - the next words out of their mouths will be what they do want. (J.B Rainsberger - from 20 mins).
Validate the software in the Production environment to see that it actually solves the problem

Wrap up

So these are my thoughts on how batch size reduction & software testing play together - I’m looking forward to reviewing these notes after I’ve attended Don’s workshop.

There are other posts brewing around the economics of batch size reduction & queuing theory, but I’ll save those for another day…

Duncs