I’ve been experimenting over the last few years with how best to manage my exploratory testing sessions and test artefacts in an agile environment. And it’s been a long and often tedious process. I’ve used notepad, excel, session based test management tool and a variety of my own inventions along the way. And although all have offered something unique, easy or suitable none have offered the complete package.
This could be because exploratory testing is performed differently by different people or it could just be that it’s incredibly tricky to manage. I suspect it’s a bit of both. I think, however, that I have finally landed on a suitable method of working that allows me to preserve my exploratory testing sessions. At least for now..
As with most agile (in our case Scrum) projects the tool is becoming increasingly more important. Although some teams have dismissed the tools in favour of boards, we’ve found it simpler to use tools to manage work flow, defects and tests. Especially so with co-located teams. In my examples of how I’m managing tests that follow I’m obviously biased in my choice of tool. I’m using our home grown iMeta Agility tool. But the process I’m explaining here will work with any scrum project management software. But do feel free to try out iMeta Agility for free here : http://agility.imeta.co.uk/
Shameless plug over it’s time to explain how I manage my exploratory testing in an agile environment. But do consider that this is just how I manage this process. This may not work for you. It’s not a Best Practice. There’s no guarantee it will work for me on the next project. But at the moment I’m finding this process is best suited to how I work.
Firstly here is typical task board view. In this case with a very simple user story called “User Story 1”. It has a series of basic “tasks” added to the story in various states. Note there is a “Testing > Validate acceptance criteria”. This story includes a hyperlink to any other supporting documentation, including test cases and test data. For me though, the acceptance criteria on the story form the basis of my none exploratory tests. If the criteria is detailed enough I find it negates the need for detailed test cases.
On this taskboard are some tasks I added called “ET Session > Explore this…….”. These are exploratory tests.
As and when I think of exploratory tests I add them to the relevant story as tasks in the not started column. This therefore highlights to me I still have some tests to run, but it also highlights to the team that this story is still not complete. Any exploratory test I think of that does not belong directly to a story in the sprint is added to the story on the backlog (this feature is not yet in the iMeta Agility tool but is planned – for now I use excel)
Over time as I explore the product using the exploratory tests I have already defined it is inevitable that I think of more. These are simply continuously added to the story until I exhaust my ideas and run them all. At which point, the story is probably done. It might be that some of the tests do not *need* to be run in which case they get shifted out and written up as high level test case ideas. These are managed separately through the same excel sheet I use for future ETs.
I keep all exploratory tests listed in excel for future stories and in my agile tool for those relevant to the current stories.
The task itself contains any of the relevant information you may need to run / manage an exploratory test including details like charter, tester, time and date etc. The task is free text so anything you want to include you can. Adding hyperlinks to supporting documents is also a good idea. As and when attachments become available in iMeta Agility I will be using this feature to attach videos of the testing sessions along with screen shots etc. It will help to keep it all together.
Having all of my notes electronically really suits how I work.
So, what about defects?
All defects I find through an ET will have their details recorded in the defect section of the exploratory test. How they are managed outside of that is one for company specific process.
So, what about metrics?
I’ve yet to have a really conclusive need to produce any of the metrics I used to produce in a traditional environment. That’s not because metrics are bad or evil. It’s just the responsibility for quality moves to the team and hence there is less of a need to gather metrics. Instead we do fast and regular deployments where the software is tested in small and manageable chunks. This allows fast feedback and smaller more manageable fixes. Metrics don’t really seem relevant other than the usual milestones and burndown reporting needs.
So, what about being able to trace tests and testing?
This is an interesting one because as the testing completes some teams build up a series of test cases for regression and future testing. Some teams produce a simple but effective automation suite to cover the growing regression. Some teams do nothing. Some teams automate as much as possible. Some teams never re-run an exploratory test.
I tend to create a new test case for any exploratory testing idea that ‘felt’ interesting enough to ever have to run again. For normal tests I automate simple versions of them for regression whilst at the same time building up a test case regression pack to run in future sprints. The automation gives a good indication of how stable the software is, leaving more time for exploratory tests. Too much automation for me and I would have no time to explore. Too little and I wouldn’t have the confidence in the software and i would be spending too much time regression testing, especially as the application grows. It’s a balance which is often tricky to maintain.
All of my test cases are high level guidance tests with limited step details placing more emphasis on the tester. It allows a bit more creativity but ultimately cuts down the amount of administration I have to do on the tests. But this works for me and is not by any means a Best Practice.
So by hijacking the task feature in iMeta Agility (or any other scrum tool) it is possible to list out, manage and record for posterity all of my exploratory testing ideas (with a little help from excel – for now. It certainly beats (for ease) any other system I’ve so far tried. Sure, it lacks the reporting power of session based test management tool and the metric generating abilities of excel, but it’s simple, clean, efficient. And more importantly it works for us, in our scrum environment.
Let me know how you manage your exploratory tests in your environment…..
Rob..