On a recent agile project the testing consisted of record and playback scripts running nightly in a permanent and ever growing regression suite. And it worked!
The reason I’m making a point of this is that in the testing community it is often stated that record and playback does not work due to maintenance overheads and inflexibility with the scripts. I even used to believe this myself until I had it working effectively with limited overhead. In fact, in the grand scheme of things it actually saved time and more importantly, added masses of quality.
When I say record and playback I also need to make an important point that many of the tests were written (rather than recorded) but written using commands and operations that the recording mechanism would use anyway. The tool of choice for this was Selenium. I installed and used the Firefox add-on ‘Selenium IDE’. To start with I recorded a few tests using the ‘story’ acceptance criteria as my guide. I then found I was editing these scripts to make them generic and future proof. After using them in this way I eventually moved to straight forward writing of the scripts in the IDE.
Development Help
With the help of Mike Blake here at iMeta we worked on integrating these tests in to an automated nightly build process. Mike’s blog covers this in depth. But basically, TFS would do a build, drop the build to the test stack, fire up Selenium R/C (server control), fire up a browser and then kick off the tests. Each test and test suite would run with all steps being recorded to a database. When a fail occurs the system would take a screenshot of the failure as well. All of this information would be published to a website which we used to analyze the results. In essence, this meant that every morning I had a full view of the system (system being the features covered by the selenium scripts).
This regression suite proved useful in an agile environment as work was constantly going on in the areas already completed. As new stories and new features were being worked on in each sprint it was inevitable that changes would be made and errors introduced. In the grand scheme of things the regression did not find that many defects, certainly no-where near as many as the exploratory testing performed did. But what it did do was give confidence in the existing solution and it did highlight some very important defects in functionality we thought was already nailed. The confidence that the tests gave was outstanding and it meant I didn’t have to spend my time looking at features already been given the green light. I could concentrate on the new features and important high risk areas.
In order for the tests to be useful though I did have to re-factor, maintain and in some instances, re-write the selenium tests. This again was inevitable as the system was growing and evolving. It’s agile after all.
Replacing Manual Tests
I basically started using the Selenium IDE as a manual test case holder. Instead of writing steps in to an excel spreadsheet to run manually, I could write these same steps in to Selenium. Using the same ‘sorts’ of commands I could quite easily construct a ‘manual’ test in Selenium, especially with a liberal dosing of comments.
The beauty of doing these scripts in Selenium was that I could run these scripts after they were written and see instantly where they may need tweaking and amending. I could also run each step as I wrote it, debug it or use the scripts to data/state load the system. None of which are possible with a manual test case in a Word document.
Printing the content of the Selenium tests also gave very accurate manual tests to run.
Overall, I did spend a considerable amount of time maintaining these scripts but this time was easily regained by the ability to hit the majority of the site in one click of the mouse. With the build being in TFS I could set the build going at any given time, so not only could I run the tests on any server through TFS, I could drop an entire build and re-run at the click of a mouse also. As the scripts were checked in to TFS I had the TFS tool set on hand to quickly and easily edit them.
Pros and Cons
Here are the pros I found working this way:
- Just as quick to produce the selenium tests as it is to write traditional manual tests
- More time freed up to perform exploratory testing
- Ability to run the tests to check their accuracy whilst writing them
- Regression is run nightly with results ready for analysis at the start of each day
- Quickly and efficiently load data or get the system to a particular ‘state’
- Can run the full regression by simply queuing a build in TFS
- Visible state of the system available each morning
- The scripts were checked in to TFS meaning TFS tools such as find and replace were available
There were some downsides though:
- The scripts often failed due to speed issues (server updating, anti-virus scans using resource etc)
- It is difficult to automate anything that stepped outside of the website (i.e. emails with passwords, exported and emailed documents etc)
- The early scripts were un-maintainable and required constant re-write. This solution only works when the UI is settling down and the feature group is becoming stable.
- Some elements in a web site are harder to automate than other (drag and drop, pop ups) so considerations will need to be made in the design and implementation.
- The test data used has to be really cleverly thought out, more so than manual data as the machine is validating not the human eye.
- Changing designs and processes can cause a large maintenance overhead.
- Analyzing the results each morning can be time consuming (but valuable)
- The scripts will only find what they are told to find so will miss many issues a manual tester would spot.
After the initial pain of getting to grips with Selenium and the limitations the IDE version has, it was possible to be scripting very detailed and accurate scripts in about the same time as it takes to write a manual one. With a decent sprinkling of comments the scripts can be very easy to read. There are also only about 6 commands you need to know to be able to automate tests. Same as a manual test case really (open, click, select, type, assert (check), pause/wait).
Next Steps
We are currently working on extracting out requirements/stories/features labels using the ECHO command so we can produce a requirements matrix from the tests.
Lessons Learnt
I’ve learnt a valuable lesson on this project. The lesson being that there is no such thing as a best practice in testing (i.e that record and playback does not work, that automation is not valuable on small projects etc) – if it works for your project, do it . Cem Kaner makes the point nicely here regarding Context Driven Testing.