Overview
Problem: How can we test our web site? One option is to use http://seleniumhq.org/.
The Firefox plug-in “Selenium IDE” can be used to record, edit and manage scripts. (See their site for details)
We have then taken this further by creating a c# test project and adding the html test scripts into source control. On a nightly/on demand bases we run selenium tests, as if they we’re ms unit tests through TFS.
This process opens the html test suites/cases and sends the commands to a selected Selenium RC server, recording the results in a SQL database.
A Dynamic Data ASP.Net site lets people view the progress of a TestRun with screenshots when an action fails.
Components
Selenium IDE
(Requires Firefox) The Selenium IDE can be used to record Test Cases and link them via a Test Suite. By default, they are stored as HTML. This is the main tool the tester will use to edit, manage and run the test cases.
Selenium RC
The “Remote Control” Java application that is used to interact with a browser based on commands it receives. This should be run on the target OS in a VM.
TestEase
TestEase uses the test cases in a Test Project and sends the commands to the selenium RC. Results are stored in a database.
Build integration
As the tests run as standard C# they can be added as unit tests and then run each night (after a deployment).
Test Ease Site / Reports
A simple web site can be used to display the results from the database. It is even possible to see screenshot when an action fails. Reporting services can be used to pull statistics from the database.
Selenium Grid
If the product under tests needs to support multiple browsers and operating systems, for example a public facing web site, Selenium Grid can be used to run the tests on multiple environments. (this has not been done for our product)
Problems
Speed changes the results. Depending on the web site, depending on if a page comes back or if the test timeout. This might be an issue with an ASP.net site because the first time a new deployment is accessed it JIT compiles the code. Workarounds include: 1, increasing the selenium timeout. 2, “Hitting” the site to force the Jitting before running the tests. 3, Force the asp compiler to JIT the site as part of a deployment.
All tests can break if the control names change (unlike UIA accessibility). By default, selenium will normally use the id of the html control as its reference. If the site layout changes this could break the tests. Workarounds:
Testers need to know about VS and source control including merging. (minor issue)
Screenshot don’t work when using Remote desktop. The VM host view must be used instead.
Selenium RC isn’t a service. Workaround: Auto login/start the RC in case the server is reset.
Firefox isn’t currently working on the RC box due to a profile issue.
Some commands in the IDE are not available in the RC (e.g. echo, pause). TestEase manages these manually.
Ajax requires “pauses” that might not be that robust
Popups can be an issue if the windows aren’t name. The RC supports multi-window that can help when testing a multiple window web site
Future Improvements
Requirement mapping. It would be nice to see the requirements for the system and how many tests cover them. It is thought that an echo or new selenium command can be used to show what requirement the test is covering.
Better configuration management. Some more work needs to be done to the configuration of the c# project. For example, pushing the settings to xml files or to build properties. This will be useful if the tester wants to run the tests on a different set of servers.
Remove the database dependency so that the tests can be run ad-hoc.
Add data driven testing. This could be achieved by using template test files. The templates could contain tokens that are replaced by the TestEase engine. The process could generate test files that can be run by the Selenium IDE with the tokens replaced.
Alternatives
Microsoft Visual Studio Web Testing,
Uia (for WinForms)
VSTS 2010 - http://www.microsoft.com/visualstudio/2010/overview.mspx