Rob Lambert's Blog

There are hundreds of Add-ons for Firefox, most of them are not suitable for helping testers whilst testing but amongst the vast number available there are some true time saving gems.

Here’s a few that I use on a regular basis, you might also find them quite useful.

1. Selenium IDE

https://addons.mozilla.org/en-US/firefox/addon/2079

Selenium is fast becoming the standard web test automation tool and there is good reason for it. You can use it for record and playback testing or get down to code level and drive it that way. The browser platform for replaying the tests is not complete yet but it is getting there. The Selenium IDE is the basic record and playback tool which is invaluable for automating website pages or for quickly loading data through the UI.

2. IE Tab

https://addons.mozilla.org/en-US/firefox/addon/1419

IE Tab is great if you use Firefox as your default browser, especially if you use sharepoint sites as you can flip the current tab to render in IE. In sharepoint this enables controls that are simply not available in Firefox and displays the pages correctly in some instances. It is also great for quickly seeing how a site will look in IE whilst you are testing, although it should not be used to substitute full IE testing.

3. Firesizer 0.65

https://addons.mozilla.org/en-US/firefox/addon/5792

This is a handy little tool that simply resizes the window to a predefined size. Great for testing how your site will look in different sizes.

4. W3C Page Validator

https://addons.mozilla.org/en-US/firefox/addon/2250

Great tool for checking your site against the W3C conformance standards. Does sometimes complain about things that are not W3C failures and does require you to validate each page in turn, but still useful.

5. Pencil Sketcher

https://addons.mozilla.org/en-US/firefox/addon/8487

Amazing little add-on which allows you to design screens and mockups really quickly and within the browser. You can add all sorts of shapes and widgets. Brilliant for mocking up screens for the test plans and cases.

6. SQL Injection

https://addons.mozilla.org/en-US/firefox/addon/6727

This is a great tool for checking SQL injection vulnerabilities on sites under test. It generates the SQL for you and adds it to each available input field.

7. Quick Restart

https://addons.mozilla.org/en-US/firefox/addon/3559

This is a very simple tool but invaluable. It basically adds a restart button to the tool bar which simply restarts the Firefox browser. Invaluable when testing cookies and sessions where lots of restarting is needed to clear them down.

8. Firebug

https://addons.mozilla.org/en-US/firefox/addon/1843

Brilliant add-on for inspecting the page under test. You can view the HTML and CSS as well but the inspection tool is very useful.

9. Molybdenum

https://addons.mozilla.org/en-US/firefox/addon/4149

This is a cool little app for managing Selenium tests. It gives a clearer interface for managing suites and cases. Still not perfect though but better than the standard Selenium IDE.

10. Regular Expressions Tester

https://addons.mozilla.org/en-US/firefox/addon/2077

Useful tool for testing Regular Expressions.

These are just a few of the useful tools that can be found to help testers be more efficient and effective whilst testing web apps.

As much as I would like to make software testing out to be glamorous, international and dangerous, more often than not, it is not. In fact sometimes it can be quite dull. We don’t get to wear disguises, play with the latest spy technology or flit around the world in fast cars. Far from it. Only the other day I drove to work in a family hatchback, sat at my desk testing all day, drove home again and got an early night because I had a headache.

However, there is one thing that testing does have in common with the CIA which I discovered the other night whilst researching something for my wife. I stumbled across the following site: http://fnopress.com/SYN/modules/mod9.htm and realized I employ many of the techniques that the CIA do. Digging further I found that Cem Kaner has also drawn the comparison with exploratory testing and the Phoenix checklist. http://www.kaner.com/pdfs/ExploringExploratoryTesting.pdf

When looking at problems and formulating plans the CIA operatives asks themselves a series of questions; these questions are known as the Phoenix checklist (link here). These questions can be applied to any problem, scenario or plan to tease out more information and find out exactly what information is present or missing. The questions are applied to problems and plans by a CIA operative in the same way a tester would apply the questions to a specification, test case, test plan or software planning discussion. It just happens the subject matter is different. It does not even stop at testing; product managers, business analysts, project managers and software designers can also use these questions (and probably do use a variation of them). The questions are not just limited to the CIA or the IT industry – they are often used in the advertising and media industries as well.

So software testing is loosely associated to the CIA, if only in terms of the questioning strategy. I’ve therefore made myself slightly more interesting by association…..

So what type of questions do software testers and the CIA operatives ask? Well here is the list of questions that the Phoenix checklist recommends. http://blog.futurelab.net/2007/01/the_phoenix_checklist.html

For Problems

• Why is it necessary to solve the problem?

• What benefits would be received by solving the problem?

• What is the unknown?

• What is it you don’t yet understand?

• What is the information you have?

• What isn’t the problem?

• Is the information sufficient? Or is it insufficient? Or redundant? Or contradictory?

• Where are the boundaries of the problem? Can you separate the various parts of the problem? Can you write them down? What are the relationships of the parts of the problem?

• What are the constants (things that can’t be changed) of the problem?

• Have you seen this problem before?

• Have you seen this problem in a slightly different form?

• Do you know a related problem?

• Try to think of a familiar problem having the same or similar unknown?

• Suppose you find a problem related to yours that has already been solved. Can you use it? Can you use its method?

• Can you restate your problem? How many different ways can you restate it? More general? More specific?

• Can the rules be changed?

• What are the best, worst, and most probable cases you can imagine?

For Plans

• Can you solve the whole problem? Part of the problem?

• What would you like the resolution to be? Can you picture it?

• How much of the unknown can you determine?

• Can you derive something useful from the information you have?

• Have you used all the information?

• Have you taken into account all essential notions in the problem?

• Can you separate the steps in the problem-solving process? Can you determine the correctness of each step?

• What creative thinking techniques can you use to generate ideas? How many different techniques?

• Can you see the result? How many different kinds of results can you see?

• How many different ways have you tried to solve the problem?

• Can you intuit the problem? Can you check the result?

• What should be done? How should it be done?

• Where should it be done?

• When should it be done?

• Who should do it?

• What do you need to do at this time?

• Who will be responsible for what?

• Can you use this problem to solve some other problem?

• What is the unique set of qualities that makes this problem different from another?

• What milestones can best mark your progress?

• How will you know when you are successful?

Looking at the list above I have been asking these questions of specifications, test plans, test cases and the software itself throughout my whole career. Fair enough, I may not have phrased them in the same way, or asked them all at the same time in the same order, but none the less I have questioned everything in the same way. 

In summary, questioning things thoroughly at all stages of the Software Development Lifecycle is what software testers bring to the table and formulating those questions around the Phoenix Checklist is not a bad place to start.

http://www.kaner.com/pdfs/ExploringExploratoryTesting.pdf

The problem

Ensuring the website under test looks and works as designed on the multitude of browsers on the market can be a nightmare. Testing the current version of Internet Explorer and Firefox is reasonably straight forward. Throw in a Linux and Mac browser and maybe Opera and you have a serious amount of browsers to test across. Throw in previous versions of all of the browsers and this suddenly becomes a mammoth task.

Defining the browsers supported by the product up front has never been more important. Avoiding ambiguous browser statements is also a way of limiting potential problems and rafts of unnecessary browser testing. For example, “the product will work on IE 6 and above” – are we including all future unknown versions of IE? Betas? Or just the ones out on the market at the moment? Clear the ambiguity by stating definitive versions.

Given the requirements and browser support is sorted and defined, we have the problem of testing the website on each one.

The solutions

Unless time exists in abundance then choosing a subset of end to end tests and running these on each browser will be the most effective route. Drilling down in to the detail if time is available. But how do you manage the browsers?

• One way is to utilise Virtual machines to create operating systems with different browser mixes
• The downside to this is that it is time consuming and you end up with a large amount of virtual machines to manage and maintain.
• Multiple boot machines is another way.
• The downside to this is that you need more hardware and the management of machines becomes a bind.
• Emulators of each browser
• Emulators tend to be slow.
• Not all browsers have emulators available.
• Run multiple versions of each browser on the same machine
• This is not possible for all browsers.
• The registry maintenance becomes a bind

A very cool way I have found recently to partially test how a site will ‘look’ is to let someone else do the hard work. There are loads of services now that simply visit a given URL using several machines with different browsers and simply take snapshots of the page. The snapshots are posted back to a website or emailed to you where you can view how your page looks.

• This is a great way of seeing quickly what your site looks like.
• This technique does not tell you how your website ‘works’ and ‘acts’ within that browser.
• It will also not tell you what your site looks like past a secure log in.
• Another downside is that you need to pass it every URL possible from your site – of which there may be many.
• This tool does not allow you to see what your site looks like whilst in development/test, unless you expose your test stacks to the outside world or publish your site to the web.

So, is it actually a useful process? Well, yes – as a quick snapshot of what your site will look like (assuming you can provide suitable URLs) it is invaluable. In terms of proving that you are indeed supporting the browser, well, not so useful.

Another solution?

I’ve been investigating ways of using Selenium RC to launch my Selenium automation scripts in multiple browsers using Selenium Core. It has the potential to solve the problem of whether the site works (not look or feel – unless you sit and watch the script) but at the moment it does not support enough browsers (Selenium browser support listings) so cannot offer a complete solution.

Other automation tools can also offer this type of solution (launching different browsers) but again, the coverage of browsers is not complete.

So at the moment I am going to have to combine a mixture of the above. Virtual machines with different browsers installed, selenium scripts running end to end on supported browsers and screen grabs from an online service mixed with good old fashioned testing.

Unless anyone can think of a more efficient and effective way of browser testing? Suggestions more than welcome.

http://browsershots.org/

http://healyourchurchwebsite.com/2007/07/03/how-to-test-your-site-across-multiple-browsers/

The problem

I came across this issue the other day and with the help of a friendly developer we narrowed this down to a post-back timeout.

Selenium states “[error] Element 100_page1_Page1Information_button1 not found” when clearly the button is there on the screen.

I can click the button and the test can continue but Selenium barfs due apparently not finding the button.

The reason

The command before this one in the script changes a drop down which then does a post back to refresh a list shown on the same page. The problem is that the Selenium test is moving to the next command (push the button) whilst the post-back is still taking place.

The solution

A simple solution exists – slow the selenium playback script down. Re-run the test and the next step in the script should hopefully occur after the post back has happened.

“The more we can make testing intellectually rich and fluid, the more likely we will hit upon the right tests at the right time. That’s where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test.” – Snippet from James Bach’s article on Exploratory Testing which can be found at http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=2255

Exploratory testing (ET) is a very powerful way of finding defects fast but can it really replace scripted testing? For some companies exploratory testing is the only type of testing performed but for most companies a mix of scripted, automated and exploratory is a good place to be. In my experience stepping off the test case track and exploring the software often generates a significant number of defects. If ET is performed in a controlled way there is no reason why this should not form part of your test strategy and plan.

So what is exploratory testing?

Exploratory testing is:

• “Any testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.” state Cem Kaner and Andy Tinkham in their joint publication about exploratory testing (click here to view the PDF document).
• A controlled way of exploring a small concentrated section of the application under test .
• An audited and recorded way of testing.
• Repeatable, reportable and traceable.
• A valid testing technique used with exceptional success.
• Something most testers will have performed in the past whether knowingly or not.
• A technique only experienced testers should be performing if exploratory is used as a main form of testing (experience can be defined as having product knowledge, domain knowledge, hardware knowledge, testing technique experience) .

Exploratory testing is not:

• Random exploration of the software under test with no agenda or charter. (this is more commonly known as ad-hoc testing)
• A way of pretending to test and doing nothing all day.
• A task that should be carried out by junior or inexperienced testers.
• A new concept or buzzword. ET has been around for a very long time as a concept, it is only recently that people have started to appreciate the power of it.
• Exploring to write test cases to replace design, specification or requirements documentation. Writing tests whilst testing is never ideal if there are no other supporting documents to reference. How can a tester be sure the feature is working correctly with no reference? ET assumes a reference exists to how the solution should work (spec, email, requirements doc) – it just happens no script has been produced in advance which limits the testing path.

When and where could it be used?

Exploratory testing can be used on any project at any time but lends itself more to agile or iterative projects where requirements and features may be rapidly changing. Saying that though, making use of say one day a week for ET will be very valuable even in traditional waterfall models. Some projects may require test artifacts to be provided in which case it may not be suitable to use ET. Other projects may have no experienced testers available in which case scripted testing may be the most effective use of test resource.

With the rise of agile methodologies and on projects where requirements can be fluid, ET can often be the most effective form of testing. Re-writing test cases when designs change, scripting up front when there are little or no details available and not knowing which tests you can/cannot run in which order are all reasons to look at exploratory testing as an effective test approach.

A good way of knowing when to use exploratory testing is for the tester to jot down potential areas to investigate and then on a given day spend a given time exploring that area. An experienced tester will find many areas which, when exploratory testing is exercised, will show many more defects than a pre-scripted test will find. The key point here is to spend a ‘given’ amount of time on the area. If no defects are found then move on.

How can we report on exploratory testing?

Repeatable, accountable and traceable exploratory tests are often overlooked and this is where the lines between ET and Ad-Hoc testing can cross. There must be a charter (aim) of the test, a dedicated time slot and a dedicated tester – the rest of the information gathered at the time is open for debate.

Simple approaches of recording test steps in a text editor or spreadsheet offer the repeatable tests, but only if all information is carefully recorded by the tester.

A wonderful tool that makes use of this text based technique but adds on reportable and accountable ways of analyzing the results is a session based testing tool from Jonathon and James Bach. Click here to go to the Satisfice website. Results from the testing can be linked to a requirement/area and then reported on.

However, the most effective way I have found over the last few years when testing websites is to use one of the many web capture (record and playback) tools available like Selenium. Recording your exploratory session using a tool such as this allows repeated testing (where the software under test allows) but more importantly provides a full script of what was tested, in what order, with what data and what the outcomes were. Any errors encountered are recorded in the test and any other notes can be inserted as comments.

For more information on Exploratory Testing check out the following resources:

http://www.kaner.com/pdfs/ExploringExploratoryTesting.pdf – PDF document

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=2255

http://www.informit.com/articles/article.aspx?p=405514

http://www.satisfice.com/

http://selenium.openqa.org/