Rob Lambert's Blog

You may be thinking 'here he goes again'. Not content with associating testing to the CIA the other week, he is now trying to create a tenuous link to Toyota. Bear with me though. There is a very real link between Toyota and software testing and that link is quality. In fact, any part of the software development lifecycle or any department within the development cycle can learn from Toyota. It is simply a case of process improvement.

People who know me, know that I love cars. More specifically that I love Japanese cars and even more specifically Toyotas. The reasons are simple, some models are quick, insane fun, frugal, comfortable and more importantly, reliable. Don’t just take my word for it the JD Power survey (a customer satisfaction survey) has placed Toyota (and luxury arm, Lexus) in the top 5 for the last 6 years.

The JD Power results for 2007.

So we have established Toyota have an enviable reputation for quality. We also know that software testing is about driving quality and reliability in to the software process and the product under test. There are many ways the two are linked together but the one I am going to concentrate on is Kaizen.

Kaizen is the Japanese term which literally means “continuous improvement” or "Good Change”. It is a methodology, a mind set even, that instills quality in to every single aspect of the workplace.

Toyota have taken Kaizen and made it a fundamental policy within its company which has resulted in them being famous for quality and reliability.

Every employee at Toyota has a say in how their area/department is run with quality and efficiency being the core goal. Anything at all that will improve efficiency and quality is implemented – no matter how small it is. It is not unheard of for the conveyor belts at Toyota’s Nagoya factory to be stopped as someone has spotted something which could be a potential quality issue. A process is then performed to work out how to solve the potential quality issue before the belts are resumed and work can continue. The end product is a car (and build process) that has an enviable reputation for quality.

Broadening the picture shows that almost every single Japanese car manufacturer has amazing quality levels and this is because in Japan, Kaizen is part of the culture. 

Kaizen is what we in the IT industry often call "process improvement". Process improvement is something all companies should be doing to stay ahead in a competitive industry and to improve efficiency in general. Improving quality in the build, the documentation, the review process, the tools we all use daily, communication and anything else that can affect the way we work is essential.

Using Wikipedia's description of Kaizen we can clearly see that this fits nicely with how process improvement should work.

• standardize an operation
• measure the standardized operation
• gauge measurements against requirements
• innovate to meet requirements and increase productivity
• standardize the new, improved operations
• continue cycle ad infinitum.

It seems so simple but it is something many companies and individuals fail to grasp and implement. In order to improve there must be something to measure. In order to measure you must know what it is you are wanting to improve.

Most software testers have a thing for metrics and stats. They know deep down that metrics should be taken with a pinch of salt and that on face value, they are often not the 'whole' picture of the facts, but gathered over time they show a valuable measurement that can be used for improvement. Defects per build, defects per functional area, test case pass failure rates etc all show information regarding efficiency and quality.

A high number of defects per functional area may show that the specification needs improving or that there is a misinterpretation of implementation between development and test. It may just show that this area is very complicated and needs more resource, maybe even more detailed planning. Whatever the interpretation of the stats there are always areas for potential improvement.

Even the regular processes that people may take for granted could yield some areas of improvement, like daily stand-ups and team meetings right through to creating build environment and test stacks. There is always something that can be standardized, measured and then improved upon. Test case writing and detail levels are a classic example.

Often the first hurdle to overcome is the empowerment of employees to have their say and to feel confident in putting forward suggestions for improvement. End of project reviews, end of iteration reviews and general feedback forums are essential to getting a balanced view point. Making sure people feel comfortable giving feedback which at times can often be negative, is a crucial step forward.

However, Kaizen and process improvement can be taken too far. Companies that often pride themselves on a relaxed, none bureaucratic environment can soon lose this with too much process improvement. With process improvement often comes endless forms, communication documents, processes, rules and regulations. The key is finding the right balance, getting it just right so that quality and efficiency go up, but the work environment remains as you wish.

If Kaizen can be held responsible for high quality and reliable cars then process improvement can surely lead to high quality and reliable software and a software company having that reputation can’t be a bad thing.

http://www.gembapantarei.com/2006/08/kaizen_in_software_development_start_by_seeing_the_7_wastes.html

http://en.wikipedia.org/wiki/Toyota_Production_System

http://en.wikipedia.org/wiki/Big_Three_automobile_manufacturers

http://newyorkcto.blogspot.com/2007/06/kaizen-meeting-in-agile.html

http://www.oracle.com/technology/pub/articles/dev2arch/2008/05/kaizen-bpm-agile.html

Exhaustive or complete testing is practically impossible unless you have a really simple product or have all the time and money in the world. Even simple applications become increasingly complicated to exhaustively test, especially when lots of variables are introduced. These variables could be anything from simple print options to different hardware specifications. So faced with many variables, not enough time or resource and the need to release a high quality product then what should a tester do? Pairing, that's what.

So what is pairing?

To quote testing Guru James Bach “…..you could create tests that pair each value of each of the variables with each value of each other variable at least once”.

Some examples of unrealistic testing combinations are:

• In the first four moves in a game of chess there are 318,979,564,000 possible combinations.
• A simple two field combination that accepts one digit (0-9) in each would give 100 combinations.

There is a really good example in The Art of Software Testing by Glenford J. Myers (Amazon link) regarding what at first glance looks like a simple program (a real world one). Some exploring of potential combinations leads to the reality that there are 100 trillion paths through the program. That is a lot of testing.

So, it is not only impractical in some cases to test each combination but it can also be a waste of valuable resource? Using a pair technique would allow a tester to validate the behavior of the fields and then be off finding issues in other areas where there is potentially more value added. Even pairing some variables will results in many more tests than time allows. This then comes down to risk assessment and prioritization but in general a simple pairing of variables will allow a significant number of tests to be run to give confidence in that application.

So how easy is it to pair?

One of the easiest ways to pair is to simply download one that already exists. James Bach has created a superb one which takes in the variables as a comma separated file and then pushes out an excel file with the combinations listed. In an example James uses the number of variables for one program was 10,000,000,000, which when paired came out at a much more palatable 177. You can read more at his pages as well as download the tool. You can also find a very cool data generator tool called Perlclip on James' web site also - it is invaluable! http://www.satisfice.com/tools.shtml

Microsoft also have their own tool called PICT33 which doesn’t seem to be showing on their site, but the MSI can be run from here: (Warning – it will run the MSI)

http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi 

There are also loads of tools here:

http://pairwise.org/tools.asp

If you are faced with an overwhelming list of variables, not enough time to test them all, not enough budget to draft in an army of testers and a deadline to deliver a quality solution then pairing is the most effective way of giving confidence in the solution. A decent pairing tool will be invaluable.

There are hundreds of Add-ons for Firefox, most of them are not suitable for helping testers whilst testing but amongst the vast number available there are some true time saving gems.

Here’s a few that I use on a regular basis, you might also find them quite useful.

1. Selenium IDE

https://addons.mozilla.org/en-US/firefox/addon/2079

Selenium is fast becoming the standard web test automation tool and there is good reason for it. You can use it for record and playback testing or get down to code level and drive it that way. The browser platform for replaying the tests is not complete yet but it is getting there. The Selenium IDE is the basic record and playback tool which is invaluable for automating website pages or for quickly loading data through the UI.

2. IE Tab

https://addons.mozilla.org/en-US/firefox/addon/1419

IE Tab is great if you use Firefox as your default browser, especially if you use sharepoint sites as you can flip the current tab to render in IE. In sharepoint this enables controls that are simply not available in Firefox and displays the pages correctly in some instances. It is also great for quickly seeing how a site will look in IE whilst you are testing, although it should not be used to substitute full IE testing.

3. Firesizer 0.65

https://addons.mozilla.org/en-US/firefox/addon/5792

This is a handy little tool that simply resizes the window to a predefined size. Great for testing how your site will look in different sizes.

4. W3C Page Validator

https://addons.mozilla.org/en-US/firefox/addon/2250

Great tool for checking your site against the W3C conformance standards. Does sometimes complain about things that are not W3C failures and does require you to validate each page in turn, but still useful.

5. Pencil Sketcher

https://addons.mozilla.org/en-US/firefox/addon/8487

Amazing little add-on which allows you to design screens and mockups really quickly and within the browser. You can add all sorts of shapes and widgets. Brilliant for mocking up screens for the test plans and cases.

6. SQL Injection

https://addons.mozilla.org/en-US/firefox/addon/6727

This is a great tool for checking SQL injection vulnerabilities on sites under test. It generates the SQL for you and adds it to each available input field.

7. Quick Restart

https://addons.mozilla.org/en-US/firefox/addon/3559

This is a very simple tool but invaluable. It basically adds a restart button to the tool bar which simply restarts the Firefox browser. Invaluable when testing cookies and sessions where lots of restarting is needed to clear them down.

8. Firebug

https://addons.mozilla.org/en-US/firefox/addon/1843

Brilliant add-on for inspecting the page under test. You can view the HTML and CSS as well but the inspection tool is very useful.

9. Molybdenum

https://addons.mozilla.org/en-US/firefox/addon/4149

This is a cool little app for managing Selenium tests. It gives a clearer interface for managing suites and cases. Still not perfect though but better than the standard Selenium IDE.

10. Regular Expressions Tester

https://addons.mozilla.org/en-US/firefox/addon/2077

Useful tool for testing Regular Expressions.

These are just a few of the useful tools that can be found to help testers be more efficient and effective whilst testing web apps.

As much as I would like to make software testing out to be glamorous, international and dangerous, more often than not, it is not. In fact sometimes it can be quite dull. We don’t get to wear disguises, play with the latest spy technology or flit around the world in fast cars. Far from it. Only the other day I drove to work in a family hatchback, sat at my desk testing all day, drove home again and got an early night because I had a headache.

However, there is one thing that testing does have in common with the CIA which I discovered the other night whilst researching something for my wife. I stumbled across the following site: http://fnopress.com/SYN/modules/mod9.htm and realized I employ many of the techniques that the CIA do. Digging further I found that Cem Kaner has also drawn the comparison with exploratory testing and the Phoenix checklist. http://www.kaner.com/pdfs/ExploringExploratoryTesting.pdf

When looking at problems and formulating plans the CIA operatives asks themselves a series of questions; these questions are known as the Phoenix checklist (link here). These questions can be applied to any problem, scenario or plan to tease out more information and find out exactly what information is present or missing. The questions are applied to problems and plans by a CIA operative in the same way a tester would apply the questions to a specification, test case, test plan or software planning discussion. It just happens the subject matter is different. It does not even stop at testing; product managers, business analysts, project managers and software designers can also use these questions (and probably do use a variation of them). The questions are not just limited to the CIA or the IT industry – they are often used in the advertising and media industries as well.

So software testing is loosely associated to the CIA, if only in terms of the questioning strategy. I’ve therefore made myself slightly more interesting by association…..

So what type of questions do software testers and the CIA operatives ask? Well here is the list of questions that the Phoenix checklist recommends. http://blog.futurelab.net/2007/01/the_phoenix_checklist.html

For Problems

• Why is it necessary to solve the problem?

• What benefits would be received by solving the problem?

• What is the unknown?

• What is it you don’t yet understand?

• What is the information you have?

• What isn’t the problem?

• Is the information sufficient? Or is it insufficient? Or redundant? Or contradictory?

• Where are the boundaries of the problem? Can you separate the various parts of the problem? Can you write them down? What are the relationships of the parts of the problem?

• What are the constants (things that can’t be changed) of the problem?

• Have you seen this problem before?

• Have you seen this problem in a slightly different form?

• Do you know a related problem?

• Try to think of a familiar problem having the same or similar unknown?

• Suppose you find a problem related to yours that has already been solved. Can you use it? Can you use its method?

• Can you restate your problem? How many different ways can you restate it? More general? More specific?

• Can the rules be changed?

• What are the best, worst, and most probable cases you can imagine?

For Plans

• Can you solve the whole problem? Part of the problem?

• What would you like the resolution to be? Can you picture it?

• How much of the unknown can you determine?

• Can you derive something useful from the information you have?

• Have you used all the information?

• Have you taken into account all essential notions in the problem?

• Can you separate the steps in the problem-solving process? Can you determine the correctness of each step?

• What creative thinking techniques can you use to generate ideas? How many different techniques?

• Can you see the result? How many different kinds of results can you see?

• How many different ways have you tried to solve the problem?

• Can you intuit the problem? Can you check the result?

• What should be done? How should it be done?

• Where should it be done?

• When should it be done?

• Who should do it?

• What do you need to do at this time?

• Who will be responsible for what?

• Can you use this problem to solve some other problem?

• What is the unique set of qualities that makes this problem different from another?

• What milestones can best mark your progress?

• How will you know when you are successful?

Looking at the list above I have been asking these questions of specifications, test plans, test cases and the software itself throughout my whole career. Fair enough, I may not have phrased them in the same way, or asked them all at the same time in the same order, but none the less I have questioned everything in the same way. 

In summary, questioning things thoroughly at all stages of the Software Development Lifecycle is what software testers bring to the table and formulating those questions around the Phoenix Checklist is not a bad place to start.

http://www.kaner.com/pdfs/ExploringExploratoryTesting.pdf

The problem

Ensuring the website under test looks and works as designed on the multitude of browsers on the market can be a nightmare. Testing the current version of Internet Explorer and Firefox is reasonably straight forward. Throw in a Linux and Mac browser and maybe Opera and you have a serious amount of browsers to test across. Throw in previous versions of all of the browsers and this suddenly becomes a mammoth task.

Defining the browsers supported by the product up front has never been more important. Avoiding ambiguous browser statements is also a way of limiting potential problems and rafts of unnecessary browser testing. For example, “the product will work on IE 6 and above” – are we including all future unknown versions of IE? Betas? Or just the ones out on the market at the moment? Clear the ambiguity by stating definitive versions.

Given the requirements and browser support is sorted and defined, we have the problem of testing the website on each one.

The solutions

Unless time exists in abundance then choosing a subset of end to end tests and running these on each browser will be the most effective route. Drilling down in to the detail if time is available. But how do you manage the browsers?

• One way is to utilise Virtual machines to create operating systems with different browser mixes
• The downside to this is that it is time consuming and you end up with a large amount of virtual machines to manage and maintain.
• Multiple boot machines is another way.
• The downside to this is that you need more hardware and the management of machines becomes a bind.
• Emulators of each browser
• Emulators tend to be slow.
• Not all browsers have emulators available.
• Run multiple versions of each browser on the same machine
• This is not possible for all browsers.
• The registry maintenance becomes a bind

A very cool way I have found recently to partially test how a site will ‘look’ is to let someone else do the hard work. There are loads of services now that simply visit a given URL using several machines with different browsers and simply take snapshots of the page. The snapshots are posted back to a website or emailed to you where you can view how your page looks.

• This is a great way of seeing quickly what your site looks like.
• This technique does not tell you how your website ‘works’ and ‘acts’ within that browser.
• It will also not tell you what your site looks like past a secure log in.
• Another downside is that you need to pass it every URL possible from your site – of which there may be many.
• This tool does not allow you to see what your site looks like whilst in development/test, unless you expose your test stacks to the outside world or publish your site to the web.

So, is it actually a useful process? Well, yes – as a quick snapshot of what your site will look like (assuming you can provide suitable URLs) it is invaluable. In terms of proving that you are indeed supporting the browser, well, not so useful.

Another solution?

I’ve been investigating ways of using Selenium RC to launch my Selenium automation scripts in multiple browsers using Selenium Core. It has the potential to solve the problem of whether the site works (not look or feel – unless you sit and watch the script) but at the moment it does not support enough browsers (Selenium browser support listings) so cannot offer a complete solution.

Other automation tools can also offer this type of solution (launching different browsers) but again, the coverage of browsers is not complete.

So at the moment I am going to have to combine a mixture of the above. Virtual machines with different browsers installed, selenium scripts running end to end on supported browsers and screen grabs from an online service mixed with good old fashioned testing.

Unless anyone can think of a more efficient and effective way of browser testing? Suggestions more than welcome.

http://browsershots.org/

http://healyourchurchwebsite.com/2007/07/03/how-to-test-your-site-across-multiple-browsers/

The problem

I came across this issue the other day and with the help of a friendly developer we narrowed this down to a post-back timeout.

Selenium states “[error] Element 100_page1_Page1Information_button1 not found” when clearly the button is there on the screen.

I can click the button and the test can continue but Selenium barfs due apparently not finding the button.

The reason

The command before this one in the script changes a drop down which then does a post back to refresh a list shown on the same page. The problem is that the Selenium test is moving to the next command (push the button) whilst the post-back is still taking place.

The solution

A simple solution exists – slow the selenium playback script down. Re-run the test and the next step in the script should hopefully occur after the post back has happened.

“The more we can make testing intellectually rich and fluid, the more likely we will hit upon the right tests at the right time. That’s where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test.” – Snippet from James Bach’s article on Exploratory Testing which can be found at http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=2255

Exploratory testing (ET) is a very powerful way of finding defects fast but can it really replace scripted testing? For some companies exploratory testing is the only type of testing performed but for most companies a mix of scripted, automated and exploratory is a good place to be. In my experience stepping off the test case track and exploring the software often generates a significant number of defects. If ET is performed in a controlled way there is no reason why this should not form part of your test strategy and plan.

So what is exploratory testing?

Exploratory testing is:

• “Any testing to the extent that the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests.” state Cem Kaner and Andy Tinkham in their joint publication about exploratory testing (click here to view the PDF document).
• A controlled way of exploring a small concentrated section of the application under test .
• An audited and recorded way of testing.
• Repeatable, reportable and traceable.
• A valid testing technique used with exceptional success.
• Something most testers will have performed in the past whether knowingly or not.
• A technique only experienced testers should be performing if exploratory is used as a main form of testing (experience can be defined as having product knowledge, domain knowledge, hardware knowledge, testing technique experience) .

Exploratory testing is not:

• Random exploration of the software under test with no agenda or charter. (this is more commonly known as ad-hoc testing)
• A way of pretending to test and doing nothing all day.
• A task that should be carried out by junior or inexperienced testers.
• A new concept or buzzword. ET has been around for a very long time as a concept, it is only recently that people have started to appreciate the power of it.
• Exploring to write test cases to replace design, specification or requirements documentation. Writing tests whilst testing is never ideal if there are no other supporting documents to reference. How can a tester be sure the feature is working correctly with no reference? ET assumes a reference exists to how the solution should work (spec, email, requirements doc) – it just happens no script has been produced in advance which limits the testing path.

When and where could it be used?

Exploratory testing can be used on any project at any time but lends itself more to agile or iterative projects where requirements and features may be rapidly changing. Saying that though, making use of say one day a week for ET will be very valuable even in traditional waterfall models. Some projects may require test artifacts to be provided in which case it may not be suitable to use ET. Other projects may have no experienced testers available in which case scripted testing may be the most effective use of test resource.

With the rise of agile methodologies and on projects where requirements can be fluid, ET can often be the most effective form of testing. Re-writing test cases when designs change, scripting up front when there are little or no details available and not knowing which tests you can/cannot run in which order are all reasons to look at exploratory testing as an effective test approach.

A good way of knowing when to use exploratory testing is for the tester to jot down potential areas to investigate and then on a given day spend a given time exploring that area. An experienced tester will find many areas which, when exploratory testing is exercised, will show many more defects than a pre-scripted test will find. The key point here is to spend a ‘given’ amount of time on the area. If no defects are found then move on.

How can we report on exploratory testing?

Repeatable, accountable and traceable exploratory tests are often overlooked and this is where the lines between ET and Ad-Hoc testing can cross. There must be a charter (aim) of the test, a dedicated time slot and a dedicated tester – the rest of the information gathered at the time is open for debate.

Simple approaches of recording test steps in a text editor or spreadsheet offer the repeatable tests, but only if all information is carefully recorded by the tester.

A wonderful tool that makes use of this text based technique but adds on reportable and accountable ways of analyzing the results is a session based testing tool from Jonathon and James Bach. Click here to go to the Satisfice website. Results from the testing can be linked to a requirement/area and then reported on.

However, the most effective way I have found over the last few years when testing websites is to use one of the many web capture (record and playback) tools available like Selenium. Recording your exploratory session using a tool such as this allows repeated testing (where the software under test allows) but more importantly provides a full script of what was tested, in what order, with what data and what the outcomes were. Any errors encountered are recorded in the test and any other notes can be inserted as comments.

For more information on Exploratory Testing check out the following resources:

http://www.kaner.com/pdfs/ExploringExploratoryTesting.pdf – PDF document

http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=2255

http://www.informit.com/articles/article.aspx?p=405514

http://www.satisfice.com/

http://selenium.openqa.org/

When adding a Manual Test (word doc) in Visual Studio Test Edition a template is generated allowing steps/data to be added as per the default Microsoft template. It may be the case that this template needs to be changed to conform to your test teams template.

The following steps explain how to change this manual test case document to your own company word template. At the moment I have not yet found a workaround to allow Excel to be used as a template.

1. Creating the template

The easiest way to create your new template is to edit the existing Visual Studio template.

• Locate the template on your computer. The zip file you need to copy is called something along the lines of ManualTestWordFormat.zip and is commonly located here: Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\<Language you want to edit>\1033
• Copy the zip file out and save it somewhere easy to find.
• Open the zip – it should contain two files ManualTest.mht and ManualTest.vstemplate
• Extract the files and then open the ManualTest.mht. By default it will open in MS Word.
  • Once open, modify the contents to be the same as your desired test case. This will be your template so avoid adding test specific information like testers name or project etc. Keeping it generic means it can be maintained easily.
  • Save the file without changing the name
• Open the .vstemplate file in notepad and change the following values Description, TemplateID, ProjectItem, Name (if you changed the manualtest.mht name) and Default Name (if you changed the manualtest.mht name) to be something pertinent to the template.
• Then remove the information in the TestProjectData tag – leave it blank.
• You MUST make sure that the TemplateID is unique so change at least 1 character in it.
  • Save the file without changing the name
• Zip them back in to the ManualTestWordFormat.zip file and keep a copy safe. This is now your template to share with other testers.
• IMPORTANT: This is only your template for the <Language you want to edit> . You will need to make several copies of this template and simply change the <ProjectType>Language</ProjectType> to be the correct language of your template. So, if you were replacing the C# template then you would need to make sure this file contained <ProjectType>CSharp</ProjectType>

2. Replace in Visual Studio

• For Visual Studio to pick up the new template it is important to add it to the Visual Studio directory but firstly you will need to remove the old template. Do this by going to: Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplatesCache and deleting the template from each of the cached folders you want to replace.
• Then paste the new zip file (with the relevant language to the following locations.
  • Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\CSharp\1033
  • Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\VisualBasic\1033
  • Program Files\Microsoft Visual Studio 8\Common7\IDE\ItemTemplates\VisualC\1033
• Choose to replace each of the existing files. This will replace the existing manual template with your specific one. Make sure that Visual Studio is closed whilst you do this. Any previously created manual tests (with the old template) may need to be removed from your solution as they may cause issues.

3. Load the files

• Open the Visual Studio command prompt as administrator
  • Start > All Programs > Microsoft Visual Studio 2005 > Visual Studio Tools > Visual Studio 2005 Command Prompt
• Type in and execute the following command:
• devenv /InstallVSTemplates
• It may take some time to complete.

4. Test the template

• Now this is complete open up Visual Studio and in Test Manager/Solution create a new Manual Test using Word Template. When Word opens it should now be the new template.

Software testing is more than validating the software simply does what it should do (functional testing) we also have to make sure it does what it should do in a certain way (non-functional testing). Performance testing is one of these non-functional testing technique with a huge array of tools and techniques available to testers.

At a basic level setting up simple performance measures on the machines under test can offer an amazing insight into the performance of a solution.

The following is a very simple introduction to gathering simple performance metrics from the machines under test.

This guide is using Windows Performance monitoring and was also created on a Windows 2003 server. Other operating systems may be slightly different.

Here goes:

Log on to machine you want to gather the metrics from.

Start > Control Panel > Administrative Tools > Performance

Click on the Performance Logs and Alerts > Counter Logs

Performance window

Right click in right hand pane

Choose New Log Setting

New Log dialogue

Enter a name (this name will become the name of the log file)

Log dialogue

Choose ‘Add Counters’

Add counters window

Choose the performance object and then the relevant counters required (i.e. CPU, Memory, etc). When finished click close.

Counters summary window

Choose suitable Sample data rate interval for the testing being done.

Sampling too often may mean you have too much data to waft through, yet sampling too infrequently may mean you miss vital clues to performance. This value may have a slight impact on the performance of the machine also as it is grabbing data and writing it to a file so this is a consideration when analysing the results.

Click apply when you have finished.

Configuring the log files

Set the location of the log file. This file will contain all of the data used for analysis. Choose a log file type. I always use Excel as powerful charts and graphs can be produced off the back of the metrics gathered.

Schedule Window

Move on to the Schedule Tab and choose a suitable schedule. Starting the file can be done automatically or manually. For greater control it is often worth starting them automatically. A batch file or script could be written to control this.

Performance window with configured log

The new log will show in the right hand pane of the performance window.

Right clicking can stop/start the counter. Once the log has run it will output a file to the selected location.

Opening the created log file from the destination set up before looks like this in Excel.

More visual metrics can be produced such as graphs and charts off the back of this data.

This is a very quick and easy way for manual testers to see basic performance figures of the machines under test.

Tying these metrics in with common user actions/scenarios and gathering the metrics through automation tools is the next logical step. Ensuring that the machines and network configurations are as near to Live as possible is essential if performance testing is a formal part of your test strategy.

A performance test tool such as Visual Studio Load Test can offer all of this functionality as do many other testing tools. For those with no access to these sorts of tools this is the next best thing.