Michael Blake's Blog

We’ve got a VS2010 TFS Beta 2 server going on a VM. Here are some more hints…

- Follow this guide or Google for one:

- Create a DOMAIN user called DEV\TFSReports. The TFS server must also be on a domain.

- An install with no projects takes ~18GB (windows, SQL, WSS, TFS) so make sure you have at least a 30GIG drive.

- Make sure you have Windows 2008 SP2 an all its updates.

- I’m not sure who you should login as while installing. A domain account helps so you don’t keep getting prompted when adding domain users. I used the DEV\TFSReports.

- Make sure the reports user can login locally. http://blogs.msdn.com/ablock/archive/2008/09/18/setting-the-properties-the-log-on-as-a-service-and-allow-log-on-locally.aspx. If you don’t, SharePoint won’t work.

- Don’t add any users to SQL.

- Don’t worry about SharePoint app pool warning when configuring TFS.

- Add DEV\Developers (or people you want to create projects) to the Contributors on the Reporting Service admin site http:\\tfs2010\Reports (Note; the Reports site is not listed in IIS). This fixes the TF30224 error when a user tries to create a new project.

- If SharePoint fails, you can try to re add the extensions from Programs and Features > click Add features (not repair). http://social.msdn.microsoft.com/Forums/en-US/tfsprerelease/thread/e3f80d66-2d8a-46b1-a7da-efb61d86abbc

- You need to add domain users to a few places before creating a project: http://msdn.microsoft.com/en-us/library/dd547204(VS.100).aspx

- I didn’t manage to create the WSS site under site due to SharePoint permission issues. (I’ll update this blog when I figure it out)

- Don’t worry about the object not set (Beta 2?) error when adding a project, just re add the project to VS.

- Make sure you add Task before checking the portal/reports. http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=499387. This will get rid of the rsInvaildReportParameter for the WorkItemTypeParam after 30mins or so.

The scrum stuff looks good but its hard to tell until you use it. http://msdn.microsoft.com/en-us/library/dd997796(VS.100).aspx 

Next steps include getting a Build server and a Test Controller going. Lab management requires Hyper-V which Joe Gough is sorting out.

Cool. M

Comments welcome.

We want to be able to control the version of each of our releases. This was being done by leaving the Assembly Info to 1.0.*

Soon it was time to release and we needed greater control of the version number. On a previous project we had a Version.xml file that was stored in source control. This was “Got” on each build, read in, +1’d, updated and checked back in. It also updated a comment (RegEx replacement) in the AssemblyInfo.cs file with the new version number. We also share the Version.cs file by adding it as a link to each project (the dropdown on the Add button when adding an existing file to project)

The drawbacks to this were that you had to prefix the comment when you checked in with “*** NO_CI” and you had to manage the workspaces to get the latest. It was also to set the build number as you had to perform a check in.

This time, there was a check-in policy that prevented checking the version file back in. Instead of solving this (it looked like its was going to be a custom task) we have moved to a version file next to the Drop folder of the build server (on a network share).

Now we can have different version files on the network share and the release manager can update the version numbers as required. The revision number is still incremented by the msbuild scripts. This file is also updated once the build completed with the location of the drop. The last good drop path is used when automatically deploying.

Note we have also had problems with the difference between FileVersion and AssemblyVersion. http://support.microsoft.com/kb/556041

Branching

So that we can support new work and bug fixes, we have used TFS Source control to branch releases.

After you have created a branch, you need to create a new build type. It’s just as quick to compare the previous build type and just update the version number and path. You must then also create a \\projectbuildserver\Public\Versions\Branch_2.1.3.xml file and set the version number up. The project contains information about updating the xml file (note posted).

This means that each branch is under CI with shippable build popping out after each check in.

Note that labels are removed when the builds are removed. So if the retention policy removes them, the label is lost. To stop this, keep all builds (if you have the space) or mark a build as retain indefinitely on the completed builds window.

Once the build is completed, the version.xml file is updated with the location of the build so that a deployment process can always get the latest.

kk. Comments welcome.

We rocked up at NxtGeng to listen to Ben Hall chat about TDD. It’s nice to know we are on the same kinda wavelength. For example, Tests first, passing in dependencies, using Rhino Mocks, ASP.net MVC,  Ioc (StrutureMap), and WebUI Testing (although we use Selenium so non-developer testers can maintain the scripts). One app that we will start to use is IETester. This can emulate different versions of IE. Word.

http://www.nxtgenug.net/

http://blog.benhall.me.uk/

http://www.my-debugbar.com/wiki/IETester/HomePage

Tom and me had a trip to Themes Valley Park to look at some 2010 stuff. It turned out that this was a replacement course and they mainly talked about 2008. It wasn’t all bad though, getting out the office, etc. Here are my highlights:

Team Build Screensaver: http://teambuildscreensaver.codeplex.com/

If you have permissions, you can setup a box to show the team the state of the builds. It’s open source so I’m sure you could add sound effects when the builds change state if you wanted to.

TFS Branching Guide II advice: http://www.codeplex.com/TFSBranchingGuideII

The general idea was that you have a main branch and a dev branch. New features are developed in dev then merged into the main. once stable, main is branched to a releases folder. eg DEV –> MAIN –> Release 1.2. There is no concept of UAT or ‘Wobbly’. Only one feature is merged from DEV at a time, freezing MAIN. If you follow this kind of pattern then 2010 should be easier to use as it can show branch history and where change sets have been applied in a visual graph.

Expression Blend Sketch flow: http://www.microsoft.com/expression/products/Sketchflow_Overview.aspx

A quick demo on how to story board / prototype with a customer using WPF/Silverlight. This includes linking screens together in a workflow and sending the prototype to multiple customers so they can annotate it and provide feedback.

Generate Test Data: http://msdn.microsoft.com/en-us/library/dd193262.aspx

If you have a licence for Visual Studio for Developers then you can install the database edition. This provides database source control and can generate diff scripts between projects or live databases. It can also be used to populate a scheme with test data to help overcome data protection issues.

Process Templates: http://msdn.microsoft.com/en-us/teamsystem/aa718795.aspx

There templates that can be added to the server as required. A true agile version will be coming in 2010 including a burn down charts etc.

Server Proxy http://msdn.microsoft.com/en-us/library/ms252490(VS.80).aspx

It is possible to install a TFS Proxy Server to cache source code when working with teams on a WAN.

Other

Other info included an Eclipse add-in, reports and what they mean, code coverage, database source control, licensing/CALs including the web interface. I’ll post the sides when I get hold of them.

Thanks, m.

Really long but good guide on .net testing, focusing on WPF:

http://windowsclient.net/wpf/white-papers/wpf-app-quality-guide.aspx

This includes comments about a UIA Testing API: http://www.codeplex.com/TestApi (found via http://www.codeplex.com/wpf)

I look forward to getting some of this stuff in our nightly builds as it can be used on most win forms apps.

I like Dan’s idea of local testing every time you build but I'm really lazy… I don’t want to make the effort install NUnit and getting tripped up with the #if statements. Some people have suggested a file watcher on the test dlls and then run mstest when they change.

As luck would have it, I came across Gallio. This is like NUnitGUI but work with other test frameworks (including MSTest). It didn’t however auto run the tests when the get recompiled :( So based on you don’t ask you don’t get (http://groups.google.com/group/gallio-dev/browse_thread/thread/bd0be2deef146c98) the team added this in.

Rich has download the latest version and is using while developing to make sure no tests fail. I’m sure there are more feature including VS integration. See http://www.gallio.org/

(iMeta ppl; a version can be found in MIS)

Thanks again to Jeff Brown and Graham Hay.

Friday afternoon goolging found these links…

http://www.infoq.com/news/2009/03/VSTS-2010

http://msdn.microsoft.com/en-us/magazine/2009.01.vsts.aspx

http://www.teamsystemcafe.net/VSTS.aspx

As always, looking for the next Microsoft release of a product to play with…

I’m looking forward to the new Visual Studio 2010 (with Camario). The Lab management means that testers can spin up a new test stack (using Hyper-V or ESX) with any prerequisites already install, take a build (deployed using workflow) and start testing. They can raise defects with a full environment details attached and even a “snapshot” of the VM. They will also be able to add automated steps to their test cases using recorded UIA.

We kind of do this already... We have templates to create environments but for speed and to save space we snapshot the VM’s. Each night (or on demand from a TFS build) a msbuild script calls a VMTool that rolls back to a snapshot (that contains the prereqs). The msbuild script is also currently used to deploy the latest build to the “clean” VM box. A tester can then test the build on the VM, and raise defect in TFS. We can also run automation (currently only a web site) using Selenium RC by reading the scripts and running them as standard mstests, just line unit tests in a CI build. Other tests are standard c# tests but we hope to move this to a framework soon.

The one thing I look forward to is a complete management system that we can write a mix of manual and automated tests in one place. It would also be nice to attach the delta of the VM to the defect so the developer can see what happen when the bug was found. The bit I’m not too sure about is when the developer uses the VM diff attached to snapshot it could rollback the VM I’m currently using. I guess this can be solved by good communication. I might also start video recording tests to provide that the bug did exist and attached the compress video/link with a time of crash to the defect.

Nice to see that we are kinda on the same wave length as MS when it comes to VM testing.

http://www.4fem.info/?p=14 & http://seleniumhq.org/ 

http://www.microsoft.com/visualstudio/en-us/products/2010/default.mspx

Goal

Regression test the latest version of a product every night. (You need to know about TFS and MSBuild)

Steps

Check there is a CI TFS Build to make sure build failures are detected as early as possible.

Create a clean target VM(s) with the TFS Build User as admin. Snapshot this with a good name.

Add a Release build per product with a common drop folder location around 01:00

If you have a version file then make sure the checkin comment has ***NO_CI** in it (thanks Phil) to stop the CI kicking in.

http://team-foundation-server.blogspot.com/2008/11/check-in-comment-to-dynamically-enable.html

Add a Deploy and Test build that starts around 02:00. This builds a “Testers Solution” that contains System Tests written as unit tests (see later).

Edit the build project file to kicks off a DePLoYeR script...

Deployer Scripts

These have been written as MSBuild scripts and can be tested locally. The TFSBuild.proj has BuildStep tags (http://billwg.wordpress.com/) to provide feedback in the build steps so we can tell what the deployer is doing (because some actions take a while)...

1. Get the current “version” xml

2. Write a VM Tool that rolls back a VM to a snapshot ($/iMeta Tools/VMWare/iMeta.VMWare/iMeta.VMWare.Tools)

Note: the VM API needs a VMUser as the password is stored in plain text :(

Note: VM names and snapshots are case sensitive

3. Read in the current version xml from the build that has just been leeched

4. Copy from the drop and unzip using 7za

5. Use XmlFile.SetValue to change the config files as required

6. Ping the server to help in the log files

7. Xcopy the files to the target box (with /E /I /Y)

8. Use psExec to run stuff remotely (http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx). Eg:

$(PsToolsPath)\psexec \\$(TargetServer) -u $(UserName) -p $(Password) -i -w "$(WorkingPath)\deployments\TEST" "$(WorkingPath)\deployments\TEST\AutoInstaller.bat"

You may have to pipe the results to a text file although this isn’t ideal.

(gotcha : make sure you run the exec on the build machine as the TFS build service as the first time you run it you must agree to the licence)

9. Use cacls to change permissions of folders

10. Use psservice to start services

11. Make sure that IIS is using .net 2.0 (Psexec aspnet_regiis -s W3SVC/1/ROOT)

12. Clean up temp files.

Note, you can use Robocopy but only if the build server is 2008 or has the Resource Kit. It also returns “bad” exit codes that you will need to ignore in your scripts.

When testing the script, use the console view, not RDP to see the results of psexec.

Testers Solution

Once all that has completed we can run our unit tests on the fresh box and the result be linked to the nightly build reports.

The tests can be written using Selenium and results saved to a database. If you have the tester edition This should provide the tester with a green/red light in the morning.

Peace out - Mike Tester Blake

Overview

Problem: How can we test our web site? One option is to use http://seleniumhq.org/.

The Firefox plug-in “Selenium IDE” can be used to record, edit and manage scripts. (See their site for details)

We have then taken this further by creating a c# test project and adding the html test scripts into source control. On a nightly/on demand bases we run selenium tests, as if they we’re ms unit tests through TFS.

This process opens the html test suites/cases and sends the commands to a selected Selenium RC server, recording the results in a SQL database.

A Dynamic Data ASP.Net site lets people view the progress of a TestRun with screenshots when an action fails.

Components

Selenium IDE

(Requires Firefox) The Selenium IDE can be used to record Test Cases and link them via a Test Suite. By default, they are stored as HTML. This is the main tool the tester will use to edit, manage and run the test cases.

Selenium RC

The “Remote Control” Java application that is used to interact with a browser based on commands it receives. This should be run on the target OS in a VM.

TestEase

TestEase uses the test cases in a Test Project and sends the commands to the selenium RC. Results are stored in a database.

Build integration

As the tests run as standard C# they can be added as unit tests and then run each night (after a deployment).

Test Ease Site / Reports

A simple web site can be used to display the results from the database. It is even possible to see screenshot when an action fails. Reporting services can be used to pull statistics from the database.

Selenium Grid

If the product under tests needs to support multiple browsers and operating systems, for example a public facing web site, Selenium Grid can be used to run the tests on multiple environments. (this has not been done for our product)

Problems

Speed changes the results. Depending on the web site, depending on if a page comes back or if the test timeout. This might be an issue with an ASP.net site because the first time a new deployment is accessed it JIT compiles the code. Workarounds include: 1, increasing the selenium timeout. 2, “Hitting” the site to force the Jitting before running the tests. 3, Force the asp compiler to JIT the site as part of a deployment.

All tests can break if the control names change (unlike UIA accessibility). By default, selenium will normally use the id of the html control as its reference. If the site layout changes this could break the tests. Workarounds:

Testers need to know about VS and source control including merging. (minor issue)

Screenshot don’t work when using Remote desktop. The VM host view must be used instead.

Selenium RC isn’t a service. Workaround: Auto login/start the RC in case the server is reset.

Firefox isn’t currently working on the RC box due to a profile issue.

Some commands in the IDE are not available in the RC (e.g. echo, pause). TestEase manages these manually.

Ajax requires “pauses” that might not be that robust

Popups can be an issue if the windows aren’t name. The RC supports multi-window that can help when testing a multiple window web site

Future Improvements

Requirement mapping. It would be nice to see the requirements for the system and how many tests cover them. It is thought that an echo or new selenium command can be used to show what requirement the test is covering.

Better configuration management. Some more work needs to be done to the configuration of the c# project. For example, pushing the settings to xml files or to build properties. This will be useful if the tester wants to run the tests on a different set of servers.

Remove the database dependency so that the tests can be run ad-hoc.

Add data driven testing. This could be achieved by using template test files. The templates could contain tokens that are replaced by the TestEase engine. The process could generate test files that can be run by the Selenium IDE with the tokens replaced.

Alternatives

Microsoft Visual Studio Web Testing,

Uia (for WinForms)

VSTS 2010 - http://www.microsoft.com/visualstudio/2010/overview.mspx