Daniel Gray's Blog

A colleague ran into a very interesting issue whilst working on his teams release build script. He was trying to get the custom built version number setup before a clean of the old source had taken place (so the version number could become the build number). To achieve this he was calling a sub script that (for use by a different target) setup a list of all the AssemblyInfo.cs files that needed to be updated with the version number. Later on in the build (after the clean / get) he reran this same script (for a different target) to write the version numbers into the AssembyInfo file. At this point the script would fail as one of the files wasn't there (a remnant of a previous packaging process). Much head scratching ensued as we tried to figure out how the heck an item list in a different script, invoked with MSBuild was populated from the first time it ran...

A theory was constructed that something was being cached, or data was moving in a way we didn't expect. Either the MSBuild task held onto scripts, or the item list was passed back to the calling script. The later theory was quickly discounted, which just left us with the MSBuild task caching script instances.

After a bit of digging around in the Microsoft.Build.Tasks assembly I managed to reach down into where it was actually executing MSBuild... not (as I'd thought) by spinning up a seperate process, but by calling back into it's own build engine. Fair enough, saves having to start a new process (although it explained some unusual file locking we'd seen as well). Further digging into the build engine found the following:

   1:  Project project = null;

   2:  Project existingProject = null;

   3:  Project project3 = (Project) this.projectsLoadedByHost[info.FullName];

   4:  if (project3 != null)

   5:  {

   6:     existingProject = project3;

   7:     if (project3.GlobalProperties.IsEquivalent(globalProperties))

   8:     {

   9:        project = project3;

  10:     }

  11:  }

Scripts are held firstly by the name of the script, and secondly BY THE PROPERTIES THEY ARE PASSED...

So recalling the same script with a different target but the same properties doesn't invoke a new instance of the script. No, the old one is reused and since Items are populated only when the script is first invoked the list of AssemblyInfo.cs files was as it was BEFORE the clean / get, not (as we wanted) afterwards.

In this instance the simple fix was to add an extra property to each of the two calls to ensure that the property lists were different ... and voila! Problem solved.

And that STD is a Secret Testing Defect.

If you've not seen PEX yet it's a tool from Microsoft Research that automates white box testing. That is to say you write your code, point PEX at it and away it goes, generating test values for your method that will exercise your code to the fullest. These test values are then written out as full unit tests and can be included in your regular testing regime. But how does PEX find these values? Extispicy? Dousing rods? No, it runs your code... fair enough really, if you've not used a formal specification language then the only definition of your code available to PEX is the code itself. Reread that, the only definition of your code available to PEX is the code itself. So what if your code is wrong (and trust me it will be)? PEX is writing tests based on what your code does, or appears to do. So if there's a mistake in your code, there'll be a mistake in what PEX expects. Even worse, PEX might make assumptions about your code on the basis of how it understands it.

Take the following single line method:

   1:          public static decimal ConvertCelsiusToFahrenheit(int celsius)

   2:          {

   3:              return Round(((9 / 5) * celsius) + 32);

   4:          }

If run PEX explorations against this it generates us the following set of values (I say set, I mean value):

Fair enough really, it's one line of code, and a linear function, and it's right. Zero degrees Celsius does convert to 32 degrees Fahrenheit. That set of inputs doesn't break our code. But there is an input that does, well more than one input really, in fact any value of celsius other than 0 gives an incorrect calculation. PEX has picked the first value it can use, made an assumption about how our code works and in doing so missed the one error that's crept into the code.

Were I writing the tests I'd have written 3 tests with the following values of Celsius -1, 0 and 1. That would've spotted the mistake (have you spotted it yet? I'll post it at the end) and given me much more certainty about the correctness of my method.

I can see some of the value in using tools like PEX, but I'd much rather that people wrote their own damned tests. You know how it's supposed to work (you wrote it after all), so you should be able to test it at a unit level, and if not you're doing something wrong.

The mistake in the method was in the (9 / 5), replacing it with (9m / 5m) solves our problem.

I was intrigued to read Hadis' post on the exception testing capabilities in xUnit. It's often struck me that the use of attributes on test methods is a bit blunt given the scope for exceptions to be thrown from different locations in test code. After reading how xUnit achieved the testing of expected exceptions I was inspired to implement this in a way that NUnit (and MSTest) could also use. The implementation is really rather simple:

   1:  using System;

   2:  using System.Collections.Generic;

   3:  using System.Linq;

   4:  using System.Text;

   5:   

   6:  namespace UnitTestExtensions

   7:  {

   8:      public delegate void ThrownDelegate();

   9:   

  10:      [global::System.Serializable]

  11:      public class ExceptionNotThrownException : Exception

  12:      {

  13:          public ExceptionNotThrownException() { }

  14:          public ExceptionNotThrownException(string message) : base(message) { }

  15:          public ExceptionNotThrownException(string message, Exception inner) : base(message, inner) { }

  16:          protected ExceptionNotThrownException(

  17:            System.Runtime.Serialization.SerializationInfo info,

  18:            System.Runtime.Serialization.StreamingContext context)

  19:              : base(info, context) { }

  20:      }

  21:   

  22:      public static class AssertExtensions

  23:      {

  24:          public static T Thrown<T>(ThrownDelegate d) where T : Exception

  25:          {

  26:              try

  27:              {

  28:                  d.Invoke();

  29:                  throw new ExceptionNotThrownException("Exception of type " + typeof(T) + " not thrown");

  30:              }

  31:              catch (T e)

  32:              {

  33:                  return e;

  34:              }

  35:          }

  36:      }

  37:  }

And a couple of simple tests demonstrate it working:

   1:  using System;

   2:  using System.Collections.Generic;

   3:  using System.Linq;

   4:  using System.Text;

   5:  using NUnit.Framework;

   6:  using UnitTestExtensions;

   7:   

   8:  namespace Tests

   9:  {

  10:      [TestFixture]

  11:      public class NUnitAssertExtensionsTests

  12:      {

  13:          [Test]

  14:          public void Thrown_Catches_And_Returns_Expected_Exception()

  15:          {

  16:              TestClass t = new TestClass();

  17:              ArgumentNullException a = AssertExtensions.Thrown<ArgumentNullException>(()=>t.Chuck());

  18:              Assert.AreEqual("Name", a.ParamName);

  19:          }

  20:   

  21:          [Test]

  22:          public void Thrown_Throws_ExceptionNotThrownException_When_Exception_Not_Thrown()

  23:          {

  24:              TestClass t = new TestClass();

  25:              AssertExtensions.Thrown<ExceptionNotThrownException>(()=>

  26:                  AssertExtensions.Thrown<ArgumentNullException>(() => t.NoChuck()));

  27:          }

  28:      }

  29:   

  30:      public class TestClass

  31:      {

  32:          public void Chuck()

  33:          {

  34:              throw new ArgumentNullException("Name");

  35:          }

  36:   

  37:          public void NoChuck()

  38:          {

  39:   

  40:          }

  41:      }

  42:  }

I've used the ExceptionNotThrownException rather than a call to Assert.Fail as I'd like this to work in any testing framework.

If you read Hadis' post on  ASP.NET MVCD you'll have noticed the rather unpleasant return to the bad old days of ASP code embedding in views. Even when we're following the MVC pattern closely you're still going to find a mess of C# syntax littering your beautifully crafted HTML. I don't there's much worse than having to open a script block just to finish a loop (the <% } %> alluded to in the title).

Fortunately the designers of the new MVC framework had the foresight to allow the use of different ViewEngines that allow HTML / code rendering in with far more pleasant syntax. Take, for example, Spark (my personal favourite). The entire ASP.NET rendering engine is ditched in Spark, and we're given a whole new syntax that melds control of rendering into your HTML seamlessly. Taking Hadis' original example of looping around a list we go from

   1:  <%foreach (Customer customer in ViewData.Model)

   2:  {%>

   3:      <tr>

   4:          <td style="width: 98px">

   5:              <%=Html.ActionLink(customer.Firstname, "Edit", new { id = customer.Id}) %>

   6:               

   7:          </td>

   8:          <td style="width: 112px">

   9:              <%=customer.Lastname %>

  10:               

  11:          </td>

  12:      </tr>

  13:  <% } %>

to

   1:  <for each="Customer customer in Customers">

   2:      <tr>

   3:          <td style="width: 98px">

   4:              ${Html.ActionLink(customer.Firstname, "Edit", new { id = customer.Id})}

   5:               

   6:          </td>

   7:          <td style="width: 112px">

   8:              ${customer.Lastname}

   9:               

  10:          </td>

  11:      </tr>    

  12:  </for>

There's a lot more to Spark than just this, conditional attributes on existing HTML elements (remember it goes through a whole different parser... they're doing some real magic under the hood), which enables statements like:

   1:  <p if='!user.IsLoggedIn()'>Login form</p>

   2:  <p elseif='user.HasRole(RoleType.Administrator)'>Hello - admin</p>

   3:  <p elseif='user.HasRole(RoleType.Registered)'>Hello - registered user</p>

or how about an each attribute on HTML elements so we don't need to use the <for/> element? Our updated HTML becomes:

   1:  <tr each="Customer customer in Customers">

   2:      <td style="width: 98px">

   3:          ${Html.ActionLink(customer.Firstname, "Edit", new { id = customer.Id})}

   4:           

   5:      </td>

   6:      <td style="width: 112px">

   7:          ${customer.Lastname}

   8:           

   9:      </td>

  10:  </tr>    

The one downside is that you'll lose the existing ASP.NET capabilities when it comes to master pages (they have their own version), user controls (they have something similar but not exactly the same) and built in ASP.NET controls. However, since we're coming at how we build our web applications from a different angle by using MVC, it's not necessarily a bad thing. It does mean that a lot of stuff that ASP.NET controls just did for us we need to relearn, however the state of Javascript extension packages has moved on considerably in the past few years (consider JQuery and their recent success in getting included in future release of ASP.NET).

If you're having a play with MVC give Spark a try, or one of the many other ViewEngines that are out there.

Which do you consider better as a URL for your blog post:

http://blogs.imeta.co.uk/dgray/archive/2008/10/01/409.aspx

or

http://blogs.imeta.co.uk/dgray/archive/2008/10/10/preventing-tfs-from-creating-work-items-when-a-build-fails.aspx

I know which I prefer, and I'm sure that search engines will give you a better ranking for the later. To set this up under Subtext (we're using version 1.9.5.177) simply go to the Options Tab, then under Preferences find the "Auto-Generate Friendly URL" checkbox. All future posts will come out with a much friendlier URL.

A review of work items today brought up hundreds of previously unnoticed work items for build failures. Our process doesn't currently handle this, so for the time being we've decided to simply disable the feature. Surprisingly a quick Google revealed nothing, so here's the very straightforward answer. Just add

<!-- SkipWorkItemCreation - Don't create a work item if the build fails -->
<SkipWorkItemCreation>true</SkipWorkItemCreation>

To your TFSBuild.proj file in one of the property groups and work item creation is surpressed - simple as just!

Just had an interesting conversation with a couple of colleagues about why I'm trying to get interoperability in my tests between MSTest and NUnit. I thought I'd covered this in an earlier post, but if not here's another summary.

The current implementation of MSTest inside of Visual Studio doesn't seem to fit (at least my interpretation of) how Test Driven Development should fit into your daily development cycle. Developing with unit tests needs to have a constant rhythm of code, test, code, test. Ideally the test part should be transparent, at least not seriously interrupting the flow of your coding. MSTest fails on this front as far as I'm concerned, things that are missing are:

• When I rebuild my test assembly - rerun the last set of tests I ran
• When you run my tests, do it out of the way - I don't want to know that you've run them until you find something that fails
• If I've built my tests to run fast (as they should), I want you to run them fast

However, MSTest wins hands down over all other testing tools thanks to its' TFS integration. The publishing of results back into TFS and subsequent analysis in the datacube for results and code coverage is just superb.

This leaves me with a difficult problem - I want to use something like NUnit whilst coding, but also to use MSTest whilst building centrally. Hence why I'm trying to have my cake and eat it with the dual test framework idea.

I've not had the opportunity to use it on anything of any scale or importance yet, but I've confirmed the theory with a few simple projects I've got running in my home TFS Server. The NUnit GUI sits quietly in the background whilst I code, monitoring changes to the test assemblies, happily re-running the tests when its rebuilt, only making itself known when there's a failure. My TeamBuilds then switch over to running MSTest and I get the flow of data back into TFS that I was expecting.

As I get to use this pattern more often I'll keep you all updated on my progress (fingers crossed I can keep using it).

I swear Hadi has some kind of blog post idea stealing telepathy power, that's the second time in 4 days he's pre-empted a post I was about to make with one of his own. I'd been prompted to write on the subject with my recent acquisition of a third monitor, the resurrection of a cheap PC I bought in Tesco for £150, Synergy, Media Monkey and a new 1TB Western Digital external drive. All of this has culminated in finally having my full music library with me at work (and not stealing MP3 decoding CPU cycles from my development machine), something which I'm really pleased about.

Like Hadi the music I listen to is split depending on the type of work I'm doing. For mechanical, heavy-coding sessions I'll listen to something that follows musically, so mainly Industrial bands like Skinny Puppy, KMFDM, Nine Inch Nails, :Wumpscut:, Angelspit or Razed in Black. If I'm working on design (especially anything with any layers of abstraction) then again the music follows this, Juno Reactor, Aphex Twin, Coil, Godspeed You Black Emperor!. If I'm writing something (blogs, training "bits", wikis, etc.) the music will tend to be quite wordy as well, Sneaker Pimps, Suede, Sugarcubes, Therapy?, Tori Amos.

Although the one overriding factor in all my music choices is the mood I'm in, music (for me at least) is a reflector and a director of emotion.

"If ever I would stop thinking about music and politics, I would tell you that music is the expression of emotion, whilst politics is merely the decoy of perception"

- Michael Franti, Music & Politics

When I made my previous post about testing with two frameworks I'd inadvertently made a good decision to write the tests using the MS Test attributes on my classes, and then redirected them to the appropriate NUnit attributes, rather than the other way around.

Why was this a good decision? Because Microsoft have more attributes in common use than NUnit. Here's the comparative list:

That which doesn't hurt us...

The nice thing about the extra attributes found in MSTest is that we can either ignore them or circumvent them. For example DeploymentItem is only used by MSTest because of the way they move files into a test directory prior to running the tests, no such functionality is required in NUnit as it quite happily runs inside your binary output folders. The same can be said of all the meta data attributes (Owner, Priority, etc) which don't have a functional equivalent in NUnit, they're useful to decorate our tests inside Team System, but for our desktop testing they're largely irrelevant.

Unfortunately there's nothing we can do about the AssemblyInitialize, AssemblyCleanup, DataSource and HostType, if you're going down the dual testing framework route your going to simply have to not use them.

How to include a MSTest meta data attribute and not make NUnit hurl chunks

Quite straightforward really, we define ourselves a placeholder attribute that has constructors that match the signatures of the extra attributes and then redirect to this attribute in our using block.

   1:  namespace TestHelpers

   2:  {

   3:     [AttributeUsage(AttributeTargets.All)]

   4:     public class NullAttribute : Attribute

   5:     {

   6:        public NullAttribute(string v) { }

   7:     }

   8:  }

   1:  #if NUNIT   

   2:  using NUnit.Framework;

   3:  using TestClass = NUnit.Framework.TestFixtureAttribute;

   4:  using TestMethod = NUnit.Framework.TestAttribute;

   5:  using TestInitialize = NUnit.Framework.SetUpAttribute;

   6:  using TestCleanup = NUnit.Framework.TearDownAttribute;

   7:  using ClassInitialize = NUnit.Framework.TestFixtureSetUpAttribute;

   8:  using ClassCleanup = NUnit.Framework.TestFixtureTearDownAttribute;

   9:  using DeploymentItem = TestHelpers.NullAttribute;

  10:  #else

  11:  using Microsoft.VisualStudio.TestTools.UnitTesting;

  12:  #endif

  13:   

  14:  using System;

  15:  using System.Collections.Generic;

  16:  using System.Linq;

  17:  using System.Text;

  18:  using Library;

  19:   

  20:  namespace UnitTests

  21:  {

  22:     [TestClass]

  23:     public class TemperatureConverterTests

  24:     {

  25:        [TestMethod]

  26:        [DeploymentItem("ConversionChart.xml")]

  27:        public void FahrenheitToCelsius_0Degress_Fahrenheit()

  28:        {

  29:           Assert.AreEqual(-18d,Math.Round(_converter.FahrenheitToCelsius(0),0));

  30:        }

  31:     }

  32:  }

Lather, rinse, repeat the "using DeploymentItem = TestHelpers.NullAttribute;" for all the additional attributes you're using and everything should compile, and NUnit will happily ignore the strange attributes it knows absolutely nothing about.

Final safety tip

Put your conditional using block at the TOP of your list of usings. Doing this prevents the IDE from inserting new usings (inserted using resolve) into the part of the IF block you're currently using. It can get very confusing trying to figure out why your tests don't compile in release, but do in debug.

Our upgrade from TFS2005 to TFS2008 is coming up this weekend, and there have been a number of important improvements in the TFS build system. Microsoft have been listening to some of the criticism over the way 2005 worked, and seemed to have addressed the majority of them. Here's an (non-exhaustive) list of new features:

• Separation of build types (including CI policy, retention policy) from the build project (the .proj file)
• Inclusion of automated builds
• Unit testing without the VSMDI
• Build retention policies
• Build queuing
• Improved control over builds in the VS2008 GUI

Whilst none of these new features are hugely exciting, they represent the kind of fine grained polish that's been much needed on the build front for a while now, and should make working with builds in TFS a little easier.

Buck Hodges has an excellent collection of links (note: these cover Beta 2, so some of the detail may be incorrect, but the overall thrust is accurate) that go into far more detail then I've got the time to do here. Well worth a read if you're involved in the build process on your team, or you're just interested in how things have moved along.