Steve Strong's Blog

TekPub's Mastering LINQ Challenge

Steve Strong — Sat, 09 Jan 2010 21:06:31 GMT

Just saw a good little LINQ quiz on Justin Etheredge's blog - I saw it way too late to get the TekPub subscription, but thought it looked fun anyhow. Here's my solution:

   static void Main(string[] args)

   {

     int max = 100;



     var primes = Enumerable.Range(1, max)

      .Where(i => i > 1)

      .Aggregate(Enumerable.Range(2, max - 1).ToArray(), (sieve, i) =>

      {

        if ((i > Math.Sqrt(max)) || (sieve[i - 2] == 0)) return sieve;

        for (int m = 2; m <= max / i; m++)

          sieve[i * m - 2] = 0;

        return sieve;

      })

      .Where(n => n != 0)

      ;



     foreach (var p in primes)

     {

       Console.WriteLine(p);

     }



   }

Arguably it's not pure LINQ; there's more going on inside the aggregate lambda than I'd like, but it does the job and for those into this sort of thing you should recognise it as an implementation of the Sieve of Eratosthenes.

Linq

Using the new Linq to NH Provider and migrating from the old one

Steve Strong — Wed, 16 Dec 2009 22:56:51 GMT

Using the new Linq provider is pretty simple. It all hangs of a Query() extension method on ISession, so you can do things like the following:

from c in session.Query<Customer>() select c

In my tests, I've tended to wrap the session.Query() call behind a simple facade, along the lines of:

  public class Northwind

  {

   private readonly ISession _session;



   public Northwind(ISession session)

   {

   _session = session;

   }



   public IQueryable<Customer> Customers   { get { return _session.Query<Customer>(); }

  }

Of course, that's entirely optional, but I find the resulting code easier to read:

from c in db.Customers select c

Once you know how to hook into the session (which as you can see is pretty simple), the rest is just straightforward Linq code, and entirely up to you! Right now I'm not exposing any extension points, but they'll be coming soon (plus another post to describe how to use them).

The version 1 provider used an ISession extension method call Linq() to provide its hook. I purposefully used a different name, since there's no reason at all why you can't use both providers within the same project or, indeed, within the same session. So that gives a couple of migration options for folk that want to move to the new provider:

Leave all your current queries using .Linq(), and start using .Query() for new ones.
Start changing existing queries from .Linq() to .Query(), just by changing them or by a simple search & replace. The rest of the expression will (hopefully!) just work.
Drop your reference to the original Linq provider assembly, and create your own extension method:

public static IQueryable<T> Linq<T>(this ISession session) { return session.Query<T>(); }

This lets you switch to the new provider without changing a line of your code - and if you find it all goes to hell, you just re-add the V1 reference and comment out your extension method. Should work like a treat.

Other than that, I don't think there's much to tell - usage really should be pretty simple. Oh, one thing that springs to mind - although you can use the V1 provider and the new provider within the same project (or session), don't try to compose queries from them together; that's really going to do weird stuff!

Linq, NHibernate

Linq to NHibernate Progress Report - A Christmas Gift?

Steve Strong — Wed, 16 Dec 2009 22:32:41 GMT

Time for another progress report, and this one's a biggie :)

Barring a couple of pretty minor things that won't take much fixing, all the original Linq tests have now been ported over to the new provider and are all passing. That means the new provider is now (from the perspective of the tests, at least) in better shape that the version 1 provider. It can do everything the original provider could, plus a whole bunch more. A couple of example queries that now work just fine are:

  from e in db.Employees

  from et in e.Territories

  where e.Address.City == "Seattle"

  select new {e.FirstName, e.LastName, et.Region.Description};

  from c in db.Customers

  join o in db.Orders on c.CustomerId equals o.Customer.CustomerId into orders

  select new {c.ContactName, OrderCount = orders.Average(x => x.Freight)};

  from o in db.Orders

  from p in db.Products

  join d in db.OrderLines on new {o.OrderId, p.ProductId} equals new {d.Order.OrderId, d.Product.ProductId}

   into details

  from d in details

  select new {o.OrderId, p.ProductId, d.UnitPrice}

  from c in db.Customers

  join o in db.Orders on c.CustomerId equals o.Customer.CustomerId

  group o by c into x

  select new { CustomerName = x.Key.ContactName, Order = x }

(ignore whether those queries make any real sense, it's just the form of them that matters)

So, more importantly, what doesn't work? Well, out of the tests that we've currently got, not a huge amount. Some important areas that are missing are:

Nested selects to produce hierarchical output. I've got some prototype code for doing this, so it will be supported at some point but isn't there right now. Of course, since NH already understands relationships, nested selects are far less important than they are for something like Linq to SQL.
Group joins that produce hierarchical output - these essentially boil down to the same code as the nested select case, so support for these will probably come at around the same time. Group joins that don't introduce a hierarchy should work just fine.
Set operations, such as Union and Intersect. Union you can obviously do yourself in client code with no particular overhead. Intersect would really be better in the provider :)
Left outer join style queries, such as:

from e in db.Employees join o in db.Orders on e equals o.Employee into ords from o in ords.DefaultIfEmpty() select new {e.FirstName, e.LastName, Order = o}; This is a fairly widely used construct, so is close to top of the list for future support
Let expressions, such as:

from c in db.Customers join o in db.Orders on c.CustomerId equals o.Customer.CustomerId into ords let z = c.Address.City + c.Address.Country from o in ords select new {c.ContactName, o.OrderId, z}; Again, this is quite high on the TODO list.
Support for custom functions. This is actually fully implemented internally, but I just want to review the API usage before I tell you all how to use it - it probably needs a little cleanup from it's current state, but I don't anticipate any major changes.

I've also got a number of TODOs, but mainly cleanup rather than functional, plus I need to work through the error handling to ensure that any queries that are passed in that the provider can't handle are rejected gracefully rather than just barfing (which is the likely case right now).

The only gotcha that I know of is a query that runs just fine but doesn't necessarily return the correct number of results. Specifically, queries like this:

  from user in db.Users

  select new

  {

    user.Name,

    RoleName = user.Role.Name

  };

Right now, this generates a join to the Role table, so any users that don't have a role are not returned. This differs from Linq to SQL where a left join is generated, giving a null RoleName for any users without a role. I believe the Linq to SQL implementation to be correct - the above query doesn't explicitly have any form of filtering (where clause, join clause etc), so you should get back all the users in the database. This one is top of the list, and hopefully will be fixed soon (it's not hard, I've just run out of time!).

In summary, I'm getting pretty happy with the state of the provider and think that it's now ready for general usage - although there are still some important query forms to support, I think there's sufficient there now to do useful work. If you've got good test coverage, then I'd even be happy for it to go live. If you don't have good coverage, then don't come crying to me :)

Of course, this is all in the trunk, so anyone wanting to play either needs to get the trunk source and build it, or take the much easier option of having Horn do the work. Horn builds the trunk on a daily basis, so look for a package built after around 2300GMT on the 16/12/2009 (the package URL on Horn has the datetime stamp in it, so it's pretty easy to spot).

I'm on holiday for the next couple of weeks, and will only have intermittent internet access. However, ping me either by email or twitter with comments / bugs / suggestions, and I'll do my best to reply. Normal service (whatever that is) will return around Jan 4th.

Linq, NHibernate

Similarities between directing a film & running a software team?

Steve Strong — Mon, 23 Nov 2009 13:35:28 GMT

The other day I was listening to an interview with film director Joseph Strick, and was particularly struck by two comments he made that, to me at least, resonated with my thoughts about running skilled software teams:

Always let them do something first without telling them what to do, because they may do something fantastic, and if you give them direction before you hear them you may be crushing something that could have been much better.

and

Nations advance in proportion to their freedom... the more free the culture, the more people are encouraged to think, and thinking means leaving them alone to think.

I'm a big proponent of getting a team of highly intelligent, motivated individuals and essentially letting them get on with it. The level to which creativity is crushed through any type of micromanagement with these type of people is incredible, and I quite like hearing similar sentiments from such a different sphere of life. I'd be interested in hearing other views on this...

Teams, Software Development

Linq to NHibernate Progress Report

Steve Strong — Thu, 22 Oct 2009 20:55:05 GMT

Another sprint finished (although with the amount of other stuff I've been doing lately, a "crawl" may be a better description), so time for brief update.

This one doesn't really add any visible functional improvements, rather I've been concentrating on improving the interaction between the Linq provider and the NHibernate core. So what has changed? Firstly, the Linq expressions now tie into the NHibernate HQL Query cache correctly. So running a query for the second (and subsequent) time results in a cache hit for the NH query plan, resulting in considerably less processing taking place prior to us actually hitting the database.

Secondly, and vital for the first to be useful, automatic parameterisation of queries now takes place. By this, I mean that a query such as:

from c in Customers where c.Address.City = "London"

now generates HQL of:

select c from Customers c where c.Address.City = :p1

which in turn generates parameterised SQL that gets cached within the database (at least, it does for SQL Server)

Hence, a subsequent query of

from c in Customers where c.Address.City = "Madrid"

will result in cache hits right through the chain, and so should give great performance. Back to getting more queries working tomorrow.

Linq, NHibernate

VS2010 "Add Reference" Dialog Box

Steve Strong — Thu, 22 Oct 2009 09:06:36 GMT

The Add Reference dialog box has to be one of the most hated things of Visual Studio; certainly, it seems to get more than its fair share of tweets along the lines of "Agghh, hit 'Add Reference' by mistake. I'm off for a coffee"!

The reason for the hatred was that the dialog's default tab was the full list of .Net assemblies available on your machine. That's quite a bunch of metadata for VS to go and read and unfortunately it blocked the UI whilst doing so. What made this doubly frustrating, at least for me, is that I rarely wanted that tab - I'm normally after either the Project or Browse tabs.

Well, this week has seen the release of VS2010 Beta 2, and I think there will be much rejoicing. Finally, Add Reference has had some good loving. Here are some of its better bits:

The default tab when starting VS is now the Projects tab. As a result, it loads pretty much instantly.
It now remembers which tab you were on last for this VS session.
It now remembers the size of the dialog and any adjustments made to column sizes within each tab, even across restarts of VS.
It now has a new "Recent" tab. You can guess what that's for. And the contents survive restarts.

It's not quite perfect (for example, here are some issues that relate to the .Net tab), but it's a vast improvement.

UPDATE: It would appear that I'm blind, since the Recent tab has apparently been around since VS2005. And I've just check on my VS2008 install and, sure enough, there it is. I'm now doubting everything I thought I knew :)

VS2010

Sporadic Digest of Interesting Stuff

Steve Strong — Tue, 20 Oct 2009 15:11:56 GMT

There's a ton of stuff for this digest, which is hardly a surprise considering it's now around 6 months since the last! So, in no particular order:

Here's a good video highlighting some of the key aspects of HTML 5. It's around an hour long and, particularly if you're an HTML5 noob, well worth the watch.
And whilst on the topic of HTML5, here's some info on Drag 'n' Drop support. And in the interests of impartiality, here's a great rant on some problems with it. Note that I've not tried this API yet, so I've no opinion on its quality.
Still web-based, here's an interesting gotcha on how JavaScript manages zeros. It makes perfect sense once you know, but I could see it being a great source of bugs for the uninformed.
More web, and this time very out of date, and that's a First Look at ASP.NET MVC. Hey - there might be someone out there who's not seen it!
And another old web thing, but again possibly of use to someone, and that's the "new" Website Spark program from MS that provides (amongst some other things) a pile of licences to qualifying companies.
Here's a funky collaborative web-based text editor - I've used it with a couple of colleagues, and it does exactly what it says on the tin. Of course, now the Google Wave invites are coming out, perhaps it's just a relic of yesteryear whose only use is to provide another "when I were a kid, it were all fields" story to bore your grandkids with :)
Is it time to abandon online banking? I'm guessing not yet, but those quantum computers do seem to be making slow but steady progress.
If you follow the OSS world at all, you may have heard the odd rumour about some new kid on the SCM block with the catchy name of Git. For those that don't know, Git's not actually that new - he was sired by none other than Mr. Torvalds in 2005; I particularly like Linus' first design criteria, as described on Git's Wikipedia page. Ayende has been playing with Git for a while, and a couple of interesting posts can be found here & here. For those that want to learn more, there's a free online book available here.
And whilst talking OSS (you could be fooled into thinking that these posts have thought applied to them!), here's an intro and the first post on How to Contribute to OSS projects by Krzysztof Koźmic. Now in his intro, he promises three posts but I only see one to date so @kkozmic, if you're listening, get typing :)
More from Ayende, this time on Soft Deletes vs Append Only models. Getting stuff like this correct (whatever that means) is critical since the ramifications tend to ripple throughout the codebase, making it a relatively hard decision to change in the future. Also be sure to read the comments, since they link to a couple of other good sources of information, particularly this presentation from Greg Young and this one from Udi Dahan.
For anyone doubting the ability of OSS projects to provide support, here's an awesome thread on the responsiveness of a (good) oss project. Of course, this might seem an odd thing to blog about since my company offers commercial NHibernate support, but I don't see it as a conflict - although the OSS support for a lot of projects (NHibernate included) is fantastic, some companies simply need more. In particular, they need someone that they can shout at, and we're happy to provide that service :)
On a completely different topic, here why you should test updates from your platform vendors before pushing them across all your machine. I didn't, and my iPhone suffered from the described problems for a number of weeks before the fixes were released :(
Anyone fancy transferring a Blu-Ray movie in 30 seconds? Then perhaps the Light Peak optical interconnect being developed by Intel is what you need. Of course, what devices you can stick at the end of the interconnect to handle that sort of bandwidth is another question :)
Finally, and this one is up to date, I'm sure you've all heard that VS2010 has just gone to Beta 2, with public downloads available on Wednesday. To tie in with this, there's a great series on the history of Visual Studio from Rico Mariani which is well worth a read. Plus, John Robbins has done a set of posts of various aspects of VS2010:
- Debugger In-Depth First Look
- IntelliTrace In-Depth First Look
- Code Analysis In-Depth First Look
- Sampling and Instrumentation Profiling In Depth First Look
- Concurrency Resource Profiling In Depth First Look

Interesting Stuff

Linq to NHibernate Progress Report

Steve Strong — Fri, 25 Sep 2009 21:18:42 GMT

Just done another commit to the trunk with a bit more functionality working. Here's a couple of tests that now pass:

var q = from c in db.Customers join o in db.Orders on c.CustomerId equals o.Customer.CustomerId select new { c.ContactName, o.OrderId };

var q = from c in db.Customers join o in db.Orders on c.CustomerId equals o.Customer.CustomerId into ords join e in db.Employees on c.Address.City equals e.Address.City into emps select new {c.ContactName, ords = ords.Count(), emps = emps.Count()};

var q = from c in db.Customers join o in db.Orders on c.CustomerId equals o.Customer.CustomerId into orders select new {c.ContactName, OrderCount = orders.Average(x => x.Freight)};

As before, I'm just working to get the tests passing and it currently wouldn't be hard to get variations of these to fail; for example, I don't yet handle joins on compound keys. However, also as before, I'm getting more confident with the structure and things seem to be moving along nicely.
Next steps are to look at parameterization and polymorphism - the Linq side of things actually does pretty much everything it does here, the changes I need to do are within the HQL parsing subsystem. They shouldn't be hard, so I'm hoping to get them nailed quite quickly. Oh yeah, also need to get some Asserts into the tests. Right now, they are a pretty fragile safety net :) I know how to do this now in a fairly simple way, so this should also be simple.

Linq, NHibernate

Professional NHibernate Support

Steve Strong — Thu, 17 Sep 2009 11:59:52 GMT

As most of you probably know, iMeta has been actively participating in the NHibernate project for most of this year, firstly with porting the AST parser over from Hibernate and now with active development on a new Linq provider. Around early May, we started discussing ways that we could derive some commercial benefit from our involvement in a way that would provide benefit to the community and income for iMeta. As a result we are pursuing a number of options, one of which is to offer a support package.

Due to time constraints (we have paying customers, and they have a nasty habit of taking priority!), we hadn't launched this yet, but following Ayende's announcement a few days ago now feels like an appropriate time to do so. We have had a good discussion with Ayende about what we're doing, and I'm happy to say that he fully supports us. A bit of competition is, after all, a good thing for both customers and NHibernate. Although we are slightly gutted that he beat us to it :)

Fortunately, we have taken a different approach to Ayende, so this does offer the community a real choice about how they best support their NHibernate-based developments. See here for details on the program, pricing information, how to sign up etc.

As well as announcing our package, I'm also delighted to say that it comes with the backing and support of Fabio Maulo - although iMeta has a great deal of NHibernate experience, we're the first to admit that we don't know it all and having Fabio behind us to assist with the thornier problems is a great asset. Plus it's also great to see someone who has dedicated so much of his own time to the project getting something back in return.

I hope that between the great community support through places like the NHUsers mailing list and StackOverflow, Ayende's support offering and now ours, there is a real choice for NHibernate users out there and less of a barrier for adopting this great product.

NHibernate

Linq to NHibernate Progress Report

Steve Strong — Tue, 15 Sep 2009 18:36:57 GMT

I've been busy for the last 3 or 4 weeks getting back into the swing of things with the new Linq to NHibernate parser, based on the HQL AST parser that went live in NHibernate 2.1.

Right now, I am getting some basic structures in place and getting familiar with the re-linq QueryModel. This certainly helps things along, removing a lot of the niggely issues that dealing with a raw Linq expression tree brings. It is, however, a large library itself and takes some getting your head around.

In contrast with the discovery process that I was going through back in June, where all the code I wrote was very much throw-away and nothing got committed, this time it's all for real. Anyone interested can take a look at the latest NH trunk, and I'd welcome any feedback that folk may have.

As anyone who has written a Linq provider surely knows, Linq is a big beast. So rather than trying to solve the whole problem at once, I'm very much taking a TDD-style approach of just writing enough to get a set of tests working. I've no doubt that there will be some queries that cause significant internal changes, but I'm also happy that the structure that's gradually building will be fairly easy to adapt / refactor.

For those interested on the test side of things (which will give a good guide as to the current capabilities), take a look at LinqQuerySamples.cs. Note that there are some tests that are still marked with the Ignore attribute, but all the others are working. Also note that the tests themselves still need some work - most of them don't have any form of assertions, so the only reason I'm happy with them is that I know (from visual inspection) that they are generating the correct HQL. One of the jobs in the next few days is to get the assertions in place so that the tests start forming my safety net.

On previous posts, some people were asking how they could contribute. As far as the core provider itself goes, there probably isn't much that another pair of hands can do - it's pretty dense code and I think we'd just keep stepping on each others toes. The other side of the project, however, is to get a substantial number of test cases together - if you want to help move this forward, then contributions on that side would be much appreciated. Ping me either by email or Twitter and we can work out how to move that forward.

What else to say? Well, I wouldn't plan on using this in production for the next couple of months, but after that I think it might well be viable. Today, it's very much "just for fun" - if you've got serious work that needs to go live in the near future, then stick with the existing 1.0 provider. Hopefully they'll be more to report in the next couple of weeks.

NHibernate, Linq