This one was biting webpoll - taken a while to track down, and it's the sort of thing that other people might be doing, so here's the deal:
Under the covers, metaCore holds a datarow (or an heirarchical collection of datarows for complex objects). Each datarow is in turn associated with a datatable. For objects that are returned from a single search, each of these datarows is associated with the same datatable (pretty much exactly as you'd expect). Now, the DataTable object is not threadsafe under write operations (it's documented as such, so much as I'd like to, I can't really winge at MS).
The way that webpoll works is that it initially loads all of the nodes that are to be polled from the database. It then kicks off a thread for ech node to perform the polling. Periodically, each thread updates its node. And periodically, it all goes bang :)
Since the node objects all came from a single search, they all shared the same underlying table, so the updates from the various threads were doing bad things under the covers. Two possible fixes:
* The obvious one, but probably slower and prone to bugs: Stick "locks" around all updates.
* The less obvious one: intead of doing "NODEList nl = NODE.Factory.Search()", do "NODEList nl = NODE.Factory.CreateListClone(NODE.Factory.Search())". This takes advantage of a non-obvious consequence of doing a clone operation on a list - each of the objects in the cloned list have their own data table, rather than sharing. At that point, thread safety is not an issue, since there is no shared state.
Obviously the Clone approach is slower up front, but for any long-running, multithreaded process, avoiding the locks is likely to be a good thing. Plus locks are generally quite hard to code - you only need to miss one scenario, and the bug is still there (albeit less frequent, and hence harder to fix).
A third approach would be to make metaCore thread-safe on writes, but I've decided against this; thread safety is generally expensive to perform, and the scenarios in which this particular issue crops up are few and far between. Hence I've opted for performance, and put the onus on you guys for thread safety :)
Remember, this is only a problem is you have multiple threads *writing* to data objects that *were created from the same search*. Read-only data caches are just fine, as is multiple threads writing to unrelated objects.