Category Archives: Uncategorized

Using extension method resolution rules to decorate awaiters

This post is a mixture of three things:

  • Revision of how extension methods are resolved
  • Application of this to task awaiting in async methods
  • A rant about void not being a type

Compared with my last few posts, there’s almost nothing to do with genuine asynchronous behaviour here. It’s to do with how the language supports asynchronous behaviour, and how we can hijack that support :)

Extension methods redux

I’m sure almost all of you could recite the C# 4 spec section 7.6.5.2 off by heart, but for the few readers who can’t (Newton Microkitchen Breakfast Club, I’m looking at you) here’s a quick summary.

The compiler looks for extension methods (the ones that "pretend" to be instance methods on other types, and are declared in non-generic top-level static classes) when it comes across a method invocation expression1 and finds no applicable methods. We’ll assume we’ve got to that point.

The compiler then looks in successive contexts for extension methods. It only considers non-generic static types directly declared in namespaces (as opposed to being nested classes) but it’s the order in which the namespaces are searched which is interesting. Imagine that the compiler is looking at code in a namespace X.Y.Z. That has to be within at least one namespace declaration, and can have up three, like this:

namespace X
{
    namespace Y
    {
        namespace Z
        {
            // Code being compiled
        }
    }
}

The compiler starts with the "innermost" namespace, and works outwards to the global namespace. At each level, it first considers types within that namespace, then types within any using namespace directives within the namespace declaration. So, to give a really full example, consider this:

using UD.U0;

namespace X
{
    using UD.U1;

    namespace Y
    {
       using UD.U2;

        namespace Z
        {
            using UD.U3;

            // Code being compiled
        }
    }
}

The namespaces would be searched in this order:

  • Z
  • UD.U3
  • Y
  • UD.U2
  • X
  • UD.U1
  • "global"
  • UD.U0

Note that UD itself would not be searched. If a namespace declaration contains more than one using namespace directive, they’re considered as a set of directives – the order doesn’t matter, and all types within the referenced namespaces are considered equally.

As soon as an eligible method has been found, this brings the search to a halt – even if a "better" method might be available elsewhere. This allows us to effectively prioritise extension methods within a particular namespace by including a using namespace directive in a more deeply nested namespace declaration than the methods we want to ignore.

Async methods and extensions on Task/Task<T>

So, where am I heading with all of this? Well, I wanted to work out a way of getting the compiler to use my extension methods for Task and Task<T> instead of the ones that come in the CTP library. The GetAwaiter() methods are in a type called AsyncCtpThreadingExtensions, and they both return System.Runtime.CompilerServices.TaskAwaiter instances. You can tell this just by decompiling your own code, and see what it calls when you "await" a task.

Now, we can create our own complete awaiter methods, as shown in my previous post… but it’s potentially more useful just to be able to add diagnosis tools without changing the actual behaviour. For the sake of brevity, here are some extension methods and supporting types just for Task<T> – the full code targets the non-generic Task type as well.

using System;
using System.Runtime.CompilerServices;
using System.Threading.Tasks;

namespace JonSkeet.Diagnostics
{
    public static class DiagnosticTaskExtensions
    {
        /// <summary>
        /// Associates a task with a user-specified name before GetAwaiter is called
        /// </summary>
        public static NamedTask<T> WithName<T>(this Task<T> task, string name)
        {
            return new NamedTask<T>(task, name);
        }

        /// <summary>
        /// Gets a diagnostic awaiter for a task, based only on its ID.
        /// </summary>
        public static NamedAwaiter<T> GetAwaiter<T>(this Task<T> task)
        {
            return new NamedTask<T>(task, "[" + task.Id + "]").GetAwaiter();
        }

        public struct NamedTask<T>
        {
            private readonly Task<T> task;
            private readonly string name;

            public NamedTask(Task<T> task, string name)
            {
                this.task = task;
                this.name = name;
            }

            public NamedAwaiter<T> GetAwaiter()
            {
                Console.WriteLine("GetAwaiter called for task \"{0}\"", name);
                return new NamedAwaiter<T>(AsyncCtpThreadingExtensions.GetAwaiter(task), name);
            }
        }

        public struct NamedAwaiter<T>
        {
            private readonly TaskAwaiter<T> awaiter;
            private readonly string name;

            public NamedAwaiter(TaskAwaiter<T> awaiter, string name)
            {
                this.awaiter = awaiter;
                this.name = name;
            }

            public bool BeginAwait(Action continuation)
            {
                Console.WriteLine("BeginAwait called for task \"{0}\"…", name);
                bool ret = awaiter.BeginAwait(continuation);
                Console.WriteLine("… BeginAwait for task \"{0}\" returning {1}", name, ret);
                return ret;
            }

            public T EndAwait()
            {
                Console.WriteLine("EndAwait called for task \"{0}\"", name);
                // We could potentially report the result here
                return awaiter.EndAwait();
            }
        }
    }
}

So this lets us give a task a name for clarity (optionally), and logs when the GetAwaiter/BeginAwait/EndAwait methods get called.

The neat bit is how easy this is to use. Consider this code:

using System;
using System.Net;
using System.Threading.Tasks;

namespace Demo
{
    using JonSkeet.Diagnostics;

    class Program
    {
        static void Main(string[] args)
        {
            Task<int> task = SumPageSizes();
            Console.WriteLine("Result: {0}", task.Result);
        }

        static async Task<int> SumPageSizes()
        {
            Task<int> t1 = FetchPageSize("http://www.microsoft.com");
            Task<int> t2 = FetchPageSize("http://csharpindepth.com");

            return await t1.WithName("MS web fetch") +
                   await t2.WithName("C# in Depth web fetch");
        }

        static async Task<int> FetchPageSize(string url)
        {
            string page = await new WebClient().DownloadStringTaskAsync(url);
            return page.Length;
        }
    }
}

The JonSkeet.Diagnostics namespace effectively has higher priority when we’re looking for extension methods, so our GetAwaiter is used instead of the ones in the CTP (which we delegate to, of course).

Remove the using namespace directive for JonSkeet.Diagnostics, remove the calls to WithName, and it all compiles and runs as normal. If you don’t want to have to do anything to the code, you could put the using namespace directive within #if DEBUG / #endif and write a small extension method in the System.Threading.Tasks namespace like this:

namespace System.Threading.Tasks
{
    public static class NamedTaskExtensions
    {
        public static Task<T> WithName<T>(this Task<T> task, string name)
        {
            return task;
        }
    }
}

… and bingo, diagnostics only in debug mode. The no-op WithName method will be ignored for the higher-priority version one in debug builds, and will be harmless in a release build.

The diagnostics themselves can be quite enlightening, by the way. For example, here’s the result of the previous program:

GetAwaiter called for task "[1]"
BeginAwait called for task "[1]"…
… BeginAwait for task "[1]" returning True
GetAwaiter called for task "[2]"
BeginAwait called for task "[2]"…
… BeginAwait for task "[2]" returning True
GetAwaiter called for task "MS web fetch"
BeginAwait called for task "MS web fetch"…
… BeginAwait for task "MS web fetch" returning True
EndAwait called for task "[2]"
EndAwait called for task "[1]"
EndAwait called for task "MS web fetch"
GetAwaiter called for task "C# in Depth web fetch"
BeginAwait called for task "C# in Depth web fetch"…
… BeginAwait for task "C# in Depth web fetch" returning False
EndAwait called for task "C# in Depth web fetch"
Result: 6009

This shows us waiting to fetch both web pages, and both of those awaits being asynchronous. (Note that we launched the tasks before any diagnostics were displayed – it’s only awaiting the tasks that causes all of this to kick in.) After both of those "fetch and take the length" tasks have started, we await the result of the first one (for microsoft.com). This corresponds to task 1 – but task 2 (fetching csharpindepth.com) finishes first. When the microsoft.com page has finished fetching, the length is computed and that task completes. Now when we await the result of fetching the length of csharpindepth.com, we see that it’s already finished, and the await completes synchronously.

Obviously this was a small example, I deliberately left two tasks with just task IDs, and there could be a lot more information (such as timestamps and thread IDs, to start with) but I suspect this sort of thing could be invaluable when trying to work out what’s going on in async code.

And finally… a short rant

I’ve written all the diagnostic code twice. Not because it was wrong the first time, but because it only covered Task<T>, not Task. I couldn’t write it just on Task, because then EndAwait would have had the wrong signature… but the code was pretty much a case of "cut, paste, remove <T> everywhere".

I’ve never been terribly bothered by the void type before, and it not being a "proper" type like unit in functional programming languages. Now, I suddenly begin to see the point.

Perhaps the TPL should have introduced the Unit type before the Rx team got in there. With a single Task<T> type, I suspect there’d be significantly less code duplication in the framework (including the async CTP).

Is it enough to make me wish we didn’t have void at all? Maybe. Maybe not. Perhaps with sufficient knowledge in the CLR, there wouldn’t have to be any stack penalty for copying a "pretend" return value onto the stack every time we call a method which would currently return void. I’ll certainly be keeping an eye out for other places where it would make life easier.

Conclusion

I don’t normally advocate language tricks like the extension method "priority boost" described here. I love talking about them, but I think they’re nasty enough to avoid most of the time.

But in this case the diagnostic benefit is potentially huge! I don’t know how it would fit into the full framework – or where it would dump its diagnostics to – but I’d really like to see something like this in the final release, particularly with the ability to associate a name with a task.

Even if you don’t want to actually use this, I hope you’ve enjoyed it as an intellectual exercise and a bit of reinforcement about how GetAwait/BeginAwait/EndAwait works.


1 It has to be a method invocation on an expression, too. So if you’re writing code within an IEnumerable<T> implementation and you want to call the LINQ Count() method, you have to call this.Count() rather than just Count(), for example.

Reply to a reply… tweaking refactoring

This is a reply to Ben Alabaster’s blog post, which is itself a reply. You can follow the trail yourself. I’ll assume you’ve read the post – I’m not going to go over anything already written there, other than my comments.

I took issue with three aspects of Ben’s refactoring:

  • The use of float for currency
  • The fact that "BaseRate" effectively doesn’t have a well-defined unit; in some cases it’s "dollars per hour" and in others it’s "dollars per pay cheque, regardless of hours" smells
  • The use of properties for the pay functions

I’ll tackle the last one first, because I was stupid. I suggested using public static readonly fields instead of properties. This was dumb. All we need is a simple static class with public static methods – we can use them with exactly the same source code as before for the Employee construction, but without all the fancy lambda expressions:

public static class PayCalculations
{
    public static float BasicWithoutOvertime(float hours, float baseRate)
    {
        return hours * baseRate;
    }

    public static float BasicWithOvertime(float hours, float baseRate)
    {
        if (hours < 40) return hours * baseRate;
        return ((hours – 40f) * 1.5f + 40f) * baseRate;
    }

    public static float Salary(float hours, float baseRate)
    {
        return baseRate;
    }

    public static float Contractor(float hours, float baseRate)
    {
        /* Base rate */
        float subtotal = Math.Min(hours, 40) * baseRate;
        hours -= Math.Min(hours, 40);
        /* Time plus a half */
        if (hours > 0) subtotal += 1.5f * Math.Min(hours, 20) * baseRate;
        hours -= Math.Min(hours, 20);
        /* Double time */
        if (hours > 0) subtotal += 2.0f * Math.Min(hours, 20) * baseRate;
        hours -= Math.Min(hours, 20);
        /* Double time plus a half */
        if (hours > 0) subtotal += 2.5f * hours * baseRate;

        return subtotal;
    }
}

Less code, less nesting, less use of fancy C# 3 features… generally nicer. The construction code remains the same, because it uses method group conversions to build the delegates.

Fixing the "float should be decimal" problem is easy, of course. Let’s move on to the units and "wrong" values. The problem is that the BaseRate property means different things for different employees, and in some cases it’s not even needed at all. That’s a reasonably strong indicator that it’s in the wrong place. Let’s accept that all employees’ pay may depend on the number of hours they’ve worked that week, but that’s all. Everything else depends on the particular contract that the employee is using, and that can vary. So let’s put the variance into what creates the function – so we can build a "salaried employee on $2935 week" function, a "per hour worker on $40.25 basic without overtime" etc. This is effectively like creating an IPayContract interface and multiple implementations, then creating instances of those implementations which have specific values. Except we’re using delegates… so having ripped out the lambda expressions, I’m going to put them back in :) But this time we’re just going to use a Func<decimal, decimal> as we only to know how much to pay given a certain number of hours worked. (The first decimal here could potentially be float or double instead, but if anyone ever did claim to work 3.1 hours, they’d probably want pay that reflected it.)

Here are the pay calculations:

public static class PayCalculations
{
    public static Func<decimal, decimal> BasicWithoutOvertime(decimal dollarsPerHour)
    {
        return hours => dollarsPerHour * hours;
    }

    public static Func<decimal, decimal> BasicWithOvertime(decimal dollarsPerHour)
    {
        // Use an alternative approach just for LOLs
        return hours => {
            decimal basicHours = Math.Min(hours, 40);
            decimal overtimeHours = Math.Max(hours – 40, 0);
            return (basicHours * dollarsPerHour) + (overtimeHours * dollarsPerHour * 1.5m);
        };
    }

    public static Func<decimal, decimal> Salary(decimal dollarsPerWeek)
    {
        // This *looks* like the units are wrong… but it’s okay, see text.
        return hours => dollarsPerWeek;
    }

    public static Func<decimal, decimal> Contractor(decimal baseRate)
    {
        return hours => {
            // 0-40 hours
            decimal basicHours = Math.Min(hours, 40);
            // 40-60 hours
            decimal overtime = Math.Min(Math.Max(hours – 40, 0), 20);
            // 60-80 hours
            decimal doubleTime = Math.Min(Math.Max(hours – 60, 0), 20);
            // 80+ hours
            decimal chargingThroughTheNoseTime = Math.Max(hours – 80, 0);

            return (basicHours * baseRate)
                 + (overtime * baseRate * 1.5m)
                 + (doubleTime * baseRate * 2m)
                 + (chargingThroughTheNoseTime * baseRate * 2.5m);
        };
    }
}

And now, when we construct the employees, we don’t have to specify a base rate which was only meaningful in some cases – instead, we give that value to the pay calculator instead:

List<Employee> employees = new List<Employee>
{
    new Employee("John", "MacIntyre", PayCalculations.BasicWithoutOvertime(40.25m)),
    new Employee("Ben", "Alabaster", PayCalculations.BasicWithOvertime(40.25m)),
    new Employee("Cory", "Fowler", PayCalculations.Salary(2935m)),
    new Employee("John", "Doe", PayCalculations.Contractor(150m)),
    new Employee("Jane", "Doe", hours => 3500m),
    new Employee("Joe", "Bloggs", hours => 34.25m * Math.Max(hours, 15))
};

Now, look at the Salary method and the comment in it… I’m still concerned about the units. We’re meant to be returning a simple dollar value (and in another system I’d probably bake that information into the types used) but we’ve still got dollarsPerWeek. What’s wrong here? Well, it all boils down to an assumption: we’re running this once a week. We’ve got a constant of 1 week… so we could rewrite the method like this:

public static Func<decimal, decimal> Salary(decimal dollarsPerWeek)
{
    decimal weeksWorked = 1m; // We run payroll once per week
    return hours => dollarsPerWeek * weeksWorked;
}

Now the units work… although it looks a bit silly. Of course, it makes our assumption very explicit – and easy to challenge. Maybe we actually run payroll once per month… in which case the fact that we’re expressing the salary in dollars per week is wrong – but very obviously wrong, which is a good thing.

Conclusion

It doesn’t feel right to have a blog post with no conclusion. Lessons applied here:

  • Remember all the tools available, not just the shiny ones. Using method group conversions made the initial "constant function" approach simpler to understand.
  • Units are important! If the same field effectively represents different units for different instances, there’s something wrong
  • If a field is only relevant for some instances of a type, it’s probably in the wrong place
  • Don’t use floats for currency. See any number of Stack Overflow questions for reasons why :)

EDIT: As noted by Barry Dorrans, there’s a lot of scope for introducing constants in here, for further goodness. I’m not going to bother updating the code just for Barry though. That way madness lies.

C# 4, part 5: Other bits and bobs which probably don’t merit inclusion

Okay, I know I said that part 4 would be the last part in this series… but since then I’ve not only thought about iterator block parameter checking, but a few other things. Some of these I simply forgot about before, and some I hadn’t thought of yet. I’m not sure any of these are actually worthy of inclusion, but they may provoke further thought.

Tuple returns

I’ve been reading Programming Erlang and I suspect that being able to return tuples (i.e. multiple values, strongly typed but without an overall predefined type) would be a good thing. For instance, in a tuple-returning world, int.TryParse could be redesigned to return both the true/false and the parsed value. It could have a signature like this:

public static (int, bool) TryParse(string text)

… and then be called like this:

int value;
bool parsed;

(value, parsed) = int.TryParse(“Foo”);

Now, a few things to work out:

How do we ignore values we’re not interested in?

Part of the problem with out parameters is that sometimes you don’t actually care about the value – but you still have to declare and pass in a parameter. Suppose we could use ? as a placeholder for “I don’t care”. (This is _ in Erlang pattern matching, IIRC. Same kind of business.)

What could you do with a tuple?

We could potentially make tuples first class citizens, so that you could declare variables of that type, a bit like anonymous types, but with anonymous property names as well, used just for matching later. Or we could force matching at the point of method call, which would restrict the use a bit further but leave less other rules to be worked out.

Either way, I’d hope to be able to set either fields or properties by parameter matching.

What’s the value of the overall expression?

This really depends on the answer to the previous question. If tuples are first class types, then the result of the expression would normally be the tuple itself. However, I wonder whether there’s more that can be done. For instance, thinking about our TryParse example, it’s useful to be able to write (currently):

if (int.TryParse(“Foo”out value))
{
   …
}

Suppose we were able to designate one of the matched elements of the tuple to be the expression result, e.g. using _ to be slightly Perl-like:

if ((value, _) = int.TryParse(“Foo”))
{
   …
}

Would that be worth doing?

More information required…

I suspect that people who know more about the use of tuples in other languages would be able to say more about this. Some overlap with anonymous types is clearly relevant too, and would need to be carefully considered. I’m not wedded to any of the syntax shown above, of course – I’m just interested in how/where it could be useful.

Named method/constructor arguments

One of the features I like about F# is that you can specify the names of arguments, without worrying about the order. This means that it becomes even more important to name methods appropriately, but it would make method calls with many parameters simpler to read. Currently it’s common practice to use one parameter per line and a comment to indicate the use, e.g.

foo.Complicated(10,        // Number of elements to return
                “bar”,     // Name of collection
                x => x+1,  // Step for element
                3.5        // Load factor
               );

In fact, this example is relatively simple because all the parameter types are different – look at the more complicated overloads of Enumerable.GroupBy for rather more hellish examples. It’s incredibly ugly, and the compiler isn’t able to check anything. Now suppose we could instead write:

foo.Complicated(maxElements = 10,
                collectionName = “bar”,
                step = x => x+1,
                load = 3.5);

Personally I think that’s clearer and less error-prone. The arguments could be reordered with few issues, and the compiler could check that we really were using the right parameter names. One potential issue is in terms of side-effects, where evaluating one argument had a side-effect which affected the evaluation of another argument. At that point reordering is a breaking change. I suspect the compiler would need to stick to the specified textual order, and then rework things on the stack as required to get the appropriate order for the method call. A bit nasty.

Event handler subscription in object initializers

I only thought of this one today, when coming up with an example for a screencast on object initializers. I suspect most uses of object initializers will be to with custom classes (although I recently used them for XmlWriterSettings to great effect) which would make the screencast harder to understand. I was wondering what common framework classes had lots of writable properties, and I hit on the idea of building a UI. It shouldn’t surprise me that this works quite nicely, but you can build up a hierarchical UI quite pleasantly. For example:

Form form = new Form
{
    Size = new Size(300, 300),           
    Controls =
    {
        new Button
        {
            Location = new Point(10, 10),
            Text = “Hello”,
        },
        new ListBox
        {
            Location = new Point(10, 50),
            Items =
            {
                “First”,
                “Second”,
                “Third”
            }
        }
    }
};
Application.Run(form);

This is somewhat reminiscent of Groovy builders (and no doubt many other things, of course). However, one thing you can’t currently do is attach an event handler in an object initializer. The obvious syntax would be something like:

new Button
{
    Location = new Point(10, 10),
    Text = “Hello”,
    Click += (sender, args) => Save()
}

where I happen to have used a lambda expression, but didn’t need to – a normal method group conversion or any other way of constructing a delegate would have done just as well.

I mailed the C# team about this, and although it’s been considered before it’s really not useful in many situations. However, the syntax has been left open – there’s no other use of += within object initializers, so it could always be revisited if someone comes up with a killer pattern.

Immutable object initialization

I’ve been thinking about this partly as a result of object initialization in general, and the previous point about named arguments. As has been noted before, C# doesn’t really help you to build immutable objects – either as from the point of view of building the type, or then instantiating it. Basically you’ve got the constructor call, and that’s it. A static method could set private properties and then return the object for popsicle immutability, but it still feels slightly grim.

Someone (possibly Marc Gravell – not sure) suggested to me that there ought to be some way of indicating when an object initializer had finished. At the time I think I rejected the idea, but now I like it. There’s already the ISupportInitialize interface, but that feels slightly too heavy to me – in particular, it has two methods rather than just one. What I think could be nice would be:

  • A new interface with a single CompleteInitialization method.
  • Readonly automatic properties which would either make the property only writable during a constructor call if the new interface weren’t implemented or would insert an execution-time check that CompleteInitialization hadn’t been called already.
  • I’d anticipate the C# compiler implementing the new interface itself automatically in some way which supported inheritance reasonably, unless specifically implemented by the developer.
  • Members other than constructors couldn’t set readonly automatic properties on this, to avoid accidents.
  • The CLR should have some interaction so it knew which fields it could treat as being readonly after initialization had been completed.
  • Object initializers would call CompleteInitialization automatically at the end of the block.

It’s a bit messy, and I’m sure I haven’t thought of everything – but I suspect something along these lines would be a good idea at some point. It’s reminiscent of an earlier wacky idea I had which went further, but this would be specifically to support immutability. Without it, complex immutable types end up with nightmarish constructor calls.

Conclusion

So there we have it – some relatively half-baked ideas which will hopefully provoke a bit more thought – both from readers and myself. It’s interesting to note that aside from event subscription, they all have a fair number of questions and complexity around them, which is off-putting to start with. I would feel more comfortable about event subscription being added than any of the others, because it’s relatively simple and independent. The others feel like more dangerous features – even if they’re more useful too.

The value of a language specification

Last Friday evening was the inaugural Reading Geek Night. More about that another time, but if you’re in or around Reading in the UK, and fancy meeting up with some smart people (and me) to discuss software in various shapes and forms, let me know.

After most people had gone home, a few of us including Stuart Caborn were talking about specs. Stuart remembers how I ended up writing some annotations in the C# Annotated Standard: we were debugging some code, and I noticed an unboxing conversion which was unboxing an enum as an int. It worked, but I was surprised. I consulted the spec, and found that according to the spec it really shouldn’t have worked. (Furthermore, the spec suggested a case which couldn’t possibly be valid. I can’t remember the details now, but I can dig them up if anyone was interested.) I’d had one or two conversations with Jon Jagger (the C# ECMA Task Group convenor at the time) before, so I mailed him. Jon invited me to join in the book project, and I took to it with gusto. I reading most of C# 2 ECMA spec over the course of a few weeks, writing annotations as I went along.

This is not what most people would consider normal behaviour. When I recently gave a talk about C# 3, I was delighted to hear someone else mention that they had checked the spec about some aspect of the language. Finally, I wasn’t alone! However, such people are clearly the exception rather than the rule.

I genuinely don’t think that matters too much. I really don’t expect many developers to read the spec – certainly not thoroughly. I think it’s important to know that there is a spec, and be able to consult it when in doubt. I want to be able to know what every line of code is doing, in terms of which variable it’s going to access, which method it’s going to call, the order of execution of a post-increment as a method argument, etc.

That’s not to say I actually learn all of the rules by rote – even for something as simple as operator precedence, I sometimes put brackets in when they’re not required, for example. I’d rather not rely on me or a maintenance engineer having to remember too many details. But if someone else has written some obscure line of code, I’m pretty confident that I’ll be able to understand it with the help of the spec, and refactor it into something more readable.

Now, Stuart challenged the value of the spec. If his code was misbehaving he wouldn’t consult the spec – he’d consult either books or (more likely) the unit tests. Realistically, the vast majority of C# is being compiled by the Microsoft compiler, so the idea of having a spec available for other implementations isn’t actually important to that many developers in terms of business. (It may be psychologically and politically important, and I’m not trying to knock the great work that the Mono project has done – but I’ve never used Mono professionally, and I suspect that’s the case for most people.) Either the code works or it doesn’t, and if it doesn’t work the tests should say so.

I counter that not having a spec is like not having documentation for a library – if you start relying on unspecified behaviour, you can come unstuck when that behaviour changes in a legitimate way. A good example of this is depending on a particular hash algorithm being used for GetHashCode; the algorithm for string.GetHashCode() changed between .NET 1.1 and 2.0, and I’ve seen a few people get burned, having stored the generated hash values in a database. Suddenly nothing matches any more… because they ignored what the documentation said.

Stuart’s response: if the tests still work, the changes haven’t broken anything. If the tests don’t work, we can go and fix the code so they start to work again. I’ll concede that it’s unlikely that implementation changes in the compiler will actually break any code (and it’s also very unlikely that specification changes will break code – the C# design team are pretty fanatical about not introducing breaking changes).

I can see Stuart’s point of view, but it just feels so very wrong. I suspect a lot of that is down to my personality type – how I really hate working without enough information (or what I consider to be enough information). Today I fixed a bug with an ASP.NET application which was producing incorrect JavaScript. It was working on some machines and not working on others. I thought I’d found out why (a different version of a library in the GAC) but that was ruled out after examining another machine which was working contrary to my hypothesis. I’m reasonably confident that my fix will work, but I really don’t like the fact that I don’t understand the issue in the first place. It’s very hard to piece together the necessary information – which is like working on a language that doesn’t have a spec.

Eventually, I came up with an answer which I think Stuart more or less accepted. I’m in one of the groups the spec is aimed at. I write about C#, hoping to explain it to other people. One of my aims with C# in Depth is to give enough information to make the spec even more irrelevant to most developers when it comes to the changes in C# 2 and 3. Without wishing to denigrate existing C# books too much, I’ve often found that the kind of details which I wanted to investigate further just weren’t covered in the books – to get the answer, I had to go to the spec. I really hope that if I’d had my own book, I’d have been able to consult that for most of those issues. However, I simply couldn’t have written the book without the spec.

I’ve had experience of writing about a language without a spec. When I was helping out with Groovy in Action, I often found myself frustrated by the fact that the Groovy spec is far from finished. This shouldn’t be surprising to anyone – Microsoft have a significant team of really smart people who are paid to immerse themselves thoroughly in C# and make sure the language is all it can be, in terms of design, documentation and implementation. Designing a language well is hard – I haven’t been part of designing any languages, but I can get some idea of the difficulty based on what I’ve seen of the languages I’ve used. The loving care required to make sure that all the behaviour that should be pinned down is indeed described, while leaving rigidly defined areas of doubt and uncertainty where that’s appropriate, must be phenomenal. I don’t doubt that the Groovy team is talented, but coming up with a good spec is probably too much of a resource drain, unfortunately.

I haven’t covered everything I feel about specs in this post, but I’m going to finish now before I officially begin to ramble. Apologies to Stuart if I’ve misrepresented his views – and I should point out that this was late at night after Stuart had a few beers, which may be relevant. In short then (and including points I haven’t gone into):

  • The existence of a specification is important, even if it’s not consulted by every developer. Even if I were never ill, I’d be glad that the National Health Service existed.
  • I’d be very worried if the language/compiler team itself didn’t have a good spec, and if they do, there’s no reason to hide it. As an example of how important this is, just read Martin Fowler writing about JRuby/IronRuby: “Soon-to-be-ThoughtWorker Ola Bini, a JRuby committer, reckons that it’s almost impossible to figure out how to implement a Ruby runtime without looking at source code of the MRI.” That screams to me of “the implementation is the documentation” which I regard as a very unhealthy state of affairs.
  • Specifications are vital for authors (whether of books or web articles) who need to present accurate information based on more than just the current behaviour.
  • Sometimes you can trust a specification more than tests – with a memory model spec, it’s possible to reason about whether or not my code is thread-safe. It could pass all tests but still not handle a bizarre race condition. (Of course, a better memory model spec for .NET would be welcome.)
  • Unit tests are never going to catch every flaw. They can give you a great deal of confidence, but not certainty. (Example: how many people explicitly check that their text handling code will work just as well when provided with non-interned strings, rather than strings which were originally specified as literals? If the string interning behaviour changes in a valid way, are you absolutely sure your code won’t fail?)
  • I’m on the fence about the value of having the ECMA spec as well as the Microsoft one. I can see how it could be important in certain business situations – but as a developer, I don’t care that much. I’ve had very few qualms about changing my standard reference from ECMA C# 2 to MS C# 3. It’s unclear to me (as someone completely outside the process) how much influence ECMA has at this stage on the design of the language itself. Were ECMA committee members explicitly consulted during the C# 3 design process? Clearly making a significant change to the language now would be likely to make all existing compilers “broken” – so what can the ECMA team do beyond reframing the existing rules? As I say, I’m an outside in this matter, so I can’t really judge – but I think it’s a valid question to ask.

Anyway, that’s about a sermon’s-worth of preaching about specifications – time for bed.

The CLI memory model, and specific specifications


A while ago, I was directed to a disturbing (in my view)
post
on GrantRi’s blog, to do with the .NET memory model. I’m slightly less disturbed
having read his clarification,
but there’s a fairly deep problem here. Here’s part of a sample class:


string name;
    
public void WriteNameLength()
{
    string localName = name;
    if (localName!=null)
    {
        Console.WriteLine (localName.Length);
    }
}


Now, other threads may be changing the value of name all over the place,
and there’s an issue in WriteNameLength in terms of whether or not it shows
the “latest” value, but my question is: can the above throw a NullReferenceException?



It looks like it can’t, because even if name becomes null during the method,
surely the value of localName can’t change – it’s either null or it’s not, and
we don’t try to dereference it if it’s null.



Unfortunately, it looks from Grant’s blog post as if a JIT should be free to treat the above as:


public void WriteNameLength()
{
    if (name!=null)
    {
        Console.WriteLine (name);
    }
}

Now the above clearly can throw an exception, if name becomes null in another thread after the “if” and before the dereference (and if that change is noticed by the thread running WriteNameLength).


This surprises me – just as it surprised lots of people reading Grant’s blog. It surprised me so much that I checked the CLI specification, and couldn’t work out whether it was correct or not. This is even more worrying – so I mailed Grant, and his (very speedy) reply was along the lines of “I’m not an expert, but it looks to me like the spec is too vague to say for sure whether this is legitimate or not.” (I apologise if I’ve misrepresented the reply – in some ways it doesn’t matter though.)


When trying to write performant, reliable systems, it is surely crucial to have a memory model specification which can be reasoned with. The Java memory model was reasonably well defined before 1.5, and then (after years of detailed discussion) it was updated in way which I believe was designed to give backward compatibility but lay out very clear rules. Surely the CLI deserves a specification with a similar level of detail – one which both JIT developers and application developers can use to make sure that there are no surprises amongst informed individuals. (There will always be people who write multi-threaded programs while remaining blissfully unaware of the importance of a memory model. It’s very hard to cater for them without crippling JIT optimisation, effectively synchronising all the time. I’m not too worried about that.)


Usually, when I’m writing multi-threaded code, I err on the side of caution – I tend to use locks when I could get away with volatile variables, for instance, just because I need to think slightly less hard to make sure everything’s correct. There are people for whom that’s not just good enough – their performance requirements make every context switch, every locking operation, every optimisation restriction valuable enough to really need to know the details of the memory model. There should be an effort on the part of MS and/or the ECMA committee to clearly and specifically define what the CLI memory model does and doesn’t guarantee. I doubt that anyone reading this blog is in a position to instigate such an effort – but if you are, please give it careful consideration.

Why development is still hard in 2006


Life has become considerably easier for developers over the years, particularly with the advent of managed code (or whatever the equivalent terminology is for Java). Memory usage is something which one only needs to be aware of rather than constantly being "in your face" in the way it tends to be in C. However, that doesn't mean that all is rosy, or that we can solely concentrate on actual business problems. I thought it might be worth a quick run-down of the problems I tend to find getting the way of more interesting work. In every case things have become a lot simpler than they were a while ago, and in many cases there are promising new technologies or research efforts underway to improve the situation further. To be honest, I doubt that any of those improvements will be enough to remove the relevant item from the list. I expect that if I come back in five years or so, the list may well be largely the same. So, have I missed any "biggies"? Am I shockingly stupid for regarding any of these as "hard"? I haven't included user interface design in here, partly because I have so little experience of it. We seem to keep changing our minds as an industry about what's "good", and we still seem to keep coming up with UIs with fundamental problems like not being properly resizable, so I guess we haven't cracked it yet – but I don't think I can give much insight into the actual problems. Anyway, on to the list…


Installation and updates


Installation is one of those things which tends to get forgotten until near the end of a development cycle – or at least, it's rare to get it right until you're about to ship. To some extent, this is due to the fact that it relies on knowing exactly what will need to be installed. In an ideal world, installation should be very simple in terms of being totally transactional, so that if anything goes wrong it can be rolled back reliably. If you're just adding files to the local file system, that's not too far from reality – but for many types of installation there's a lot more involved. What if you need to install a new database schema, and the database goes down after you've done that part of the installation but haven't yet finished the other parts? What if installing the application requires registering on a remote server? Basically, as soon as anything other than the local box is needed, it's hard to absolutely guarantee a clean rollback. Installation is often given to junior engineers and regarded as a less prestigious part of the project to work on, but it's absolutely crucial in terms of customer satisfaction and system stability.



Updates can be even worse – you may need to repair a "broken" system, maintain the customer's configuration from the previous installation, notice any "customisations" they have made to the previous installation, possibly upgrade from multiple versions with one installer, etc. Rollback of an unsuccessful upgrade is even trickier than normal installation, as you'd ideally want to roll back to the previous system state – an upgrade almost never involves just adding files, as you usually want to replace previous components.



Finally, installation can be a very platform-specific area to work in. Even if you're only installing on Windows, there are "gotchas" for each edition – and then you need to potentially check that the right service packs have been installed, etc. When you come to cross-platform installation, life is even worse. Checking that any dependencies are installed (and in the way you expect them to be), making your application available in the appropriate way for that system, integrating with whatever installer services are the norm – it's enough to drive a person crazy.


Versioning


Tied in with installation is versioning across communicating systems. I've only recently had to deal with this – it's certainly not something that all developers are likely to need. When you do need it, however, it's a pain. Suppose version 2 of your application needs to be able to talk with version 1 and vice versa. Undoubtedly v2 will have features that v1 doesn't support, and it may implement the v1 features in a slightly different way. The details of what is communicated in what situation are tricky to get right. This is one of those problems which isn't too hard to handle for any particular small case, but the difficulty lies in being rigorous in the definition of what you're allowed to do without a component (or whatever you use as your unit of versioning) needing to really change version. You may be able to add some data, using a default when it's not provided, but not change a method signature, for example. Likewise, depending on what technology you're using for the communication, you may need to lay down rules about exactly how data is sent between the systems. Once those rules have been precisely defined, you then need to be utterly meticulous about sticking to them. Following rules is tedious, and all developers can be forgetful on occasion.



Oh, and then there's the testing, of course. Do you support a connected system which includes two installations of v1, one installation of v2, and one installation of v3? What if for v3 you've decided to drop some of the v1 functionality? When writing v1, you need to be aware of future possibilities so you can handle them cleanly. The principle of YAGNI is less applicable here than normal, because we can't accurately predict the future. YAGNI is fine when you can implement a feature later on, but it's less useful when you don't get to force all your customers to upgrade all their systems. While you don't need to predict everything you'll implement later, you may well need to build features in now to accommodate changes later on.


Internationalisation


Up-front warning: I'm not an expert on i18n. That's one of the problems – there are very few people who are. I know what Unicode surrogates are, and I know that very little of my own code handles them properly. I know a bit about some of the more common encodings available. I know of little gotchas like the capitalised form of "i" not being "I" in Turkish (having been bitten by that one in a previous job) – that when you consider some manipulation of text data, you'd better know whether it should be done in the system locale, the user's locale, the database's locale, a different specific locale, whatever. I know that a UI designed without taking into account that labels will take up different widths in different languages is likely to fall flat when it's localised. I know that repeatedly replacing "  " (two spaces) with " " (one space) until you can't find "  " (two spaces) any more can lead to an infinite loop in .NET, as String.Replace treats zero-width characters differently to String.IndexOf.



These are just wrinkles I've come up with off the top of my head – I'm sure I could think of plenty more if I wanted to provide a longer list. All of that is without being in any way an expert. Goodness knows what bizarre stories someone genuinely knowledgeable could tell. Now, although I'm not an expert, I'm reasonably intelligent. I don't expect all the other developers on my team to have all the expertise I'm missing. Heck, I don't expect there are that many projects which have even a single genuine i18n expert. Even if they did, that expert would have to review virtually all the code of the project: how many classes do you write which really don't have any text manipulation? It's not just text which ends up in front of a user which you need to be careful with…



I strongly suspect that almost all applications are broken to a greater or lesser extent when it comes to i18n. How many validation routines take surrogates into account? How many servers have the same bug which we happened to find and fix, trying to do a case-insensitive comparison of header names by upper-casing the name in the system locale? How many systems are going to correctly handle sorting characters in Japanese text, taking the kana into account? I don't know whether it would be more depressing to think I was just singularly incompetent, or whether it's worse to believe that everyone else is just as ignorant of these issues as I am.


Date and time handling


I've broken this part of i18n out into its own topic because it's so nasty even if you only have one culture to deal with. When do you use UTC, and when do you use a local time? How easy is it to get at the local time? When should you use the local time of the user, and when should you use system times? What about daylight saving times, which can lead to some local date/time combinations being ambiguous and others being impossible? How do you gracefully cope with the system time changing abruptly?



There are four core problems here, as far as I can see. Firstly, there's working out what to do in any particular situation. Sometimes the answer is obvious, and we've learned a lot over time about best practices – keeping date/times in UTC as long as possible, for instance. In other cases the answer is harder to work out, or different users may have different goals or expectations, leading to no one correct solution.



The second problem – one which is often ignored – is communicating your decisions. Agreeing on some terminology can help, but everyone needs to be willing to take a bit of time to internalize the "rules". This is the case in many areas of development, but I've found that date and time handling tends to be particularly tricky, just because unless you're really precise about what you mean, different people will interpret your words in different ways. The more actors in the system, the worse it gets: if you're considering the situation where you have a server in Australia administered by someone in London, with a helpdesk operator in Germany answering a call from someone in France, the chances of everyone agreeing on exactly when something happened are really slim.



Thirdly, the commonly available libraries are pretty rubbish at the moment. Java allows you to do the right thing, but because it's taken several goes to get it right, there are deprecated methods everywhere. The decision to make months 0-based makes sense in some ways, but catches pretty much everyone out sooner or later, and can make tests harder to read. The precise behaviour of calendars in terms of setting/adding/rolling different inter-related fields is fairly precisely defined, but not easy to understand. There's no simple access to the UTC timezone without magic strings. The use of inheritance between java.util.Date, java.sql.Date and java.sql.Timestamp can lead to tests failing unexpectedly. At least you can generally do what you want though, with a bit of work – .NET is far worse in this respect. In 1.1, there was no way of knowing whether a DateTime was local or UTC; calling ToLocalTime() repeatedly would keep changing the time by the timezone offset on every call. In .NET 2.0 there's the DateTime.Kind property which helps, but it's a bit of a sticking plaster. There's still no way (in the framework itself) of getting at the system's list of timezones. I dare say this will improve over time, but I can't see why it's taken so long to get even this far. I'm sure there are smart people at Microsoft who know the kind of thing required for writing applications which will be have users in different timezones to the system itself – why weren't they more involved in designing the API?



Fourthly, there's the real world, where politicians may arbitrarily decide to change daylight savings etc. There has been talk about changing Britain's timezone to be one hour ahead of where it is now. How would that affect all the software in the world? How many systems would need to know about the change? How would they all find out about it? It feels to me like the same kind of scale of change as altering the currency of a country – possibly worse, as there are lots of applications which deal with times but don't ever need to deal with money.


Resource handling


I said at the start of this article that memory handling wasn't much of an issue now if you're using .NET or Java. You need to be slightly careful to make sure you don't have orphans due to events, static members etc, and you need to be aware of what's going on in order to avoid making gross inefficiencies for no reason, but most of the time you don't need to worry about things. This isn't the case when it comes to other resources, such as file handles, network connections, etc.



I've read posts by C++ developers who maintain that C++ has effectively solved the situation with RAII and auto_ptr. I don't know enough about C++ to say to what extent this is true, but in .NET and Java, without deterministic finalization (for pretty compelling reasons, in my view) you still need to handle non-memory resources manually. Now, C# provides the very useful using statement (which I deeply miss when working in Java) to make life easier, but there's still the manual element of making sure you always use it in the right place. There's still got to be a sense of someone "owning" the resource, and that object (and only that object) releasing it, and nothing else trying to use the released resource reference afterwards. A good example of this problem is when creating an image from a stream in .NET. Whenever I use a stream in .NET, I habitually start wrapping it in a using statement – but if I'm providing the stream to Image.FromStream, I have to notice in the documentation that I've got to keep the stream open for the lifetime of the Image. The documentation doesn't make it clear whether or not disposing the image will close the stream for me. Furthermore, making the transfer of ownership from the calling code to the image atomic is far from straightforward.



This is the area in which I have the most hope for the future. Possibly the successor to .NET will have resource clean-up all sorted out. I dare say it'll be a long time coming, but I still plan to be developing in 15 years' time. I hope at that point I can look back and shake my head at the hoops we have to go through today.


Concurrency


Increasingly, developers need to know about threading. Gone are the days where most developers could rely on their application being the only one running on the box at the time, and it being okay to just make the user wait for a while if a time-consumering operation was required. Like i18n, I'm not a threading expert. I probably know more about it than most developers due to investigating it more (I find the whole business fascinating) but that doesn't make me an expert. I've tried to write about the topic in an accessible way, but there are huge areas I haven't written about, simply because I don't know about them. Every so often I'll come across an optimisation I wouldn't have thought would be valid which could call into question code I thought was reasonably safe. So, to start with, there's a lot to know.



Then there's a lot of care to be taken. In some ways, avoiding deadlocks is straightforward: keep locks for as short a time as possible to avoid contention, and if you ever take out more than one lock, make sure that the code paths that will take out those multiple locks always acquire them in the same order. The reality of implementing that strategy is much harder than it sounds, in my experience – certainly when the system gets large.



Then there's the technology side of things – the facilities provided to us by the platform we're working on. These have improved by leaps and bounds over the years, and things like the CCR sound like they'll make life easier. All I'd say is that we're not there yet. While every call from a background thread to a UI thread needs some manual coordination, there's still work to do. One problem is that to get things right, you tend to need to know a certain amount of what's going on under the covers: while I expect life to get easier for developers, I think they'll still to understand a bit about tricky things like memory models and the strange optimisations that are permissible.


Error handling


Exceptions are lovely. I generally agree with Joel Spolsky, but I completely disagree with his view on exceptions. That's not to say he doesn't make some good points, but I consider his solutions to the problems of exceptions to be worse than the problems themselves. Returning error codes has proved to be a dangerous way of working – it's far too easy to forget to check the code. The equivalent with exceptions is catching an exception and then ignoring it – and that happens in real code, far more often than it should, but at least it requires actual code to do the wrong thing.



So, why is error handling still in my list? Because we haven't become good at using exceptions yet. We still find it tricky at times to know the right point to catch an exception, and in what rare circumstances it's right to catch everything. Also, there's more to error handling than just exception handling. How forgiving should we make our systems? How do we report errors to the user? How do we give users error information which is precise enough for our support team but which doesn't scare the user to death? Oh, and how do we educate developers not to catch exceptions and ignore them without having a really good reason?



Part of this may be technological. Java tried an experiment with checked exceptions, and although I was a fan of them for a few years, I've changed my mind over time. I think the experiment was worth trying, and there were some benefits that ought to be captured by the Next Big Thing, but the overall effect wasn't all it could be. I'm not smart enough to come up with the Next Big Thing myself, but I'm hoping it will improve reliability without giving the developer more grief.


API creep


If the previous topic was a bit ropey, this one barely made it on the list at all. It should definitely be on a list, however, and the link is tenuous but just about visible, so it can live here for the moment.



I've commented before how CVs these days have shopping lists of technologies on them. Regardless of how accurate those CVs are, the technologies themselves certainly exist and are being used by someone, somewhere. Just take one topic: XML, for example. How many XML APIs/technologies do you know? How many more do you know of even if you haven't used them? Here's a list off the top of my head, without reference to the net:



DOM, SAX, JDOM, dom4j, Xerces, Xalan, STaX, MarkupBuilder (and related), XPath, XQuery, XSLT, xpp3, Jaxen, JAXP, XmlReader (and related), Xstream.



Yikes! Just keeping up with all the XML APIs would be a full-time job, and that's just XML! Trying to stay on top of the standard libraries of both .NET and Java is equally tricky. How is anyone meant to cope? My personal answer is to focus on the technology I need to solve the problem at hand, but to try to keep an ear to the ground to at least have a passing awareness of interesting things I may want to use in the future. It's impossible to gauge how successful I am at that, but I know that it's a time-consuming business, and I see no sign of the software industry slowing down. Don't think I'm not grateful for all the work that these technologies save me – I'm just recognising that the variety available comes with a penalty.


Conclusion


Here in 2006, life is still tricky in software development. From a career point of view, that's a good thing – I'm pretty good at what I do, and if everything became trivial, I guess I wouldn't have as much employment value. On the other hand, some of these problems have been with us a long time and we're making lamentably slow progress towards making them no-brainers. Someone remind me to come back to this list in 2011…


Elegant comparisons with the null coalescing operator


A while ago I commented on how I’d like a return? statement, which
only returned if the return value was non-null. The purpose of this was to remove
the irritation of implementing Equals and IComparable.CompareTo
on classes with several properties. For an example of the kind of thing I mean,
consider an Address class with properties Country,
State, City, ZipCode and HouseNumber.
(Apologies to readers who aren’t American – while I feel a traitor to my country for
using state instead of county and zip code instead of post code, I’m guessing there are
more readers from the US than from elsewhere.)



This Address class needs (for whatever reason) to be comparable to itself,
comparing the properties in the order described above, in normal string comparison order.
Let’s see how annoying that is without doing anything clever. (I haven’t included
any property implementations or constructors, but I’m sure you can all guess what they’d
look like. Similarly, I haven’t overridden object.Equals or object.Hashcode,
but the implementations are trivial.)


using System;

public sealed class Address : IComparable<Address>
{
    string country;
    string state;
    string city;
    string zipCode;
    int houseNumber;
    
    public int CompareTo(Address other)
    {
        if (other==null)
        {
            return 1;
        }
        int ret = country.CompareTo(other.country);
        if (ret != 0)
        {
            return ret;
        }
        ret = state.CompareTo(other.state);
        if (ret != 0)
        {
            return ret;
        }
        ret = city.CompareTo(other.city);
        if (ret != 0)
        {
            return ret;
        }
        ret = zipCode.CompareTo(other.zipCode);
        if (ret != 0)
        {
            return ret;
        }
        return houseNumber.CompareTo(other.houseNumber);
    }
}


That’s ignoring the possibility of any of the properties being null. If
we want to include that possibility, it’s worth having a static helper method which
copes with nulls, along the lines of object.Equals(object, object).



Now, if we don’t care about doing more comparisons than we really want to and
potentially creating an array each time, it wouldn’t be hard to implement a series
of overloaded methods along the lines of:


public static int ReturnFirstNonZeroElement(int first,
                                            int second,
                                            int third)
{
    return first != 0 ? first :
           second != 0 ? second :
           third;
}


(The array part would be when you implement ReturnFirstNonZeroElement(params int[] elements)
after you’d got enough overloads to get bored.)



That still ends up being a lot of code though, and it’s doing unnecessary comparisons.
I’m not keen on micro-optimisation, of course, but it’s the inelegance of it that
bothers me. It feels like there must be a way of doing it nicely. With
C# 2.0 and the null coalescing operator, we do. (At this point I’m reminded that
the irritation actually came when writing Java, which of course doesn’t have anything
similar. Grr.) For those who are unaware of the null coalescing operator (and it’s one of the
least well publicised new features in C# 2.0) see
my brief coverage of it.
Now consider the following helper method:


public static int? CompareFirstPass<T>(IComparable<T> first, T second) 
    where T : IComparable<T>
{
    if (first==null)
    {
        return -1;
    }
    // Assume CompareTo deals with second being null correctly
    int comparison = first.CompareTo(second);
    return comparison==0 ? (int?)null : comparison;
}


In short, this returns the result of the comparison if it’s non-zero, or null otherwise.
Now, with the null coalescing operator, this allows the Address class implementation
of CompareTo to be rewritten as:


public int CompareTo(Address other)
{        
    return other==null ? 1 :
           Helper.CompareFirstPass(country, other.country) ??
           Helper.CompareFirstPass(state, other.state) ??
           Helper.CompareFirstPass(city, other.city) ??
           Helper.CompareFirstPass(zipCode, other.zipCode) ??
           houseNumber.CompareTo(other.houseNumber);
}

It’s short, simple and efficient. Now, doesn’t that make you feel better? :)

Broken windows and unit testing

There’s quite possibly only one person in the world reading this blog who doesn’t think it’s got anything to do with Vista. The windows in the title have nothing to do with Microsoft, and I’m making no assertions whatsoever about how much unit testing gets done there.


The one person who understands the title without reading the article is Stuart, who lent me The Tipping Point before callously leaving for ThoughtWorks, a move which has signficantly reduced my fun at work, with the slight compensation that my fashionable stripy linen trousers don’t get mocked quite as much. The Tipping Point is a marvellous book, particularly relevant for anyone interested in cultural change and how to bring it about. I’m not going to go into too much detail about the main premises of the book, but there are two examples which are fascinating in and of themselves and show a possible path for anyone battling with introducing agile development practices (and unit testing in particular) into an existing environment and codebase.


The first example is of a very straightforward study: look at unused buildings, and how the number of broken windows varies over time, depending on what is done with them. It turns out that a building with no broken windows stays “pristine” for a long time, but that when just a few windows have been broken, many more are likely to be broken in a short space of time, as if the actions of the initial vandals give permission to other people to break more windows.


The second example is of subway trains in New York, and how an appalling level of graffiti on them in the 80s was vastly reduced in the 90s. Rather than trying to tackle the whole problem in one go by throwing vast resources at the system, or by making all the trains moderately clean, just a few trains were selected to start with. Once they had been cleaned up, they were never allowed to run if they had graffiti on them. Furthermore, the train operators noticed a pattern in terms of how long it would take the “artists” in question to apply the graffiti, and they waited until three nights’ work had been put in before cleaning the paint off. Having transformed one set of trains, those trains were easier to keep clean due to the “broken windows” effect above and the demotivating aspects of the cleaning. It was then possible to move onto the next set, get them clean and “stable”, then move on again.


I’m sure my readership (pretentious, eh?) is bright enough to see where this is leading in terms of unit testing, but this would be a fairly pointless post if I stopped there. Here are some guidelines I’ve found to be helpful in “test infecting” code, encouraging good practice from those who might otherwise be sloppy (including myself) and keeping code clean once it’s been straightened out in the first place. None of them are original, but I believe the examples from The Tipping Point cast them in a slightly different light.


Test what you work with


If you need to make a change in legacy code (i.e. code without tests), write tests for the existing functionality first. You don’t need to test all of it, but do your best to test any code near the points you’ll be changing. If you can’t test what’s already there because it’s a Big Ball of Mud then refactor it very carefully until you can test it. Don’t start adding the features you need until you’ve put the tests in for the refactored functionality, however tempting it may be.


Anyone who later comes to work on the code should be aware that there are unit tests around it, and they’re much more likely to add their own for whatever they’re doing than they would be if they were having to put it under test for the first time themselves.


Refactor aggressively


Once code is under test, even the nastiest messes can gradually get under control, usually. If that weren’t the case, refactoring wouldn’t be much use, as we tend to write terrible code when we first try. (At least, I do. I haven’t seen much evidence of developers whose sense of design is so natural that elegance flows from their fingers straight into the computer without at least a certain amount of faffing. Even if they got it right for the current situation, the solution isn’t likely to look nearly as elegant in a month’s time when the requirements have changed.)


If people have to modify code which is hard to work with, they’ll tend to add just enough code to do what they want, holding their nose while they do it. That’s likely to just add to the problem in the long run. If you’ve refactored to a sane design to start with, contributing new elegant code (after a couple of attempts) is not too daunting a task.


Don’t tinker with no purpose


This almost goes against the point above, but not quite. If you don’t need to work in an area, it’s not worth tinkering with it. Unless someone (preferrably you) will actually benefit from the refactoring, you’re only likely to provoke negative feelings from colleagues if you start messing around. I had a situation like this recently, where I could see a redundant class. It would have taken maybe half an hour to remove it, and the change would have been very safe. However, I wasn’t really using the class directly. Not performing the refactoring didn’t hurt the testing or implementation of the classes I was actually changing, nor was it likely to do so in the short term. I was quite ready to start tinkering anyway, until a colleague pointed out the futility of it. Instead, I added a comment suggesting that the class could go away, so that whoever really does end up in that area next at least has something to think about right from the start. This is as much about community as any technical merit – instead of giving the impression that anything I had my own personal “not invented here” syndrome (and not enough “real work” to do), the comment will hopefully provoke further thought into the design decisions involved, which may affect not just that area of code but others that colleagues work on. Good-will and respect from colleagues can be hard won and easily lost, especially if you’re as arrogant as I can sometimes be.


Don’t value consistency too highly


The other day I was working on some code which was basically using the wrong naming convention – the C# convention in Java code. No harm was being done, except everything looked a bit odd in the Java context. Now, in order to refactor some other code towards proper encapsulation, I needed to add a method in the class with the badly named methods. Initially, I decided to be consistent with the rest of the class. I was roundly (and deservedly) told off by the code reviewer (so much for the idea of me being her mentor – learning is pretty much always a two-way street). As she pointed out, if I added another unconventional name, there’d be even less motivation for anyone else to get things right in the future. Instead of being a tiny part of the solution, I’d be adding to the problem. Now, if anyone works in that class, I hope they’ll notice the inconsistency and be encouraged to add any extra methods with the right convention. If they’re changing the use of an existing method, perhaps they’ll rename it to the right convention. In this way, the problem can gradually get smaller until someone can bite the bullet and make it all consistent with the correct convention. In this case, the broken windows story is almost reversed – it’s as if I’ve broken a window by going against the convention of the class, hoping that all the rest of the windows will be broken over time too.


This was a tough one for me, because I’ve always been of the view that consistency of convention is usually more important than the merit of the convention. The key here is that the class in question was inconsistent already – with the rest of the codebase. It was only consistent in a very localised way. It took me longer to understand that than it should have done – thanks Emma!


Conclusion


Predicting and modifying human behaviour is an important part of software engineering which is often overlooked. It goes beyond the normal “office politics” of jockeying for position – a lot of this is just as valid when working solo on personal code. Part of it is a matter of making the right thing to do the easy thing to do, too. If we can persuade people that it’s easier to write code test-first, they’ll tend to do it. Other parts involve making people feel bad when they’re being sloppy – which follows naturally from working hard to get a particular piece of code absolutely clean just for one moment in time.


With the right consideration for how future developers may be affected by changes we make today – not just in terms of functionality or even design, but in attitude, we can help to build a brighter future for our codebases.


The 7 Deadly Sins of Software Development

Introduction


Recently, Eric Gunnerson made a post in his blog with the idea of “the seven deadly sins of programmers”. Eric is posting his ideas for such a list one at a time, periodically. He invited others to write their own lists, however, and it was such an intriguing idea that I couldn’t resist. The list is in descending order of importance (I figure not everyone will make it to the bottom of this post) but the order is fairly arbitrary anyway. Hopefully none of this will be a surprise to most of my readership, but it’s nice to write as a sort of manifesto anyway. I’ve included a “personal guilt rating” out of 10 for each of these sins. I’m very willing to change these in the light of feedback from those with experience of working with me :)


#1 – Overengineering (in complexity and/or performance)


Personal guilt rating: historically 8, currently 3


It’s amazing how much people care about performance. They care about the smallest things, even if there’s not a chance that those things will have a significant impact in reality. A good example of this is implementing a singleton. I’ve seen the double-checked lock algorithm (as described on the page, except often broken) repeatedly brought forward as the best way to go. There’s rarely any mention of the fact that you have to get it just right (in terms of making the variable volatile or using explicit memory barriers) in order for it to be properly thread-safe. There’s no way of making it work in Java, so anyone porting code later will end up with a bug they’re very unlikely to be aware of. Yet people use it over simply locking every time on the grounds of performance. Now, acquiring an uncontested lock is very, very cheap in .NET. Yes, it’ll be slightly more expensive on multi-processor boxes, but it’s still mind-bogglingly quick. The time taken to lock, check a variable for nullity and then unlock is unlikely to cause a significant issue in almost any real world application. There may be a very, very few where it would become a problem – but developers of those applications can profile and change the code when they’ve proved that it’s a problem. Until that time, using double-checked locking just adds complexity for no tangible benefit.


That’s just a single example. People are always asking performance questions on the newsgroups which just won’t make a difference. For example, if you have a string reference in an Object expression, is it faster to cast it to String or to call the toString method? Now, don’t get me wrong – I find it very interesting to investigate this kind of thing, but it’s only as a matter of interest – not because I think it should affect what code should be written. Whatever the result, the simplest code which achieves the result in the most readable fashion is the best code until performance has proved to be an issue.


I should stress that this doesn’t extend to being stupid about performance. If I’m going to concatenate an unknown number of strings together, I’ll use StringBuilder of course – that’s what it’s designed for, and I’ve seen that it can make a huge difference in real world situations. That’s the key though – it’s evidence based optimisation. In this case it’s general past evidence, whereas for many other situations I’d use application-specific evidence, applying an optimisation which may make the code harder to read only when I’ve proved that the particular application in question is suffering to a significant extent.


The other side of this issue is complexity in terms of what the solution can achieve. These days, I mostly code for testability, knowing that if I can test that my code does what it wants to, chances are it’ll be flexible enough to meet my needs without being overly complicated. I look for complexity all over the place, particularly in terms of trying to anticipate a complicated requirement without knowing that the requirement will actually be used. For instance, if I’m writing some kind of collection (not that that happens often), I won’t add sorting capabilities until I know they’re needed. It just creates more work if you write unnecessary code. Of course, when writing libraries for public consumption, things become much trickier – you basically can’t tell how a library will be used ahead of time, so you may well end up adding features which are rarely used. The closer the communication between the code and its clients, the better. (This is also relevant in terms of performance. One area I did spend time optimising was the enhanced locking capabilities part of my miscellaneous utility library. Locking is cheap, so any replacement for “standard” locking should also be cheap, otherwise it discourages its own use.)


When I look at a design and see that it’s simple, I’m happy. When I look at implementation and see that it’s simple, I’m happy. It’s not “clever” to write complicated code. Anyone can write complicated code – particularly if they don’t check that it works. What takes time is writing simple code which is still powerful.


#2 – Not considering the code’s readership


Personal guilt rating: 4


This is actually closely linked to the first sin, but with a more human face. We know that code is generally in maintenance for longer than it’s under initial development. We know that companies often (not always, thankfully) put their “top developers” (however they decide to judge that) onto feature development rather than maintenance. These make it essential that we consider “the next guy” when writing our code. This doesn’t necessarily mean reams of documentation – indeed, too much documentation is as bad as too little documentation, as it can make the code harder to read (if it’s inline code documentation) or be hard to find your way around (if it’s external documents). One project I was working on decided to extract one document describing data formats from the main system architecture document, and we found that the extracted document became absolutely crucial to both testers and coders. That document was kept accurate, and was short enough to be easy to follow. The document from which it was extracted was rarely used.


Of course, the simpler the code, the less documentation is required. Likewise, well-written unit tests can often express the correct behaviour (and expected use) of a class more succinctly than lots of documentation – as well as being kept accurate automatically.


There are times when writing good documentation is very difficult indeed. Recently I wrote a class which did exactly the right thing, and did it in an elegant manner. However, explaining what its purpose was even in person was difficult. Understanding it from just the documentation would be almost impossible – unless the reader looked at what problem was being solved and worked through what was required, which would lead fairly naturally to the same kind of solution. I’m not proud of this. I’m proud of the class itself, but I don’t like finding myself stuck for words.


Sometimes, simple code (in terms of number of characters) is quite complicated. At one point on a project I had to find whether only a single bit was set in a long. I’m no good at remembering the little tricks involved for this kind of thing, but I’m aware they exist, and using them can be a lot more reliable than writing bit-twiddling loops to do things in a long-winded fashion. In this case, we found the appropriate trick on the web, and included a link in the code. Without the link (or at least a description in a comment) the code would have been effectively incomprehensible to anyone who didn’t recognise the trick.


Code reviews definitely help readability – when they’re done properly. In some ways, they can be better than pair programming for this, particularly if the original developer of the code doesn’t try to explain anything to the reviewer. At this point the reviewer is able to take on the role of the first person who has to maintain the code – if anything isn’t clear just from what’s available in terms of code and documentation (as opposed to human intervention) then that probably needs a bit of work. Not verbally explaining what’s happening when you’re the author is incredibly difficult to do, and I’m very bad at it. It adds time to the review, and when you’re under a lot of pressure (and who isn’t?) it can be very frustrating to watch someone painstakingly understanding your code line by line. This is not to say that code reviews shouldn’t be the source of discussion as well, of course – when I’m working with people I respect, I rarely come through a review without some changes to my code, and the reverse is true too. After all, what are the chances that anyone gets something absolutely right to start with?


#3 – Assuming your code works


Personal guilt rating: historically 9, currently 3


Ever since I heard about unit testing I’ve seen the appeal, but I didn’t start to use it regularly until 2005 when I joined Clearswift and met Stuart. Until then, I hadn’t heard about mock objects which I find absolutely crucial in unit testing. I had read articles and seen examples of unit tests, but everything seemed to fall apart when I tried to write my own – everything seemed to need something else. Of course, taken to extremes this is often a fault of the code itself, where some classes require a complete system to be up and running before they can do anything. Unit testing such a beast is difficult to say the least. In other situations you merely need to be able to specify a collaborator, which is where mock objects come in.


So, since finding out about mock objects I’ve been more and more keen on unit testing. I know it’s not a silver bullet – I know that more testing is required, both at an integration level between components, and at a system level, sometimes including manual tests. However, once I started unit testing regularly, I got a sense of just how often code which I’d assumed would work simply wouldn’t. I’ve seen examples of code which could never have possibly worked, with any input – so they can’t possibly have been used, or the results weren’t actually important in the first place.


These days, I get frustrated when I either have to work with code which isn’t under test and can’t be easily put under test due to the design (see point 7 on Eric’s list, excessive coupling) or when I’m writing code which is necessarily difficult to test. Obviously I try to minimise the amount of the code which really can’t be tested – but sometimes it’s a significant amount. Urgh.


I currently don’t have many tests around my Miscellaneous Utility Library. I’ve resolved to add tests for any new features I add or bugs I find though. Someone mailed me about a bug within the Utf32String class. In the process of writing a unit test to demonstrate that bug I found at least two or three others – and that wasn’t even trying to exercise the whole code, just enough to get to the original bug. I only mention this as a defence against the “I don’t need unit tests – my code doesn’t have bugs in it” mentality. I would really like to think that I’m a pretty good coder – but everyone mistakes. Unit tests won’t catch all of them, but it gets an awful lot. It also acts as documentation and gives you a much better base on which to refactor code into the right shape, of course…


#4 – Using the wrong tool for the job


Personal guilt rating: 3


“When all you have is a hammer, everything looks like a nail.” That’s the typical quote used when tackling this topic. I hope this sin is fairly self-explanatory. If all you know is Java and C#, you’ll find it a lot harder to solve problems which are best solved with scripts, for instance. If all you know is C, you’ll find writing a complex web app a lot harder than you would with Java or .NET. (I know it’s doable, and I used to do it – but it was really painful.) If you’re writing a device driver, you’ll find life sucks pretty hard if you don’t know C or C++, I suspect.


Using the wrong tool for the job can be really damaging in terms of maintenance. While bad code can often be refactored into good code over time (with a lot of effort) there are often significant implications in changing implementation language/technology.


This is a really good reason to make sure you keep yourself educated. You don’t need to necessarily keep up to date with all the buzzword technologies – and indeed you’d find you did nothing else if you tried to keep up with everything – but there’s always plenty to learn. Recently I’ve been looking at Windows PowerShell and Groovy. Next on my list is Squeak (Smalltalk). I’ve been promising myself that I’d learn Smalltalk for years – at a recent Scrum training course I met yet another Smalltalk evangelist, who had come from the Java side of things. It’s got to be worth trying…


#5 – Excessive code pride


Personal guilt rating: 2


I was recently pair programming and looking at some code Stuart had written a couple of weeks before. There were various bits I wasn’t sure about, but thought were probably a bit smelly, and I asked my pairing partner to include a lot of comments such as // TODO: Stuart to justify this code and // TODO: Stuart to explain why this test is useful in the slightest. It’s worth bearing in mind at this point that Stuart is significantly senior to me. With some people, comments like this would have been a career-limiting move. Stuart, however, is a professional. He knows that code can usually be improved – and that however hard we try, we sometimes take our eyes off the ball. Stuart had enough pride in his code to feel a need to fix it once the flaws had been pointed out, but not enough pride to blind him to the flaws in the first place. This appropriate level of pride is vital when you’re working with others, in my view. I don’t mind if people change my code – assuming they improve it. I expect that to happen at a good code review, if I haven’t been pairing. A code review which is more of a rubber stamp than anything else is just a waste of time.


This doesn’t mean I will always agree with others, of course. If I think my design/code is better than their suggested (or even committed!) change I’m happy to put my case robustly (and with no deference to seniority) – but usually if someone’s put in sufficient effort to understand and want to change my code in the first place, chances are they’ll think of something I haven’t.


Write code you can be proud of – but don’t be proud to the point of stubbornness. Be prepared to take ownership of code you write in terms of being responsible for your own problems – but don’t try to own it in terms of keeping other people out of it.


#6 – Failing to acknowledge weaknesses


Personal guilt rating: 2


ASP.NET, JSP, SQL, Perl, COBOL, Ruby, security, encryption, UML, VB… what do they all have in common? I wouldn’t claim to “know” any of them, even though I use some on a regular basis. They are just some of my many technological weak spots. Ask me a question about C# or Java in terms of the languages and I’ll be fairly confident about my chances of knowing the answer (less so with generics). For lots of other stuff, I can get by. For the rest, I would need to immediately turn to a book or a colleague who knows the subject. It’s important to know what you know and what you don’t.


The result of believing you know more than you actually do is usually writing code which just about works, at least in your situation, but which is unidiomatic, probably non-performant, and quite possibly fails on anything other than the data you’ve given it.


Ask for help when you need it, and don’t be afraid to admit to not being an expert on everything. No-one is an expert at everything, even though Don Box does a pretty good impression of such a person…


#7 – Speaking with an accent


Personal guilt rating: 6 (but conscious and deliberate in some places)


Some of the worst Java code I’ve seen has come from C++ developers who then learned Java. This code typically brings idioms of C/C++ such as tests like if (0==x) (which is safer than if (x==0) in C as missing out an equals sign would just cause an accidental assignement rather than a compiler error. Similarly, Java code which assumes that double and Double mean the same thing (as they do in C#) can end up behaving contrary to expectations.


This is related to sin #6, in terms of people’s natural reaction to their own ignorance: find something similar that they’re not ignorant about, and hope/assume that the similarities are enough to carry them through. In a way, this can make it a better idea to learn VB.NET if you know Java, and C# if you know VB. (There are other pros and cons, of course – this is only one tiny aspect.)


One way of trying to make yourself think in the language you’re actually writing in rather than the similar language you’re more familiar with is to use the naming conventions of the target language. For instance, .NET methods are conventionally PascalCased whereas Java methods are conventionally camelCased. If you see Pascal casing in Java code, look for other C# idioms. Likewise, if you see method_names_with_underscores in either language, look for C/C++ idioms. (The most obvious C idiom is likely to be checking return codes instead of using exceptions.)


Naming conventions are the most obvious “tell” of an accent, but sometimes it may be worth going against them deliberately. For instance, I like the .NET conventions of prefixing interfaces with I, and of using Pascal casing for constants instead of JAVA_SHOUTY_CONSTANTS. It’s important when you favour this sort of “breakaway” behaviour that you consult with the rest of the team. The default should always be the conventions of the language you’re working in, but if the whole team decides that using parts of a different convention helps more than it reduces consistency with other libraries, that’s reasonable.


What isn’t so reasonable is breaking the coding idioms of a language – simply because they other languages’ idioms tend not to work. For example, RAII just doesn’t work in Java, and isn’t automatic in C# either (you can “fake it” with a using statement, but I’m not sure that really counts as RAII). Idiomatic code tends to be easier to read (particularly for those who are genuinely familiar with the language to start with) and less semantically troublesome than “imported” styles from other languages.


Conclusion


So, that’s the lot. Ask me in a year and I may well have a different list, but this seems like a good starting point. I’ve committed every sin in here at least once, so don’t feel too guilty if you have too. If you agree with them and are still breaking them, that’s worth feeling a bit guilty about. If you disagree with them to start with, that’s fair enough – add a comment to say why :)