Voting on implementation effort

I was recently at the International Conference on Information Systems Design at the National University of Ireland in Galway. One of the more interesting presentations was from Hans Mitlöhner. He suggested that you could use ordinal rankings and voting rules to develop effort estimates for software development projects with comparable accuracy to the more complicated but well known COCOMO method* (here are the slides and the paper). It was an interesting idea but I think it requires a slightly different focus to be an effective estimation technique.

The idea was that you could think of each of the different COCOMO attributes as a voter. Each of these attributes (or voters) ranks the new project relative to the other projects. So for instance you rank all the projects based on the required reliability, then on programmers' capability and so on. These rankings (or votes) are then aggregated based on voting rules: simple majority, maximin, copeland or borda. Based on these rankings you can come up with an accurate estimate for how much effort it will take to develop the software. Using, for instance, simple majority voting and just 5 of the COCOMO attributes (reliability, turn-around time, analyst's capability, programmer's capability, schedule constraints) this method produced estimates of comparable to the full COCOMO method (i.e. 30% accuracy).

It is presumably easier to rank projects rather than to calculate the attributes completely. Coming up with a numeric estimate for some of the COCOMO attributes is a time-consuming and produces precise numbers for things that are somewhat ambiguous in nature. For that reason its plausible that coming up with a simple ranking will in many circumstances be easier and more transparent than using the normal COCOMO method.

In my view, there are two problems with the method:

• Most companies won't have enough projects to come up with a representative sample that could be used to apply this method. This estimation method seems to require a reasonable number of cases to be used as reference points that the people in the project are familiar with before it is applicable.
  
• Even if they do have enough projects that the method could be applied effectively, would there be people in the company with enough knowledge about each of those cases that they could ranking each project accurately on each of the attributes.

Where I think this method may, however, be useful is in aiding the estimation of work effort of "cards" in extreme programming. In extreme programming techniques changes are written out on cards which are then assigned an effort estimate. These effort estimates are usually based on the intuition of the software developer. In my current workplace we have a problem that these estimates aren't terribly accurate and that there is a large optimistic bias (we usually estimate too little time for each card rather than too much).

Each week we work through a fairly large number of these cards, maybe twenty or so. The developers, especially the developer that worked on the card, can give a fairly accurate ranking of the card's difficulty across different criteria. This means that, while this effort estimation method may not be useful at the level of whole projects, it may be possible to apply it at the level of individual cards. Especially if there was software support for the ranking process to make it easier to apply the method.

The next question I guess is do we really want better estimates of development effort? Some people say no... but that's a topic for another time.

*COCOMO is a technique for estimating the effort required to implement a particular piece of software and was originally developed by Barry Boehm. For more information see this Wikipedia article.