Justin du Coeur (jducoeur) wrote,
Justin du Coeur

Is there a good "prioritization annealing" algorithm?

This is a very long-term question -- there's no chance anything will be done about it in the next six months, and it's likely to be a year. But it occurred to me, and it's an interesting question, so I toss it out to the Lazyweb for pointers and/or thoughts.


In Agile development, Estimation is almost always a group project -- we figure out how much effort a given User Story is likely to take by playing Planning Poker or something similar. But Prioritization -- deciding what order we're going to tackle things in -- typically isn't: most often, it is a single Product Manager sweating bullets, rearranging the Story Stack every hour or two and getting tagged to describe the current state of the world at the beginning of each sprint.

(Note: I am assuming a proper prioritized Story Stack -- the User Stories are arranged in strict order of priority, defined however you like, and the team pulls the top stories off the stack for each sprint unless there is a good reason to do otherwise. I have never seen another system that actually works *well*, at least for the near-term stories. For more distant stuff you can be more coarse-grained, and simply mark things High/Medium/Low, or something like that.)

There are good reasons for this approach, and I think it's always the way to make the final decisions. But what about when you have a project like Querki, with literally hundreds of user stories (indeed, I suspect I'll go over a thousand before the end of the year), not nearly enough resources, and you want deep community involvement in the prioritization process?

I've seen sites that do something simplistic for this, with a simple thumbs-up, or maybe up/down voting. I don't think it actually works all that well, though, because it's prone to all sorts of systematic problems -- it's very easy to game, it's prone to the highly-voted items getting more up-votes, the newest items tend to become trendy, and so on. The Law of Large Numbers helps, but my sense is that it's a pretty weak system for making *good* decisions. (It's perfectly fine for making the community *feel* involved, but not as much for working with them to make good prioritization decisions.)

So it occurred to me that it might make more sense, instead of treating it as "voting", to literally have the community participate in prioritization -- just at a small enough grain to be practical.

A Possible Design

Here's a strawman scenario for your consideration:

As an active member of the community, I go to the "Prioritize" page. The page offers me five random User Stories in a drag-and-drop list. I can click on each to see more information about it, and I rearrange the list to match my *personal* opinion of what their priority order is. I press "Done" to record my vote, or maybe "Done and Give Me Another" to keep playing the game with a new list. My opinion then gets fed into an overall Querki-wide list, which combines all of these votes to come up with a current master state of things.

There are many possible tweaks -- tracking my personal "priority order" or not, being able to see and/or work with a larger list, letting me simply declare "Pass" on the entire list because I don't care about any of these -- but that's the core idea I'm pondering: taking lots of little easy-to-digest lists and combining them into a big one.

Personally, I think this would not only be a good way to get the community genuinely involved, I believe it would actually be kinda *fun*. I could actually see myself getting a lot more involved doing this than I would in the usual upvote systems.

The Problem

Plain and simply, the issue is: how does it actually *work*? What is the algorithm to take these individual snippets of prioritization, and combine them into a single gigantic list that represents the gestalt feelings of the community?

I'm sure that I could come up with *some* sort of adequate approach if I sleep on it for a day or two. But I'm wondering whether there are any well-established algorithms for this sort of problem. In airy theory, the question is how to take a collection of inconsistent ordered sublists, and combine them into a single best-fit complete list. Intuitively, it feels kind of like a statistical problem, but despite three years at Memento, my statistics are quite weak. I'd love to find that there are already some well-established best practices, and maybe even a proof of a "best" algorithm within certain constraints.

Suggestions? This is a case where, while I probably could reinvent the wheel, I suspect it's smarter not to...
Tags: agile, programming, querki

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded