You've got stories. You run sprints. You even have a daily standup. You're agile and doing Scrum, right?! So why's it not working? Most implementations I've seen have absolutely missed the point, and are failing to reap the benefits. Why does this happen, and what can we do about it?

Mike Cohn's book User Stories Applied is seen as the definitive work on user stories. It's a worthwhile read (though a bit dull in places), but it is well worth noting that he was writing it from an XP practitioner's viewpoint. There is a chapter for using stories with Scrum, but the implementing Scrum using that chapter as a step-by-step guide is likely to be counter-productive. This is unfortunate, since the general story concept is good, and the book is seen as something of an authority. Many blog posts and conference talks on "how to do scrum with stories" are based on his writing (or based on writing that's based on...), which leads to a lot of unfortunate implementations.

So, what's good? Firstly, the concept of stories is good. They're intended to capture user-centric requirements, but so little up-front detail that they force collaboration between the person with the requirement (or a good proxy) and the implementer. They provide deliverable units of work that can be given rough estimates, allowing you to establish an indication of how long a project will take and how close you are to completion. I cannot stress how important it is that you read the main body of the book, and understand what a user (or good proxy) is, and what a story isn't. A story is NOT a design document. It is not set in stone. It is a living, changing thing. It might even spawn other stories or be replaced by them. It is a place-holder for a conversation. It's all about the customer collaboration and responding to change that the agile manifesto talks about.

It's also important to realise that a story might be something other than a requirement for the end user. He talks about requirements of the purchaser of the software, and you need to work out who that might be in your environment if you're not selling software. He talks about what a good proxy for the end-user is, if you can't actually talk to end-users. There's also the concept of constraints: these stories sit on the board for the entire project, should be considered with all work. For example: "Throughout the process documentation will be produced suitable for an ISO 9001 audit" (dictated by the purchaser), or "New screens must appear within 2s in 95% of cases" (a testable translation of the customer saying "it must be fast.")

I should note that I really do like his ideas. And, after reading the book, please, please, please go and read as much of Cohn's blog as you can. It elaborates on, extends, and just plain deviates from what the book says. The rest of this blog post is about what happens if you slavishly implement that chapter on Scrum as if it's some kind of infallible rule book, without actually thinking about any of the "why."

So, where do things go wrong?

In Scrum you need to have a sprint goal. Everyone is supposed to be working towards that goal, which will tend to be a "done" iterative, potentially releasable (though not required to actually be released), improvement on the product.

Cohn's description of Scrum assumes a sprint backlog populated by taking the top N estimation points worth of items from the product backlog, where N is the number of points the team thinks that they can deliver in a sprint. In effect the "sprint goal" becomes "deliver N points."

Your scrum team is supposed to be working towards a coherent combined goal for the sprint. They're supposed to be working together to achieve it, and it's supposed to be a deliverable "Done" increment. If you take the top N points off the backlog, it's quite unlikely that you're going to be working together, instead you'll be doing a bunch of arbitrary unconnected pieces of work. You don't have a shared purpose, everyone just has their number of points. Without the goal, you lose the sense of team, and any hope of the team self-organising to achieve it. Your team will be less productive than it should be, and probably produce lower quality work.

Further, and to state the obvious, story point estimates are estimates. Over a sufficiently long period of time they give an indication of velocity, and thus the ability to plan work. The key phrase is "sufficiently long period of time." To use the literal interpretation of Cohn's "Scrum" system your sprints MUST be sufficiently long that the estimation inaccuracies average out over the course of one sprint. If they aren't, you will repeatedly have sprints that complete all their stories far too early and sprints that have huge quantities of carry over. This is, I assume, why he picks 30 days as his sprint length (any longer and it exceeds the suggested max in the Scrum guide.)

To further complicate your workload, your stakeholders may well be fighting over whose stories get done. You aren't setting sprint goals, you're taking on N points, so your product owner will be under pressure to ensure the sprint is a representative mix from all the stakeholders. This will further erode any hope of a sprint backlog that has the team working together, and put pressure on to make stories small since a "big" story wont allow a fair mix to be put into the sprint. A large number of unrelated small stories have an awful lot of overhead!

You decide to run 2 week long sprints, because the the agile manifesto seems to say that "a couple of weeks" is a preference. Plus you've heard that a bunch of other people think that's a nice length. But you find that your sprints are oscillating wildly between too little and too much work. No matter how you try to establish a good value of N, it's just not working. So, what's wrong? Well, clearly the estimates must be bad! You were told that the team will get better at estimating over time, but you still need to make them more accurate! How do you make them more accurate? Obviously they need more detail up front: how can you be expected to provide an accurate estimate if there isn't any detail in the story? How can you be expected to deliver when so much of the "sprint" is spent trying to pin down the "customer" about what they actually wanted? Clearly the "story" needs to be a fully fleshed out blueprint, preferably complete with implementation notes, with strictly defined "acceptance criteria" before it can be estimated and put in the product backlog.

This utterly defeats the point of stories, you no longer have the placeholder for a conversation, you have a design spec. Your developers will deliver as many points as they can by implementing exactly what was written, the customer will be handed exactly what was written in the spec, and they'll have to like it even if it's not really what they had in mind. Your developers will even start to demand that they have pixel perfect designs plus necessary assets attached to every story before they'll put an estimate on it. This is exactly what you were trying to escape from! It's waterfall, but without the time and effort spent on making sure that the design documents are correct.

So, does this mean stories are useless for Scrum? No, of course not. Just don't blindly follow someone else's implementation and fall into the fast waterfall trap. While, over a sufficiently long period, estimations are a powerful planning tool, they should not be used to decide what work goes into the Sprint backlog. Based on the product owner's prioritisation your team should pull work from the product backlog to build a sprint with a coherent, meaningful, goal. If you can't work out what the goal is, or the goal isn't a useful product increment, it isn't a sprint. It is entirely up to the team what and how much of that work they're prepared to put into a sprint, though the number of points vs average velocity may provide a warning signal that you've got either too much or too little work.

Another point of note is that user stories are not the whole picture. In reality, helpful while it is, not all of your work will comfortably fit the user story format. It's incredibly awkward to try to force it to do so, or you end up actually dropping important requirements because you cannot phrase them from the end user perspective. That'd be bad, so it's actually fine if your backlog does not consist entirely of user stories. Cohn advocates use of the FDD style features syntax for backlog items that don't fit the user story format. Yes, really, it's acceptable and normal to have things that aren't formatted as user stories in your backlog!

Anyone (including me) who writes about implementing a process such as Scrum is going to be influenced by what happened in their organisation. Anything they were already doing well and didn't need to change is likely to be missed from their account, while any significant change will receive much attention and may have led to them implementing a significant compromise or strict controls to modify people's behaviour. These will be different for every organisation, but what are common are the underlying principles and the "why" of Scrum, of stories, and of the Agile Manifesto. If you understand and communicate these, and evaluate any change that you intend to make against them, you stand a much better chance of avoiding the fast waterfall trap.