Success in Scrum: Real Metrics or Wild-Ass Guesses?

There is a lot of controversy over how to measure success in a Scrum project.  Some say it’s impossible and opt to build software via the traditional waterfall methodology where great metrics abound. But are the plethora of metrics available in waterfall development truly “great”…or are they just familiar?

In traditional Waterfall projects, requirements are finalized in advance of development and the project team then estimates how many resources will be needed and how long it will take to complete all the features defined in the requirements document.  Thus, the three points of the iron triangle are formed – scope, budget, and schedule – and now the success of the project can be measured by how close the project comes to completing all requirements within the estimated cost and timeline.

Taking a closer look, these metrics inevitably fall apart because:

  • Requirements always change as the project progresses and
  • Time estimates always change as software complexities are revealed

A project’s budget then simply becomes a function of luck:

  • How successful were we in padding our estimates so we can handle the changing requirements and hidden software complexities?
  • Did we budget enough money to bring in extra help if that’s even a possibility?

Metrics built on chance are not a good way to measure success, yet organizations have been using them for years to measure just this!

Compare this with Scrum where we don’t have a pre-defined full set of requirements but instead help the customer define the most critical 20% of functions that can provide 80% of the value to customers and then, based on that, build project teams to accomplish that development.  We iterate over 1-4 weeks and show new features to the customer at the end of each iteration. When the system is reviewed, changes inevitably arise, but now they are caught early.  Customers begin to use the application sooner, more fully defining the direction the software should go in, ultimately allowing us to build exactly what they need.

Now don’t get me wrong – there are still budgets, timelines, and an overall scope of a project in Scrum BUT focusing on maximizing value and eliminating waste allows for a process fluidity that makes measurements seemingly less defined.  Acknowledging and accepting the fluidity allows us to utilize better metrics.  Instead of looking at how closely a project team adhered to a particular set of requirements in a set time and on a predefined budget, Scrum metrics get upper management much closer to measuring success:

Project Team Velocity

Looking at the sprint burndown rate allows a project team’s success to be measured over time.  The sprint burndown rate measures how quickly a team addresses a block of work.  The goal here is to increase project pace over time as a team gets better at working together.

Overall Project Velocity

Looking at the epic burndown rate over time allows the entire project to be measured including the participation of all stakeholders.  The epic burndown rate measures how quickly the entire organization can define, clarify, and prioritize requirements and get the work completed.  The goal here is to get a health report on how well the organization is supporting the project.

Customer Satisfaction

This is the ultimate goal of every software project, right? Regularly interacting with customers helps determine their satisfaction with both the process and the results, providing a critical metric tied directly to the success of a project. Is the project team building the software the customer needs at the current time (as opposed to the software the customer thought that they needed at the start of the project)?  Is the software value maximized for time and cost?

These metrics used in Scrum may not be the same comfortable metrics that everyone knows and loves from waterfall development, but I believe they do a better job achieving our ultimate goal – making great software, delighting our customers, and enhancing business value!

Written by 

Jessica Mazzeo is Crossfire's trusty CEO and leads our software development operations. She has been responsible for the growth of our software business since 2000. She lives and breathes all things software which makes her very entertaining at some dinner parties but not so much at others.

Leave a Reply

Your email address will not be published. Required fields are marked *