Many organizations are transitioning from waterfall to agile for getter products out faster and meet customers’ expectations. But they tend to use same metrics to measure project performance.
We are all familiar with common project management metrics focused on Scope/Time/Cost – traditional/waterfall KPIs – where we answer q’s like ones below –
- Scope: Is the project’s scope in line with business expectations? Are the requirements changing? (# of CRs, KLOC, etc.)
- Time: Are we falling behind schedule? (Schedule Variance (SV))
- Cost: Are we leveraging ppl to maximum extent? (Resource over utilized, under utilized, overtime hours worked, Cost Variance (CV))
- Quality: How is the testing progressing? How are the problems identifies, reviewed and fixed? (#total defects, #defects/testing cycle, defects fix rate, etc)
- Project Performance: Have we delivered value for the amount spent? (think Earned Value Management)
- Risk: What new risks (qualitative, quantitative) have we identified? How are we mitigating risks?
Traditional project metrics never measure value delivered to customer. Also, the focus in traditional metrics is measuring activities, GANTT chart, % complete, etc.
How is Agile metrics different?
Agile metrics are empirical i.e. measurable. Either the story is done or not done unlike traditional project metric like 50% complete! Please note that in Agile there is no credit for partial completion – either it is done or not done (0% or 100%). With Agile, value is delivered incrementally to customer. The stories that deliver highest value to customer is delivered before others.
Basic measures for Agile projects
These basic metrics are primarily for the team to report on regular basis.
- Velocity = # of story points / sprint
- Actual stories planned vs. done
- Testing/QA Reports – Indicates quality of product. e.g. if the quality is bad with many defects, management can decide to augment more resources
- Technical Debt – code that is not fixe e.g. bugs, workarounds, etc.
- Burn down chart – rate at which work is being completed. Create two charts – Sprint Burn down chart & Release Burn down chart
- Resource utilization – helps answer the type and cost of resources utilized, and projected need for future sprints
Agile builds quality in definition of “doneness” (is the acceptance criteria met?), and sprint review (customer and stakeholder feedback).
Sr. Management is more interested in the Business Value of Project being delivered
How do you assess that and report that?
Agile should be assessed against KPIs which are outcome focused i.e. tied to objective (or business goals) of the project and target based. These must be forward-looking indicators to provide meaningful data to business from value accrued to date, financials, customer growth, customer acquisitions, etc. Agile project scorecard should be broadly report on 5 key areas.
1. Product/Service Value
Let me illustrate this with an example. Say, you have a business goal to “Increase customer acquisition”, which represents 40% of your business case.
- “User Registration” epic represents 60% of business value
- “Register via LinkedIn” user story realizes 30% of epic value
- Therefore, % product value delivered would be = .4 * .6 * .3 = 7.2% to date
How? using traditional EVM calculations. Earned Value Management has been a recognized project management technique, standardized by Project Management Institute’s (PMI). EVM integrates the areas of technical performance, schedule and actual cost to provide metrics for work actually accomplished. Just like traditional EVM, we need agile inputs to set the baseline prior to measuring performance –
- # of planned iterations in a release;
- total number of planned story points in a release;
- planned budget for the release
Let’s measure EVM with an example – let’s say, project has completed iteration 1 out of 5 iterations. Planned budget = $300K
|Feature||Estimate (story points)||Completed (story points)||Actual Cost ($K)|
|Product Detail page||10||10||20|
|Credit Card Verification||10|
|Order Confirmation Email||20|
EVM calculations –
- Planned % complete = Iteration # / total iterations = 1/4 =25%
- Actual % complete = Points delivered / total points = 40/200 = 20%
- Planned Value = Planned % complete * budget at completion = 25% * $300K = $75K
- Earned Value (EV) = Actual % complete * budget at completion = 20% * $300K = $60K
- Cost Performance Index (CPI) = Earned Value / Actual Cost = $60K / $65K = 0.92
- Schedule Performance Index (SPI) = Earned Value / Planned Value = $60K / $75K = 0.8
- Estimate at Completion (EAC) = budget at completion / CPI = $300k / .92 = $326K ($26K over budget)
- Estimated Completion = Planned iterations / SPI = 4 / .8 = 5 iterations
Reviewing these metrics tell you that project is in trouble because (a) EV < PV; (b) CPI <1, i.e. over budget, indicating you are not efficiently spending your dollars; (c) SPI < 1, i.e. behind schedule. $26K over budget.
When we planned, we said 4 iterations, now we will need another iteration!
In a portfolio of projects (i.e. programs), you can roll up Earned Value calculations for each project –
|Program||Total Budget||Planned Value||Earned Value||Actual Cost||CPI||SPI||EAC|
Looking at the overall program, this is in trouble too – over budget and behind schedule!
For additional information, refer to Lean & Agile Earned Value Management – David F. Rico
3. Customer KPIs
- # of leads generated
- # of new customers
- Customer Satisfaction Survey results
- Customer Retention
- # of visitors to site
- # of awards
4. Financial KPIs
- Cash Flow
- Net Present Value (NPV)
5. Business Change indicators
- Team Morale – can be measured through surveys, seeking 1:1 feedback
- User adoption
- Stakeholder engagement
Defining proper KPIs is crucial for project success and team should work collaboratively with business to jointly define “what constitutes success?”. Once defined, KPIs allow team members and stakeholders to work towards a common goal, no second guessing about what defines the project “success”.
Reporting in business language like project scorecard above, for example, will have business paying attention and engaged. Think of this as a way to bring business on board with agile way of working.
How do you measure the business value of agile adoption? Please chime in with your thoughts.