How do you measure Business value for Agile Projects?

Many organizations are transitioning from waterfall to agile for getter products out faster and meet customers’ expectations. But they tend to use same metrics to measure project performance.

We are all familiar with common project management metrics focused on Scope/Time/Cost – traditional/waterfall KPIs – where we answer q’s like ones below –

  • Scope: Is the project’s scope in line with business expectations? Are the requirements changing? (# of CRs, KLOC, etc.)
  • Time: Are we falling behind schedule? (Schedule Variance (SV))
  • Cost: Are we leveraging ppl to maximum extent? (Resource over utilized, under utilized, overtime hours worked, Cost Variance (CV))
  • Quality: How is the testing progressing? How are the problems identifies, reviewed and fixed? (#total defects, #defects/testing cycle, defects fix rate, etc)
  • Project Performance: Have we delivered value for the amount spent? (think Earned Value Management)
  • Risk: What new risks (qualitative, quantitative) have we identified? How are we mitigating risks?

Traditional project metrics never measure value delivered to customer. Also, the focus in traditional metrics is measuring activities, GANTT chart, % complete, etc.

How is Agile metrics different?

Agile metrics are empirical i.e. measurable. Either the story is done or not done unlike traditional project metric like 50% complete! Please note that in Agile there is no credit for partial completion – either it is done or not done (0% or 100%). With Agile, value is delivered incrementally to customer. The stories that deliver highest value to customer is delivered before others.

Basic measures for Agile projects

These basic metrics are primarily for the team to report on regular basis.

  1. Velocity = # of story points / sprint
  2. Actual stories planned vs. done
  3. Testing/QA Reports – Indicates quality of product. e.g. if the quality is bad with many defects, management can decide to augment more resources
  4. Technical Debt – code that is not fixe e.g. bugs, workarounds, etc.
  5. Burn down chart – rate at which work is being completed. Create two charts – Sprint Burn down chart & Release Burn down chart
  6. Resource utilization – helps answer the type and cost of resources utilized, and projected need for future sprints

Agile builds quality in definition of “doneness” (is the acceptance criteria met?), and sprint review (customer and stakeholder feedback).

 

Sr. Management is more interested in the Business Value of Project being delivered

How do you assess that and report that?

Agile should be assessed against KPIs which are outcome focused i.e. tied to objective (or business goals) of the project and target based. These must be forward-looking indicators to provide meaningful data to business from value accrued to date, financials, customer growth, customer acquisitions, etc. Agile project scorecard should be broadly report on 5 key areas.

Agile project scorecard
Executive level Reporting Dashboard for Agile Project

 

1. Product/Service Value

Let me illustrate this with an example. Say, you have a business goal to “Increase customer acquisition”, which represents 40% of your business case.

  • “User Registration” epic represents 60% of business value
  • “Register via LinkedIn” user story realizes 30% of epic value
  • Therefore, % product value delivered would be = .4 * .6 * .3 = 7.2% to date
Business value of project
Business value of project

2.Timelines

How? using traditional EVM calculations. Earned Value Management has been a recognized project management technique, standardized by  Project Management Institute’s (PMI). EVM integrates the areas of technical performance, schedule and actual cost to provide metrics for work actually accomplished. Just like traditional EVM, we need agile inputs to set the baseline prior to measuring performance –

  • # of planned iterations in a release;
  • total number of planned story points in a release;
  • planned budget for the release

Let’s measure EVM with an example – let’s say, project has completed iteration 1 out of 5 iterations. Planned budget = $300K

Feature Estimate (story points) Completed (story points) Actual Cost ($K)
Home page 10 10 15
Landing page 20 20 30
Product Detail page 10 10 20
Login 20    
Registration 20    
My Account 10    
Cart 30    
Check-out Process 20    
Invoice Calculation 10    
Credit Card Verification 10    
Payment Handling 20    
Order Confirmation Email 20    
Totals 200 40 65

EVM calculations – 

  • Planned % complete = Iteration # / total iterations = 1/4 =25%
  • Actual % complete = Points delivered / total points = 40/200 = 20%
  • Planned Value = Planned % complete * budget at completion = 25% * $300K = $75K
  • Earned Value (EV) = Actual % complete * budget at completion = 20% * $300K = $60K
  • Cost Performance Index (CPI) = Earned Value / Actual Cost = $60K / $65K = 0.92
  • Schedule Performance Index (SPI) = Earned Value / Planned Value = $60K / $75K =  0.8
  • Estimate at Completion (EAC) = budget at completion / CPI = $300k / .92 = $326K ($26K over budget)
  • Estimated Completion = Planned iterations / SPI = 4 / .8 = 5 iterations

Reviewing these metrics tell you that project is in trouble because (a) EV < PV;  (b) CPI <1, i.e. over budget, indicating you are not efficiently spending your dollars; (c) SPI < 1, i.e. behind schedule. $26K over budget.

When we planned, we said 4 iterations, now we will need another iteration!

In a portfolio of projects (i.e. programs), you can roll up Earned Value calculations for each project –

Program Total Budget Planned Value Earned Value Actual Cost CPI SPI EAC
Project 1 $300K $75K $60K $65K 0.92 0.80 $325K
Project 2 $150K $50K $35K $80K 0.44 0.70 $343K
Project 3 $2100K $700K $680K $720K 0.94 0.97 $2224K
Total $2550K $825K $775K $865K 0.90 0.94 $2846K

Looking at the overall program, this is in trouble too – over budget and behind schedule!

For additional information, refer to Lean & Agile Earned Value Management – David F. Rico

3. Customer KPIs

  1. # of leads generated
  2. # of new customers
  3. Customer Satisfaction Survey results
  4. Customer Retention
  5. # of visitors to site
  6. # of awards

4. Financial KPIs

  1. Cash Flow
  2. Net Present Value (NPV)

5. Business Change indicators

  1. Team Morale – can be measured through surveys, seeking 1:1 feedback
  2. User adoption
  3. Stakeholder engagement

Conclusion

Defining proper KPIs is crucial for project success and team should work collaboratively with business to jointly define “what constitutes success?”. Once defined, KPIs allow team members and stakeholders to work towards a common goal, no second guessing about what defines the project “success”.

Reporting  in business language like project scorecard above, for example, will have business paying attention and engaged. Think of this as a way to bring business on board with agile way of working.

How do you measure the business value of agile adoption? Please chime in with your thoughts.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s