资源描述
Metrics and BenchmarkingPresented toWayne KellyPrincipal ConsultantQuality Assurance PracticeContentsIntroduction to MetricsWhat is Metrics?Why Metrics?Project Metrics Program ObjectivesMetrics-Customer NeedMeasures and MetricsImprovement CycleMeasures Data ElementsTools for MeasurementMetrics CalculationMetrics RoadmapMetrics Analysis-Examples2It is not sufficient to produce software defect metrics-like how many raised,how many outstanding,fixed by severity and urgency.We must prove that the activities of the test team are a cost effective and efficient use of staff.Test Metrics Without Tears-David VaughnQuote3 of 37Introduction to Metrics4 of 37What Is Metrics?Metrics-Derived from the word“metre”-Quantitative indicator of performance of a s/w process or product.Metrics is a cannot-do-without project management toolVisibility into project executionAnalysis of process strengths and weaknessesyou cannot manage or improve what you cannot measure“You cant control what you cant measure”-Tom DeMarco5 of 37Establish a measurement and continous improvement culture across the company.Improved quality of software accepted into productionHelp management and employees make well-informed and decisions Alert management about risks to project goals,schedules and software qualityProvides an objective assessment of the state of the software Why Metrics?6 of 37Why Metrics?ContinuedMetrics provide inputs for future estimations and planningThe actual performance of the past in quantitative terms becomes a starting point for future estimatesHelps in setting goals and targetsBetter resource planning Metrics identify the areas for improvementsFor example,metrics can identify the phase which introduces the maximum number of errors Metrics can be used to eliminate problem areas and root causes7 of 37SeniorManagementProject ManagementProjectTeamsBenchmarkingSetting improvement goalsPerformance across groupsHow large is project,Reduce defectsBetter estimates at task levelConsistency,predictabilityBetter estimation,project controlEffort,defect,schedule,sizeCost of Quality,Effort&Defect Distribution,Quality,ProductivityEstimation Accuracy,Quality,Productivity,Development Cost,ROIWhy is it MeasuredWhat is Measured Measurements drive continuous improvement across the organization8 of 37Metrics Program ObjectivesMust be clearly defined and POSITIVEDriven by critical business objectivesMeasures at the enterprise,program,project and application levelsProvide quantitative information of key drivers(i.e.schedule,estimating accuracy,defects)that improves decision making in time to affect the business outcome and successful management of a project9 of 37CUSTOMER PERSPECTIVECUSTOMER PERSPECTIVEWHAT CUSTOMER NEEDS MUST WE SERVE?”LEARNING PERSPECTIVELEARNING PERSPECTIVETO ACHIEVE OUR GOALS,HOW MUST OUR ORGANIZATION LEARN,INNOVATE AND IMPROVE?”“WHAT FINANCIAL OBJECTIVES MUST WE ACCOMPLISH?WHAT IS THE COSTPer FUNCTION POINT?”FINANCIAL PERSPECTIVEFINANCIAL PERSPECTIVE“TO SATISFY OUR CUSTOMERS,IN WHICH INTERNAL BUSINESS PROCESSES MUST WE EXCEL?”INTERNAL PERSPECTIVEINTERNAL PERSPECTIVEMultiple perspectives of a Metrics program10 of 37Measures and Metrics11 of 37MeasuresMeasures are the fundamental building blocks of a measurement systemMeasuresDefectsSizeEffortScheduleCost12 of 37MetricsMetrics are derived out of measuresMeasuresMetricsDefectsSizeEffortScheduleCostProductivityDeliveredQualityDefectRemovalEfficiencyCost ofQualityReview EffectivenessInternal Perspective13 of 37MetricsMetrics are derived out of measuresMeasuresMetricsDefectsSizeEffortScheduleCostProductivityDeliveredQualityDefectRemovalEfficiencyCost ofQualityReview effectivenessReturn onInvestmentTotal Cost of OwnershipInternal PerspectiveFinancial Perspective14 of 37Improvement CycleGatherAnalyseActImproveMeasuresMetricsDefectsSizeEffortScheduleCostProductivityDeliveredQualityDefectRemovalEfficiencyCost of QualityRevieweffectivenessReturn on InvestmentTotal Cost ofOwnershipLearningPerspective15 of 37Measures-Data ElementsDefects#of defectsDefect descriptionStage injected&detectedSeverityStatus Effort Planned vs.actualActivities(Task,review,rework)Resource groupsWeek ending datePhase wise person effortMeasuresMetricsDefectsSizeEffortScheduleCost Schedule Start date(Planned,Actual)End date(Planned,Actual)Milestone date(Planned,Actual)SizeProgram referenceLanguage used Source Lines of Code(Initial and Final)FP out of SLOCCost Per PhasePlan Vs ActGroup16 of 37Measures Tools Used to Automatically Capture the Data ElementsDefects Excel Template ClearQuest Defect LogsEffort Excel TemplateTime TrackingMS ProjectPlanViewMeasuresMetricsDefectsSizeEffortScheduleCost Schedule WBS Excel TemplateMS ProjectPlanViewSizeSLOC counters depending on the languageLabor UnitCost Project Budgets17 of 37Measures SizeExample of tools18 of 37Measures EffortExample of Tools19 of 37Measures DefectExample of Tools20 of 37Metrics Are Calculated Using MeasuresMetricsUnit of MeasureMeasuresFormulaProductivityQualityDefect RemovalEfficiencyCost of QualityReviewEffectivenessFP/pmDelivered defects/pm%SizeEffortDefectsSizeDefectsEffortDefectsSize of the product/Total Effort in person monthsDefects found duringAcceptance Testing/SizeOf the productTotal defects detected before QA/Total defects detected including QAReview+Test+Training+Rework effort *100/Total effort for the projectDefects detected by means of reviews/Defects detected by means of reviews+testing21 of 37Metrics Roadmap22 of 37Metric Program Critical Success FactorsDefine clear objectives and keep them POSITIVEDriven by key business issues Ensure that the process is rolled out effectively target a subset,test,validate and improve Begin with manual methods then,consider automation(DataDrill)Train,Train,TrainProvide time for the processes to mature;variability tends to be very high initially expect noise and changeAssist the project teams in understanding,analyzing and improving the processes through metric reportingCommunicate,Communicate,Communicate23 of 37Obtain Executive SponsorDefine the business and IT objectives for the measurement and improvement program(must be data driven)Conduct JAD session to identify possible measuresSelect measures that align with program objectives and improve IT performance Identify the information requirement,author,data source and toolsIdentify the core set or phased implementation plan for measures and validate with author,contributor and user that they align with project,department and organizational goalsImplementing A Metrics Program24 of 37Develop the metric collection and reporting solution including automation,collection,graphing and reporting processesTest,fine tune,verify and pilot all reports and measuresPackage multiple measures into a metrics releaseIdentify and train a core team of authors,contributors and users in the interpretation and use of all measures to improve business resultsProduction rollout beginsMonitor,improve,retire and implement new measures as the organization matures in process and measurement capabilitiesCapture,communicate and inform all of Program results Integrate into a Business performance ProgramImplementing A Metrics Program25 of 37 Alignment with the IT Governance and/or PMO Program Alignment with certification objectives(CMM/CMMI/Iso,etc)Definition of measurement roles and responsibilities Integration into the team,department and organizational processes Consistency in the coding and collection of metrics Production-izing the collection and reporting process Overall report design and interpretation TQM leadership and facilitation Alignment with the organizational scorecard Relevant indusrty figures for benchmarking objectivesMetrics Program Dependencies26 of 37The Business Performance Program27 of 371Initial2Repeatable3 Defined4 Managed5 OptimizingData definition and collection is standardized across the Organization Data is used to understand the project QuantitativelyEstablish PCBFocus on reducing variabilityData is collected and used in all defined processesData is systematically shared across projects Standardize metrics collection Improve metrics reliability Organization level Process DatabaseIndividual projects plan and manage data Start collection of measuresProject level metrics Data is used to evaluate and select process improvements Improve mean capabilityMetrics Roadmapin Terms of CMM LevelsPredictability Goes Up28 of 37Level 2-What you can expectDeploymentMetrics at Project level for Metrics at Project level for monitoring and controlling monitoring and controlling SQA is identified to help SQA is identified to help projects in metrics collectionprojects in metrics collectionToo much of follow-up for Too much of follow-up for datadataRegular status tracking of Regular status tracking of projects using metricsprojects using metricsBenefits Culture is being establishedCulture is being establishedProject level decisions are Project level decisions are taken based on metricstaken based on metricsTracking becomes easier in Tracking becomes easier in projectsprojectsAwareness/CultureMetrics identification and Metrics identification and definitiondefinitionOrientation on various metricsOrientation on various metricsA little cynicism from project A little cynicism from project sidesideFear of metrics usage against Fear of metrics usage against individualsindividualsInfrastructureUse of simple tools for Use of simple tools for metricsmetricsManual collation of data Manual collation of data from projectsfrom projects29 of 37Level 3-What you can expectDeploymentSupport from SQA for Support from SQA for metrics gets reducedmetrics gets reducedCapturing metrics becomes a Capturing metrics becomes a way of lifeway of lifeImprovement in goodness of Improvement in goodness of datadataBenefits Decisions are taken at Decisions are taken at organization level based on organization level based on metricsmetricsProcess improvements are Process improvements are driven based on metricsdriven based on metricsImprovements in Effort Improvements in Effort estimation accuracy,Cost of estimation accuracy,Cost of Quality,Schedule adherence,Quality,Schedule adherence,Productivity,Quality,etc.Productivity,Quality,etc.Awareness/CulturePurpose of metrics Purpose of metrics collection is known to collection is known to everyoneeveryoneCynicism goes awayCynicism goes awayMany are trained in Metrics,Many are trained in Metrics,FP counting,Estimation FP counting,Estimation using past projects datausing past projects dataInfrastructureIT wide repository for IT wide repository for metricsmetricsStandardized tools across Standardized tools across projectsprojectsCollation of data becomes Collation of data becomes easiereasier30 of 37Metrics Analysis-Examples31 of 37Example Effort analysis32 of 37Examples Defect analysis33 of 37Examples Code changes34 of 37Examples Defect analysis35 of 37Examples Defect categorization36 of 37Questions and AnswersYour Favorite and Most Effective GraphsWayne Kelly Principal ConsultantLiza Roberts Business Development M804-334-289537 of 37
展开阅读全文