Critical Systems ValidationObjectivesTopics coveredValidation of critical systemsValidation costsReliability validationThe reliability measurement processReliability validation activitiesStatistical testingReliability measurement problemsOperational profilesAn operational profileOperational profile generationReliability predictionEqual-step reliability growthObserved reliability growthRandom-step reliability growthGrowth model selectionSlide 19Safety assuranceSafety confidenceSafety reviewsReview guidanceSafety argumentsConstruction of a safety argumentInsulin delivery codeSafety argument modelProgram pathsProcess assuranceSafety related process activitiesHazard analysisHazard log entryRun-time safety checkingInsulin administration with assertionsSecurity assessmentSecurity validationSecurity checklistSafety and dependability casesThe system safety caseComponents of a safety caseArgument structureInsulin pump argumentClaim hierarchyKey pointsSlide 45©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 1Critical Systems Validation©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 2ObjectivesTo explain how system reliability can be measured and how reliability growth models can be used for reliability predictionTo describe safety arguments and how these are usedTo discuss the problems of safety assuranceTo introduce safety cases and how these are used in safety validation©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 3Topics coveredReliability validationSafety assuranceSecurity assessmentSafety and dependability cases©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 4Validation of critical systemsThe verification and validation costs for critical systems involves additional validation processes and analysis than for non-critical systems:•The costs and consequences of failure are high so it is cheaper to find and remove faults than to pay for system failure;•You may have to make a formal case to customers or to a regulator that the system meets its dependability requirements. This dependability case may require specific V & V activities to be carried out.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 5Validation costsBecause of the additional activities involved, the validation costs for critical systems are usually significantly higher than for non-critical systems.Normally, V & V costs take up more than 50% of the total system development costs.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 6Reliability validationReliability validation involves exercising the program to assess whether or not it has reached the required level of reliability.This cannot normally be included as part of a normal defect testing process because data for defect testing is (usually) atypical of actual usage data.Reliability measurement therefore requires a specially designed data set that replicates the pattern of inputs to be processed by the system.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 7The reliability measurement processComputeobservedreliabilityApply tests tosystemPrepare testdata setIdentifyoperationalprofi les©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 8Reliability validation activitiesEstablish the operational profile for the system.Construct test data reflecting the operational profile.Test the system and observe the number of failures and the times of these failures.Compute the reliability after a statistically significant number of failures have been observed.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 9Statistical testingTesting software for reliability rather than fault detection.Measuring the number of errors allows the reliability of the software to be predicted. Note that, for statistical reasons, more errors than are allowed for in the reliability specification must be induced.An acceptable level of reliability should be specified and the software tested and amended until that level of reliability is reached.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 10Reliability measurement problemsOperational profile uncertainty•The operational profile may not be an accurate reflection of the real use of the system.High costs of test data generation•Costs can be very high if the test data for the system cannot be generated automatically.Statistical uncertainty•You need a statistically significant number of failures to compute the reliability but highly reliable systems will rarely fail.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 11Operational profilesAn operational profile is a set of test data whose frequency matches the actual frequency of these inputs from ‘normal’ usage of the system. A close match with actual usage is necessary otherwise the measured reliability will not be reflected in the actual usage of the system.It can be generated from real data collected from an existing system or (more often) depends on assumptions made about the pattern of usage of a system.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 12An operational profile...Number ofinputsInput classes©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 13Operational profile generationShould be generated automatically whenever possible.Automatic profile generation is difficult for interactive systems.May be straightforward for ‘normal’ inputs but it is difficult to predict ‘unlikely’ inputs and to create test data for them.©Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 24 Slide 14Reliability predictionA reliability growth model is a mathematical model of the system reliability change as it is tested and faults are removed.It is used as a means of reliability prediction by extrapolating from current data•Simplifies test planning and customer negotiations.•You can predict when testing will be completed and demonstrate to customers whether or not the reliability growth will ever be achieved.Prediction depends on the use of statistical testing to measure the reliability of a system