Evaluation of Biometric Identification SystemsThe Biometric Test CenterSummary of PresentationThree Basic Biometric OperationsThe Three Biometric OperationsEnrollmentVerificationIdentificationCOMPARE operationDistance MeasuresVariations on the basic measurement planBinningMeasures of EffectivenessDistributions of Large Ensemble of CandidatesIntegrated DistributionsCross-over ThresholdChanging the Device ThresholdThe d-prime MeasureComparison Rate MeasuresPenetration RateExample: fingerprintsPowerPoint PresentationBin Error RateCollection VariablesSlide 25Liveness IssueCollection Variables -- FingerprintsCollection Variables - Hand GeometryCollection Variables - Iris IdentificationCollection Variables - Palm PrintCollection Variables - Face RecognitionCollection Variables - Voice RecognitionExample - Miros Face Recognition SystemExample - FaceitTM Face Recognition SystemEvaluation StrategiesCommon FactorsConvenience FactorsCollecting a Template Database for TestingPractical DatabasesSome ResultsHand Geometry for INSPASSSlide 42FaceitTM General CommentsFaceitTM Face RecognitionSlide 45FaceitTM Study SummaryMirosTM Face RecognitionIriscanTM Recognition SystemSlide 49IriscanTM ObservationsConclusionsWeb Sites1Evaluation of Biometric Identification SystemsDr. Bill Barrett, CISE department andUS National Biometric Test CenterSan Jose State Universityemail: [email protected] Biometric Test Center•Funded by several federal agencies•Centered in a disinterested university setting•Provide objective evaluations of commercial biometric instruments•Provide consulting services to sponsors regarding the most effective application of biometric instruments•No funds accepted from vendors•No independent competing research3Summary of Presentation•Three Basic Biometric Operations•Measures of effectiveness - the ROC curve•Comparison Rate Measures•Collection Variables•Evaluation strategies•Testing issues•Some results•Conclusion4Three Basic Biometric Operations5•Enrollment: first time in.•Verification: does this credit card belong to this person?•Identification: who is this person, anyway?The Three Biometric Operations6Enrollment7Verification8Identificationmatchmatchno matchno match9COMPARE operation•Yields a DISTANCE measure between candidate C and template T•d = distance(C, T)•d LARGE: C probably is NOT T•d SMALL: C probably IS T•NOTE: is reversed for fingerprints10Distance Measures•Euclidean•Hamming•Mahalonobis11Variations on the basic measurement plan•3 strikes and you’re out•Multiple templates of same person•Template replacement over time•Template averaging•Binning12Binning•Find some way to segment the templates, e.g.•male/female•particular finger•loop vs. whorl vs. arch•May have to include the same template in different bins•Improves search performance, may reduce search accuracy (more false non-matches)13Measures of Effectiveness14Distributions of Large Ensemble of Candidates15Integrated Distributions16Cross-over Threshold•tc = cross-over threshold•where probability of false match: I = probability of false rejection: A17Changing the Device Threshold•td > tc : reduces false rejection: A increases false match: I(bank ATM choice)•td < tc : increases false rejection: Areduces false match: I(prison guard choice)18The d-prime Measure2/)('222121d•Measures the overall quality of a biometric instrument.•d’ usually in the range of 2 to 10, logarithmic, like the Richter Scale.•Assumes normal distribution.19Comparison Rate Measures20Penetration Rate•Percentage of templates that must be individually compared to a candidate, given some binning.•Search problem: usually exhaustive search, with some comparison algorithm, no reliable tree or hash classification.•Low penetration rate implies faster searching21Example: fingerprintsAFIS (FBI automated classification system) classifies by:•Left loop/ right loop•Arch/whorl•UnknownThen•Exhaustive search of the subset of prints22Jain, Hong, Pankanti & Bolle,An Identity-Authentication System Using Fingerprints,Proc. IEEE vol. 85, No. 9, Sept. 199723Bin Error Rate•Probability that a search for a matching template will fail owing to an incorrect bin placement•Related to confidence in the binning strategy•AFIS Bin error typically < 1%24Collection Variables25Collection Variables•Physical variations during biometric collection that may change the measurement•Translation/scaling/rotation usually compensated in software•Tend to increase the width of the authentics distribution, and thus•...make it easier to get a false rejection•...cause a smaller d’26Liveness IssueCan the device detect that the subject is live?•Fake face recognition with a photograph?•...or a rubber print image (fingerprint)?•...or a glass eye (iris encoding)?27Collection Variables -- Fingerprints•Pressure•Angle of contact•Stray fluids, film buildup•Liveness28Collection Variables - Hand Geometry•Finger positioning (usually constrained by pins)•Rings•Aging•Liveness29Collection Variables -Iris Identification•Lateral angle of head•Focus quality•Some people have very dark irises; hard to distinguish from pupil•Outer diameter of iris difficult to establish•Eyelids, lashes may interfere•NO sunglasses•Liveness can be established from live video30Collection Variables -Palm Print•Pressure•Stray fluids, film buildup•Liveness31Collection Variables -Face Recognition•3D angles•Lighting•Background•Expression•Hairline•Artifacts (beard, glasses)•Aging•Liveness: smiling, blinking32Collection Variables -Voice Recognition•Speed of delivery•Articulation•Nervousness•Aging•Laryngitis•Liveness: choose speech segments for the user to repeat, i.e. “Say 8. Say Q. Say X”33Example - Miros Face Recognition System•Lighting is specified•Static background, subtracted from candidate image to segment face•Camera mounted to a wall - standing candidate•Height of eyes above floor used as an auxiliary measure•Verification only recommended•Liveness - can be fooled with a color photograph34Example - FaceitTM Face Recognition System•No particular lighting specified; it expects similar lighting & expression of candidate and template•Face segmented from background using live video•Face lateral angles not well tolerated•Liveness: blinking, smiling test35Evaluation Strategies36Common Factors•Bio