Testing COSY’s Interval ArithmeticJun Yu and George CorlissMarquette University, Milwaukee(Martin Berz and Kyoko Makino)Outline:Testing strategyRun the tests with the old COSYShow how Maple catches the containment failureRerun the tests with the new COSYHow Sun fairedOpportunities for improvementCOSY responsesNot many (3) errors in COSY’s interval arithmeticWe welcome suggestions for improving our testsFunded in part by a subcontract from Michigan State UniversityPresented at Miami Beach Workshop on Taylor Models, December 16, 2002Why Test COSY?Spring 2002, reliable computing list [email protected] had active discussion ofCOSY InfinityRaised concerns about the reliability of interval and Taylor model arithmeticsCOSY Infinity [1], by Berz et al. available from http://cosy.pa.msu.edu [2]Arbitrary order beam dynamics simulation and analysis code using intervalarithmetic and Taylor models for the validated solution of systems of ordinarydifferential equationsBerz commissioned execution-based testing of COSY interval arithmeticWhat Is Testing?“The purpose of testing is to find errors” - Myers [8].Execution based testing cannot show the absence of errors, but can onlydemonstrate their presenceKit [7] or Kaner et al. [5] offer best practice in industrial software qualityassuranceError if a program fails to do what it is supposed to do - MyersError if a program does what it should not do - MyersWhat Is “Correct?”Fundamental tenet: Thou shalt not lie!It is an error to• violate containment• assert a mathematical falsehoodTwo violations of containment in COSY’s interval arithmetic:1. power when the exponent is not an integer, but very close to it2. (with warning) tan when the interval argument crosses discontinuityWhat Is “Correct?”Assertions of a mathematical falsehood for1. asin and acos on intervals containing ±1Questions ofDomains for interval operationsTightnessSpeedEase of useetc.Not errors. Opportunities for improved performanceTest Strategy: GoalGoal is limited: Identify violations of containment or assertions of mathematicalfalsehood.Developed a set of test cases consisting of1. an interval vector [x]2. an expression f(x)Expected results are computed a posteriori in Mapled[f([x])] is the result of challenging the COSY interval arithmetic to evaluate fon the interval [x]Seek examples x ∈ [x] for which f(x) is not ind[f([x])]Do not need to know the true containment set of f([x])Test Strategy: Maple as “Referee”Use Maple as the “referee” of containment1. Read each test case into a COSY driver2. Construct COSY intervals for the arguments3. Evaluate the expression using COSY interval arithmetic4. Write binary values of the arguments and the COSY result5. Read the binary arguments and COSY results into Maple6. Perform many point evaluations f(x) for x ∈ [x]7. Compare Maple’s f(x) with COSY enclosurefor (i = 0; i <= 10; i++) {y = INF(X) + (SUP(X) - INF(X)) * i/10.0fx = f(y)ERROR if fx is outside COSY result}Test Strategy: Roundoff?Most challenging aspect: Prevent inevitable roundoff errors from contaminatingour resultsConsider an example: Test COSY’s sin on [0.1, 0.6]Impossible to test that, since 0.1 and 0.6 are not exactly representableCannot express the question, “What is sin [0.1, 0.6]?” to COSY’s sinRoundoff errors may be introduced1. Read test cases into COSY driver2. Construct COSY interval3. Extract COSY interval bounds4. Write arguments and COSY results to a file5. Read arguments and COSY results into Maple6. Construct Maple variable precision representations7. Perform Maple operations8. Report from MapleTest Strategy: Test CasesIf an interval arithmetic package gets individual operations and intrinsicfunctions right, it will get complicated expressions right, tooTested primarily expressions composed of a single operation or intrinsic functionFor elementary operations, not matter how wide the arguments, extrema arealways at endpoints, except for division by zeroFor elementary functions, extrema are always at endpoints, except for a modestset of exceptions (e.g., sin and cos on arguments that span π or π/2), whichwe can enumerate and testMost likely to find violations of containment at endpoints of an argumentintervalsConjecture: For any rational function, if we get no violation of containment forall possible combinations of argument endpoints (and do not divide by zero),we can get no violation of containment from interior pointsProbably straightforward application of the Maximum Principle from complexvariable theoryTest Strategy: 2200+ Test CasesMost came from TOMS 737 [6]. Kearfott et al. tested their Fortran 77 INTLIBinterval arithmetic operations with a combination of specially constructed andrandomly generated argumentsAlso used 30 multi-operation expressions taken from tests of a validatedquadrature package by Corliss and Rall [3]To increase the coverage of our tests of binary operations, each pair ofarguments was used in several combinations. For example for addition andsubtraction, argument intervals [a] and [b] give test cases• [a] + [b], [a] − [b], [−a] + [b], [−a] − [b]• [−a] + [−b], [−a] − [−b], [a] + [−b], [a] − [−b]• [b] + [a], [b] − [a], [−b] + [a], [−b] − [a]• [−b] + [−a], [−b] − [−a], [b] + [−a], [b] − [−a]Test Strategy: 2200+ Test CasesFor multiplication, with 0 ≤ [a, a] and 0 ≤hb, bi, we test 16 combinations:• [a, a] ×hb, bi, [−a, a] ×hb, bi, [−a, −a] ×hb, bi, [−a, a] ×hb, bi• [a, a] ×h−b, bi, [−a, a] ×h−b, bi, [−a, −a] ×h−b, bi, [−a, a] ×h−b, bi• [a, a] ×h−b, −bi, [−a, a] ×h−b, −bi, [−a, −a] ×h−b, −bi,[−a, a] ×h−b, −bi• [a, a] ×h−b, bi, [−a, a] ×h−b, bi, [−a, −a] ×h−b, bi, [−a, a] ×h−b, biRun Tests on June 8 COSYPOWER near an integerSee 2arith inpPOW.txtruntest POWSee 2arith resPOW.txtMaple 2arith binPOW.txtTAN crossing discontinuitySee 2arith inpTAN.txtruntest TANSee 2arithresTAN.txtASIN or ACOS at ±1See 2arith inpASIN1.txtruntest ASIN1similarly for [-1, 1] and ACOSNoteworthy: List is short and fixableHow Did Sun’s F95 Compiler Do?Same tests ported to Sun’s F95 compilerError: tanh (negative), e.g., tanh ([-4.879, -4.267])Fails by 1-2 UPL’sSee 1sun.f95, Maple/testfort.mwsSun fixed within a weekDiscrepancy between production and developmentPOWER: COSY ResponseMartin Berz:Observation: When the power operation is called with an