Single sample hypothesis testing, IIOutlineBrief reviewKey ConceptsWhy bother with Ha at all?OutlineA Tale of Two TailsThe p-value for a test of Ho against:How do you decide to use a one- or two-tailed approach?One Tail or Two? The moderate approach:Examples of the moderate approachAge DistributionCost of an Ivy EducationNumber of DoodlesOne tail or two? The moderately conservative approach:One tail or two? The moderately conservative approach:The moderately conservative approach: a possible exampleOne tail or two: The conservative approachOutlineSignificance testing for small samplesExample t-testSpectrophotometer calibrationCalculate the test statisticCalculate the test statisticFind the p-valueFind the p-valueReport the resultsOutlineSignificance and multiple tests (from the last lecture)Significance and multiple testsA special case of multiple tests: data snoopingData snooping: Disease clustersData snooping: One-tailed vs. two-tailed significance testingConsequences of data snooping: 1-tailed vs. 2-tailed testsData snooping & the switch to a 2-tailed testSwitching to a 2-tailed testData snooping and the switch to a 1-tailed testCorrecting for one- vs. two-tailed testsA special case of multiple tests: data snoopingOutlineWhat do our results mean?Was the result significant?Was the result important?Importance and what you are studyingImportance and what you are studyingWas the result important?Picking NPicking NDoes the difference prove the point the study was designed to test?OutlineDecisions, Decisions...Decision theory and tradeoffs between types of errorsDecision theory and tradeoffs between types of errorsA tableA tableMore on the tradeoff between Type I and Type II errorsMore on the tradeoff between Type I and Type II errorsMore on the tradeoff between Type I and Type II errorsMore on the tradeoff between Type I and Type II errorsMore on the tradeoff between Type I and Type II errorsMore on the tradeoff between Type I and Type II errorsMoving the criterion around changes the % of false alarms (a) and “hits” (1-b)Type I and Type II errorsStatistical powerAn exampleHow to figure out the power of a significance test (p. 471)First, find the criterion for rejecting the null hypothesis with a=0.05Step 2Step 2How to increase powerSingle sample hypothesis testing, II9.073/02/2004Outline• Very brief review• One-tailed vs. two-tailed tests• Small sample testing• Significance & multiple tests II: Data snooping• What do our results mean?• Decision theory and powerBrief review• Null and alternative hypothesis– Null: only chance effects– Alternative: systematic + chance effects• Assume the null is true• Given this assumption, how likely is it that we’d see values at least as extreme as the ones we got?• If it’s highly unlikely, reject the null hypothesis, and say the results are statistically significant.– The results are due to a combination of chance and a systematic effect.Key Concepts•H0and Haare contradictory (mutually exclusive)• Support for Hacan only be obtained indirectly -- by rejecting H0• Rationale:– We can never prove anything true, but we can prove something false– We know the value of the parameter given H0but not given HaWhy bother with Haat all?• The alternative hypothesis describes the condition that is contrary to the null hypothesis, and this can be directional or non-directional– Directional: The effect only occurs in a specific direction -- increases or decreases– Non-directional:The effect may be greater or less than a population parameterOutline• Very brief review• One-tailed vs. two-tailed tests• Small sample testing• Significance & multiple tests II: Data snooping• What do our results mean?• Decision theory and powerA Tale of Two Tails• Directional hypotheses are called one-tailed– We are only interested in deviations at one tail of the distribution• Non-directional hypotheses are called two-tailed– We are interested in any significant deviations from H0The p-value for a test of Ho: = o against:Ha: µ> µo is probHa: µ< µo is probHa: µ≠ µo is probz|z|zFigure by MIT OCW.µµHow do you decide to use a one- or two-tailed approach?pzobt2p• A one-tailed approach is more liberal -- it is more likely to declare a result significant.–tcrit= 1.69 5%, one-tailed–tcrit= 2.03 5%, two-tailed• There’s no one right answer as to which test to use. People will debate this point.One Tail or Two? The moderate approach:• If there’s a strong, prior, theoretical expectation that the effect will be in a particular direction (A>B), then you may use a one-tailed approach. Otherwise, use a two-tailed test.• Because only an A>B result is interesting, concentrate your attention on whether there is evidence for a difference in that direction.– E.G. does this new educational reform improve students’ test scores?– Does this drug reduce depression?Examples of the moderate approach• Is the age of this class different than the average age at MIT?• Do you pay less for an education at a state university than you do at an Ivy League college?• Is this class more boring than the norm for an MIT class?Age DistributionAgeProbability DensityCost of an Ivy Education CostProbability DensityNumber of DoodlesDoodlesProbability DensityOne tail or two? The moderately conservative approach:• The problem with the moderate approach is that you probably would actually find it interesting if the result went the other way, in many cases.– If the new educational reform leads to worsetest scores, we’d want to know!– If the new drug actually increases symptoms of depression, we’d want to know!One tail or two? The moderately conservative approach: • Only use a one-tailed test if you not only have a strong hypothesis about the directionality of the results (A>B) but if it could also be argued that a result in the “wrong tail” (A<B) is meaningless, and might as well be due to chance.• Put another way, only use a one-tailed test if you would not have been tempted, if the result went the “wrong” way, to switch to a two-tailed test (or switch the direction of your one-tailed test).• It’s tough to meet this criterion.The moderately conservative approach: a possible example• It’s known how well students typically do on a intro statistics class. • You test a new self-paced study guide, in addition to the instruction the students usually get, and have reason to believe this will