Jacklyn Giampa Lecture 9 2 25 16 Step 3 of the data analysis plan For most research that is done today inferential statistics are used such as a t test and analysis of variance Purpose of inferential statistics To aid the researcher in making a decision about whether the differences in the DV are large enough to reflect a true effect of the IV or are actually a result of chance alone Null hypothesis significance testing 1 H0 null hypothesis a hypothesis of no difference In reality the samples come from the same population and differ only due to chance There is no effect of the IV Of if I redo this study I might or might not get the same results 2 Ha alternative hypothesis says the means are truly different as a result of the effects of the different levels of my IV The samples come from different populations There is an effect of the IV If I redo this I will find the same results Type I II Errors and Power Pg 387 of text book H0 false H0 true Reject H0 H0 is false Ha is more likely Correct decision power 1 Type I error P type I error alpha Fail to reject H0 is true Type II error Power the probability of correctly rejecting a false null hypothesis There really is an effect of the IV and it was correctly measured Used to describe the ability of a particular statistical test to detect a true effect of an IV Sensitivity similar to power used to describe the likelihood that a design will be able to correctly detect a true effect of the IV Type I error when you claim there is an effect of the IV but in reality the differences were due to chance alone Type II error when you miss a true effect of the IV What effects power 1 Sample size sample size increases power increases 2 Effect size size increases power increases 3 Statistical test used 4 Alpha affects power increases power increases Alpha Traditionally set alpha at 05 at the highest or somethings at 01 Arbitrary but where we set it at Why would you want alpha to be low Less likely to make a type I error Want to avoid saying IV had an effect when it did not If alpha is high we increase the amount of research in our data base that implies particular IVs have effects on DVs when the difference observed are only due to chance Potentially damaging when using these incorrect results in an applies way Relationship between alpha beta and power Interrelated Alpha and beta have reciprocal relationship as one goes up one goes down As the chance of type I error decreases the chance of type II error increases As chance of Type I error increases the chance of type II error decreases Power 1 beta the relationship between beta and power is also reciprocal As one goes up the other goes down as power increases the chance of a type II error decreases The relationship between alpha and power is direct As alpha goes up power moves in the same direction Alpha increases power increases Mnemonic crank shaft as you turn it alpha and power move together beta does the opposite Statistical heresy You as a researcher have control over your alpha While the traditional levels for alpha are 05 or 01 there could be practical situations that allow for a higher level for alpha 05 is truly arbitrary Example number 1 Situation 1 new drug is cheaper than old drug and has no major side effects You have a new drug for severe depression You have two groups old vs new drug New drug group shows considerable improvement over old drug but the significance level is 0556 Say there is no significant difference between the old and the new drug What do we do Use new drug or stick to old But 05 is arbitrary so in this case you should follow your findings Change mental alpha to 06 Effect If you increase alpha beta will decrease but power will increase Borderline significant or marginally significant cannot claim it is 100 statistically significant Situation 2 new drug is expensive and have bad side effects Again analyze and get 0556 What do we do If this is a potentially dangerous drug change your decision Lower 0556 because you need to be absolutely sure Lower alpha to 01 to make sure it will be effective Want to avoid type I error Lowering alpha would increase beta and decrease power Would not want to risk people to the dangerous side effects unless certain that the drug was more effective than the old drug Power Power tells us the probability of correctly detecting a real effect of an IV As effect size of magnitude increases power increases As sample size increases power increases These are aspects you CAN CONTROL Easiest way to increase power is to increase sample size Power tables allow you to estimate the sample size needed to obtain a particular level of power based on the estimated magnitude of effect for the IV you are studying and the alpha level you are using Ideally you want a power value of 80 What does power REALLY mean Suppose You conclude your results are not significant SPSS tells you your power level is 30 A study with this sample size alpha 05 and this effect size would detect an effect only 3 10 times the study was done 7 10 times you would miss seeing a real effect with this level of power Be cautious claiming no effect in this case making a type II error is at large Three steps to a statistical decision 1 Assume the null hypothesis 2 Calculate the probability of results as more extreme than those obtained under the null 3 Decide whether you are willing to accept this risk of error Decide to reject or fail to reject sometimes called retaining the null hypothesis hypothesis Finale Statistical significance does not mean a result is practically significant could be a small magnitude of effect Does not mean a result is interpretable could still be threats to internal validity Does not mean that the null is true or false it is all a gamble a game of chance Does not mean the result can be replicated could still be a result from a Type I error