Research Hypotheses and Multiple Regression: 2PowerPoint PresentationSlide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12Slide 13Research Hypotheses and Multiple Regression: 2•Comparing model performance across populations•Comparing model performance across criteriaComparing model performance across groups This involves the same basic idea as comparing a bivariate correlation across groups• only now we’re working with multiple predictors in amultivariate modelThis sort of analysis has multiple important uses …• theoretical – different behavioral models for different groups?• psychometric – important part of evaluating if “measures” are equivalent for different groups (such as gender, race, across cultures or within cultures over time) is unravelingthe multivariate relationships among measures & behaviors• applied – prediction models must not be “biased”Comparing model performance across groups There are three different questions involved in this comparison … Does the predictor set “work better” for one group than another?• Asked by comparing R2 of predictor set from the 2 groups ?• we will build a separate model for each group (allowingdifferent regression weights for each group) • then use Fisher’s Z-test to compare the resulting R2sAre the models “substitutable”?• use a cross-validation technique to compare the models• use Steiger’s t-test to compare R2 of “direct” & “crossed” modelsAre the regression weights of the 2 groups “different” ?• use Z-tests to compare the weights predictor-by-predictor• or using interaction terms to test for group differencesThings to remember when doing these tests!!!• the more collinear the variables being substituted, the more collinear they will be -- for this reason there can be strong collinearity between two models that share no predictors• the weaker the two models (lower R²), the less likely they are to be differentially correlated with the criterion• nonnill-H0: tests are possible -- and might be more informative!!• these are not very powerful tests !!!• compared to avoiding a Type II error when looking for a given r , you need nearly twice the sample size to avoid a Type II error when looking for an r-r of the same magnitude• these tests are also less powerful than tests comparing nested modelsSo, be sure to consider sample size, power and the magnitude of the r-difference between the non-nested models you compare !Group #1 (larger n)“direct model” R²D1 y’1 = b1x + b1z + a1“direct model” R²D2 y’2 = b2x + b2z + a2Group #2 (smaller n)Comparing multiple regression models across groups  3 ?sDoes the predictor set “work better” for one group than another? Compare R²D1 & R²D2 using Fisher’s Z-test• Retain H0: predictor set “works equally” for 2 groups• Reject H0: predictor set “works better” for higher R2 groupRemember!!We are comparing the R2 “fit” of the models…But, be sure to use R in the computator!!!!Group #1 (larger n)“G1 direct model” R²D1 y’1 = b1x + b1z + a1“G1 crossed model” R²X1 y’1 = b2x + b2z + a2using Hotelling’s t-test or Steiger’s Z-test will need rDX -- correlation between models – from each groupGroup #2 (smaller n)Are the multiple regression models “substitutable” across groups?Apply the model (bs & a) from Group 2 to the data from Group 1“G1 crossed model” R²X2 y’1 = b2x + b2z + a2Apply the model (bs & a) from Group 1 to the data from Group 2“ G2 direct model” R²D2 y’2 = b2x + b2z + a2Compare R²D2 & R²X2Compare R²D1 & R²X1Are the regression weights of the 2 groups “different” ?•test of an interaction of predictor and grouping variable•Z-tests using pooled standard error termsAsking if a single predictor has a different regression weight for two different groups is equivalent to asking if there is an interaction between that predictor and group membership.(Please note that asking about a regression slope difference and about a correlation difference are two different things – you know how to use Fisher’s Test to compare correlations across groups)This approach uses a single model, applied to the full sample…Criterion’ = b1predictor + b2group + b3predictor*group + aIf b3 is significant, then there is a difference between then predictor regression weights of the two groups.However, this approach gets cumbersome when applied to models with multiple predictors. With 3 predictors we would look at the model … y’ = b1G + b2P1 + b3G*P1 + b4P2 + b5G*P2 + b6P3 + b7G*P3 +aEach interaction term is designed to tell us if a particular predictor has a regression slope difference across the groups.Because the collinearity among the interaction terms and between a predictor’s term and other predictor’s interaction terms all influence the interaction b weights, there has been dissatisfaction with how well this approach works for multiple predictors.Also, because his approach does not involve constructing different models for each group, it does not allow…•the comparison of the “fit” of the two models•an examination of the “substitutability” of the two modelsAnother approach is to apply a significance test to each predictor’s b weights from the two models – to directly test for a significant difference. (Again, this is different from comparing the same correlation from 2 groups).The most common formula is … bG1 - bG2 Z = ------------------ SE b-differenceHowever, there are competing formulas for “SE b-difference “The most common formula (e.g., Cohen, 1983) is… (dfbG1 * SEbG12) + (dfbG2 * SEbG22) SE b-difference = --------------------------------------------- √ dfbG1 + dfbG2However, work by two research groups has demonstrated that, for large sample studies (both N > 30) this Standard Error estimator is negatively biased (produces error estimates that are too small), so that the resulting Z-values are too large, promoting Type I & Type 3 errors.•Brame, Paternost, Mazerolle & Piquero (1998)•Clogg, Petrova & Haritou (1995) Leading to the formulas … SE b-difference = √ ( SEbG12 + SEbG22 )and… bG1 - bG2 Z = --------------------------- √ ( SEbG12 + SEbG22 )Match the question with the most direct test…Practice is better correlated to performance for novices than for experts.The structure