New version page

WMU EVAL 6970 - Lecture Notes

Upgrade to remove ads
Upgrade to remove ads
Unformatted text preview:

Slide 1AgendaFormulating a ProblemFormulating a ProblemFormulating a ProblemFormulating a ProblemPrimary Coding TopicsStudy Eligibility CriteriaScreening FormDevelopment of Coding ProtocolDevelopment of Coding ProtocolDevelopment of Coding ProtocolFlat File StructureHierarchical StructureMore Complex StructureMultiple Flat File Structure“Working” with Flat FilesWhat About Sub-Samples?Coding MechanicsCoding Directly to DatabaseDatabases with FormsDatabases with FormsReliability of CodingCoefficient of AgreementCoefficient of AgreementCoefficient of AgreementCohen’s KappaCohen’s KappaCohen’s KappaTraining and Calibrating CodersCommon MistakesManaging the BibliographyResearch Design ReviewResearch Design ReviewResearch Design ReviewToday’s In-Class ActivityProblem #1Problem #2Problem #3EVAL 6970: Meta-AnalysisFormulating a Problem, Coding the Literature, and Review of Research DesignsDr. Chris L. S. CorynSpring 2011Agenda•Formulating a problem•Coding the literature•Review of research designs•In-class activity•Next meetingFormulating a Problem•Like any research, meta-analysis should begin with a careful statement of the topic to be investigated or the question to be answered•This statement will guide study selection, coding of information, and data analysisFormulating a Problem•The problem statement needs to be straightforward and complete, but at this stage, need not be highly detailed•The problem statement will become clearer and more concise when eligibility criteria are developedFormulating a Problem“How effective are challenge programs in reducing the subsequent antisocial behavior of juveniles with behavior problems? What are the characteristics of the least and most successful programs? Do these programs have favorable effects on other outcomes such as relations with peers, locus-of-control, and self-esteem?”—Lipsey & Wilson (2001)Formulating a Problem•The statement of problem on the prior slide yields a preliminary specification of the research literature at issue (studies of the effects of challenge programs on juveniles with behavior problems), the major category of independent variables (program characteristics), and key dependent variables (antisocial behavior, interpersonal relationships, locus-of-control, and self-esteem)Primary Coding Topics•Eligibility criteria and screening form•Development of coding protocol•Hierarchical nature of data•Assessing reliability of coding•Training of coders•Common mistakesStudy Eligibility Criteria•Flow from research question•Identify specifics of:–Defining features of the program/policy/intervention–Eligible designs and required methods–Key sample features–Required statistical data–Geographic/linguistic restrictions, if any–Time frame, if any•Also explicitly states what is excludedScreening Form•Develop a screening form with clearly defined criteria•Complete form for all studies retrieved as potentially eligible•Modify criteria after examining sample of studies (controversial)•Double-code eligibility•Maintain database on results for each study screenedDevelopment of Coding Protocol•Goal of protocol–Describe studies–Differentiate studies–Extract findings (effect sizes if possible)•Coding forms and manual–Both importantDevelopment of Coding Protocol•Types of information to code–Report identification–Study setting–Participants–Method–Treatment or experimental manipulation–Dependent measures–Effect sizes–Confidence ratingsDevelopment of Coding Protocol•Iterative nature of development•Structuring data–Data hierarchical (findings within studies)–Coding protocol needs to allow for this complexity–Analysis of effect sizes needs to respect this structure–Flat file–Relational hierarchical fileFlat File StructureNote that there is only one record (row) per studyMultiple effect sizes handled by having multiplevariables, one for each potential effect sizeID Paradigm ES1 DV1 ES2 DV2 ES3 DV3 ES4 DV422 2 0.77 323 2 0.77 331 1 -0.1 5 -0.05 5 -0.2 1136 2 0.94 340 1 0.96 1182 1 0.29 11185 1 0.65 5 0.58 5 0.48 5 0.068 5186 1 0.83 5204 2 0.88 3229 2 0.97 3246 2 0.91 3274 2 0.86 3 -0.31 3 0.79 3 1.17 3295 2 7.03 3 6.46 3 . 3 0.57 .626 1 0.87 3 -0.04 3 0.1 3 0.9 31366 2 0.5 3Hierarchical StructureNote that a single record in the file above is “related” to five records in the file to the rightStudy Level Data FileEffect Size Level Data FileID PubYear MeanAge TxStyle100 92 15.5 27049 82 14.5 1OutcomeID ESNum Type TxN CgN ES100 1 1 24 24 -0.39100 2 1 24 24 0100 3 1 24 24 0.09100 4 1 24 24 -1.05100 5 1 24 24 -0.447049 1 2 30 30 0.347049 2 4 30 30 0.787049 3 1 30 30 0More Complex StructureStudy Level Data File Outcome Level Data FileEffect Size Level Data FileNote that study 100 has 2 records in the outcomes data file and 6 outcomes in the effect size data file, 2 for each outcome measured at different points in time (Months)ID PubYear MeanAge TxStyle100 92 15.5 27049 82 14.5 1ID OutNum Constrct Scale100 1 2 1100 2 6 1100 3 4 27049 1 2 47049 2 6 3ID OutNum ESNum Months TxN CgN ES100 1 1 0 24 24 -0.39100 1 2 6 22 22 0100 2 3 0 24 24 0.09100 2 4 6 22 22 -1.05100 3 5 0 24 24 -0.44100 3 6 6 22 21 0.347049 1 2 0 30 30 0.787049 1 6 12 29 28 0.787049 2 2 0 30 30 0Multiple Flat File Structure•Advantages–Can “grow” to any number of effect sizes–Reduces coding task (faster coding)–Simplifies data cleanup–Smaller data files to manipulate•Disadvantages–Complex to implement–Data must be manipulated prior to analysis–Must be able to select a single effect size per study for any analysis•When to use–Large number of effect sizes per study are possible“Working” with Flat FilesStudy Data FileOutcome Data FileEffect Size Data FileComposite Data FileCreatecompositedata fileSelect subset of effect sizes of interest to current analysis(e.g., a specific outcome atposttest)Verify that there is only asingle effect size per studyyesWorking Analysis FilePermanent Data FilesAverage effect sizes,further selectbased explicit criteria, orselect randomlynoWhat About Sub-Samples?•What if you are interested in coding effect sizes separately for different sub-samples, such as, boys and girls or high-risk and low-risk youth?–Just say “no”!•Often not enough of such data for meaningful analysis•Complicates coding and data structure–If you must, plan your data structure carefully•Include a full sample effect size for each dependent


View Full Document
Download Lecture Notes
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Lecture Notes and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Lecture Notes 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?