Slide 1Slide 2Purposes of ProposalSlide 4Slide 5Slide 6Slide 7Methods ChoicesSlide 9Slide 10Primary or Secondary DataSlide 12Slide 13General Guidelines on when to use different approachesGuidelines (cont)Rest of CourseProposals & Research DesignMeasurementSamplingSurvey methodsBasic Statistics for Survey AnalysisExperimentsOther Approaches- Observation, qualitative methods1. Evaluation Research Proposal Format2. Introduction to Research DesignPrimary & Secondary DataQualitative & Quantitative ApproachesSurvey & ExperimentsOn-Site & Household designs3. Common Problems/Approaches in PRTRTopics for TodayPurposes of ProposalCommunicate with ClientDemonstrate your grasp of problemPlan the study in advance, so others can evaluate the study approach–will it work?–have you overlooked something?–will results be useful to client?– Can we afford it?Proposal Format1. Problem Statement - define program to be evaluated/problem to be studied, users & uses of results. Justify importance of the problem/study.2. Objectives : Concise listing . In evaluation studies, the objectives usually focus on the key elements of program to be evaluated & the evaluation criteria. These are the study objectives NOT the program objectives.3. Background/Literature Review - place for more extensive history/structure of program. Focus on aspects most relevant to proposed evaluation. Discuss previous studies or the relevant methods.4. Methods - details on procedures for achieving objectives - data gathering and analysis, population, sampling, measures, etc. Who will do what to whom, when, where, how and why?5. Attachments - budget, timeline, measurement instruments, etc.NOTE: Most “programs” must be narrowed to specific components to be evaluated. Think of a “Program of studies” rather than a single evaluation study. The proposal should define this specific study & how it fits into a broader program of studies.Sample Objectives1. Estimate benefits and costs of program2. Estimate economic impacts of program on local community (social, environmental, fiscal).3. Determine effects of program on target population.4. Describe users and non-users of program5. Assess community recreation needs, preferences6. Determine market/financial feasibility of program7. Evaluate adequacy or performance of programDescribe a sample or populationIdentify/test relationships between variables in a population: statistical cause-effectQuantify the relationshipAverage income of MI Snowmobilers in 1998 is $45K SB with higher incomes spend more moneyAfter safety program, SB’s have fewer accidentsSB spend per day = $25 + .4 * Income Typical Research ObjectivesVariables : any characteristic that varies across individuals in a population (i.e. takes on different values for different individuals).Dependent variable is the one you are trying to predict or explain, usually the focus of your studyIndependent variables are the ones that help explain the dependent variable.In Program evaluation, the outcomes are generally the dependent variables and characteristics of program or target populations are independent. In a cause effect relationship, cause is independent & effect dependent.Variable TerminologyMethods ChoicesOverall Approach/Design–Qualitative or Quantitative–Primary or secondary data–Survey, experiment, case study, etc.Who to study - population, sample–individuals, market segments, populationsWhat to study - concepts, measures–behavior, knowledge, attitudesCost vs Benefit of StudyQualitative vs Quantitative ApproachesQualitativeFocus GroupIn-Depth InterviewCase StudyParticipant observationSecondary data analysisQuantitativeSurveysExperimentsStructured observationSecondary data analysisQualitative vs QuantitativePurposePerspectiveProceduresQuantitativeGen’l LawsTest HypothesesPredict behaviorOutsider-ObjectiveStructured formal measures probability samples statistical analysisQualitativeUnique/Individual caseUnderstandingMeanings/IntentionsInsider-SubjectiveUnstructured open ended measures judgement samples interpretation of dataPrimary or Secondary DataSecondary data are data that were collected for some purpose other than your study, e.g. government records, internal documents, previous surveysChoice between Primary /Secondary Data–Costs (time, money, personnel)–Relevance, accuracy, adequacy of dataHow ....WhereGatheredHousehold On-Site LaboratoryPersonalInterviewSurveys Surveys,Field ExpmtsFocus GroupsTelephone/ComputerSurveys ComputerInterviewsComputerInterviewsSelf-Admin.Quest.Surveys,Field ExpmtsExperimentsObservation& TracesNA ObservableCharacteristicsObservableCharacteristicsSecondarySourcesNA InternalRecordsNAResearch Designs/Data Collection ApproachesSurvey vs ExperimentSurvey - measure things as they are, snapshot of population at one point in time, generally refers to questionnaires(telephone, self-administered, personal interview)Experiment - manipulate at least one variable (treatment) to evaluate response, to study cause-effect relationships(field and lab experiments)General Guidelines on when to use different approaches1. Describing a population - surveys2. Describing users/visitors - on-site survey3. Describing non-users, potential users or general population - household survey4. Describing observable characteristics of visitors - on-site observation5. Measuring impacts, cause-effect relationships - experimentsGuidelines (cont)6. Anytime suitable secondary data exists - secondary data7. Short, simple household studies - phone8. Captive audience or very interested population - self-administered survey9. Testing new ideas - experimentation or focus groups10. In-depth study - in-depth personal interviews, focus groups, case