Outcome and Impact Evaluations Pages 167 196 Anything from the power point is in green INCOMPLETE READING BUT STILL A GREAT DEAL OF INFO There are two types of evaluations Outcome evaluation to establish whether a policy is associated with some intended outcome or set of outcomes o They describe the types and levels of outcomes associated with a particular policy o They do not allow us to make casual claims o They identify types levels or changes in an outcome or set of outcomes and their association with some policy o It prompts policy makers and criminal justice administrators to clarify relevant criteria for identifying policy effectiveness like some sort of panel o Provides the initial groundwork for making causal claims but provides little to no comparative framework for establishing causality Impact evaluation To determine whether the association is casual o Go one step further by establishing whether a policy not only is associated with the outcomes but also whether it actually produces or causes them o Directly establishes whether policy produces specific outcomes and does so by using control groups or conditions It employs a comparative framework which then turns in to an outcome evaluation in to a study so it could produce a more credible claim about casualty o The basic idea is to examine what would have happened if the policy was not implemented also referred to as the counterfactual condition Basically an estimation of whether it be quality or quantity of circumstances that would have occurred had the policy not been proposed Researchers traditionally use statistical matching techniques such as propensity score matching in order to estimate the counterfactual With an implementation evaluation we establish accountability in part by showing that expected activities services or practices that were under taken as intended An outcome is the state of the target population or the social conditions that a policy is An output refers to the services or activities undertaken by a policy prison reentry expected to have changed program example page 172 o A prisoner reentry program with job training The job training is an output and outcomes are how well the inmates learn skills in the training program and whether they obtain jobs or continue to commit crime o Service quality is a type of outcome Service quality o Harry Hatry emphasized that it should be viewed an an important type of outcome for simple reason Even though quality of service characteristics does not measure a final result they are important to program customers and thus can be considered intermediate outcomes an agency should track One type of outcome relevant to criminal justice policies is service quality o Allows us to consider a broader spectrum of criminal justice outcomes o Can be considered an intermediate outcome used to hold agencies accountable o Examples include Timeliness of service provision Accuracy of assistance or information Courteousness of service delivery Condition and safety of facilities Customer satisfaction A selection effect can have a negative impact on data because it causes inappropriate comparisons o Especially when it comes to figuring out whether or not a policy or program is actually working in terms of its effectiveness or impact on recidivism rates among inmates or clients A good amount of service quality dimensions should be considered as an outcome because either they are important to the public or because they relate to overachieving goals such as public safety and justice o Examples of this consists of how fast police respond or how courteous juvenile probation officer are towards juvenile parents and etc What messes up a lot of criminal justice policies are the apples to oranges comparisons What we want are apples to apples comparisons o Researchers traditionally use statistical matching techniques such as propensity Researchers were able to provide an apples to apples comparison page score matching 177 Evaluators should o Identify which outcomes should be given more weight For any given set outcomes evaluators ideally should ideally identify those that should be given more weight Let s also consider that policies typically have more than one objective nonetheless dictates that some attempt made to identify those that matter the most o So it is not always possible to reach to an agreement on which outcomes matters most o Build on theory evaluations to distinguish between short and intermediate term outcomes and longer term outcomes o Distinguish between in program outcomes and after program outcomes o Use the best data possible to create outcome measures For instance screening assessments for probation officers to use on clients Two basic methodologies for assessing whether a policy produces an impact are experiments and quasi experiments Experiments consist of random assignment of subjects to either treatment or control groups o Considered the gold standard of research because of their ability to create equivalency between treatment and control groups o Establish the by which other research methodologies are judged o Randomization is critical to experiments because we want to know what that the only thing that created any difference in outcomes is the intervention and not some characteristic unique to the treatment group If successful there will be equivalence will be present in the results It allows one to create simple comparisons o The objective is to show that an intervention produced or did not produced at a specific level or change in outcome Quasi experiments are typically similar to experiments except that they lack randomization o The goal then is to attempt to create equivalency between the treatment and control groups Methods of doing this include Statistical techniques Matching procedures multivariate statistical approaches regression discontinuity designs pre post analyses and time series analyses o Lacks the feature of randomization o The goal is to create as much equivalence as possible between control groups and treatment groups The problem with this is that it is hard to achieve especially data for creating equivalence does not exist While the goal of impact evaluations is to determine if a policy produces the desired effect many policies may have unintended effects which should be identified in order to avoid creating an unbalanced assessment of a policy Impact assessments by nature are comparative The trick to is identifying the right comparison