Broadly speaking, the primary purpose of any evaluation is to find out how successful the program, intervention, or a new strategy was in achieving the overarching goals. Evaluation should tell you what worked and what could be improved to make the program more effective. In this blog series we’ll explore different types of evaluation frameworks, their specific characteristics, perspectives, measures they use, and applications. This introductory blog defines four types of evaluation frameworks and specifies their foci by providing brief examples of the core components. We will discuss each evaluation type through a lens of a large-scale and long-term college readiness program Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP).
In this type of evaluation,the focus is on the process of program implementation rather than its outcomes. The evaluators assess whether the program is being implemented as planned and if the activities and methods used are appropriate for achieving the desired outcomes. This type of evaluation is useful for identifying areas where improvements can be made to the program's delivery and implementation. Throughout the duration of a GEAR UP program (7-year period) the evaluators assess whether the program follows the plan of activities by reviewing student participation and academic records, doing observations of activities, conducting surveys or interviews to gather feedback from the partners on their experiences with the program. The evaluators report the findings to the program staff and other partners outlining what works well and what elements of the program could be modified to better align with the goals and objectives.
In this type of evaluation, the focus is on measuring the program's actual outcomes of the participants. The evaluators assess the effectiveness of the program in terms of the changes that have occurred in the students’ outcomes as a result of the program's activities. This type of evaluation is useful for determining whether the program has achieved its intended outcomes and whether any adjustments are needed to improve the outcomes. For example, if one of the expected outcomes of participation in GEAR UP activities is 10% increase in the number of students who pass a required math class, the evaluators check student academic records to see if it has been accomplished. Based on the results the evaluators could recommend adjusting the activities to close the gap between the expected and achieved outcomes.
In this type of evaluation, the focus is on measuring the extent to which a program or a project has achieved its goals and objectives. This type of evaluation measures progress towards achieving specific goals, which may be short-term or long-term. Goals and outcomes are both important evaluation components, but they serve different purposes. Outcomes are measurable results of a program while goals describe broad intents that provide a framework for the program’s activities. You will learn more about the differences and specifics of each type in the upcoming series. Let’s say one of the goals is to increase GEAR UP students’ and their families’ knowledge of postsecondary education options, preparation, and financing. The evaluators would need to administer pre- and post-surveys to test the knowledge and perceptions of the students’ college readiness before and after the activities targeted towards achieving this goal. Thus, they will assess if there has been any change in the participants’ knowledge in this area.
In this type of evaluation, the focus is to determine the effectiveness of a program in achieving its intended goals and outcomes. It differs from the other types as it attempts to answer questions about the causal relationship between the intervention and its impact, by comparing the outcomes of a group that received the intervention to a group that did not receive the intervention or received a different intervention. In our example, to examine the impact of GEAR UP services the evaluators may ask specific questions to assess whether a statistically significant relationship exists between participation in the provided services and student academic performance. They would compare certain academic indicators (e.g., GPA, Math grades) of the students who participated in the activities and those who did not participate. Thus, the evaluators can make conclusions about the program effectiveness. The choice of which type of evaluation to use depends on the specific program or project and the questions that need to be answered. When evaluating a long-term program that spans several years (such as GEAR UP) these evaluations are used as a sequence or a combination of evaluation activities. In the following entries of this series, we will address each evaluation type in more detail and provide specific examples.
Nadzeya (Nadia) Kardash, Ph.D., is an Associate Researcher with the Research, Evaluation & Dissemination Department in the Center for Educational Opportunity Programs. She currently conducts research and evaluation of the CEOP’s federally funded college access programs including GEAR UP, TRIO, and other college access programs.
Follow @CEOPmedia on Twitter to learn more about how our Research, Evaluation, and Dissemination team leverages data and strategic dissemination to improve program outcomes while improving the visibility of college access programs.
