Week 3: Evaluating Programs & Human Performance

1). Evaluation is the process of determining the merit, worth and value of things, and evaluations are the products of that process (Dempsey & Reiser). Evaluation is the making of a judgment about the amount, number, or value of something, or an assessment. An assessment is the evaluation or estimation of the nature, quality, or ability of someone or something. So both assessment and evaluation is entangled into one definition of being able to reflect on the quality of ones learning’s.

Many evaluation models were developed in the 1970s and 1980s. These evaluation models were to have a profound impact on how designers would come to use the evaluation process. The new models were used on projects that included extensive development work, multiple organizations and agencies, and multiple forms of instructional delivery. These projects had multiple goals that were to be achieved over time. Examples were teacher corps projects aimed at reforming teacher education and math projects that attempted to redefine what and how children learned about mathematics. These projects often employed new models of evaluation. Perhaps the most influential model of that era was the CIPP model developed by Stufflebeam in 1971(Dempsey & Reiser).

Stufflebeam’s CIPP evaluation model stands for context, input, process, and product. These are four distinct type of evaluation, and they all can be done in a single comprehensive evaluation or a single type can be done as a stand-alone evaluation. Context evaluation is the assessment of the environment in which an innovation or program will be used to determine the need and objectives for the innovation and to identify the factors in the environment that will impact the success of its use. This analysis is frequently called a needs assessment, and it is used in making program planning decisions. According to stufflebeams CIPP model the evaluator should be present from the beginning of the project, and should assist in the conduct of the needs assessment (Dempsey & Reiser).

The second step or component of the CIPP model is input evaluation. Here evaluation questions are raised about the resources that will be used to develop and conduct the innovation/program. The third step or component of CIPP is process and evaluation. This corresponds closely to formative evaluation. The process evaluation is used to examine the ways in which an innovation/program is being developed, the way it is implemented, and the initial effectiveness, and effectiveness after revisions. The fourth and final component of CIPP is a product evaluation, which focuses on the success of the innovation/program in producing the desired outcomes. Product evaluation includes measuring the outcome variables specified in the program objectives, identifying unintended outcomes, assessing program merit, and conducting cost analyses. Product evaluation is used when making summative evaluation decisions (Dempsey & Reiser).

Rossi’s five domain evaluation model started in the late 1970s and is continuing today. Peter Rossi and his colleagues developed this model. According to this model, each evaluation should be tailored to fit local needs, resources, and type of program. The Rossi model emphasizes five primary evaluation domains.  First is needs assessment. A need is the gap between the actual and desired state of affairs. Second is program theory assessment. If the program is not based on sound social, psychological, and educational theory, it cannot be expected to work. Third is implementation assessment. If a program is not properly operated and delivered, it has no chance of succeeding. The fourth evaluation domain is synonymous with the traditional social science model of evaluation, and the fifth domain is synonymous with the economic model of evaluation. The fourth domain is impact assessment, which is cause and effect of the assessment. The fifth domain is efficiency assessment, which checks if the assessment is cost effective (Dempsey & Reiser).

Kirkpatrick’s training evaluation model was published initially in four articles in 1959. Kirkpatrick’s training evaluation model has four levels. Level one is reaction. The first level is the assessment of learner’s reactions or attitudes towards the learning experience. Anonymous questionnaires should be used to get honest reactions from learners about the training. Level two is learning. In the second level evaluation, the goal is to determine what the participants in the training program learned. Level three is behavior. In the third level the evaluator’s goal is to determine whether the training program participants change their on-the-job behavior. Level four is results. In the fourth level the evaluator’s goal is to find out if the training leads to final results which include outcomes that affect the performance of the organization (Dempsey & Reiser).

Brinkerhoff’s success case method is a specialized model that focuses on finding out what about a training or other organizational interventions work. The SCM follows five steps. First you focus and plan the success case study. Second, construct a visual impact model. Third, conduct a survey research study to identify the best cases and the worst cases. Fourth, schedule and conduct in-depth interviews, usually via telephone for approximately forty-five minutes per interview (Dempsey & Reiser).

Patton’s utilization-focused evaluation was constructed by Michael Patton in the 1970s. The U-FE follows several steps. Because U-FE is a participatory evaluation approach, the client and primary users will be actively involved in structuring, conducting and using the evaluation and its results. There are nine major steps. Step one, conduct a readiness assessment; step two, identify the primary intended users and develop a working relationship with them; step three, conduct a situational analysis; step four, identify the primary intended users; step five. focus the evaluation; step six, design the evaluation; step seven, collect, analyze, and interpret the evaluation data; step eight, continually facilitate evaluation use; step nine, conduct a meta-evaluation (Dempsey & Reiser).

2). Evaluation is often looked at from four different levels listed below. Note that the farther down the list, the more valid the evaluation.

• Reaction - What does the learner feel about the training?
• Learning - What facts, knowledge, etc., did the learner gain?
• Behaviors - What skills did the learner develop, that is, what new information is the learner using on the job?

Results or effectiveness - What results occurred, that is, did the learner apply the new skills to the necessary tasks in the organization and, if so, what results were achieved?

3). A performance problem that I have in my area of work is that my students don’t have any motivation to read. I have students that have the potential to read but they have no drive to learn. They don’t practice at home because their parents don’t encourage them. I believe using incentives to help motivate students to read would be an excellent gesture for a non-instructional solution. For first grade we had free pizza incentives and treasure box incentives if they read so many books and it really helps encourage their reading practice at home.

Reference:

Reiser , R. A. & Dempsey, J. V. (2002).Trends and Issues in Instructional Design and Technology. Upper Saddle River, N.J.: Merrill/Prentice Hall, c2002