Why evaluate your program?

How do we measure up in practice to what we set ourselves in our goals? This useful model, developed within the youth justice system, is very suitable for any program – your program

Evaluations are not just exercises in collecting numbers and other information. Nor are they pass/fail tests of your group's effectiveness or your program's mission. They are ways to check your program's impact and identify which parts need to be modified or eliminated to better achieve your goals. Evaluations can help you discover whether you carried out the activities you originally planned, reached your audience, changed attitudes or behaviors, increased knowledge, and addressed the problem you sought to correct.

An evaluation also will provide valuable information to help determine whether changes to your program are needed and, if so, what those changes should be.

Kinds of evaluation
Evaluations generally are conducted for two reasons: first, to review what you're doing in implementing your program (a process evaluation), and second, to identify and measure the program's results (an outcome evaluation). The following are some common evaluation techniques:

Interviews with participants, key leaders in the community, volunteers, and program staff.
Analysis of existing data (i.e., information) such as census information, crime reports, and economic and social statistics gathered by local and State governments and private groups.
Onsite program reviews by the evaluator.
Reviews of program materials and reports.
Surveys of key groups.

Starting a program evaluation
Evaluation works only if you decide up front what you want to evaluate and how you will do it. At the beginning of your project, spend time thinking through your evaluation. Identify what information you need.

For example, if the goal of your program is crime prevention, you will want to know whether it does one or all of the following:

Reduces crime.
Reduces fear of crime.
Has a lasting impact.
Attracts support and resources.
Makes people feel safe and better about being in their school or community.
You also may want to know whether your program is cost effective.

Think about how you will collect the information, what data need to be gathered at the program's start for comparison purposes, and who will take responsibility for gathering information. Just as you have to specify the goals of your program, you also must specify the goals of your evaluation. Are you primarily interested in how your program operates (e.g., “What activities are carried out?" “What services are provided to whom?")? If so, you'll conduct a process evaluation and design your evaluation activities to collect information that will answer those questions. Or, are you interested in the results (e.g., “Is there less graffiti in your target neighborhood for a longer period of time?" “Are program participants learning what you wanted them to learn?")? If this is the case, you'll conduct an outcome evaluation and design your evaluation activities to gather information that will help answer those questions. In many cases, you'll want to conduct both types of evaluations. Figure out up front what you want to know. This will save you a lot of time later and help ensure that you get the information you need.

It's also important to keep in mind that there's no such thing as a generic evaluation – especially if you're looking at program outcomes. For example, although your general questions may be similar (e.g., “Did we achieve our goals?"), your specific evaluation questions will be different for each type of program (e.g., “Did the graffiti removal program reduce the amount of graffiti on city buildings?"). Unless you frame your evaluation questions carefully, you run the risk of collecting the wrong type of information.

Ideally, one person or group will take responsibility for the evaluation. Even though it's important for evaluators to remain objective, it's also important for people who are involved with the program to be involved with the evaluation – especially thinking through what questions need to be answered, what data will be collected, and how. Conducting an evaluation can be an excellent opportunity for partnerships with colleges, universities, local businesses, or other organizations that have research capabilities, especially if the evaluation requires “number crunching" or rigorous assessment. These partners may be willing to donate time and expertise to help process and analyze the information, especially if you can help supply volunteers. Graduate students often will work for free in exchange for access to fresh data. Try to get your partners involved early so that everyone is confident that the evaluation will produce helpful results.

Critical items for conducting effective evaluations

Build an evaluation into your program from the start
Plan early. The most important step is to commit to an evaluation when you start your program. Once you do, it will become part of the routine operations of your program. If your program is already up and running, find a way to add an evaluation component as soon as possible. Although there are ways to add an evaluation midcourse, building an evaluation into your program from the beginning will give you a better chance to gather the data needed to measure important changes caused by your program.

Thinking through evaluation issues early helps you check some of the assumptions you're making about your program goals and operations. If your goal is to improve participants' knowledge about the dangers of drug use, the only way you'll know if their knowledge improved is to measure their knowledge before and after the program. The same can be true for a graffiti removal program. If your goal is to reduce the amount of graffiti in your neighborhood, you need to evaluate how much was there before your program began, how much still exists after your program ends, and how long the neighborhood stays clean. The key point here is that in order to be able to collect the information you're going to need, you have to build evaluation into your program from the start.

Design your evaluation based on specific questions
Come up with a good, clear set of evaluation questions, which might change (or grow) as you go along. Make sure the questions you're asking are appropriate for your program and that they're clearly linked to the program's goals and objectives. For example, your English teacher's goal is to improve your knowledge and use of English. It wouldn't be too fair to you (or the teacher) if the evaluation of the class was a math test.

An evaluation design is basically a plan for collecting data: what information you will collect, when you will collect it, how, from whom, and so on. Base your design on what you want to know about your program (i.e., your evaluation questions). These might be process evaluation questions (e.g., “How many youth did we reach with educational messages?"), outcome evaluation questions (e.g., “Did test scores improve?" “Did delinquency decline?"), or both.

Collect data consistently and accurately
Whenever you collect information about something or “measure" something – you want to make sure your measurements are consistent and accurate. If you want to measure and compare your height over time, for example, it would be hard to do if one month you measure with your shoes on and the next month you measure with your shoes off. Imagine what would happen if you were watching your weight and the scale you were using wasn't very accurate or you used different scales every time? Would you want to rely on these shaky measurements to show your hard-won progress?

The same is true for measurements in evaluation. For example, if the number of people attending a series of youth events is one of your measures, decide whether you are going to count everyone or just the youth in attendance. Whatever you decide, you should count the number of people at each event the same way. Also, make sure to note differences, if any, in the way you measure. For example, you may have read test questions aloud to all second and third graders, but allowed all fourth and fifth graders to read questions to themselves. This might affect your evaluation outcome.

Keep in mind that factors beyond your control may affect your measurement. If you count attendance at an outdoor event, the number of participants may depend on the weather. If the day of your event is cold and rainy, attendance will most likely be low. Take this into account when you evaluate attendance and if you compare the attendance record with previous events. Also, find out if anything else was happening in the community that might have affected your project's results. If you were conducting an anti-graffiti campaign and the city's anti-gang task force also launched a major anti-gang/anti-graffiti effort at the same time, it would not be appropriate for you to claim credit for all graffiti reductions.

Measure only what you can use, and use what you measure
Avoid counting everything in sight on the theory that a widely cast net will find some positive result, no matter how far fetched. Resist that temptation. Important findings could get buried under irrelevant statistics.

Ask yourself at the start, “What will this information tell us about whether the project has met its goals? What would I change if the answer were yes? If the answer were no? Do the things I've decided to measure really relate to the program and its goals?" For example, it would be pretty difficult to explain how counting traffic accidents at an intersection near a school would relate to your drug abuse prevention initiative. But counting traffic accidents involving young drivers who were found to be drunk or drugged might be a good measure of the impact of your drunk-driving elimination campaign.

Evaluation results that aren't used represent money, time, and energy poured down the drain. Youth taking part in the program, volunteers helping with it, agencies and groups funding it, and media reporting on it are all potential audiences for your evaluation findings. Good news can be widely publicized; news that is not so good can also be shared so it can contribute to solutions that may already be under way. Your evaluation can also help others trying to conduct similar programs.

You also can use what you measure to improve your project, reassess your goals and objectives, discover whether your project had the impact that was expected, and help to lay the foundation for replicating your project somewhere else.

Challenges and rewards of an evaluation
It is sometimes easy to delay conducting an evaluation while the “real program" gets going or to assume that evaluation is impossible without a lot of money. Although it's true that it's generally a good idea to evaluate a program that's in full operation and that some evaluations can be expensive, you may be able to conduct all or most of the evaluation yourself, and you may be able to find volunteers to help.

The most important reward gained from an evaluation should be either confirmation that the program is meeting (or has met) its goals and objectives or recognition of things that need to be changed. It's a wonderful outcome to be able to document your success for others, but often evaluations have mixed outcomes – some successes and some areas that may require change due to weak or nonexistent findings. Also, sharing your findings (successes, obstacles to overcome, and lessons learned) with others conducting similar programs can often be helpful to both your efforts.

Perhaps the second most important reward of an evaluation are the hidden bonuses – the findings about attitudes, events, and results that were unexpected – which can be highly instructive. These may relate to your goals or to other relevant issues. Don't ignore them – and don't forget to look for them. For example, at the end of your anti-graffiti campaign, people may reveal in a survey that, although the graffiti cleanup was a success, they are now bothered by the playground filled with trash. The cleanup of the park could be the focus of your next action campaign. Maybe the same people who complained would be willing to volunteer. Perhaps a mayor's office that previously didn't have funding would be willing to help you when you can prove your program works.

Remember that including an evaluation step in your overall plan from the very beginning is smart. After the evaluation is completed, ask yourself what could have been done better to reach your goals, to involve more people in your project, and to spread your crime prevention message to a wider audience. Then, make the necessary adjustments to strengthen your program.

Learning to evaluate your program is a good skill and one you can apply to all aspects of your life. An evaluation can help keep your program going in the right direction. Good luck with your evaluations – use them to show others your program works!