- Step 1: Create a logic model that conceptualizes the program, including key outcomes.
- Step 2: Define the evaluation’s goals and purpose with input from stakeholders.
- Step 3: Determine the evaluation design based on these evaluation goals.
- Review and renew the program and evaluation goals with stakeholders on an ongoing basis.
Over time, it will be important to review and renew the evaluation goals with stakeholders.
The first step in an evaluation is to develop a logic model. A logic model describes the elements of a STEM teacher leader program, connections among them, and expected outcomes. Logic models help solidify a clear, shared understanding of how the program should work for all stakeholders and serve as a foundation for evaluation.
Below is an example logic model template, which begins with problem/goals/purpose and ends with long-term impact.
To complete a logic model, program leaders can lead stakeholders through the following steps, focusing on the “why,” “how,” and “what” for each:
- Determine the program goals/purpose (the why) and the objective (the what). What assumptions are needed for the project to accomplish its goal?
- List the program strategies/activities that will be implemented to accomplish the objective. What are these activities and why they are important? What contextual factors might affect program implementation?
- List the expected immediate outputs or direct results of each activity. Why and how will these results occur?
- List the intermediate outcomes or results as they connect to the objectives. Why and how will these results occur? What contextual factors might impact program outcomes?
- List the long-term impact (past three years if applicable). Why will this be the impact? What might be key unintended consequences?
Stakeholders of STEM teacher leader programs can include program funders, program staff leaders, teacher participants, and district and school leaders. Each group will vary in their view of the evaluation purpose and have different evaluation goals based on their interests. District leaders may be interested in the degree to which participants are taught and learn key leadership skills, and school leaders may want to know if participants will be able to take on new leadership roles such as leading professional learning communities. Funders of the program, in contrast, may want to find out if the program improves commitment to STEM subjects and advances student achievement.
Program leaders should seek the input from all stakeholders so that their goals for the evaluation can be taken into consideration and appropriately balanced:
- Collect information from stakeholders prior to evaluation planning and regularly update this information gathering process.
- Convene a meeting of STEM teacher leader program stakeholders to develop consensus on goals. Plan to discuss different evaluation aspects at this meeting including:
- Program goals and program design
- Data that have been collected in advance of the meeting on evaluation goals
- Evaluation budget
- Projected timeline for the evaluation
- Initial options for the design to help stakeholders envision how different evaluation goals would actually play out
- Outcomes, impact, and unintended consequences
- Information needed to support sustainability decisions
- Use the goals of the STEM teacher leader program outlined in the logic model to set goals for the program’s evaluation. General program goals need to be translated into more specific objectives that can be measured as part of the evaluation. The graphic below illustrates the step of going from program goals to evaluation goals for a STEM teacher leader program that had four primary program goals.
|Program Goals||Evaluation Goals|
|Improve inquiry-based teaching in STEM||→||Measure change in teaching practices pre- and post- program participation|
|Improve student achievement in STEM||→||Examine proficiency levels on state-level assessments of students of teachers who received professional development|
|Develop teacher leadership skills||→||Assess growth in leadership skills|
|Build career teacher career paths||→||Document movement in participants’ positions and responsibilities|
The evaluation design should then be created based on evaluation goals. There are three general types of evaluation designs—non-experimental, quasi-experimental, and experimental—each of which has its relative advantages and disadvantages.
Experimental and quasi-experimental designs are most appropriate when the objective is to determine the impact of a program. Experimental designs are strongest because participants are randomly assigned to the program (treatment group) or not (comparison group). Random assignment maximizes the chances that the two groups are equal on factors that could influence their response to the program. Differences that emerge between the groups can be attributed to the program itself rather than other factors.
Experimental designs are not always appropriate or feasible (logistically or financially). In this case, a quasi-experimental design may be preferred. Even without random assignment, these designs can counter some alternative explanations for outcomes besides the program itself through the use of statistical controls.
In formative evaluations where impact is not a focus, a comparison group may not be necessary. In such cases, non-experimental designs may be more appropriate.
Goals may change over time for a program, particularly if the program has been implemented over a number of years. Review evaluation goals and determine whether they too need to be adjusted to reflect program changes.
For example, one STEM teacher leader program set a new focal area for the program each year. This focal area also became the focus of its evaluation that year. They drafted a new planning document annually with new evaluation questions, data collection plans, and analyses.
This reexamination of goals should occur at critical times in the life of the program or the evaluation: such as at the start of the new phase of the program, at the end of a reporting cycle when evaluation results come in, or when new teacher leader participants are selected for the program.