What factors influence the costs and other resources needed to conduct a program evaluation?
Building STEM Teacher Leadership
Reflections by the STEM teacher leader community on opportunities and gaps in STEM teacher leader development efforts
Ideally, programs should set goals for their evaluation, select an evaluation design that is consistent with those goals, and devote/seek sufficient funding to carry out that design. However, in reality, the level of funding available for an evaluation can be a key constraint and may define the evaluation’s design and scope.
Fortunately, even scarce resources can have a big impact if invested strategically, with each evaluation building on what you’ve previously learned about your program. Here is advice for:
- Who will conduct the evaluation?
- What are common cost drivers and cost savers in a program evaluation?
- What strategies should I use to build an evaluation budget?
You may rely on your own program staff to conduct an internal evaluation, or you can hire an outside evaluator, typically from a research firm or college/university, to conduct an external evaluation.
Having program staff conduct an internal evaluation can be a cost saver, but it is essential that the staff members have training and experience in evaluation-related tasks. Without such experience, the accuracy and usefulness of your evaluation results can be jeopardized.
Programs should also consider hiring external evaluators when:
- It is a legal requirement/condition of their funding
- They have the financial resources to support hiring an external evaluator
- The evaluation is summative or highly public
- Third-party objectivity will be viewed as more credible
- Having an outside perspective is important
Programs should look for an external evaluator who has expertise and/or specialized competencies that program staff don’t have in-house. These may include a background in research design, experience developing surveys and collecting data, and an understanding of research permissions. While most effective evaluators won’t need a great deal of oversight, it’s necessary for program staff to develop goals in partnership with external evaluators to make sure program needs are understood, as well as to clearly communicate expectations.
STEM teacher leader programs have conducted a mix of internal and external evaluation efforts, but the scope of these internal evaluations varied greatly based on staff bandwidth and program resources. Here are some examples of difference experiences with internal and external evaluators:
- One program began to discern the impact of its teacher leaders on their schools through internal evaluation, compiling teachers’ responses to surveys, journals, and logs, and they conducted classroom observations. The program then turned to an external evaluator and requested an evaluation that used a quasi-experimental research design to answer the question of teacher impact.
- A second program with more evaluation capacity also conducted both internal and external evaluations. In this case, instead of using an external evaluator to enhance their capability to answer a single research question, the organization leveraged outside expertise to answer a more complex question. The internal evaluation used an existing, national survey to assess program participants’ professional mindsets relative to the national average. The external evaluator used more intensive data collections to focus on member teachers’ identities and classroom contexts.
In program evaluation, certain activities are significant drivers of the budget and shouldn’t be underestimated in terms of their cost. To help offset the more cost-demanding aspects of evaluation, programs can leverage cost-saving strategies. The table below shows common cost drivers and cost savers.
Evaluation Cost Drivers
Evaluation Cost Savers
STEM teacher leader programs have encountered these common cost drivers. In particular, programs expressed the challenge of securing the necessary permissions to conduct the evaluation. Among these were Data Sharing Agreements (contracts documenting which data are being shared between parties and how they can be used) with each entity providing data (including in each district if the program operates in multiple districts) and approval from Institutional Review Boards (IRB) to conduct research involving human subjects.
Programs also relied on the cost-saving strategies. For example, one found ways to build evaluation activities into routine operations. Evaluation activities felt foreign to staff at first, but they were becoming more integrated into the program culture, and staff began to see them as just part of the job.
Simple benchmark ratios for the proportion of program funds that should be devoted to evaluation aren’t adequate, because evaluation costs vary greatly based on program characteristics and evaluation goals. Budgeting allows you to think through the implementation of your evaluation, and create an accurate and meaningful budget. Evaluation budgets typically include the following cost categories:
- Staffing: the cost to support the amount of time program staff will spend on the evaluation. This should account for both their pay rate and any accompanying benefits. Estimate the time required by staff member by evaluation task.
- Travel: airfare, ground transportation, lodging, meals, and incidentals for travel to and from clients/partners for meetings, sites for data collection, and conferences for dissemination.
- Other Direct Costs: communications services, printing and postage, materials, supplies, incentives, and equipment. To best predict these costs, estimate them by event/activity.
Even if you’re working with an external evaluator, don’t forget to budget time for your own staff to support an external evaluation! This time can include regular meetings with the evaluator, serving as a liaison between the evaluator and program sites, participating in the dissemination of results—and even providing supports for data collection.
To get a sense of whether your budget is reasonable, look at the budgets of evaluations similar in size and scope, and review the budget with other stakeholders who have been engaged in the evaluation planning. It can be helpful to cost out more than one evaluation scenario, such as creating a high and low estimate, or costing out different initial options for the evaluation design to see the budget impacts.