Skip to Content

What kinds of tools or data collection instruments have STEM teacher leader programs used in their evaluations? Which may be relevant for my program?

Building STEM Teacher Leadership

Reflections by the STEM teacher leader community on opportunities and gaps in STEM teacher leader development efforts

 

Choosing the right instrument is critical in an evaluation. Relevant instruments allow programs to measure their intended outcomes accurately, consistently, and in a manner well suited to the needs of the program and its evaluation.

To select the best measurement instruments for evaluating a STEM teacher leader program, many factors should be taken into consideration, including:

  • Overall purpose of the program
  • Specific teacher and student outcomes the program is measuring
  • The evaluation’s targeted audience

The goals of the program dictate the outcomes the program is interested in measuring. For example, if a program focuses on developing leadership skills in teachers, measuring student outcomes would not be meaningful. However, if improved student achievement in STEM is a program’s goal, instruments of student data would be very useful.

Another item to consider is whether to use an “off the shelf” tool or to design your own instrument. Advantages of using a published tool are the reliability and validity, ease of use and comparability in results; the major drawback, however, is inflexibility. If you choose to develop your own tool, you can customize the information collected to tightly align with your program. In this case, your evaluation costs will be increased, and you lose the reliability and comparability of a published tool.

The table below represents types of data collection tools that STEM teacher leader programs have used in their evaluations. For each type of instrument, examples, the relevant outcome, the purpose, and the type of program for which the instrument may be particularly relevant are provided.

Teacher/Participant Data
Category Examples Outcome Measured Utility / Purpose Type of Program
Participant Evaluation Evaluation of professional development Quality of professional development User feedback and needs Formative assessment to improve the quality of future professional development in the program Any program that offers professional development
Opinion Surveys Survey of teachers who work with the teacher leader Survey of school leaders who work with the teacher leader Perceived impact of the teacher leader Feedback for the teacher leader about their effectiveness Detect unmet needs that might be addressed in the future Projects who train teacher leaders as coaches to impact others in their sphere of influence
Disposition Surveys Determining growth vs fixed mindset Discovering attitudes toward math and science Change in disposition before and after the program Formative assessment to indicate if adjustments need to be made to affect targeted dispositions Projects whose goals include changing participants’ dispositions
Teacher/Participant Data (Continued)
Category Examples Outcome Measured Utility / Purpose Type of Program
Rubric for Leadership Evaluation rubric for teacher leaders (e.g., North Carolina) Leadership skills and actions of participants Identify a level of competence and build a professional capacity based on the continuum Leader self-assessment Any program that evaluates teacher leaders
Content Knowledge Assessment Assessment of math or science (e.g. Learning Mathematics for Teaching) Content knowledge among teacher leaders before and after participation in the program Formative assessment to determine if the program has had the desired impact on teacher content knowledge Any program using math and science content within their program’s focus area
Count of Leadership Activities Number of op-ed articles authored Number of professional development sessions facilitated Teacher leaders’ engagement in leadership activities Establish that teacher leaders contribute to the learning of other educators Demonstrate sustainability of the overall STEM program Programs who encourage participants to engage in leadership activities
Classroom Observations Use a calibrated tool to look for particular outcomes (e.g., 5×8 card) Impact on classroom practice Formative assessment of the degree to which the program impacts classroom practice Any program that seeks to impact classroom instruction
Artifacts / Portfolio Artifacts collect by Teachers Leaders such as video, photos, and planning documents Evidence of teacher leadership activities Exemplars to use with future participants Capture events that are difficult to measure in other ways Any program that uses a cycle of inquiry (set a goal & rationale, objective to accomplish goal, implement learning activity, reflect on evidence of learning, and set a new goal)

 

Student Data
Category Examples Outcome Measured Utility / Purpose Type of Program
Opinion Surveys Survey of students who participate in the program Perceived impact of the program as it relates to the student’s social and emotional learning Feedback for the program’s leader about their effectiveness in meeting students’ needs Detect unmet student needs that might be addressed in the future Projects who want students to be impacted by the program
Disposition Surveys Determining growth vs fixed mindset Discovering attitudes toward math and science Change in student’s dispositions before and after the program Determine if the project changed the student’s disposition Projects whose goals include changing student’s dispositions
Content Knowledge Assessment Assessment of math or science (e.g. Learning Mathematics for Teaching) Provide an understanding of content knowledge Understand major concepts, assumptions, processes of inquiry, and ways of knowing that are central to the discipline(s) Any program using math and science content within their program’s focus area
Standardized Assessment Data State, national or international assessments (e.g., PARCC, Smarter Balanced, PISA, ACT, SAT, AP) Student achievement in STEM disciplines Detect systematic differences over time in STEM disciplines Any program that seeks to impact student achievement
Participant Evaluation Student evaluation of STEM activity (e.g. a STEM camp) Continually evaluate the effects of student’s participation and overall understanding Reflect on student’s participation and adapts practice to meet the participation needs of the program Any program that models a cycle of inquiry (set a goal & rationale, objective to accomplish goal, implement learning activity, reflect on evidence of learning, and set a new goal)