Evaluation Resources Definitions
The fields used in the database “Evaluation Resources: Supporting Effective Educator Development” are defined below. You can use these fields to narrow your search results and find resources that address specific areas of interest.
Plan and Design
- Randomized Controlled Trials (RCT)
- Randomized Controlled Trials (RCT) study designs are evaluations that randomly assign study participants to either a “control group” that does not receive the intervention or to a “treatment group” that does receive the intervention.
- Quasi-Experimental Design (QED)
- Quasi-Experimental Designs (QED) cover a wide variety of research designs. Unlike randomized controlled trials, participants are not randomly assigned to a treatment group or control group prior to the intervention. Instead, in a Quasi-Experimental Design, the treatment group includes individuals who received the intervention. The comparison group is made up of individuals who did not receive the intervention but had similar characteristics to the treatment group. Note that a special type of Quasi-Experimental Group Design, Regression Discontinuity Design, is in a separate category below.
- Regression Discontinuity Design (RDD)
- Regression Discontinuity Designs are a special type of Quasi-Experimental Group Design. Regression Discontinuity Designs can be applied when participants are accepted into a program intervention based on a cut-off score. Individuals just below the cut-off serve as a good comparison group for those individuals just above the cut-off, or vice versa.
Identify and Follow Participants
- Identifying Comparison Groups
- Guidance on how to identify or select a comparison group for your study. A comparison group should be as similar as possible to the treatment group. “Baseline Equivalence” is the technical term used by the WWC to measure how similar the comparison group is to the treatment group.
- Determining Sample Sizes
- Guidance on how to determine the appropriate sample size for your study.
- Recruiting Study Participants
- Guidance on how to recruit sites and participants (including gaining “buy-in”) for your study.
- Addressing Changes in your Sample
- Consider how people leaving or entering your study will affect the integrity of your study, and guidance as to how to respond if this is the case.
- Reducing Bias in Comparison Groups
- Guidance on how to identify and avoid bias in your research design. Bias exists when it is unclear whether the intervention or another factor is responsible for the difference in outcomes between the treatment and comparison groups. “Confounding factors” is the technical term for these other factors in WWC.
Collect and Store Data
- Selecting Appropriate Outcome Measures
- Ensure that your outcome measures are appropriate for your research design. WWC refers to these issues as “Outcome Eligibility.”
- Acquiring Administrative Data
- Consider issues related to identifying, getting permission to use, and obtaining existing administrative data for a research study. This may be from states, districts, schools, or other entities.
- Collecting New Data
- Guidance related to original data collection for a research study, such as designing and fielding surveys.
- Combining Data Systems
- Consider issues in combining data systems for analysis. For example, this includes issues such as combining data from different sources on similar variables or pooling data across districts or states.
- Understanding Data Analytic Models
- Models used to analyze the data to determine if the program had a casual impact on the outcome of interest. Analytic models such as Hierarchical Linear Modeling, Propensity Score Matching, and Difference-in-Differences are addressed.
- Addressing Analysis Challenges
- Consider adjustments that researchers make to ensure that the results of their analytic models are valid. For example, this can include methods to deal with missing data, testing multiple outcomes, and cluster corrections.
- Interpreting Findings
- Explore how to report findings. For example, this includes issues such as interpreting effect sizes, improvement index, statistical significance of results, and/or interaction effects.
- Visualizing Data
- Consider methods to visually depict data to more easily interpret its meaning.
Special Evaluation Topics
- Low-Cost Approaches
- Explore low-cost approaches to evaluation. For example, this includes naturally occurring experiments and use of administrative data or publicly available data sources.
- Short-Cycle Methods
- Explore short-cycle approaches to evaluation. For example, this includes rapid-cycle evaluation, continuous improvement methods, and Plan-Do-Study-Act cycles.
- Educator Development Focus
- Consider evaluation strategies in the context of educator development. This includes teacher or principal recruitment, selection, and preparation; professional development for teachers of academic subjects; and/or advanced certification and advanced credentialing.
- References WWC Standards
- View documents that reference What Works Clearinghouse (WWC) standards.
Examples of Outcome Measures
- Student Achievement
- View examples of outcome measures related to student achievement outcomes. Measures may include: graduation rates, reading and/or writing, math, science, college entrance exams, grade progression, or civic knowledge & skills.
- Student Behavior
- View examples of outcome measures related to student behavior. Measures may include: student attitudes, student attendance, student retention, discipline and behavior outcomes, student dispositions, self – efficacy, or classroom climate perception.
- View examples of outcome measures related to teachers. Measures may include: teacher quality/effectiveness measures (e.g., value-added measures), classroom observations, lesson plans, surveys of teacher knowledge and/or attitudes, teacher retention, or teacher evaluation scores.
- View examples of outcome measures related to principals and/or schools. Measures may include: school instructional climate surveys, principal evaluation scores, school performance measures (e.g., test scores), or principal retention.
- View examples of outcome measures related districts. Measures may include: district value added scores, district leadership participation, or training ratings.