Evaluation report

The work of the Oregon Direct Access to Achievement Project was evaluated by an outside contractor, who completed the process in Spring 2011. The executive summary and the final evaluation report are now available for download below.

Click HERE to download the complete report.

Executive Summary

The Oregon Direct Access to Achievement (DATA) Project was designed to increase classroom-level Data-Driven Decision Making (DDDM). Professional development included job-embedded training and traditional seminar training. The impact of this training on teachers was monitored utilizing a triadic assessment framework. This assessment framework examined teacher concerns (emotional response), efficacy (motivational response), and knowledge. Results of this assessment framework were used to provide recommendations for training efforts in order to create “prescriptive professional development.” The results of this prescriptive professional development were dramatic at the teacher level and student level.

Teachers reported significant decreases in resistance to DDDM and noticeable increases in teacher efficacy while maintaining relatively high levels of concerns about the logistics of DDDM (Management Concerns) and the impact of DDDM on students (Consequence Concerns). Management Concerns and Consequence Concerns indicated that teachers are likely beginning to incorporate DDDM in their classrooms, and will likely continue to increase implementation as resistance decreases. Qualitative data findings also supported these quantitative findings. Data team activity logs supported the finding that teachers’ team meetings primarily focused on logistical issues and were beginning to focus on the connection of data to instructional practice and student learning. In trainers’ notes regarding next steps, they indicated that trainings were increasingly focused on building teachers’ DDDM knowledge base and teachers’ understanding of how to connect data to instructional practice. Trainers noted that this focus needed to continue.

After only two years of work at the teacher level, the results of the assessment framework and qualitative analysis of activity logs indicated that teachers have made tremendous and swift progress toward adoption of classroom-level DDDM, connecting data to teaching. Experts in organizational change note that large-scale change of this type takes a minimum of three to five years (Hall & Hord, 2011). The change noted here is unmatched in the DDDM literature and may provide a model for helping teachers connect data to instructional practice. The connection of data to instructional action results in a differentiated learning environment as well as increased student performance (Armstrong & Anthes, 2001; Rallis & MacMullen, 2000; Schueurich & Skrla, 2003). It is recommended that ODP continue training and adhere to the suggestions for training provided in this report. The continuation of this successful project is of paramount importance as research continues to note teachers’ unwillingness and unpreparedness to engage in DDDM (e.g., Bernhardt, 1998; Choppin, 2002; Samuel, 2008; Volante & Fazio, 2007; Wayman, 2005).

Student-level data analysis indicated that students likely benefited from these increases in teachers’ DDDM. At the onset of the ODP initiative, non-ODP schools performed at higher levels in both reading and mathematics. Over the course of the project, ODP students experienced greater gains in the percentage of those who met or exceeded standards in both reading and mathematics. Overall, reading is not an area of weakness in Oregon, but the gap between student reading performance in non-ODP schools and ODP schools decreased. ODP students also experienced significant gains in percentage of students who met or exceeded standards in reading each year of this initiative.

Mathematics is a subject area in need of greater attention in Oregon as compared to reading. In mathematics, ODP schools experienced significant gains in percentage of students who met or exceeded standards each year of this initiative, and ODP schools made substantial progress in closing the gap between non-ODP schools and ODP schools. In fact, there was no significant difference between non-ODP and ODP schools by 2011 on mathematics scores. It is important to note that the greatest gains in reading and mathematics occurred after the first year of job-embedded DDDM trainings provided by certified Oregon trainers and supported through Sustainability Grants to seven regional ESD collaborative partnerships.

Such promising student gains are exciting in light of teacher status in the change process. Teacher change is in its infancy, but reflects substantial movement with regard to change related to DDDM and implementation of DDDM. Such powerful outcomes for both students and teachers speak to the efficacy of this initiative’s design and potential. It is highly recommended that further funding be procured to continue this initiative, and that this work continue in accordance with the suggestions for future practice provided in this report.