by Chad Finlay and Sherol Manavi

For full report click here.

As school districts and states increasingly focus on developing ways to hold administrators and teachers accountable for student performance, measurements based on value-added models (VAM) have recently received much scrutiny. Because value-added models claim to measure the impact schools and teachers have upon their students, independent of outside factors such as SES and past performance that are beyond their control, VAMs are a promising option for policymakers hoping to establish an effective accountability system that will improve student achievement. But due to the logistical, statistical, and political complexity of value-added models, they have been met with significant criticism that has slowed or severely limited their successful implementation.

Our client, Glenn Daley, Director of Los Angeles Unified School District's Office of Evaluation and Research, has been tasked with helping the District develop metrics to be included in their new "Performance Measurement and Accountability System." Mindful of the potential of value added models to assist the nation's second largest school district in better measuring the impact of its schools and teachers upon their students, the Client would like to understand the problems states and districts have encountered in their value-added model efforts.

Based upon a review of various states and districts actively using value-added models, this report describes problems that threaten the long-term use of VAM and offers the Client guidance on how to avoid them.

Findings
An important general theme that emerged from our research is that problems threatening VAM could occur at any time - mature value-added model efforts were as much at risk as relatively newer systems. Tennessee and Dallas, the first adopters of value-added models, experienced the most recent and severe problems, suggesting that successful, long-term use of VAM requires continual vigilance.

Before detailing the problems encountered in value-added models, we summarized some basic characteristics in VAM. Key findings include:

• The more established VAM participants tend to be districts, while recent participants tend to be states
• VAM participants adopted value-added models because they felt current accountability systems were unfair to low-performing schools or did not adequately measure school or teacher effectiveness
• Very few VAM participants use value-added models for performance pay. About half release it publicly, the other half keep it private, and no participant publicly releases teacher level data.

Our findings are grouped into two types of problems: 1) problems caused by the complexity of value-added models and 2) problems caused by the power struggle between states/districts and their teachers' unions.

Problems due to the value-added models' complexity generally were caused by the failure to gain the trust of stakeholders. Opponents of value-added models often complained about the lack of transparency in the accuracy of data, the statistical model, and how value-added models were used to evaluate schools and teachers. Distrust was further exacerbated when responses to complaints about VAM were too technical in nature, instead of addressing concerns in ways more accessible and meaningful to a lay-person.

If these core areas of distrust were not addressed, then even mature value-added modeling efforts were at risk. In addition, since gaining the trust of stakeholders was so crucial, efforts to later attach high stakes to VAM were met with significant resistance as states and districts did not adequately utilize the interim time to garner the confidence of stakeholders.

Since all teachers' unions of the VAM participants opposed using teacher level value-added models for high stakes, the participants were forced to shift to school level or grade level value-added models. This potentially reduced the effectiveness of the value-added models as a tool for accountability since teachers are a key component of accountability systems.

Recommendations
From these findings, the report offers recommendations in addressing the problems due to the complexity of value-added models and the resistance of teachers' unions to using VAM at the teacher level.

To address the problems of trust due to the complexity of VAM, we recommend implementing continual, independent oversight from external parties trusted by those being evaluated. This would include auditing of the data to ensure accuracy, ongoing assessment of the statistical models in use, and verification that the results are both correct and correctly applied. In addition, schools and teachers should be provided with on-site experts on value-added models who would provide training and mentoring in understanding VAM results and how to effectively utilize them to better student performance. Ideally, this expertise and mentoring would include peers to increase the level of trust and credibility of the professional development.

The constraints imposed by teacher union resistance to high-stakes value-added models at the teacher level are formidable. Even if problems regarding the complexity of VAM are overcome, there are still significant concerns that value-added models are not capable of adequately measuring teacher effects. It is likely that teacher level VAM along with high stakes remains out of reach for the foreseeable future.

However, we do not recommend attempting to use no/low stakes VAM at the teacher level as a "Trojan Horse" to later switch to a high stakes system. Without high stakes attached to teacher level value-added models, we found that there was not a sufficient incentive for teachers and administrators to take it seriously, and thus develop the necessary communication and processes crucial for long-term implementation. Instead, we recommend a different type of "beachhead," one that centers around pairing teacher level VAM for bonus pay to teachers at underserved schools. Since this type of bonus pay has been generally accepted by teachers' unions, it could demonstrate and showcase teacher level value-added models. In addition, it gives proponents an economic argument that the funds used for such efforts are being efficiently allocated to the most effective teachers.