APP 2013 Highest Honors: Increasing Student Learning: Teacher Bonus Program in Rio de Janeiro, Brazil
By: Chloe Cornuejols, Soledad De Gregorio, Nobuko Goto, Cody Reneau
For the full report, click here.
The Secretaria Municipal de Educação of Rio de Janeiro and The World Bank asked the research team to investigate the impacts of a teacher bonus program that was implemented in Rio de Janeiro in 2009. Known as the Prêmio Anual de Desempenho (Annual Performance Award), the program awards teachers and school staff a bonus if their school reaches a pre-determined performance goal. The objective of the program is to increase student learning by adding a variable component to teacher and staff pay that encourages them to improve the quality of the educational environment.
Using a combination of test scores and the school’s promotion (passing) rate, an index measure of performance is created annually for each school. The goal for receipt of the bonus is set for each school every year based on a percentage increase from the school’s previous-year performance. The program has no individual teacher-level assessment or payment.
Our analysis found significant improvements in student performance among participating schools.
The clearest evidence for increased performance was in test scores. Mathematics test scores among fifth graders rose 11% in the first year of the program and increased a total of 17% by the third year. Older children also saw gains. Ninth graders achieved a 6% increase in math and a 7% increase in Portuguese between 2007 and 2011.
The improvements in test scores are significant even when controlling for class size, socioeconomic status, and size of the school. When compared with schools in Rio that were not part of the program, students of participating schools performed significantly better. It is important to note that the Municipality of Rio is undertaking numerous other efforts to improve education, and thus we cannot conclude that all progress is due to the bonus program alone. Our findings, however, indicate that the policies are going in the correct direction and that the bonus program can even be upgraded to increase its positive effects.
We examine the design of the bonus program and present evidence regarding its efficacy. Based on our evaluation we find that this program is well designed because it creates an adequate incentive to improve educational quality. The annual goals are set appropriately, with a near-ideal likelihood of bonus attainment for all schools. We also find that the bonus is rewarding schools that are actually increasing student learning as measured by high school enrollment rates. We go on to consider some potential weaknesses that may threaten the effectiveness of the program. The bonus design is not optimal, as we found evidence of free riding, socioeconomic equity problems, and vulnerability to random variation in the quality of students.
Based on our evaluation, our team recommends retaining the bonus program. We also recommend a number of modifications to improve it, in order to address the weaknesses identified. Our recommendations are divided into short, medium and long-term divisions, reflecting 1-2 year, 2-6 year, and greater than 6-year timeframes. As the federal government of Brazil has committed to double education spending by 2020, our long-term goals coincide with this 7-year-long budget expansion.
We recommend that the Secretaria Municipal de Educação consider the following adjustments and new components:
In the short term (1-2 years)
• Introduce flexibility in target setting for high-performing schools.
• Conduct further research on equity issues.
• Perform continuous evaluation of the program and monitoring of long-term external measures such as high school enrollment in order to measure real-world educational impacts.
• Design teacher-level measurement systems.
In the medium term (3-6 years)
• Introduce a secondary individual bonus based on additional measures at the teacher level. This bonus will allow teachers to receive an additional sum independent of school performance and should combat free riding issues.
• Conduct random experiments of individual teacher performance measures through different combinations of the three measures: classroom observations, supervisor evaluations, and teacher tests.
• Perform continuous evaluation of the program and long-term external measures.
In the long term (6+ years)
• After the experiment is evaluated, the individual bonus program should be fully introduced using the most effective combination of measures, if any is found.
• Continuous evaluation of the program and long-term external measures.