13 December 2019
There was a methodological error in the initial randomisation for the 2014–15 trial of the Magic Breakfast project. The error affected the assignment of schools to intervention and comparison groups, which meant that schools were sorted on Local Authority establishment code within strata (defined by school characteristics: ages served, prior attainment, and the percentage of pupils with English as Additional Language). This means that London and urban schools were disproportionately likely to be assigned to receive the Magic Breakfast intervention. The trial was therefore not a randomised controlled trial as was intended, but instead has a comparison group design.
The mistake was made, discovered and reported to EEF by the principal investigator at IFS. As a result of discovering this error, IFS have re-analysed the data to account for the new information and apply a more appropriate methodology. They find that the original conclusions remain broadly the same at Key Stage 1 (age 7): offering schools support to establish a free, universal breakfast club boosts attainment at Key Stage 1 by around two months’ expected progress, although this result now has a lower security rating. There is no impact at Key Stage 2 (Year 6 pupils). The original estimate was also not statistically significantly different from zero, though the new estimate is smaller and less secure.
The error in the randomisation code was first discovered in 2016 while preparing the original report for publication. At the time, the principal investigator – who discovered the error – believed that it did not meaningfully affect the balance of the intervention and comparison groups and that the trial was therefore ‘as good as random’. At a difficult personal time which clouded her judgement, she therefore made the executive decision to submit corrected randomisation code alongside the report, in an effort to ensure that others adapting the code for their own purposes would not be affected by the error. Upon realising the extent of the implications the error had for the trial, the principal investigator immediately informed colleagues and began the process of revising the results.
The results of the new analysis, along with a full description of the error and the updated methodology, are contained in a peer-reviewed amended evaluation report. A range of alternative specifications is available in the Appendix. The findings from the process evaluation, conducted separately by the National Children’s Bureau, are not affected in any way by the methodological error in the impact evaluation.
The principal investigator takes full responsibility for the error, and is grateful for the help of colleagues at EEF and the independent peer reviewers for assistance in establishing the reliability of the revised results.