Using graduate earnings data in the regulation of higher education providers

Executive summary

This report explores options for incorporating administrative records on the earnings outcomes of higher education graduates into the Office for Students’ (OfS’s) regulation of the higher education sector in England. It focuses on developing metrics that can effectively measure higher education providers’ (HEPs’) success in equipping graduates with valuable labour market skills.

The OfS currently uses three key metrics, known as B3 metrics, to assess providers’ performance: continuation (whether students continue their studies), completion (whether students complete their qualifications) and progression (whether graduates move into managerial or professional employment or further study). These metrics are compared against absolute thresholds; provider-specific benchmarks are also reported but only serve as contextual information.

In developing a new earnings-based metric, we outline both conceptual and practical criteria. Conceptually, an ideal metric should reflect the impact a course has on graduates’ earnings and capture only factors that providers can control. Practical considerations include fairness across providers, accuracy, creating positive incentives for providers, regulatory practicality, timeliness and transparency.

A key outcome from the work is that we do on balance recommend using an earnings metric calculated from administrative data in regulation. We believe that the best option would be to integrate this new metric into the current OfS regulatory framework in such a way that it complements – rather than replaces – the existing B3 progression metric. Should it be used, this new metric could enable the OfS to develop a more comprehensive view of providers’ performance in preparing students for the labour market. 

Unlike the current B3 progression metric, all the options we consider in detail would use the difference from an institution-specific benchmark as the headline measure, targeting the causal effect of a course on graduates’ earnings relative to other courses in the same subject area. This is because, in our view, only this type of metric can fulfil our conceptual and practical criteria to a sufficient degree.

We organise our discussion of different options for an earnings metric of this type around a baseline approach and various potential variations of it. Variations we examine include different approaches to pooling cohorts, transforming earnings outcomes, controlling for prior attainment and demographics, accounting for location when earning, and controlling for course-level characteristics. While our baseline estimates prove robust to many of these variations, some alternative specifications yield noteworthy differences. These alternatives provide policymakers with a range of options to consider when implementing an earnings metric.

Our proposed baseline metric is based on the highest earnings of graduates three to five years after graduation, although using data anywhere from two to six years after graduation, and averaging rather than taking the maximum, would be perfectly reasonable in our view. We recommend pooling data across two or three cohorts of graduates to maximise statistical power. Crucially, the metric would control for prior attainment and various demographic characteristics as well as subject studied, in line with the OfS’s current approach to benchmarking.

We recommend estimating the model excluding very low earners, as very low recorded earnings are unlikely to reflect graduates’ true earnings potential. Our baseline approach is to exclude those whose maximum earnings three to five years after graduation are below £3,000 in all years. To be transparent about the share of graduates excluded by this threshold, we recommend reporting this share alongside our main estimates for each course. If a much higher minimum earnings threshold such as full-time earnings on the national living wage was chosen, we would recommend estimating separate models for the share of graduates earning below (or above) that threshold, which could be adopted as a third progression indicator. 

Our analysis reveals a persistent relationship between course selectivity and earnings outcomes across most specifications. We examine what we believe to be credible alternative approaches to our baseline approach that would remove this correlation, which could be considered by the regulator. However, as the assumptions underlying these approaches are strong, our recommendation is instead to use one of these approaches to generate an additional ‘selectivity-adjusted benchmark’ to be reported alongside our main benchmark.

We recommend not to control for graduates’ location when earning, as enabling geographic mobility is one channel through which a course may impact graduates’ earnings. Controlling for this would therefore move the earnings metric away from measuring the impact of a course on the earnings of its graduates. Instead, we recommend controlling for graduates’ home region and reporting contextual information about graduates’ location decisions separately, such as the share of graduates staying in the same local area as the provider or the share moving to London. However, we acknowledge that making an adjustment for the location of a provider’s graduates may be desirable from a policy perspective. If so, we would recommend adjusting earnings for regional price levels, though this would require robust regional price indices that are yet to be developed.  

We advise caution in reporting earnings metrics for part-time and Other Undergraduate courses. Estimates for these level–mode combinations will be substantially less robust than those for first degrees, partly due to the much lower number of students taught, and partly due to the larger share of mature students for whom school records are unavailable. In our view, there is therefore a good case not to proceed with earnings metrics for these level–mode combinations at present. 

Importantly, any earnings metric should be treated as one of many sources of evidence on provider performance. Equipping graduates with skills that are valuable in the labour market is only one aspect of course quality. Earnings also only capture one aspect of graduates’ success in the labour market; other factors such as job security or fulfilment at work may be as or more important for graduates. No earnings metric could fully account for the complexity of the higher education landscape. We recommend that the OfS continue considering the wider context in individual cases.
Our recommendations aim to balance various practical and conceptual criteria, providing a framework for implementing an earnings metric that could be useful to the OfS in assessing and regulating higher education providers. For a comprehensive list of recommendations and options considered, please refer to Section 10 of the report.

Acknowledgements 

We gratefully acknowledge and thank members of our Advisory Group for their constructive input to this project. The Advisory Group included Professor Dame Alison Wolf (King’s College London) and Nick Hillman (Higher Education Policy Institute), as well as representatives from the 10 Downing Street Data Science Team; the Association of Colleges; the Association of Employment and Learning Providers; the Department for Culture, Media and Sport; the Department for Education; Guild HE; HM Treasury; Independent Higher Education; the Office for Students; and Universities UK.