We consider two recent suggestions for how to perform an empirically motivated Monte Carlo study to help select a treatment effect estimator under unconfoundedness. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real‐world setting where estimator performance is known. Both approaches are worse than random at selecting estimators that minimize absolute bias. They are better when selecting estimators that minimize mean squared error. However, using a simple bootstrap is at least as good and often better. For now, researchers would be best advised to use a range of estimators and compare estimates for robustness.
Authors

Research Associate University College London and Brown University
Toru is a Research Associate of the IFS, a Professor of Economics at UCL and an Associate Professor in the Department of Economics at Brown University

Research Fellow University of Warwick
Arun is a Research Fellow at IFS, an Associate Professor of Economics at the University of Warwick and a Commissioner at the Wealth Tax Commission.

Tymon Słoczyński
Journal article details
- DOI
- 10.1002/jae.2724
- Publisher
- Wiley
- Issue
- Volume 34, Issue 6, October 2019, pages 893-910
Suggested citation
A, Advani and T, Kitagawa and T, Słoczyński. (2019). 'Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies' 34(6/2019), pp.893–910.
More from IFS
Understand this issue

Gender norms, violence and adolescent girls’ trajectories: Evidence from India
24 October 2022

Do tariffs work?
We discuss the economic consequences of tariffs, why governments use them, and whether they actually achieve their intended goals.
23 January 2025

What is this government’s ‘theory of growth’? Nobody knows
"Shifting the performance of an entire economy requires a long-term, consistent and persistent direction." Paul Johnson writes for the Times.
20 January 2025
Policy analysis

ABC of SV: Limited Information Likelihood Inference in Stochastic Volatility Jump-Diffusion Models
We develop novel methods for estimation and filtering of continuous-time models with stochastic volatility and jumps using so-called Approximate Bayesian Compu- tation which build likelihoods based on limited information.
12 August 2014

Assessing the economic benefits of education: reconciling microeconomic and macroeconomic approaches
This CAYT report discusses the strengths and limitations of several approaches to assessing the effect of education on productivity.
14 March 2013

Misreported schooling, multiple measures and returns to educational qualifications
We provide a number of contributions of policy, practical and methodological interest to the study of the returns to educational qualifications in the presence of misreporting.
1 February 2012
Academic research

Prediction sets and conformal inference with censored outcomes
This paper provides estimation methods of such prediction sets given observed conditioning covariates when 𝑌 is censored or measured in intervals.
21 January 2025

Robust estimation and inference in panels with interactive fixed effects
We consider estimation and inference for a regression coefficient in panels with interactive fixed effects (i.e., with a factor structure).
13 December 2024

Individual welfare analysis: Random quasilinear utility, independence and confidence bounds
We introduce a novel framework for individual-level welfare analysis.
13 December 2024