Downloads
![Image representing the file: CWP561818.pdf](/sites/default/files/output_url_files/CWP561818.pdf_0.jpg)
CWP561818.pdf
PDF | 367.05 KB
Currently there is little practical advice on which treatment effect estimator to use when trying to adjust for observable differences. A recent suggestion is to compare the performance of estimators in simulations that somehow mimic the empirical context. Two ways to run such ‘empirical Monte Carlo studies’ (EMCS) have been proposed. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real-world setting where estimator performance is known. We find that in our setting both EMCS approaches are worse than random at selecting estimators which minimise absolute bias. They are better when selecting estimators that minimise mean squared error. However, using a simple bootstrap is at least as good and often better. For now researchers would be best advised to use a range of estimators and compare estimates for robustness.
Authors
![Toru Kitagawa](/sites/default/files/styles/square_desktop/public/2022-05/Toru_Kitagawa.jpg?itok=3kinno1S)
Research Associate University College London and Brown University
Toru is a Research Associate of the IFS, a Professor of Economics at UCL and an Associate Professor in the Department of Economics at Brown University
![Arun Advani](/sites/default/files/styles/square_desktop/public/2022-05/Arun_Advani.jpg?itok=i5KiRxAs)
Research Fellow University of Warwick
Arun is a Research Fellow at IFS, an Associate Professor of Economics at the University of Warwick and a Commissioner at the Wealth Tax Commission.
![Person graphic](/sites/default/files/styles/square_desktop/public/2022-06/IFS-person-graphic.png?itok=hWCtTSrz)
Tymon Słoczyński
Working Paper details
- DOI
- 10.1920/wp.cem.2018.5618
- Publisher
- The IFS
Suggested citation
A, Advani and T, Kitagawa and T, Słoczyński. (2018). Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies. London: The IFS. Available at: https://ifs.org.uk/publications/mostly-harmless-simulations-internal-validity-empirical-monte-carlo-studies (accessed: 30 June 2024).
Related documents
More from IFS
Understand this issue
![School girls in Rajasthan](/sites/default/files/styles/square_desktop/public/2022-11/Schoolgirls-in-Rajasthan_0.jpg?itok=aboMI9Wt)
Gender norms, violence and adolescent girls’ trajectories: Evidence from India
24 October 2022
![Isabel Stockton](/sites/default/files/styles/square_desktop/public/2024-06/Isabel-public-finances.jpg?itok=JfdJNN7F)
What are the challenges in getting debt on a falling path?
28 June 2024
![Microphone](/sites/default/files/styles/square_desktop/public/2024-06/Microphone.jpg?itok=soM7Wvbz)
Election Special: Your questions answered
27 June 2024
Policy analysis
![Publication graphic](/sites/default/files/styles/portrait/public/2022-06/IFS-publication-graphic.png?itok=QoQz8AN4)
ABC of SV: Limited Information Likelihood Inference in Stochastic Volatility Jump-Diffusion Models
We develop novel methods for estimation and filtering of continuous-time models with stochastic volatility and jumps using so-called Approximate Bayesian Compu- tation which build likelihoods based on limited information.
12 August 2014
![Publication graphic](/sites/default/files/styles/portrait/public/2022-06/IFS-publication-graphic.png?itok=QoQz8AN4)
Assessing the economic benefits of education: reconciling microeconomic and macroeconomic approaches
This CAYT report discusses the strengths and limitations of several approaches to assessing the effect of education on productivity.
14 March 2013
![Publication graphic](/sites/default/files/styles/portrait/public/2022-06/IFS-publication-graphic.png?itok=QoQz8AN4)
Misreported schooling, multiple measures and returns to educational qualifications
We provide a number of contributions of policy, practical and methodological interest to the study of the returns to educational qualifications in the presence of misreporting.
1 February 2012
Academic research
![Working Paper Cover](/sites/default/files/styles/portrait/public/2024-05/CWP1124-Inference-for-rank-rank-regressions_Page_01.jpg?itok=iJl8Ja1B)
Inference for rank-rank regressions
28 May 2024
![Journal Article Cover](/sites/default/files/styles/portrait/public/2024-02/Fiscal%20Studies%20-%202024%20-%20%20-%20Issue%20Information_Page_1.jpg?itok=GfdQz4AB)
Understanding Society: minimising selection biases in data collection using mobile apps
2 February 2024
![Working paper cover](/sites/default/files/styles/portrait/public/2024-06/WP202428-The-impact-of%20labour-demand-shocks-when-occupational-labour-supplies-are-heterogeneous.jpg?itok=Erq9-V9O)
The impact of labour demand shocks when occupational labour supplies are heterogeneous
28 June 2024