In this paper we evaluate the premise from the recent literature on Monte Carlo studies that an empirically motivated simulation exercise is informative about the actual ranking of various estimators when applied to a particular problem. We consider two alternative designs and provide an empirical test for both of them. We conclude that a necessary condition for the simulations to be informative about the true ranking is that the treatment effect in simulations must be equal to the (unknown) true effect. This severely limits the usefulness of such procedures, since were the effect known, the procedure would not be necessary.
Authors
![Arun Advani](/sites/default/files/styles/square_desktop/public/2022-05/Arun_Advani.jpg?itok=i5KiRxAs)
Research Fellow University of Warwick
Arun is a Research Fellow at IFS, an Associate Professor of Economics at the University of Warwick and a Commissioner at the Wealth Tax Commission.
![Person graphic](/sites/default/files/styles/square_desktop/public/2022-06/IFS-person-graphic.png?itok=hWCtTSrz)
Tymon Słoczyński
Working Paper details
- DOI
- 10.1920/wp.cem.2013.6413
- Publisher
- IFS
Suggested citation
Advani, A and Słoczyński, T. (2013). Mostly harmless simulations? On the internal validity of empirical Monte Carlo studies. London: IFS. Available at: https://ifs.org.uk/publications/mostly-harmless-simulations-internal-validity-empirical-monte-carlo-studies-1 (accessed: 1 July 2024).
Related documents
More from IFS
Understand this issue
![School girls in Rajasthan](/sites/default/files/styles/square_desktop/public/2022-11/Schoolgirls-in-Rajasthan_0.jpg?itok=aboMI9Wt)
Gender norms, violence and adolescent girls’ trajectories: Evidence from India
24 October 2022
![Isabel Stockton](/sites/default/files/styles/square_desktop/public/2024-06/Isabel-public-finances.jpg?itok=JfdJNN7F)
What are the challenges in getting debt on a falling path?
28 June 2024
![Microphone](/sites/default/files/styles/square_desktop/public/2024-06/Microphone.jpg?itok=soM7Wvbz)
Election Special: Your questions answered
27 June 2024
Policy analysis
![Carl Emmerson](/sites/default/files/styles/square_desktop/public/2022-06/Carl_Emmerson.jpg?itok=6jM06LTY)
IFS Deputy Director Carl Emmerson appointed to the UK Statistics Authority Methodological Assurance Review Panel
14 April 2023
![Publication graphic](/sites/default/files/styles/portrait/public/2022-06/IFS-publication-graphic.png?itok=QoQz8AN4)
ABC of SV: Limited Information Likelihood Inference in Stochastic Volatility Jump-Diffusion Models
We develop novel methods for estimation and filtering of continuous-time models with stochastic volatility and jumps using so-called Approximate Bayesian Compu- tation which build likelihoods based on limited information.
12 August 2014
![Hospital](/sites/default/files/styles/square_desktop/public/2022-08/Hospital2.jpg?itok=Jt37JXbP)
Is there really an NHS productivity crisis?
17 November 2023
Academic research
![Working Paper Cover](/sites/default/files/styles/portrait/public/2024-05/CWP1124-Inference-for-rank-rank-regressions_Page_01.jpg?itok=iJl8Ja1B)
Inference for rank-rank regressions
28 May 2024
![Journal Article Cover](/sites/default/files/styles/portrait/public/2024-02/Fiscal%20Studies%20-%202024%20-%20%20-%20Issue%20Information_Page_1.jpg?itok=GfdQz4AB)
Sample composition and representativeness on Understanding Society
2 February 2024
![Working paper cover](/sites/default/files/styles/portrait/public/2024-06/WP202428-The-impact-of%20labour-demand-shocks-when-occupational-labour-supplies-are-heterogeneous.jpg?itok=Erq9-V9O)
The impact of labour demand shocks when occupational labour supplies are heterogeneous
28 June 2024