Publications

2025

1. Assessing Reproducibility in Economics Using Standardized Crowd-sourced Analysis

   Abel Brodeur, Seung Yong Sung, Edward Miguel, and 2 more authors

   May 2025

   Abs DOI

   This paper presents a framework to standardize crowd-sourced computational reproductions in economics through the Social Science Reproduction Platform (SSRP). The approach address four main challenges for computational reproductions: a lack of standardization, aggregation issues, existing incentives for “adversarial” interactions, and the loss of knowledge from analyses that are never published. We then summarize the first 487 reproductions uploaded on the SSRP. The results show substantial heterogeneity in the ability to successfully reproduce empirical results in economics research, with approximately 30% of recent studies meeting at least a basic definition of being computationally reproducible.

2. TROV - A Model and Vocabulary for Describing Transparent Research Objects

   Meng Li, Timothy McPhillips, Craig Willis, and 6 more authors

   International Journal of Digital Curation, Feb 2025

   Abs DOI

   The Transparent Research Object Vocabulary (TROV) is a key element of the Transparency Certified (TRACE) approach to ensuring research trustworthiness. In contrast with methods that entail repeating computations in part or in full to verify that the descriptions of methods included in a publication are sufficient to reproduce reported results, the TRACE approach depends on a controlled computing environment termed a Transparent Research System (TRS) to guarantee that accurate, sufficiently complete, and otherwise trustworthy records are captured when results are obtained in the first place. Records identifying (1) the digital artifacts and computations that yielded a research result, (2) the TRS that witnessed the artifacts and supervised the computations, and (3) the specific conditions enforced by the TRS that warrant trust in these records, together constitute a Transparent Research Object (TRO). Digital signatures provided by the TRS and by a trusted third-party timestamp authority (TSA) guarantee the integrity and authenticity of the TRO. The controlled vocabulary TROV provides means to declare and query the properties of a TRO, to enumerate the dimensions of trustworthiness the TRS asserts for a TRO, and to verify that each such assertion is warranted by the documented capabilities of the TRS. Our approach for describing, publishing, and working with TROs imposes no restrictions on how computational artifacts are packaged or otherwise shared, and aims to be interoperable with, rather than to replace, current and future Research Object standards, archival formats, and repository layouts.

3. Report of the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2025
4. Using Containers to Validate Research on Confidential Data at Scale

   Lars Vilhuber

   Harvard Data Science Review, Jun 2025

   Abs DOI

   I describe past experience with the validation server process over 10 years and several hundred users, as a means to provide proxy access to confidential data. As a modern replacement, I propose the use of containers—simulated computers that encapsulate entire software and file structures. The use of containers ensures reproducibility, reliable portability, and enables scalability. Infrastructure can be outsourced to commercial providers or users, at little to no cost to data providers. The only likely limitation to full automation is the absence of automated output vetting algorithms at statistical agencies.

2024

1. Report of the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2024

   DOI
2. Mass Reproducibility and Replicability: A New Hope

   Abel Brodeur, Derek Mikola, Nikolai Cook, and 346 more authors

   May 2024

   Abs

   This study pushes our understanding of research reliability by reproducing and replicating claims from 110 papers in leading economic and political science journals. The analysis involves computational reproducibility checks and robustness assessments. It reveals several patterns. First, we uncover a high rate of fully computationally reproducible results (over 85%). Second, excluding minor issues like missing packages or broken pathways, we uncover coding errors for about 25% of studies, with some studies containing multiple errors. Third, we test the robustness of the results to 5,511 re-analyses. We find a robustness reproducibility of about 70%. Robustness reproducibility rates are relatively higher for re-analyses that introduce new data and lower for re-analyses that change the sample or the definition of the dependent variable. Fourth, 52% of re-analysis effect size estimates are smaller than the original published estimates and the average statistical significance of a re-analysis is 77% of the original. Lastly, we rely on six teams of researchers working independently to answer eight additional research questions on the determinants of robustness reproducibility. Most teams find a negative relationship between replicators’ experience and reproducibility, while finding no relationship between reproducibility and the provision of intermediate or even raw data combined with the necessary cleaning codes.

3. Reproduce to Validate: A Comprehensive Study on the Reproducibility of Economics Research

   Sylvérie Herbert, Hautahi Kingi, Flavio Stanchi, and 1 more author

   Canadian Journal of Economics/Revue canadienne d’économique, Aug 2024

   Abs DOI

   Abstract Journals have pushed for transparency of research through data availability policies. Such data policies improve availability of data and code, but what is the impact on reproducibility? We present results from a large reproduction exercise for articles published in the American Economic Journal: Applied Economics, which has had a data availability policy since its inception in 2009. Out of 363 published articles, we assessed 274 articles. All articles provided some materials. We excluded 122 articles that required confidential or proprietary data or that required the replicator to otherwise obtain the data (44.5% of assessed articles). We attempted to reproduce 152 articles and were able to fully reproduce the results of 68 (44.7% of attempted reproductions). A further 66 (43.4% of attempted reproductions) were partially reproduced. Many articles required complex code changes even when at least partially reproduced. We collect bibliometric characteristics of authors, but find no evidence for author characteristics as determinants of reproducibility. There does not appear to be a citation bonus for reproducibility. The data availability policy of this journal was effective to ensure availability of materials, but is insufficient to ensure reproduction without additional work by replicators. , Résumé Les journaux militent pour la transparence de la recherche par le biais de politiques sur la disponibilité des données . De telles politiques sur les données améliorent la disponibilité des données et du code, mais quelle en est l’incidence sur la reproductibilité? Nous présentons les résultats d’un grand exercice de reproduction pour des articles publiés dans le American Economic Journal: Applied Economics , qui a une politique de disponibilité des données depuis sa création en 2009. Parmi les 363 articles publiés, nous en évaluons 274. Tous les articles avaient fourni certains documents. Nous avons exclu 122 articles qui nécessitaient des données confidentielles ou exclusives, ou qui exigeaient que la personne chargée de la réplication obtienne autrement les données (44,5% des articles évalués). Nous avons tenté de reproduire 152 articles et avons réussi à entièrement reproduire les résultats de 68 articles (44,7% des reproductions tentées). Nous avons aussi partiellement reproduit 66 autres articles (43,4% des reproductions tentées). De nombreux articles nécessitaient de complexes changements de code, même lorsqu’ils étaient au moins partiellement reproduits. Nous avons recueilli les caractéristiques bibliométriques des auteurs, mais n’avons pas constaté qu’elles étaient déterminantes pour la reproductibilité. La reproductibilité ne semble offrir aucun avantage en matière de citations. La politique de disponibilité des données de ce journal a été efficace pour assurer la disponibilité des documents, mais insuffisante pour assurer la reproduction sans travail supplémentaire des personnes chargées de la réplication.

4. Crowdsourcing Digital Public Goods: A Field Experiment on Metadata Contributions

   Linfeng Li, Yan Chen, Margaret C. Levenstein, and 1 more author

   Nov 2024

   Abs DOI

   This study explores why people choose to contribute metadata, which is data about data. Using a field experiment conducted with more than 3,000 authors of AEA journal articles, our control message reduces the uncertainty about the future value of metadata, whereas those from the treatment conditions additionally make the private or social benefits of metadata salient. Surprisingly, we find that participants in the control condition provide significantly more metadata compared to those in the treatments. This suggests that simply knowing that metadata will have value is sufficient to motivate people to contribute. Our results also highlight the importance of interface design in online field experiments.

5. A Guide for Social Science Journal Editors on Easing into Open Science

   Priya Silverstein, Colin Elman, Amanda Montoya, and 36 more authors

   Research Integrity and Peer Review, Feb 2024

   Abs DOI

   Journal editors have a large amount of power to advance open science in their respective fields by incentivizing and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org) has collated several resources on embedding open science in journal editing (www.dpjedi.org/resources). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we have created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx)).

6. Protecting Confidential Data through Non-Statistical Methods

   Lars Vilhuber

   In Handbook of Sharing Confidential Data: Differential Privacy, Secure Multiparty Computation, and Synthetic Data, Feb 2024

   Abs

   "Statistical agencies, research organizations, companies, and other data stewards that seek to share data with the public face a challenging dilemma. They need to protect the privacy and confidentiality of data subjects’ and their attributes while providing data products that are useful for their intended purposes. In an age when information on data subjects is available from a wide range of data sources, as are the computational resources to obtain that information, this challenge is increasingly difficult. The Handbook of Sharing Confidential Data helps data stewards understand how tools from the data confidentiality literature-specifically, synthetic data, formal privacy, and secure computation-can be used to manage trade-offs in disclosure risk and data usefulness"–

7. Protecting Confidential Data through Non-Statistical Methods

   Lars Vilhuber

   Oct 2024

   Abs

   This chapter will rely on and update previous overviews of how researchers, citizens, and administrators can reliably and securely access confidential data, i.e., data that cannot be simply published as “open data”. I will discuss various legal, technical, and practical ways of securing access to data that is needed for computations. This obviously depends on the type and complexity of the computations but also depends on the who, how, and where access is needed.

2023

1. Report of the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2023

   DOI
2. Assessing Utility of Differential Privacy for RCTs

   Soumya Mukherjee, Aratrika Mustafi, Aleksandra Slavković, and 1 more author

   Sep 2023

   Abs DOI

   Randomized control trials, RCTs, have become a powerful tool for assessing the impact of interventions and policies in many contexts. They are considered the gold-standard for inference in the biomedical fields and in many social sciences. Researchers have published an increasing number of studies that rely on RCTs for at least part of the inference, and these studies typically include the response data collected, de-identified and sometimes protected through traditional disclosure limitation methods. In this paper, we empirically assess the impact of strong privacy-preservation methodology (with \ac\textbraceleft DP\textbraceright guarantees), on published analyses from RCTs, leveraging the availability of replication packages (research compendia) in economics and policy analysis. We provide simulations studies and demonstrate how we can replicate the analysis in a published economics article on privacy-protected data under various parametrizations. We find that relatively straightforward DP-based methods allow for inference-valid protection of the published data, though computational issues may limit more complex analyses from using these methods. The results have applicability to researchers wishing to share RCT data, especially in the context of low- and middle-income countries, with strong privacy protection.

3. Reproducibility and Transparency versus Privacy and Confidentiality: Reflections from a Data Editor

   Lars Vilhuber

   Journal of Econometrics, Jun 2023

   DOI
4. Reproducibility and Transparency versus Privacy and Confidentiality: Reflections from a Data Editor

   Lars Vilhuber

   Jun 2023

   Abs DOI

   Transparency and reproducibility are often seen in opposition to privacy and confidentiality. Data that need to be kept confidential are seen as an impediment to reproducibility, and privacy would seem to inhibit transparency. I bring a more nuanced view to the discussion, and show, using examples from over 1,000 reproducibility assessments, that confidential data can very well be used in reproducible and transparent research. The key insight is that access to most confidential data, while tedious, is open to hundreds if not thousands of researchers. In cases where few researchers can consider accessing such data in the future, reproducibility services, such as those provided by some journals, can provide some evidence for effective reproducibility even when the same data may not be available for future research.

2022

1. Report by the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2022

   DOI
2. Teaching for Large-Scale Reproducibility Verification

   Lars Vilhuber, Hyuk Harry Son, Meredith Welch, and 2 more authors

   Journal of Statistics and Data Science Education, Sep 2022

   Abs DOI

   We describe a unique environment in which undergraduate students from various STEM and social science disciplines are trained in data provenance and reproducible methods, and then apply that knowledge to real, conditionally accepted manuscripts and associated replication packages. We describe in detail the recruitment, training, and regular activities. While the activity is not part of a regular curriculum, the skills and knowledge taught through explicit training of reproducible methods and principles, and reinforced through repeated application in a real-life workflow, contribute to the education of these undergraduate students, and prepare them for post-graduation jobs and further studies.

2021

1. Report by the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2021

   DOI
2. Disclosure Limitation and Confidentiality Protection in Linked Data

   John M. Abowd, Ian M. Schmutte, and Lars Vilhuber

   In Administrative Records for Survey Methodology, Apr 2021

   Abs

   Confidentiality protection for linked administrative data is a combination of access modalities and statistical disclosure limitation. We review traditional statistical disclosure limitation methods and newer methods based on synthetic data, input noise infusion and formal privacy. We discuss how these methods are integrated with access modalities by providing three detailed examples. The first example is the linkages in the Health and Retirement Study to Social Security Administration data. The second example is the linkage of the Survey of Income and Program Participation to administrative data from the Internal Revenue Service and the Social Security Administration. The third example is the Longitudinal Employer-Household Dynamics data, which links state unemployment insurance records for workers and firms to a wide variety of censuses and surveys at the U.S. Census Bureau. For examples, we discuss access modalities, disclosure limitation methods, the effectiveness of those methods, and the resulting analytical validity. The final sections discuss recent advances in access modalities for linked administrative data.

3. Handbook on Using Administrative Data for Research and Evidence-based Policy

   Shawn Cole, Iqbal Dhaliwal, Anja Sautmann, and 1 more author

   Jan 2021

   Abs DOI

   The Handbook serves as a go-to reference for researchers seeking to use administrative data and for data providers looking to make their data accessible for research. The handbook is published online under an open licensing model and freely available to all. It provides information, best practices, and case studies on how to create privacy-protected access to, handle, and analyze administrative data, with the aim of pushing the research frontier as well as informing evidence-based policy innovations.

4. Using Administrative Data for Research and Evidence-Based Policy: An Introduction

   Shawn Cole, Iqbal Dhaliwal, Anja Sautmann, and 1 more author

   In Handbook on Using Administrative Data for Research and Evidence-based Policy, Jan 2021

   DOI
5. Balancing Privacy and Data Usability: An Overview of Disclosure Avoidance Methods

   Ian M. Schmutte and Lars Vilhuber

   In Handbook on Using Administrative Data for Research and Evidence-based Policy, Jan 2021

   Abs DOI

   The Five Safes framework (safe projects, safe people, safe settings, safe data, and safe outputs) is one way of thinking about security of different aspects of a project, and is used throughout the Handbook and in research with administrative data. Within the Five Safes framework, data providers need to create safe data that can be provided to trusted safe people for use within safe settings, as part of safe projects. Finally, any findings that are shared publicly must be safe outputs. The processes used to create safe data and safe outputs (manipulations that render data less sensitive and therefore more appropriate for public release) are generally referred to as statistical disclosure limitation (SDL). This chapter describes techniques traditionally used within the field of SDL, pointing at methods as well as metrics to assess the resultant statistical quality and sensitivity of the data, and offers technical guidance applicable to any data provider or researcher looking for practical tools to apply to their own data to reduce the risk to privacy.

6. Physically Protecting Sensitive Data

   Jim Shen and Lars Vilhuber

   In Handbook on Using Administrative Data for Research and Evidence-based Policy, Jan 2021

   Abs DOI

   Keeping sensitive data safe relies heavily on the physical environments in which data are stored, processed, transmitted, and accessed, and from which researchers can access computers that store and process the data. However, it is also the setting that is most dependent on rapidly evolving technology. The chapter provides snapshot of the technologies available and in use as of 2020, and characterizes the technologies along a multi-dimensional scale, allowing for some comparability across methods.

2020

1. Report by the AEA Data Editor

   Lars Vilhuber, James Turitto, and Keesler Welch

   AEA Papers and Proceedings, May 2020

   DOI
2. Applying Data Synthesis for Longitudinal Business Data across Three Countries

   M. Jahangir Alam, Benoit Dostie, Jörg Drechsler, and 1 more author

   Statistics in Transition New Series, May 2020

   DOI
3. Data for: Requesting Replication Materials via Email

   Lars Vilhuber

   Nov 2020

   Abs DOI

   No description provided.

4. Migrating Historical AEA Supplements

   Lars Vilhuber

   May 2020
5. Reproducibility and Replicability in Economics

   Lars Vilhuber

   Harvard Data Science Review, Dec 2020

   DOI

2019

1. Report by the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2019

   DOI
2. Why the Economics Profession Must Actively Participate in the Privacy Protection Debate

   John M. Abowd, Ian M. Schmutte, William N. Sexton, and 1 more author

   AEA Papers and Proceedings, May 2019

   Abs DOI

   When Google or the U.S. Census Bureau publish detailed statistics on browsing habits or neighborhood characteristics, some privacy is lost for everybody while supplying public information. To date, economists have not focused on the privacy loss inherent in data publication. In their stead, these issues have been advanced almost exclusively by computer scientists who are primarily interested in technical problems associated with protecting privacy. Economists should join the discussion, first, to determine where to balance privacy protection against data quality; a social choice problem. Furthermore, economists must ensure new privacy models preserve the validity of public data for economic research.

2018

1. Disclosure Limitation and Confidentiality Protection in Linked Data

   John M. Abowd, Ian M. Schmutte, and Lars Vilhuber

   Jan 2018

   Abs

   Confidentiality protection for linked administrative data is a combination of access modalities and statistical disclosure limitation. We review traditional statistical disclosure limitation methods and newer methods based on synthetic data, input noise infusion and formal privacy. We discuss how these methods are integrated with access modalities by providing three detailed examples. The first example is the linkages in the Health and Retirement Study to Social Security Administration data. The second example is the linkage of the Survey of Income and Program Participation to administrative data from the Internal Revenue Service and the Social Security Administration. The third example is the Longitudinal Employer-Household Dynamics data, which links state unemployment insurance records for workers and firms to a wide variety of censuses and surveys at the U.S. Census Bureau. For examples, we discuss access modalities, disclosure limitation methods, the effectiveness of those methods, and the resulting analytical validity. The final sections discuss recent advances in access modalities for linked administrative data.

2. The Reproducibility of Economics Research: A Case Study

   Hautahi Kingi, Flavio Stanchi, Lars Vilhuber, and 1 more author

   Jan 2018

2017

1. Making Confidential Data Part of Reproducible Research

   C Lagoze and L Vilhuber

   Chance, Jan 2017

   Abs

   The rise of data-centric research practices has uncovered shortcomings in the traditional scholarly communication system. The foundation of that system, the peer-reviewed publication,“[the] selective distribution of ink on paper, or… electronic facsimiles of the same”(Bourne, et al., 2011), does not adequately support what has become an essential element of scholarship; the reproducibility of research results. That is, duplicating a ...