Publications

2026

1. Report of the Oversight Committee for Registry of Randomized Controlled Trials

   Lars Vilhuber

   AEA Papers and Proceedings, May 2026

   DOI
2. Reproducibility and Robustness of Economics and Political Science Research

   Abel Brodeur, Derek Mikola, Nikolai Cook, and 23 more authors

   Nature, Apr 2026

   Abs DOI

   This study pushes our understanding of research reliability by reproducing and replicating claims from 110 papers in leading economic and political science journals. The analysis involves computational reproducibility checks and robustness assessments. It reveals several patterns. First, we uncover a high rate of fully computationally reproducible results (over 85%). Second, excluding minor issues like missing packages or broken pathways, we uncover coding errors for about 25% of studies, with some studies containing multiple errors. Third, we test the robustness of the results to 5,511 re-analyses. We find a robustness reproducibility of about 70%. Robustness reproducibility rates are relatively higher for re-analyses that introduce new data and lower for re-analyses that change the sample or the definition of the dependent variable. Fourth, 52% of re-analysis effect size estimates are smaller than the original published estimates and the average statistical significance of a re-analysis is 77% of the original. Lastly, we rely on six teams of researchers working independently to answer eight additional research questions on the determinants of robustness reproducibility. Most teams find a negative relationship between replicators’ experience and reproducibility, while finding no relationship between reproducibility and the provision of intermediate or even raw data combined with the necessary cleaning codes.

3. AI-assisted Teams Outperform AI-led Teams but Not Human-Only Teams in Assessing Research Reproducibility in Quantitative Social Science

   Abel Brodeur, David Valenta, Alexandru Marcoci, and 1 more author

   Proceedings of the National Academy of Sciences, Jun 2026

   Abs DOI

   Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time.

4. Randomness in Large Language Models: What Researchers Need to Know (and Report)

   Guillaume Coqueret, Joan Llull, Florian Oswald, and 3 more authors

   Jul 2026

   Abs DOI

   Large language models (LLMs) are increasingly used to generate data for research. Typical use cases are classifications, annotations, information extraction, and generation of numerical scores. Unlike conventional measurements, LLM outputs can vary across repeated requests even when the prompt and apparent model settings remain unchanged. This variation arises from deliberate sampling, silent model updates, numerical rounding, or expert routing. Setting a dedicated temperature parameter to zero removes deliberate sampling when that option is available, but it does not eliminate the other sources of randomness. Exact reproduction is therefore generally not possible when using proprietary application programming interfaces. Local execution of open-weight models offers greater control, but reproducibility still depends on the complete hardware and software stack. We illustrate these issues through sentiment classifications of corporate filings and examine their consequences for downstream regression results. We then propose a reporting standard for articles and replication packages, as well as guidance for data editors and authors. Together, these findings and recommendations establish that LLM outputs should be treated as draws from a distribution rather than as fixed measurements.

5. Open Science in Den Wirtschaftswissenschaften: Transparenz, Reproduzierbarkeit Und Zugang

   Klaus M. Schmidt, Levent Neyse, Marianne Saam, and 3 more authors

   Apr 2026

   Abs

   Der Beitrag diskutiert Open Science in den Wirtschaftswissenschaften als Bündel von Praktiken zur Verbesserung von Transparenz, Reproduzierbarkeit und Zugänglichkeit wissenschaftlicher Forschung. Der Artikel zeigt, dass Präregistrierungen und Registered Reports, Open Data und Open Code sowie Open Access die Glaubwürdigkeit empirischer Forschung stärken können, zugleich aber disziplinspezifische Grenzen und Zielkonflikte berücksichtigen müssen. Für die Wirtschaftswissenschaften und die Forschungsförderung folgt daraus die Notwendigkeit verlässlicher Infrastrukturen, klarer Standards und nachhaltiger institutioneller Unterstützung, insbesondere für nicht-kommerzielle Open-Access-Modelle wie Diamond Open Access.

6. Open Science in Den Wirtschaftswissenschaften: Transparenz, Reproduzierbarkeit Und Zugang

   Klaus M. Schmidt, Levent Neyse, Marianne Saam, and 3 more authors

   Perspektiven der Wirtschaftspolitik, Jun 2026

   DOI
7. Improving Results Reporting in RCT Registries

   Jack Cavanagh, Sarah Kopper, and Lars Vilhuber

   submitted, Jun 2026

   Abs

   The complete registration of trials is a critical component of transparent science. Completeness requires not only initial registration, but also post-study updates. We show this practice is lacking in the largest social science registry – the AEA RCT Registry – and conduct a mixed-methods study to first determine drivers and then test interventions to improve updating rates. In a qualitative study of Registry users we find salience and lack of incentives to be the largest barriers. From an RCT we find limited evidence of differences between email nudges, but substantial reporting increases relative to baseline from the overall supernormal number of reminders.

2025

1. Assessing Reproducibility in Economics Using Standardized Crowd-sourced Analysis

   Abel Brodeur, Seung Yong Sung, Edward Miguel, and 2 more authors

   May 2025

   Abs DOI

   This paper presents a framework to standardize crowd-sourced computational reproductions in economics through the Social Science Reproduction Platform (SSRP). The approach address four main challenges for computational reproductions: a lack of standardization, aggregation issues, existing incentives for “adversarial” interactions, and the loss of knowledge from analyses that are never published. We then summarize the first 487 reproductions uploaded on the SSRP. The results show substantial heterogeneity in the ability to successfully reproduce empirical results in economics research, with approximately 30% of recent studies meeting at least a basic definition of being computationally reproducible.

2. TROV - A Model and Vocabulary for Describing Transparent Research Objects

   Meng Li, Timothy McPhillips, Craig Willis, and 6 more authors

   International Journal of Digital Curation, Feb 2025

   Abs DOI

   The Transparent Research Object Vocabulary (TROV) is a key element of the Transparency Certified (TRACE) approach to ensuring research trustworthiness. In contrast with methods that entail repeating computations in part or in full to verify that the descriptions of methods included in a publication are sufficient to reproduce reported results, the TRACE approach depends on a controlled computing environment termed a Transparent Research System (TRS) to guarantee that accurate, sufficiently complete, and otherwise trustworthy records are captured when results are obtained in the first place. Records identifying (1) the digital artifacts and computations that yielded a research result, (2) the TRS that witnessed the artifacts and supervised the computations, and (3) the specific conditions enforced by the TRS that warrant trust in these records, together constitute a Transparent Research Object (TRO). Digital signatures provided by the TRS and by a trusted third-party timestamp authority (TSA) guarantee the integrity and authenticity of the TRO. The controlled vocabulary TROV provides means to declare and query the properties of a TRO, to enumerate the dimensions of trustworthiness the TRS asserts for a TRO, and to verify that each such assertion is warranted by the documented capabilities of the TRS. Our approach for describing, publishing, and working with TROs imposes no restrictions on how computational artifacts are packaged or otherwise shared, and aims to be interoperable with, rather than to replace, current and future Research Object standards, archival formats, and repository layouts.

3. Report of the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2025
4. Using Containers to Validate Research on Confidential Data at Scale

   Lars Vilhuber

   Harvard Data Science Review, Jun 2025

   Abs DOI

   I describe past experience with the validation server process over 10 years and several hundred users, as a means to provide proxy access to confidential data. As a modern replacement, I propose the use of containers—simulated computers that encapsulate entire software and file structures. The use of containers ensures reproducibility, reliable portability, and enables scalability. Infrastructure can be outsourced to commercial providers or users, at little to no cost to data providers. The only likely limitation to full automation is the absence of automated output vetting algorithms at statistical agencies.

5. Comparing Human-Only, AI-assisted, and AI-led Teams on Assessing Research Reproducibility in Quantitative Social Science

   Abel Brodeur, David Valenta, Alexandru Marcoci, and 1 more author

   Jan 2025

   Abs

   Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification.

6. Reproducibility and Open Science in Economics

   Lars Vilhuber

   Revue économique, Jan 2025

   Abs DOI Code

   Drawing on my experience as the American Economic Association’s data editor, I examine the current state of open science in economics as facilitated by and related to reproducibility. I touch on the tension between accessibility, sharing, and preservation. The guiding theme is the accessibility of the key ingredients for scholarship: manuscripts, data, software, and the necessary technology to combine the latter two in order to produce knowledge. I analyze how economic research balances openness with necessary restrictions, particularly regarding administrative and confidential data. I argue that a large degree of openness is nevertheless present, with extensive networks that include thousands of researchers supporting collaborative science.

2024

1. Report of the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2024

   DOI
2. Mass Reproducibility and Replicability: A New Hope

   Abel Brodeur, Derek Mikola, Nikolai Cook, and 346 more authors

   May 2024

   Abs

   This study pushes our understanding of research reliability by reproducing and replicating claims from 110 papers in leading economic and political science journals. The analysis involves computational reproducibility checks and robustness assessments. It reveals several patterns. First, we uncover a high rate of fully computationally reproducible results (over 85%). Second, excluding minor issues like missing packages or broken pathways, we uncover coding errors for about 25% of studies, with some studies containing multiple errors. Third, we test the robustness of the results to 5,511 re-analyses. We find a robustness reproducibility of about 70%. Robustness reproducibility rates are relatively higher for re-analyses that introduce new data and lower for re-analyses that change the sample or the definition of the dependent variable. Fourth, 52% of re-analysis effect size estimates are smaller than the original published estimates and the average statistical significance of a re-analysis is 77% of the original. Lastly, we rely on six teams of researchers working independently to answer eight additional research questions on the determinants of robustness reproducibility. Most teams find a negative relationship between replicators’ experience and reproducibility, while finding no relationship between reproducibility and the provision of intermediate or even raw data combined with the necessary cleaning codes.

3. Reproduce to Validate: A Comprehensive Study on the Reproducibility of Economics Research

   Sylvérie Herbert, Hautahi Kingi, Flavio Stanchi, and 1 more author

   Canadian Journal of Economics/Revue canadienne d’économique, Aug 2024

   Abs DOI

   Abstract Journals have pushed for transparency of research through data availability policies. Such data policies improve availability of data and code, but what is the impact on reproducibility? We present results from a large reproduction exercise for articles published in the American Economic Journal: Applied Economics, which has had a data availability policy since its inception in 2009. Out of 363 published articles, we assessed 274 articles. All articles provided some materials. We excluded 122 articles that required confidential or proprietary data or that required the replicator to otherwise obtain the data (44.5% of assessed articles). We attempted to reproduce 152 articles and were able to fully reproduce the results of 68 (44.7% of attempted reproductions). A further 66 (43.4% of attempted reproductions) were partially reproduced. Many articles required complex code changes even when at least partially reproduced. We collect bibliometric characteristics of authors, but find no evidence for author characteristics as determinants of reproducibility. There does not appear to be a citation bonus for reproducibility. The data availability policy of this journal was effective to ensure availability of materials, but is insufficient to ensure reproduction without additional work by replicators. , Résumé Les journaux militent pour la transparence de la recherche par le biais de politiques sur la disponibilité des données . De telles politiques sur les données améliorent la disponibilité des données et du code, mais quelle en est l’incidence sur la reproductibilité? Nous présentons les résultats d’un grand exercice de reproduction pour des articles publiés dans le American Economic Journal: Applied Economics , qui a une politique de disponibilité des données depuis sa création en 2009. Parmi les 363 articles publiés, nous en évaluons 274. Tous les articles avaient fourni certains documents. Nous avons exclu 122 articles qui nécessitaient des données confidentielles ou exclusives, ou qui exigeaient que la personne chargée de la réplication obtienne autrement les données (44,5% des articles évalués). Nous avons tenté de reproduire 152 articles et avons réussi à entièrement reproduire les résultats de 68 articles (44,7% des reproductions tentées). Nous avons aussi partiellement reproduit 66 autres articles (43,4% des reproductions tentées). De nombreux articles nécessitaient de complexes changements de code, même lorsqu’ils étaient au moins partiellement reproduits. Nous avons recueilli les caractéristiques bibliométriques des auteurs, mais n’avons pas constaté qu’elles étaient déterminantes pour la reproductibilité. La reproductibilité ne semble offrir aucun avantage en matière de citations. La politique de disponibilité des données de ce journal a été efficace pour assurer la disponibilité des documents, mais insuffisante pour assurer la reproduction sans travail supplémentaire des personnes chargées de la réplication.

4. Crowdsourcing Digital Public Goods: A Field Experiment on Metadata Contributions

   Linfeng Li, Yan Chen, Margaret C. Levenstein, and 1 more author

   Nov 2024

   Abs DOI

   This study explores why people choose to contribute metadata, which is data about data. Using a field experiment conducted with more than 3,000 authors of AEA journal articles, our control message reduces the uncertainty about the future value of metadata, whereas those from the treatment conditions additionally make the private or social benefits of metadata salient. Surprisingly, we find that participants in the control condition provide significantly more metadata compared to those in the treatments. This suggests that simply knowing that metadata will have value is sufficient to motivate people to contribute. Our results also highlight the importance of interface design in online field experiments.

5. A Guide for Social Science Journal Editors on Easing into Open Science

   Priya Silverstein, Colin Elman, Amanda Montoya, and 36 more authors

   Research Integrity and Peer Review, Feb 2024

   Abs DOI

   Journal editors have a large amount of power to advance open science in their respective fields by incentivizing and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org) has collated several resources on embedding open science in journal editing (www.dpjedi.org/resources). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we have created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx)).

6. Protecting Confidential Data through Non-Statistical Methods

   Lars Vilhuber

   In Handbook of Sharing Confidential Data: Differential Privacy, Secure Multiparty Computation, and Synthetic Data, Feb 2024

   Abs

   "Statistical agencies, research organizations, companies, and other data stewards that seek to share data with the public face a challenging dilemma. They need to protect the privacy and confidentiality of data subjects’ and their attributes while providing data products that are useful for their intended purposes. In an age when information on data subjects is available from a wide range of data sources, as are the computational resources to obtain that information, this challenge is increasingly difficult. The Handbook of Sharing Confidential Data helps data stewards understand how tools from the data confidentiality literature-specifically, synthetic data, formal privacy, and secure computation-can be used to manage trade-offs in disclosure risk and data usefulness"–

7. Protecting Confidential Data through Non-Statistical Methods

   Lars Vilhuber

   Oct 2024

   Abs

   This chapter will rely on and update previous overviews of how researchers, citizens, and administrators can reliably and securely access confidential data, i.e., data that cannot be simply published as “open data”. I will discuss various legal, technical, and practical ways of securing access to data that is needed for computations. This obviously depends on the type and complexity of the computations but also depends on the who, how, and where access is needed.

2023

1. Report of the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2023

   DOI
2. Assessing Utility of Differential Privacy for RCTs

   Soumya Mukherjee, Aratrika Mustafi, Aleksandra Slavković, and 1 more author

   Sep 2023

   Abs DOI

   Randomized control trials, RCTs, have become a powerful tool for assessing the impact of interventions and policies in many contexts. They are considered the gold-standard for inference in the biomedical fields and in many social sciences. Researchers have published an increasing number of studies that rely on RCTs for at least part of the inference, and these studies typically include the response data collected, de-identified and sometimes protected through traditional disclosure limitation methods. In this paper, we empirically assess the impact of strong privacy-preservation methodology (with \ac\textbraceleft DP\textbraceright guarantees), on published analyses from RCTs, leveraging the availability of replication packages (research compendia) in economics and policy analysis. We provide simulations studies and demonstrate how we can replicate the analysis in a published economics article on privacy-protected data under various parametrizations. We find that relatively straightforward DP-based methods allow for inference-valid protection of the published data, though computational issues may limit more complex analyses from using these methods. The results have applicability to researchers wishing to share RCT data, especially in the context of low- and middle-income countries, with strong privacy protection.

3. Reproducibility and Transparency versus Privacy and Confidentiality: Reflections from a Data Editor

   Lars Vilhuber

   Journal of Econometrics, Jun 2023

   DOI
4. Reproducibility and Transparency versus Privacy and Confidentiality: Reflections from a Data Editor

   Lars Vilhuber

   Jun 2023

   Abs DOI

   Transparency and reproducibility are often seen in opposition to privacy and confidentiality. Data that need to be kept confidential are seen as an impediment to reproducibility, and privacy would seem to inhibit transparency. I bring a more nuanced view to the discussion, and show, using examples from over 1,000 reproducibility assessments, that confidential data can very well be used in reproducible and transparent research. The key insight is that access to most confidential data, while tedious, is open to hundreds if not thousands of researchers. In cases where few researchers can consider accessing such data in the future, reproducibility services, such as those provided by some journals, can provide some evidence for effective reproducibility even when the same data may not be available for future research.

5. The Reproducibility of Economics Research: A Case Study

   Sylverie Herbert, Hautahi Kingi, Flavio Stanchi, and 1 more author

   Jun 2023

   Abs DOI

   Given the importance of reproducibility for the scientific ethos, more and more journals have pushed for transparency of research through data availability policies. If the introduction and implementation of such data policies improve the availability of researchers’ code and data, what is the impact on reproducibility? We describe and present the results of a large reproduction exercise in which we assess the reproducibility of research articles published in the American Economic Journal: Applied Economics, which has implemented a data availability policy since 2005. Our replication success rate is relatively moderate, with 37.78% of replication attempts successful. 68 of 162 eligible replication attempts successfully replicated the article’s analysis (41.98%) conditional on non-confidential data. A further 69 (42.59%) were at least partially successful. A total of 98 out of 303 (32.34%) relied on confidential or proprietary data, and were thus not reproducible by this project. We also conduct several bibliometric analyses of reproducible vs. non-reproducible articles and show that replicable papers do not provide citation bonuses for authors.

6. Reinforcing Reproducibility and Replicability: An Introduction

   Lars Vilhuber, Ian Schmutte, Aleksandr Michuda, and 1 more author

   Harvard Data Science Review, Jul 2023

   Abs DOI

   The purpose of scientific publishing is the dissemination of robust research findings, exposing them to the scrutiny of peers. The key to this endeavor is documenting the provenance of those findings. Scientific practices during the course of research and subsequent publication, peer review, and dissemination practices and tools, all interact to (hopefully) enable a meaningful discourse about the veracity of scientific claims. However, while all practices and tools contribute to the final output, some are less often discussed than others, and perceptions, usage, and acceptance differ in myriad ways across disciplines. In this special theme, and in a subsequent column called "Reinforcing Reproducibility and Replicability," we will explore these topics, with expert providers and expert users providing their input.

2022

1. Report by the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2022

   DOI
2. Teaching for Large-Scale Reproducibility Verification

   Lars Vilhuber, Hyuk Harry Son, Meredith Welch, and 2 more authors

   Journal of Statistics and Data Science Education, Sep 2022

   Abs DOI

   We describe a unique environment in which undergraduate students from various STEM and social science disciplines are trained in data provenance and reproducible methods, and then apply that knowledge to real, conditionally accepted manuscripts and associated replication packages. We describe in detail the recruitment, training, and regular activities. While the activity is not part of a regular curriculum, the skills and knowledge taught through explicit training of reproducible methods and principles, and reinforced through repeated application in a real-life workflow, contribute to the education of these undergraduate students, and prepare them for post-graduation jobs and further studies.

3. Data and Code Availability Standard

   Miklós Koren, Marie Connolly, Joan Lull, and 1 more author

   Dec 2022

   Version Number: 1.0

   Abs DOI

   DCAS is a standard for sharing research code and data, endorsed by leading journals in social sciences. It is maintained by the Social Science Data Editors.

4. A Template README for Social Science Replication Packages

   Lars Vilhuber, Marie Connolly, Miklós Koren, and 2 more authors

   Nov 2022

2021

1. Report by the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2021

   DOI
2. Disclosure Limitation and Confidentiality Protection in Linked Data

   John M. Abowd, Ian M. Schmutte, and Lars Vilhuber

   In Administrative Records for Survey Methodology, Apr 2021

   Abs

   Confidentiality protection for linked administrative data is a combination of access modalities and statistical disclosure limitation. We review traditional statistical disclosure limitation methods and newer methods based on synthetic data, input noise infusion and formal privacy. We discuss how these methods are integrated with access modalities by providing three detailed examples. The first example is the linkages in the Health and Retirement Study to Social Security Administration data. The second example is the linkage of the Survey of Income and Program Participation to administrative data from the Internal Revenue Service and the Social Security Administration. The third example is the Longitudinal Employer-Household Dynamics data, which links state unemployment insurance records for workers and firms to a wide variety of censuses and surveys at the U.S. Census Bureau. For examples, we discuss access modalities, disclosure limitation methods, the effectiveness of those methods, and the resulting analytical validity. The final sections discuss recent advances in access modalities for linked administrative data.

3. Handbook on Using Administrative Data for Research and Evidence-based Policy

   Shawn Cole, Iqbal Dhaliwal, Anja Sautmann, and 1 more author

   Jan 2021

   Abs DOI

   The Handbook serves as a go-to reference for researchers seeking to use administrative data and for data providers looking to make their data accessible for research. The handbook is published online under an open licensing model and freely available to all. It provides information, best practices, and case studies on how to create privacy-protected access to, handle, and analyze administrative data, with the aim of pushing the research frontier as well as informing evidence-based policy innovations.

4. Using Administrative Data for Research and Evidence-Based Policy: An Introduction

   Shawn Cole, Iqbal Dhaliwal, Anja Sautmann, and 1 more author

   In Handbook on Using Administrative Data for Research and Evidence-based Policy, Jan 2021

   DOI
5. Balancing Privacy and Data Usability: An Overview of Disclosure Avoidance Methods

   Ian M. Schmutte and Lars Vilhuber

   In Handbook on Using Administrative Data for Research and Evidence-based Policy, Jan 2021

   Abs DOI

   The Five Safes framework (safe projects, safe people, safe settings, safe data, and safe outputs) is one way of thinking about security of different aspects of a project, and is used throughout the Handbook and in research with administrative data. Within the Five Safes framework, data providers need to create safe data that can be provided to trusted safe people for use within safe settings, as part of safe projects. Finally, any findings that are shared publicly must be safe outputs. The processes used to create safe data and safe outputs (manipulations that render data less sensitive and therefore more appropriate for public release) are generally referred to as statistical disclosure limitation (SDL). This chapter describes techniques traditionally used within the field of SDL, pointing at methods as well as metrics to assess the resultant statistical quality and sensitivity of the data, and offers technical guidance applicable to any data provider or researcher looking for practical tools to apply to their own data to reduce the risk to privacy.

6. Physically Protecting Sensitive Data

   Jim Shen and Lars Vilhuber

   In Handbook on Using Administrative Data for Research and Evidence-based Policy, Jan 2021

   Abs DOI

   Keeping sensitive data safe relies heavily on the physical environments in which data are stored, processed, transmitted, and accessed, and from which researchers can access computers that store and process the data. However, it is also the setting that is most dependent on rapidly evolving technology. The chapter provides snapshot of the technologies available and in use as of 2020, and characterizes the technologies along a multi-dimensional scale, allowing for some comparability across methods.

7. Metajelo: A Metadata Package for Journals to Support External Linked Objects

   Carl Lagoze and Lars Vilhuber

   International Journal of Digital Curation, Jan 2021

   Abs DOI

   We propose a metadata package that is intended to provide academic journals with a lightweight means of registering, at the time of publication, the existence and disposition of supplementary materials. Information about the supplementary materials is, in most cases, critical for the reproducibility and replicability of scholarly results. In many instances, these materials are curated by a third party, which may or may not follow developing standards for the identification and description of those materials. As such, the vocabulary described here complements existing initiatives that specify vocabularies to describe the supplementary materials or the repositories and archives in which they have been deposited. Where possible, it reuses elements of relevant other vocabularies, facilitating coexistence with them. Furthermore, it provides an "at publication" record of reproducibility characteristics of a particular article that has been selected for publication. The proposed metadata package documents the key characteristics that journals care about in the case of supplementary materials that are held by third parties: existence, accessibility, and permanence. It does so in a robust, time-invariant fashion at the time of publication, when the editorial decisions are made. It also allows for better documentation of less accessible (non-public data), by treating it symmetrically from the point of view of the journal, therefore increasing the transparency of what up until now has been very opaque.

2020

1. Report by the AEA Data Editor

   Lars Vilhuber, James Turitto, and Keesler Welch

   AEA Papers and Proceedings, May 2020

   DOI
2. Applying Data Synthesis for Longitudinal Business Data across Three Countries

   M. Jahangir Alam, Benoit Dostie, Jörg Drechsler, and 1 more author

   Statistics in Transition New Series, May 2020

   DOI
3. Data for: Requesting Replication Materials via Email

   Lars Vilhuber

   Nov 2020

   Abs DOI

   No description provided.

4. Migrating Historical AEA Supplements

   Lars Vilhuber

   May 2020
5. Reproducibility and Replicability in Economics

   Lars Vilhuber

   Harvard Data Science Review, Dec 2020

   DOI
6. A Template README for Social Science Replication Packages

   Lars Vilhuber, Marie Connolly, Miklós Koren, and 2 more authors

   Dec 2020

   Version Number: v1.0.0

   Abs DOI

   The typical README in social science journals serves the purpose of guiding a reader through the available material and a route to replicating the results in the research paper, including the description of the origins of data and/or description of programs. As such, a good README file should first provide a brief overview of the available material and a brief guide as to how to proceed from beginning to end, before then diving into the specifics. These template files structure such a README in a way that is compliant with the typical data and code workflow in the social sciences.

2019

1. Report by the AEA Data Editor

   Lars Vilhuber

   AEA Papers and Proceedings, May 2019

   DOI
2. Why the Economics Profession Must Actively Participate in the Privacy Protection Debate

   John M. Abowd, Ian M. Schmutte, William N. Sexton, and 1 more author

   AEA Papers and Proceedings, May 2019

   Abs DOI

   When Google or the U.S. Census Bureau publish detailed statistics on browsing habits or neighborhood characteristics, some privacy is lost for everybody while supplying public information. To date, economists have not focused on the privacy loss inherent in data publication. In their stead, these issues have been advanced almost exclusively by computer scientists who are primarily interested in technical problems associated with protecting privacy. Economists should join the discussion, first, to determine where to balance privacy protection against data quality; a social choice problem. Furthermore, economists must ensure new privacy models preserve the validity of public data for economic research.

3. Metajelo: A Metadata Package for Journals to Support External Linked Objects

   Carl Lagoze and Lars Vilhuber

   May 2019
4. Making Confidential Data Part of Reproducible Research

   Lars Vilhuber

   In Methods to Foster Transparency and Reproducibility of Federal Statistics: Proceedings of a Workshop, May 2019

   Abs DOI

   In 2014 the National Science Foundation (NSF) provided support to the National Academies of Sciences, Engineering, and Medicine for a series of Forums on Open Science in response to a government-wide directive to support increased public access to the results of research funded by the federal government. However, the breadth of the work resulting from the series precluded a focus on any specific topic or discussion about how to improve public access. Thus, the main goal of the Workshop on Transparency and Reproducibility in Federal Statistics was to develop some understanding of what principles and practices are, or would be, supportive of making federal statistics more understandable and reviewable, both by agency staff and the public. This publication summarizes the presentations and discussions from the workshop.

2018

1. Disclosure Limitation and Confidentiality Protection in Linked Data

   John M. Abowd, Ian M. Schmutte, and Lars Vilhuber

   Jan 2018

   Abs

   Confidentiality protection for linked administrative data is a combination of access modalities and statistical disclosure limitation. We review traditional statistical disclosure limitation methods and newer methods based on synthetic data, input noise infusion and formal privacy. We discuss how these methods are integrated with access modalities by providing three detailed examples. The first example is the linkages in the Health and Retirement Study to Social Security Administration data. The second example is the linkage of the Survey of Income and Program Participation to administrative data from the Internal Revenue Service and the Social Security Administration. The third example is the Longitudinal Employer-Household Dynamics data, which links state unemployment insurance records for workers and firms to a wide variety of censuses and surveys at the U.S. Census Bureau. For examples, we discuss access modalities, disclosure limitation methods, the effectiveness of those methods, and the resulting analytical validity. The final sections discuss recent advances in access modalities for linked administrative data.

2. The Reproducibility of Economics Research: A Case Study

   Hautahi Kingi, Flavio Stanchi, Lars Vilhuber, and 1 more author

   Jan 2018
3. Reproducibility and Replicability in Economics

   Lars Vilhuber

   Jan 2018

2017

1. Making Confidential Data Part of Reproducible Research

   C Lagoze and L Vilhuber

   Chance, Jan 2017

   Abs

   The rise of data-centric research practices has uncovered shortcomings in the traditional scholarly communication system. The foundation of that system, the peer-reviewed publication,“[the] selective distribution of ink on paper, or… electronic facsimiles of the same”(Bourne, et al., 2011), does not adequately support what has become an essential element of scholarship; the reproducibility of research results. That is, duplicating a ...