About the Conference on Reproducibility and Replicability in Economics and the Social Sciences


The Conference on Reproducibility and Replicability in Economics and the Social Sciences is a series of virtual and in-person panels on the topics of reproducibility, replicability, and transparency in the social sciences. The purpose of scientific publishing is the dissemination of robust research findings, exposing them to the scrutiny of peers and other interested parties. Scientific articles should accurately and completely provide information on the origin and provenance of data and on the analytical and computational methods used. Yet in recent years, doubts about the adequacy of the information provided in scientific articles and their addenda have been voiced. The conferences will address the following topics: the initiation of research, the conduct of research, the preparation of research for publication, and the scrutiny after publication. Undergraduates, graduate students, and career researchers will be able to learn about best practices for transparent, reproducible, and scientifically sound research in the social sciences.

Session 1: Institutional support: Should journals verify reproducibility?

2022-09-27 @ 4:15 PM Eastern
Panelists: Guido Imbens (Stanford and Econometrica), Tim Salmon (SMU and Economic Inquiry), Toni Whited (University of Michigan and Journal of Financial Economics)
Different journals have different approaches towards enforcement of their data availability policies, ranging from a thorough and complete verification including running code and checking the output, to a cursory review of the files provided to make sure they appear satisfactory, to simply receiving the data and code package and archiving it on a website or a repository. What drives the choice of approach? What are the reasons behind such choices?

Session 2: Reproducibility and ethics approval - a conversation with IRBs

2022-10-25 @ 4:15 PM Eastern
online
Panelists: Michelle Meyer (Geisinger), Others (TBD)
One of the most crucial dimensions that Internal Review Boards are interested in are the protocols that researchers have in place to protect their subjects' privacy. This often leads to researchers writing in their IRB protocols that they will destroy their data once their project is complete. Understandably, however, destruction of data makes it impossible to verify and replicate work, which is increasingly becoming a vital part of modern science. How should data privacy be handled in the wake of the replication crisis? What protocols and standards should be put in place to minimize the risk of data leakage? Or should data be destroyed after some time span?

Session 3: Should teaching reproducibility be a part of undergraduate education or curriculum? (at SEA)

2022-11-19 @ 4:15 PM Eastern
SEA (Registration)
Panelists: Diego Mendez-Carbajo (Fed of St. Louis), Richard Ball (Project TIER), Others (TBD)
Panelists will discuss teaching reproducibility (TIER Protocol), the involvement of undergraduates for replications based on restricted-access data, and other topics.

Session 4: Reproducibility and confidential or proprietary data: can it be done?

2022-12-13 **exceptionally** @ 12:15 PM Eastern
online
Panelists: Lars Vilhuber (Cornell and AEA Data Editor), Paulo GuimarĂ£es (Banco de Portugal, BPLIM), John Horton (MIT)
What happens to reproducibility when data are confidential or proprietary? Many journals can only ask that detailed access procedures be provided in a ReadMe file, but what mechanims could be used to conduct computational reproducibility checks on such data? Should authors temporarily share their data with the journal for the purposes of reproducibility verification, even if they are not part of the public data replication package? Is it feasible to use a network of "insiders" to run code provided as part of a data replication package to assess reproducibility? Could a "certified run" be used?

Session 5: Disciplinary support: why is reproducibility not uniformly required across disciplines?

2023-01-31 @ 4:15 PM Eastern
online
Panelists: Kim Weeden (Cornell, Sociology), Betsy Sinclair (WashU, Political Science), Hilary Hoynes (Berkeley, Economics)
Why do learned societies decide (or not) to implement data (and code) availability policies? What influences the level of enforcement, and the choice of "enforcer" (data editor, administrative staff, referees)? What are reasons NOT to require data sharing or code sharing?

Session 6: Institutional support: How do journal reproducibility verification services work?

2023-02-28 @ 4:15 PM Eastern
online
When journals conduct active verification of replication packages, including accessing data and running code, how does that work? Can journals with limited resources still assess reproducibility? What depth of verification is optimal? Do journals provide a clear indication of whether an article was successfully reproduced?

Session 7: Why can or should academic institutions publish replication packages?

2023-03-28 @ 4:15 PM Eastern
online

Session 8: Whether and How to integrate reproducibility into Social Science graduate education?

2023-04-25 @ 4:15 PM Eastern
online

Session 9: Should funders require reproducible archives?

2023-05-30 @ 4:15 PM Eastern
online
Panelists: Josh Greenberg (Sloan), Sebastian Martinez (3ie), Stuart Buck (Good Science Project)

Session 10: Reproducibility, confidentiality, and open data mandates (at CEA (tentative))

2023-06-xx @ 4:15 PM Eastern
CEA (tentative)
Many granting agencies have adopted open data mandates. What is the interplay between reproducibility and those mandates? How can researchers be supported to meet those mandates, both in general, and specifically when data are confidential. At first glance, confidentiality and open data seem irreconciable, but could we find practices that both respect confidentiality and provide enough information and transparency to foster reproducibility?

Session 11: Whether and How to integrate reproducibility into Social Science graduate education? (2nd round) (at WEAI (tentative))

2023-07-xx @ 4:15 PM Eastern
WEAI (tentative)

Format


During the Session

Each session is a 60-minute long online videoconference, tentatively scheduled for Tuesdays at 4:15 PM Eastern. Each panelist will speak for 10-15 minutes, followed by 15 minutes of discussion and audience questions, selected by the moderator. Certain sessions are in person at indicated conferences, in which case the availability of an online version depends on the conference facilities.

After the Session

Each session is recorded, and a link to the recording made available shortly afterwards on this page. Panelists are also asked to write a 5-7 page version of their statement, which will be collected as an online "book" on this site, and potentially as part of one or more special issues in journals.



Contact

For more information, or if you are a presenter and have questions, please contact us.