2023-09-26

The Plan

9:00

  • Konnichi wa こんにちは
  • Overview of reproducibility and replicability in economics, including replication packages (presentation)

10:00-10:50 … Let’s make a replication package (hands-on using a simple example, from directory structure to (almost) deposit at a trusted repository) 0

11:00-11:50 … Principles and structure of replication packages (including README files) 1 2 For confidential data: 1b

11:50-13:00 … Lunch time

13:00-13:50 … Foundations of reproducible programming practices (in any programming language) 3 4

14:00-14:50 … Continuation and open discussion about specific examples (if you have one, bring your own project!) 5

15:00-15:50 … Discussions on institutional support for replicability in Japanese context (including Q&A) 15:50-16:00 … Closing remarks

Overview

  • Reproducible practices
  • Ideal directory and data structure
  • Some programming practices

Part 1

Ideal structure

Generic project setup

Basic project setup

Structure your project

  • Data inputs
  • Data outputs
  • Code
  • Paper/text/etc.

Version your project (git)!

Track metadata

  • cite articles you reference
  • cite data sources you use
    • We will get back to data citations later!

Project setup examples

/inputs
/outputs
/code
/paper
/datos/
    /brutos
    /limpiados
    /finales
/codigo
/articulo

It doesn’t really matter, as long as it is logical. We will get to how this translates to confidential or big data in a moment!

Computational Empathy

Consider how the next person will (be able to) compute

  • You don’t know who that is
  • You don’t know what they don’t know
  • Will not have any of your add-on packages/ libraries/ etc. pre-installed
  • Don’t force them to do tedious things

It might be “Future You!”

It IS you

The replicator is the first (?) reader of the instructions who will need to reproduce the analysis.

Streamlining

  • Master script preferred
    • Least amount of manual effort
  • No manual copying of results
    • (dynamic documents!)
    • Write out/save tables and figures using packages
  • Clear instructions
  • No manual install of packages
    • Use a script to create all directories, install all necessary packages/requirements/etc.

Reproducibility

  • No manual manipulation
    • “Change the parameter to 0.2, then run the code again”
    • Use functions, ado files, programs, macros, subroutines
    • Use loops, parameters, parameter files to call those subroutines
    • Use placeholders (globals, macros, libnames, etc.) for common locations ($CONFDATA, $TABLES, $CODE)
  • Compute all numbers in package
    • No manual calculation of numbers
  • Use cross-platform programming practices

Cross-platform programming practices 1

Use programming-language specific code as much as possible

Avoid

system("unzip C:\data\myfile.zip")

or

shell unzip "C:\data\myfile.zip"

Cross-platform programming practices 1

Most languages have appropriate code:

R:

unzip(zipfile, files = NULL, list = FALSE, overwrite = TRUE,
      junkpaths = FALSE, exdir = ".", unzip = "internal",
      setTimes = FALSE)

Stata:

unzipfile "zipfile.zip" [, replace]

Cross-platform programming practices 2

Use neutral pathnames (mostly forward slashes)

R: Use functions to combine paths (and/or use forward slashes), packages to make code more portable.

basepath <- rprojroot::find_root(rprojroot::has_file("README.md"))
data <- read.dta(file.path(basepath,"path","data.dta"))

Stata: always use forward slashes, even on Windows

global data "/my/computer"
use "$data/path/data.dta"

More complex Data structures

Back to the TIER protocol

TIER Protocol again

Generic suggested data structure

Simplified replication package structure

When data are big/in the cloud

Data are big

When data are confidential

Confidential data

When data are confidential

Confidential data in enclaves

Project setup examples

This may no longer work:

/data/
    /raw
    /clean
    /final
/code
/article

But this might

/project123/
     /data/
        /raw
        /clean
        /final
     /code
     /article
/confidential        (read-only)
     /taxes          (read-only)
     /wages          (read-only)

Stata configuration files

File structure thus becomes more complex, but fundamentally not so different:

global taxdata    "/confidential/taxes"  
global salarydata "/confidential/wages"  
global outputdata "/project/data/clean" // this is where you would write the data you create in this project
global results    "/project/article"    // All tables for inclusion in your paper go here
global programs   "/project/code"       // All programs (which you might "include") are to be found here

Stata configuration files

Or even more robust:

global basedir     "/project123"
global confbase    "/data/provided"
global project     "$basedir/project"

global taxdata    "$confbase/taxes"  
global salarydata "$confbase/wages"  
global outputdata "$project/data/clean" // this is where you would write the data you create in this project
global results    "$project/article"    // All tables for inclusion in your paper go here
global programs   "$project/code"       // All programs (which you might "include") are to be found here

Try it out!

  • Set up a directory structure
  • Set up a simple project
  • Share it with us!

Tomorrow

Follow-up