data-doc

Document datasets, variables, sources, and merge keys for replication

SKILL.md

GitHub

Data Documentation Assistant

Help document datasets systematically for replication packages, co-author handoffs, and future reference.

Dataset Documentation Template

When documenting a dataset, capture:

1. Dataset Overview

  • Name: (e.g., firm_year_panel.parquet)
  • Location: (relative path from project root)
  • Created by: (script that generates it)
  • Last updated:
  • Unit of observation: (firm-year, loan-quarter, etc.)
  • N observations:
  • Time coverage:

2. Source Data

Source Access Raw File Notes
WRDS Compustat Subscription raw/compustat_funda.csv Annual fundamentals
Revelio Labs Licensed raw/revelio_positions.parquet Via BU server
Hand-collected Manual raw/manual_coding.xlsx See coding protocol

3. Key Variables

Variable Type Description Source Notes
gvkey str Compustat firm identifier Compustat Primary key
fyear int Fiscal year Compustat
at float Total assets ($ millions) Compustat Winsorized 1/99
treated int =1 if treated firm Constructed See section 4

4. Variable Construction

For constructed/derived variables, document:

Variable: treated
Definition: =1 if firm received first PE investment in year t
Construction:
  1. Merge PitchBook deals to Compustat on EIN
  2. Keep first deal per firm
  3. Flag year of first investment
Script: 2a_construct_treatment.py, lines 45-78

5. Sample Filters

Document all filters applied:

Filter Observations Dropped Remaining
Raw data - 150,000
Drop financials (SIC 6000-6999) 25,000 125,000
Require non-missing assets 5,000 120,000
Require 2+ years in panel 10,000 110,000

6. Merge Keys

Dataset A Dataset B Key(s) Match Rate Notes
Compustat CRSP gvkey 95% Via CCM link table
Compustat PitchBook EIN 72% Manual cleaning needed
Revelio Compustat company_name 68% Fuzzy match, see script

Quick Commands

  • "document this dataset" - Generate full template for a dataset
  • "variable list" - Create variable table only
  • "merge documentation" - Focus on merge keys and match rates
  • "sample flow" - Generate sample filter table
  • "codebook" - Formal codebook format for replication package

Codebook Format (for Replication Packages)

================================================================================
CODEBOOK: firm_year_panel.dta
Generated: 2026-01-15
================================================================================

IDENTIFICATION
  gvkey         Compustat permanent firm identifier
  fyear         Fiscal year

OUTCOME VARIABLES
  roa           Return on assets = ni/at (winsorized 1/99)
  investment    Capex/lagged assets = capx/L.at

TREATMENT VARIABLES
  post          =1 for years after treatment
  treated       =1 for firms ever treated
  treat_post    Interaction: treated × post (DiD coefficient)

CONTROLS
  size          Log total assets = ln(at)
  leverage      Book leverage = (dltt+dlc)/at
  mtb           Market-to-book = (prcc_f×csho)/ceq

FIXED EFFECTS
  ff48          Fama-French 48 industry classification
  state         State of incorporation
================================================================================

Data Provenance Checklist

Before finalizing a dataset, verify:

  • All source files documented with access instructions
  • Variable definitions are unambiguous
  • Sample filters documented with observation counts
  • Merge match rates reported
  • Winsorization/trimming documented
  • Missing value treatment documented
  • Script that creates dataset is identified
  • Date of creation recorded