data-summary-statistics

Use when you have a curated relational dataset (structure-organism pairs) and need to quantify how structures distribute across a categorical variable (e.g., organism prevalence).

SKILL.md

GitHub

data-summary-statistics

Summary

Compute and report frequency distributions and summary counts across categorical bins to characterize the composition and coverage of a large structure-organism dataset. This skill validates data integrity by comparing observed bin membership against reference gold-standard counts.

When to use

Apply this skill when you have a curated relational dataset (structure-organism pairs) and need to quantify how structures distribute across a categorical variable (e.g., organism prevalence). Use it as a validation checkpoint to detect data loss, miscuration, or processing errors by comparing bin counts against published benchmarks.

When NOT to use

  • Input is a pre-computed summary table or aggregated report (you would be double-summarizing).
  • The analysis goal is to identify which specific structures or organisms are anomalous rather than to validate global bin distributions.
  • Organism prevalence counts have not been computed or validated upstream; use organism-count binning as a prerequisite first.

Inputs

  • Flat-file table of structure-organism pairs (TSV or TSV.GZ format)
  • Column defining unique structure identifier (SMILES or InChI)
  • Column defining organism association for each pair
  • Reference gold-standard bin counts (from published dataset version)

Outputs

  • Summary table with four rows (one per frequency bin) and columns for bin label, count, and discrepancy flag
  • Text or markdown report documenting bin membership and any divergence from reference counts
  • Validation flag (pass/fail) indicating whether observed counts match reference within tolerance

How to apply

Load the flat-file table of structure-organism pairs (e.g., from LOTUS as .tsv.gz). Group all unique 2D structures by their organism count (the number of distinct organisms in which each structure appears). Define four frequency bins: singleton structures (1 organism), low-diversity structures (2–10 organisms), medium-diversity structures (11–100 organisms), and high-diversity structures (>100 organisms). Count the number of unique structures falling into each bin. Generate a summary report tabulating bin membership counts and compute any numeric discrepancies between observed counts and the gold-standard reference counts reported in the literature or prior dataset releases. Use R or Python (pandas, groupby, value_counts) to aggregate and validate.

Related tools

Examples

python3 -c "import pandas as pd; df = pd.read_csv('lotus_pairs.tsv.gz', sep='\t', compression='gzip'); bins = df.groupby(df.groupby('structure_id').size().rename('org_count')).size(); print(f'Singleton: {bins[1]}, Low (2-10): {bins[2:11].sum()}, Medium (11-100): {bins[11:101].sum()}, High (>100): {bins[101:].sum()}')"

Evaluation signals

  • Observed bin counts for all four frequency categories (1, 2–10, 11–100, >100 organisms) are non-zero and match published LOTUS gold-standard counts to within ±1–2% (accounting for rounding and incremental updates).
  • Total count of unique 2D structures across all bins matches the expected curated structure count (e.g., 153956 unique 2D structures for LOTUS).
  • No structure appears in more than one bin; bin membership is mutually exclusive and exhaustive.
  • Summary report is human-readable, clearly labeled by bin, and includes numeric discrepancy values flagged when observed ≠ reference.
  • All input rows are accounted for in the output; no structures are dropped during grouping or binning.

Limitations

  • Binning thresholds (1, 2–10, 11–100, >100) are fixed and may not suit datasets with very different organism prevalence distributions.
  • The skill depends on accurate upstream organism counting; errors in organism deduplication upstream will propagate into bin counts.
  • Comparison to gold-standard reference counts assumes the reference is correct and applicable to the current dataset version; dataset growth or curation changes will render older benchmarks obsolete.
  • The skill does not identify which structures or organisms are anomalous—only whether the global distribution is as expected.

Evidence

  • [other] Group all unique 2D structures by their associated organism count. Bin structures into four frequency bins: singleton structures (1 organism), low-diversity structures (2–10 organisms), medium-diversity structures (11–100 organisms), and high-diversity structures (>100 organisms).: "Group all unique 2D structures by their associated organism count. Bin structures into four frequency bins: singleton structures (1 organism), low-diversity structures (2–10 organisms),"
  • [other] Count the number of structures in each bin and compare against the reported gold-standard counts.: "Count the number of structures in each bin and compare against the reported gold-standard counts."
  • [other] LOTUS is a comprehensive collection of documented structure-organism pairs designed to enable computational understanding of organisms and their chemistry.: "LOTUS is a comprehensive collection of documented structure-organism pairs designed to enable computational understanding of organisms and their chemistry."
  • [other] 231330 | 153956 (3D|2D) unique curated structures: "231330 | 153956 (3D|2D) unique curated structures"
  • [other] Load the LOTUS 2D structure-organism pairs table from the published flat file.: "Load the LOTUS 2D structure-organism pairs table from the published flat file."