Data Validation and Quality

This skill provides comprehensive approaches to validating data quality, implementing testing frameworks, and ensuring data reliability throughout your pipelines.

When to use this skill

Use this skill when you need to:

Implement data quality checks in pipelines
Create data validation frameworks
Set up automated data testing
Monitor data quality metrics
Ensure compliance with data standards

Prerequisites

Understanding of data schemas and business rules
Access to data quality tools (Great Expectations, Soda, custom)
Knowledge of statistical methods for data analysis
Integration points in data pipelines

Guidelines

Validation Categories

Schema Validation

Column existence and order
Data type correctness
Nullability constraints
Format validation (regex patterns)

import pandera as pa
from pandera import Column, Check

schema = pa.DataFrameSchema({
    "id": Column(int, nullable=False),
    "email": Column(str, Check.str_matches(r"^[^@]+@[^@]+$"), nullable=False),
    "age": Column(int, Check.greater_than(0), Check.less_than(150), nullable=True),
    "created_at": Column(pa.DateTime, nullable=False)
})

validated_df = schema.validate(df)

Content Validation

Range checks (min/max values)
Uniqueness constraints
Referential integrity
Business rule validation

Statistical Validation

Distribution analysis
Outlier detection
Trend analysis
Anomaly detection

Validation Frameworks

Great Expectations

import great_expectations as gx

context = gx.get_context()
suite = context.add_expectation_suite("my_suite")

# Add expectations
validator = context.get_validator(
    batch_request=batch_request,
    expectation_suite=suite
)

validator.expect_column_values_to_not_be_null("id")
validator.expect_column_values_to_be_between("age", 0, 150)
validator.expect_column_values_to_match_regex("email", r"^[^@]+@[^@]+$")
validator.expect_column_mean_to_be_between("salary", 50000, 150000)

validator.save_expectation_suite()

Soda Core

# checks.yml
checks for dataset:
  - row_count > 0
  - duplicate_count(id) = 0
  - missing_count(email) < 5
  - invalid_percent(email) < 1%:
      valid format: email
  - avg(age) between 18 and 100

Validation Strategies

Pre-Load Validation

Validate raw data before transformation
Reject or quarantine bad records
Log validation failures

Post-Transform Validation

Validate after business rules applied
Ensure data warehouse constraints
Check aggregate correctness

Continuous Monitoring

Schedule validation jobs
Track quality metrics over time
Alert on degradation

Data Quality Dimensions

Completeness

Percentage of populated fields
Required field coverage
Historical completeness trends

Accuracy

Match with source of truth
Business rule compliance
Cross-system consistency

Consistency

Format standardization
Unit consistency
Naming conventions

Timeliness

Data freshness
Update frequency
Latency monitoring

Implementation Best Practices

Validation Layers

Source → Schema Validation → Business Rules → Statistical Checks → Target
              ↓                      ↓                  ↓
         Reject Bad Data       Flag Warnings      Alert Anomalies

Error Handling

Categorize failures (critical, warning, info)
Implement quarantine tables for bad records
Provide detailed error messages
Enable quick failure recovery

Testing

Unit tests for validation rules
Integration tests for full pipelines
Data diff tests between environments
Regression testing for schema changes

Monitoring and Alerting

Metrics to Track

Validation pass/fail rates
Data quality scores
Processing times
Error frequencies by category

Alert Conditions

Critical validation failures
Quality score drops below threshold
Unusual error patterns
SLA breaches

Examples

See the examples/ directory for:

great-expectations-suite.py - Complete GE suite setup
pandera-schema.py - Runtime schema validation
custom-validator.py - Building custom validation framework
quality-dashboard.py - Quality metrics visualization

data-validation

SKILL.mdskills/data/data-validation/SKILL.md