exasol-distributed-ml

Distributed machine learning, data mining, and iterative HPC with Exasol. Covers end-to-end ML pipelines (DISTRIBUTE BY + SET scripts + BucketFS), per-entity federated training with partial_fit and ct

SKILL.md

GitHub

Exasol Distributed ML and HPC

Trigger when the user mentions: distributed ML, machine learning, train model, batch inference, prediction, feature engineering, hyperparameter, PyTorch, TensorFlow, scikit-learn, RAPIDS, GPU model, model deployment, distributed training, ensemble, anomaly detection, forecasting, clustering at scale, k-means, gradient descent, iterative algorithm, frequent itemset, association rules, market basket, Apriori, FP-Growth, data mining, SON algorithm, partial_fit, federated training, or any pattern where data is trained or scored inside Exasol.

Routing Algorithm

Choose the narrowest matching route. Load all routes that apply — they are designed to be read together.

Route 1 — Pipeline architecture, algorithms, and patterns

Trigger phrases: distributed training, end-to-end ML, feature engineering, batch inference, ensemble, k-means, gradient descent, frequent itemset, association rules, market basket, Apriori, FP-Growth, data mining, federated training, per-entity model, anomaly detection, forecasting, hyperparameter search, map-reduce

→ Load: references/distributed-ml-patterns.md

Route 2 — Model storage, versioning, and lifecycle

Trigger phrases: save model, ONNX, joblib, pickle, model versioning, load model in UDF, update model, latest.json, model registry, model path, BucketFS model

→ Load: references/model-lifecycle.md

Route 3 — GPU, CUDA, and RAPIDS

Trigger phrases: GPU UDF, CUDA SLC, PyTorch, TensorFlow, RAPIDS, cuDF, cuML, GPU acceleration, TorchScript, GPU cluster

→ Load: references/gpu-acceleration.md

Route 4 — Performance, tuning, and memory

Trigger phrases: slow UDF, OOM, out of memory, memory_limit, data skew in ML, profile SET script, chunking, group size, partial_fit convergence, ctx.reset, multi-pass, epoch loop

→ Load: references/ml-performance.md

Defer to Existing Skills

Do not re-explain these — activate the relevant skill instead:

  • UDF API basics (ctx.emit, ctx.get_dataframe, SCALAR vs SET syntax, CREATE SCRIPT templates) → exasol-udfs
  • SLC build/deploy (exaslct CLI, flavor selection, Dockerfile customization) → exasol-udfs references/slc-reference.md
  • Lua execute script orchestration (query/pquery API, error handling) → exasol-udfs references/lua-execute-scripts.md
  • BucketFS file operations (cp/ls/rm via exapump) → exasol-bucketfs
  • DISTRIBUTE BY table design and query profiling → exasol-database

Key Principles

  • Prefer partial_fit algorithms (SGD, MLP, MiniBatchKMeans) — stream chunks with ctx.get_dataframe(num_rows=CHUNK_SIZE), loop epochs with ctx.reset() for convergence
  • Avoid ctx.get_dataframe(num_rows='all') — it materializes the entire group in memory
  • For algorithms without partial_fit (RandomForest, GradientBoosting): use the map-reduce ensemble pattern (Section 6 of distributed-ml-patterns.md) rather than loading all data at once
  • Load models at module level (outside run()), never inside run()
  • Orchestrate iterative algorithms with Lua execute scripts (query) rather than external drivers when possible
  • DISTRIBUTE BY on the training/inference key keeps related rows on the same node and eliminates shuffles