Technical analysis and engineering decisions from building enterprise data reliability on Databricks.
Documented rules don't stop a confident subagent. Hooks do. The runtime-enforcement layer of the agentic governance stack — what a protected-branch guard actually looks like…
Read more →The standard advice for governing AI coding agents is to write a good CLAUDE.md. That answers orientation. It does not answer governance, error rates, or…
Read more →Not every drop in column entropy is a data quality failure. The harder problem is distinguishing unintended collapse from intended convergence — the signal a…
Read more →Schema and freshness checks pass while semantic-layer metrics silently degrade. Distributional validation — grounded in Shannon entropy — is the missing trust layer for dbt…
Read more →We ran a structured sequence of preregistered experiments across three real-world datasets totaling nearly 6.6 million rows to demonstrate that Shannon entropy catches data quality…
Read more →Schema checks validate shape. Entropy measures information content. Here’s why the gap between them is where your pipeline silently fails — and how to close…
Read more →
