Lakewright solutions / brief

Automate SFTP CSV drops into Apache Iceberg tables

The vendor CSV drop — daily files, shifting schemas, no contract — becomes a governed feed: detected, deduplicated, loaded to Iceberg, and watched for new arrivals under a human-approved plan.

What makes vendor drops hard?

Encodings (UTF-16, Windows-1252), delimiter surprises, renamed columns, re-sent files, and gzip/zip wrappers. Lakewright detects encoding and dialect by evidence, expands archives transparently, journals every file by content digest (re-sends are no-ops), and surfaces schema drift instead of loading it silently.

The steps

lakewright scan  sftp-drop/
lakewright approve plans/vendor_returns.plan.yaml
lakewright serve --plans plans --rest $CATALOG   # the ever-running watcher

FAQ

What happens when the vendor renames a column?The new shape doesn't match the approved plan — it's reported as drift and held for review, not fumbled into the warehouse.