Projects

Sector: Finance

Production

On-prem financial lakehouse

A self-hosted DuckLake lakehouse for a regulated financial cooperative.

Architecture diagram

A regulated financial cooperative had all its analytics trapped in the Oracle core. I designed an on-prem lakehouse on DuckLake that frees it, automates debt certificates, and enables the regulatory dashboard, on a 100% open-source self-hosted stack.

Data protection was not a later add-on: it was a design focus from the start, with the sector’s legal and contractual rigor (data-subject rights, minimization, confidentiality). Its technical expression —anonymizing a ~740-table core while preserving referential integrity— was the hardest part of the project.

Key metrics

Rows in Silver (20.3M in the core fact)
~40M Rows in Silver (20.3M in the core fact)
Tables anonymized with referential integrity
~740 Tables anonymized with referential integrity
Self-hosted OSS stack
100% Self-hosted OSS stack
Critical production bug found and fixed
1 Critical production bug found and fixed

Stack

  • DuckLake
  • DuckDB
  • PostgreSQL
  • SeaweedFS
  • Parquet
  • Windmill
  • n8n

Decisions and trade-offs

  1. Data protection as a design focus, not a patch

    Legal-contractual rigor (data-subject rights, minimization, confidentiality) drove the architecture from the start. It showed up as anonymizing a ~740-table core while preserving referential integrity: the hardest part, and a real differentiator for a regulated sector.

  2. DuckLake (catalog + Parquet + DuckDB)

    Betting on DuckLake —2025 leading edge— produced an open lakehouse over a PostgreSQL catalog, Parquet on SeaweedFS, and the DuckDB engine, validated end to end against the original Oracle. No proprietary DW or cloud lock-in.

  3. The JOIN that issued invalid certificates

    During validation, invalid debt certificates surfaced from a broken JOIN in the legacy process. Found and fixed: a production bug with legal impact, caught by comparing the lakehouse against the source.

  4. End-to-end financial rigor

    Decimal arithmetic (never float for money), tiered provisions, and regulatory controls modeled into the layers, not left to the consumption side.