A reference architecture for a governed data lakehouse spanning multiple cloud providers, using open table formats to avoid duplicating data across engines.
Storage and table format
Object storage (S3/ADLS) holds the physical data, with Apache Iceberg as the open table format layer providing ACID transactions, schema evolution, and time travel without locking the data into a single query engine.
Compute layer flexibility
Because Iceberg is engine-agnostic, the same physical data can be queried by Spark, Trino, Snowflake, or cloud-native engines (Athena, Synapse) depending on the workload — batch ETL, ad hoc analytics, or BI dashboards — without data movement.
Governance and access control
A centralized data catalog with column-level and row-level access policies is enforced at the catalog layer so governance travels with the data regardless of which compute engine is querying it.
Leave a Reply