What Is a Data Lake? Architecture, Tradeoffs, and Use Cases

A data lake is persisted storage for data that arrives in many shapes and at different stages of preparation. It commonly uses object storage for files and tables, then lets several compute engines read that storage. The useful part is not simply keeping raw files. It is making data durable, discoverable, governed, and usable without coupling every producer to one database schema.

This article separates three related architectures, describes the parts of a lake, and gives a way to decide whether building one is justified.

Lake, warehouse, and lakehouse are different designs

A data lake keeps source data, derived data, and often large files in low-cost object storage. Data can be written before every downstream question is known. A lake may contain CSV, JSON, Parquet, images, event logs, and table data. Schema-on-read is common: a reader interprets a file's fields at query time. That flexibility creates a governance obligation rather than removing one.

A data warehouse is a managed analytical store organized around curated tables. Ingestion normally validates and transforms data before it becomes a reporting model. Warehouses can also store semi-structured values, but their central promise is a reliable, queryable representation for a defined audience.

A lakehouse applies database-like table semantics to files in a lake. A table format can record snapshots, schemas, partitions, and file membership so readers see a consistent table rather than an arbitrary directory of files. The label does not make every lake a warehouse, nor does it make every warehouse a lakehouse.

Design	Primary contract	Best fit	Main tradeoff
Data lake	Durable, flexible persisted data	Retention, varied source data, multiple consumers	Quality and discovery need deliberate work
Data warehouse	Curated analytical tables	Repeatable reporting and shared metrics	Earlier modeling decisions and loading discipline
Lakehouse	File-based tables with table metadata	Shared object storage with table semantics	More components and operational choices

Many teams use more than one: retain source extracts in a lake, publish governed reporting tables in a warehouse, and run exploration or model training against selected lake tables.

The practical layers of a data lake

The following pieces matter more than a particular cloud provider or query engine.

Ingestion

Ingestion copies or receives data from databases, applications, SaaS exports, files, and event streams. Batch ingestion is easier to reconcile and replay. Streaming ingestion reduces the delay before data is available, but introduces ordering, late-event, and recovery decisions.

Keep the original delivery boundary clear. Record where a load came from, when it was received, what time range it represents, and whether it completed. A retry should not silently duplicate a batch. For change-data capture, retain source positions or transaction identifiers where the source provides them.

Object storage and file layout

Object storage is the usual lake foundation because it separates durable bytes from compute. It is suited to large immutable objects and can apply lifecycle rules, but it is not a relational database. Listing many small files, replacing objects, and coordinating concurrent writers all need care.

Use a layout that people and jobs can understand. Separate landing data from validated or published data. Partition only by fields that readers reliably filter on, such as a business date. Excessive partitioning creates small files and metadata work; no partitioning can make routine scans expensive.

Columnar files such as Parquet are often useful for analytical tables because engines can read selected columns and skip some files. They do not replace a schema, catalog, or access policy.

Catalog and metadata

A catalog maps a logical table or dataset name to its location, schema, owner, tags, and sometimes lineage. It answers basic questions: which dataset is approved for a metric, which files it includes, and who is responsible for it.

Treat metadata as part of the delivery. At minimum, capture an owner, purpose, source, refresh expectation, sensitivity classification, and a way to contact someone when a load fails. Without those facts, a lake accumulates files that cannot be safely reused.

Table formats

Raw files are appropriate for immutable source deliveries. Reusable analytical tables need more: an agreed schema, a snapshot of included files, and a safe way to evolve both. Open table formats such as Apache Iceberg, Delta Lake, and Apache Hudi address versions of this problem, but their supported features depend on the writer, catalog, and reader you choose.

Before adopting one, test the operations that matter: append, overwrite, merge or delete if needed, concurrent writes, schema evolution, snapshot reads, and cleanup of obsolete files. A format is a protocol across tools, not merely a file extension.

Compute

Compute engines query, transform, and sometimes train models from lake storage. A team may use SQL engines for reporting, distributed jobs for large transformations, and local engines for analysis. Storage/compute separation is useful when workloads vary, but it also means each engine needs compatible credentials, table support, and cost controls.

Do not assume every engine sees the same table behavior. Confirm which catalog it reads, which table-format features it supports, and how it handles time zones, decimals, and nested values.

Governance and security

Governance starts before storage. Classify sensitive fields, minimize what is copied, and define retention periods. Enforce access through identities and policies rather than path names alone. Encryption, audit logs, masking, and row or column restrictions may be available in a platform, but availability and enforcement differ by service and engine.

Published datasets should have a clear contract: meaning, grain, freshness expectation, quality checks, and owner. A quality check can be simple, such as rejecting a batch with an unexpected schema or recording a row-count change for review. The point is to make a failure visible before it becomes a dashboard argument.

Costs are a system property

Object storage is often inexpensive relative to interactive compute, but storage price alone is not a lake budget. Include ingestion and egress, catalog operations, file compaction, query scans, temporary compute, retries, observability, and the staff time to operate the system.

The cost unit users feel is usually a completed question, not a stored gigabyte. Put guardrails around broad scans, retain only necessary raw copies, compact small files where the table format and engine support it, and set lifecycle rules that match legal and analytical needs. Measure actual workload patterns before committing to a tiering or partition strategy.

Failure modes to design against

Failure mode	What it looks like	Practical response
Data swamp	Files have unclear meaning, owner, or status	Require catalog metadata and separate landing from published zones
Silent duplication	Retries create overlapping extracts	Use load identifiers, idempotent writes, and reconciliation checks
Small-file sprawl	Queries spend time opening metadata and files	Batch writes and schedule compaction where supported
Broken access boundaries	Broad paths expose sensitive data	Apply least privilege, classify fields, and audit access
Unbounded scans	A routine query reads far more data than intended	Partition for real filters, use columnar files, and monitor query cost
Conflicting definitions	Teams calculate the same metric differently	Publish owned datasets and document their grain and definitions

Federation is a different access pattern

Federated query tools can read several live sources through one query interface. For example, an analyst might join a PostgreSQL operational table with a MySQL system during an investigation. This can avoid a one-off export and is valuable for discovery or limited cross-system analysis.

Federation is not a replacement for persisted lake storage. A federated query depends on source availability, credentials, source-side load, and the connector's ability to push work down. It also does not by itself retain a historical snapshot, standardize data quality, or establish a reusable governed table.

Tangents is an example of a federated access pattern: it can provide a unified way to inspect and query connected sources. Use that pattern when the question needs live source data and the workload is bounded. Persist data in a lake when retention, repeatability, shared transformation, or independent analytical compute is the requirement.

When a lake is a good fit

Choose a lake when several of these statements are true:

• You need to retain source data that is too varied or too large for one curated model.
• Multiple teams need different downstream representations of the same inputs.
• Historical replay, auditability, or model-training datasets matter.
• You can assign owners for ingestion, cataloging, access, and quality.
• You have a concrete plan for the engines and tables people will actually use.

When not to build one

Do not begin with a lake because "we might need all our data later." A warehouse or operational database is often the better first choice when one team needs a small set of stable reports, data volume is modest, and source systems can publish clean tables directly.

Avoid a lake when no team can own its metadata and access controls. A folder of exports is not a data platform. Also avoid using a lake as an excuse to postpone data modeling indefinitely: downstream users still need documented, tested datasets.

A selection checklist

Before implementation, answer these questions in writing:

1. Which data must be persisted, and for how long?
2. Who owns each source and each published dataset?
3. Which workloads need raw files, curated tables, or live federation?
4. How will a reader discover a dataset's meaning, freshness, and sensitivity?
5. Which write, schema, and snapshot semantics must work across chosen tools?
6. What will stop an accidental broad scan or an uncontrolled copy of sensitive data?
7. What is the smallest initial dataset that proves the operating model?

The selection principle is simple: choose a lake for durable, governed shared storage with real downstream reuse; choose a warehouse for curated analytical consumption; choose federation for bounded access to live systems. Start with the narrowest architecture that satisfies the data contract you can actually operate.