Medallion Architecture Explained (Bronze, Silver, Gold)

Medallion architecture (Bronze / Silver / Gold): what it is

Medallion architecture is a naming + workflow convention for structuring a data platform into three tiers. It is not a separate installable technology (i.e., there’s no “Medallion” button in tools like Databricks/Snowflake/BigQuery).

Its goal is debuggability and repeatability: split a pipeline into stages where each stage produces durable, reviewable outputs.

---

Tier definitions and intended contents

Bronze (raw / recovery layer)

Purpose: Hold a reliable copy of source data in original form as it arrives.

• No business logic:
  • no filtering
  • no renaming
  • no type casting
  • no corrections
• If a source column contains issues (e.g., trailing spaces or damaged strings), Bronze keeps them as-is.
• Adds only upload/ingestion metadata, such as:
  • timestamp of receipt
  • source system name
  • file/partition name
  • ideally a package ID
• Append-only / recovery point:
  • Enables rebuilding downstream data when later stages have bugs.
  • Avoids dependence on overwritten production tables or limited vendor retention windows.

Practical tips:

• Partition Bronze by “date received” (not business/request dates) to support accurate reprocessing.
• Store semi-structured data as JSON/variant rather than splitting into columns immediately.
• Apply a storage policy/retention strategy for continuously growing Bronze.

Does Bronze mean every version forever?

• Not strictly “forever,” but typically keep every event/image received (e.g., CDC change events or nightly full-frame snapshots).

---

Silver (clean / entity layer)

Purpose: Convert raw into valid, consistent data.

• Cleaned, typed, deduplicated
• Conforms to a shared model

What happens here:

• Primary cleanup work, including:
  • convert strings → dates/decimals
  • remove spaces
  • standardize codes (e.g., US vs USA vs United States)
  • filter invalid records
  • remove duplicates to avoid double counting (common with at-least-once ingestion)
• Unify entities from multiple sources:
  • same business key
  • same format/value
  • regardless of which source pipeline provided it

What Silver should avoid:

• Don’t mix business definitions into Silver.
  • Casting types is cleanup (OK for Silver).
  • Defining “active customer” based on last 90 days is a business rule (belongs in Gold).

Also recommended for:

• Structures for slowly changing dimensions (SCD) history (e.g., “active from/to” tariff plan history).

---

Gold (business-ready / consumer layer)

Purpose: Provide ready-to-use tables aligned to business questions and metrics.

• Solidify metric definitions and aggregations, with examples like:
  • MRR
  • active user
  • attributed conversion
• Built “from the question to be answered” (not directly from source events).
• Can be shaped for consumers:
  • star schema for BI tools
  • denormalized tables / feature exports optimized for dashboard speed or downstream exports

Access guidance:

• Most of the organization should access Gold directly (analysts/executives).
• Silver is for data teams/analytics engineers.
• Bronze is for engineers and data re-creation.

Rebuild characteristics:

• Gold is typically cheaper/faster to rebuild because it depends on already-cleaned Silver.

---

Four rules to keep the architecture robust

1. Data only moves downstream

   • Bronze → Silver → Gold.
   • Gold should not read from Bronze.
   • If cross-tier access is needed, implement the missing logic in Silver.

2. Each level can be replayed from the previous one

   • If Bronze exists, you can regenerate everything downstream.

3. Transformations must be idempotent

   • Re-running the same task with the same input should not change results.
   • Use strategies like merge by key / overwrite partitions instead of blind appends.

4. Quality checks at level boundaries

   • Validate entering Silver (source correctness).
   • Validate entering Gold (business/logical correctness).
   • Pre-decide whether failures stop the pipeline or isolate bad rows—especially during incidents.

---

Common mistakes highlighted

1. Treating level names like folders instead of contracts

   • Example: Silver being just “select all from bronze” adds no value.

2. Putting business logic in Silver

   • Business definitions then propagate into every Gold table, making future changes costly.

3. Forcing exactly three levels even when unnecessary

   • Layer count isn’t sacred; you can use intermediate/differently structured zones if each has a clear purpose.

4. No storage/compaction policy

   • Bronze small files/old partitions can slow queries and increase costs; retention/compression/periodic review is needed.

5. No owner per level

   • Without responsibility, tables drift and duplicate/fragment over time across pipelines.

---

When to skip medallion architecture

Skip it if the use case doesn’t justify the added storage and complexity, for example:

• Small team with one dashboard and multiple sources → two-tier (“raw + storefront”) may suffice.
• Real-time sub-second latency workloads → 3-layer medallion may add unacceptable latency; streaming designs may differ.
• The source system is already clean/up-to-date → copying it as “raw” may be wasted effort.

The pattern is most valuable when you need:

• root-cause debugging for incorrect numbers
• the ability to reprocess without going back to the source

---

Main speaker/source(s)

• “Data Guy” (closing speaker reference)