Lineage, Audit Logs & Delta Sharing

Why lineage and audit matter

Data lineage

What lineage tracks

Unity Catalog automatically captures lineage when notebooks, jobs, or pipelines read from and write to tables:

Lineage Element	What It Shows
Table-level lineage	Which tables feed into which tables
Column-level lineage	Which source columns map to which destination columns
Notebook/job lineage	Which notebook or job created/modified a table
Owner	Who owns the table (user or group)
History	When the table was created, last modified, version history
Dependencies	Upstream tables this table depends on

Viewing lineage in Catalog Explorer

In Catalog Explorer, select any table and click the Lineage tab to see:

• Upstream — tables and columns that feed into this table
• Downstream — tables and dashboards that consume this table
• Notebooks/jobs — the code that created the relationship

When Mei Lin investigates a data quality issue in Freshmart’s daily revenue report, she traces lineage back through:

gold.daily_revenue ← silver.cleaned_transactions ← bronze.raw_pos_data

The issue is in the bronze layer — a partner changed their CSV format.

Querying lineage via system tables

-- View table lineage
SELECT * FROM system.access.table_lineage
WHERE target_table_full_name = 'prod_sales.curated.daily_revenue';

-- View column lineage
SELECT * FROM system.access.column_lineage
WHERE target_table_full_name = 'prod_sales.curated.daily_revenue';

💡 Exam tip: Lineage is automatic but not universal

Unity Catalog captures lineage automatically for:

• ✅ Spark SQL queries (SELECT INTO, INSERT INTO, MERGE)
• ✅ DataFrame operations (read → transform → write)
• ✅ Lakeflow Spark Declarative Pipeline dependencies
• ✅ Notebook-driven ETL

Lineage is NOT captured for:

• ❌ External tools that bypass Spark (direct file access)
• ❌ Legacy Hive metastore tables (not registered in Unity Catalog)

If the exam asks “how to ensure lineage is tracked,” the answer is: use Unity Catalog tables and run transformations through Databricks compute.

Audit logging

What gets logged

Unity Catalog logs every significant action to system tables:

Event Type	Examples
Data access	SELECT queries, table reads
Data modification	INSERT, UPDATE, DELETE, MERGE
Permission changes	GRANT, REVOKE
Schema changes	CREATE TABLE, ALTER TABLE, DROP TABLE
Authentication	Login events, service principal access
Admin actions	Cluster creation, job scheduling

Querying audit logs

-- Who accessed the customers table in the last 7 days?
SELECT
  event_time,
  user_identity.email AS user_email,
  action_name,
  request_params.full_name_arg AS table_name
FROM system.access.audit
WHERE request_params.full_name_arg = 'prod_sales.curated.customers'
  AND event_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
ORDER BY event_time DESC;

Dr. Sarah Okafor runs audit queries weekly at Athena Group to verify that only authorised users accessed sensitive financial tables.

ℹ️ Audit log retention and export

System audit tables have a default retention period. For long-term compliance:

• Export audit logs to a Delta table in your own storage for permanent retention
• Stream to Azure Monitor for real-time alerting on suspicious activity
• Integrate with SIEM tools for security operations

Exam tip: If the question mentions “long-term audit retention” or “compliance archive,” the answer involves exporting audit logs to your own managed storage — not relying on the system table defaults.

Delta Sharing

What is Delta Sharing?

Delta Sharing is an open protocol for secure, live data sharing:

• Provider — the organisation sharing data (creates shares, adds tables, generates recipient tokens)
• Recipient — the organisation receiving data (connects using a credential file or Databricks-to-Databricks sharing)
• Share — a named collection of tables made available to recipients

Two sharing modes

Feature	Open Sharing	Databricks-to-Databricks
Recipient needs Databricks?	No — any Spark/pandas/Power BI	Yes — Databricks workspace
Authentication	Bearer token (credential file)	Unity Catalog identity
Governance	Read-only access to shared tables	Full UC governance (lineage, audit)
Best for	External partners, customers	Internal cross-workspace, Databricks partners
Live data?	Yes — reads current version	Yes — reads current version

Setting up Delta Sharing

-- Step 1: Create a share
CREATE SHARE IF NOT EXISTS partner_freshmart
  COMMENT 'Product catalog shared with Freshmart suppliers';

-- Step 2: Add tables to the share
ALTER SHARE partner_freshmart
  ADD TABLE prod_sales.curated.product_catalog;

-- Step 3: Create a recipient
CREATE RECIPIENT IF NOT EXISTS freshmart_supplier_a
  COMMENT 'Supplier A needs product catalog for inventory planning';

-- Step 4: Grant the recipient access to the share
GRANT SELECT ON SHARE partner_freshmart TO RECIPIENT freshmart_supplier_a;

Secure sharing strategy

When designing a Delta Sharing strategy, consider:

Decision	Recommendation
What to share	Only curated/gold tables — never raw data
Granularity	One share per partner or use case
Column filtering	Share views that exclude sensitive columns
Audit	Monitor share access via audit logs
Revocation	Remove recipient access immediately when partnership ends

Tomás shares anonymised fraud pattern data with NovaPay’s banking partners via Delta Sharing. Partners get a read-only view of fraud trends without accessing NovaPay’s customer data.

Question

What is data lineage in Unity Catalog and how is it captured?

Click or press Enter to reveal answer

Answer

Lineage tracks table-to-table and column-to-column dependencies automatically when data flows through Databricks compute. View in Catalog Explorer or query system.access.table_lineage. Not captured for external tool access bypassing Spark.

Click to flip back

Question

What is Delta Sharing and what are its two modes?

Click or press Enter to reveal answer

Answer

Delta Sharing is an open protocol for secure data sharing. Open Sharing: recipients don't need Databricks (use token + any Spark/pandas tool). Databricks-to-Databricks: full UC governance, identity-based. Both provide live, read-only access.

Click to flip back

Question

How do you query Unity Catalog audit logs?

Click or press Enter to reveal answer

Answer

Query system.access.audit with SQL. Filter by user_identity.email, action_name, request_params.full_name_arg, and event_time. Export to your own storage for long-term compliance retention.

Click to flip back

🎬 Video coming soon

Knowledge check

Knowledge Check

Mei Lin discovers incorrect revenue figures in Freshmart's executive dashboard. She needs to trace the data back to its source to find where the error was introduced. Which Unity Catalog feature should she use?

AAudit logs to see who modified the tableBData lineage in Catalog Explorer to trace upstream dependenciesCDelta Sharing to compare data with the source systemDRow filters to check if data is being filtered incorrectly

Check Answer

Knowledge Check

Dr. Sarah Okafor needs to share product inventory data with Athena Group's logistics partner. The partner uses Snowflake, not Databricks. The data must be live (not a copy) and read-only. Which approach should she use?

AExport CSV files to a shared Azure Blob container dailyBSet up Delta Sharing with Open Sharing mode and a bearer tokenCCreate a foreign catalog connection from the partner's Snowflake to Athena's DatabricksDUse Databricks-to-Databricks sharing with Unity Catalog identity

Check Answer

Knowledge Check

Tomás needs to prove to NovaPay's compliance auditor that no unauthorised users accessed the fraud_alerts table in the last 30 days. Which Unity Catalog feature provides this evidence?

AData lineage trackingBSystem audit logs (system.access.audit)CDelta Sharing recipient logsDColumn mask access history

Check Answer

---

Next up: Data Modeling: Ingestion Design — choosing ingestion tools, loading methods, table formats, and managed vs external tables.