Testing & Databricks Asset Bundles

Testing strategy

Testing layers

Test Type	What It Tests	How	When
Unit test	Individual functions/transforms	pytest with mock data	Every commit
Integration test	Components working together	Test tables in dev workspace	Every PR merge
End-to-end test	Full pipeline bronze → gold	Run pipeline on test data in staging	Before production deploy
UAT	Business rules and output quality	Stakeholders validate sample output	Before production release

Unit testing example

# test_transforms.py
from transforms import clean_amount, validate_date

def test_clean_amount_removes_negatives():
    assert clean_amount(-50) is None
    assert clean_amount(100) == 100.0

def test_validate_date_rejects_future():
    assert validate_date("2099-01-01") is False
    assert validate_date("2026-04-01") is True

Integration testing

# Run in a dev workspace with test data
test_df = spark.createDataFrame([
    (1, "Alice", 100.0, "2026-04-01"),
    (2, None, -50.0, "2099-01-01"),  # should be filtered out
], ["id", "name", "amount", "date"])

result = run_silver_pipeline(test_df)
assert result.count() == 1  # only valid row
assert result.filter("name = 'Alice'").count() == 1

Databricks Asset Bundles (DABs)

Asset Bundles package your entire project into a deployable unit:

# databricks.yml — bundle configuration
bundle:
  name: freshmart-etl

workspace:
  host: https://adb-1234567890.1.azuredatabricks.net

resources:
  jobs:
    nightly_etl:
      name: "Freshmart Nightly ETL"
      tasks:
        - task_key: ingest
          notebook_task:
            notebook_path: ./notebooks/01_ingest.py
          job_cluster_key: etl_cluster
        - task_key: transform
          depends_on:
            - task_key: ingest
          notebook_task:
            notebook_path: ./notebooks/02_transform.py
          job_cluster_key: etl_cluster

  pipelines:
    quality_pipeline:
      name: "Freshmart Quality Pipeline"
      target: freshmart_silver
      libraries:
        - notebook:
            path: ./pipelines/quality_checks.sql

environments:
  dev:
    workspace:
      host: https://adb-dev.azuredatabricks.net
  prod:
    workspace:
      host: https://adb-prod.azuredatabricks.net

Deploy via CLI

# Validate the bundle
databricks bundle validate

# Deploy to dev environment
databricks bundle deploy --target dev

# Run a specific job
databricks bundle run nightly_etl --target dev

# Deploy to production
databricks bundle deploy --target prod

Deploy via REST API

import requests

# Deploy using the Databricks REST API
response = requests.post(
    f"{workspace_url}/api/2.1/jobs/create",
    headers={"Authorization": f"Bearer {token}"},
    json=job_config
)

ℹ️ CI/CD with Asset Bundles

A typical CI/CD pipeline:

1. Developer pushes code to feature branch
2. CI pipeline (GitHub Actions/Azure DevOps) runs:
   • databricks bundle validate — check config syntax
   • pytest — run unit tests
   • databricks bundle deploy --target dev — deploy to dev
   • Integration tests in dev workspace
3. PR merged → deploy to staging, run E2E tests
4. Release → databricks bundle deploy --target prod

Question

What are the four testing levels for data engineering?

Click or press Enter to reveal answer

Answer

Unit tests (individual functions, every commit), integration tests (connected components, every PR), end-to-end tests (full pipeline, before deploy), UAT (business validation, before release).

Click to flip back

Question

What are Databricks Asset Bundles?

Click or press Enter to reveal answer

Answer

DABs package notebooks, jobs, pipelines, and configuration into a deployable unit defined in databricks.yml. Deploy via CLI (databricks bundle deploy) or REST API. Supports multiple environments (dev/staging/prod).

Click to flip back

Question

How do you deploy a bundle to different environments?

Click or press Enter to reveal answer

Answer

Define environments in databricks.yml with different workspace hosts. Deploy with: databricks bundle deploy --target dev (or staging/prod). Each target has its own workspace configuration.

Click to flip back

🎬 Video coming soon

Knowledge check

Knowledge Check

Dr. Sarah Okafor needs to deploy Athena Group's ETL pipeline to three environments (dev, staging, prod) with the same code but different workspace URLs. Which tool should she use?

AManually copy notebooks to each workspaceBDatabricks Asset Bundles with target environmentsCGit branches per environment (dev, staging, prod branches)DAzure Data Factory linked services per environment

Check Answer

---

Next up: Monitoring Clusters & Troubleshooting — cluster monitoring, job repair, and Spark troubleshooting.