Databricks Billing Setup

​

What each level gives you

​

How the data flows

​

Key terms used in this article

• Level 1 (Billing): DBU usage from system.billing.usage, priced from your system.billing.list_prices export (or split from your cloud bill if you pay through a marketplace).
• Level 2 (Enrichment): Cloud VM lines gain Databricks cluster, job, and job run dimensions by matching instance ids, so DBU cost and VM cost describe the same workload.
• System tables: Databricks-managed tables under system.billing, system.access, and system.compute. You enable the schemas, then the export job reads them.
• Unity Catalog external location: The landing path your export job writes to, backed by storage credentials in your cloud account.

​

Prerequisites

• A Costory account.
• A Unity Catalog admin (plan about 30 to 60 minutes for first-time setup).
• A storage location you control, in the same cloud as the workspace.
• For Level 2: AWS, GCP, or Azure billing already in Costory, and on GCP the detailed (resource-level) billing export to BigQuery.

​

Level 1: Databricks DBU billing

• Databricks as a provider in Costory, with spend by product (Jobs, SQL, All-Purpose, DLT, Model Serving, Apps, and similar lines), SKU, and workspace.
• Breakdowns by cluster, job, job run, SQL warehouse, DLT pipeline, serving endpoint, notebook, app, and instance pool, from Databricks usage metadata.
• Custom tags from Databricks become dimensions you can use in allocation, dashboards, alerts, and reports.
• Serverless workloads are fully covered at Level 1. They run in Databricks’ account, and the DBU price already includes the compute. There are no VMs in your cloud bill for them.
• Corrections: Databricks retractions and restatements in system.billing.usage are ingested as-is. Costory nets them so you always see net usage, not only original rows.
• Freshness: Usage usually appears in Costory within about 12 hours after your daily export. See Data refresh.
• History: Run the notebook once with lookback_days=365 to load up to 12 months.

​

How DBUs are priced

​

Level 2: VM enrichment

• Group total cost (DBUs + VMs) by job name, cluster, workspace, or your Databricks custom tags
• Open a job run and see both DBU lines and the EC2 / Azure VM / Compute Engine lines for the nodes that executed it
• Compare DBU-only spend to infrastructure spend for the same cluster (useful when VM waste or instance type choice dominates)
• Split instance pool idle from time pools spent on real workloads

• Matching uses instance ids from Databricks compute system tables, not cloud cost allocation tags. It works on history, including dates before you activated tags on AWS.
• When an instance moves between clusters, its cost is split by the minutes or hours it spent on each: hour by hour on AWS and GCP, day by day on Azure (Azure cost exports are daily).
• Instances in pools are attributed to the cluster or job that pulled them. Time an instance sits ready but unassigned is pool idle on that pool, so idle pool capacity is visible instead of disappearing into anonymous VM spend.
• Setup: one export job per Databricks region with include_compute=true, plus cloud billing already in Costory. On GCP, enable the detailed billing export so Costory can read VM resource ids.
• History follows what Databricks retains in system.compute tables.

​

What is not covered

​

Export notebook

​

Job parameters

​

Output layout

<target_path>/
  billing/billing_usage/ingestion_date=YYYY-MM-DD/*.parquet
  billing/list_prices/*.parquet
  billing/workspaces/*.parquet
  compute/region=<region>/node_timeline_hourly/usage_date=YYYY-MM-DD/*.parquet
  compute/region=<region>/instance_events/event_date=YYYY-MM-DD/*.parquet
  compute/region=<region>/clusters/*.parquet

dbutils.widgets.text("target_path", "")
dbutils.widgets.text("lookback_days", "3")
dbutils.widgets.dropdown("include_billing", "true", ["true", "false"])
dbutils.widgets.dropdown("include_compute", "false", ["true", "false"])
dbutils.widgets.text("region", "")

TARGET = dbutils.widgets.get("target_path").rstrip("/")
DAYS = int(dbutils.widgets.get("lookback_days"))
INCLUDE_BILLING = dbutils.widgets.get("include_billing") == "true"
INCLUDE_COMPUTE = dbutils.widgets.get("include_compute") == "true"
REGION = dbutils.widgets.get("region").strip()

assert TARGET, "target_path is required"
assert not INCLUDE_COMPUTE or REGION, "region is required when include_compute=true"

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
SINCE = f"current_date() - INTERVAL {DAYS} DAYS"

BILLING = {
    "billing_usage": (f"SELECT * FROM system.billing.usage WHERE ingestion_date >= {SINCE}", "ingestion_date"),
    "list_prices": ("SELECT * FROM system.billing.list_prices", None),
    "workspaces": ("SELECT * FROM system.access.workspaces_latest", None),
}

COMPUTE = {
    "node_timeline_hourly": (f"""
        SELECT workspace_id, cluster_id, instance_id, driver, node_type,
               date_trunc('HOUR', start_time) AS usage_hour,
               to_date(start_time)            AS usage_date,
               COUNT(*)                       AS minutes
        FROM system.compute.node_timeline
        WHERE start_time >= {SINCE}
        GROUP BY ALL""", "usage_date"),
    "instance_events": (f"""
        SELECT *, to_date(event_time) AS event_date
        FROM system.compute.instance_events
        WHERE event_time >= {SINCE}""", "event_date"),
    "clusters": ("SELECT * FROM system.compute.clusters", None),
}

def export(tables, base):
    for name, (query, partition_col) in tables.items():
        writer = spark.sql(query).write.mode("overwrite")
        if partition_col:
            writer = writer.partitionBy(partition_col)
        writer.parquet(f"{base}/{name}/")

if INCLUDE_BILLING:
    export(BILLING, f"{TARGET}/billing")
if INCLUDE_COMPUTE:
    export(COMPUTE, f"{TARGET}/compute/region={REGION}")

​

FAQ

​

Related