> ## Documentation Index > Fetch the complete documentation index at: https://docs.costory.io/llms.txt > Use this file to discover all available pages before exploring further. # Databricks Billing Setup > Connect Databricks DBU billing and full workload cost (DBUs plus cloud VMs) in Costory. Level 2 ties each instance to the cluster and job that ran on it. Your DBU (Databricks Unit, the usage quantity on the Databricks bill. Costory prices DBUs from system.billing.list\_prices unless marketplace mode uses the cloud bill.) spend sits on a Databricks invoice or a marketplace (Third-party software bought through AWS, GCP, or Azure Marketplace.) line on AWS, GCP, or Azure. The VMs and disks behind your clusters sit on the same cloud bill, often with no link back to the job or notebook that used them. The Databricks connector (Storage-based integration. Level 1 ingests DBU usage; Level 2 attributes cloud VMs to cluster and job by instance id.) puts Databricks costs next to the rest of your cloud spend in [Explorer](https://docs.costory.io/features/cost-explorer.md). You connect at two levels. **Level 1** brings DBU billing into Costory with workspace, job, cluster, and tag breakdowns. **Level 2** joins those DBUs to the cloud VM lines for the same workloads, so you see the **full cost of Databricks**: license (DBUs) plus infrastructure (EC2, Azure VMs, Compute Engine) on one cluster, one job run, or one notebook. Start with Level 1 and add Level 2 when you are ready. Both work on AWS, GCP, and Azure. **Costory never gets access to your Databricks account.** You run a small daily export job on your side. It writes Parquet files to storage you own. Costory reads that location with a read-only grant. We do not hold Databricks API keys or workspace credentials. ## What each level gives you "Right away" means your cloud billing export (AWS CUR, Azure Cost Management, or GCP BigQuery billing) is already connected to Costory, with no Databricks setup. | What you want to see | Right away | Level 1 | Level 2 | | - | - | - | - | | Databricks DBU cost by workspace, product, job, cluster, warehouse, and your tags | No | Yes | Yes | | VM cost per cluster or job, for clusters without pools | Partly, through tags Databricks puts on VMs (AWS only after you activate those tags, from that date forward) | Same | Yes, including history, with no tag activation | | VM cost per cluster or job, for clusters using pools | No, only the pool is visible | No | Yes | | Idle cost of instance pools | No | No | Yes | | Total cost of a job or cluster (DBUs + VMs) | No | No | Yes | | Which cluster or job used a given cloud instance (by instance id, over time) | No | No | Yes | Level 1 answers “what did we spend in DBUs, and on which job or warehouse?” Level 2 answers “what did that workload **actually** cost once you count the nodes it ran on?” Costory reads `system.compute.node_timeline` and related tables, matches each **instance id** to the cluster and job that held it (hour by hour on AWS and GCP, day by day on Azure), and rolls VM spend together with DBUs from Level 1. You can trace a spike on an EC2 line back to a **job run**, see **pool idle** separately from active work, and build dashboards on **total Databricks cost** per team or tag, not DBUs alone. That enrichment is not the same as [Enrichments](https://docs.costory.io/features/tagging/enrichments.md), which are lookup files you publish from git. The export notebook supports `include_compute=true` for the compute tables Level 2 uses. ## How the data flows **1. Pick storage you own** Choose an S3 bucket, Azure Data Lake Storage Gen2 container, or GCS bucket in the **same cloud** as your Databricks workspace. **2. Run a daily export job** Import the [export notebook](#export-notebook) into Databricks and schedule it on a small cluster. It exports Databricks system tables as Parquet. The job usually runs a few minutes per day. **3. Grant Costory read-only access** Give Costory `GetObject` / `ListBucket` (AWS), `Storage Blob Data Reader` (Azure), or `roles/storage.objectViewer` (GCP) on the prefix you use. That is the only credential Costory holds for Databricks. **4. Costory ingests and keeps history** Costory reads the export daily and stores history in its own data store. You can delete files in your bucket after a short retention window. Once ingested, history stays in Costory. ## Key terms used in this article * **Level 1 (Billing):** DBU usage from `system.billing.usage`, priced from your `system.billing.list_prices` export (or split from your cloud bill if you pay through a marketplace). * **Level 2 (Enrichment):** Cloud VM lines gain Databricks cluster, job, and job run dimensions by matching instance ids, so DBU cost and VM cost describe the same workload. * **System tables:** Databricks-managed tables under `system.billing`, `system.access`, and `system.compute`. You enable the schemas, then the export job reads them. * **Unity Catalog external location:** The landing path your export job writes to, backed by storage credentials in your cloud account. See the [Glossary](https://docs.costory.io/docs/glossary.md) for a full list of terms. ## Prerequisites * A Costory account. * A Unity Catalog admin (plan about **30 to 60 minutes** for first-time setup). * A storage location you control, in the same cloud as the workspace. * For Level 2: [AWS](https://docs.costory.io/setup/billing/aws.md), [GCP](https://docs.costory.io/setup/billing/gcp.md), or [Azure](https://docs.costory.io/setup/billing/azure.md) billing already in Costory, and on GCP the [detailed (resource-level) billing export](https://cloud.google.com/billing/docs/how-to/export-data-bigquery-setup#detailed_usage_cost_data) to BigQuery. **Manual setup** **1. Enable system schemas** In Databricks, enable the Unity Catalog system schemas `billing` and `access`. For Level 2, also enable `compute`. See [Databricks system tables](https://docs.databricks.com/en/admin/system-tables/index.html). **2. Create the storage location** Create or reuse a bucket or container in the same cloud as your workspace. Register it as a Unity Catalog **external location** with write access for the identity that runs the export job. Example landing prefixes (you can use a different root; tell Costory the **billing root** path you chose): | Cloud | Example prefix | | - | - | | AWS | `s3:///costory/databricks/` | | Azure | `abfss://@.dfs.core.windows.net/costory/databricks/` | | GCP | `gs:///costory/databricks/` | **3. Import and schedule the export notebook** Copy the [export notebook](#export-notebook) into a Databricks notebook and create a daily job. * First run: `lookback_days=365` to load up to **12 months** of history. * Daily run: `lookback_days=3` (rewrites the last few ingestion dates so Databricks corrections land correctly). * Set `include_billing=true` in **one workspace only** (billing data is account-wide). * For Level 2: set `include_compute=true` in one workspace **per region**, with `include_billing=false` on those jobs, and set `region` to a label such as `eu-west-1`. **4. Grant Costory read-only access** | Cloud | Read access for Costory | | - | - | | AWS | Trust the Costory role with `s3:GetObject` and `s3:ListBucket` on your prefix | | Azure | `Storage Blob Data Reader` for the Costory app on the container (Costory support confirms the exact principal when you connect) | | GCP | `roles/storage.objectViewer` on the bucket for the Costory service account | Data stays in your account. Costory only reads it. **5. Send Costory your connection details** Contact [Costory support](mailto:contact@\[REDACTED]) with: * The **billing root** path (for example `gs:///costory/billing/` or `gs:///costory/databricks/billing/`). * How you pay for Databricks: **direct contract** (Databricks invoice) or **marketplace** (AWS, Azure, or GCP). * For a direct contract, your optional **discount %** if you want paid-cost metrics to apply it (list price stays visible for comparison). [Costory support](mailto:contact@\[REDACTED]) completes the datasource connection after you grant read access. **Terraform** Not supported. ## Level 1: Databricks DBU billing * **Databricks as a provider** in Costory, with spend by product (Jobs, SQL, All-Purpose, DLT, Model Serving, Apps, and similar lines), SKU, and workspace. * **Breakdowns** by cluster, job, job run, SQL warehouse, DLT pipeline, serving endpoint, notebook, app, and instance pool, from Databricks usage metadata. * **Custom tags** from Databricks become dimensions you can use in [allocation](https://docs.costory.io/features/tagging/shared-cost-allocation.md), [dashboards](https://docs.costory.io/features/dashboards), [alerts](https://docs.costory.io/features/alerts.md), and [reports](https://docs.costory.io/features/slack-reports). * **Serverless** workloads are fully covered at Level 1. They run in Databricks' account, and the DBU price already includes the compute. There are no VMs in your cloud bill for them. * **Corrections:** Databricks retractions and restatements in `system.billing.usage` are ingested as-is. Costory nets them so you always see net usage, not only original rows. * **Freshness:** Usage usually appears in Costory within about **12 hours** after your daily export. See [Data refresh](https://docs.costory.io/docs/data-refresh.md). * **History:** Run the notebook once with `lookback_days=365` to load up to 12 months. Costory shows a **workspace name** when your export includes `system.access.workspaces_latest`. If the `access` schema is not enabled yet, you still get `workspace_id`. ### How DBUs are priced Costory uses the most accurate price available and does not count DBUs twice. | Where you pay for Databricks | Cost shown in Costory | | - | - | | Direct contract (Databricks invoice) | Databricks list price from your `system.billing.list_prices` export, minus an optional **discount %** you set on the connector. [List Cost](https://docs.costory.io/docs/cost-metrics) stays visible so you can compare. | | AWS Marketplace, Azure Databricks, or GCP Marketplace | Your real price from the cloud bill. Databricks data splits that cost across workspaces, jobs, clusters, and tags. Tell Costory you pay through a marketplace so we do not invent a second DBU cost on top of the cloud bill. | For direct contracts, every paid-cost metric applies your discount %. List price is always shown as well. See [Cost metrics](https://docs.costory.io/docs/cost-metrics). ## Level 2: VM enrichment Level 2 shows the full price of each workload: you see DBUs and cloud VMs on the same cluster, job, and job run instead of reconciling two bills by hand. Costory ties every billable **instance id** in your cloud account to the Databricks **cluster**, **job**, and **job run** that used it, then adds those VM dollars to the DBUs you already see from Level 1. In [Explorer](https://docs.costory.io/features/cost-explorer.md) you can: * Group **total cost** (DBUs + VMs) by job name, cluster, workspace, or your Databricks custom tags * Open a **job run** and see both DBU lines and the EC2 / Azure VM / Compute Engine lines for the nodes that executed it * Compare **DBU-only** spend to **infrastructure** spend for the same cluster (useful when VM waste or instance type choice dominates) * Split **instance pool idle** from time pools spent on real workloads How it works: * Matching uses **instance ids** from Databricks compute system tables, not cloud cost allocation tags. It works on history, including dates before you activated tags on AWS. * When an instance moves between clusters, its cost is split by the minutes or hours it spent on each: hour by hour on AWS and GCP, day by day on Azure (Azure cost exports are daily). * Instances in **pools** are attributed to the cluster or job that pulled them. Time an instance sits ready but unassigned is **pool idle** on that pool, so idle pool capacity is visible instead of disappearing into anonymous VM spend. * Setup: one export job per Databricks **region** with `include_compute=true`, plus [cloud billing](https://docs.costory.io/setup/billing) already in Costory. On GCP, enable the detailed billing export so Costory can read VM resource ids. * History follows what Databricks retains in `system.compute` tables. ## What is not covered | Item | Covered? | Notes | | - | - | - | | Serverless | Yes (Level 1) | No VMs in your cloud bill; DBU price includes compute | | Classic / Pro SQL warehouse VMs | No | Databricks does not expose warehouse instances; DBUs are still broken down per warehouse at Level 1 | | Disks attached to clusters | Partly (Level 2) | Linked to the VM where possible, otherwise shared proportionally | | Networking / storage (S3, ADLS, GCS) | No | Use your usual cloud tags | | GCP workspaces on legacy GKE compute | Possibly not | GKE compute was deprecated by Databricks in March 2025 | | Regions without a compute export job | No (Level 2) | One export job per region | ## Export notebook Each run rewrites only the last few days of partitioned tables and fully refreshes small reference tables, so storage stays small and nothing is duplicated. > [!NOTE] > > `system.billing.usage` can include an `identity_metadata` column (for example user emails). Costory does not need it for DBU billing or allocation by cluster, job, or custom tags. You can drop that column in the export query if your security policy requires it. ### Job parameters | Parameter | Value | | - | - | | `target_path` | Your storage location: `s3://…`, `abfss://…`, or `gs://…` | | `lookback_days` | `365` for the first run, then `3` | | `include_billing` | `true` in **one** workspace only (billing data is account-wide) | | `include_compute` | `true` in one workspace **per region** (Level 2 only) | | `region` | Region label, e.g. `eu-west-1`. Required when `include_compute=true` | ### Output layout ``` / billing/billing_usage/ingestion_date=YYYY-MM-DD/*.parquet billing/list_prices/*.parquet billing/workspaces/*.parquet compute/region=/node_timeline_hourly/usage_date=YYYY-MM-DD/*.parquet compute/region=/instance_events/event_date=YYYY-MM-DD/*.parquet compute/region=/clusters/*.parquet ``` Tell Costory the **billing root** you used (for example `/billing/`). You can use any prefix layout (such as `costory/billing/` or `costory/databricks/billing/`). Costory is configured with the path you provide. > **Export notebook (Python)** > > ```python theme={null} > dbutils.widgets.text("target_path", "") > dbutils.widgets.text("lookback_days", "3") > dbutils.widgets.dropdown("include_billing", "true", ["true", "false"]) > dbutils.widgets.dropdown("include_compute", "false", ["true", "false"]) > dbutils.widgets.text("region", "") > > TARGET = dbutils.widgets.get("target_path").rstrip("/") > DAYS = int(dbutils.widgets.get("lookback_days")) > INCLUDE_BILLING = dbutils.widgets.get("include_billing") == "true" > INCLUDE_COMPUTE = dbutils.widgets.get("include_compute") == "true" > REGION = dbutils.widgets.get("region").strip() > > assert TARGET, "target_path is required" > assert not INCLUDE_COMPUTE or REGION, "region is required when include_compute=true" > > spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic") > SINCE = f"current_date() - INTERVAL {DAYS} DAYS" > > BILLING = { > "billing_usage": (f"SELECT * FROM system.billing.usage WHERE ingestion_date >= {SINCE}", "ingestion_date"), > "list_prices": ("SELECT * FROM system.billing.list_prices", None), > "workspaces": ("SELECT * FROM system.access.workspaces_latest", None), > } > > COMPUTE = { > "node_timeline_hourly": (f""" > SELECT workspace_id, cluster_id, instance_id, driver, node_type, > date_trunc('HOUR', start_time) AS usage_hour, > to_date(start_time) AS usage_date, > COUNT(*) AS minutes > FROM system.compute.node_timeline > WHERE start_time >= {SINCE} > GROUP BY ALL""", "usage_date"), > "instance_events": (f""" > SELECT *, to_date(event_time) AS event_date > FROM system.compute.instance_events > WHERE event_time >= {SINCE}""", "event_date"), > "clusters": ("SELECT * FROM system.compute.clusters", None), > } > > def export(tables, base): > for name, (query, partition_col) in tables.items(): > writer = spark.sql(query).write.mode("overwrite") > if partition_col: > writer = writer.partitionBy(partition_col) > writer.parquet(f"{base}/{name}/") > > if INCLUDE_BILLING: > export(BILLING, f"{TARGET}/billing") > if INCLUDE_COMPUTE: > export(COMPUTE, f"{TARGET}/compute/region={REGION}") > ``` ## FAQ > **Why does Costory not connect to the Databricks API?** > > Your export job runs inside your workspace and writes to storage you control. Costory only needs read access to that prefix. That keeps Databricks credentials in your account and limits Costory to pull-only access on data you choose to export. > **Workspace names show as ids only** > > Enable the `system.access` schema and include `workspaces` in the export (the notebook does this when `include_billing=true`). Until then, group and filter by `workspace_id`. > **Will I be double-charged if I also pay Databricks through GCP or AWS?** > > No, if you tell Costory you use marketplace billing. Costory uses your cloud bill for the dollar amount and uses Databricks usage data for breakdown dimensions. If you pay on a direct contract, Costory uses list price from `list_prices` plus an optional discount %. > **How much does the export job cost?** > > A few minutes per day on a small cluster is typical. Billing exports are a small fraction of normal Databricks spend. > **Can I delete old files in my bucket?** > > Yes. Costory ingests incrementally and keeps full history on its side. Many teams keep only about 7 to 10 days in the bucket after Costory has read them. ## Related > **[Billing Data overview](https://docs.costory.io/setup/billing)** > > All supported billing providers and shared setup concepts. > **[AWS billing](https://docs.costory.io/setup/billing/aws.md)** > > Required for Level 2 VM enrichment on AWS. > **[GCP billing](https://docs.costory.io/setup/billing/gcp.md)** > > Required for Level 2 on GCP, including detailed export. > **[Azure billing](https://docs.costory.io/setup/billing/azure.md)** > > Required for Level 2 VM enrichment on Azure. > **[Dimensions](https://docs.costory.io/features/tagging/dimensions.md)** > > Use Databricks custom tags in allocation and reports. > **[Explorer](https://docs.costory.io/features/cost-explorer.md)** > > Explore Databricks DBU costs with the rest of your stack. **Databricks reference docs:** [Billable usage system table](https://docs.databricks.com/en/admin/system-tables/billing), [Pricing system table](https://docs.databricks.com/en/admin/system-tables/pricing), [Compute system tables](https://docs.databricks.com/en/admin/system-tables/compute), [System tables overview](https://docs.databricks.com/en/admin/system-tables/index.html). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.