Databricks Terraform Onboarding

The Espresso Databricks module manages the Databricks service principal, workspace assignments, system-table grants, bootstrap SQL warehouse, Espresso credentials, and Warehouse Agent settings.

Prerequisites

• Terraform 1.11 or newer

• Databricks account-admin access

• Databricks metastore-admin access

• Identity-federated workspaces attached to Unity Catalog

• Serverless SQL support in each workspace

• An Espresso organization API key

In Espresso, select an existing account, open Tools → API Keys, choose Generate API key → Organization key, and copy the complete ok_ secret. It cannot be displayed again. Store it in your secret manager and expose it to Terraform as ESPRESSO_API_KEY.

Configure an account-level Databricks provider using a bootstrap identity that can manage every workspace passed to the module:

terraform {
  required_version = ">= 1.11.0"

  required_providers {
    databricks = {
      source  = "databricks/databricks"
      version = ">= 1.122.0"
    }
    espresso = {
      source  = "espressocomputing/espresso"
      version = ">= 0.1.2"
    }
  }
}

provider "databricks" {
  host          = "https://accounts.cloud.databricks.com"
  account_id    = var.databricks_account_id
  client_id     = var.databricks_client_id
  client_secret = var.databricks_client_secret
}

provider "espresso" {}

Configure the module

Create one Espresso account per Databricks workspace. Warehouse Agent settings are disabled unless explicitly enabled.

module "espresso_databricks" {
  source  = "espressocomputing/databricks/espresso"
  version = "~> 0.1"

  providers = {
    databricks = databricks
    espresso   = espresso
  }

  workspaces = {
    "1234567890123456" = {
      espresso_slug  = "databricks_acme_production"
      display_name   = "ACME Production Databricks"
      workspace_url  = "https://dbc-example.cloud.databricks.com"
      workspace_name = "production"

      warehouse_agent = {
        enabled     = true
        auto_opt_in = true
      }

      managed_warehouses = {
        "ANALYTICS_WH" = {
          enabled = true
        }
      }
    }
  }
}

The module grants USE_CATALOG on the system catalog and grants USE_SCHEMA and SELECT on every system schema discovered during planning. Run Terraform again after Databricks adds a system schema so the new grant is created.

When several workspaces share a Unity Catalog metastore, set system_table_access_workspace_ids to one representative workspace ID per metastore. This prevents Terraform from managing the same grants through multiple workspace APIs.

Secret storage

The module creates a Databricks OAuth secret for the Espresso service principal. The Databricks provider returns that generated secret to Terraform, so it is stored in Terraform state. Use an encrypted remote backend and restrict state access.

The Espresso provider treats its client_secret argument as write-only and does not add another copy to state. When importing an existing Databricks secret, the module also accepts its plaintext through the sensitive, ephemeral existing_service_principal_client_secret input. The Databricks resource itself remains represented in state, and any secret Terraform generates during later rotation is stored there.

Use a separate bootstrap identity for the Databricks provider. Terraform cannot safely rotate the same OAuth secret it needs to authenticate the rotation operation.

Existing onboarding resources

If the service principal, OAuth secret, workspace assignment, grants, or bootstrap warehouse already exist, import them into the module addresses before applying. Review the plan until it contains imports and expected in-place changes only. Do not allow Terraform to replace an active service principal or bootstrap warehouse during adoption.

After Terraform finishes, verify the Databricks connection and enabled warehouses in Espresso.