Airflow

Overview

This guide walks you through updating your Apache Airflow Databricks connection to point to Espresso's infrastructure. The key change is replacing your Databricks host (e.g. <workspace>.cloud.databricks.com) with the corresponding Espresso host (<workspace>.cloud.databricks.espressocomputing.com). The http_path stays the same.

Prerequisites

• Access to your Airflow environment (UI or config files)

• The package apache-airflow-providers-databricks installed

• Your Databricks connection details (Server Hostname, HTTP Path, credentials)

Steps

Option A: Update via the Airflow UI

1. Navigate to Admin → Connections in the Airflow web UI.

2. Find and click on your Databricks connection (default ID: databricks_default).

3. Set Host to your Espresso endpoint, e.g. <workspace>.cloud.databricks.espressocomputing.com.

4. Leave the Extra field as-is. The http_path does not change:

{
    "http_path": "/sql/1.0/warehouses/<warehouse_id>"
}

1. Leave Login / Password (or the token in Extra) unchanged. Your existing Databricks credentials continue to work through the proxy.

2. Click Save.

Option B: Update via Environment Variable (JSON format)

If you manage connections through environment variables, update the variable as follows:

export AIRFLOW_CONN_DATABRICKS_DEFAULT='{
    "conn_type": "databricks",
    "host": "<workspace>.cloud.databricks.espressocomputing.com",
    "password": "<personal_access_token>",
    "extra": {
        "http_path": "/sql/1.0/warehouses/<warehouse_id>"
    }
}'

Option C: Update via Environment Variable (URI format)

export AIRFLOW_CONN_DATABRICKS_DEFAULT='databricks://:<personal_access_token>@<workspace>.cloud.databricks.espressocomputing.com?http_path=%2Fsql%2F1.0%2Fwarehouses%2F<warehouse_id>'

Note: All URI components should be URL-encoded.

Verifying the Connection

After updating, test the connection by:

1. Clicking Test on the connection page in the Airflow UI (Airflow 2.5+), or

2. Running a simple DAG task that queries through the proxy (e.g. a DatabricksSqlOperator with SELECT 1).

Troubleshooting

• Authentication errors — Your existing Databricks credentials should work unchanged. If you use service principal OAuth, confirm the relevant Extra fields (service_principal_oauth, client ID/secret) are still present.

• Connection timeouts — Verify that your Airflow environment has network access to <workspace>.cloud.databricks.espressocomputing.com.

• Warehouse not found — Confirm the http_path in Extra is unchanged and still points at an existing SQL warehouse.