> For the complete documentation index, see [llms.txt](https://docs.espresso.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.espresso.ai/databricks-optimizer/databricks-spark-optimizer.md).

# Databricks Spark Optimizer

Espresso's Spark optimizer continuously right-sizes your Databricks Spark jobs. We look at how each job uses CPU, memory, and I/O, and then pick the instance type and worker count that will run the job at lowest cost. 

We then monitor your configs on an ongoing basis, handling sizing up and down as your jobs and data change.

### Onboarding

#### 1. Savings estimate (read-only)

You create an OpenShare of your system tables to our metastore. No credentials leave your workspace, no service principal is created, and no management permissions are granted.

We share a per-job report showing current vs. proposed configuration and projected savings. You see the full savings estimate before granting any permission to change configurations.

[Follow the instructions here.](/databricks-optimizer/databricks-sql-onboarding.md)

#### 2. Deployment

Most tools produce recommendations and hand them back to you, creating work for your team: an engineer has to review, validate the config, edit code, and redeploy. Our permissions let us make those changes for you automatically.

* A workspace admin runs our onboarding notebook. It creates a service principal named espresso-ai-optimizer, grants the permissions listed below, and returns a JSON credentials blob.
* You share those credentials with Espresso via your preferred secure channel.

Permissions granted to the service principal:

* SELECT on system tables - for continuous monitoring of job telemetry.
* CAN\_MANAGE on clusters, jobs, instance pools, and pipelines, plus CAN\_USE on cluster policies - to apply instance-type and worker-count changes in place.

### Security & Compliance

**Network access**

Espresso connects to your Databricks workspace API, authenticating as the service principal you issue. Front-end Private Link and workspace IP access lists control which networks can reach the API - a workspace-level setting that sits in front of authentication and operates independently of what any service principal is allowed to do.

In hybrid access, Databricks' term for Private Link combined with a workspace IP access list, internal users reach the workspace over Private Link, and external API clients like Espresso reach the workspace over the public path, gated by the allow list. You add Espresso's egress IPs to the allow list, and from there the service principal's permissions take over. No component of Espresso runs inside your network.

If your workspace is configured for no public access – Private Link only, with public\_access\_enabled = false on the private access settings – the workspace rejects public network connections before authentication. In that case, we will set up a cross-tenant Azure Private Link connection from Espresso's infrastructure into your workspace. Our services still run where they always run; only the network path changes. This is a one-time setup negotiated with your networking team.

**Data and credentials**

The only data that leaves the workspace is Databricks system table telemetry: job and cluster metadata, node timelines (CPU / memory / I/O), usage, list prices, and query history.

Handling of query text (statement\_text) is configured to match your data governance policy.

Credentials are revocable whenever by disabling the service principal or removing its OAuth token.

**Compliance**

Espresso is SOC 2 Type II compliant. Our attestation report is available upon request.