> For the complete documentation index, see [llms.txt](https://docs.espresso.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.espresso.ai/databricks-optimizer/databricks-spark-optimizer.md). # Databricks Spark Optimizer ### What Espresso AI Does Espresso continuously right-sizes your Databricks Spark jobs. We look at how each job uses CPU, memory, and I/O, and then pick the instance type and worker count that will run the job at lowest cost. We figure out the best config for each of your jobs and continuously adapt to changing data and workload patterns. ### How We Optimize Cost
For every Spark job there is a cost-vs-runtime Pareto frontier across compute configurations. Most jobs sit above the frontier, which means Espresso can change configs to cut costs without impacting performance. We consistently find three patterns in customer environments: * **Under-provisioned jobs with sporadic failures**. Jobs that were right-sized originally become memory- or CPU-starved as data grows, fail intermittently, and cost the customer in retry overhead. We upsize them, and stop failures. * **Over-provisioned "just in case."** Jobs running on far bigger instances than they need. We downsize them safely. * **Wrong shape, not just wrong size**. A job is starving for memory while sitting on idle CPU (or vice versa). We move it to a different instance family - cost drops and runtime improves. We collect usage patterns from Databricks system tables: per-job CPU, memory, and I/O-wait traces. Then, we use machine learning to estimate cost and runtime for candidate configurations from those usage logs, so we can move each job to the Pareto frontier without testing every option live. We pick the configuration that matches your SLA, with built-in headroom and ongoing monitoring for data growth. Finally, we apply the change. The next run of the job picks up the new configuration automatically, saving you money. ### Integration Process #### Stage 1 - Savings estimate (read-only) You create a Delta share of your system tables to our metastore. No credentials leave your workspace, no service principal is created, and no management permissions are granted. We share a per-job report showing current vs. proposed configuration and projected savings. You see the full savings estimate before granting any permission to change configurations. #### Stage 2 - Onboarding (continuous deployment) Most tools produce recommendations and hand them back to you, creating work for your team: an engineer has to review, validate the config, edit code, and redeploy. Our permissions let us make those changes for you automatically. * A workspace admin runs our onboarding notebook. It creates a service principal named espresso-ai-optimizer, grants the permissions listed below, and returns a JSON credentials blob. * You share those credentials with Espresso via your preferred secure channel. Permissions granted to the service principal: * SELECT on system tables - for continuous monitoring of job telemetry. * CAN\_MANAGE on clusters, jobs, instance pools, and pipelines, plus CAN\_USE on cluster policies - to apply instance-type and worker-count changes in place. ### Security & Compliance **Network access** Espresso connects to your Databricks workspace API, authenticating as the service principal you issue. Front-end Private Link and workspace IP access lists control which networks can reach the API - a workspace-level setting that sits in front of authentication and operates independently of what any service principal is allowed to do. In hybrid access, Databricks' term for Private Link combined with a workspace IP access list, internal users reach the workspace over Private Link, and external API clients like Espresso reach the workspace over the public path, gated by the allow list. You add Espresso's egress IPs to the allow list, and from there the service principal's permissions take over. No component of Espresso runs inside your network. If your workspace is configured for no public access – Private Link only, with public\_access\_enabled = false on the private access settings – the workspace rejects public network connections before authentication. In that case, we will set up a cross-tenant Azure Private Link connection from Espresso's infrastructure into your workspace. Our services still run where they always run; only the network path changes. This is a one-time setup negotiated with your networking team. **Data and credentials** The only data that leaves the workspace is Databricks system table telemetry: job and cluster metadata, node timelines (CPU / memory / I/O), usage, list prices, and query history. Handling of query text (statement\_text) is configured to match your data governance policy. Credentials are revocable whenever by disabling the service principal or removing its OAuth token. **Compliance** Espresso is SOC 2 Type II compliant. Our attestation report is available upon request. We’ve gone through comprehensive security reviews with a number of our customers, including Barclays and IBM.