# Welcome to Espresso AI Espresso AI automates data warehouse performance engineering with ML to reduce Snowflake and Databricks costs. It uses intelligent agents that run continuously to optimize scaling, scheduling, and query execution, delivering savings without manual tuning. ## What it does * Saves up to 70% on Snowflake and Databricks spend through real-time optimization. * Runs autonomous agents that act like a 24/7 team of expert data engineers. * Charges only on realized savings (no onboarding costs, no minimums, and no commitments). ## How it works We built a system of intelligent agents that optimize how your data warehouse scales, schedules, and runs queries. Each agent targets a specific layer: scaling and sizing, workload placement, and query structure. Together, they help your warehouse- and your data engineering team - do more with less. * Autoscaling Agent: Smarter multicluster scaling per workload using predictive models on metadata logs. * Scheduling Agent: Routes queries in real time to reduce idle compute and maximize utilization. * Query Agent (Private Preview): Optimizes SQL using LLMs with formal verification to preserve correctness. ## Why teams use it * Lower cloud data warehouse bills without sacrificing performance. * Fast, easy setup (run a SQL command and update configs). * Continuous optimization across all workloads and query sources. * By charging only on realized savings, ROI is guaranteed. # Snowflake Optimizer # Terraform provider ## Authentication In the Espresso dashboard, select an existing account, open **Tools → API Keys**, choose **Generate API key → Organization key**, and copy the complete `ok_` secret. It cannot be displayed again. Set it as `ESPRESSO_API_KEY`. ## Snowflake credentials ```hcl resource "espresso_account" "production" { slug = "acme_snowflake_production" display_name = "Acme Snowflake Production" product = "snowflake" } data "espresso_snowflake_public_key" "production" { account = espresso_account.production.slug } resource "snowflake_service_user" "espresso" { name = "ESPRESSO_AI_USER" default_role = "ESPRESSO_AI_ROLE" default_warehouse = "ESPRESSO_AI_WH" rsa_public_key = data.espresso_snowflake_public_key.production.public_key } resource "espresso_snowflake_credentials" "production" { account = espresso_account.production.slug snowflake_account = "acme-org-acme-production" host = "acme-org-acme-production.snowflakecomputing.com" username = snowflake_service_user.espresso.name role = "ESPRESSO_AI_ROLE" warehouse = "ESPRESSO_AI_WH" } ``` `espresso_snowflake_public_key` only reads the public half of an existing Espresso keypair. Configure the keypair through Espresso onboarding before reading it, then assign the value to the Snowflake service user before creating `espresso_snowflake_credentials`. The credentials resource always uses the stored keypair and tests the Snowflake connection before saving the remaining connection settings. Omit `host` to derive it from `snowflake_account`. An account's `display_name` can be updated in place. Its `slug` and `product` are immutable. Removing an account or credentials resource from Terraform stops managing it but does not delete the account or stored credentials from Espresso. ## Warehouse Agent settings ```hcl resource "espresso_snowflake_warehouse_agent" "production" { account = espresso_account.production.slug enabled = true auto_opt_in = true notes = "Managed by Terraform" } locals { transforming = { min_clusters = 1 max_clusters = 4 scaling_policy = "STANDARD" } } resource "snowflake_warehouse" "transforming" { name = "TRANSFORMING" min_cluster_count = local.transforming.min_clusters max_cluster_count = local.transforming.max_clusters scaling_policy = local.transforming.scaling_policy lifecycle { ignore_changes = [min_cluster_count, max_cluster_count, scaling_policy] } } resource "espresso_snowflake_warehouse_agent_warehouse" "transforming" { account = espresso_account.production.slug name = snowflake_warehouse.transforming.name enabled = true min_clusters = local.transforming.min_clusters max_clusters = local.transforming.max_clusters scaling_policy = local.transforming.scaling_policy } ``` The lifecycle list prevents the Snowflake and Espresso providers from fighting over Warehouse Agent settings. Terraform lifecycle values cannot be conditional. To return control safely, first set the Espresso warehouse's `enabled` to `false` and apply, then remove its `ignore_changes` entries and apply again. The Snowflake provider then reconciles the warehouse to the configured values. Each Warehouse Agent warehouse configuration is managed as a discrete resource. Its settings fields are optional, so an `account` and `name` can adopt the current values without changing them. Removing a Warehouse Agent resource stops Terraform management without changing the current Espresso settings or the underlying warehouse. # Snowflake Metadata This page describes what Snowflake metadata Espresso reads and why. In general, we look at workload metadata to be able to understand and optimize your traffic patterns. Espresso will *never* access your Snowflake data.\ \ Most customers, including public companies, are comfortable sharing all of the below data under NDA. #### Query History We read Snowflake's [query history](https://docs.snowflake.com/en/sql-reference/functions/query_history) table to understand what workloads you're running and when, as well as statistics about those workloads (e.g. runtime, time in queue, bytes scanned).\ \ This table contains user-generated query text. If your query text is sensitive, we're happy to sign an NDA or a BAA before sharing data. We can also remove query text from the data we collect when we generate a savings estimate upon request.\ \ All other data in this table is machine-generated. #### Query Attribution History This table gives a per-query breakdown of compute costs. #### Warehouse Event History This table logs warehouse events, such as warehouses turning on and off and clusters spinning up and down. #### **Rate Sheet Daily** **Contract Items** These tables contains Snowflake contract information. We use them to calculate financial projections and to let you know if you're on track with your committed spend. We can remove them from the data we collect for a savings estimate upon request. #### **Warehouse Metering History** **Metering Daily History** **Remaining Balance Daily** These tables store hourly credit usage broken down by warehouse and daily credit usage, respectively. We use then to tune our models for new accounts and to check for outliers and certain edge cases. #### Warehouse Settings This lists your warehouses and their settings (e.g. size, type). #### Current Account ID This is your Snowflake's account ID. We use it to uniquely identify your account.
# Snowflake Savings Estimate Instructions Using workload metadata, we simulate your environment and produce an estimate of how much we can save you. The estimate looks like this:
## How do I get an estimate? We need a few things to generate the estimate: query metadata, warehouse metadata, and Snowflake usage metadata. For a full list of what we need, see [this page](/snowflake-optimizer/snowflake-metadata). The fastest way to share those is to [set up a Snowflake account for Espresso](https://espresso.ai/snowflake-optimizer-onboarding). If you'd prefer to share your metadata without setting up an account, you can also use a Snowflake Python worksheet. Log into the [Espresso AI dashboard](https://dashboard.espressocomputing.com) with your work email to generate a command you can run. This creates a secure upload token tied to your email address. ## How do I know the savings are accurate? Our models are continuously calibrated with production Snowflake data to ensure our savings numbers are accurate. The best way for you to judge accuracy is to compare our upfront savings estimate to the savings you see in production when we first turn on. We also encourage users to run A/B tests once we've been on for a few months: shut Espresso off for a week and see how your actual spend compares to our savings calculation. ## NDA and support Espresso AI is happy to sign an NDA. Contact with your NDA or any questions. # Scheduler Credit Attribution Espresso attributes Snowflake credits by preserving routing metadata in query text, reconstructing which original warehouse each query came from, then slicing billed cluster uptime across active and idle intervals. ## 1. Query comments carry routing context Every proxied query gets one `Espresso Metadata` line comment. `old_warehouse` is the warehouse the user selected; `new_warehouse` is where Espresso ran it. ```sql // Espresso Metadata: {"query_id": "01b4...", "old_warehouse": "ANALYTICS_WH", "new_warehouse": "_ESPRESSO_POOL_XS_1", "routing_enabled": true} ``` * Snowflake stores the comment in `QUERY_HISTORY.QUERY_TEXT`; attribution parses `old_warehouse` from it. * When query text is censored for export, the metadata comment can still be retained. ## 2. Cluster uptime is split into query and idle slices Snowflake bills for warehouse uptime, not just query runtime. For an interval where a cluster is up, Espresso builds boundaries from query starts, query ends, resize/routing state, and metering windows. Each slice is either active, with one or more running queries, or idle. A model allocates shared active slices across the running queries and allocates idle slices using surrounding warehouse activity and routing context.
Overlapping queries on one running cluster with a trailing idle slice

Idle spans are first-class slices; the model attributes them alongside active overlap spans instead of dropping them.

The widths and ratios in the diagrams are illustrative; they show the accounting shape, not a literal allocation for a specific workload. ## 3. Slices roll up to attributed metering rows After slicing, Espresso rolls each slice into the original warehouse namespace from the metadata comment and session/routing state. The output is `espresso.reporting.warehouse_metering_history`, shaped like Snowflake metering history so existing BI queries can read attributed credits.
Uptime slices rolled up into attributed warehouse rows

The model allocates both shared active slices and idle slices into warehouse rows; attributed totals still match the metered total.

# Snowflake PrivateLink Configure Snowflake private connectivity for Espresso AI. Espresso AI can connect to Snowflake privately instead of using the public Snowflake endpoint. On AWS, Snowflake uses AWS PrivateLink. On Azure, Snowflake uses Azure Private Link. This page applies to Espresso-managed connections to your Snowflake account. For proxy traffic from your own tools, use the proxy onboarding guides. ## How it works Snowflake private connectivity gives each Snowflake account private connection hostnames, including a Snowflake account URL and an OCSP URL. Espresso configures its Snowflake connection to use the private Snowflake account URL as its Snowflake host. Espresso creates and operates the cloud-side private endpoint from the Espresso environment. You do not need to create another private endpoint, update DNS, or route Espresso through your VPC. ## Prerequisites * Your Snowflake account is on Business Critical or higher. * Your Snowflake administrator can use the `ACCOUNTADMIN` role. * The Espresso Snowflake service user and role are already created. * You have not enforced private-only Snowflake access for Espresso yet. ## 1. Send your PrivateLink configuration to Espresso Before you open a Snowflake Support case, run this in Snowflake: ```sql USE ROLE ACCOUNTADMIN; SELECT SYSTEM$GET_PRIVATELINK_CONFIG(); ``` Send the full JSON output to Espresso. Espresso will use the fields needed for your deployment, including: | Value | Why Espresso needs it | | ------------------------------------ | ------------------------------------------------------------------- | | `privatelink-account-name` | Snowflake account identifier for Espresso's Snowflake connection. | | `privatelink-account-url` | Private Snowflake hostname Espresso will use. | | `regionless-privatelink-account-url` | Optional alternative hostname if you use organization/account URLs. | | `privatelink-ocsp-url` | OCSP hostname that Snowflake clients use for certificate checks. | | `regionless-privatelink-ocsp-url` | Optional OCSP hostname for the regionless account URL. | | `privatelink-vpce-id` | Snowflake endpoint service ID for AWS PrivateLink. | | `privatelink-pls-id` | Snowflake Private Link Service ID for Azure Private Link. | Espresso will give you a private endpoint resource ID to include in your Snowflake Support case. On AWS, this is the VPC endpoint ID. On Azure, this is the full Azure private endpoint resource ID. ## 2. Ask Snowflake to authorize the connection ### AWS accounts For Snowflake accounts hosted on AWS, Snowflake must authorize Espresso's AWS account and private endpoint before Espresso can use the connection. After Espresso gives you the private endpoint resource ID, open a Snowflake Support case and ask Snowflake to authorize Espresso AI's AWS account and private endpoint for PrivateLink access to your Snowflake account. In the [Espresso AI dashboard](https://dashboard.espressocomputing.com/), go to `Proxy Onboarding` and copy Espresso AI's AWS Account ID. Suggested support case text: ``` Please authorize Espresso AI for AWS PrivateLink access to our Snowflake account as a managed cloud service / third-party vendor. Espresso AI AWS account ID: Private endpoint resource ID (VPC endpoint ID): ``` Snowflake's self-service AWS PrivateLink flow does not support third-party vendor account authorization. Snowflake Support must complete this step. ### Azure accounts For Snowflake accounts hosted on Azure, the equivalent private connectivity service is Azure Private Link. After Espresso gives you the private endpoint resource ID, open a Snowflake Support case and ask Snowflake to allow Espresso AI to establish an Azure Private Link connection to your Snowflake account. In the [Espresso AI dashboard](https://dashboard.espressocomputing.com/), go to `Proxy Onboarding` and copy Espresso AI's Azure subscription ID for Snowflake Private Link. If you do not see it, ask Espresso for the Azure subscription ID to use. If Espresso will connect to more than one Snowflake account, include every Snowflake account URL in the support case. Suggested support case text: ``` Please allow this Azure subscription to establish an Azure Private Link connection to our Snowflake account. Azure subscription ID: Private endpoint resource ID: Snowflake account URL(s): ``` Snowflake uses the resource ID to approve the specific Espresso-managed private endpoint. ## 3. Tell Espresso when Snowflake approves the request Send Espresso Snowflake's confirmation. Espresso will finish configuring the connection and validate that traffic uses the private endpoint. ## Wait for Espresso validation Do not enable Snowflake private-only enforcement or remove existing network policy access for Espresso until Espresso confirms validation. ## After validation After Espresso confirms the connection is using PrivateLink, you can restrict Snowflake public access for the Espresso service user according to your Snowflake network policy standards. If you make this change, tell Espresso before enforcement so we can verify the cutover window and avoid interrupting warehouse optimization. ## Snowflake references * [AWS PrivateLink and Snowflake](https://docs.snowflake.com/en/user-guide/admin-security-privatelink) * [Azure Private Link and Snowflake](https://docs.snowflake.com/en/user-guide/privatelink-azure) * [SYSTEM$GET\_PRIVATELINK\_CONFIG](https://docs.snowflake.com/en/sql-reference/functions/system_get_privatelink_config) # Snowflake Proxy Onboarding This page explains how to point your tools at the Espresso AI proxy. ## General To use our proxy, configure your tools to point to us instead of directly to Snowflake.\ ‍ 1. Find the Snowflake URL you shared with Espresso AI, e.g. `.snowflakecomputing.com`. 2. Replace it with `.espressocomputing.com` in each tool you are onboarding. ## Tool-specific guides * [Snowflake Web App users](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-snowflake-web-app) * [Python](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-python) * [dbt Cloud](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-dbt-cloud) * [Fivetran](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-fivetran) * [Hex](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-hex) ## Scheduler Warehouse access * [Grant access to all warehouses](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-warehouse-access) ## On-prem Deployment * [Proxy on-prem Terraform deployment (AWS)](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-terraform-deployment-aws) * [Proxy on-prem Terraform deployment (Azure)](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-terraform-deployment-azure) * [Proxy on-prem Helm chart deployment](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-helm-deployment) # Authentication Snowflake authentication continues to work as normal. The proxy forwards authentication traffic to Snowflake without modifying it, so credentials are never inspected, stored, or altered. Snowflake performs all authentication exactly as it does without the proxy, and every authentication method it supports is unaffected by routing. ## Supported authentication methods * **Username and password**: Credentials pass through unchanged and are validated by Snowflake as usual. * **Multi-factor authentication (MFA)**: MFA prompts and challenges are handled directly between you and Snowflake. The proxy does not interfere with the MFA flow. * **Federated authentication & SSO**: SAML/SSO redirects to your identity provider continue to work normally, since the proxy passes these requests through untouched. * **Key-pair authentication**: Signed JWTs are forwarded to Snowflake for verification. Your private keys never leave your environment and are never seen by the proxy. * **Programmatic access tokens (PATs)**: Tokens are forwarded as-is and validated by Snowflake. * **OAuth (Snowflake OAuth and External OAuth)**: OAuth token exchange and validation occur directly with Snowflake and your authorization server. The proxy does not modify tokens or intercept the OAuth flow. * **Workload Identity Federation (WIF)**: Service-to-service identity tokens are passed through to Snowflake unchanged for verification. ## IP allowlists If you use an IP allowlist, please let us know. Snowflake will see requests originating from the proxy's IP address, but we can mirror your allowlist on the proxy so requests are approved and rejected exactly per your existing policy. Please add the following IPs to your allowlist for all users: ``` 18.233.13.51 34.195.242.31 34.231.116.52 34.231.212.71 34.234.123.175 35.169.148.94 52.87.110.223 54.161.160.239 ``` # Snowflake Web App Users To use Espresso AI optimizations in combination with the proxy, install the [Espresso AI Chrome extension](https://chromewebstore.google.com/detail/espresso-ai/gakppoblkclomegbggekigkdhdpfgpmp). # Python If you have custom Python code, add the `host` parameter to your connection initialization, e.g. ```python import snowflake.connector conn = snowflake.connector.connect( user="", authenticator="oauth", token="", account="", host=".espressocomputing.com", ) ``` # Airflow ### Overview This guide walks you through updating your Apache Airflow Snowflake connection to point to Espresso's infrastructure. The key change is replacing your Snowflake host (`.snowflakecomputing.com`) with the corresponding Espresso host (`.espressocomputing.com`). ### Prerequisites * Access to your Airflow environment (UI or config files) * Your Espresso account details (host, account name, database, warehouse, etc.) * The package `apache-airflow-providers-snowflake` must be version >= 6.0.0 for the host parameter to work. ### Steps #### Option A: Update via the Airflow UI 1. Navigate to **Admin → Connection**s in the Airflow web UI. 2. Find and click on your Snowflake connection (default ID: `snowflake_default`). 3. In the **Extra** field, add or update the `host` parameter to your Espresso endpoint: {% code overflow="wrap" %} ```json { "account": "", "database": "", "warehouse": "", "host": ".espressocomputing.com" } ``` {% endcode %} 4. Update **Login** and **Password** if your Espresso credentials differ from your previous Snowflake credentials. 5. Click **Save**. #### Option B: Update via Environment Variable (JSON format) If you manage connections through environment variables, update the variable as follows: {% code overflow="wrap" %} ```json export AIRFLOW_CONN_SNOWFLAKE_DEFAULT='{ "conn_type": "snowflake", "login": "", "password": "", "schema": "", "extra": { "account": "", "database": "", "warehouse": "", "host": ".espressocomputing.com" } }' ``` {% endcode %} #### Option C: Update via Environment Variable (URI format) If using the URI format (common in Airflow versions prior to 2.3.0): {% code overflow="wrap" %} ```json export AIRFLOW_CONN_SNOWFLAKE_DEFAULT='snowflake://:@/?account=&database=&warehouse=&host=.espressocomputing.com' ``` {% endcode %} **Note:** All URI components should be URL-encoded. ### Verifying the Connection After updating, test the connection by: 1. Clicking **Test** on the connection page in the Airflow UI (Airflow 2.5+), or 2. Running a simple DAG task that queries Espresso (e.g., a `SnowflakeOperator` with `SELECT 1`). ### Troubleshooting * **Authentication errors** — Double-check that your Login and Password match your Espresso credentials. If using key pair or OAuth authentication, ensure the relevant Extra fields (`private_key_file`, `authenticator`, etc.) are also updated. * **Connection timeouts** — Verify that your Airflow environment has network access to `.espressocomputing.com`. * **Schema/database not found** — Confirm that the `database`, `schema`, and `warehouse` values exist in your Espresso account. # dbt Cloud 1. Open your dbt Cloud project and go to **Environments**.
dbt Cloud environments page
2. For each environment on the list, select the environment.
dbt Cloud environment list
3. Click the settings icon.
dbt Cloud settings icon
4. Click **Edit**.
dbt Cloud edit button
5. Set the Snowflake host in **Extended Attributes** (bottom of the page).
dbt Cloud extended attributes
6. Click **Save**.
dbt Cloud save button
# dbt Core You can connect to Snowflake through a custom domain using the `host` parameter in `profiles.yaml`: ```yaml my_project: target: dev outputs: dev: type: snowflake account: myorg-myaccount host: $account.espressocomputing.com user: # ... ``` Each project can be configured to use its own profile if needed. See the [dbt docs](https://docs.getdbt.com/docs/platform/connect-data-platform/connect-snowflake?version=2.0\&name=Fusion#custom-domain-url) for more information. # Fivetran 1. Go to **Destinations**.
Fivetran destinations
2. Select your Snowflake connection and update the host to `.espressocomputing.com`.
Fivetran update host
3. Click **Save & Test** in the bottom right corner. # Hex 1. In Hex, open **Data connections** and select **Snowflake**. 2. Click the **Proxy** checkbox.
Hex Snowflake proxy settings
3. In **Account identifier**, enter the full proxy host: `.espressocomputing.com`. 4. Click **Connect**. # Looker 1. Follow the [instructions here](https://docs.cloud.google.com/looker/docs/db-config-snowflake#creating-the-connection-to-your-database) to set up a connection to Snowflake in Looker. 2. For the hostname parameter, change `.snowflakecomputing.com` to `.espressocomputing.com`. 3. Press **Test** to test your connection. # Warehouse access To onboard to the scheduler feature, ensure users have access to all the warehouses we route to. Granting warehouse access does not grant any data access. It only permits users to operate the warehouses they already have data access to use. ## Fast grant for all warehouses For fast onboarding, run the following query to grant `OPERATE` on every warehouse to the `PUBLIC` role: ```sql EXECUTE IMMEDIATE $$ DECLARE rs RESULTSET; BEGIN SHOW WAREHOUSES; rs := ( SELECT "name" AS WAREHOUSE_NAME FROM TABLE(RESULT_SCAN(LAST_QUERY_ID(-1))) ); FOR r IN rs DO EXECUTE IMMEDIATE 'GRANT USAGE ON WAREHOUSE "' || r.WAREHOUSE_NAME || '" TO ROLE PUBLIC'; END FOR; END; $$; ``` # Self-hosted Terraform deployment (AWS) This guide explains how to deploy Espresso AI's Proxy Service on your AWS infrastructure with Terraform. You can deploy either: 1. In a dedicated VPC that Terraform creates. 2. In an existing VPC that you provide. ## Prerequisites * Access to an AWS account with IAM permissions for VPC, EKS, IAM roles, EC2/load balancers, Route53 (if used), and Secrets Manager (if used). * In the [Espresso AI dashboard](https://dashboard.espressocomputing.com/), go to `Proxy Onboarding` and: * Enter your AWS account ID so we can grant ECR access for the Proxy image. * Copy your customer name. * Copy Espresso AI's AWS Account ID. This is needed for the ECR url. * Generate an API key for Espresso API authentication. ## What this module creates * VPC (optional) or uses your existing VPC/subnets. * EKS cluster and node group. * Karpenter for node autoscaling. * AWS Load Balancer Controller. * Proxy deployment, service, and HPA in Kubernetes. * Optional Route53 record. * Optional managed API key flow via AWS Secrets Manager + External Secrets. ## Example usage ### Dedicated VPC + managed secret + DNS ```hcl variable "proxy_api_key_value" { description = "Managed proxy API key value for Secrets Manager sync." type = string sensitive = true } module "proxy_on_prem" { source = "github.com/espressocomputing/espresso-ai-proxy-tf//aws?ref=v0.4.3" region = "us-east-1" customer = "" create_dedicated_vpc = true vpc_config = { cidr = "10.80.0.0/16" public_subnet_cidrs = ["10.80.0.0/20", "10.80.16.0/20"] private_subnet_cidrs = ["10.80.32.0/20", "10.80.48.0/20"] availability_zones = ["us-east-1a", "us-east-1b"] } eks_config = { cluster_endpoint_public_access = true cluster_endpoint_public_access_cidrs = ["203.0.113.10/32"] } proxy_config = { repository = ".dkr.ecr.us-east-1.amazonaws.com/proxy" image = "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" proxy_host = "customer.example.com" api_key_secret_mode = "MANAGED_AWS_SECRETS_MANAGER" } proxy_api_key_value = var.proxy_api_key_value alb_config = { certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-2222-3333-4444-555555555555" ingress_host = "proxy.customer.example.com" } dns_config = { create_record = true zone_id = "Z123EXAMPLE456" record_name = "proxy.customer.example.com" } } ``` ### Existing VPC + bring-your-own Kubernetes secret ```hcl module "proxy_on_prem" { source = "github.com/espressocomputing/espresso-ai-proxy-tf//aws?ref=v0.4.3" region = "us-east-1" customer = "" create_dedicated_vpc = false existing_vpc_config = { vpc_id = "vpc-0123456789abcdef0" private_subnet_ids = ["subnet-01aaaa", "subnet-02bbbb"] public_subnet_ids = ["subnet-03cccc", "subnet-04dddd"] } eks_config = { cluster_endpoint_public_access = true cluster_endpoint_public_access_cidrs = ["203.0.113.10/32"] } proxy_config = { repository = ".dkr.ecr.us-east-1.amazonaws.com/proxy" image = "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" proxy_host = "customer.example.com" api_key_secret_mode = "BYO_K8S_SECRET" api_key_secret_name = "espresso-ai" } alb_config = { certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-2222-3333-4444-555555555555" ingress_host = "proxy.customer.example.com" } } ``` ## Argument reference ### Top-level arguments * `region`: Required. AWS region for deployment. * `customer`: Required. Customer identifier used in naming and `API_URL` suffixing. * `create_dedicated_vpc`: Optional. Creates dedicated VPC (`true`) or uses existing VPC (`false`). Default: `true`. * `vpc_config`: Optional/conditional. Required when `create_dedicated_vpc = true`. * `existing_vpc_config`: Optional/conditional. Required when `create_dedicated_vpc = false`. * `eks_config`: Optional. EKS cluster and node group settings. * `karpenter_config`: Optional. Karpenter NodePool tuning. * `proxy_config`: Required. Proxy runtime configuration. * `proxy_api_key_value`: Optional/conditional, sensitive. Required when `proxy_config.api_key_secret_mode = MANAGED_AWS_SECRETS_MANAGER`. * `alb_config`: Optional. ALB ingress configuration. * `dns_config`: Optional. Route53 alias record configuration. * `autoscaling_config`: Optional. Proxy HPA configuration. * `tags`: Optional. Additional AWS tags. Default: `{}`. ### `vpc_config` * `vpc_name`: Optional. Default: `espresso-ai-proxy-vpc`. * `cidr`: Required in dedicated VPC mode. * `public_subnet_cidrs`: Required in dedicated VPC mode. * `private_subnet_cidrs`: Required in dedicated VPC mode. * `availability_zones`: Required in dedicated VPC mode and must align with subnet counts. ### `existing_vpc_config` * `vpc_id`: Required in existing VPC mode. * `private_subnet_ids`: Required in existing VPC mode. * `public_subnet_ids`: Optional. Default: `[]`. ### `eks_config` * `cluster_name`: Optional. Default: `espresso-ai-proxy`. * `cluster_version`: Optional. Default: `1.35`. * `bootstrap_self_managed_addons`: Optional. Default: `false`. * `cluster_endpoint_public_access`: Optional. Default: `true`. * `cluster_endpoint_private_access`: Optional. Default: `true`. * `cluster_endpoint_public_access_cidrs`: Required when public endpoint access is enabled. * `create_cloudwatch_log_group`: Optional. Default: `false`. * `cloudwatch_log_group_retention_in_days`: Optional. Default: `90`. * `instance_types`: Optional. Default: `["c8i.2xlarge", "c8i.4xlarge"]`. * `node_group_min_size`: Optional. Default: `2`. * `node_group_desired_size`: Optional. Default: `2`. * `node_group_max_size`: Optional. Default: `10`. ### `karpenter_config` * `instance_types`: Optional. Default: `["c8i.2xlarge", "c8i.4xlarge"]`. * `capacity_types`: Optional. Default: `["on-demand"]`. * `cpu_limit`: Optional. Default: `64`. * `memory_limit`: Optional. Default: `256Gi`. * `node_cap`: Optional. Default: `10`. ### `proxy_config` * `image`: Required. Proxy container image URI in Espresso AI's ECR. * `replicas`: Optional. Default: `2`. * `proxy_host`: Required. Your base domain (e.g. `example.com`), injected as `PROXY_HOST`. Use the registrable base domain only — not a full hostname (`proxy.example.com`), scheme, or port. * `otel_collector`: Optional. OTEL Collector sidecar configuration. See [`otel_collector`](#otel_collector) below. * `api_key_secret_name`: Optional. Kubernetes secret name for API key injection. Default: `espresso-ai`. * API key secret key name is fixed to `ESPRESSO_AI_API_KEY` and is not configurable. * `api_key_secret_mode`: Optional. `BYO_K8S_SECRET` or `MANAGED_AWS_SECRETS_MANAGER`. Default: `BYO_K8S_SECRET`. * `api_key_aws_secret_name`: Optional. AWS Secrets Manager secret name used in managed mode. Default: `/espresso-ai/proxy/api-key`. * `api_url`: Optional. Base URL. Default: `https://api.espressocomputing.com:25831`. * `env_vars`: Optional. Map of environment variable key/value pairs. Currently supported keys: | key | type | definition | | -------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `EXCLUDE_QUERY_TEXT` | `bool` | Default: `false`. Whether to exclude query text on requests to Espresso AI's API. *Note: Enabling this will limit supported functionality.* | ### `otel_collector` Nested object under `proxy_config`. When `enabled = true`, the module deploys an OpenTelemetry Collector sidecar in the proxy pod and renders a ConfigMap with its pipeline. The proxy is automatically pointed at `http://localhost:4318`; the sidecar forwards to your-owned OTLP backend. * `enabled`: Optional. Default: `true`. Set to `false` to disable the sidecar; the proxy will then emit OTLP directly to `otel_exporter_otlp_endpoint`. * `image`: Optional. Full image reference for the collector container. Default: `otel/opentelemetry-collector-contrib:0.152.0`. * `customer_endpoint`: Optional. OTLP endpoint for the customer's own observability backend. Leave empty (default) to disable the customer exporter; the Espresso pipeline still runs. * `customer_protocol`: Optional. `grpc` (renders the `otlp/customer` exporter) or `http` (renders `otlphttp/customer`). Default: `grpc`. * `customer_signals`: Optional. Signals to mirror to the customer exporter. Any subset of `metrics`, `logs`. Default: all three. * `customer_auth_secret_name`: Optional. Existing Kubernetes Secret in the proxy namespace whose value is mounted as `CUSTOMER_OTLP_AUTH` and sent as the customer exporter's `Authorization` header. Leave empty for unauthenticated endpoints. * `customer_auth_secret_key`: Optional. Key within `customer_auth_secret_name`. Default: `authorization`. * `customer_tls_insecure`: Optional. Disable TLS verification on the customer exporter. Default: `false`. Example — also mirror metrics (not logs) to the customer's own OTLP backend with bearer-token auth: ```hcl proxy_config = { repository = ".dkr.ecr.us-east-1.amazonaws.com/proxy" image = "0.1-dev-..." proxy_host = "customer.example.com" otel_collector = { customer_endpoint = "https://otlp.observability.customer.example.com:4317" customer_protocol = "grpc" customer_signals = ["metrics", "logs"] customer_auth_secret_name = "customer-otlp-auth" } } ``` The `customer-otlp-auth` Secret must exist in the `proxy` namespace and contain an `authorization` key whose value is the full header (e.g. `Bearer eyJ...`). For the full list of metrics, spans, and resource attributes the proxy emits — useful for building dashboards and alerts against the customer exporter — see [Proxy telemetry reference](/snowflake-optimizer/proxy-onboarding/proxy-telemetry-reference). ### `alb_config` * `enable_ingress`: Optional. Enables ALB ingress. Default: `true`. * `certificate_arn`: Required when ingress is enabled. * `ingress_host`: Optional. Host rule. * `scheme`: Optional. `internet-facing` or `internal`. Default: `internet-facing`. ### `dns_config` * `create_record`: Optional. Creates Route53 alias. Default: `false`. * `zone_id`: Required when `create_record = true`. * `record_name`: Optional. Falls back to ingress host if omitted. ### `autoscaling_config` * `min_replicas`: Optional. Default: `2`. * `max_replicas`: Optional. Default: `10`. * `target_cpu_utilization`: Optional. Default: `70`. ## Secret modes * `BYO_K8S_SECRET` (default): Proxy reads from an existing Kubernetes secret (`api_key_secret_name`) using fixed key `ESPRESSO_AI_API_KEY`. * `MANAGED_AWS_SECRETS_MANAGER`: Module provisions AWS Secrets Manager secret, IRSA, External Secrets Operator, and syncs to Kubernetes secret. ## Outputs The module exports: * `vpc_id` * `public_subnet_ids` * `private_subnet_ids` * `eks_cluster_name` * `eks_cluster_endpoint` * `eks_cluster_security_group_id` * `proxy_namespace` * `proxy_service_name` * `proxy_service_load_balancer_hostname` * `proxy_ingress_load_balancer_hostname` * `proxy_hpa_name` * `proxy_dns_fqdn` ## How to deploy Deployment typically takes around 20-30 minutes. ```bash terraform init terraform plan terraform apply ``` ## Best practices * Manage sensitive variables via environment variables or `.tfvars`. ## Version Migrations The v0.1.0 → v0.2.0 change is a path-only refactor: the AWS configuration moved from the repo root into an `aws/` subdirectory, so the source URL needs `//aws`. No resource addresses changed, so existing state continues to apply cleanly. ```bash module "proxy_on_prem" { source = "github.com/espressocomputing/espresso-ai-proxy-tf//aws?ref=v0.2.0" # ^^^^^ new ... } ``` ```bash terraform init -upgrade terraform plan terraform apply ``` `plan` should report zero changes. If it shows any resource being destroyed, recreated, or replaced, stop and investigate before running `apply`. # Self-hosted Terraform deployment (Azure) This guide explains how to deploy Espresso AI's Proxy Service on your Azure infrastructure with Terraform. You can deploy either: 1. In a dedicated VNet that Terraform creates. 2. In an existing VNet that you provide. ## Prerequisites * Access to an Azure subscription with permissions to create resource groups, VNets, AKS clusters, public IPs, role assignments, user-assigned managed identities, and (optionally) Key Vault, Azure DNS records, and Azure Front Door. * In the [Espresso AI dashboard](https://dashboard.espressocomputing.com/), go to `Proxy Onboarding` and: * Enter your Azure Subscription ID so we can grant ACR access for the Proxy image. We will generate a username and password for you to be able to pull the image from our ACR. * Copy your customer name. * If running on Azure, copy Espresso AI's Azure Account ID. This is needed for the ACR url. * Generate an API key for Espresso API authentication. ## What this module creates * Resource group (optional) or uses your existing resource group. * VNet with a node subnet (optional) or uses your existing VNet/subnet. * AKS cluster with a system-assigned identity, Workload Identity + OIDC issuer enabled, and an autoscaling node pool. * Static public IP for ingress (placed in the AKS-managed node resource group). * `ingress-nginx` controller wired to the static public IP, with Azure LB annotations tuned for AKS Standard LB. * Proxy deployment, service, and HPA in Kubernetes. * TLS for the Ingress, via either: * cert-manager + Let's Encrypt (HTTP-01 by default, DNS-01 via Azure DNS for wildcard hosts), or * a bring-your-own `kubernetes.io/tls` secret you pre-create in the proxy namespace. * Optional Azure DNS A record pointing at the ingress public IP. * Optional managed API key flow via Azure Key Vault + External Secrets Operator (federated workload identity). * Optional Azure Front Door fronting the AKS LB (recommended when clients enforce strict OCSP behavior, e.g. Snowflake's connector). ## Example usage ### Dedicated VNet + cert-manager (Let's Encrypt) + Azure DNS record ```hcl module "proxy_on_prem" { source = "github.com/espressocomputing/espresso-ai-proxy-tf//azure?ref=v0.4.3" location = "eastus" customer = "" resource_group_config = { create = true name = "espresso-ai-proxy-rg" } create_dedicated_vnet = true vnet_config = { address_space = ["10.240.0.0/16"] node_subnet_cidr = "10.240.0.0/22" } aks_config = { api_server_authorized_ranges = ["203.0.113.10/32"] } proxy_config = { repository = ".azurecr.io/proxy" image = "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" proxy_host = "customer.example.com" } ingress_config = { enable_ingress = true ingress_host = "proxy.customer.example.com" letsencrypt_email = "ops@customer.example.com" } dns_config = { create_record = true zone_name = "customer.example.com" zone_resource_group_name = "dns-rg" record_name = "proxy.customer.example.com" } } ``` ### Existing VNet + bring-your-own TLS secret + managed API key in Key Vault ```hcl variable "proxy_api_key_value" { description = "Managed proxy API key value for Key Vault sync." type = string sensitive = true } module "proxy_on_prem" { source = "github.com/espressocomputing/espresso-ai-proxy-tf//azure?ref=v0.4.3" location = "eastus" customer = "" resource_group_config = { create = false name = "existing-platform-rg" } create_dedicated_vnet = false existing_vnet_config = { vnet_id = "/subscriptions//resourceGroups/network-rg/providers/Microsoft.Network/virtualNetworks/platform-vnet" node_subnet_id = "/subscriptions//resourceGroups/network-rg/providers/Microsoft.Network/virtualNetworks/platform-vnet/subnets/aks-nodes" } aks_config = { api_server_authorized_ranges = ["203.0.113.10/32"] } proxy_config = { repository = ".azurecr.io/proxy" image = "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" proxy_host = "customer.example.com" api_key_secret_mode = "MANAGED_AZURE_KEY_VAULT" api_key_azure_key_vault_secret = "espresso-ai-proxy-api-key" } proxy_api_key_value = var.proxy_api_key_value ingress_config = { enable_ingress = true ingress_host = "proxy.customer.example.com" tls_secret_name = "proxy-tls" } } ``` ### Wildcard ingress + Azure Front Door + DNS-01 wildcard cert Use this shape when fronting Snowflake's connector (or any client with strict OCSP behavior). AFD terminates TLS for clients with a DigiCert-issued managed cert, and the LE cert on the AKS LB only secures the AFD-to-origin hop. ```hcl module "proxy_on_prem" { source = "github.com/espressocomputing/espresso-ai-proxy-tf//azure?ref=v0.4.3" location = "eastus" customer = "" aks_config = { api_server_authorized_ranges = ["203.0.113.10/32"] } proxy_config = { repository = ".azurecr.io/proxy" image = "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" proxy_host = "customer.example.com" } ingress_config = { enable_ingress = true ingress_host = "*.customer.example.com" letsencrypt_email = "ops@customer.example.com" front_door = { enabled = true sku_name = "Standard_AzureFrontDoor" } } # Wildcard hosts can only be issued by Let's Encrypt via DNS-01. letsencrypt_dns01_azure_dns = { zone_name = "customer.example.com" zone_resource_group_name = "dns-rg" } } ``` ## Argument reference ### Top-level arguments * `location`: Required. Azure region for deployment. * `customer`: Required. Customer identifier used in naming and `API_URL` suffixing. * `resource_group_config`: Optional. Set `create = false` and supply `name` to deploy into an existing resource group. Default: creates `espresso-ai-proxy-rg`. * `create_dedicated_vnet`: Optional. Creates a dedicated VNet (`true`) or uses an existing VNet (`false`). Default: `true`. * `vnet_config`: Optional/conditional. Used when `create_dedicated_vnet = true`. * `existing_vnet_config`: Optional/conditional. Required when `create_dedicated_vnet = false`. * `aks_config`: Optional. AKS cluster and node pool settings. * `proxy_config`: Required. Proxy runtime configuration. * `proxy_api_key_value`: Optional/conditional, sensitive. Required when `proxy_config.api_key_secret_mode = MANAGED_AZURE_KEY_VAULT`. * `ingress_config`: Optional. NGINX ingress configuration, TLS provisioning, and optional Azure Front Door fronting. * `dns_config`: Optional. Azure DNS A-record configuration. * `letsencrypt_dns01_azure_dns`: Optional. Switches cert-manager to the DNS-01 solver via Azure DNS. Required when `ingress_config.ingress_host` is a wildcard. * `autoscaling_config`: Optional. Proxy HPA configuration. * `tags`: Optional. Additional Azure tags. Default: `{}`. ### `resource_group_config` * `create`: Optional. Default: `true`. * `name`: Optional. Default: `espresso-ai-proxy-rg`. Must be non-empty. When `create = false`, an existing resource group with this name must exist. ### `vnet_config` * `vnet_name`: Optional. Default: `espresso-ai-proxy-vnet`. * `address_space`: Optional. Default: `["10.240.0.0/16"]`. Must contain at least one valid CIDR when `create_dedicated_vnet = true`. * `node_subnet_cidr`: Optional. Default: `10.240.0.0/22`. Must be a valid CIDR within `address_space`. ### `existing_vnet_config` * `vnet_id`: Required in existing-VNet mode. * `node_subnet_id`: Required in existing-VNet mode. ### `aks_config` * `cluster_name`: Optional. Default: `espresso-ai-proxy`. Used as the resource name prefix throughout the module. * `kubernetes_version`: Optional. Default: `1.35`. * `api_server_authorized_ranges`: Optional. Required when `enable_private_cluster = false`. CIDRs allowed to reach the AKS API server. * `enable_private_cluster`: Optional. Default: `false`. When `true`, the API server has only a private endpoint. * `pod_cidr`: Optional. Default: `10.244.0.0/16`. CNI Overlay pod range; not part of the VNet. * `service_cidr`: Optional. Default: `10.245.0.0/16`. ClusterIP service range. * `dns_service_ip`: Optional. Default: `10.245.0.10`. Must lie within `service_cidr`. * `vm_size`: Optional. Default: `Standard_D8s_v5`. * `node_pool_min_count`: Optional. Default: `2`. * `node_pool_max_count`: Optional. Default: `10`. Must be ≥ `node_pool_min_count`. * `enable_log_analytics`: Optional. Default: `false`. When `true`, attaches a Log Analytics workspace and enables Container Insights. * `log_analytics_retention_days`: Optional. Default: `90`. ### `proxy_config` * `image`: Required. Proxy container image URI in Espresso AI's ACR. * `replicas`: Optional. Default: `2`. * `proxy_host`: Required. Your base domain (e.g. `example.com`), injected as `PROXY_HOST`. Use the registrable base domain only — not a full hostname (`proxy.example.com`), scheme, or port. * `otel_collector`: Optional. OTEL Collector sidecar configuration. See [`otel_collector`](#otel_collector) below. * `api_key_secret_name`: Optional. Kubernetes secret name in the proxy namespace from which `ESPRESSO_AI_API_KEY` is mounted. Default: `espresso-ai`. * API key secret key name is fixed to `ESPRESSO_AI_API_KEY` and is not configurable. * `api_key_secret_mode`: Optional. `BYO_K8S_SECRET` or `MANAGED_AZURE_KEY_VAULT`. Default: `BYO_K8S_SECRET`. * `api_key_azure_key_vault_secret`: Optional. Key Vault secret name used in managed mode. Default: `espresso-ai-proxy-api-key`. Required (non-empty) when `api_key_secret_mode = MANAGED_AZURE_KEY_VAULT`. * `api_url`: Optional. Base URL. Default: `https://api.espressocomputing.com:25831`. * `env_vars`: Optional. Map of environment variable key/value pairs. Currently supported keys: | key | type | definition | | -------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `EXCLUDE_QUERY_TEXT` | `bool` | Default: `false`. Whether to exclude query text on requests to Espresso AI's API. *Note: Enabling this will limit supported functionality.* | ### `otel_collector` Nested object under `proxy_config`. When `enabled = true` (the default, starting in `v0.4.0`), the module deploys an OpenTelemetry Collector sidecar in the proxy pod and renders a ConfigMap with its pipeline. The proxy is automatically pointed at `http://localhost:4318`; the sidecar forwards to your-owned OTLP backend. * `enabled`: Optional. Default: `true`. Set to `false` to disable the sidecar; the proxy will then emit OTLP directly to `otel_exporter_otlp_endpoint`. * `image`: Optional. Full image reference for the collector container. Default: `otel/opentelemetry-collector-contrib:0.152.0`. * `customer_endpoint`: Optional. OTLP endpoint for the customer's own observability backend. Leave empty (default) to disable the customer exporter; the Espresso pipeline still runs. * `customer_protocol`: Optional. `grpc` (renders the `otlp/customer` exporter) or `http` (renders `otlphttp/customer`). Default: `grpc`. * `customer_signals`: Optional. Signals to mirror to the customer exporter. Any subset of `metrics`, `logs`. Default: all three. * `customer_auth_secret_name`: Optional. Existing Kubernetes Secret in the proxy namespace whose value is mounted as `CUSTOMER_OTLP_AUTH` and sent as the customer exporter's `Authorization` header. Leave empty for unauthenticated endpoints. * `customer_auth_secret_key`: Optional. Key within `customer_auth_secret_name`. Default: `authorization`. * `customer_tls_insecure`: Optional. Disable TLS verification on the customer exporter. Default: `false`. Example — also mirror metrics (not logs) to the customer's own OTLP backend with bearer-token auth: ```hcl proxy_config = { repository = ".azurecr.io/proxy" image = "0.1-dev-..." proxy_host = "customer.example.com" otel_collector = { customer_endpoint = "https://otlp.observability.customer.example.com:4317" customer_protocol = "grpc" customer_signals = ["metrics", "logs"] customer_auth_secret_name = "customer-otlp-auth" } } ``` The `customer-otlp-auth` Secret must exist in the `proxy` namespace and contain an `authorization` key whose value is the full header (e.g. `Bearer eyJ...`). For the full list of metrics, spans, and resource attributes the proxy emits — useful for building dashboards and alerts against the customer exporter — see [Proxy telemetry reference](/snowflake-optimizer/proxy-onboarding/proxy-telemetry-reference). ### `ingress_config` * `enable_ingress`: Optional. Enables the nginx Ingress. Default: `true`. * `ingress_host`: Required when `enable_ingress = true`. Hostname (or wildcard, e.g. `*.example.com`) the Ingress serves. * `letsencrypt_email`: Optional. When set, installs cert-manager and a Let's Encrypt `ClusterIssuer`, and cert-manager auto-issues/renews a `kubernetes.io/tls` secret for the Ingress. Mutually exclusive with `tls_secret_name`. Exactly one must be provided when `enable_ingress = true`. * `use_letsencrypt_staging`: Optional. Default: `false`. Switches the issuer to Let's Encrypt staging (useful while iterating to avoid hitting prod rate limits). * `tls_secret_name`: Optional. Bring-your-own `kubernetes.io/tls` secret name in the proxy namespace (e.g. synced from a Key Vault cert via the Secrets Store CSI driver). Mutually exclusive with `letsencrypt_email`. * `front_door`: Optional. Azure Front Door fronting configuration: * `enabled`: Optional. Default: `false`. * `sku_name`: Optional. `Standard_AzureFrontDoor` or `Premium_AzureFrontDoor`. Default: `Standard_AzureFrontDoor`. Premium adds WAF and Private Link to origin. ### `dns_config` * `create_record`: Optional. Creates an Azure DNS A record pointing at the ingress public IP. Default: `false`. * `zone_name`: Required when `create_record = true`. Apex of the existing Azure DNS zone (e.g. `customer.example.com`). * `zone_resource_group_name`: Required when `create_record = true`. Resource group of the DNS zone. * `record_name`: Optional. Falls back to `ingress_config.ingress_host` when omitted. * `ttl`: Optional. Default: `300`. ### `letsencrypt_dns01_azure_dns` When set, cert-manager uses DNS-01 (which supports wildcard certs) instead of HTTP-01. Required when `ingress_config.ingress_host` is a wildcard, since Let's Encrypt only issues wildcards via DNS-01. The module provisions a user-assigned managed identity, federates it to cert-manager's controller service account, and grants it `DNS Zone Contributor` on the named zone — no static credentials are needed. * `zone_name`: Required. * `zone_resource_group_name`: Required. ### `autoscaling_config` * `min_replicas`: Optional. Default: `2`. * `max_replicas`: Optional. Default: `10`. Must be ≥ `min_replicas`. * `target_cpu_utilization`: Optional. Default: `70`. Must be between 1 and 100. ## Secret modes * `BYO_K8S_SECRET` (default): Proxy reads from an existing Kubernetes secret (`api_key_secret_name`) in the proxy namespace using fixed key `ESPRESSO_AI_API_KEY`. * `MANAGED_AZURE_KEY_VAULT`: Module provisions an Azure Key Vault, writes the API key as a secret, federates a user-assigned managed identity to the External Secrets Operator service account, installs ESO, and creates an `ExternalSecret` that syncs the Key Vault secret into a `kubernetes.io/tls`-style Kubernetes secret in the proxy namespace. ## TLS modes * **Let's Encrypt (default path).** Set `ingress_config.letsencrypt_email`. cert-manager runs an HTTP-01 challenge through the nginx Ingress and writes a `proxy-tls` secret in the proxy namespace. Renewals are automatic. * **Let's Encrypt with DNS-01.** Add `letsencrypt_dns01_azure_dns` to switch the solver to Azure DNS. Required for wildcard hosts. The module wires up a federated managed identity scoped to `DNS Zone Contributor` on the target zone — no service-principal credentials needed. * **Bring-your-own TLS secret.** Set `ingress_config.tls_secret_name` (and pre-create that secret in the proxy namespace) — useful when an existing process syncs a Key Vault cert via the Secrets Store CSI driver, or when cert-manager is managed outside this module. * **Front Door fronting.** When `ingress_config.front_door.enabled = true`, AFD terminates TLS for end clients with a DigiCert-issued managed cert (which has working OCSP). The LE/BYO cert on the AKS LB then only secures the AFD-to-origin hop. Use this when the client enforces strict OCSP (e.g. Snowflake's connector). ## Outputs The module exports: * `resource_group_name` * `vnet_id` * `node_subnet_id` * `aks_cluster_name` * `aks_node_resource_group` * `aks_oidc_issuer_url` * `aks_kubelet_identity_object_id` — exposed for advanced cases (e.g. granting `AcrPull` on a private registry of your own). Not needed for the standard flow, which uses Espresso AI's ACR. * `proxy_namespace` * `proxy_service_name` * `proxy_ingress_public_ip` * `proxy_hpa_name` * `proxy_dns_fqdn` * `proxy_api_key_key_vault_name` — only set when `MANAGED_AZURE_KEY_VAULT` is enabled. * `front_door_endpoint_hostname` — point a CNAME from your custom domain at this. Only set when AFD fronting is enabled. * `front_door_custom_domain_validation_token` — publish at `_dnsauth.` so AFD will issue the managed cert. Only set when AFD fronting is enabled. * `front_door_route_id`, `front_door_custom_domain_id`, `front_door_custom_domain_association_id` — AFD resource IDs, only set when AFD fronting is enabled. ### Customer-side DNS work after AFD provisioning When `ingress_config.front_door.enabled = true`, after `terraform apply` finishes, publish: 1. `CNAME → ` 2. `TXT _dnsauth. → ` Both values are exposed as outputs above. ## How to deploy Deployment typically takes around 20-30 minutes (longer when Azure Front Door is enabled and waiting on managed-cert validation). ```bash terraform init terraform plan terraform apply ``` ACR pull is handled on the Espresso AI side — once your tenant ID is registered in the dashboard, our onboarding workflow grants your AKS cluster pull access on the `proxy` repository in Espresso AI's ACR. No `az role assignment` is needed in your subscription. ## Best practices * Manage sensitive variables (e.g. `proxy_api_key_value`) via environment variables or `.tfvars` files excluded from source control. * Keep `aks_config.api_server_authorized_ranges` tight when running a public API server, or set `enable_private_cluster = true` and reach the cluster via a peered network/jumpbox. * For wildcard ingress hosts, always pair `letsencrypt_email` with `letsencrypt_dns01_azure_dns` — HTTP-01 cannot validate a wildcard. * When fronting clients with strict OCSP behavior (Snowflake's connector, in particular), enable `ingress_config.front_door` so end clients see AFD's DigiCert cert instead of the LE cert on the AKS LB. # Self-hosted Helm chart deployment Use this Helm chart to deploy Espresso AI's Proxy Service into an existing Kubernetes cluster. * **`aws`** — EKS (or self-managed Kubernetes on AWS) fronted by the AWS Load Balancer Controller (ALB). * **`azure`** — AKS (or self-managed Kubernetes on Azure) fronted by the Application Gateway Ingress Controller (AGIC). * **`generic`** — any Kubernetes cluster with an ingress controller you manage yourself (NGINX, Traefik, HAProxy, Contour, etc.), or no ingress at all. The non-ingress parts of the chart (Deployment, Service, HPA, ServiceAccount) are identical across all three providers. ## Prerequisites Common to all deployments: * Kubernetes cluster access (`kubectl` context points to the target cluster). * An existing Kubernetes Secret containing `ESPRESSO_AI_API_KEY`. * A container image for the proxy that is reachable from the cluster's nodes. * In the [Espresso AI dashboard](https://dashboard.espressocomputing.com/), go to `Proxy Onboarding` and: * Provide the information needed for image access (see per-provider notes below). * Copy your customer name. * If running on AWS, copy Espresso AI's AWS Account ID. This is needed for the ECR url. * If running on Azure, copy Espresso AI's Azure Account ID. This is needed for the ACR url. * Generate an API key for Espresso API authentication. Per-provider additions: * **AWS** — In the dashboard, enter your AWS account ID so Espresso AI can grant ECR access for the Proxy image. If you plan to expose the proxy via ingress, install the AWS Load Balancer Controller and have an ACM certificate ARN ready. * **Azure** — Enter your Azure Subscription ID so we can grant ACR access for the Proxy image. We will generate a username and password for you to be able to pull the image from our ACR. The chart's `azure` ingress provider is specifically for the Application Gateway Ingress Controller (AGIC); install AGIC on your AKS cluster if you plan to use it, and have your Application Gateway SSL certificate name ready if terminating TLS at the gateway. If you front AKS with Azure Front Door (AFD) over a different in-cluster ingress controller (e.g., NGINX), use the `generic` provider instead and point your AFD origin at that ingress — the chart's `ingress.healthcheck.enabled` option emits a hostless `/healthcheck` rule that AFD probes can hit, regardless of provider. * **Generic** — Contact Espresso AI for image distribution details. Have the ingress controller of your choice installed and the corresponding `ingressClassName` available. ## Required values These are required regardless of provider: * `image.repository` * `image.tag` * `customer` (non-empty) * `env.PROXY_HOST` (non-empty) — your base domain (e.g. `example.com`), not a full hostname or URL * `apiKeySecret.name` (must reference an existing Kubernetes Secret with key `ESPRESSO_AI_API_KEY`) If `ingress.enabled: true`, also set: * `ingress.provider` (`generic`, `aws`, or `azure`) * `ingress.host` (the hostname clients will use) * For `aws`: `ingress.aws.certificateArn` (recommended) * For `azure`: `ingress.azure.appgwSslCertificate` (when terminating TLS at the gateway) or `ingress.tls` (when terminating TLS via a Kubernetes Secret) * For `generic`: `ingress.className` and, if using TLS, `ingress.tls` The API key secret key is fixed to `ESPRESSO_AI_API_KEY` and is not configurable. ## How to deploy Add/update the chart repository: ```bash helm repo add espresso-ai-proxy-chart https://espressocomputing.github.io/espresso-ai-proxy-chart helm repo update ``` Install/upgrade with your `values.yaml`: ```bash helm upgrade --install proxy espresso-ai-proxy-chart/proxy \ --namespace proxy \ --create-namespace \ --version 0.4.0 \ -f values.yaml ``` Create the API key secret (example): ```bash kubectl -n proxy create secret generic espresso-ai \ --from-literal=ESPRESSO_AI_API_KEY='' ``` ## Example values per provider ### AWS (EKS + ALB) ```yaml customer: "value from Dashboard" image: repository: .dkr.ecr.us-east-1.amazonaws.com/proxy tag: "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" env: PROXY_HOST: customer.example.com apiKeySecret: name: espresso-api service: type: ClusterIP port: 5050 ingress: enabled: true provider: aws host: proxy.customer.example.com # className defaults to "alb" when provider is aws and className is unset. aws: certificateArn: arn:aws:acm:us-east-1:123456789012:certificate/11111111-2222-3333-4444-555555555555 scheme: internet-facing targetType: ip listenPorts: '[{"HTTPS":443}]' sslRedirect: "443" healthcheckPath: /healthcheck annotations: {} autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70 ``` ### Azure (AKS + Application Gateway Ingress Controller) ```yaml customer: "value from Dashboard" image: repository: .azurecr.io/espresso/proxy tag: "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" env: PROXY_HOST: customer.example.com apiKeySecret: name: espresso-api service: type: ClusterIP port: 5050 ingress: enabled: true provider: azure host: proxy.customer.example.com # className defaults to "azure/application-gateway" when provider is azure and className is unset. azure: sslRedirect: true healthProbePath: /healthcheck # Name of an SSL certificate already uploaded to your Application Gateway. # Use either appgwSslCertificate (TLS at the gateway) or ingress.tls (TLS via a K8s Secret), not both. appgwSslCertificate: proxy-cert annotations: {} # Optional: extra hostless /healthcheck rule for probes that don't send the Host header # (e.g., Azure Front Door health probes). healthcheck: enabled: true path: /healthcheck pathType: Prefix autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70 ``` If you prefer to terminate TLS in the cluster instead of at the Application Gateway, omit `ingress.azure.appgwSslCertificate` and use the `ingress.tls` shorthand (see below). **Azure Front Door (AFD).** The example above assumes AGIC is the edge. If your edge is AFD over an in-cluster ingress controller (e.g., NGINX on AKS), use `provider: generic` with your controller's `className` instead — the AGIC-specific annotations don't apply. The `ingress.healthcheck` block shown above is still useful: AFD origin health probes do not forward the application Host header, so the hostless `/healthcheck` rule lets them succeed against any provider. ### Generic Kubernetes (any ingress controller) This example uses an NGINX ingress controller and TLS via a Kubernetes Secret. ```yaml customer: "value from Dashboard" image: repository: .dkr.ecr.us-east-1.amazonaws.com/proxy tag: "0.1-dev-c6cb3f5e933cc1d6871195b9d4ffcfea149d4321f1bdf96a8352c112740f32f3" env: PROXY_HOST: customer.example.com apiKeySecret: name: espresso-api service: type: ClusterIP port: 5050 ingress: enabled: true provider: generic className: nginx host: proxy.customer.example.com path: / pathType: Prefix # Shorthand: chart fills hosts from ingress.host if you omit it. tls: secretName: proxy-tls # Or pass through any annotations your controller needs: annotations: nginx.ingress.kubernetes.io/proxy-body-size: "10m" autoscaling: enabled: true minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 70 ``` If you don't need an ingress at all (for example, exposing the Service via a `LoadBalancer` type or accessing it from inside the cluster), set `ingress.enabled: false` and `service.type` to whatever fits your environment. ## Core configuration ### Image | Field | Description | Required | Default | | ------------------ | ----------------------------------------- | -------- | -------------- | | `image.repository` | Container image repository for the proxy. | Yes | None | | `image.tag` | Container image tag. | Yes | None | | `image.pullPolicy` | Kubernetes image pull policy. | No | `IfNotPresent` | ### Environment | Field | Description | Required | Default | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------- | | `customer` | Customer identifier used by the proxy. | Yes | None | | `env.PROXY_HOST` | Your base domain (e.g. `example.com`), injected as `PROXY_HOST`. Use the registrable base domain only — not a full hostname (`proxy.example.com`), scheme, or port. | Yes | None | | `apiUrl` | Base API URL used to derive runtime `API_URL` (`/` unless overridden). | No | `https://api.espressocomputing.com:25831` | | `env.API_URL` | Optional full override for `API_URL`. | No | `/` | | `env.OTEL_EXPORTER_OTLP_ENDPOINT` | Optional telemetry OTLP endpoint override. Defaults to `http://localhost:4318` when `otelCollector.enabled: true` (so the proxy hits the in-pod sidecar) and to `https://metrics.espressocomputing.com:443` otherwise. | No | See description | | `env.EXCLUDE_QUERY_TEXT` | Whether to exclude query text on requests to Espresso AI's API. *Note: enabling this will limit supported functionality.* | No | `false` | | `extraEnv` | Raw list of Kubernetes env entries injected into the proxy container, rendered after the chart-managed env vars. Use it for values the `env` map can't express — anything that needs `valueFrom` (`fieldRef`, `secretKeyRef`, `configMapKeyRef`). | No | `[]` | Any other key/value pairs you put under `env` are passed through to the container as environment variables, except for the chart-managed names listed above and `ESPRESSO_AI_API_KEY`. #### `env` vs `extraEnv` `env` is a simple `name: value` **map** for literal values, and is the right place for almost everything. `extraEnv` is the escape hatch for env vars whose value comes from a non-literal source via `valueFrom`: ```yaml extraEnv: - name: NODE_IP valueFrom: fieldRef: fieldPath: status.hostIP ``` `extraEnv` entries are rendered **after** the chart's managed env vars. Kubernetes `$(VAR)` substitution only resolves variables defined earlier in the same container, so a chart-managed env var cannot reference an `extraEnv` var via `$(VAR)`. If one `extraEnv` var must reference another via `$(VAR)`, define both in `extraEnv` with the source listed before the consumer. Each entry needs its `name` and `value` (or `valueFrom`) on the **same list item**. Splitting them across two items — `- name: X` then `- value: Y` — produces an entry with no name and fails admission with `spec.template.spec.containers[N].env[M].name: Required value`. ### API key secret | Field | Description | Required | Default | | --------------------- | -------------------------------------------------------------- | -------------------- | ------- | | `apiKeySecret.name` | Existing Kubernetes Secret name that stores the proxy API key. | Yes | None | | `ESPRESSO_AI_API_KEY` | Fixed key the chart reads from the Kubernetes Secret. | Yes (in Secret data) | Fixed | ### Service | Field | Description | Required | Default | | --------------------- | ---------------------------------------------------------------------------------- | -------- | ----------- | | `service.type` | Service type (`ClusterIP`, `NodePort`, `LoadBalancer`). | No | `ClusterIP` | | `service.port` | Service port. | No | `5050` | | `service.annotations` | Extra annotations on the Service (e.g., cloud LB hints when using `LoadBalancer`). | No | `{}` | ### Ingress (common fields) These fields apply to every provider when `ingress.enabled: true`. | Field | Description | Required | Default | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | ------------------------ | | `ingress.enabled` | Whether to render an Ingress resource. | No | `false` | | `ingress.provider` | Ingress flavor: `generic`, `aws`, or `azure`. | Yes (if enabled) | `generic` | | `ingress.className` | `ingressClassName` on the Ingress. Defaults: `alb` for `aws`, `azure/application-gateway` for `azure`, empty for `generic`. | Conditional | None / provider-specific | | `ingress.host` | Hostname rule for the ingress. | No (recommended) | None | | `ingress.path` | Path for the primary rule. | No | `/` | | `ingress.pathType` | `pathType` for the primary rule. | No | `Prefix` | | `ingress.annotations` | Extra annotations merged onto the Ingress (after provider-specific annotations). | No | `{}` | | `ingress.tls` | Either a standard ingress TLS list, or a shorthand `{secretName, hosts}`. When `hosts` is omitted from the shorthand, `ingress.host` is used. | No | `null` | | `ingress.healthcheck.enabled` | Render an extra hostless rule (no `host`) routing `/healthcheck` to the Service. Useful for probes that don't send the application's Host header. | No | `false` | | `ingress.healthcheck.path` | Path used by the hostless healthcheck rule. | No | `/healthcheck` | | `ingress.healthcheck.pathType` | `pathType` used by the hostless healthcheck rule. | No | `Prefix` | If `ingress.provider` is set to anything other than `generic`, `aws`, or `azure`, the chart fails the install with an explanatory error. ### Ingress — AWS (ALB) When `ingress.provider: aws`, the chart emits AWS Load Balancer Controller annotations from `ingress.aws.*`. The previous `ingress.alb.*` block is still read as a backward-compatible alias if both are present, with `ingress.aws.*` winning on conflicts. | Field | Description | Required | Default | | ----------------------------- | --------------------------------------------- | ----------- | ----------------- | | `ingress.aws.certificateArn` | ACM certificate ARN for HTTPS listener. | Recommended | None | | `ingress.aws.scheme` | ALB scheme (`internet-facing` or `internal`). | No | `internet-facing` | | `ingress.aws.targetType` | ALB target type. | No | `ip` | | `ingress.aws.listenPorts` | ALB listen ports JSON. | No | `[{"HTTPS":443}]` | | `ingress.aws.sslRedirect` | ALB SSL redirect port. | No | `"443"` | | `ingress.aws.healthcheckPath` | ALB target group health check path. | No | `/healthcheck` | | `ingress.aws.annotations` | Extra ALB-specific annotations. | No | `{}` | ### Ingress — Azure (Application Gateway) When `ingress.provider: azure`, the chart emits AGIC annotations from `ingress.azure.*`. | Field | Description | Required | Default | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | -------------- | | `ingress.azure.healthProbePath` | Custom health probe path used by Application Gateway. Renders as `appgw.ingress.kubernetes.io/health-probe-path`. | No | `/healthcheck` | | `ingress.azure.sslRedirect` | When `true` (default), renders `appgw.ingress.kubernetes.io/ssl-redirect: "true"`. Set to `false` to disable. | No | `true` | | `ingress.azure.appgwSslCertificate` | Name of an SSL certificate already uploaded to the Application Gateway. Renders as `appgw.ingress.kubernetes.io/appgw-ssl-certificate`. Use this *or* `ingress.tls`, not both. | No | None | | `ingress.azure.annotations` | Extra AGIC annotations. | No | `{}` | ### Ingress — Generic When `ingress.provider: generic`, no cloud-specific annotations are added. Set `ingress.className` to your controller's class (e.g., `nginx`, `traefik`) and pass any controller-specific configuration through `ingress.annotations`. TLS works the same way as a stock Kubernetes Ingress, including the shorthand: ```yaml ingress: tls: secretName: proxy-tls # hosts: [proxy.customer.example.com] # optional; defaults to ingress.host ``` ### Autoscaling | Field | Description | Required | Default | | -------------------------------------------- | ---------------------------------------------------- | -------- | ------- | | `autoscaling.enabled` | Whether to render the HPA. | No | `true` | | `replicaCount` | Initial deployment replica count before HPA adjusts. | No | `2` | | `autoscaling.minReplicas` | Minimum replicas for HPA. | No | `2` | | `autoscaling.maxReplicas` | Maximum replicas for HPA. | No | `10` | | `autoscaling.targetCPUUtilizationPercentage` | CPU utilization target for HPA scaling decisions. | No | `70` | ### Probes Both readiness and liveness probes are enabled by default and hit `/healthcheck` on the container port. They can be tuned or disabled under `probes.readiness` and `probes.liveness`. ### Telemetry collector When `otelCollector.enabled: true`, the chart adds an OpenTelemetry Collector container to the proxy pod and renders a ConfigMap with a pipeline configuration. The proxy emits OTLP/HTTP to `localhost:4318`, and the sidecar fans the traffic out to two exporters: * The **Espresso exporter** always sends to `otelCollector.espresso.endpoint` (default `https://metrics.espressocomputing.com:443`) for every pipeline. This is how Espresso AI receives your proxy's telemetry, and it is independent of the customer endpoint — leaving the customer exporter unset does not affect it. * The optional **customer exporter** sends to `otelCollector.customer.endpoint` for the signals listed in `otelCollector.customer.signals`. Leave the endpoint empty to disable it entirely (the Espresso pipeline still runs). | Field | Description | Required | Default | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------- | | `otelCollector.enabled` | Whether to deploy the OTEL Collector sidecar and ConfigMap. | No | `true` | | `otelCollector.image.repository` | Collector image repository. | No | `otel/opentelemetry-collector-contrib` | | `otelCollector.image.tag` | Collector image tag. | No | `0.152.0` | | `otelCollector.image.pullPolicy` | Kubernetes image pull policy for the collector. | No | `IfNotPresent` | | `otelCollector.resources` | Standard `requests` / `limits` block for the collector container. | No | `50m` / `128Mi` requests, `200m` / `256Mi` limits | | `otelCollector.env` | Raw list of Kubernetes env entries injected into the collector sidecar container. Reference them inside the collector config with the collector's own `${env:NAME}` substitution (see below). | No | `[]` | | `otelCollector.espresso.endpoint` | OTLP endpoint for Espresso AI's backend. The Espresso exporter always sends here, for every pipeline. | No | `https://metrics.espressocomputing.com:443` | | `otelCollector.customer.endpoint` | OTLP endpoint for the customer's own observability backend. Leave empty to disable the customer exporter entirely (the Espresso pipeline still runs). | No | `""` | | `otelCollector.customer.protocol` | Wire protocol for the customer exporter. `grpc` renders `otlp/customer`; `http` renders `otlphttp/customer`. | No | `grpc` | | `otelCollector.customer.signals` | Signals to mirror to the customer exporter. Any subset of `metrics`, `logs`. Signals not listed here go only to Espresso. | No | `[metrics, logs]` | | `otelCollector.customer.authSecret.name` | Kubernetes Secret holding the value for the customer endpoint's `Authorization` header. Leave empty for unauthenticated endpoints. | No | `""` | | `otelCollector.customer.authSecret.key` | Key within `customer.authSecret.name` whose value is mounted as `CUSTOMER_OTLP_AUTH`. | No | `authorization` | | `otelCollector.customer.tls.insecure` | Disable TLS verification on the customer exporter. | No | `false` | Example — also mirror metrics (not logs) to your own OTLP backend with bearer-token auth: ```yaml otelCollector: customer: endpoint: https://otlp.observability.customer.example.com:4317 protocol: grpc signals: - metrics authSecret: name: customer-otlp-auth key: authorization ``` Where `customer-otlp-auth` is a Kubernetes Secret in the proxy namespace whose `authorization` key contains the full header value (e.g. `Bearer eyJ...`). #### Per-node customer collector If you run a customer collector on each node (a DaemonSet, say) and want the sidecar to ship to the collector on its own node, the customer endpoint has to resolve to a per-node address. Use `otelCollector.env` to surface the node's IP, then reference it from `otelCollector.customer.endpoint`: ```yaml otelCollector: enabled: true env: - name: NODE_IP valueFrom: fieldRef: fieldPath: status.hostIP customer: endpoint: "http://${env:NODE_IP}:4318" protocol: http # use grpc + port 4317 for an OTLP/gRPC collector signals: [metrics, logs] tls: insecure: true # plaintext OTLP/HTTP on the node ``` Espresso keeps receiving telemetry directly through the Espresso exporter; the customer also gets a copy on each node. **Two substitution layers.** Note which one resolves `NODE_IP` here. There are two distinct layers in play: * `$(VAR)` is expanded by the **kubelet** in a container's `env` and `args`. * `${env:NAME}` is expanded by the **OpenTelemetry Collector** when it reads its config. The customer endpoint lives in the collector's ConfigMap, which the kubelet never touches, so a per-node value there must use `${env:NAME}` — the collector reads `NODE_IP` from its own process environment. Unlike the proxy `extraEnv` `$(VAR)` case above, ordering within `otelCollector.env` does not matter, because the collector just reads the environment it was given. For the full list of metrics, spans, and resource attributes the proxy emits — useful for building dashboards and alerts against the customer exporter — see [Proxy telemetry reference](/snowflake-optimizer/proxy-onboarding/proxy-telemetry-reference). ### Resources, scheduling, service account `resources`, `nodeSelector`, `tolerations`, `affinity`, and `serviceAccount` follow standard Helm-chart conventions; see `values.yaml` for the defaults. ## Managed secret note This chart does not create AWS Secrets Manager, Azure Key Vault, or External Secrets resources by itself. For managed secret sync from a cloud secrets manager, provision your sync resource (e.g., External Secrets Operator with an AWS Secrets Manager or Azure Key Vault `SecretStore`) separately and set: * `apiKeySecret.name` to the Kubernetes Secret generated by the sync (key must be `ESPRESSO_AI_API_KEY`). ## Validation checklist * Pods are running: `kubectl -n proxy get pods` * Service exists: `kubectl -n proxy get svc` * HPA exists (when `autoscaling.enabled`): `kubectl -n proxy get hpa` * Ingress exists (if enabled): `kubectl -n proxy get ingress` * App health endpoint responds on `/healthcheck` # Telemetry reference The Espresso AI proxy emits OpenTelemetry metrics, and logs over OTLP. When deployed with chart `v0.3.0` / proxy-tf `v0.4.0` or later, that traffic flows through the in-pod OTEL Collector sidecar and can be mirrored to a customer-owned OTLP backend (see [`otelCollector`](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-helm-deployment#telemetry-collector) for Helm and [`otel_collector`](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-terraform-deployment-aws#otel_collector) for Terraform). This page lists what to expect on that customer-side stream so you can build dashboards and alerts against it. ## Resource attributes The following resource attributes are attached to every signal emitted by the proxy: | Attribute | Value | Source | | ------------------------ | ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `service.name` | `proxy` | Baked into the proxy image. | | `deployment.environment` | `prod` | Derived from the `ENV` env var, which the chart and Terraform module set to `PROD`. | | `customer` | The customer identifier you supplied during onboarding. | Derived from the `CUSTOMER` env var, which the chart and Terraform module wire from the `customer` value. | Standard OpenTelemetry SDK resource attributes (`telemetry.sdk.name`, `telemetry.sdk.language=python`, `telemetry.sdk.version`, `process.runtime.*`, host attributes) are also present. ## Metrics The proxy emits two custom metrics on top of any standard runtime metrics produced by the OpenTelemetry Python SDK. | Metric | Instrument | Unit | Description | Attributes | | ------------------------------ | ---------------- | -------------- | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | `proxy_httpx_pool_connections` | Observable gauge | `{connection}` | Snapshot of the httpx connection pool used to talk to Snowflake, sliced by connection state. | `pool` (logical pool name), `state` ∈ `open`, `active`, `idle` | | `proxy_httpx_pool_requests` | Observable gauge | `{request}` | Snapshot of the httpx pool's request queue. Emitted only when the pool exposes its request queue. | `pool`, `state` = `queued` | | `proxy_endpoint_responses` | counter | `{response}` | Status Code counter for the Proxy service's responses. | `status_code` | Use these to alert on pool saturation (`state=queued` rising, or `state=active` approaching the pool's configured limit). ## Logs The proxy ships Python `logging` records as OTLP log records. Each log record carries the resource attributes above, the standard `severity_text` / `severity_number` and the originating logger name. ## How to subscribe To receive these signals at your own OTLP backend, set the customer exporter on the OTEL Collector sidecar: * **Helm** — set `otelCollector.customer.endpoint` (and optionally `customer.protocol`, `customer.signals`, `customer.authSecret`, `customer.tls.insecure`). See [Telemetry collector](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-helm-deployment#telemetry-collector). * **Terraform** — set `proxy_config.otel_collector.customer_endpoint` (and the matching `customer_*` fields). See the `otel_collector` reference on the [AWS](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-terraform-deployment-aws#otel_collector) and [Azure](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-terraform-deployment-azure#otel_collector) pages. By default the customer exporter mirrors both signals (`metrics`, `logs`); narrow with `customer.signals` / `customer_signals` if you only want a subset. To point each sidecar at a customer collector running on its own node, inject the node IP with `otelCollector.env` and reference it from the endpoint via the collector's `${env:NAME}` substitution. See [Per-node customer collector](/snowflake-optimizer/proxy-onboarding/proxy-onboarding-helm-deployment#telemetry-collector). # Snowflake Architecture Espresso's architecture is designed to maximize availability and to protect the security of your data. Our system is trained on metadata and we only use metadata in production. All metadata is encrypted in transit and at rest. Your data passes through our Snowflake proxy for proxy-enabled features (the Scheduling Agent and the Query Agent). Data is encrypted in transit and is never accessed, logged, or stored by our system. Enterprise customers can self-host the proxy to prevent data from being transmitted outside of their VPC. ## Network Connectivity We support TLS encryption, PrivateLink on AWS and Azure, and Private Service Connect on GCP. (This applies to any connection in blue on our architecture diagram.) If you use a Snowflake allowlist, please allow the following IPs: ``` 18.233.13.51 34.195.242.31 34.231.116.52 34.231.212.71 34.234.123.175 35.169.148.94 52.87.110.223 54.161.160.239 ``` ## Warehouse Agent Espresso's warehouse agent connects directly to your Snowflake account using a Snowflake service user.
## Snowflake Proxy: Standard Deployment Our Scheduling Agent and Query Agent run over a proxy. In our standard deployment users connect directly to the proxy, which forwards requests to Snowflake and returns results to the user. Customer data passes in transit through the proxy but is never inspected or stored.
## Snowflake Proxy: Self-Hosted Proxy Customers who do not want their data to leave their environment, even in transit, can self-host the proxy.
## Self-Hosted Proxy: Query-Text-Less Operation For deployments that require stronger privacy guarantees, we support a query-text-less proxy mode via `EXCLUDE_QUERY_TEXT=true`. In this mode, the proxy only sends routing and control-plane metadata to our backend. It degrades optimization and routing signal quality by replacing non-`USE` queries with a query hash, while still sending `USE` statements as raw SQL when needed for tracking accuracy. In this mode, we currently send: * Login metadata for account/session initialization: * Warehouse * Username * Database and schema names * Hashed session tokens for session tracking * Query context metadata for query routing: * Query hash and hash version * Original warehouse (when available) * Database and schema from request context * Routing SQL statements (e.g., `USE ...`) needed for routing and execution continuity We may update this list to collect more metadata information, but guarantee to not collect general query text in this mode. ## Snowflake Proxy: Self-Hosted Deployment Enterprise customers can self-host Espresso's entire architecture.
# Snowflake Autoscaler Espresso AI uses ML models to optimize Snowflake workloads in real time. ## How it works 1. Espresso trains custom models on your Snowflake metadata. This usually takes between 15 minutes and 48 hours, depending on the number of queries and warehouses in your account. 2. Once the models are trained, you receive a savings estimate. 3. You also get access to the Espresso dashboard, where you can turn optimization on across your account or for specific warehouses. ## Create a Snowflake service user and role Run the following commands in Snowflake to set up access for Espresso. We only access metadata. You can find the list of metadata we read on [Snowflake Metadata](/snowflake-optimizer/snowflake-metadata). ```sql -- Create the Espresso AI user with our public key CREATE USER espresso_ai_user TYPE=SERVICE RSA_PUBLIC_KEY='MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6GfMQyT/1ZQS1wwTeF3Q lbHJyBSuIio+UlnZixffvo9UwP/ild6R4AqOEma39Ty1zuLibbyNSgjTYqNXv7QNiEgR50SvEo27N6EJ I1EOnJPREzi060E64eXMrc1mwPrERxtEtNXgJBUs3Y2aKsGKoo900jkjK08CrMNBM1uzyXhLBS5a4sND Slef4JjyZCBl4iHTqmeqD0xZJCC2/Rlr40UGPq+Ae/zXyyDyQkWE69ytZQRjPXvv8b2x6C9JtKYqtklJ ljzFy74eQZV4m9hxtH+r1Z32zyhjNGhYaRF/0yD3gNWg135cexfyd5M8PKPt1Km4VAV6oPr8QYEfvS50 0QIDAQAB'; CREATE ROLE IF NOT EXISTS ESPRESSO_AI_USER COMMENT = 'Used by Espresso AI'; GRANT ROLE ESPRESSO_AI_USER TO USER ESPRESSO_AI_USER; -- Allow Espresso AI to query Snowflake metadata GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE ESPRESSO_AI_USER; -- Allow Espresso AI to securely export data ALTER USER ESPRESSO_AI_USER SET PREVENT_UNLOAD_TO_INLINE_URL = false; -- Allow Espresso AI to monitor warehouses GRANT MONITOR USAGE ON ACCOUNT TO ROLE ESPRESSO_AI_USER; -- Give Espresso AI the ability to modify warehouses GRANT MANAGE WAREHOUSES ON ACCOUNT TO ROLE ESPRESSO_AI_USER; -- Set up a warehouse for Espresso AI queries to run on CREATE WAREHOUSE IF NOT EXISTS ESPRESSO_AI_WH WAREHOUSE_SIZE = XSMALL AUTO_SUSPEND = 60 INITIALLY_SUSPENDED = TRUE COMMENT = 'Used by Espresso AI'; GRANT MONITOR, OPERATE, USAGE, MODIFY ON WAREHOUSE ESPRESSO_AI_WH TO ROLE ESPRESSO_AI_USER; ``` ## Share your account details After you run the Snowflake commands, fill out the form below with your account information. Your account hostname is the URL you use to sign in to Snowflake, for example `https://.snowflakecomputing.com/`. {% @espresso-web-form/web-form submitUrl="" submitMethod="POST" submitFormat="urlencoded" submitButtonLabel="Get started" successMessage="Thanks. You should receive a savings estimate and dashboard access shortly." errorMessage="Something went wrong while submitting the form." fieldsJson="\[{"name":"Email","label":"Email","type":"email","required":true,"placeholder":"Email\*"},{"name":"Snowflake-Account-Hostname","label":"Snowflake account hostname","type":"text","required":true,"placeholder":"Snowflake Account Hostname\*"},{"name":"Role-name","label":"Role name","type":"text","required":true,"placeholder":"Role name (e.g. ESPRESSO\_USER)\*"},{"name":"utm\_source","type":"hidden","defaultValue":""},{"name":"utm\_medium","type":"hidden","defaultValue":""},{"name":"utm\_campaign","type":"hidden","defaultValue":""},{"name":"utm\_term","type":"hidden","defaultValue":""},{"name":"utm\_content","type":"hidden","defaultValue":""},{"name":"gclid","type":"hidden","defaultValue":""},{"name":"li\_fat\_id","type":"hidden","defaultValue":""}]" %} ## Optional: IP allowlist If you use Snowflake network policies, allow these IPs: * `18.233.13.51` * `34.195.242.31` * `34.231.116.52` * `34.231.212.71` * `34.234.123.175` * `35.169.148.94` * `52.87.110.223` * `54.161.160.239` ## Questions? [Book a call](https://espresso.ai/demo) if you have questions or want an NDA in place. # Databricks Optimizer # Terraform provider ## Authentication In the Espresso dashboard, select an existing account, open **Tools → API Keys**, choose **Generate API key → Organization key**, and copy the complete `ok_` secret. It cannot be displayed again. Set it as `ESPRESSO_API_KEY`. ## One account per Databricks workspace An `espresso_account` is the Espresso account boundary. Key the Terraform resources by Databricks workspace ID and give each workspace a permanent Espresso slug: ```hcl variable "databricks_workspaces" { type = map(object({ espresso_slug = string display_name = string workspace_url = string })) } resource "espresso_account" "workspace" { for_each = var.databricks_workspaces slug = each.value.espresso_slug display_name = each.value.display_name product = "databricks" } resource "espresso_databricks_warehouse_agent" "workspace" { for_each = var.databricks_workspaces account = espresso_account.workspace[each.key].slug enabled = false auto_opt_in = false } output "espresso_account_by_workspace_id" { value = { for workspace_id, account in espresso_account.workspace : workspace_id => account.slug } } ``` For example: ```hcl databricks_workspaces = { "1234567890123456" = { espresso_slug = "acme_production" display_name = "Acme Production" workspace_url = "https://1234567890123456.cloud.databricks.com" } "9876543210987654" = { espresso_slug = "acme_staging" display_name = "Acme Staging" workspace_url = "https://9876543210987654.cloud.databricks.com" } } ``` Espresso prepends `databricks_` when a Databricks slug omits it, so these accounts are stored as `databricks_acme_production` and `databricks_acme_staging`. Every global and warehouse setting for workspace `1234567890123456` must use `espresso_account.workspace["1234567890123456"].slug` as its `account`. An account's `display_name` can be updated in place. Its `slug` and `product` are immutable. Removing an account resource from Terraform stops managing it but leaves the account in Espresso. Databricks onboarding must still be run for each account. ## Databricks credentials The credentials resource authenticates with Databricks, verifies access to the configured SQL warehouse, and saves the connection in Espresso: ```hcl resource "espresso_databricks_credentials" "workspace" { for_each = var.databricks_workspaces account = espresso_account.workspace[each.key].slug workspace_url = each.value.workspace_url workspace_id = each.key workspace_name = each.value.display_name client_id = databricks_service_principal.espresso.application_id client_secret = databricks_service_principal_secret.espresso.secret service_principal_id = databricks_service_principal.espresso.id service_principal_name = databricks_service_principal.espresso.display_name warehouse_id = databricks_sql_endpoint.espresso[each.key].id warehouse_name = databricks_sql_endpoint.espresso[each.key].name } ``` `client_secret` is write-only in the Espresso provider and is not retained in that resource's state. The Databricks provider retains the generated service-principal secret in Terraform state, so use encrypted remote state with tightly restricted access. See [Databricks Terraform Onboarding](/databricks-optimizer/databricks-terraform-onboarding) for a complete configuration that creates the Databricks identity, permissions, SQL warehouse, and Espresso credentials. ## Warehouse Agent settings ```hcl locals { shared_workspace_id = "1234567890123456" shared = { min_clusters = 1 max_clusters = 8 } } resource "databricks_sql_endpoint" "shared" { name = "Shared SQL" min_num_clusters = local.shared.min_clusters max_num_clusters = local.shared.max_clusters cluster_size = "Large" warehouse_type = "PRO" lifecycle { ignore_changes = [min_num_clusters, max_num_clusters] } } resource "espresso_databricks_warehouse_agent_warehouse" "shared" { account = espresso_account.workspace[local.shared_workspace_id].slug name = databricks_sql_endpoint.shared.name enabled = true min_clusters = local.shared.min_clusters max_clusters = local.shared.max_clusters } ``` The lifecycle list prevents the Databricks and Espresso providers from fighting over Warehouse Agent settings. Terraform lifecycle values cannot be conditional. To return control safely, first set the Espresso warehouse's `enabled` to `false` and apply, then remove its `ignore_changes` entries and apply again. The Databricks provider then reconciles the warehouse to the configured values. Each Warehouse Agent warehouse configuration is managed as a discrete resource. Its settings fields are optional, so an `account` and `name` can adopt the current values without changing them. Removing a Warehouse Agent resource stops Terraform management without changing the current Espresso settings or the underlying warehouse. # Databricks Savings Estimate Follow these instructions to find out how much we can save you! ### What is a Savings Estimate? Using workload metadata, we simulate your environment and produce an estimate of how much we can save you. The estimate looks like this:
### How do I know the savings are accurate? Our models are continuously calibrated with production Databricks data to ensure our savings numbers are accurate. The best way for you to judge accuracy is to compare our upfront savings estimate to the savings you see in production when we first turn on. We also encourage users to run A/B tests once we've been on for a few months: shut Espresso off for a week and see how your actual spend compares to our savings calculation. ## How do I get an estimate? We need a few things to generate the estimate: query metadata, warehouse metadata, and Databricks usage metadata. The fastest way to share those is to [set up a Databricks service principal for Espresso](https://docs.espresso.ai/~/revisions/WipBY1leQllUvtR0ovja/databricks-optimizer/databricks-sql-onboarding-1). If you'd prefer to share your metadata without setting up an account, you can also [securely share metadata](https://docs.espresso.ai/~/revisions/WipBY1leQllUvtR0ovja/databricks-optimizer/databricks-sql-onboarding) with our Databricks account via OpenSharing (previously Delta Sharing). ## NDA and support Espresso AI is happy to sign an NDA. Contact [savings@espresso.ai](mailto:savings@espress.ai) with your NDA or any questions. # Databricks Metadata Share #### Steps 1. Visit and click on your metastore. 2. If unchecked, check the box saying "Allow Delta Sharing with parties outside your organization." 3. Set your "Organization name" to the name of your company/organization 4. In the first cell of a Databricks notebook, install/upgrade the SDK: `%pip install databricks-sdk --upgrade` 5. When finished, create a second cell and restart the Python kernel: `%restart_python` 6. Copy-paste and run the following in a third cell to securely share your metadata with Espresso's Databricks account. This will allow us to generate savings estimates for your Databricks account. ```python from datetime import UTC, datetime, timedelta from databricks.sdk import WorkspaceClient from databricks.sdk.service import sharing from pyspark.sql import SparkSession TABLES_TO_SHARE = [ ("system", "query", "history", "start_time"), ("system", "compute", "warehouse_events", "event_time"), ("system", "compute", "warehouses", None), ("system", "compute", "clusters", None), ("system", "compute", "node_timeline", "start_time"), ("system", "compute", "node_types", None), ("system", "billing", "list_prices", None), ("system", "billing", "usage", "usage_start_time"), ("system", "access", "audit", "event_time"), ("system", "access", "workspaces_latest", None), ("system", "lakeflow", "job_run_timeline", "period_start_time"), ("system", "lakeflow", "job_task_run_timeline", "period_start_time"), ("system", "serving", "served_entities", None), ("system", "serving", "endpoint_usage", "request_time"), ("system", "information_schema", "metastores", None), ] CENSORED_QUERY_TEXT = "''" REDACTION_GROUPS = { "query_text": { ("system", "query", "history"): [ ("statement_text", CENSORED_QUERY_TEXT), ("error_message", CENSORED_QUERY_TEXT), ], }, "job_names": { ("system", "lakeflow", "job_run_timeline"): [("run_name", "CONCAT('job_', job_id)")], ("system", "lakeflow", "job_task_run_timeline"): [ ("task_key", "CONCAT('task_', SUBSTR(SHA2(task_key, 256), 1, 12))") ], }, "resource_names": { ("system", "compute", "clusters"): [ ("cluster_name", "CONCAT('cluster_', cluster_id)") ], ("system", "compute", "warehouses"): [ ("warehouse_name", "CONCAT('warehouse_', warehouse_id)") ], }, } ENABLED_REDACTIONS = set() def redacted_select(table): computed = {} for group in ENABLED_REDACTIONS: computed.update(dict(REDACTION_GROUPS[group].get(table, []))) if not computed: return "*" excepted = ", ".join(computed) replacements = [f"{expr} as {col}" for col, expr in computed.items()] return ", ".join([f"* EXCEPT ({excepted})", *replacements]) TABLE_FILTERS = { ("system", "access", "audit"): [ "action_name NOT IN ('listHistoryQueries', 'oidcTokenAuthorization', 'tokenLogin', 'workspaceInHouseOAuthClientAuthentication', 'mintOAuthToken', 'getTable', 'generateTemporaryTableCredential', 'authzEval', 'aadTokenLogin', 'metadataAndPermissionsSnapshot', 'getPipeline', 'listCatalogs', 'metadataSnapshot', 'setTaskValue', 'getVolume', 'loadTable', 'getCatalog', 'config', 'getTableById', 'getSchema')", "service_name NOT IN ('syslog', 'capsule8-alerts-dataplane', 'clamAVScanService-dataplane', 'monit')", ], } def is_user_workspace_admin(client): current_user = client.current_user.me() return current_user.groups is not None and any( group.display == "admins" for group in current_user.groups ) def is_user_metastore_admin(client): current_user = client.current_user.me() metastore = client.metastores.summary() if metastore.owner == current_user.user_name: return True return current_user.groups is not None and any( g.display == metastore.owner for g in current_user.groups ) def check_permissions(client): missing = [] if not is_user_workspace_admin(client): missing.append( "WORKSPACE ADMIN required: ask a workspace admin to add you to the 'admins' group." ) if not is_user_metastore_admin(client): missing.append( "METASTORE ADMIN required: visit https://accounts.cloud.databricks.com/data, " "open your metastore's Configuration tab, and set yourself as Metastore Admin." ) if missing: raise PermissionError("Cannot run share_data.py:\n - " + "\n - ".join(missing)) spark = SparkSession.getActiveSession() or SparkSession.builder.getOrCreate() def get_current_catalog_storage_root(client): current_catalog = spark.sql("SELECT current_catalog()").collect()[0][0] return client.catalogs.get(name=current_catalog).storage_root def get_or_create_warehouse(client): for warehouse in client.warehouses.list(): if warehouse.name == "ESPRESSO_AI_WAREHOUSE": return warehouse.id created_warehouse = client.warehouses.create_and_wait( name="ESPRESSO_AI_WAREHOUSE", cluster_size="X-Small", auto_stop_mins=5, enable_serverless_compute=True, min_num_clusters=1, max_num_clusters=1, ) return created_warehouse.id def get_or_create_catalog(client): CATALOG_NAME = "espresso_ai_system_metadata" if CATALOG_NAME in [c.name for c in client.catalogs.list()]: return CATALOG_NAME comment = "System metadata catalog for Espresso AI" try: client.catalogs.create(name=CATALOG_NAME, comment=comment) except Exception as e: if "Metastore storage root URL does not exist" in str(e): if not (storage_root := get_current_catalog_storage_root(client)): raise RuntimeError("No metastorage root or current catalog storage root") client.catalogs.create( name=CATALOG_NAME, comment=comment, storage_root=storage_root ) else: raise return CATALOG_NAME def get_or_create_metadata_tables(client, warehouse_id, catalog_name): cutoff_timestamp = (datetime.now(UTC) - timedelta(days=60)).isoformat() # `information_schema` is auto-created and read-only in every UC catalog, # so we can't mirror into it — write those tables to `metadata` instead. schema_names = list( {("metadata" if s == "information_schema" else s) for _, s, _, _ in TABLES_TO_SHARE} ) existing_schemas = {s.name for s in client.schemas.list(catalog_name=catalog_name)} for schema_name in schema_names: if schema_name not in existing_schemas: client.schemas.create( catalog_name=catalog_name, name=schema_name, comment=f"Mirror of system.{schema_name}", ) for catalog, schema, table_name, timestamp_col in TABLES_TO_SHARE: target_schema = "metadata" if schema == "information_schema" else schema target_table = f"{catalog_name}.{target_schema}.{table_name}" conditions = TABLE_FILTERS.get((catalog, schema, table_name), []).copy() if timestamp_col: conditions.insert(0, f"{timestamp_col} >= '{cutoff_timestamp}'") where = f"WHERE {' AND '.join(conditions)}" if conditions else "" select = redacted_select((catalog, schema, table_name)) sql = f""" CREATE MATERIALIZED VIEW IF NOT EXISTS {target_table} AS SELECT {select} FROM {catalog}.{schema}.{table_name} {where} """ client.statement_execution.execute_statement( warehouse_id=warehouse_id, catalog=catalog_name, schema=target_schema, statement=sql, ) return schema_names def create_share_with_system_catalog_schemas(client, catalog_name, schema_names): SHARE_NAME = "espresso_ai_system_data" existing_shares = {s.name for s in client.shares.list_shares()} if SHARE_NAME in existing_shares: share = client.shares.get(name=SHARE_NAME, include_shared_data=True) else: share = client.shares.create( name=SHARE_NAME, comment="System catalog data for Espresso AI" ) existing = {obj.name for obj in (share.objects or [])} updates = [ sharing.SharedDataObjectUpdate( action=sharing.SharedDataObjectUpdateAction.ADD, data_object=sharing.SharedDataObject( name=f"{catalog_name}.{schema_name}", data_object_type=sharing.SharedDataObjectDataObjectType.SCHEMA, ), ) for schema_name in schema_names if f"{catalog_name}.{schema_name}" not in existing ] if updates: client.shares.update(name=SHARE_NAME, updates=updates) return SHARE_NAME def get_or_create_recipient(client): RECIPIENT_NAME = "espresso_ai" SHARING_IDENTIFIER = "aws:us-west-2:6a0451ec-2d11-48b7-8fde-ffaf14401682" existing_recipients = {r.name: r for r in client.recipients.list()} if RECIPIENT_NAME in existing_recipients: if ( existing_recipients[RECIPIENT_NAME].data_recipient_global_metastore_id == SHARING_IDENTIFIER ): return RECIPIENT_NAME client.recipients.delete(name=RECIPIENT_NAME) client.recipients.create( name=RECIPIENT_NAME, authentication_type=sharing.AuthenticationType.DATABRICKS, data_recipient_global_metastore_id=SHARING_IDENTIFIER, comment="Espresso AI optimizer service", ) return RECIPIENT_NAME def grant_recipient_access(client, share_name, recipient_name): client.shares.update_permissions( name=share_name, changes=[sharing.PermissionsChange(principal=recipient_name, add=["SELECT"])], ) if __name__ == "__main__": client = WorkspaceClient() check_permissions(client) warehouse_id = get_or_create_warehouse(client) catalog_name = get_or_create_catalog(client) schema_names = get_or_create_metadata_tables(client, warehouse_id, catalog_name) share_name = create_share_with_system_catalog_schemas(client, catalog_name, schema_names) recipient_name = get_or_create_recipient(client) grant_recipient_access(client, share_name, recipient_name) print("\n🎉 Delta Sharing setup complete!") print("=" * 50) print(f"Share: {share_name}") print(f"Recipient: {recipient_name}") print(f"Catalog: {catalog_name}") print("=" * 50) ``` ## Questions? [Book a Call](https://espresso.ai/demo) or email . # Databricks SQL Onboarding Espresso AI makes real-time decisions powered by ML models to optimize your Databricks SQL workloads. Please run the following commands in Databricks SQL. This will set up a user account we can use to access the metadata we need for our models as well as optimize your Databricks SQL account's operations. To manage onboarding as code instead, use the [Espresso Databricks Terraform module](/databricks-optimizer/databricks-terraform-onboarding). Note that we **never** access, log, or store any data from Databricks SQL. We only look at metadata. Prerequisites: Make sure you are a Databricks workspace admin, account admin, and metastore admin. ## Steps 1. Open your Databricks workspace, create a Python notebook, and attach it to Serverless compute. 2. In the first cell, install/upgrade the SDK: ```python %pip install databricks-sdk --upgrade ``` 3. When finished, create a second cell and restart the Python kernel: ```python %restart_python ``` 4. In the next cell, copy-paste the code below. When you run this, if you have multiple workspaces, you will be prompted to grant a temporary service principal account admin credentials. This simply lets us grant permissions across workspaces, and we delete that temporary service principal immediately in the script. If you only have one workspace to manage, change the first line to `MULTI_WORKSPACE = False` before running: ```python MULTI_WORKSPACE = True import json import time from datetime import datetime from databricks.sdk import AccountClient, WorkspaceClient from databricks.sdk.errors import BadRequest from databricks.sdk.service import catalog, jobs from databricks.sdk.service.iam import ( AccessControlRequest, ComplexValue, Patch, PatchOp, PatchSchema, PermissionLevel, WorkspacePermission, ) from databricks.sdk.service.workspace import ImportFormat, Language from pydantic import BaseModel, field_validator class DatabricksOAuthToken(BaseModel): id: str oauth_secret: str created_at: datetime expires_at: datetime client_id: str @field_validator("oauth_secret", "client_id", mode="before") @classmethod def strip_whitespace(cls, v: str) -> str: """Strip whitespace from string fields.""" return v.strip() if isinstance(v, str) else v class DatabricksCredentials(BaseModel): oauth_token: DatabricksOAuthToken workspace_url: str service_principal_name: str = "espresso-ai-optimizer" service_principal_id: str warehouse_id: str | None = None warehouse_name: str | None = None workspace_id: str | None = None workspace_name: str | None = None def to_json(self) -> str: return json.dumps(self.model_dump(mode="json")) SYNC_SP_NAME = "espresso-ai-permission-sync" MANAGED_RESOURCES = [ ("warehouses", "warehouses", "list", "id"), ("clusters", "clusters", "list", "cluster_id"), ("jobs", "jobs", "list", "job_id"), ("pipelines", "pipelines", "list_pipelines", "pipeline_id"), ("instance-pools", "instance_pools", "list", "instance_pool_id"), ] SYNC_NOTEBOOK = """# Databricks notebook source from databricks.sdk import WorkspaceClient from databricks.sdk.service.iam import AccessControlRequest, PermissionLevel APP_ID = "{app_id}" client = WorkspaceClient() acl = [ AccessControlRequest( service_principal_name=APP_ID, permission_level=PermissionLevel.CAN_MANAGE ) ] for object_type, service, list_method, id_attr in [ {resources} ]: for obj in getattr(getattr(client, service), list_method)(): if getattr(obj, "creator_user_name", None) == APP_ID: continue if object_type == "jobs" and obj.settings.name == "{sync_job}": continue client.permissions.update( request_object_type=object_type, request_object_id=str(getattr(obj, id_attr)), access_control_list=acl, ) """ def _is_duplicate_error(e): msg = str(e).lower() return "already exists" in msg or "duplicate" in msg def _patch_sp_entitlements(client, sp): client.service_principals.patch( id=sp.id, operations=[ Patch( op=PatchOp.ADD, path="entitlements", value=[ {"value": "databricks-sql-access"}, {"value": "allow-cluster-create"}, ], ), ], schemas=[PatchSchema.URN_IETF_PARAMS_SCIM_API_MESSAGES_2_0_PATCH_OP], ) def get_or_create_service_principal(client, name="espresso-ai-optimizer"): if sps := list(client.service_principals.list(filter=f"displayName eq '{name}'")): sp = sps[0] _patch_sp_entitlements(client, sp) return sp return client.service_principals.create( display_name=name, active=True, entitlements=[ ComplexValue(value="allow-cluster-create"), ComplexValue(value="databricks-sql-access"), ], ) def find_service_principal(client, name="espresso-ai-optimizer", timeout_secs=60): deadline = time.monotonic() + timeout_secs last_err = None while time.monotonic() < deadline: try: sps = list(client.service_principals.list(filter=f"displayName eq '{name}'")) if sps: return sps[0] except Exception as e: # A freshly-assigned principal may not be able to authenticate to the # workspace yet; keep polling until the assignment propagates. last_err = e time.sleep(5) raise RuntimeError( f"Service principal {name!r} not visible after {timeout_secs}s. " "Workspace assignment may not have propagated." + (f" Last error: {last_err}" if last_err else "") ) def create_oauth_token(client, service_principal): token = client.service_principal_secrets_proxy.create( service_principal_id=service_principal.id, lifetime=f"{2 * 365 * 24 * 60 * 60}s", # 2 years ) return DatabricksOAuthToken( id=token.id, oauth_secret=token.secret, created_at=token.create_time, expires_at=token.expire_time, client_id=service_principal.application_id, ) def grant_service_principal_resource_permissions(client, service_principal): acl = [ AccessControlRequest( service_principal_name=service_principal.application_id, permission_level=PermissionLevel.CAN_MANAGE, ) ] for object_type, service, list_method, id_attr in MANAGED_RESOURCES: try: objects = list(getattr(getattr(client, service), list_method)()) except Exception as e: print(f" {object_type}: ⚠️ could not list ({e})") continue granted = 0 errors = [] for obj in objects: object_id = getattr(obj, id_attr, None) if object_id is None or ( object_type == "jobs" and obj.settings.name == SYNC_SP_NAME ): continue try: client.permissions.update( request_object_type=object_type, request_object_id=str(object_id), access_control_list=acl, ) granted += 1 except Exception as e: errors.append(f"{object_id}: {e}") for err in errors: print(f" ⚠️ failed to grant on {err}") def make_service_principal_workspace_admin(client, sp): try: client.groups.patch( id=next(client.groups.list(filter="displayName eq 'admins'")).id, operations=[Patch(op=PatchOp.ADD, value={"members": [{"value": sp.id}]})], schemas=[PatchSchema.URN_IETF_PARAMS_SCIM_API_MESSAGES_2_0_PATCH_OP], ) except Exception as e: if not _is_duplicate_error(e): raise def schedule_permission_sync(client, service_principal): sync_sp = get_or_create_service_principal(client, SYNC_SP_NAME) make_service_principal_workspace_admin(client, sync_sp) for old in list(client.jobs.list(name=SYNC_SP_NAME)): client.jobs.delete(job_id=old.job_id) secret = client.service_principal_secrets_proxy.create( service_principal_id=sync_sp.id, lifetime=f"{60 * 60}s" ) sync_client = WorkspaceClient( host=client.config.host, client_id=sync_sp.application_id, client_secret=secret.secret, ) find_service_principal(sync_client, SYNC_SP_NAME) sync_client.workspace.mkdirs(home := f"/Users/{sync_sp.application_id}") notebook_path = f"{home}/{SYNC_SP_NAME}" sync_client.workspace.upload( notebook_path, SYNC_NOTEBOOK.format( app_id=service_principal.application_id, sync_job=SYNC_SP_NAME, resources="\n".join(f" {r}," for r in MANAGED_RESOURCES), ).encode(), format=ImportFormat.SOURCE, language=Language.PYTHON, overwrite=True, ) sync_client.jobs.create( name=SYNC_SP_NAME, tasks=[ jobs.Task( task_key="sync", notebook_task=jobs.NotebookTask(notebook_path=notebook_path), ) ], schedule=jobs.CronSchedule(quartz_cron_expression="0 0 * * * ?", timezone_id="UTC"), performance_target=jobs.PerformanceTarget.STANDARD, ) client.service_principal_secrets_proxy.delete( service_principal_id=sync_sp.id, secret_id=secret.id ) print(f" Hourly permission sync scheduled, owned by {SYNC_SP_NAME}") def get_or_create_warehouse(client): for warehouse in client.warehouses.list(): if warehouse.name == "ESPRESSO_AI_WAREHOUSE": return warehouse.id return client.warehouses.create_and_wait( name="ESPRESSO_AI_WAREHOUSE", cluster_size="X-Small", auto_stop_mins=1, enable_serverless_compute=True, min_num_clusters=1, max_num_clusters=1, ).id def allow_service_principal_to_read_system_logs(client, service_principal): errors = [] def grant(asset_name, asset_type, privilege): try: client.grants.update( full_name=asset_name, securable_type=asset_type.value, changes=[ catalog.PermissionsChange( add=[privilege], principal=service_principal.application_id ) ], ) except Exception as e: errors.append(str(e).lower()) grant("system", catalog.SecurableType.CATALOG, catalog.Privilege.USE_CATALOG) for schema in client.schemas.list(catalog_name="system"): schema_full_name = f"system.{schema.name}" grant(schema_full_name, catalog.SecurableType.SCHEMA, catalog.Privilege.USE_SCHEMA) grant(schema_full_name, catalog.SecurableType.SCHEMA, catalog.Privilege.SELECT) if not errors: return if any("account admin" in e for e in errors): print("\n⚠️ ACCOUNT ADMIN required: Ask an account admin to grant you access.") if any("manage on catalog" in e or "metastore" in e for e in errors): accounts_url = client.config.environment.deployment_url("accounts") print(f"⚠️ METASTORE ADMIN required: Visit {accounts_url}/data") raise RuntimeError( "Failed to grant system table access (resolve the above and re-run):\n - " + "\n - ".join(errors) ) def wait_for_account_admin(accounts_url, account_id, client_id, secret): deadline = time.monotonic() + 300 # 5 minutes while time.monotonic() < deadline: try: ac = AccountClient( host=accounts_url, account_id=account_id, client_id=client_id, client_secret=secret, ) workspaces = [ (ws.workspace_id, ws.workspace_name, ws.deployment_name) for ws in ac.workspaces.list() ] print("Account admin access confirmed.") return ac, workspaces except Exception: time.sleep(5) raise TimeoutError( "Account admin not granted within 5 minutes. Grant the role via the link above and re-run." ) def filter_to_current_metastore(account_client, workspace_client, workspaces): metastore_id = workspace_client.metastores.current().metastore_id metastore_ws_ids = set(account_client.metastore_assignments.list(metastore_id)) return [ws for ws in workspaces if ws[0] in metastore_ws_ids] def check_workspaces_federated(account_client, workspaces): not_federated = [] for ws_id, ws_name, _ in workspaces: try: account_client.workspace_assignment.list(workspace_id=ws_id) except BadRequest as e: if "permission assignment apis are not available" not in str(e).lower(): raise not_federated.append((ws_id, ws_name)) if not_federated: details = "\n - ".join(f"{name} ({wid})" for wid, name in not_federated) raise PermissionError( "IDENTITY FEDERATION required on the following workspaces:\n" f" - {details}\n\n" "Enable identity federation via the account console:\n" " Workspaces → → Configuration tab → " "'Identity federation' must show 'Enabled'.\n" ) def _assign_workspace_permission(account_client, ws_id, principal_id, permission): deadline = time.monotonic() + 60 while True: try: account_client.workspace_assignment.update( workspace_id=int(ws_id), principal_id=int(principal_id), permissions=[permission], ) return except Exception as e: if _is_duplicate_error(e): return if time.monotonic() >= deadline: raise time.sleep(5) def assign_sp_to_workspaces( account_client, sp_id, workspaces, permission=WorkspacePermission.USER ): for ws_id, ws_name, _ in workspaces: _assign_workspace_permission(account_client, ws_id, sp_id, permission) print(f" Assigned {permission.value} on {ws_name} ({ws_id})") def setup_workspace( account_client, temp_sp, temp_secret, workspace_url, workspace_id, sp_name, ): _assign_workspace_permission( account_client, workspace_id, temp_sp.id, WorkspacePermission.ADMIN ) admin_client = WorkspaceClient( host=workspace_url, client_id=temp_sp.application_id, client_secret=temp_secret, ) sp = find_service_principal(admin_client, sp_name) _patch_sp_entitlements(admin_client, sp) wh_id = get_or_create_warehouse(admin_client) grant_service_principal_resource_permissions(admin_client, sp) schedule_permission_sync(admin_client, sp) allow_service_principal_to_read_system_logs(admin_client, sp) return wh_id if __name__ == "__main__": from importlib import import_module SparkSession = import_module("pyspark.sql").SparkSession spark = SparkSession.getActiveSession() or SparkSession.builder.getOrCreate() workspace_url = f"https://{spark.conf.get('spark.databricks.workspaceUrl') or ''}" client = WorkspaceClient() environment = client.config.environment accounts_url = environment.deployment_url("accounts") service_principal = get_or_create_service_principal(client) oauth_token = create_oauth_token(client, service_principal) warehouse_id = get_or_create_warehouse(client) grant_service_principal_resource_permissions(client, service_principal) schedule_permission_sync(client, service_principal) allow_service_principal_to_read_system_logs(client, service_principal) credentials_by_workspace: dict[str, DatabricksCredentials] = {} if not MULTI_WORKSPACE: workspace_id = str(client.get_workspace_id()) workspace_name = spark.sql( f"SELECT workspace_name FROM system.access.workspaces_latest " f"WHERE workspace_id = {workspace_id}" ).collect()[0][0] credentials_by_workspace[workspace_name] = DatabricksCredentials( oauth_token=oauth_token, workspace_url=workspace_url, workspace_id=workspace_id, workspace_name=workspace_name, service_principal_name=service_principal.display_name, service_principal_id=service_principal.id, warehouse_id=warehouse_id, warehouse_name="ESPRESSO_AI_WAREHOUSE", ) else: account_id = spark.sql( "SELECT account_id FROM system.billing.usage LIMIT 1" ).collect()[0][0] temp_sp = get_or_create_service_principal(client, "espresso-ai-temp") temp_secret = client.service_principal_secrets_proxy.create( service_principal_id=temp_sp.id, lifetime=f"{60 * 60}s" ).secret roles_url = ( f"{accounts_url}/user-management/" f"serviceprincipals/{temp_sp.id}/roles?account_id={account_id}" ) print( f"\n📋 To continue, grant the temporary service principal account admin access." f"\n Open this link and add the 'Account admin' role:\n\n {roles_url}\n" ) account_client, workspaces = wait_for_account_admin( accounts_url, account_id, temp_sp.application_id, temp_secret ) workspaces = filter_to_current_metastore(account_client, client, workspaces) check_workspaces_federated(account_client, workspaces) assign_sp_to_workspaces(account_client, int(service_principal.id), workspaces) current_ws_id = str(client.get_workspace_id()) for ws_id, ws_name, deployment_name in workspaces: ws_id_str = str(ws_id) workspace_url_for_ws = environment.deployment_url(deployment_name) if ws_id_str == current_ws_id: ws_warehouse_id = warehouse_id else: ws_warehouse_id = setup_workspace( account_client=account_client, temp_sp=temp_sp, temp_secret=temp_secret, workspace_url=workspace_url_for_ws, workspace_id=ws_id_str, sp_name=service_principal.display_name, ) credentials_by_workspace[ws_name] = DatabricksCredentials( oauth_token=oauth_token, workspace_url=workspace_url_for_ws, workspace_id=ws_id_str, workspace_name=ws_name, service_principal_name=service_principal.display_name, service_principal_id=service_principal.id, warehouse_id=ws_warehouse_id, warehouse_name="ESPRESSO_AI_WAREHOUSE" if ws_warehouse_id else None, ) delete_url = ( f"{accounts_url}/user-management/" f"serviceprincipals/{temp_sp.id}?account_id={account_id}" ) print( f"\n🧹 When you're done, delete the temporary 'espresso-ai-temp' service" f"\n principal — it still has account admin. Open this link, then" f"\n ⋮ menu → Delete → Confirm delete:\n\n {delete_url}\n" ) print("\n🎉 Setup complete! Here are the Databricks credentials to send to Espresso AI:") print("=" * 50) print( json.dumps( { ws: cred.model_dump(mode="json") for ws, cred in credentials_by_workspace.items() }, indent=2, ) ) print("=" * 50) ``` 5. Copy the JSON credentials printed between the ===== lines and share it securely with Espresso AI. You can securely [upload the output here](https://www.dropbox.com/request/IGcHPWby1x9tPPv8hXWr). ## What the Script Does: * Create or reuse a service principal named "espresso-ai-optimizer" for Espresso AI. * Grant that service principal the ability to manage SQL Warehouses. * Grant SELECT access on Databricks system tables used for usage and cost analysis (e.g. query history, warehouse events, node timeline). * If we're granting access to multiple workspaces, we create a temporary service principal "espresso-ai-temp" with account admin privileges to grant "espresso-ai-optimizer" the same permissions across the workspaces. We then delete "espresso-ai-temp." * Create an hourly job that checks for any new warehouses and grants CAN\_MANAGE permissions to our service principal. * Print a JSON blob with credentials and your workspace URL for you to share securely with Espresso AI. ## Troubleshooting The script provides error messages for permission issues: 1. "You are not a workspace admin" → Contact one of the listed admins for access 2. "ACCOUNT ADMIN PERMISSIONS REQUIRED" → You need account admin to create service principals 3. "METASTORE ADMIN PERMISSIONS REQUIRED" →You need metastore admin to grant system table access 4. "Account admin not granted within 5 minutes. Grant the role via the link above and re-run." → You need to grant the temporary service principal account admin privileges. This SP will be deleted. Quick permission checks: * Verify that you started a serverless notebook, not a notebook on a specific cluster * Verify you're a workspace admin: * Click on the circle with your initials * Click on "Settings" * You should see "Workspace Admin" section in addition to a "User" section. * Verify you're an account admin: * You should be able to login to and see the console to manage your account. * Verify you're a metastore admin: * Visit * Click on the name of the metastore listed, which will open up a "Configuration" page. * Set yourself as the "Metastore Admin". Please rerun the script once you have the required permissions. ## Questions? [Book a Call](https://espresso.ai/demo) # Databricks Terraform Onboarding The Espresso Databricks module manages the Databricks service principal, workspace assignments, system-table grants, bootstrap SQL warehouse, Espresso credentials, and Warehouse Agent settings. ## Prerequisites * Terraform 1.11 or newer * Databricks account-admin access * Databricks metastore-admin access * Identity-federated workspaces attached to Unity Catalog * Serverless SQL support in each workspace * An Espresso organization API key In Espresso, select an existing account, open **Tools → API Keys**, choose **Generate API key → Organization key**, and copy the complete `ok_` secret. It cannot be displayed again. Store it in your secret manager and expose it to Terraform as `ESPRESSO_API_KEY`. Configure an account-level Databricks provider using a bootstrap identity that can manage every workspace passed to the module: ```hcl terraform { required_version = ">= 1.11.0" required_providers { databricks = { source = "databricks/databricks" version = ">= 1.122.0" } espresso = { source = "espressocomputing/espresso" version = ">= 0.1.2" } } } provider "databricks" { host = "https://accounts.cloud.databricks.com" account_id = var.databricks_account_id client_id = var.databricks_client_id client_secret = var.databricks_client_secret } provider "espresso" {} ``` ## Configure the module Create one Espresso account per Databricks workspace. Warehouse Agent settings are disabled unless explicitly enabled. ```hcl module "espresso_databricks" { source = "espressocomputing/databricks/espresso" version = "~> 0.1" providers = { databricks = databricks espresso = espresso } workspaces = { "1234567890123456" = { espresso_slug = "databricks_acme_production" display_name = "ACME Production Databricks" workspace_url = "https://dbc-example.cloud.databricks.com" workspace_name = "production" warehouse_agent = { enabled = true auto_opt_in = true } managed_warehouses = { "ANALYTICS_WH" = { enabled = true } } } } } ``` The module grants `USE_CATALOG` on the `system` catalog and grants `USE_SCHEMA` and `SELECT` on every system schema discovered during planning. Run Terraform again after Databricks adds a system schema so the new grant is created. When several workspaces share a Unity Catalog metastore, set `system_table_access_workspace_ids` to one representative workspace ID per metastore. This prevents Terraform from managing the same grants through multiple workspace APIs. ## Secret storage The module creates a Databricks OAuth secret for the Espresso service principal. The Databricks provider returns that generated secret to Terraform, so it is stored in Terraform state. Use an encrypted remote backend and restrict state access. The Espresso provider treats its `client_secret` argument as write-only and does not add another copy to state. When importing an existing Databricks secret, the module also accepts its plaintext through the sensitive, ephemeral `existing_service_principal_client_secret` input. The Databricks resource itself remains represented in state, and any secret Terraform generates during later rotation is stored there. Use a separate bootstrap identity for the Databricks provider. Terraform cannot safely rotate the same OAuth secret it needs to authenticate the rotation operation. ## Existing onboarding resources If the service principal, OAuth secret, workspace assignment, grants, or bootstrap warehouse already exist, import them into the module addresses before applying. Review the plan until it contains imports and expected in-place changes only. Do not allow Terraform to replace an active service principal or bootstrap warehouse during adoption. After Terraform finishes, verify the Databricks connection and enabled warehouses in Espresso.