[Ultimate]{class="badge positive"}

Databricks

Databricks is a cloud-based platform designed for data analytics, machine learning, and AI. You can use Databricks to integrate and provide a holistic environment for building, deploying, and managing data solutions at scale.

Use the Databricks source to connect your account and ingest your Databricks data to Adobe Experience Platform.

Prerequisites

Complete the prerequisite steps to successfully connect your Databricks account to Experience Platform.

Retrieve your container credentials

Retrieve your Experience Platform Azure Blob Storage credentials to enable your Databricks account to access it later.

To retrieve your credentials, make a GET request to the /credentials endpoint of the Connectors API.

API format

GET /data/foundation/connectors/landingzone/credentials?type=dlz_databricks_source

Request

The following request retrieves the credentials for your Experience Platform Azure Blob Storage.

curl -X GET \
  'https://platform.adobe.io/data/foundation/connectors/landingzone/credentials?type=dlz_databricks_source' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -H 'Content-Type: application/json' \

Response

A successful response provides your credentials (containerName, SASToken, storageAccountName) for later use in Apache Spark configuration for Databricks.

{
    "containerName": "dlz-databricks-container",
    "SASToken": "sv=2020-10-02&si=dlz-b1f4060b-6bbd-4043-9bd9-a5f5be72de30&sr=c&sp=racwdlm&sig=zVQfmuElZJzOKkUk8z5lChrJ3YQUE2h6EShDZOsVeMc%3D",
    "storageAccountName": "sndbxdtlndga8m7ajbvgc64k",
    "SASUri": "https://sndbxdtlndga8m7ajbvgc64k.blob.core.windows.net/dlz-databricks-container?sv=2020-10-02&si=dlz-b1f4060b-6bbd-4043-9bd9-a5f5be72de30&sr=c&sp=racwdlm&sig=zVQfmuElZJzOKkUk8z5lChrJ3YQUE2h6EShDZOsVeMc%3D",
    "expiryDate": "2025-07-05"
}

Refresh your credentials

To refresh your credentials, make a POST request and include action=refresh as a query parameter.

API format

POST /data/foundation/connectors/landingzone/credentials?type=dlz_databricks_source&action=refresh

Request

The following request refreshes the credentials for your Azure Blob Storage.

curl -X POST \
  'https://platform.adobe.io/data/foundation/connectors/landingzone/credentials?type=dlz_databricks_source&action=refresh' \
  -H 'Authorization: Bearer {ACCESS_TOKEN}' \
  -H 'x-api-key: {API_KEY}' \
  -H 'x-gw-ims-org-id: {ORG_ID}' \
  -H 'x-sandbox-name: {SANDBOX_NAME}' \
  -H 'Content-Type: application/json' \

Response

A successful response returns your new credentials.

{
    "containerName": "dlz-databricks-container",
    "SASToken": "sv=2020-10-02&si=dlz-6e17e5d6-de18-4efc-88c7-45f37d242617&sr=c&sp=racwdlm&sig=wvA4K3fcEmqAA%2FPvcMhB%2FA8y8RLwVJ7zhdWbxvT1uFM%3D",
    "storageAccountName": "sndbxdtlndga8m7ajbvgc64k",
    "SASUri": "https://sndbxdtlndga8m7ajbvgc64k.blob.core.windows.net/dlz-databricks-container?sv=2020-10-02&si=dlz-6e17e5d6-de18-4efc-88c7-45f37d242617&sr=c&sp=racwdlm&sig=wvA4K3fcEmqAA%2FPvcMhB%2FA8y8RLwVJ7zhdWbxvT1uFM%3D",
    "expiryDate": "2025-07-20"
}

Configure access to your Azure Blob Storage

Next, you must ensure that your Databricks cluster has access to the Experience Platform Azure Blob Storage account. In doing so, you can use Azure Blob Storage as an interim location for writing delta lake table data.

To provide access, you must configure an SAS token on the Databricks cluster as part of your Apache Spark configuration.

In your Databricks interface, select Advanced options and then input the following in the Spark config input box.

fs.azure.sas.{CONTAINER_NAME}.{STORAGE-ACCOUNT}.blob.core.windows.net {SAS-TOKEN}

If unprovided, the copy activity in the flow run fails and returns the following error:

Unable to access container '{CONTAINER_NAME}' in account '{STORAGE_ACCOUNT}.blob.core.windows.net' using anonymous credentials. No credentials found in the configuration. Public access is not permitted on this storage account.

Unity Catalog governance and cluster access mode

The Databricks source connector performs read and write (I/O) operations on your Databricks cluster during ingestion. These operations require Unity Catalog object permissions and, depending on your cluster’s access mode, additional cluster-level permissions. Configure both before connecting.

Grant Unity Catalog permissions

Grant the following Unity Catalog permissions on the source objects to the identity used by the connection (a service principal):

Configure your cluster’s access mode

Select the access mode used by your Databricks cluster and complete the corresponding configuration.

Standard (Shared) access mode

Standard (Shared) access mode enforces Table Access Control. In addition to the Unity Catalog grants above, the connector’s I/O operations require the ANY FILE privilege. Grant it to the connector’s service principal:

GRANT SELECT ON ANY FILE TO `{SERVICE_PRINCIPAL}`;
GRANT MODIFY ON ANY FILE TO `{SERVICE_PRINCIPAL}`;

If the permissions required for your cluster’s access mode are not in place, ingestion fails during execution with the following error:

[INSUFFICIENT_PERMISSIONS] ... on any file. SQLSTATE: 42501.

Connect Databricks to Experience Platform

Now that you have completed the prerequisite steps, you can now proceed and connect your Databricks account to Experience Platform: