Campaign: Slow data transfers from Adobe Campaign to Databricks (DBX)
This article describes a performance issue where data transfers from Adobe Campaign to Databricks are significantly slower than expected. The behavior occurs when Campaign cannot use its bulk-load mechanism and instead falls back to a slower row-by-row insertion strategy.
Description description
In some Adobe Campaign workflows, data written to Databricks can be extremely slow, even when other database transfer paths perform normally.
Typical symptoms include:
- Transfers from Adobe Campaign or federated data sources into Databricks take much longer than expected.
- Reverse or alternate data flows may still perform normally.
- The issue is reproducible across multiple workflows or environments.
- Workflow or audit logs contain a warning similar to:
Storage account used for bulk loading is not found. Using default bulk load strategy
This warning indicates that Campaign could not find the linked storage configuration required for the bulk-load path. As a result, the connector falls back to a default row-by-row insertion method over ODBC.
Resolution resolution
The slow data transfer performance observed when writing data from Adobe Campaign to Databricks is a direct consequence of the Databricks external account not having a linked cloud storage account (Azure Blob or AWS S3) configured.
Root Cause
When Campaign writes data to Databricks, it attempts to use a bulk load path that relies on a linked cloud storage account as an intermediate staging area. The process is designed as follows:
- Campaign serializes outbound data as compressed CSV files.
- Campaign uploads these files to the linked cloud storage account (Azure Blob or S3).
- Campaign issues a Databricks COPY INTO command, instructing Databricks to ingest the staged files directly from cloud storage.
- Databricks reads the files in parallel at native speed.
When no linked storage account is configured, Campaign cannot use this bulk load path. The system logs the following warning:
Storage account used for bulk loading is not found. Using default bulk load strategy
This warning is visible in the audit log of the workflow. After this warning, Campaign falls back to sending individual row-level INSERT statements over the ODBC connection. Each row requires a separate network round-trip to the Databricks SQL Warehouse, which results in performance that degrades linearly with data volume.
This fallback behavior is not a degraded “slower mode” — it is not suitable for production-scale data volumes. It exists only as a last-resort mechanism.
So, Adding a linked storage account (Azure Blob or AWS S3) to the Databricks external account will resolve this performance issue. With a storage account configured, Campaign will use the COPY INTO bulk load path, which is Databricks’ recommended data ingestion mechanism.