Azure Event Hubs source

Last update: Tue Dec 05 2023 00:00:00 GMT+0000 (Coordinated Universal Time)

Topics:
Sources

CREATED FOR:

Developer

IMPORTANT

The Azure Event Hubs source is available in the sources catalog to users who have purchased Real-Time Customer Data Platform Ultimate.

Adobe Experience Platform provides native connectivity for cloud providers like AWS, Google Cloud Platform, and Azure. You can bring your data from these systems into Platform.

Cloud storage sources can bring your own data into Platform without the need to download, format, or upload. Ingested data can be formatted as XDM JSON, XDM Parquet, or delimited. Every step of the process is integrated into the Sources workflow. Platform allows you to bring in data from Event Hubs in real time.

Scaling with Event Hubs

The scale factor of your Event Hubs instance must be increased if you need to ingress high volume data, increase parallelism, or raise the speed of the ingestion Platform.

Ingress higher volume data

Currently, the maximum volume of data that you can bring from your Event Hubs account to Platform is 2000 records per second. To scale up and ingest higher volume data, please contact your Adobe representative.

Increase parallelism on Event Hubs and Platform

Parallelism refers to the simultaneous execution of the same tasks on multiple processing units in order to increase speed and performance. You can increase parallelism on the Event Hubs side by increasing partition or by acquiring more processing units for your Event Hubs account. See this Event Hubs document on scaling for more information.

To increase the rate of speed of ingestion on the Platform side, Platform must increase the number of tasks in the source connector to read from your Event Hubs partitions. Once you have increased parallelism on the Event Hubs side, please contact your Adobe representative to scale Platform tasks based on your new partition. Currently, this process is not automated.

Use a virtual network to connect to Event Hubs to Platform

You can set up a virtual network to connect Event Hubs to Platform while having your firewall measures enabled. To set up a virtual network, head to this Event Hubs network rule set document and follow the steps listed below:

Select Try It from the REST API panel;
Authenticate your Azure account using your credentials in the same browser;
Select the Event Hubs namespace, resource group, and subscription that you want to bring to Platform and then select RUN;
In the JSON body that appears, add the following Platform subnet under virtualNetworkRules inside properties:

IMPORTANT

You must make a backup of the JSON body that you receive, prior to updating virtualNetworkRules with the Platform subnet as it contains your existing IP filtering rules. Otherwise, the rules will be deleted after the call.

{
    "subnet": {
        "id": "/subscriptions/93f21779-b1fd-49ee-8547-2cdbc979a44f/resourceGroups/ethos_12_prod_va7_network/providers/Microsoft.Network/virtualNetworks/ethos_12_prod_va7_network_10_19_144_0_22/subnets/ethos_12_prod_va7_network_10_19_144_0_22"
    },
    "ignoreMissingVnetServiceEndpoint": true
}

See the list below for different regions of Platform subnets:

VA7: North America

{
  "properties": {
    "defaultAction": "Deny",
    "virtualNetworkRules": [
      {
        "subnet": {
          "id": "/subscriptions/93f21779-b1fd-49ee-8547-2cdbc979a44f/resourceGroups/ethos_12_prod_va7_network/providers/Microsoft.Network/virtualNetworks/ethos_12_prod_va7_network_10_19_144_0_22/subnets/ethos_12_prod_va7_network_10_19_144_0_22"
        },
        "ignoreMissingVnetServiceEndpoint": true
      },
    ],
    "ipRules": []
  }
}

NLD2: Europe

{
  "properties": {
    "defaultAction": "Deny",
    "virtualNetworkRules": [
      {
        "subnet": {
            "id": "/subscriptions/40bde086-46ad-44c3-afba-c306f54b64ec/resourceGroups/ethos_12_prod_nld2_network/providers/Microsoft.Network/virtualNetworks/ethos_12_prod_nld2-vnet/subnets/ethos_12_prod_nld2_network_10_20_40_0_23"
        },
        "ignoreMissingVnetServiceEndpoint": true
      },
    ],
    "ipRules": []
  }
}

AUS5: Australia

{
  "properties": {
    "defaultAction": "Deny",
    "virtualNetworkRules": [
      {
        "subnet": {
          "id": "/subscriptions/1618ef18-9edc-48bf-88dd-61cc979629b5/resourceGroups/ethos_12_prod_aus5_network/providers/Microsoft.Network/virtualNetworks/ethos_12_prod_aus5-vnet/subnets/ethos_12_prod_aus5_network_10_21_116_0_22"
        },
        "ignoreMissingVnetServiceEndpoint": true
      },
    ],
    "ipRules": []
  }
}

See the following Event Hubs document for more information on network rule sets.

Connect Event Hubs to Platform

The documentation below provides information on how to connect Event Hubs to Platform using APIs or the user interface:

Using APIs

Using the UI

recommendation-more-help

337b99bb-92fb-42ae-b6b7-c7042161d089