Data replication wf-data-replication
Principle
In the context of an Enterprise (FFDA) deployment, data replication ensures that the two databases, Campaign local database (PostgreSQL) and Cloud database (Snowflake), are operational in parallel and remain synchronized in real-time.
The Cloud database (Snowflake) is optimized for handling large data batches, such as updating 1 million addresses. Meanwhile, the Campaign local database (PostgreSQL) is better suited for individual or small-volume operations, like updating a single seed address. Synchronization occurs automatically and transparently in the background, ensuring data in Campaign local database (PostgreSQL) is duplicated in the Cloud database (Snowflake) in real-time, keeping both databases synchronized. Data synchronization involves schemas and tables, and data.
➡️ Discover how data replication works in video
Replication modes modes
Data replication can occur in different modes depending on the use case.
- On-the-fly replication handles cases where real-time duplication is essential. It relies on specific technical threads to replicate data immediately for use cases like creating a diffusion or updating a seed address.
- Scheduled replication is used when immediate synchronization is not required. Scheduled replication uses specific technical workflows that run every hour to synchronize data, such as typology rules.
Replication Policies
Replication policies define how much data is replicated from a Campaign local database (PostgreSQL) table. These policies depend on the size of the table and the specific use case. Some tables will have incremental updates when others will be entirely replicated. There are three main types of replication policies:
- XS: This policy is used for tables with relatively small sizes. The entire table is replicated in one shot. Incremental replication avoids repeatedly replicating the same data by using a timestamp pointer to replicate only recent changes.
- SingleRow: This policy replicates only one row at a time. It is typically used for on-the-fly replication involving current Campaign objects and related objects.
- SomeRows: This policy is designed for replicating a limited subset of data using query definitions or filters. It is used for larger tables where selective replication is necessary.
Replication Workflows workflows
Campaign v8 relies on specific technical workflows to manage scheduled data replication. These technical workflows are available from the Administration > Production > Technical workflows > Full FFDA Replication node of Campaign Explorer. They must not be modified.
Technical workflows execute processes or jobs, scheduled on a regular basis on the server. The full list of technical workflows is detailed in this page.
Technical workflows ensuring data replication are as follows:
If needed, you can launch data synchronization manually. To do this, right-click on the Scheduler activity and select Execute pending task(s) now.
In addition to the built-in Replicate Reference tables technical workflow, you can force data replication in your workflows using one of these methods
-
Add a specific Javascript code activity with the following code:
code language-none nms.replicationStrategy.StartReplicateStagingData("dem:sampleTable")
-
Add a specific nlmodule activity with the following command:
code language-none nlserver ffdaReplicateStaging -stagingSchema -instance:acc1
APIs
APIs enable replication of both custom and out-of-the-box data from Campaign local database (PostgreSQL) to Cloud database (Snowflake). These APIs allow you to bypass predefined workflows and customize replication for specific requirements, such as replicating custom tables.
Example:
var dataSource = "nms:extAccount:ffda";
var xml = xtk.builder.CopyXxlData(
<params dataSource={dataSource} policy="xs">
<srcSchema name="cus:recipient"/>
</params>
);
Replication queues
When high volumes of replication requests occur simultaneously, performance issues may arise in the Cloud database (Snowflake) due to table-level locks during MERGE operations. To mitigate this, centralized replication workflows group requests into queues.
Each queue is handled by a technical workflow, which manages replication for a specific table, executing pending requests as a single MERGE operation. These workflows are triggered every 20 seconds to process new replication requests:
nms:delivery table.nms:dlvExclusion table.nms:dlvRemoteIdRel table.Replicate nmsTrackingUrl queue in concurrency (ffdaReplicateQueueTrackingUrl_2)
nms:trackingUrl table, utilizing two workflows to improve efficiency by processing requests based on different priorities.Tutorial video
This video presents the key concepts of which databases Adobe Campaign v8 uses, why data is being replicated, which data is being replicated and how the replication process works.
Additional Campaign v8 Client Console tutorials are available here.