Deduplication deduplication

Add a Deduplication activity to your workflow.

In the Fields to identify duplicates section, click the Add attribute button to specify the fields for which the identical values allow the duplicates to be identified, such as: email address, first name, last name, etc. The order of the fields allows you to specify those to process first.

In the Deduplication settings section, select the number of unique Duplicates to keep. The default value for this field is 1. The value 0 allows you to keep all the duplicates.

For example, if records A and B are considered duplicates of record Y, and a record C is considered as a duplicate of record Z:

• If the value of the field is 1: only the Y and Z records are kept.
• If the value of the field is 0: all the records are kept.
• If the value of the field is 2: records C and Z are kept and two records from A, B, and Y are kept, by chance or depending on the deduplication method selected thereafter.

Select the Deduplication method to use:

• Random selection: Randomly selects the record to be kept out of the duplicates.
• Using an expression: Keep the records in which the value of the expression entered is the smallest or the biggest.
• Non-empty values: Keep the records for which the expression is not empty.
• Following a list of values: Define a value priority for one or more fields. To define the values, click Attribute to select a field or create an expression, then add the value(s) into the appropriate table. To define a new field, click the Add button located above the list of values.

Check the Generate complement option if you wish to exploit the remaining population. The complement consists of all the duplicates. An additional transition will then be added to the activity.