DocumentationExperience PlatformModel data in schemas

Model data in schemas

Last update: Thu Oct 03 2024 00:00:00 GMT+0000 (Coordinated Universal Time)

CREATED FOR:

  • Beginner
  • Intermediate
  • Developer

In this lesson, you will model Luma’s data into schemas. This is one of the longest lessons in the tutorial, so get a glass of water and buckle up!

Standardization and interoperability are key concepts behind Adobe Experience Platform. Experience Data Model (XDM) is an effort to standardize customer experience data and define schemas for customer experience management.

XDM is a publicly documented specification designed to improve the power of digital experiences. It provides common structures and definitions for any application to use to communicate with Platform services. By adhering to XDM standards, all customer experience data can be incorporated into a common representation that can deliver insights in a faster, more integrated way. You can gain valuable insights from customer actions, define customer audiences through segments, and express customer attributes for personalization purposes.

XDM is the foundational framework that allows Adobe Experience Cloud, powered by Experience Platform, to deliver the right message to the right person, on the right channel, at exactly the right moment. The methodology on which Experience Platform is built, XDM System, operationalizes Experience Data Model schemas for use by Platform services.

Data Architects will need to create schemas outside of this tutorial, but Data Engineers will work closely with the schemas created by the Data Architect.

Before you begin the exercises, watch this short video to learn more about schemas and the Experience Data Model (XDM):

https://video.tv.adobe.com/v/27105?learn=on

TIP
For a deeper dive into data modeling in Experience Platform, we recommend watching the playlist Model Your Customer Experience Data with XDM, available for free on Experience League!

Permissions required

In the Configure Permissions lesson, you set up all the access controls required to complete this lesson.

Create Loyalty Schema via UI

In this exercise, we will create a schema for Luma’s loyalty data.

  1. Go to the Platform user interface and ensure that your sandbox is selected.

  2. Go to Schemas in the left navigation.

  3. Select the Create schema button on the top right.
    Schema with OOTB Field group

  4. In the Create schema workflow, select Individual Profile as the base class for your schema, since we will be modeling attributes of an individual customer (points, status, and so on).

  5. Select Next.
    Select base class

  6. Enter Luma Loyalty Schema in the Schema display name text field. In the canvas below, you can also review and verify the base schema structure that is provided by the class you have chosen.

  7. Select Finish to create your schema.
    Finish creating the loyalty schema

Add standard field groups

Once the schema is created, you will be redirected to the Schema editor where you can add fields to the schema. You can add individual fields directly to the schema or use field groups. It is important to note that all individual fields are still associated with a class or field group. You can choose from a large set of industry-standard field groups provided by Adobe or create your own. As you start modeling your own data in Experience Platform, it is good to become familiar with the industry-standard field groups provided by Adobe. Whenever possible, it is a best practice to use them as they sometimes power downstream services, such as Customer AI, Attribution AI, and Adobe Analytics.

When working with your own data, a significant step will be to determine which of your own data should be captured in Platform and how it should be modeled. This large topic is discussed in more depth in the playlist Model Your Customer Experience Data with XDM. In this tutorial, I will just guide you through the implementation of some pre-determined schemas.

To add field groups:

  1. Select Add under the Field Groups heading.
    Add a new field group

  2. In the Add Field groups modal, select the following field groups:

    1. Demographic Details for basic customer data like name and birthdate
    2. Personal Contact Details for basic contact details like email address and phone number

  3. You can preview the fields contributed in the field group by selecting the icon on the right side of the row.
    Select standard field groups

  4. Check the Industry > Retail box to expose industry-specific field groups.

  5. Select Loyalty Details to add the loyalty program fields.

  6. Select Add field groups to add all three field groups to the schema.
    Add standard field groups to loyalty schema

Now, take some time to explore the current state of the schema. The field groups have added standard fields related to a person, their contact details, and loyalty program status. You may find these two field groups useful when you create schemas for your own company’s data. Select a specific field group row or check the box next to the field group name to see how the visualization changes.

To save the schema, select Save.
Save the schema

NOTE
It is okay if a field group adds a field for a data point that you don’t collect. For example, “faxPhone” might be a field for which Luma doesn’t collect data. That’s fine. Just because a field is defined in the schema doesn’t mean that data for it must be ingested later on. You can also remove the field from the schema.

Add a custom field group

Now, let’s create a custom field group.

While the loyalty field group contained a loyaltyID field, Luma would like to manage all of their system identifiers in a single group to help ensure consistency across their schemas.

Field groups must be created in the schema workflow. You can either:

  • Add a new custom field to your schema first and then create a custom field group, or
  • Create a custom field group first and then add fields to it.

In this tutorial, we start with creating a custom field group.

To create the field group:

  1. Select Add under the Schema Field Groups heading
    Add a new field group
  2. Select Create new field group
  3. Use Luma Identity profile field group as the Display name
  4. Use system identifiers for XDM Individual Profile class as the Description
  5. Select Add field groups
    Add a new field group

The new, empty field group is added to your schema. The + buttons can be used to add new fields to any location in the hierarchy. In our case we want to add fields at the root level:

  1. Select + next to the name of the schema. This adds a new field under your tenant id namespace to manage conflicts between your custom fields and any standard fields.

  2. In the Field properties sidebar add the details of the new field:

    1. Field name: systemIdentifier
    2. Display name: System Identifier
    3. Type: Object
    4. In the Field Group dropdown select the Luma Identity profile field group that we have created.
      Add a new field group
    5. Select Apply
      Apply new field properties

Now add two fields under the systemIdentifier object:

  1. First field

    1. Field name: loyaltyId
    2. Display name: Loyalty Id
    3. Type: String

  2. Second field

    1. Field Name: crmId
    2. Display Name: CRM Id
    3. Type: String

Your new field group should look like this. Select the Save button to save your schema, but leave the schema open for the next exercise.
Loyalty field group complete

Create a data type

Field groups, such as your new Luma Identity profile field group, can be reused in other schemas, allowing you to enforce standard data definitions across multiple systems. But they can only be reused in schemas that share a class, in this case the XDM Individual Profile class.

The data type is another multi-field construct which can be reused in schemas across multiple classes. Let’s convert our new systemIdentifier object into a data type:

With the Luma Loyalty Schema still open, select the systemIdentifier object and select Convert to new data type

Loyalty field group Complete

If you Cancel out of the schema and navigate to the Data types tab, you will see your newly created data type. We will use this data type later in the lesson.

Loyalty field group Complete

Create CRM Schema via API

Now we will create a schema using the API.

TIP
If you prefer to skip the API exercise, you can create the following schema using the user interface method:
  1. Use the Individual Profile class
  2. Name it Luma CRM Schema
  3. Use the following field groups: Demographic Details, Personal Contact Details, and Luma Identity profile field group

First we create the empty schema:

  1. Open Postman

  2. If you don’t have an access token, open the request OAuth: Request Access Token and select Send to request a new access token.

  3. Open your environment variables and change the value of CONTAINER_ID from global to tenant. Remember, you must use tenant whenever you want to interact with your own custom elements in Platform, such as creating a schema.

  4. Select Save
    Change the CONTAINER_ID to tenant

  5. Open the request Schema Registry API > Schemas > Create a new custom schema.

  6. Open the Body tab and paste the following code and select Send to make the API call. This call creates a new schema using the same XDM Individual Profile base class:

    code language-json 
    {
  "type": "object",
  "title": "Luma CRM Schema",
  "description": "Schema for CRM data of Luma Retail ",
  "allOf": [{
    "$ref": "https://ns.adobe.com/xdm/context/profile"
  }]
}
    note note
    NOTE
    The namespace references in this and subsequent code samples (for example https://ns.adobe.com/xdm/context/profile), can be obtained by using list API calls with the CONTAINER_ID and accept header set to the correct values. Some are also easily accessible in the user interface.

  7. You should get a 201 Created response

  8. Copy meta:altId from Response body. We will use it later in another exercise.
    Create the CRM schema

  9. The new schema should be visible in the user interface but without any field groups
    Create the CRM schema

NOTE
The meta:altId or schema id can also be obtained by making the API request Schema Registry API > Schemas > Retrieve a list of schemas within the specified container. with the CONTAINER_ID set to tenant and an accept header application/vnd.adobe.xdm+json.
TIP
Common issues with this call and likely fixes:
  • No auth token: Run the OAuth: Request Access Token request to generate a new token
  • 401: Not Authorized to PUT/POST/PATCH/DELETE for this path : /global/schemas/: Update the CONTAINER_ID environment variable from global to tenant
  • 403: PALM Access Denied. POST access is denied for this resource from access control: Verify your user permissions in the Admin Console

Add standard field groups

Now it’s time to add the field groups to the schema:

  1. In Postman, Open the request Schema Registry API > Schemas > Update one or more attributes of a custom schema specified by ID.

  2. In the Params tab, paste the meta:altId value from the previous response as the SCHEMA_ID

  3. Open the Body tab and paste the following code and select Send to make the API call. This call adds the standard field groups to your Luma CRM Schema:

    code language-json 
    [{
    "op": "add",
    "path": "/allOf/-",
    "value": {
      "$ref": "https://ns.adobe.com/xdm/context/profile-personal-details"
    }
  },
  {
    "op": "add",
    "path": "/allOf/-",
    "value": {
      "$ref": "https://ns.adobe.com/xdm/context/profile-person-details"
    }
  }
]

  4. You should get a 200 OK status for the response and the field groups should be visible as part of your schema in the UI

    Standard field groups added

Add custom field group

Now let’s add our Luma Identity profile field group to the schema. First, we need to find the id of our new field group, using a list API:

  1. Open the request Schema Registry API > Field groups > Retrieve a list of field groups within the specified container.

  2. Select the Send button to retrieve a list of all of the custom field groups in your account

  3. Grab the $id value of the Luma Identity profile field group (yours will be different from the value in this screenshot)
    Retrieve the list of field groups

  4. Open the request Schema Registry API > Schemas > Update one or more attributes of a custom schema specified by ID. again

  5. The Params tab should still have the $id of your schema

  6. Open the Body tab and paste the following code, replacing the $ref value with the $id of your own Luma Identity profile field group:

    code language-json 
    [{
  "op": "add",
  "path": "/allOf/-",
  "value": {
    "$ref": "REPLACE_WITH_YOUR_OWN_FIELD_GROUP_ID"
  }
}]

  7. Select Send
    Adding the Identity Field group

Verify that the field group has been added to the schema by checking both the API response and in the interface.

Create Offline Purchase Events Schema

Now let’s create a schema based on the Experience Event class for Luma’s offline purchase data. Since you are now getting familiar with the schema editor user interface, I will reduce the number of screenshots in the instructions:

  1. Create a schema with the Experience Event class.
  2. Name your schema Luma Offline Purchase Events Schema.
  3. Add the standard field group Commerce Details to capture common order details. Spend a few minutes exploring the objects inside.
  4. Search for Luma Identity profile field group. It is not available! Remember that field groups are tied to a class, and since we are using a different class for this schema we can’t use it. We need to add a new field group for the XDM ExperienceEvent class containing the identity fields. Our data type will make that really easy!
  5. Select the Create new field group radio button
  6. Enter the Display name as Luma Identity ExperienceEvent field group and select the Add field groups button
  7. Select + next to the name of the schema.
  8. As the Field Name, enter systemIdentifier.
  9. As the Display Name, enter System Identifier.
  10. As the Type, select System Identifier which is the custom data type you created earlier.
  11. As the Field Group select Luma Identity ExperienceEvent field group.
  12. Select the Apply button.
  13. Select the Save button.

Note how the data type added all of the fields!

Add the data type to the field group

Also, select XDM ExperienceEvent under the Class heading and inspect some of the fields contributed by this class. Note that _id and timestamp fields are required when using the XDM ExperienceEvent class—these fields must be populated for every record you ingest when using this schema:

Experience Event Base Structure

Create Web Events Schema

Now we are going to create one more schema for Luma’s website data. By this point you should be an expert at creating schemas! Build the following schema with these properties

Property
Value
Class
Experience Event
Schema Name
Luma Web Events Schema
Field group
AEP Web SDK ExperienceEvent
Field group
Consumer Experience Event

Select the Consumer Experience Event field group. This field group contains the commerce and productListItems objects that were also in Commerce Details. Indeed Consumer Experience Event is a combination of several other standard field groups that are also available separately. AEP Web SDK ExperienceEvent field group also contains other field groups, including some of the same ones in Consumer Experience Event. Fortunately, they blend together seamlessly.

Notice that we didn’t add the Luma Identity ExperienceEvent field group to this schema. This is because the Web SDK has a different way of collecting identities. If you select the XDM ExperienceEvent class in the Composition section of the schema editor, you will notice that one of the fields it adds by default is called IdentityMap. IdentityMap is used by various Adobe applications to link to Platform. You will see how identities are sent to Platform via identityMap in the streaming ingestion lesson.

Create Product Catalog Schema

By using the Commerce Details and Consumer Experience Event field groups, Luma reports some details of product-related events via the standard productListItems data type. But they also have additional product detail fields that they would like to send to Platform. Instead of capturing all of these fields in their point-of-sale and e-commerce systems, Luma would prefer to ingest these fields directly from their product catalog system. A “schema relationship” allows you to define a relationship between two schemas for the purposes of classification or lookups. Luma will use a relationship to classify their product details. We will begin the process now and complete it at the end of the next lesson.

NOTE
If you are an existing Analytics or Target customer, classifying entities with schema relationships is analogous to SAINT classifications or uploading your product catalog for Recommendations

First we must create a schema for Luma’s product catalog using a custom class:

  1. Select the Create schema button.

  2. In the Create schema workflow, select Other option.
    Create new Schema

  3. Select the Create class button

  4. Name it Luma Product Catalog Class

  5. Leave the Behavior as Record

  6. Select the Create button.
    Create new Class

  7. The Luma Product Catalog Class you have created appears in the Classes table below. Make sure the class is selected, then select Next.
    New Class added

  8. Name the schema Luma Product Catalog Schema.

  9. Create a new field group called Luma Product Catalog field group with the following fields:

    1. productName: Product Name: String
    2. productCategory: Product Category: String
    3. productColor: Product Color: String
    4. productSku: Product SKU: String | Required
    5. productSize: Product Size: String
    6. productPrice: Product Price: Double

  10. Save the schema

Your new schema should look like this. Note how the productSku field is listed in the Required fields section:
Product Schema

The next step is to define the relationship between the two ExperienceEvent schemas and the Luma Product Catalog Schema, however there are a few additional steps that we must take in the next lesson before we can do that.

Additional Resources

Now that you have your schemas you can map identities!

recommendation-more-help
513160b6-bf42-4c58-abdd-4f817b1cccad