Smart tag your video assets

The expanding need for new content calls for reduced manual efforts to deliver compelling digital experiences in no time. Adobe Experience Manager as a Cloud Service supports auto-tagging of video assets using artificial intelligence. Tagging the videos manually can be time-consuming. However, Adobe Sensei powered video smart tagging feature uses artificial intelligence models to analyze video content and add tags to the video assets. Thereby reducing time for DAM users to deliver rich experiences to their customers. Adobe’s machine learning service generates two sets of tags for a video. While, one set corresponds to objects, scenes, and attributes in that video; the other set relates to actions such as drinking, running, and jogging.

Video tagging is enabled by default in Adobe Experience Manager as a Cloud Service. However, you can opt-out of video smart tagging on a folder. Videos are auto-tagged when you upload new videos or re-process existing ones. Experience Manager also creates the thumbnails and extracts metadata of the video files. The smart tags are displayed in descending order of their confidence score in asset Properties.

Smart tagging videos on upload

When you upload video assets to Adobe Experience Manager as a Cloud Service, the videos are processed. Once the processing is complete, see the Basic tab of asset Properties page. Smart tags are automatically added to the video under Smart Tags. Asset microservices leverages Adobe Sensei to create these smart tags.

Smart Tags are added to videos and seen in Basic tab of asset Properties

The applied smart tags are sorted in descending order of confidence score, combined for object and action tags, within Smart Tags.

IMPORTANT

You are advised to review these automatically generated tags to ensure that they conform to your brand and its values.

Smart tagging existing videos in DAM

The already existing video assets in DAM are not smart tagged automatically. You need to Reprocess Assets manually to generate smart tags for them.

To smart tag video assets, or folders (including subfolders) of assets that already exist in assets repository, follow these steps:

  1. Select the Adobe Experience Manager logo and then select assets from the Navigation page.

  2. Select Files to display the Assets interface.

  3. Navigate to the folder to which you want to apply smart tags.

  4. Select the entire folder or specific video assets.

  5. Select Reprocess assets icon Reprocess Assets icon and select the Full Process option.

Reprocess assets to add tags to videos existing DAM repository

Once the process completes, navigate to the Properties page of any video asset within the folder. The automatically added tags are seen in Smart Tags section in Basic tab. These applied smart tags are sorted in descending order of confidence score.

Search for tagged videos

To search for the video assets based on the auto generated smart tags, use Omnisearch:

  1. Select the search icon search icon to display the Omnisearch field.

  2. Specify a tag, in the Omnisearch field, that you have not explicitly added to a video.

  3. Search based on the tag.

The search results display the video assets based on the tag you specified.

Your search results are a combination of video assets with searched keywords in the metadata and the video assets that are smart tagged with the searched keywords. However, the search results that match all search terms in metadata fields are displayed first, followed by the search results that match any of the search terms in the smart tags. For more information, see Understand Experience Manager search results with smart tags.

Moderate video smart tags

Adobe Experience Manager allows you to curate the smart tags to:

  • remove inaccurate tags assigned to your brand videos.

  • refine tag-based searches for videos by ensuring that your video appears in search results for the most relevant tags. It, therefore, eliminates the chances of unrelated videos from showing up in search results.

  • assign a higher rank to a tag to increase its relevance with respect to a video. Promoting a tag for a video increases the chances of the video appearing in search results when a search is performed based on that tag.

To know more about how to moderate the smart tags for assets, see Manage smart tags.

Moderate video smart tags

NOTE

Any tags that are moderated using the steps in Manage smart tags are not remembered on reprocessing of the asset. The original set of tags are displayed again.

Opt out of video smart tagging

As the automated tagging of videos runs in parallel to other asset processing tasks like thumbnail creation and metadata extraction, it can be time consuming. To expedite the asset processing you can opt out of video smart tagging on upload at folder level.

To opt out of automated video smart tags generation for assets uploaded to specific folder:

  1. Open Asset Processing tab in folder Properties.

  2. In Smart Tags for Videos menu, Inherited option is selected by default and video smart tag is enabled.

    When the Inherited option is selected, the inherited folder path is also visible along with the information whether it is set to Enable or Disable.

    Disable video smart tagging

  3. Select Disable to opt out of smart tagging of videos uploaded to the folder.

IMPORTANT

If you have opted out of tagging videos on a folder at the time of upload and want to smart tag the videos after upload, then Enable Smart Tags for Videos from Asset Processing tab of the folder Properties and use Reprocess Asset option to add smart tags to the video.

Confidence score

Adobe Experience Manager applies a minimum confidence threshold for object and action smart tags to avoid having too many tags for each video asset, which slows down indexing. Your asset search results are ranked based on the confidence scores, which generally improve search results beyond what an inspection of the assigned tags of any video asset suggests. Inaccurate tags often have low confidence scores so they seldom appear at the top of the Smart Tags list for assets.

The default threshold for action and object tags in Adobe Experience Manager is 0.7 (should be value between 0 and 1). If some video assets are not tagged by a specific tag, then it indicates that the algorithm is less than 70% confident in the predicted tags. The default threshold might not always be optimal for all the users. You can, therefore, change the confidence score value in OSGI configuration.

To add the confidence score OSGI configuration to the project deployed to Adobe Experience Manager as a Cloud Service through Cloud Manager:

  • In the Adobe Experience Manager project (ui.config since Archetype 24, or previously ui.apps) the config.author OSGi configuration, include a config file named com.adobe.cq.assetcompute.impl.senseisdk.SenseiSdkImpl.cfg.json with the following contents:
{
  "minVideoActionConfidenceScore":0.5,
  "minVideoObjectConfidenceScore":0.5,
}
NOTE

Manual tags are assigned a confidence of 100% (maximum confidence). Therefore, if there are video assets with manual tags that match the search query, they are displayed before smart tags matching the search query.

Limitations

  • You cannot train the service that applies Smart Tags to videos using any specific videos. It works with default Adobe Sensei settings.

  • Tagging progress is not displayed.

  • Only the videos smaller than 300 MB in file size are auto-tagged. The Adobe Sensei service skips video files that are larger in size.

  • Only the videos in the file formats and supported codecs mentioned in Smart Tags are tagged.

On this page