Tokenize

Last update: 2022-10-04
  • Created for:
  • User
    Admin
IMPORTANT

Read more about Data Workbench’s End-of-life announcement.

The Tokenize transformation iteratively applies a regular expression against the input string.

However, unlike RETransform, Tokenize does not have to match the entire string: the regular expression used for the Tokenize transformation can match a subset of the input. After a match is found, Tokenize applies the regular expression again, starting at the character after the end of the last match.

Parameter Description Default
Name Descriptive name of the transformation. You can enter any name here.
Case Sensitive True or false. Specifies whether the match is case-sensitive.
Comments Optional. Notes about the transformation.
Condition The conditions under which this transformation is applied.
Default The default value to use if the condition is met and the input value is either not available or the regular expression does not match the input value.
Expression The regular expression used for matching.
Outputs The names of the output strings. You can have multiple outputs for a given input string. The number of outputs must correspond to the number of capturing sub-patterns in the regular expression.

In the following example, the Tokenize transformation uses a regular expression to capture the names of the query strings (in cs-uri-query) and output the captured sub-pattern (the query name) to x-pull-query-name.

For the query string “a=b&c=d,” the output would be a vector containing “a” and “c.”

For information about regular expressions, see Regular Expressions.

Previous page
Next page

On this page