Read more about Data Workbench’s End-of-life announcement.
The Tokenize transformation iteratively applies a regular expression against the input string.
However, unlike RETransform, Tokenize does not have to match the entire string: the regular expression used for the Tokenize transformation can match a subset of the input. After a match is found, Tokenize applies the regular expression again, starting at the character after the end of the last match.
|Name||Descriptive name of the transformation. You can enter any name here.|
|Case Sensitive||True or false. Specifies whether the match is case-sensitive.|
|Comments||Optional. Notes about the transformation.|
|Condition||The conditions under which this transformation is applied.|
|Default||The default value to use if the condition is met and the input value is either not available or the regular expression does not match the input value.|
|Expression||The regular expression used for matching.|
|Outputs||The names of the output strings. You can have multiple outputs for a given input string. The number of outputs must correspond to the number of capturing sub-patterns in the regular expression.|
In the following example, the Tokenize transformation uses a regular expression to capture the names of the query strings (in cs-uri-query) and output the captured sub-pattern (the query name) to x-pull-query-name.
For the query string “a=b&c=d,” the output would be a vector containing “a” and “c.”
For information about regular expressions, see Regular Expressions.