Bot Rules let you remove traffic from your report suite that is generated by known spiders and bots. Removing bot traffic can provide a more accurate measurement of user activity on your website.
After bot rules are defined, all incoming traffic is compared against the defined rules. Traffic that matches any of these rules is not collected in the report suite and is not included in traffic metrics.
Removing bot traffic typically reduces the volume of traffic and conversion metrics. Many customers find that removing bot traffic results in increased conversion rates and increases in other usability metrics.
Bot traffic data is stored in a separate repository for display in the Bots and Bot Pages reports.
Before removing bot traffic, communicate with stakeholders to make sure they can make the necessary adjustments to key performance indicators as a result of this change. If possible, we recommend first removing bot traffic from a small report suite to estimate the potential impact.
The following video demonstrates how to configure bot rules:
To update or upload bot rules:
Go to Analytics > Admin > Report Suites.
Select the report suite where you want to update bot rules, then select Edit Settings > General > Bot Rules.
Use any of the following options to update or upload bot rules for the report suite:
Select Enable IAB Bot Filtering Rules to remove bots in IAB’s (International Advertising Bureau’s) International Spiders & Bots List to remove bot traffic.
We recommend that you select this option at a minimum.
For more information, see the section below, Standard IAB bot rules.
Select Add Rule to define and add custom bot rules based on user agents, IP addresses or IP ranges.
For more information, see the section below, Custom bot rules.
Next to the Select CSV Bot file to import area, select Choose File, then select the CSV file that defines the bot rules.
For more information, see the section below, Upload bot rules.
Standard IAB bot rules can be turned on by checking the Enable IAB Bot Filtering Rules checkbox. This selection will remove bots in IAB’s (International Advertising Bureau’s) International Spiders & Bots List to remove bot traffic. Adobe updates this list from the IAB on a monthly basis.
Adobe is unable to provide the detailed IAB bot list to customers, though you can use the Bots Report to view a list of bots that have accessed your site. To submit a bot to the IAB list, visit IAB.
For information about how to enable standard IAB bot rules in a report suite, see Update or upload bot rules.
The user interface allows for 500 rules to be manually defined. After this limit is reached, rules must be managed in bulk through the Import File and Export Bot Rules options.
Custom bot rules let you filter traffic based conditions you define. To begin the process of enabling custom bot rules in a report suite, see Update or upload bot rules.
Custom bot rules are defined using the following condition types:
Multiple conditions can be defined for a single rule. Multiple conditions are matched using “or”. For example, if you provide a value for User Agent and IP Address, the traffic is considered bot traffic if either condition is met.
A User Agent condition checks the user agent value to see if it starts with or contains the specified string. If contains is selected, the substring is matched if it occurs anywhere in the user agent.
Optional values can be included in the does not contain list to define values that the user agent must not contain for a successful match. Multiple values can be specified by including one value per line. If the user agent meets the criteria specified in the match string, but also contains a string on the does not contain list, it is not considered a match.
The contains field is limited to 100 characters. The does not contain list is limited to 255 characters minus a separator character for each new line. (This is equal to the number of strings - 1. If you specify 4 does not contain strings, 3 separator characters are required.) All string matches are case-insensitive.
Matches an IP address or multiple addresses in the same block using wildcards (*). Provide the numeric values of the IP address you want to match. Substitute * for any values you want to match using a wildcard. The following list contains examples of IP address match string:
Provide the start and end ranges of the IP addresses to match. Substitute * for any values you want to match using a wildcard.
To bulk import bot rules, you can upload a CSV file that defines the rules.
To begin the process of uploading bot rules to a report suite, see Update or upload bot rules.
Create a CSV file with the following columns, on row 1 of the spreadsheet and in the order presented:
|Column 1, Row 1
|Column 2, Row 1
|Column 3, Row 1
|Column 4, Row 1
|Column 5, Row 1
|Column 6, Row 1
(contains or starts with)
|User Agent Include
|User Agent Exclude
You can define three types of bot rules:
Each row in the import file can contain only one of the following bot definitions:
To match a bot using a combination of rules joined with an OR (for example, user agent or IP address), provide an identical name for all rules that you want to combine in the bot name field. AND matches are not supported.
User agent contains or starts with: Provide a single user agent string to match in the Agent Include column. Specify the type of match you want performed by placing contains or starts with in the Agent Match Rule field. An optional value can be included in the Agent Exclude column that defines one or more pipe-delimited (
| ) strings that the Agent does not contain. String matches are case-insensitive. Both the IP Start and IP End columns must be empty.
Single IP address or wildcard match: To match a single IP address (
10.10.10.1) or wildcard IP address (
10.10.*.*), place the same value in both the IP Start and IP End columns. Match Rule, Agent Include, and Agent Exclude must be empty.
IP range match: Define a range of IP addresses using the IP Start and IP End columns. Wildcards can be used to match IP ranges, for example
10.10.20.*. Match Rule, Agent Include, and Agent Exclude must be empty.
On the Bot Rules page in Report Suite Manager, next to the Select CSV Bot file to import area, select Choose File, then select the CSV file that defines the bot rules that you want to import.
(Optional) Select the Overwrite existing rules checkbox to delete all existing rules and replace them with the rules defined in the upload file.
Select Import File.
In the Rule Sets area, review the rules that were imported.
To export all rules defined in the UI in a CSV format:
Go to Analytics > Admin > Report Suites.
Select the report suite that contains the bot rules that you want to export, then select Edit Settings > General > Bot Rules.
Select Export Bot Rules, then save the CSV file to your file system.
Bot Rules are applied to all analytics data. Data removed by Bot Rules is visible only in the Bots and Bot Pages Reports.
VISTA rules are applied after Bot Rules. See Processing order in the Technotes user guide.
High-Hit Visit Processing: If more than 100 hits occur in a visit, reporting determines if the time of the visit in seconds is less than or equal to the number of hits in the visit. In this situation, due to the cost of processing long, intense visits, reporting starts over with a new visit. High-hit visits are typically caused by bot attacks and are not considered normal visitor browsing.
Hits marked as
bots are billed as server calls.
The IAB bot list is based solely on user agent, so filtering based on that list is not impacted by IP obfuscation settings. For non-IAB bot filtering (custom rules), IP may be part of the filtering criteria. If filtering bots using IP, bot filtering happens after the last octet has been removed if that setting is enabled, but before the other IP obfuscation options, such as deleting the entire IP or replacing it with some unique ID.
If IP obfuscation is enabled, IP exclusion happens before the IP address is obfuscated, so customers do not need to change anything when they enable IP obfuscation.
If the last octet is removed, that is done before IP filtering. As such, the last octet is replaced with a 0, and IP exclusion rules should be updated to match IP addresses with a zero on the end. Matching * should match 0.