DocumentationCommerceCommerce on Cloud Infrastructure Guide

Block referral spam

Last update: Thu Apr 27 2023 00:00:00 GMT+0000 (Coordinated Universal Time)

CREATED FOR:

  • Admin
  • Developer

The following example shows how to configure Fastly Edge Dictionary with a custom VCL snippet to block referral spam from your Adobe Commerce on cloud infrastructure site.

NOTE
We recommend adding custom VCL configurations to a Staging environment where you can test them before running them against the Production environment.

Prerequisites:

  • Your environment must be configured to use the Fastly CDN. See Configure Fastly services.

  • Ensure that you are running the latest version of the Fastly CDN module for Magento 2. See Upgrade the Fastly Module.

  • Verify the environment configuration for the Fastly service. See Check Fastly caching.

  • You must have Admin credentials to access the Staging and Production environments.

  • Review your site logs for fake referral URLs, and make a list of domains to block.

Create a referrer blocklist

Edge Dictionaries create key-value pairs accessible to VCL functions during VCL snippet processing. In this example, you create an edge dictionary that provides the list of referrer websites to block.

  1. Log in to the Admin.

  2. Click Stores > Settings > Configuration > Advanced > System.

  3. Expand Full Page Cache > Fastly Configuration > Edge dictionaries.

  4. Create the Dictionary container:

    • Click Add container.

    • On the Container page, enter a Dictionary name—referrer_blocklist.

    • Select Activate after the change to deploy your changes to the version of the Fastly service configuration that you are editing.

    • Click Upload to attach the dictionary to your Fastly service configuration.

  5. Add the list of domain names to block to the referrer_blocklist dictionary:

    • Click the Settings icon for the referrer_blocklist dictionary.

    • Add and save key-value pairs in the new dictionary. For this example, each Key is the domain name of a referrer URL to block and Value is true.

      Add bad referrer dictionary items

    • Click Cancel to return to the system configuration page.

  6. Click Save Config.

  7. Refresh the cache according to the notification at the top of the page.

For more information about Edge Dictionaries, see Creating and using Edge Dictionaries and custom VCL snippets in the Fastly documentation.

Create a custom VCL snippet to block referrer spam

The following custom VCL snippet code (JSON format) shows the logic to check and block requests. The VCL snippet captures the host of a referrer website into a header, and then compares the host name to the list of URLs in the referrer_blocklist dictionary. If the host name matches, the request is blocked with a 403 Forbidden error.

{
  "name": "block_bad_referrer",
  "dynamic": "0",
  "type": "recv",
  "priority": "5",
  "content": "set req.http.Referer-Host = regsub(req.http.Referer, \"^https?:\/\/?([^:\/s]+).*$\", \"\\1\"); if (table.lookup(referrer_blocklist, req.http.Referer-Host)) { error 403 \"Forbidden\"; }"
}

Before creating a snippet based on this example, review the values to determine whether you need to make any changes:

  • name — Name for the VCL snippet. For this example, we used block_bad_referrer.

  • dynamic — Value 0 indicates a regular snippet to upload to the versioned VCL for the Fastly configuration.

  • priority — Determines when the VCL snippet runs. The priority is 5 to run this snippet code before any of the default Magento VCL snippets (magentomodule_*) assigned a priority of 50. Set the priority for each custom snippet higher or lower than 50 depending on when you want your snippet to run. Snippets with lower priority numbers run first.

  • type — Specifies a location to insert the snippet in the VCL version. In this example, the VCL snippet is a recv snippet. When the snippet is inserted into the VCL version, it is added to the vcl_recv subroutine, below the default Fastly VCL code and above any objects.

  • content — The snippet of VCL code to run in one line, without line breaks.

After reviewing and updating the code for your environment, use either of the following methods to add the custom VCL snippet to your Fastly service configuration:

Add the custom VCL snippet

  1. Log in to the Admin.

  2. Click Stores > Settings > Configuration > Advanced > System.

  3. Expand Full Page Cache > Fastly Configuration > Custom VCL Snippets.

  4. Click Create Custom Snippet.

  5. Add the VCL snippet values:

    • Name — block_bad_referrer

    • Type — recv

    • Priority — 5

    • VCL snippet content —

      code language-conf 
      set req.http.Referer-Host = regsub(req.http.Referer,
"^https?://?([^:/\s]+).*$", "1");
if (table.lookup(referrer_blocklist, req.http.Referer-Host)) {
  error 403 "Forbidden";
}

  6. Click Create.

    Create custom referrer block VCL snippet

  7. After the page reloads, click Upload VCL to Fastly in the Fastly Configuration section.

  8. After the upload completes, refresh the cache according to the notification at the top of the page.

Fastly validates the updated VCL version during the upload process. If the validation fails, edit your custom VCL snippet to fix any issues. Then, upload the VCL again.

NOTE
Instead of manually uploading custom VCL snippets, you can add snippets to the $MAGENTO_CLOUD_APP_DIR/var/vcl_snippets_custom directory in your environment. Snippets in this directory upload automatically when you click upload VCL to Fastly in the Commerce Admin. See Automated custom VCL snippets deployment in the Fastly CDN module for Magento 2 documentation.

Modify the custom VCL snippet

  1. Log in to the Admin.

  2. Click Stores > Settings > Configuration > Advanced > System.

  3. Expand Full Page Cache > Fastly Configuration > Custom VCL Snippets.

    Manage custom VCL snippets

  4. In the Action column, click the settings icon next to the snippet to edit.

  5. After the page reloads, click Upload VCL to Fastly in the Fastly Configuration section.

  6. After the upload completes, refresh the cache according to the notification at the top of the page.

WARNING
The Custom VCL snippets UI option shows only the snippets added through the Adobe Commerce Admin. If you add snippets using the Fastly API, use the API to manage them.

Delete the custom VCL snippet

  1. Log in to the Admin.

  2. Click Stores > Settings > Configuration > Advanced > System.

  3. Expand Full Page Cache > Fastly Configuration > Custom VCL Snippets.

    Manage custom VCL snippets

  4. In the Action column, click the trash icon next to the snippet to delete.

  5. On the next modal window, click DELETE and activate a new version.

WARNING
The Custom VCL snippets UI option shows only the snippets added through the Adobe Commerce Admin. If you add snippets using the Fastly API, use the API to manage them.
recommendation-more-help
05f2f56e-ac5d-4931-8cdb-764e60e16f26