Rewrite using Fastly VCL snippet

If you have different domains and you need separate site maps, you can create a VCL to route to the proper sitemap. Generate the sitemap.xml file in the Admin panel as described above, then create a custom Fastly VCL snippet to manage the redirect. See Custom Fastly VCL snippets.

NOTE
You can upload custom VCL snippets from the Admin UI or using the Fastly API. See Custom VCL snippet examples and tutorials.

Use a Fastly VCL snippet for redirect

Create a custom VCL snippet to rewrite the path for sitemap.xml to /media/sitemap.xml using the type and content key-value pairs.

{
  "name": "sitemapxml_rewrite",
  "dynamic": "0",
  "type": "recv",
  "priority": "90",
  "content": "if ( req.url.path ~ \"^/?sitemap.xml$\" ) { set req.url = \"/media/sitemap.xml\"; }"
}

The following example demonstrates how to rewrite the path for robots.txt and sitemap.xml to /media/robots.txt and /media/sitemap.xml

{
  "name": "sitemaprobots_rewrite",
  "dynamic": "0",
  "type": "recv",
  "priority": "90",
  "content": "if ( req.url.path ~ \"^/?sitemap.xml$\" ) { set req.url = \"/media/sitemap.xml\"; } else if (req.url.path ~ \"^/?robots.txt$\") { set req.url = \"/media/robots.txt\";}"
}

To use a Fastly VCL snippet for particular domain redirect:

Create a pub/media/domain_robots.txt file, where the domain is domain.com, and use the next VCL snippet:

{
  "name": "domain_robots",
  "dynamic": "0",
  "type": "recv",
  "priority": "90",
  "content": "if ( req.url.path == \"/robots.txt\" ) { if ( req.http.host ~ \"(domain).com$\" ) { set req.url = \"/media/\" re.group.1 \"_robots.txt\"; }}"
}

The VCL snippet routes http://domain.com/robots.txt and presents the pub/media/domain_robots.txt file.

To configure a redirect for robots.txt and sitemap.xml in a single snippet, create pub/media/domain_robots.txt and pub/media/domain_sitemap.xml files, where the domain is domain.com and use the next VCL snippet:

{
  "name": "domain_sitemaprobots",
  "dynamic": "0",
  "type": "recv",
  "priority": "90",
  "content": "if ( req.url.path == \"/robots.txt\" ) { if ( req.http.host ~ \"(domain).com$\" ) { set req.url = \"/media/\" re.group.1 \"_robots.txt\"; }} else if ( req.url.path == \"/sitemap.xml\" ) { if ( req.http.host ~ \"(domain).com$\" ) {  set req.url = \"/media/\" re.group.1 \"_sitemap.xml\"; }}"
}

In the sitemap admin config, you must specify the location of the file using pub/media/ rather than /.

Configure indexing by search engine

To activate robots.txt customizations in Production, you must enable the Indexing by search engines is On for <environment-name> option in your project settings.

Use the Cloud Console to manage environments

NOTE
  • Indexing by search engines can only be enabled in Production, but not in any of the lower environments.

  • If you are using PWA Studio and are unable to access your configured robots.txt file, add robots.txt to the Front Name Allowlist at Stores > Configuration > General > Web > UPWARD PWA Configuration.

Previous pageMultiple sites
Next pagePayPal payment methods

Commerce