Rewrite using Fastly VCL snippet
If you have different domains and you need separate site maps, you can create a VCL to route to the proper sitemap. Generate the sitemap.xml
file in the Admin panel as described above, then create a custom Fastly VCL snippet to manage the redirect. See Custom Fastly VCL snippets.
Use a Fastly VCL snippet for redirect
Create a custom VCL snippet to rewrite the path for sitemap.xml
to /media/sitemap.xml
using the type
and content
key-value pairs.
{
"name": "sitemapxml_rewrite",
"dynamic": "0",
"type": "recv",
"priority": "90",
"content": "if ( req.url.path ~ \"^/?sitemap.xml$\" ) { set req.url = \"/media/sitemap.xml\"; }"
}
The following example demonstrates how to rewrite the path for robots.txt
and sitemap.xml
to /media/robots.txt
and /media/sitemap.xml
{
"name": "sitemaprobots_rewrite",
"dynamic": "0",
"type": "recv",
"priority": "90",
"content": "if ( req.url.path ~ \"^/?sitemap.xml$\" ) { set req.url = \"/media/sitemap.xml\"; } else if (req.url.path ~ \"^/?robots.txt$\") { set req.url = \"/media/robots.txt\";}"
}
To use a Fastly VCL snippet for particular domain redirect:
Create a pub/media/domain_robots.txt
file, where the domain is domain.com
, and use the next VCL snippet:
{
"name": "domain_robots",
"dynamic": "0",
"type": "recv",
"priority": "90",
"content": "if ( req.url.path == \"/robots.txt\" ) { if ( req.http.host ~ \"(domain).com$\" ) { set req.url = \"/media/\" re.group.1 \"_robots.txt\"; }}"
}
The VCL snippet routes http://domain.com/robots.txt
and presents the pub/media/domain_robots.txt
file.
To configure a redirect for robots.txt
and sitemap.xml
in a single snippet, create pub/media/domain_robots.txt
and pub/media/domain_sitemap.xml
files, where the domain is domain.com
and use the next VCL snippet:
{
"name": "domain_sitemaprobots",
"dynamic": "0",
"type": "recv",
"priority": "90",
"content": "if ( req.url.path == \"/robots.txt\" ) { if ( req.http.host ~ \"(domain).com$\" ) { set req.url = \"/media/\" re.group.1 \"_robots.txt\"; }} else if ( req.url.path == \"/sitemap.xml\" ) { if ( req.http.host ~ \"(domain).com$\" ) { set req.url = \"/media/\" re.group.1 \"_sitemap.xml\"; }}"
}
In the sitemap
admin config, you must specify the location of the file using pub/media/
rather than /
.
Configure indexing by search engine
To activate robots.txt
customizations in Production, you must enable the Indexing by search engines is On for <environment-name>
option in your project settings.
-
Indexing by search engines can only be enabled in Production, but not in any of the lower environments.
-
If you are using PWA Studio and are unable to access your configured
robots.txt
file, addrobots.txt
to the Front Name Allowlist at Stores > Configuration > General > Web > UPWARD PWA Configuration.