Multi-site storefronts
You can set up and run several stores with a single implementation of Adobe Commerce on cloud infrastructure. See Set up multiple websites or stores.
The same best practices for configuring the robots.txt
and sitemap.xml
files for single-site storefronts applies to multi-site storefronts with two important differences:
-
Make sure that the
robots.txt
andsitemap.xml
file names contain the names of the corresponding sites. For example:domaineone_robots.txt
domaintwo_robots.txt
domainone_sitemap.xml
domaintwo_sitemap.xml
-
Use a slightly modified custom Fastly VCL snippet to redirect from the root of your sites to the
pub/media
location for both files across your sites:{ "name": "sitemaprobots_rewrite", "dynamic": "0", "type": "recv", "priority": "90", "content": "if ( req.url.path == \"/robots.txt\" ) { if ( req.http.host ~ \"(domainone|domaintwo).com$\" ) { set req.url = \"pub/media/\" re.group.1 \"_robots.txt\"; }} else if ( req.url.path == \"/sitemap.xml\" ) { if ( req.http.host ~ \"(domainone|domaintwo).com$\" ) { set req.url = \"pub/media/\" re.group.1 \"_sitemap.xml\"; }}" }
Adobe Commerce on-premises
Use the Admin application to configure the robots.txt
and sitemap.xml
files to prevent bots from scanning and indexing unnecessary content (see Search Engine Robots).
/path/to/commerce/pub/media/
or /path/to/commerce/media
, whichever is right for your installation.Security
Do not expose your Admin path in your robots.txt
file. Having the Admin path exposed is a vulnerability for site hacking and potential loss of data. Remove the Admin path from the robots.txt
file.
For steps to edit the robots.txt
file and remove all entries of the Admin path, see Marketing User Guide > SEO and Search > Search Engine Robots.