Multi-site storefronts

You can set up and run several stores with a single implementation of Adobe Commerce on cloud infrastructure. See Set up multiple websites or stores.

The same best practices for configuring the robots.txt and sitemap.xml files for single-site storefronts applies to multi-site storefronts with two important differences:

  • Make sure that the robots.txt and sitemap.xml file names contain the names of the corresponding sites. For example:

    • domaineone_robots.txt
    • domaintwo_robots.txt
    • domainone_sitemap.xml
    • domaintwo_sitemap.xml
  • Use a slightly modified custom Fastly VCL snippet to redirect from the root of your sites to the pub/media location for both files across your sites:

    {
      "name": "sitemaprobots_rewrite",
      "dynamic": "0",
      "type": "recv",
      "priority": "90",
      "content": "if ( req.url.path == \"/robots.txt\" ) { if ( req.http.host ~ \"(domainone|domaintwo).com$\" ) { set req.url = \"pub/media/\" re.group.1 \"_robots.txt\"; }} else if ( req.url.path == \"/sitemap.xml\" ) { if ( req.http.host ~ \"(domainone|domaintwo).com$\" ) {  set req.url = \"pub/media/\" re.group.1 \"_sitemap.xml\"; }}"
    }
    

Adobe Commerce on-premises

Use the Admin application to configure the robots.txt and sitemap.xml files to prevent bots from scanning and indexing unnecessary content (see Search Engine Robots).

TIP
For on-premises deployments, where you write the files depends on how you installed Adobe Commerce. Write the files to /path/to/commerce/pub/media/ or /path/to/commerce/media, whichever is right for your installation.

Security

Do not expose your Admin path in your robots.txt file. Having the Admin path exposed is a vulnerability for site hacking and potential loss of data. Remove the Admin path from the robots.txt file.

For steps to edit the robots.txt file and remove all entries of the Admin path, see Marketing User Guide > SEO and Search > Search Engine Robots.

Additional information

Next pageSecure your site and infrastructure