robots.txt not updated or displaying default settings

Last update: June 18, 2025

This article provides a solution for cases where the robots.txt file in Adobe Commerce is configured correctly, but still displays default settings or fails to update. To resolve this issue, ensure that indexing by search engines is enabled.

Description

Environment

Adobe Commerce on cloud infrastructure 2.3.x, 2.4.x

Issue

Unable to change the default robots.txt settings in Adobe Commerce.

Steps to reproduce:

  1. Access the Admin panel.
  2. Navigate to Content > Design > Configuration, then edit the Custom instruction of robots.txt field.
  3. Add custom content (for example, the text “hello”) and save the changes.
  4. Visit the robots.txt URL.

Expected result:
The robots.txt file displays the saved custom content.

Actual result:
The robots.txt file does not change and continues to show the default content.

Cause

Indexing by search engines is disabled, which prevents the custom robots.txt content from being applied.

Resolution

Enable indexing by search engines. For detailed instructions, see Configure indexing by search engine in our developer documentation.

Add site map and search engine robots

3d58f420-19b5-47a0-a122-5c9dab55ec7f