Resolve robots.txt not updating or displaying default settings
On Adobe Commerce on cloud infrastructure, robots.txt content is generated dynamically from Admin configuration and stored in the database, so you do not need to create a physical robots.txt file for this scenario.
When configured robots.txt content still displays the default content or fails to update, ensure that indexing by search engines is enabled.
Description description
Environment
Adobe Commerce on cloud infrastructure 2.4.x
Issue
Unable to change the default robots.txt settings in Adobe Commerce.
Steps to reproduce:
- Access the Admin panel.
- Navigate to Content
>Design>Configuration, then edit the Custom instruction of robots.txt field. - Add custom content (for example, the text “hello”) and save the changes.
- Visit the
robots.txtURL.
Expected result:
The robots.txt file displays the saved custom content.
Actual result:
The robots.txt file does not change and continues to show the default content.
Cause
Indexing by search engines is disabled, which prevents the custom robots.txt content from being applied.
Resolution resolution
Method 1: On the Adobe Commerce Cloud Console, configure indexing by search engines as described in our developer documentation.
Method 2: Using the magento-cloud CLI, run the following command:
magento-cloud environment:info -p <cluster> -e production restrict_robots false