robots.txt not updated or displaying default settings
The article provides a solution for when you have configured robots.txt
correctly, for example per Best practices for Adobe Commerce robots.txt but the robots.txt
is not getting updated or is displaying the default settings.
Affected products and versions
- Adobe Commerce on cloud infrastructure 2.3.x, 2.4.x
Issue
Cannot change the default robots.txt
setting.
Steps to reproduce:
- Access the Admin panel.
- Add content to Content > Design > Configuration > Edit Custom instruction of
robots.txt
file such as the text “hello” and save the changes. - Visit the
robots.txt
url.
Expected result:robots.txt
has the saved text.
Actual result:
robots.txt
file does not change.
Cause
Indexing by search engines is turned off.
Solution
Enable indexing by search engines. See Configure indexing by search engine in our developer documentation.
Related reading
- Add site map and search engine robots in our developer documentation.
recommendation-more-help
8bd06ef0-b3d5-4137-b74e-d7b00485808a