DocumentationExperience Cloud KCS

Customize robots.txt file

Last update: August 18, 2026

Description description

Search engines index all pages, but you want to exclude some pages, images and/or files from the index and allow others.

This is similar to: https://nation.marketo.com/t5/Knowledgebase/Preventing-Search-Crawlers-on-a-Page-Level-Basis/ta-p/248226

Resolution resolution

  1. Author a custom robots.txt with your web developer and verify it is correct
  2. Upload it to Design Studio and copy the URL. https://docs.marketo.com/display/public/DOCS/Add+Images+and+Files+to+Marketo
  3. Add a Redirect Rule, with Original URL: go.rest.com.au/robots.txt (This will apply to all landing pages and assets on https://go.rest.com.au) and ‘Redirect URL’ < paste the URL from Design Studio> https://docs.marketo.com/display/public/DOCS/Redirect+a+URL+Path
recommendation-more-help
experience-cloud-kcs-help-kbarticles