DocumentationExperience Cloud KCS

AEM solution for preventing search engine indexing using Robots.txt and X‑Robots‑Tag Headers

Last update: July 6, 2026

Adobe Experience Manager as a Cloud Service environments, pages, assets, and generated endpoints can appear in search results when crawl directives are missing or applied at the wrong layer. You can prevent indexing by configuring robots.txt, applying X-Robots-Tag response headers, and using dispatcher or CDN rules for specific paths, file types, or *.plain.html endpoints.

Description description

Environment

Adobe Experience Manager as a Cloud Service

Issue/Symptoms

Some Adobe Experience Manager environments, pages, assets, or system-generated endpoints can appear in search engine results when search engine blocking directives are missing, incomplete, or applied at the wrong layer. This is commonly seen in non-production environments or when specific resources such as PDF files or generated endpoints are still crawlable.

  • Non-production domains appear in search results.
  • Specific pages or assets, such as PDF files, are indexed even though indexing restrictions were expected.
  • The X-Robots-Tag response header is missing for some URLs.
  • System-generated endpoints such as *.plain.html do not return the intended search engine directives.

You can confirm the issue safely by checking the affected URLs and response headers:

  • Open the site’s /robots.txt file and confirm that the expected crawl directives are present.
  • Use a header check such as curl -I https://<domain>/<path> to verify whether the response includes X-Robots-Tag.
  • Test representative URLs for each affected resource type, such as standard pages, PDF files, forms-related paths, and *.plain.html endpoints.
  • Compare the results across the URLs that should be blocked and those that should remain crawlable.

Resolution resolution

Try the following steps to solve the issue:

  1. Identify what must be blocked from indexing, such as an entire non-production environment, selected pages, PDF assets, forms-related paths, or *.plain.html endpoints. The expected result is a clear list of affected URL types, which you can verify by checking which URLs appear in search results or should not be indexed.
  2. Configure robots.txt for environment-wide blocking when needed. The expected result is that https://<environment-domain>/robots.txt returns the intended crawl directives, which you can verify by opening the live file and reviewing its contents.
  3. Apply an X-Robots-Tag header through dispatcher or CDN configuration when you need response-level control for specific domains, paths, or resource types. The expected result is that affected responses include a directive such as X-Robots-Tag: noindex, nofollow, which you can verify with a header check on the affected URLs.
  4. Use path-based rules if only selected pages, folders, or file types should be excluded from search results instead of blocking the full environment. The expected result is that only the targeted resources return the blocking directive, which you can verify by comparing responses for targeted and non-targeted URLs.
  5. Apply X-Robots-Tag rules to PDF files or other downloadable assets if those resources are still being indexed. The expected result is that matching asset URLs consistently return the no-index header, which you can verify by testing several asset URLs directly.
  6. Exclude forms-related paths with robots.txt directives and supporting dispatcher or CDN rules when those URLs should not be crawled. The expected result is that search engines are instructed not to crawl those form URLs, which you can verify by reviewing the live robots.txt rules and any related response headers.
  7. Use CDN response rules for *.plain.html endpoints when those generated URLs need search engine blocking directives. The expected result is that these endpoints return the intended X-Robots-Tag value, which you can verify by checking the response headers after deployment.
  8. Redeploy the configuration and retest the affected URLs. The expected result is that the updated directives are visible in live responses and match the intended blocking behavior, which you can verify by rechecking both the live robots.txt output and response headers.

If the following symptoms occur and the troubleshooting steps produce the following results, submit a support request at Adobe Support:

  • The correct robots.txt content is deployed, but the live environment still serves different crawl directives.
  • The expected X-Robots-Tag header is configured, but it does not appear in live responses after redeployment.
  • Only some URLs receive the expected header even though the same rule should apply to the full path or file pattern.
  • *.plain.html or other generated endpoints continue to return unexpected search engine directives after CDN configuration updates.

When contacting Adobe Support, include the affected URL, the live response headers, the current robots.txt content, and the relevant dispatcher or CDN configuration snippet.

recommendation-more-help
experience-cloud-kcs-help-kbarticles