Custom Domain not Indexed in Search Engines

Last Updated: Sep 10, 2025

Overview

When using a custom domain for an Auth0 tenant, it is possible to experience issues with Google indexing due to a restrictive robots.txt file being hosted. This article explains why the restriction is in place.

Applies To

Custom Domain
Search Engine Indexing
robots.txt

Cause

The custom domain should be used only for Auth0 API endpoints (Authentication API, Management API), which should not be crawled or indexed by search engines.

Solution

The presence of the robots.txt resource in an Auth0 domain discourages automated web crawling mechanisms (such as search engines like Google) from crawling the domain because no content in an Auth0 domain can benefit from web indexing.