This tool fetches a site’s robots.txt file and helps surface sitemap references. Robots.txt is a crawl-control file; a sitemap is a discovery file that lists URLs you want search engines to know about.
Robots.txt is not an access-control system
A Disallow rule asks compliant crawlers not to crawl a path. It does not make the path private, and the URL can sometimes still appear in search results if discovered elsewhere. Use authentication or proper authorization for sensitive content.
What to inspect?
Look for accidental blocks of important CSS/JS or content directories, unexpected wildcard rules, and sitemap locations. Then verify the actual page status with the search engine’s inspection tools. A robots rule and a page-level noindex directive solve different problems.
Sitemap quality
A sitemap should generally contain canonical, indexable URLs that return successful responses. Thin, duplicate, redirected, error, or intentionally non-indexable URLs usually do not belong in the primary sitemap because they are poor crawl targets.
Robots rules control crawling, not privacy
A Disallow directive asks compliant crawlers not to fetch a path; it does not authenticate the URL and does not guarantee the URL can never appear in search. Sensitive resources require real access control. Use page-level indexing directives and canonicals for different jobs.
A useful sitemap is selective
Submit canonical, indexable URLs that return successful responses. Redirects, errors, duplicate variants, and intentionally non-indexable pages are poor sitemap targets because they send mixed crawling signals.