What Is robots.txt and How Do I Write One?
robots.txt is a text file at the root of your site that tells search engine crawlers which pages they may fetch. It cannot force a page out of Google, but it controls crawl budget and can block private areas like staging environments.
A minimal safe robots.txt
Allow everything by default, block nothing unless you have a reason, and declare your sitemap. Two lines do it: User-agent: * with Disallow: (empty), plus a Sitemap: line with your full sitemap URL. Common blocks include /cart, /checkout and admin paths.
The mistake that kills rankings
A stray Disallow: / blocks your entire site from Google. It happens during migrations and staging deployments. Check yours with the free Sitemap and Robots Checker, which reads your live file and flags blocks that could stop indexing.
Frequently Asked Questions
Does robots.txt remove pages from Google?
No. Use noindex for that. robots.txt only stops crawling.
Where does it live?
At https://yourdomain.com/robots.txt, nothing else counts.
Related reading
Put these tactics to work
Run InRankSEO's free SEO tools to audit your site, research keywords, and track your visibility across Google and AI search.