How to Create a Robots.txt File (With Examples for WordPress and Shopify)

By Abdur RehmanNovember 1, 20263 min read

Quick answer: robots.txt is a plain text file at yourdomain.com/robots.txt that tells search engine crawlers which parts of your site they may visit. A safe file for most sites is just three lines: User-agent: *, Disallow: (with nothing after it, meaning "allow everything"), and Sitemap: https://yourdomain.com/sitemap.xml.

Every time Googlebot, Bingbot or another crawler visits your site, it checks robots.txt first. Getting this one small file right helps search engines spend their time on your important pages. Getting it wrong can hide your whole website from Google. Here's how to create one safely.

🤖
Robots.txt Generator

Create a robots.txt file for your website, with rules for search engines and AI crawlers and your sitemap.

Open the tool →

How robots.txt works

A robots.txt file is made of groups. Each group starts with a User-agent line naming the crawler (or * for all crawlers), followed by rules:

  • Disallow: /path/ — don't crawl anything that starts with this path.
  • Allow: /path/ — an exception inside a disallowed folder.
  • Sitemap: https://… — where your XML sitemap lives (can go anywhere in the file).
  • Crawl-delay: 10 — seconds between requests. Bing and Yandex respect it; Google ignores it.

Paths are case-sensitive and match from the start of the URL. Disallow: /admin also blocks /admin-login and /administrator, so end folder paths with a slash when you mean just the folder.

How to create a robots.txt file in 3 steps

  1. Generate it. In the robots.txt generator, choose whether to allow or block all robots by default, list any folders to keep crawlers out of, add your sitemap URL, and optionally tick the box to block AI training crawlers.
  2. Download it. Click download to get a file named exactly robots.txt (all lowercase, plain text, UTF-8).
  3. Upload it to the root of your site so it opens at https://yourdomain.com/robots.txt, then open that address in your browser to check.

Robots.txt examples you can copy

Allow everything (recommended for most sites)

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

Block a few private folders

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Block AI training crawlers but allow search engines

User-agent: *
Disallow:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Google-Extended controls whether Google can use your content for its Gemini models; blocking it does not affect your Google Search rankings. Blocking AI crawlers is a business choice — some sites want the visibility that AI answers can bring, others don't want their content used for training.

Block the whole site (staging or development sites only)

User-agent: *
Disallow: /

⚠️ This is the single most common robots.txt disaster: a developer blocks the staging site, and the same file gets copied to the live site. If your traffic suddenly drops to zero after a redesign, check robots.txt first.

Robots.txt on WordPress

WordPress creates a virtual robots.txt automatically. To customise it:

  • Yoast SEO: Yoast → Tools → File editor → Create robots.txt file, paste your rules and save.
  • Rank Math: Rank Math → General Settings → Edit robots.txt.
  • Without a plugin: upload a robots.txt file to your site's root folder (where wp-config.php is) with your host's file manager or FTP. A real file overrides the virtual one.

A typical WordPress file blocks /wp-admin/ but allows /wp-admin/admin-ajax.php, because some themes and plugins need it. Don't block /wp-content/ or /wp-includes/ — Google needs your CSS and JavaScript to render pages properly.

Robots.txt on Shopify

Shopify generates a sensible robots.txt for every store, blocking cart, checkout and internal search pages. To change it, go to Online Store → Themes → Edit code → Add a new template → choose robots.txt. This creates robots.txt.liquid, where you can add rules. Most stores never need to.

Common robots.txt mistakes

  • Using robots.txt to hide a page from Google. Blocked pages can still be indexed from links. Use a noindex meta tag instead — the meta tag generator can create one — and make sure the page is not blocked, or Google will never see the tag.
  • Blocking CSS, JavaScript or images that pages need to render.
  • Putting the file in the wrong place — it must be at the root, not /blog/robots.txt.
  • Wrong file name — it must be robots.txt, lowercase, not Robots.TXT or robots.txt.txt.
  • Treating it as security. robots.txt is public and only well-behaved bots follow it. Protect private areas with a password, not a Disallow line.

How to check your robots.txt is working

  1. Visit yourdomain.com/robots.txt — you should see your rules as plain text.
  2. In Google Search Console, open Settings → robots.txt to see when Google last fetched it and whether there were errors.
  3. Use URL Inspection on an important page; it tells you if robots.txt is blocking it.

Frequently asked questions

Where do I put robots.txt?

In the root of your domain, so it loads at https://yourdomain.com/robots.txt. A robots.txt file in a subfolder is ignored. Each subdomain (like blog.yourdomain.com) needs its own file.

Does robots.txt stop a page from appearing in Google?

Not reliably. It stops Google from crawling the page, but a blocked URL can still be indexed (without a description) if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag instead.

Do I need a robots.txt file?

It's not required — without one, search engines crawl everything they can find. But a simple file that allows everything and lists your sitemap is recommended, and it stops 404 errors in your logs.

How do I block ChatGPT and other AI crawlers?

Add a separate group for each AI crawler, such as "User-agent: GPTBot" followed by "Disallow: /". The robots.txt generator can add the common AI crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended and others) with one checkbox.

How do I test my robots.txt?

Open yourdomain.com/robots.txt in a browser to check it is live, then check Google Search Console under Settings → robots.txt, which shows the version Google fetched and any errors.

Written by Abdur Rehman

ERP developer and AI & automation engineer who builds and maintains every tool on WebToolNest. More about me.