Configuring robots.txt: Rules for AI and Non-English Characters

Google Search Central establishes that robots.txt functions as an access preference protocol governed by the RFC 9309 standard. This rule fails if a crawler is not compliant with that specific internet standard or if a user treats the file as a legal data-usage policy for generative AI. Technical SEO practitioners and site owners should act on these guidelines.

Who should act on robots.txt guidelines and who should skip it

Readers who should act on robots.txt guidelines

Technical SEO practitioners should act on these guidelines to ensure crawl rules are correctly parsed and don’t cause unexpected indexing issues.
Web developers should act on these guidelines to ensure the technical implementation of the robots.txt file uses standard-compliant syntax.
International SEOs should act on these guidelines to ensure non-ASCII characters in URL paths are correctly interpreted by all crawlers.
Site owners wanting to block AI should act on these guidelines to understand the limitations of robots.txt regarding AI data usage.

Readers who should skip the robots.txt guidelines

General website visitors should skip this as they do not interact with crawler permissions.
Social media marketers should skip this as robots.txt does not affect social media reach or engagement.

What to do next about robots.txt

Audit current robots.txt against RFC 9309 standards to ensure compliance with official internet standards.
Identify specific AI crawler user-agent tokens to target them explicitly in the file.
Convert non-ASCII URL paths to percent-encoded strings to ensure compatibility across the entire global internet infrastructure.

Adhering to the RFC 9309 Standard

The robots.txt file must follow the official internet standard, RFC 90309, to ensure reliable crawler behavior. The parser reads the file line-by-line and looks for specific commands like User-agent, Allow, Disallow, and Sitemap.

If you write a rule that the parser does not understand, the system will silently discard the line it doesn’t understand and move on. This means a “creative” syntax or non-standard spacing can lead to a failure of the rule to trigger.

Google follows the RFC 9309 standard for its own parsers, but other or older crawlers may not be as flexible. Keeping your robots.txt “beautifully boring” ensures that every bot that speaks robots.txt respects your boundaries.

What is RFC 9309?

RFC 9309 is the official internet standard that defines the robots.txt file format. It ensures that well-behaved crawlers interpret your rules correctly by defining the specific syntax and characters allowed in the file.

Handling Non-English Characters and Emojis

You can use emojis and non-English characters in robots.txt, but the placement of these characters determines how they are processed. User-agent tokens can only contain letters and dashes because emojis and non-English scripts are multibyte UTF-8 characters that do not fit the allowed character set.

While Googlebot handles international scripts and emojis by automatically translating them, other crawlers may get confused by raw non-ASCII characters. To be safe across the global internet, you should use the percent-encoded version of these characters in your robots.txt rules.

For example, if you want to block a path like /🍣-gear, you must find the percent-encoded equivalent of the emoji. The parser will then recognize the path correctly regardless of the crawler’s local encoding capabilities.

Allowed vs. Disallowed in robots.txt

Item	Allowed	Not Allowed / Requirement
User-agent tokens	Letters and dashes	Emojis, multibyte UTF-8 characters
URL paths	Non-English characters, emojis	Must be percent-encoded for global compatibility
Comments	Any text	Must start with a hash (#)

The Limits of robots.txt for Generative AI

Robots.txt is an access preference protocol, not a data usage or copyright policy. Gary from Google Search Relations explains that the file only tells a bot whether it can crawl a page or not. It does not dictate what a system does with the data after it has already crawled it.

Think of robots.txt as a “Keep Out” sign on a door rather than a legal contract. It tells the person they can’t enter, but it doesn’t stop them from using a photo they already took. If you let an AI bot crawl your site, the bot processes the robots.txt as a machine-readable file to determine entry, not as a set of rules for how the bot can use the data once it has it.

To stop a specific AI crawler from accessing your pages entirely, you must target their specific user-agent tokens explicitly. You cannot write a fancy custom rule like “no-ai-training-allowed” because the bot’s parser will throw any non-standard instruction in the trash. If you want to prevent data usage, you must look toward other methods beyond the robots.txt file.

How robots.txt differs from the noindex tag

The robots.txt file and the noindex meta tag serve different purposes in crawl management. robots.txt prevents the crawler from entering the page (crawling), while noindex tells the crawler not to include the page in the index.

Because robots.txt only blocks the entry to the room, a page that is disallowed in robots.txt might still be indexed if other sources link to it. In contrast, a noindex tag allows the crawler to enter the room but prevents it from putting the content in its public library.

Best Practices for a “Beautifully Boring” robots.txt

The most effective robots.txt file is one that avoids “creative” syntax in favor of strict standard compliance. You should avoid using non-standard rules that the parser might silently discard, as this can lead to your restricted pages being crawled anyway.

Always use percent-encoding for non-English characters in your paths to ensure the rules are understood by all bots. If you need to include decorative elements like ASCII art, place them behind hash (#) symbols so they are ignored by the crawler logic. Stick to the standard to ensure your site boundaries are respected by every bot that speaks robots.txt.

What is percent-encoding?

Percent-encoding is the process of translating non-ASCII characters, such as emojis or non-English scripts, into a standard %-hex format. It ensures that crawlers can correctly interpret these characters in URL paths without encountering encoding errors.

A Developer’s Guide to Japanese Path Encoding

Suppose a developer named Sarah manages a site with a Japanese path: /🍣-gear. Sarah wants to ensure all crawlers respect this path and avoid crawling the folder. She knows she cannot put a raw emoji in a user-agent token, but she also knows that while Googlebot might handle the emoji, other crawlers might not.

Sarah converts the emoji to its percent-encoded version in the robots.txt file. The robots.txt is now globally compatible, and crawlers correctly interpret the path regardless of their local encoding capabilities. If Sarah had used the raw emoji without percent-encoding, some crawlers might have ignored the disallow line entirely and crawled the folder anyway.

Google has not announced third-party AI crawler behavior for non-standard UTF-8 characters and emojis

Google has not announced the specific behavior of all third-party AI crawlers when encountering non-standard UTF-8 characters.
The creators have not stated the exact percentage of crawlers that fail to parse non-percent-encoded emojis.

Frequently Asked Questions

Can I use emojis in my robots.txt file?

The use of emojis in robots.txt depends on where they are placed in the file. Emojis are not allowed in user-agent tokens because they are multibyte UTF-8 characters, but they can be used in URL paths if they are percent-encoded.

Does robots.txt stop AI models from using my content for training?

The robots.txt file does not stop AI models from using your content for training. It is an access preference protocol that tells a bot whether it can enter a page, but it does not dictate what the bot does with the data once it has crawled it.

What happens if I write a rule that the crawler doesn’t understand?

The parser will simply discard the line it doesn’t understand and move on to the next rule. This can result in the crawler accessing a page you intended to block because the rule was not processed by the machine.

How do I handle non-English characters in my robots.txt?

To handle non-English characters in your robots.txt, you should use the percent-encoded version of the characters in your paths. This ensures that well-behaved crawlers respect your boundaries regardless of their local encoding capabilities.

Details first shown in What can you actually put in a robots.txt? by Google Search Central.