robots.txt: When Allow and Disallow Conflict, Which One Wins?
Sometimes you want to block an entire admin folder with Disallow: /wp-admin/, but carve out one specific file underneath it with an Allow exception. When two rules give opposite instructions for the same path, which one does a crawler follow? The order they appear in the file doesn't matter — there's a fixed rule that decides it.
1. The core rule: the longer (more specific) path wins
Google and other major crawlers resolve Allow/Disallow conflicts in robots.txt based on path string length. Within the same User-agent block, if multiple rules match a given URL, the rule whose matched path pattern is longer — i.e., more specific — wins. It doesn't matter whether Allow or Disallow appears first in the file; order has no effect on the outcome.
Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
/wp-admin/ is 7 characters, while /wp-admin/admin-ajax.php is 23 characters. Since the Allow rule matches a longer path, it wins: everything under /wp-admin/ is blocked, except admin-ajax.php, which remains crawlable. This mechanism is exactly why WordPress recommends this combination by default — admin-ajax.php is a public endpoint that handles front-end AJAX requests, and some pages only work correctly if search engines can reach it.
2. What happens when the lengths are equal: Allow wins
In the rare case where both rules match paths of exactly the same length, Google's documentation explicitly states that the less restrictive rule (Allow) takes priority. In other words, ties are designed to favor "opening up" over "blocking."
3. Why length, not order
robots.txt existed for decades as an informal convention with no official standard, until the IETF formally standardized it as RFC 9309 in 2022. Before standardization, different crawlers resolved conflicts inconsistently — some followed "first rule wins," others "last rule wins." Google documented its own "longer path (more specific) wins" policy early on, and that approach was effectively adopted into RFC 9309, cementing it as the de facto industry standard. That said, there's no guarantee every crawler implements the RFC identically, so for any exception rule that really matters, it's safer to verify the actual behavior of each crawler individually.
4. Practical tips
- Make exceptions clearly longer: if you want to open up just one sub-path, make sure the Allow rule's path is unambiguously longer and more specific — in character count — than the Disallow rule.
- Don't rely on file order: the assumption that "writing this rule later makes it win" is wrong. Order has no bearing on the outcome.
- Verify actual behavior with tools: the most reliable approach is to directly test whether your intended path is actually allowed or blocked, using the robots.txt report or URL Inspection tool in Google Search Console.
Frequently Asked Questions
Q. Does Allow or Disallow always win?
A. Neither one always wins outright — whichever rule matches a longer (more specific) path takes priority. If Allow's matched path is longer, Allow wins; if Disallow's is longer, Disallow wins.
Q. Does changing the order of the rules change the outcome?
A. No. Google and other major crawlers resolve conflicts based on path length, so the order the rules appear in the file has no effect on the outcome.
Q. Do all search engine crawlers follow this rule identically?
A. Since RFC 9309 was standardized in 2022, most major crawlers follow a similar approach, but because robots.txt operated as an informal convention for so long, implementation details can still vary between crawlers. Verify important rules with an actual testing tool.
Q. What if the paths in Disallow: /wp-admin/ and Allow: /wp-admin/ are exactly identical?
A. Since the lengths are equal, the tie-breaking rule applies, and the less restrictive Allow rule takes priority.