robotparser reads empty robots.txt file as "all denied"

@terryjreedy

BPO	35457
Nosy	@terryjreedy, @berkerpeksag, @tirkarthi, @andreburgaud

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-12-11.09:30:47.872>
labels = ['3.8', 'type-feature', 'library']
title = 'robotparser reads empty robots.txt file as "all denied"'
updated_at = <Date 2020-01-02.15:45:55.440>
user = 'https://bugs.python.org/larsfuse'

bugs.python.org fields:

activity = <Date 2020-01-02.15:45:55.440>
actor = 'gallicrooster'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-12-11.09:30:47.872>
creator = 'larsfuse'
dependencies = []
files = []
hgrepos = []
issue_num = 35457
keywords = []
message_count = 6.0
messages = ['331595', '331870', '331963', '359180', '359185', '359202']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'berker.peksag', 'xtreak', 'larsfuse', 'gallicrooster']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35457'
versions = ['Python 3.8']

Linked PRs

The standard (http://www.robotstxt.org/robotstxt.html) says:

To allow all robots complete access:
User-agent: *
Disallow:
(or just create an empty "/robots.txt" file, or don't use one at all)

Here I give python an empty file:
$ curl http://10.223.68.186/robots.txt
$

Code:

rp = robotparser.RobotFileParser()
print (robotsurl)
rp.set_url(robotsurl)
rp.read()
print( "fetch /", rp.can_fetch(useragent = "*", url = "/"))
print( "fetch /admin", rp.can_fetch(useragent = "*", url = "/admin"))

Result:

$ ./test.py
http://10.223.68.186/robots.txt
('fetch /', False)
('fetch /admin', False)

And the result is, robotparser thinks the site is blocked.

https://docs.python.org/2.7/library/robotparser.html#module-robotparser
and
https://docs.python.org/3/library/urllib.robotparser.html#module-urllib.robotparser
refers users, for file structure, to http://www.robotstxt.org/orig.html.
This says nothing about the effect of an empty file, so I don't see this as a bug. Even if it was, I would be dubious about reversing the effect without a deprecation notice first, and definitely not in 2.7.

I would propose instead that the doc be changed to refer to the new file, with more and better examples, but add a note that robotparser interprets empty files as 'block all' rather than 'allow all'.

Try bringing this up on python-ideas.

(...) refers users, for file structure, to http://www.robotstxt.org/orig.html. This says nothing about the effect of an empty file, so I don't see this as a bug.

That is incorrect. From that url you can find:

The presence of an empty "/robots.txt" file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.

So this is definitely a bug.

Hi,

Is this ticket still relevant for Python 3.8?

While running some tests with an empty robotstxt file I realized that it was returning "ALLOWED" for any path (as per the current draft of the Robots Exclusion Protocol: https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1 ")

Code:

from urllib import robotparser

robots_url = "file:///tmp/empty.txt"

rp = robotparser.RobotFileParser()
print(robots_url)
rp.set_url(robots_url)
rp.read()
print( "fetch /", rp.can_fetch(useragent = "*", url = "/"))
print( "fetch /admin", rp.can_fetch(useragent = "*", url = "/admin"))

Output:

$ cat /tmp/empty.txt
$ python -V
Python 3.8.1
$ python test_robot3.py
file:///tmp/empty.txt
fetch / True
fetch /admin True

There is a behavior change. parse() sets the modified time and unless the modified time is set the can_fetch method returns false. In Python 2 the parse method was called only when the file is non-empty [0] but in Python 3 it's always called though the file is empty [1] . The change was done with 1afc169 to always read parse and then in 122541b modified function was always called during parse thus setting the modified_time to return True from can_fetch in the end.

I think the behavior of robotparser for empty file was undefined allowing these changes and it will be good to have a test for this behavior.

[0]

cpython/Lib/robotparser.py

Lines 66 to 67 in f82e59a

    
           elif self.errcode == 200 and lines: 
        
               self.parse(lines)

[1]

cpython/Lib/urllib/robotparser.py

Lines 69 to 70 in 149175c

    
           raw = f.read() 
        
           self.parse(raw.decode("utf-8").splitlines())

@XTreak

Thanks @XTreak for providing some clarification on this behavior! I can write some tests to cover this behavior, assuming that we agree that an empty file means "unlimited access". This was worded as such in the old internet draft from 1996 (section 3.2.1 in https://www.robotstxt.org/norobots-rfc.txt). The current draft is more ambiguous with "If no group satisfies either condition, or no groups are present at all, no rules apply." https://tools.ietf.org/html/draft-koster-rep-00#section-2.2.1

https://www.robotstxt.org/robotstxt.html clearly states that an empty file gives full access, but I'm getting lost in figuring out which is the official spec at the moment :-)

pythongh-79638: Treat an unreachable robots.txt as "disallow all"

gh-79638: Treat an unreachable robots.txt as "disallow all" (GH-138555)

Revert "pythongh-79638: Treat an unreachable robots.txt as "disallow …

gh-79638: Test other HTTP error codes besides 403 in test_robotparser (…

Since currently robotparser disallows access to all resource if the HTTP status code was 500-599, adding new tests is enough.

[3.13] gh-79638: Test other HTTP error codes besides 403 in test_robo…

[3.14] gh-79638: Test other HTTP error codes besides 403 in test_robo…

[3.15] gh-79638: Test other HTTP error codes besides 403 in test_robo…

gh-79638: Restore "Treat an unreachable robots.txt as disallow all" (G…

[3.13] gh-79638: Restore "Treat an unreachable robots.txt as disallow…

[3.14] gh-79638: Restore "Treat an unreachable robots.txt as disallow…

[3.15] gh-79638: Restore "Treat an unreachable robots.txt as disallow…

pythongh-79638: Restore "Treat an unreachable robots.txt as disallow …

Navigation Menu

Repository navigation

robotparser reads empty robots.txt file as "all denied" #79638

Linked PRs

3 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

robotparser reads empty robots.txt file as "all denied" #79638

Description

Linked PRs

Activity

larsfuse commented on Dec 11, 2018

terryjreedy commented on Dec 14, 2018

larsfuse commented on Dec 17, 2018

andreburgaud commented on Jan 2, 2020

tirkarthi commented on Jan 2, 2020

andreburgaud commented on Jan 2, 2020

3 remaining items

serhiy-storchaka commented on May 8, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions