Searched refs:RobotFileParser (Results 1 – 14 of 14) sorted by relevance
23 This module provides a single class, :class:`RobotFileParser`, which answers29 .. class:: RobotFileParser(url='')69 The following example demonstrates basic use of the RobotFileParser class. ::72 >>> rp = robotparser.RobotFileParser()
21 self.parser = robotparser.RobotFileParser()199 parser = robotparser.RobotFileParser()214 cls.parser = robotparser.RobotFileParser(cls.robots_txt)236 parser = robotparser.RobotFileParser(self.url('i-robot.txt'))
20 This module provides a single class, :class:`RobotFileParser`, which answers26 .. class:: RobotFileParser(url='')80 The following example demonstrates basic use of the :class:`RobotFileParser`84 >>> rp = urllib.robotparser.RobotFileParser()
18 self.parser = urllib.robotparser.RobotFileParser()310 parser = urllib.robotparser.RobotFileParser()325 cls.parser = urllib.robotparser.RobotFileParser(cls.robots_txt)349 parser = urllib.robotparser.RobotFileParser(self.url('i-robot.txt'))
19 class RobotFileParser: class
22 class RobotFileParser: class
309 :meth:`RobotFileParser.parse() <urllib.robotparser.RobotFileParser.parse>`
493 RobotFileParser now correctly returns default values for crawl_delay and
389 :meth:`RobotFileParser.parse() <urllib.robotparser.RobotFileParser.parse>`
2726 RobotFileParser now supports Crawl-delay and Request-rate extensions. Patch
4491 RobotFileParser now correctly returns default values for crawl_delay and
332 self.robots[root] = rp = robotparser.RobotFileParser()
1681 :class:`~urllib.robotparser.RobotFileParser` now supports the ``Crawl-delay`` and
2167 When encountering a password-protected robots.txt file the RobotFileParser