Searched refs:RobotFileParser (Results 1 – 3 of 3) sorted by relevance
36 parser = robotparser.RobotFileParser()237 parser = robotparser.RobotFileParser()248 parser = robotparser.RobotFileParser(
18 class RobotFileParser: class
332 self.robots[root] = rp = robotparser.RobotFileParser()