gh-60055: Allow passing a Request instance for RobotParser URLs

[ephphatha]

RobotParser only needs to know the type of the URL param in order to set the undocumented host and path properties, so when calling set_url use the pre-parsed attributes if given a Request object instance. Otherwise the only use of the url is when it is passed to urllib.request.urlopen which accepts either str or Request URLs.

Main driver for this change was to set the User-Agent header so #60055 seemed like the appropriate issue to link.

• Issue: Lib/robotparser.py doesn't accept setting a user agent string, instead it uses the default. #60055

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[ghost]

All commit authors signed the Contributor License Agreement.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #33443069 | 📁 Comparing 35bc72c against main (639a552)

  🔍 Preview build  

2 files changed

± library/urllib.robotparser.html
± whatsnew/changelog.html