QID 980491

QID 980491: Python (pip) Security Update for Scrapy (GHSA-jwqp-28gf-p498)

Security update has been released for Scrapy to fix the vulnerability.

Note: The preceding description block is extracted directly from the security advisory. Using automation, we have attempted to clean and format it as much as possible without introducing additional issues.

If you use [`HttpAuthMiddleware`](http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpauth) (i.e. the `http_user` and `http_pass` spider attributes) for HTTP authentication, all requests will expose your credentials to the request target.

This includes requests generated by Scrapy components, such as `robots.txt` requests sent by Scrapy when the `ROBOTSTXT_OBEY` setting is set to `True`, or as requests reached through redirects.

CVSS V3 rated as High - 6.5 severity.

CVSS V2 rated as Medium - 4 severity.

Solution

Upgrade to Scrapy 2.5.1 and use the new `http_auth_domain` spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead.Workaround:
If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the [`w3lib.http.basic_auth_header`](https://w3lib.readthedocs.io/en/latest/w3lib.html#w3lib.http.basic_auth_header) function to convert your credentials into a value that you can assign to the `Authorization` header of your request, instead of defining your credentials globally using [`HttpAuthMiddleware`](http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#module-scrapy.downloadermiddlewares.httpauth).

Vendor References

GHSA-jwqp-28gf-p498 - github.com/advisories/GHSA-jwqp-28gf-p498

CVEs related to QID 980491

CVE-2021-41125 |

Software Advisories

Advisory ID	Software	Component	Link
GHSA-jwqp-28gf-p498	Scrapy		github.com/advisories/GHSA-jwqp-28gf-p498

By selecting these links, you may be leaving CVEreport webspace. We have provided these links to other websites because they may have information that would be of interest to you. No inferences should be drawn on account of other sites being referenced, or not, from this page. There may be other websites that are more appropriate for your purpose. CVEreport does not necessarily endorse the views expressed, or concur with the facts presented on these sites. Further, CVEreport does not endorse any commercial products that may be mentioned on these sites. Please address comments about any linked pages to [email protected].