[Docs] mention curl2scrapy in Request.from_curl

This commit is contained in:
Aditi Dutta 2020-03-25 14:38:22 -04:00
parent 0ee04e1e91
commit 010edfe85c
3 changed files with 9 additions and 0 deletions

View File

@ -292,6 +292,9 @@ Alternatively, if you want to know the arguments needed to recreate that
request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs` request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`
function to get a dictionary with the equivalent arguments. function to get a dictionary with the equivalent arguments.
Note that to translate a cURL command into a Scrapy request,
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
As you can see, with a few inspections in the `Network`-tool we As you can see, with a few inspections in the `Network`-tool we
were able to easily replicate the dynamic requests of the scrolling were able to easily replicate the dynamic requests of the scrolling
functionality of the page. Crawling dynamic pages can be quite functionality of the page. Crawling dynamic pages can be quite

View File

@ -104,6 +104,9 @@ If you get the expected response `sometimes`, but not always, the issue is
probably not your request, but the target server. The target server might be probably not your request, but the target server. The target server might be
buggy, overloaded, or :ref:`banning <bans>` some of your requests. buggy, overloaded, or :ref:`banning <bans>` some of your requests.
Note that to translate a cURL command into a Scrapy request,
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
.. _topics-handling-response-formats: .. _topics-handling-response-formats:
Handling different response formats Handling different response formats

View File

@ -129,6 +129,9 @@ class Request(object_ref):
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`, :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
may modify the :class:`~scrapy.http.Request` object. may modify the :class:`~scrapy.http.Request` object.
To translate a cURL command into a Scrapy request,
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
""" """
request_kwargs = curl_to_request_kwargs(curl_command, ignore_unknown_options) request_kwargs = curl_to_request_kwargs(curl_command, ignore_unknown_options)
request_kwargs.update(kwargs) request_kwargs.update(kwargs)