mirror of https://github.com/scrapy/scrapy.git
[Docs] mention curl2scrapy in Request.from_curl
This commit is contained in:
parent
0ee04e1e91
commit
010edfe85c
|
|
@ -292,6 +292,9 @@ Alternatively, if you want to know the arguments needed to recreate that
|
||||||
request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`
|
request you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`
|
||||||
function to get a dictionary with the equivalent arguments.
|
function to get a dictionary with the equivalent arguments.
|
||||||
|
|
||||||
|
Note that to translate a cURL command into a Scrapy request,
|
||||||
|
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
||||||
|
|
||||||
As you can see, with a few inspections in the `Network`-tool we
|
As you can see, with a few inspections in the `Network`-tool we
|
||||||
were able to easily replicate the dynamic requests of the scrolling
|
were able to easily replicate the dynamic requests of the scrolling
|
||||||
functionality of the page. Crawling dynamic pages can be quite
|
functionality of the page. Crawling dynamic pages can be quite
|
||||||
|
|
|
||||||
|
|
@ -104,6 +104,9 @@ If you get the expected response `sometimes`, but not always, the issue is
|
||||||
probably not your request, but the target server. The target server might be
|
probably not your request, but the target server. The target server might be
|
||||||
buggy, overloaded, or :ref:`banning <bans>` some of your requests.
|
buggy, overloaded, or :ref:`banning <bans>` some of your requests.
|
||||||
|
|
||||||
|
Note that to translate a cURL command into a Scrapy request,
|
||||||
|
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
||||||
|
|
||||||
.. _topics-handling-response-formats:
|
.. _topics-handling-response-formats:
|
||||||
|
|
||||||
Handling different response formats
|
Handling different response formats
|
||||||
|
|
|
||||||
|
|
@ -129,6 +129,9 @@ class Request(object_ref):
|
||||||
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
|
:class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,
|
||||||
may modify the :class:`~scrapy.http.Request` object.
|
may modify the :class:`~scrapy.http.Request` object.
|
||||||
|
|
||||||
|
To translate a cURL command into a Scrapy request,
|
||||||
|
you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.
|
||||||
|
|
||||||
"""
|
"""
|
||||||
request_kwargs = curl_to_request_kwargs(curl_command, ignore_unknown_options)
|
request_kwargs = curl_to_request_kwargs(curl_command, ignore_unknown_options)
|
||||||
request_kwargs.update(kwargs)
|
request_kwargs.update(kwargs)
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue