RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,861 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

218 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 95172659af
Add sphinx-lint. (#6920) 2025-06-28 01:37:20 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Kevin Lloyd Bernal 1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware (#6151) 
* new 'allow_offsite' parameter in OffsiteMiddleware

* document deprecated dont_filter flag in OffsiteMiddleware

* avoid deprecating dont_filter in OffsiteMiddleware

* Copy the code to the downloader mw.

* Add tests for allow_offsite in the downloader mw.

* Mark allow_offsite with reqmeta.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-08 21:28:51 +05:00
Andrey Rakhmatullin 59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector (#6526) 
* Improve internal refs to scrapy.Selector.

* Improve internal refs to scrapy.Request.

* More scrapy.http fixes.

* Fix FormRequest refs.

* More fixes.

* Simplifications.

* Last fixes.

* Add the parsel intersphinx.
 2025-01-07 16:18:18 +05:00
Andrey Rakhmatullin 499e7e8aa6 Add from_crawler() to components that only had from_settings(). 2024-11-12 21:12:32 +05:00
Adrián Chaves 2a4b7fe0f8 dont_throttle → autothrottle_dont_adjust_delay 2024-11-07 16:17:16 +01:00
Adrián Chaves f57fc454be Replace Slot.throttle with Request.meta['dont_throttle'] 2024-11-05 19:44:30 +01:00
Adrián Chaves 812fd2368f
Allow user-defined secure cookies (#6357) 2024-05-15 11:48:43 +02:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Adrián Chaves 809bfac489 Merge branch '2.11-compression-bomb' into 2.11 2024-02-14 17:16:25 +01:00
Laerte Pereira 24634f1bb2 Attend PR comments 2024-01-31 12:29:43 -03:00
Laerte Pereira bacaf0db7a Update documentation 2024-01-31 10:14:48 -03:00
Jalil SA d25cfe5315
Add JsonResponse (#6174) 2023-12-21 10:36:21 +01:00
Adrián Chaves b095dd218f Extend Request.meta documentation (#5565) 2023-12-20 21:43:03 +04:00
Adrián Chaves 731f749556
Extend Request.meta documentation (#5565) 2023-11-30 10:54:09 +01:00
Adrián Chaves 1087bb7b2e Update the docs 2023-11-23 09:11:14 +01:00
Andrey Rakhmatullin ffbf943e9d Merge pull request #6077 from 11-aryan/11-aryan 2023-11-16 19:13:02 +04:00
Adrián Chaves 42adbb2104
Update docs/topics/request-response.rst 2023-10-02 14:43:30 +02:00
Aryan e146c3a2fc removed the entire example for dont_merge_cookies 2023-10-02 15:36:29 +05:30
Aryan fe5ef0a80a Fixed the dont_merge_cookies example in docs 2023-10-01 11:47:51 +05:30
Aaron Smith ee215a2970
Change redirect text from Response.request docs (#5937) 2023-06-21 20:05:39 +02:00
Jalil SA 865c36bdbb update docs 2023-04-28 08:56:11 -06:00
Andrey Rakhmatullin 76eba9977b
Skip or fix failing code block tests (#5826) 2023-02-14 09:40:38 +01:00
pankaj1707k 03f32c018f
Update black version and fix minor typos and punctuations 2023-02-03 11:52:20 +05:30
Pankaj Kumar 280cd6ce71
Merge branch 'scrapy:master' into blacken-docs 2023-02-03 11:38:26 +05:30
Adrián Chaves 068af85722
Add release notes for Scrapy 2.8 (#5799) 2023-02-02 04:54:02 +01:00
pankaj1707k cc9eb3fa79
Format python console blocks using blacken-docs 
Prepend '.. code-block:: pycon' to make python console blocks detectable by blacken-docs
 2023-02-01 20:37:39 +05:30
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs' 
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
 2023-02-01 16:30:57 +05:30
Adrián Chaves 50500a6b28 Implement a NO_CALLBACK value for Request.callback 2023-01-19 17:14:18 +01:00
Mikhail Korobov 82f25bc44a
Merge pull request #5681 from mattkohl-flex/master 
Documentation: typo fixes
 2022-10-18 15:27:52 +05:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680) 
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
 2022-10-17 18:04:29 +05:00
mattkohl-flex c49764ffd7 typo fixes 2022-10-17 11:15:17 +01:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Adrián Chaves aa0306a167
Cover 2.6.0 in the release notes (#5399) 2022-03-01 12:16:37 +01:00
Mikhail Korobov 9dd77b42b5
Merge pull request #1582 from starrify/add_doc_textresponse_urljoin 
added: Doc for `scrapy.http.TextResponse.urljoin`
 2022-01-23 21:40:21 +05:00
Adrián Chaves d91d82b506
Make Scrapy SFW again 2021-10-01 16:31:29 +02:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Pratik Mahankal f0e1a33225
Sort the list of Request.meta alphabetically #5061 (#5065) 2021-03-23 22:16:50 +05:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Eugenio Lacuesta e494a3f733
protocol attribute for h2 responses 2020-12-31 11:50:15 -03:00
Eugenio Lacuesta 5e9a99e6a1
Reponse.protocol as string 2020-11-11 11:15:29 -03:00
Eugenio Lacuesta 2242412556
Docs: placeholder for versionadded directive 2020-11-11 10:50:54 -03:00
Eugenio Lacuesta 61d089485c
Docs: sort versionadded directives 2020-11-11 01:31:21 -03:00