RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,620 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

10620 Commits

Author SHA1 Message Date
Andrey Rakhmatullin ce56d227e5 Fix a typo that broke PDF builds. 2024-11-18 15:39:49 +05:00
Andrey Rakhmatullin b1f9e56693 Bump version: 2.11.2 → 2.12.0 2024-11-18 13:08:05 +05:00
Adrián Chaves 10089c6fe2
2.12 release notes (#6226) 
* Cover 2.12 in the release notes up to 9bb973dc54

* Add one more highlight

* Better merge of the news entries.

* Cover 2.12 in the release notes up to 642af40.

* Cover 2.12 in the release notes up to 7a0a34b.

* Cover 2.12 in the release notes up to b4bad97.

* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.

* Cover 2.12 in the release notes up to d85c39f, small fixes.

* Cover 2.12 in the release notes up to d215669.

* Cover #6527 in the release notes.

* Address PR feedback.

* Cover recent PRs.

* Finalize the 2.12.0 release notes, small additional fixes.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-11-18 13:07:32 +05:00
Andrey Rakhmatullin 212e848402
Merge pull request #6545 from wRAR/mitmproxy-dhparam 
Commit mitmproxy-dhparam.pem.
 2024-11-15 22:24:46 +05:00
Andrey Rakhmatullin feea3a0f67 Commit mitmproxy-dhparam.pem. 2024-11-15 21:08:18 +05:00
Andrey Rakhmatullin 87b2300831
Merge pull request #6543 from scrapy/pep740-attestations 
Verified PyPI releases (a.k.a. PEP740)
 2024-11-15 12:41:43 +05:00
Daniel Graña dc4d6d16ea Verified PyPI releases (a.k.a. PEP740) 2024-11-15 00:09:00 -03:00
Andrey Rakhmatullin 30fb54f47e
Merge pull request #6542 from nicklaustrup/Tests-contracts-swallowed-exceptions 
Fix - return in finally can swallow exception (#6505)
 2024-11-15 00:48:06 +05:00
Nicholas Laustrup bfcee452b0 Added failing test cases to tests/test_contracts.py and fixed corresponding methods + removed pylint comments 2024-11-14 10:40:12 -08:00
Andrey Rakhmatullin ab5cb7c7d9
Merge pull request #6540 from wRAR/build_from_settings 
Remove build_from_settings() and deprecate from_settings() code
 2024-11-14 20:40:54 +05:00
Andrey Rakhmatullin 929d665a74 Address PR feedback. 2024-11-14 19:35:56 +05:00
Andrey Rakhmatullin 2ad5f0c12b Extract duplicated code. 2024-11-14 13:03:04 +05:00
Andrey Rakhmatullin 6aa4d2b4ab Prefer crawler.settings over settings in media pipelines. 2024-11-14 02:03:43 +05:00
Andrey Rakhmatullin 28fafbb8c5 Modernize the media pipeline initialization API. 2024-11-14 01:29:51 +05:00
Robert Palmer 261c4b61dc
Enhancement: Update docs to include IgnoreRequest details (#6506) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2024-11-13 22:47:39 +05:00
Andrey Rakhmatullin 8700a5b7a9 Deprecate build_from_crawler() calling from_settings(). 2024-11-13 00:20:10 +05:00
Andrey Rakhmatullin eda1a8a7c5 Deprecate MiddlewareManager.from_settings(). 2024-11-12 22:57:39 +05:00
Andrey Rakhmatullin 499e7e8aa6 Add from_crawler() to components that only had from_settings(). 2024-11-12 21:12:32 +05:00
Andrey Rakhmatullin f796d8780c Add tests for MediaPipeline.from_crawler() and related code. 2024-11-12 21:08:04 +05:00
Andrey Rakhmatullin 83d4939d41 Deprecate scrapy.extensions.feedexport.build_storage() and simplify _get_storage(). 2024-11-11 22:15:08 +05:00
Andrey Rakhmatullin eda3a89b3f Remove build_from_settings(). 2024-11-11 16:44:47 +05:00
Andrey Rakhmatullin b042ad255d
Address some previously ignored pylint messages. (#6531) 2024-11-11 11:49:52 +01:00
Mikhail Korobov bcef96570b
Merge pull request #6527 from Gallaecio/dont-throttle-meta 
Replace Slot.throttle with Request.meta['dont_throttle']
 2024-11-08 14:25:37 +05:00
Adrián Chaves dc3ebb6cf7 Refactor the docs 2024-11-07 16:38:48 +01:00
Adrián Chaves 2a4b7fe0f8 dont_throttle → autothrottle_dont_adjust_delay 2024-11-07 16:17:16 +01:00
Adrián Chaves b244ea7ac0 Add the missing import to the docs example 2024-11-05 20:05:58 +01:00
Adrián Chaves 5862216bb1 Fix docs example 2024-11-05 19:55:28 +01:00
Adrián Chaves f57fc454be Replace Slot.throttle with Request.meta['dont_throttle'] 2024-11-05 19:44:30 +01:00
Rohit Kumar Singh d2156696c4
Deprecate unused scrapy utils (#6519) 
* Added deprecation warnings for unused Scrapy.utils

* Grammatical corrections

* Exceptions class connected

* Deprecation of ScrapyJSONDecoder

* request_authenticate function deprecation

* Making all warning similar

* Added ignore statements for deprecation warning in tests

* Missing stacklevel attr. added

* Added Deprecation message
 2024-11-04 21:09:45 +05:00
Andrey Rakhmatullin e7f5ae0b34
Update the outdated Item docstring. (#6427) 
* Update the outdated Item doscstring.

* Fix the reference links in items.html.
 2024-11-04 16:17:56 +05:00
Andrey Rakhmatullin ce5a132f12
Run and fix linkcheck. (#6524) 2024-11-04 11:40:07 +01:00
Rohit Kumar Singh 7701e590fb
Documentation added for Spider State in extensions.rst (#6522) 
* Documentation added for Spider State in extensions.rst

* Made correction in documentation for Spiderstate

* Added appropriate intro for Spider state extension

* Added reference for spiderstate extension

* Added Spiderstate extension hyperlink refrence in jobs.rst
 2024-11-02 10:45:27 +05:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Laerte Pereira d2bdbad8c8
Deprecate `scrapy.twisted_version` (#6512) 
* Deprecate scrapy.twisted_version

* fix: typing

* remove typing

* raise default exception if attribute is not found

* remove redudant ()

* add tests

* rollback exception raised

* add filterwarnings again

* change order

* lint
 2024-10-30 00:28:35 +05:00
Rohitkr117 12b087b0f2
Added ignore statements for Windows specific typing issues (#6516) 2024-10-29 19:30:32 +01:00
Rohitkr117 65ecd5d528
Fixes for Twisted Version Check and Typing Issues (#6511) 2024-10-29 19:08:38 +01:00
ThunderMind 5bbf8124ac
Updated deprecated ast.NameConstant with ast.Constant #6305 (#6463) 2024-10-29 10:28:00 +01:00
Andrey Rakhmatullin fcb5ab6cff
Remove code for unsupported Twisted. (#6510) 2024-10-29 10:21:07 +01:00
Andrey Rakhmatullin 0523e1616d
Explictly set html_baseurl on RTD. (#6507) 2024-10-29 10:16:03 +01:00
Andrey Rakhmatullin b4bad97eae
Merge pull request #6166 from Gallaecio/py313 
Add Python 3.13 to CI
 2024-10-22 20:05:56 +05:00
Andrey Rakhmatullin d10c58ff38 Bump pyftpdlib to the version supporting Python 3.13 on Windows. 2024-10-22 19:07:21 +05:00
Andrey Rakhmatullin fffacb9dac Merge remote-tracking branch 'origin/master' into py313 2024-10-22 19:07:10 +05:00
Andrey Rakhmatullin 04d0411bf7
Filter test-time warnings. (#6501) 2024-10-21 12:30:49 +02:00
Andrey Rakhmatullin 6d65708cb7
Merge pull request #6499 from wRAR/3.8-cleanup 
Additional Python 3.7 and 3.8 cleanup, including Reppy support.
 2024-10-18 00:08:11 +05:00
Andrey Rakhmatullin 677e977207 Remove dead links to the Reppy doc from the release notes. 2024-10-18 00:03:32 +05:00
Andrey Rakhmatullin 5759b3f0f2 Drop Reppy. 2024-10-17 23:41:23 +05:00
Andrey Rakhmatullin 7e07d48cc5 Small 3.7 and 3.8 cleanup. 2024-10-17 23:22:37 +05:00
Andrey Rakhmatullin 1138a5cf99
Merge pull request #6498 from wRAR/remove-runtime-typing 
Remove --keep-runtime-typing from pyupgrade.
 2024-10-17 23:18:22 +05:00
Andrey Rakhmatullin 7196a11f53 Reorder unions with None. 2024-10-17 21:52:03 +05:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00