RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,793 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

2796 Commits

Author SHA1 Message Date
Adrián Chaves 36bf3a8cb4 Allow raising CloseSpider from seed iterables 2025-03-13 15:43:01 +01:00
Adrián Chaves b56a883d58 news.rst: Update related GitHub issues 2025-03-13 14:07:49 +01:00
Adrián Chaves b1999444ba Default seeding policy: lazy → greedy 2025-03-13 13:17:57 +01:00
Adrián Chaves e4e3dba141 Fix a typo 2025-03-13 11:25:00 +01:00
Adrián Chaves 1d1a85711d news.rst: cover start-item-related delay fix, reproducible in Scrapy 2.12 2025-03-13 11:23:52 +01:00
Adrián Chaves 724b6a620a news.rst: Remove fake bug fix 
I was convinced this used to be an issue, but I could not reproduce it in Scrapy 2.12
 2025-03-13 11:18:11 +01:00
Adrián Chaves a067253234 Allow overriding the active seeding policy 2025-03-13 08:58:12 +01:00
Adrián Chaves 38cf129241 Publish the SeedingPolicy enum 2025-03-13 02:10:20 +01:00
Adrián Chaves 69f829fa5e Implement seeding policies 2025-03-13 00:58:02 +01:00
Adrián Chaves d4dc14155c AsyncIterator → AsyncIterable 2025-03-12 21:30:56 +01:00
Adrián Chaves 62c5bd1e0c Remove partial typing from some code examples to keep them shorter 2025-03-12 17:37:42 +01:00
Adrián Chaves 6a03bf241f Improve the release notes 2025-03-12 09:46:00 +01:00
Adrián Chaves 53841ff419 Cover backward-incompatible changes in the release notes 2025-03-12 08:38:27 +01:00
Adrián Chaves bd73b5437d Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-12 02:20:29 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves ba28d96d3e
Centralize from_crawler docs (and somewhat related changes) (#6723) 2025-03-11 14:53:04 +01:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Adrián Chaves 9b7db1a068
Move some reference docs of Request to the code (#6721) 2025-03-11 12:43:50 +01:00
Adrián Chaves bee74fb753 Remove trailing whitespace 2025-03-11 11:56:44 +01:00
Adrián Chaves 47bc1422d2 Revert docs change, the warning is not preventing the docs build 2025-03-11 11:43:05 +01:00
Adrián Chaves 3d93441ea6 Merge remote-tracking branch 'scrapy/master' into async-seeds 2025-03-09 09:47:20 +01:00
Adrián Chaves 364664b0a4 def start_requests → async def yield_seeds (no backward compatibility, new tests or additional features implemented yet) 2025-03-09 09:41:06 +01:00
Andrey Rakhmatullin d2e5486d5a Remove the Splash recommendation. 2025-03-07 13:20:42 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin c200458f24
Add more docs for updating sync spider middlewares. (#6688) 2025-02-25 00:16:23 +01:00
anubhav f041f26a6f
Support dark mode in the documentation (#6653) 2025-02-06 18:07:07 +01:00
Laerte Pereira 2eb3c75c69 Remove AjaxCrawlMiddleware mention from built-in downloader middleware 2025-02-05 13:16:51 -03:00
Andrey Rakhmatullin 76a8badd24 Add a deprecation notice to the AjaxCrawlMiddleware docs. 2025-02-03 14:55:10 +05:00
Andrey Rakhmatullin 393ff96e45 Deprecate AjaxCrawlMiddleware. 2025-02-03 14:08:05 +05:00
Rotzbua 7e61ff3524
Upgrade Sphinx (#6624) 2025-01-22 18:09:42 +01:00
guillermo-bondonno 9bc0029d27
Allow updating pre-crawler settings from add-ons (#6568) 2025-01-22 12:07:44 +01:00
Rotzbua e0c828b7f6
chore(docs): refactor config (#6623) 2025-01-20 12:18:30 +01:00
Rotzbua ee4f527f47
fix(docs): pillow domain is shut down permanently 
See https://github.com/python-pillow/Pillow/issues/8585
 2025-01-19 14:58:02 +01:00
Rotzbua d7168577b8
chore(docs): migrate to RTD template v3 
notable change: Drop support for all versions of Internet Explorer.
 2025-01-19 13:50:53 +01:00
anubhav ca345a3b73
Flexible severity of logging level when items are dropped (#6608) 2025-01-15 11:08:18 +01:00
Adrián Chaves 98ba61256d
Deprecate BaseDupeFilter.log() and improve dupefilter docs (#4151) 
* Remove BaseDupeFilter.log()

It is never called because request_seen() always returns False

* Document the interface of DUPEFILTER_CLASS classes

* Remove unnecessary BaseDupeFilter comments and add a short class description

* Improve the documentation related to the DUPEFILTER_CLASS setting

* Deprecate BaseDupeFilter.log

* Update the docs

* Fix the new code example

* Remove typing to keep the example short

Otherwise, it would have required yet another import line (from __future__ or typing).

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-14 19:36:56 +05:00
Kevin Lloyd Bernal 1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware (#6151) 
* new 'allow_offsite' parameter in OffsiteMiddleware

* document deprecated dont_filter flag in OffsiteMiddleware

* avoid deprecating dont_filter in OffsiteMiddleware

* Copy the code to the downloader mw.

* Add tests for allow_offsite in the downloader mw.

* Mark allow_offsite with reqmeta.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-08 21:28:51 +05:00
Andrey Rakhmatullin 59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector (#6526) 
* Improve internal refs to scrapy.Selector.

* Improve internal refs to scrapy.Request.

* More scrapy.http fixes.

* Fix FormRequest refs.

* More fixes.

* Simplifications.

* Last fixes.

* Add the parsel intersphinx.
 2025-01-07 16:18:18 +05:00
Andrey Rakhmatullin b70443f2d0 Split ruff and pylint ignores into two categories, some pylint cleanup. 2025-01-02 01:03:42 +05:00
Adrián Chaves 21b9ba717c
Allow customizing logged software versions (#6582) 
Co-authored-by: Grammy Jiang <grammy.jiang@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-12-16 14:46:23 +01:00
Adrián Chaves 5c2df5cf2a
Contributing: add a section on finding work (#6575) 2024-12-12 11:38:30 +01:00
Andrey Rakhmatullin efb53aafdc Fix a typo that broke PDF builds. 2024-11-18 15:44:52 +05:00
Adrián Chaves 10089c6fe2
2.12 release notes (#6226) 
* Cover 2.12 in the release notes up to 9bb973dc54

* Add one more highlight

* Better merge of the news entries.

* Cover 2.12 in the release notes up to 642af40.

* Cover 2.12 in the release notes up to 7a0a34b.

* Cover 2.12 in the release notes up to b4bad97.

* Add not yet merged PRs #6463, #6507, #6511 to the 2.12 release notes.

* Cover 2.12 in the release notes up to d85c39f, small fixes.

* Cover 2.12 in the release notes up to d215669.

* Cover #6527 in the release notes.

* Address PR feedback.

* Cover recent PRs.

* Finalize the 2.12.0 release notes, small additional fixes.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-11-18 13:07:32 +05:00
Andrey Rakhmatullin ab5cb7c7d9
Merge pull request #6540 from wRAR/build_from_settings 
Remove build_from_settings() and deprecate from_settings() code
 2024-11-14 20:40:54 +05:00
Robert Palmer 261c4b61dc
Enhancement: Update docs to include IgnoreRequest details (#6506) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2024-11-13 22:47:39 +05:00
Andrey Rakhmatullin 499e7e8aa6 Add from_crawler() to components that only had from_settings(). 2024-11-12 21:12:32 +05:00
Adrián Chaves dc3ebb6cf7 Refactor the docs 2024-11-07 16:38:48 +01:00
Adrián Chaves 2a4b7fe0f8 dont_throttle → autothrottle_dont_adjust_delay 2024-11-07 16:17:16 +01:00
Adrián Chaves b244ea7ac0 Add the missing import to the docs example 2024-11-05 20:05:58 +01:00
Adrián Chaves 5862216bb1 Fix docs example 2024-11-05 19:55:28 +01:00