RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,701 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

10701 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 79480658cd Move FTP imports back to the top level. 2025-02-02 01:25:11 +05:00
Andrey Rakhmatullin 190967439d Remove an obsolete service_identity check. 2025-02-02 01:17:41 +05:00
Andrey Rakhmatullin 542764c02f Don't run tests in imported base classes. 2025-02-02 00:45:53 +05:00
Andrey Rakhmatullin 9d10ce56a0 Don't run the full test suite for special HTTP cases. 2025-02-02 00:36:05 +05:00
Andrey Rakhmatullin fcc4d30df0 Small cleanup. 2025-02-01 23:47:12 +05:00
Andrey Rakhmatullin b4a8c11453 Make download handler test base classes abstract. 2025-02-01 23:21:49 +05:00
Andrey Rakhmatullin 340819eff0
Merge pull request #6635 from wRAR/webclient-cleanup 
Remove scrapy.core.downloader.webclient._parse().
 2025-01-28 23:08:06 +04:00
Andrey Rakhmatullin 0a80871c3a Remove scrapy.core.downloader.webclient._parse(). 2025-01-28 22:22:09 +05:00
Andrey Rakhmatullin a8d9746f56
Merge pull request #6634 from wRAR/deprecate-http10 
Deprecate HTTP/1.0 code.
 2025-01-28 13:29:25 +04:00
Andrey Rakhmatullin 0d2d2892ba Silence the readBody warning. 2025-01-28 02:08:49 +05:00
Andrey Rakhmatullin bc1aeeefc9 Deprecate overriding ScrapyClientContextFactory.getContext(). 2025-01-28 02:07:47 +05:00
Andrey Rakhmatullin 16b998f9ca Sort out webclient tests. 2025-01-28 01:35:04 +05:00
Andrey Rakhmatullin d27c6b46b1 Deprecate HTTP/1.0 support. 2025-01-27 21:25:47 +05:00
Lidiane T 98a57e2418
Fix error when running `scrapy bench` (#6633) 2025-01-27 11:21:30 +01:00
Andrey Rakhmatullin cec0aeca58
Bump ruff, switch from black to ruff-format (#6631) 2025-01-27 11:07:09 +01:00
anubhav c03fb2abb8
fix: added feed_options as a keyword argument to GCSFeedStorage. (#6628) 2025-01-23 21:06:45 +05:00
Andrey Rakhmatullin d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. (#6613) 2025-01-23 09:22:18 +01:00
Rotzbua 7e61ff3524
Upgrade Sphinx (#6624) 2025-01-22 18:09:42 +01:00
Andrey Rakhmatullin 499b6c66b4
Merge pull request #6626 from Laerte/master 
fix: test_s3_export fails with boto3 >= 1.36.0
 2025-01-22 15:14:33 +04:00
guillermo-bondonno 9bc0029d27
Allow updating pre-crawler settings from add-ons (#6568) 2025-01-22 12:07:44 +01:00
Laerte Pereira 14219b1fca fix: test_s3_export fails with boto3 >= 1.36.0 2025-01-22 07:16:22 -03:00
Rotzbua e0c828b7f6
chore(docs): refactor config (#6623) 2025-01-20 12:18:30 +01:00
Andrey Rakhmatullin 8bc8f752e6
Merge pull request #6622 from Rotzbua/fix_docs_broken_link 
fix(docs): pillow domain is shut down permanently
 2025-01-19 20:33:00 +04:00
Rotzbua ee4f527f47
fix(docs): pillow domain is shut down permanently 
See https://github.com/python-pillow/Pillow/issues/8585
 2025-01-19 14:58:02 +01:00
Andrey Rakhmatullin 782e286ccf
Merge pull request #6621 from Rotzbua/chore_docs_rtd_template 
chore(docs): migrate to RTD template v3
 2025-01-19 17:35:43 +04:00
Rotzbua d7168577b8
chore(docs): migrate to RTD template v3 
notable change: Drop support for all versions of Internet Explorer.
 2025-01-19 13:50:53 +01:00
anubhav ca345a3b73
Flexible severity of logging level when items are dropped (#6608) 2025-01-15 11:08:18 +01:00
anubhav 1c1e83895c
Fix _pop_command_name (#6606) 2025-01-14 16:40:24 +01:00
Adrián Chaves 98ba61256d
Deprecate BaseDupeFilter.log() and improve dupefilter docs (#4151) 
* Remove BaseDupeFilter.log()

It is never called because request_seen() always returns False

* Document the interface of DUPEFILTER_CLASS classes

* Remove unnecessary BaseDupeFilter comments and add a short class description

* Improve the documentation related to the DUPEFILTER_CLASS setting

* Deprecate BaseDupeFilter.log

* Update the docs

* Fix the new code example

* Remove typing to keep the example short

Otherwise, it would have required yet another import line (from __future__ or typing).

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-14 19:36:56 +05:00
Ionut-Cezar Ciubotariu 402500b164
Change unknown cmd message when outside project (#3426) 
* Change unknown cmd message when outside project

* Simplification.

* Move the import to the top level.

* Reword the message.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-10 23:08:27 +05:00
Kevin Lloyd Bernal 1fc91bb462
new `allow_offsite` parameter in OffsiteMiddleware (#6151) 
* new 'allow_offsite' parameter in OffsiteMiddleware

* document deprecated dont_filter flag in OffsiteMiddleware

* avoid deprecating dont_filter in OffsiteMiddleware

* Copy the code to the downloader mw.

* Add tests for allow_offsite in the downloader mw.

* Mark allow_offsite with reqmeta.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2025-01-08 21:28:51 +05:00
Adrián Chaves b6d69e3895
Merge pull request #6610 from wRAR/coverage-subprocess 
Fix and speed up subprocess coverage tracking
 2025-01-07 18:29:31 +01:00
Andrey Rakhmatullin 3154b08e90 Improve coverage speed on Python 3.12+. 2025-01-07 20:42:03 +05:00
Andrey Rakhmatullin 7dfbecd392 Fix tracking of coverage in subprocesses. 2025-01-07 20:41:48 +05:00
Andrey Rakhmatullin 59fcb9b93c
Improve internal refs to scrapy.Request and scrapy.Selector (#6526) 
* Improve internal refs to scrapy.Selector.

* Improve internal refs to scrapy.Request.

* More scrapy.http fixes.

* Fix FormRequest refs.

* More fixes.

* Simplifications.

* Last fixes.

* Add the parsel intersphinx.
 2025-01-07 16:18:18 +05:00
Andrey Rakhmatullin 5d3aa80ad1
Switch CI to codecov/codecov-action and enable it on Windows. (#6609) 2025-01-07 10:52:26 +01:00
Andrey Rakhmatullin 4869315d10
Install libjpeg-dev on pinned envs to be able to install Pillow. (#6607) 2025-01-07 10:46:12 +01:00
Laerte Pereira f2234c5b96
Fix Crawler.request_fingerprinter typing (#6605) 2025-01-07 10:40:49 +01:00
Andrey Rakhmatullin 4d31277bc6
Explicitly mark re-exports. (#6579) 2025-01-02 23:48:14 +01:00
Andrey Rakhmatullin c330a399dc
Merge pull request #6601 from wRAR/ruff-rules-5 
Add more Ruff rules, do some pylint cleanups
 2025-01-02 17:38:15 +04:00
Andrey Rakhmatullin 176ae348c5 Reformat long REFERRER_POLICY. 2025-01-02 18:14:18 +05:00
Andrey Rakhmatullin 6ae5b92671
Drop the remaining unittest.main() blocks. (#6602) 2025-01-02 12:45:04 +01:00
Arthur b10d46d280
Fix the calculate_final_stats method (#6599) 2025-01-02 12:36:28 +01:00
Andrey Rakhmatullin dc706d4fc3 Remove useless pylint: disable lines. 2025-01-02 12:32:25 +05:00
Andrey Rakhmatullin b70443f2d0 Split ruff and pylint ignores into two categories, some pylint cleanup. 2025-01-02 01:03:42 +05:00
Andrey Rakhmatullin c87354cd46 Enable SIM Ruff rules. 2025-01-02 01:03:36 +05:00
Andrey Rakhmatullin 273620488c Enable PTH Ruff rules. 2025-01-01 22:21:51 +05:00
Andrey Rakhmatullin f44ca39fa2 Enable FLY Ruff rules. 2025-01-01 21:50:02 +05:00
Andrey Rakhmatullin 838ff99f37 Enable RUF Ruff rules. 2025-01-01 21:41:08 +05:00
Adrián Chaves ee239d2451
Merge pull request #6595 from wRAR/update-tools 
Update tool versions
 2024-12-30 09:56:08 +01:00