RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,828 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

10828 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 35212ec5b0
Merge pull request #6696 from wRAR/deprecate-test-utils 
Move most of the test utils inside tests.
 2025-03-05 22:07:15 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin d161d1d47d
Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
Andrey Rakhmatullin 93c076047b Add scrapy/utils/testproc.py to collect_ignore to silence a warning. 2025-03-02 21:19:24 +05:00
Andrey Rakhmatullin a5731c1944 Move most of the test utils inside tests. 2025-03-02 21:04:12 +05:00
Andrey Rakhmatullin 87db3f2fd6
Add SpiderLoaderProtocol. (#6694) 2025-02-28 11:18:55 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin 391af6afcc
Unknown encoding handling in HttpCompressionMiddleware, restore x-gzip support (#6618) 
* Unknown encoding handling in HttpCompressionMiddleware.

* Implement the changes for unknown encoding handling.

* Restore support for Content-Encoding: x-gzip.

* Simplify the decoding logic.

* Add tests for the unsupported encoding warning.

* Add a test for the "no zstandard" warning.
 2025-02-27 22:37:01 +05:00
Andrey Rakhmatullin c200458f24
Add more docs for updating sync spider middlewares. (#6688) 2025-02-25 00:16:23 +01:00
Matt Winter 8c34e6d9a4
curl: add support for parsing -b,--cookie (#6684) 2025-02-19 10:17:37 +01:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin 7d5b189c11
Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 
* Fix getting annotations for _parse_sitemap() at the runtime.

* Split off the callback annotations test.
 2025-02-14 20:40:06 +05:00
Andrey Rakhmatullin 00167edca0
Merge pull request #6664 from wRAR/downgrade-warnings 
Improve diagnostics for sync-only spider middlewares.
 2025-02-11 22:45:21 +04:00
Andrey Rakhmatullin ede9e9c3c3 Use full method names in all spidermw log messages. 2025-02-11 23:07:25 +05:00
Andrey Rakhmatullin d8978d405c Improve diagnostics for sync-only spider middlewares. 2025-02-09 01:33:27 +05:00
anubhav f041f26a6f
Support dark mode in the documentation (#6653) 2025-02-06 18:07:07 +01:00
Laerte Pereira 4e0a3087e4
fix: Reactor info logged twice (#6657) 
* fix: Reactor info logged twice

* Change condition syntax

* Simplify logic

* Format
 2025-02-06 15:47:39 +05:00
Andrey Rakhmatullin 02ad6bd1f6
Merge pull request #6656 from Laerte/master 
docs: Remove AjaxCrawlMiddleware mention from built-in downloader middleware
 2025-02-05 21:05:54 +04:00
Laerte Pereira 2eb3c75c69 Remove AjaxCrawlMiddleware mention from built-in downloader middleware 2025-02-05 13:16:51 -03:00
Andrey Rakhmatullin 9fd08a92a8
Merge pull request #6655 from Laerte/master 
Remove deprecated signals
 2025-02-05 16:49:56 +04:00
Laerte Pereira 9d35428770 Remove deprecated signals 2025-02-05 06:48:56 -03:00
Andrey Rakhmatullin cc480680d7
Merge pull request #6650 from wRAR/cleanup-cache-tests 
Remove a duplicate test.
 2025-02-03 23:33:07 +04:00
Andrey Rakhmatullin ba5df629a2
Refactor downloader tests (#6647) 
* Make download handler test base classes abstract.

* Small cleanup.

* Don't run the full test suite for special HTTP cases.

* Don't run tests in imported base classes.

* Remove an obsolete service_identity check.

* Move FTP imports back to the top level.

* Simplify the H2DownloadHandler import.

* Forbig pytest 8.2.x.

* Revert "Simplify the H2DownloadHandler import."

This reverts commit ed187046ac.
 2025-02-03 20:11:47 +05:00
Andrey Rakhmatullin 16e39661e9
Merge pull request #6651 from wRAR/deprecate-ajaxcrawl 
Deprecate AjaxCrawlMiddleware and stop calling escape_ajax() by default
 2025-02-03 13:57:46 +04:00
Andrey Rakhmatullin 76a8badd24 Add a deprecation notice to the AjaxCrawlMiddleware docs. 2025-02-03 14:55:10 +05:00
Andrey Rakhmatullin 4842bcbf1d Deprecate and disable escape_ajax(). 2025-02-03 14:08:05 +05:00
Andrey Rakhmatullin 393ff96e45 Deprecate AjaxCrawlMiddleware. 2025-02-03 14:08:05 +05:00
Mikhail Korobov b4c2531021
Merge pull request #6648 from wRAR/mockserver-calls 
Optimize mockserver calls
 2025-02-03 14:06:35 +05:00
Mikhail Korobov c727c6f201
Merge pull request #6646 from wRAR/engine-tests-cleanup 
Refactor EngineTest tests.
 2025-02-03 13:56:47 +05:00
Andrey Rakhmatullin df688910e0 Remove a duplicate test. 2025-02-02 18:48:26 +05:00
Andrey Rakhmatullin 783b98deda Make mockserver instances per-class. 2025-02-02 14:10:09 +05:00
Andrey Rakhmatullin 1a0dfbd32e Reuse mockserver instances in test_feedexport.py. 2025-02-02 13:28:34 +05:00
Andrey Rakhmatullin 200d76afa9 Refactor EngineTest tests. 2025-02-01 16:07:55 +05:00
Andrey Rakhmatullin 340819eff0
Merge pull request #6635 from wRAR/webclient-cleanup 
Remove scrapy.core.downloader.webclient._parse().
 2025-01-28 23:08:06 +04:00
Andrey Rakhmatullin 0a80871c3a Remove scrapy.core.downloader.webclient._parse(). 2025-01-28 22:22:09 +05:00
Andrey Rakhmatullin a8d9746f56
Merge pull request #6634 from wRAR/deprecate-http10 
Deprecate HTTP/1.0 code.
 2025-01-28 13:29:25 +04:00
Andrey Rakhmatullin 0d2d2892ba Silence the readBody warning. 2025-01-28 02:08:49 +05:00
Andrey Rakhmatullin bc1aeeefc9 Deprecate overriding ScrapyClientContextFactory.getContext(). 2025-01-28 02:07:47 +05:00
Andrey Rakhmatullin 16b998f9ca Sort out webclient tests. 2025-01-28 01:35:04 +05:00
Andrey Rakhmatullin d27c6b46b1 Deprecate HTTP/1.0 support. 2025-01-27 21:25:47 +05:00
Lidiane T 98a57e2418
Fix error when running `scrapy bench` (#6633) 2025-01-27 11:21:30 +01:00
Andrey Rakhmatullin cec0aeca58
Bump ruff, switch from black to ruff-format (#6631) 2025-01-27 11:07:09 +01:00
anubhav c03fb2abb8
fix: added feed_options as a keyword argument to GCSFeedStorage. (#6628) 2025-01-23 21:06:45 +05:00
Andrey Rakhmatullin d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. (#6613) 2025-01-23 09:22:18 +01:00
Rotzbua 7e61ff3524
Upgrade Sphinx (#6624) 2025-01-22 18:09:42 +01:00
Andrey Rakhmatullin 499b6c66b4
Merge pull request #6626 from Laerte/master 
fix: test_s3_export fails with boto3 >= 1.36.0
 2025-01-22 15:14:33 +04:00
guillermo-bondonno 9bc0029d27
Allow updating pre-crawler settings from add-ons (#6568) 2025-01-22 12:07:44 +01:00
Laerte Pereira 14219b1fca fix: test_s3_export fails with boto3 >= 1.36.0 2025-01-22 07:16:22 -03:00
Rotzbua e0c828b7f6
chore(docs): refactor config (#6623) 2025-01-20 12:18:30 +01:00
Andrey Rakhmatullin 8bc8f752e6
Merge pull request #6622 from Rotzbua/fix_docs_broken_link 
fix(docs): pillow domain is shut down permanently
 2025-01-19 20:33:00 +04:00