RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,931 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

10931 Commits

Author SHA1 Message Date
Andrey Rakhmatullin d2e5486d5a Remove the Splash recommendation. 2025-03-07 13:20:42 +05:00
Andrey Rakhmatullin 5a605969bd
Converting tests to plain asserts, part 2. (#6699) 2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin 1843a4f753
Converting tests to plain asserts, part 3. (#6700) 2025-03-06 20:50:14 +01:00
Andrey Rakhmatullin 35212ec5b0
Merge pull request #6696 from wRAR/deprecate-test-utils 
Move most of the test utils inside tests.
 2025-03-05 22:07:15 +05:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin d161d1d47d
Convert tests/test_utils* to plain asserts. (#6695) 2025-03-04 09:31:26 +01:00
Andrey Rakhmatullin 93c076047b Add scrapy/utils/testproc.py to collect_ignore to silence a warning. 2025-03-02 21:19:24 +05:00
Andrey Rakhmatullin a5731c1944 Move most of the test utils inside tests. 2025-03-02 21:04:12 +05:00
Andrey Rakhmatullin 87db3f2fd6
Add SpiderLoaderProtocol. (#6694) 2025-02-28 11:18:55 +01:00
Andrey Rakhmatullin 8d92c28a16
Switch to pytest.raises(). (#6680) 
* Switch to pytest.raises().

* Add matches= to broad pytest.raises().

* Adjust the test_nonserializable_object() regex for Python <= 3.11.

* Adjust the test_nonserializable_object() regex for PyPy.

* Adjust other test exception regexes for PyPy.

* Cleanup.
 2025-02-27 23:13:04 +05:00
Andrey Rakhmatullin 391af6afcc
Unknown encoding handling in HttpCompressionMiddleware, restore x-gzip support (#6618) 
* Unknown encoding handling in HttpCompressionMiddleware.

* Implement the changes for unknown encoding handling.

* Restore support for Content-Encoding: x-gzip.

* Simplify the decoding logic.

* Add tests for the unsupported encoding warning.

* Add a test for the "no zstandard" warning.
 2025-02-27 22:37:01 +05:00
Andrey Rakhmatullin c200458f24
Add more docs for updating sync spider middlewares. (#6688) 2025-02-25 00:16:23 +01:00
Matt Winter 8c34e6d9a4
curl: add support for parsing -b,--cookie (#6684) 2025-02-19 10:17:37 +01:00
Andrey Rakhmatullin a898331d14
Preparations for switching to direct pytest. (#6678) 2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin 7d5b189c11
Fix getting annotations for _parse_sitemap() at the runtime. (#6671) 
* Fix getting annotations for _parse_sitemap() at the runtime.

* Split off the callback annotations test.
 2025-02-14 20:40:06 +05:00
Andrey Rakhmatullin 00167edca0
Merge pull request #6664 from wRAR/downgrade-warnings 
Improve diagnostics for sync-only spider middlewares.
 2025-02-11 22:45:21 +04:00
Andrey Rakhmatullin ede9e9c3c3 Use full method names in all spidermw log messages. 2025-02-11 23:07:25 +05:00
Andrey Rakhmatullin d8978d405c Improve diagnostics for sync-only spider middlewares. 2025-02-09 01:33:27 +05:00
anubhav f041f26a6f
Support dark mode in the documentation (#6653) 2025-02-06 18:07:07 +01:00
Laerte Pereira 4e0a3087e4
fix: Reactor info logged twice (#6657) 
* fix: Reactor info logged twice

* Change condition syntax

* Simplify logic

* Format
 2025-02-06 15:47:39 +05:00
Andrey Rakhmatullin 02ad6bd1f6
Merge pull request #6656 from Laerte/master 
docs: Remove AjaxCrawlMiddleware mention from built-in downloader middleware
 2025-02-05 21:05:54 +04:00
Laerte Pereira 2eb3c75c69 Remove AjaxCrawlMiddleware mention from built-in downloader middleware 2025-02-05 13:16:51 -03:00
Andrey Rakhmatullin 9fd08a92a8
Merge pull request #6655 from Laerte/master 
Remove deprecated signals
 2025-02-05 16:49:56 +04:00
Laerte Pereira 9d35428770 Remove deprecated signals 2025-02-05 06:48:56 -03:00
Andrey Rakhmatullin cc480680d7
Merge pull request #6650 from wRAR/cleanup-cache-tests 
Remove a duplicate test.
 2025-02-03 23:33:07 +04:00
Andrey Rakhmatullin ba5df629a2
Refactor downloader tests (#6647) 
* Make download handler test base classes abstract.

* Small cleanup.

* Don't run the full test suite for special HTTP cases.

* Don't run tests in imported base classes.

* Remove an obsolete service_identity check.

* Move FTP imports back to the top level.

* Simplify the H2DownloadHandler import.

* Forbig pytest 8.2.x.

* Revert "Simplify the H2DownloadHandler import."

This reverts commit ed187046ac.
 2025-02-03 20:11:47 +05:00
Andrey Rakhmatullin 16e39661e9
Merge pull request #6651 from wRAR/deprecate-ajaxcrawl 
Deprecate AjaxCrawlMiddleware and stop calling escape_ajax() by default
 2025-02-03 13:57:46 +04:00
Andrey Rakhmatullin 76a8badd24 Add a deprecation notice to the AjaxCrawlMiddleware docs. 2025-02-03 14:55:10 +05:00
Andrey Rakhmatullin 4842bcbf1d Deprecate and disable escape_ajax(). 2025-02-03 14:08:05 +05:00
Andrey Rakhmatullin 393ff96e45 Deprecate AjaxCrawlMiddleware. 2025-02-03 14:08:05 +05:00
Mikhail Korobov b4c2531021
Merge pull request #6648 from wRAR/mockserver-calls 
Optimize mockserver calls
 2025-02-03 14:06:35 +05:00
Mikhail Korobov c727c6f201
Merge pull request #6646 from wRAR/engine-tests-cleanup 
Refactor EngineTest tests.
 2025-02-03 13:56:47 +05:00
Andrey Rakhmatullin df688910e0 Remove a duplicate test. 2025-02-02 18:48:26 +05:00
Andrey Rakhmatullin 783b98deda Make mockserver instances per-class. 2025-02-02 14:10:09 +05:00
Andrey Rakhmatullin 1a0dfbd32e Reuse mockserver instances in test_feedexport.py. 2025-02-02 13:28:34 +05:00
Andrey Rakhmatullin 200d76afa9 Refactor EngineTest tests. 2025-02-01 16:07:55 +05:00
Andrey Rakhmatullin 340819eff0
Merge pull request #6635 from wRAR/webclient-cleanup 
Remove scrapy.core.downloader.webclient._parse().
 2025-01-28 23:08:06 +04:00
Andrey Rakhmatullin 0a80871c3a Remove scrapy.core.downloader.webclient._parse(). 2025-01-28 22:22:09 +05:00
Andrey Rakhmatullin a8d9746f56
Merge pull request #6634 from wRAR/deprecate-http10 
Deprecate HTTP/1.0 code.
 2025-01-28 13:29:25 +04:00
Andrey Rakhmatullin 0d2d2892ba Silence the readBody warning. 2025-01-28 02:08:49 +05:00
Andrey Rakhmatullin bc1aeeefc9 Deprecate overriding ScrapyClientContextFactory.getContext(). 2025-01-28 02:07:47 +05:00
Andrey Rakhmatullin 16b998f9ca Sort out webclient tests. 2025-01-28 01:35:04 +05:00
Andrey Rakhmatullin d27c6b46b1 Deprecate HTTP/1.0 support. 2025-01-27 21:25:47 +05:00
Lidiane T 98a57e2418
Fix error when running `scrapy bench` (#6633) 2025-01-27 11:21:30 +01:00
Andrey Rakhmatullin cec0aeca58
Bump ruff, switch from black to ruff-format (#6631) 2025-01-27 11:07:09 +01:00
anubhav c03fb2abb8
fix: added feed_options as a keyword argument to GCSFeedStorage. (#6628) 2025-01-23 21:06:45 +05:00
Andrey Rakhmatullin d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. (#6613) 2025-01-23 09:22:18 +01:00
Rotzbua 7e61ff3524
Upgrade Sphinx (#6624) 2025-01-22 18:09:42 +01:00
Andrey Rakhmatullin 499b6c66b4
Merge pull request #6626 from Laerte/master 
fix: test_s3_export fails with boto3 >= 1.36.0
 2025-01-22 15:14:33 +04:00
guillermo-bondonno 9bc0029d27
Allow updating pre-crawler settings from add-ons (#6568) 2025-01-22 12:07:44 +01:00