RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,225 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

11225 Commits

Author SHA1 Message Date
Renne Rocha 34b216289c
Update link for reasoning value of URLLENGTH_LIMIT (#5134) 2021-05-06 16:34:05 +02:00
Eugenio Lacuesta 19c7415aae
Request type hints 2021-05-01 16:34:46 -03:00
Andrey Rahmatullin e27eff47ac
Merge pull request #5122 from hrnciar/patch-1 
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources
 2021-04-28 17:28:04 +05:00
Tomáš Hrnčiar 4f500342c8
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources 2021-04-28 11:57:44 +02:00
Eugenio Lacuesta 02ae1deaf4
Deprecate unused squeues (#5117) 2021-04-27 09:41:44 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta 6837919798
Add peek method to queues (#5112) 2021-04-26 14:55:02 -03:00
Andrey Rahmatullin 7095df1558
Merge pull request #5082 from storymode7/shell-fix 
Find bash from PATH instead of /bin/bash
 2021-04-21 14:30:50 +05:00
Eugenio Lacuesta e779ed7d93
Dupefilter type hints (#5108) 2021-04-20 21:39:07 +02:00
Eugenio Lacuesta e3f81d8d5f
Engine: remove unnecessary parameter (#5106) 2021-04-20 11:46:43 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rahmatullin 5b78a64fca
Merge pull request #5070 from ric2b/patch-1 2021-04-15 14:50:59 +05:00
Andrey Rakhmatullin ffca87ae88 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-04-14 21:00:27 +05:00
Andrey Rahmatullin 06f3d12c12
Merge pull request #5100 from wRAR/more-scraper-typing 
Additional typing for scraper and a small code change.
 2021-04-14 20:58:51 +05:00
Andrey Rakhmatullin 309a637f32 Small changes. 2021-04-14 20:26:37 +05:00
Andrey Rakhmatullin 77bff0db0a Additional typing for scraper and a small code change. 2021-04-14 19:10:13 +05:00
Andrey Rahmatullin 9db01a483c
Update scrapy/core/spidermw.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-14 19:02:38 +05:00
Andrey Rakhmatullin 61197d3dba Add/update typing, cleanup iterator/iterable inconsistencies. 2021-04-14 18:21:58 +05:00
Andrey Rakhmatullin ffc6f525ce Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-04-14 17:34:58 +05:00
Andrey Rahmatullin 9bf9ab7291
Merge pull request #5077 from wRAR/deferred-typing 
Add typing for middleware and coroutine related code.
 2021-04-13 23:29:49 +05:00
Andrey Rakhmatullin 08e4eaf973 Import Deferred directly in scrapy/utils/defer.py. 2021-04-13 22:41:01 +05:00
Andrey Rakhmatullin cef0a8b3d6 Import Deferred directly in scrapy/middleware.py. 2021-04-13 21:08:06 +05:00
Andrey Rahmatullin a8de04c823
Update scrapy/utils/defer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:30 +05:00
Andrey Rahmatullin b0e7512574
Update scrapy/middleware.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:25 +05:00
Andrey Rahmatullin 335a256752
Update scrapy/core/spidermw.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:20 +05:00
Andrey Rakhmatullin 76fa2257ef Add typing also for return values, other small fixes. 2021-04-13 20:22:02 +05:00
Andrey Rakhmatullin 5b547f0808 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-13 19:47:20 +05:00
Andrey Rahmatullin 15edb3f656
Merge pull request #5095 from wRAR/scraper-slot-cleanup 
Cleanup of slot handling in Scraper.
 2021-04-13 19:46:26 +05:00
Andrey Rahmatullin 38010997a4
Merge pull request #5098 from storymode7/docs-update 
Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation
 2021-04-13 18:52:10 +05:00
Mayank Singhal a4415e4e6f Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation 2021-04-13 17:56:44 +05:30
Adrián Chaves 18502981f7
Merge pull request #5094 from elacuesta/fix-item-class-in-engine-tests 
Engine tests: fix item class spider, add minimal type hints
 2021-04-13 11:07:15 +02:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00
Andrey Rakhmatullin 4673f05dde Cleanup of slot handling in Scraper. 2021-04-09 23:42:24 +05:00
Eugenio Lacuesta b6f77806b0
Engine tests: fix item class spider, add minimal type hints 2021-04-09 13:09:47 -03:00
Eugenio Lacuesta 91f8144552
Remove deprecated Spider.make_requests_from_url method 2021-04-07 15:16:55 -03:00
Adrián Chaves 5a75b14a5f docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17 2021-04-07 14:04:00 +02:00
Ricardo Amendoeira 8603f9d7a5
Apply changes to other examples in the same section. 2021-04-06 20:23:07 +01:00
Andrey Rakhmatullin e63188cbf7 Bump version: 2.4.1 → 2.5.0 2021-04-06 19:13:32 +05:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Mayank Singhal e7d51886ef Find bash from PATH instead of /bin/bash 2021-04-06 02:21:18 +05:30
Andrey Rakhmatullin 7dc857668f Also some typing for Scraper. 2021-04-04 16:15:33 +05:00
Andrey Rakhmatullin a0101361e2 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-04 16:01:47 +05:00
Andrey Rahmatullin 099fb6ead0
Merge pull request #5076 from wRAR/qualname 
Use __qualname__ in middleware handling.
 2021-04-04 15:52:32 +05:00
Andrey Rakhmatullin 414dd1119a Drop an unused import. 2021-04-03 17:54:55 +05:00
Andrey Rakhmatullin a9e96f9907 Add typing for middleware and coroutine related code. 2021-04-03 17:40:45 +05:00
Andrey Rakhmatullin 9e3b868dd8 Use __qualname__ in middleware handling. 2021-04-03 17:04:09 +05:00
Andrey Rakhmatullin 7bd1d888d4 More robust sync/async middleware mix checking. 2021-04-02 23:07:24 +05:00
Andrey Rakhmatullin 30ed7fa349 Some cleanup, make sync middlewares fail earlier. 2021-04-02 22:20:56 +05:00
Andrey Rakhmatullin 849472535e Update docs. 2021-04-02 20:20:35 +05:00
Mikhail Korobov 8c5a3a5189
Merge pull request #5073 from elacuesta/fix-urlencode-doseq-arg 
Fix type for urlencode's doseq argument
 2021-04-02 00:51:15 +05:00