RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,839 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

8839 Commits

Author SHA1 Message Date
Vostretsov Nikita f2f101106f ignore deprecation warnings here 2021-05-25 14:36:06 +00:00
Vostretsov Nikita 52dd719045 suppress warnings from twisted 2021-05-25 14:22:50 +00:00
Vostretsov Nikita f83d9b8d41 remove ResourceLeaked warning in pypy 2021-05-25 14:04:41 +00:00
Vostretsov Nikita 7511d8541e put flake8 options into separate file to remove pytest warnings 2021-05-18 08:47:56 +00:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Shinichi Takayanagi bd60c3f41f
More documentation for setting spider atributes 
* docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17

* Update feed-exports.rst

* Update feed-exports.rst

* Reflects the comments

* Remove redundant newline

* Update docs/topics/feed-exports.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Apply suggestions from code review

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 00:58:04 +05:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Renne Rocha 34b216289c
Update link for reasoning value of URLLENGTH_LIMIT (#5134) 2021-05-06 16:34:05 +02:00
Andrey Rahmatullin e27eff47ac
Merge pull request #5122 from hrnciar/patch-1 
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources
 2021-04-28 17:28:04 +05:00
Tomáš Hrnčiar 4f500342c8
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources 2021-04-28 11:57:44 +02:00
Eugenio Lacuesta 02ae1deaf4
Deprecate unused squeues (#5117) 2021-04-27 09:41:44 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta 6837919798
Add peek method to queues (#5112) 2021-04-26 14:55:02 -03:00
Andrey Rahmatullin 7095df1558
Merge pull request #5082 from storymode7/shell-fix 
Find bash from PATH instead of /bin/bash
 2021-04-21 14:30:50 +05:00
Eugenio Lacuesta e779ed7d93
Dupefilter type hints (#5108) 2021-04-20 21:39:07 +02:00
Eugenio Lacuesta e3f81d8d5f
Engine: remove unnecessary parameter (#5106) 2021-04-20 11:46:43 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rahmatullin 5b78a64fca
Merge pull request #5070 from ric2b/patch-1 2021-04-15 14:50:59 +05:00
Andrey Rahmatullin 06f3d12c12
Merge pull request #5100 from wRAR/more-scraper-typing 
Additional typing for scraper and a small code change.
 2021-04-14 20:58:51 +05:00
Andrey Rakhmatullin 309a637f32 Small changes. 2021-04-14 20:26:37 +05:00
Andrey Rakhmatullin 77bff0db0a Additional typing for scraper and a small code change. 2021-04-14 19:10:13 +05:00
Andrey Rahmatullin 9bf9ab7291
Merge pull request #5077 from wRAR/deferred-typing 
Add typing for middleware and coroutine related code.
 2021-04-13 23:29:49 +05:00
Andrey Rakhmatullin 08e4eaf973 Import Deferred directly in scrapy/utils/defer.py. 2021-04-13 22:41:01 +05:00
Andrey Rakhmatullin cef0a8b3d6 Import Deferred directly in scrapy/middleware.py. 2021-04-13 21:08:06 +05:00
Andrey Rahmatullin a8de04c823
Update scrapy/utils/defer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:30 +05:00
Andrey Rahmatullin b0e7512574
Update scrapy/middleware.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:25 +05:00
Andrey Rahmatullin 335a256752
Update scrapy/core/spidermw.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:20 +05:00
Andrey Rakhmatullin 76fa2257ef Add typing also for return values, other small fixes. 2021-04-13 20:22:02 +05:00
Andrey Rakhmatullin 5b547f0808 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-13 19:47:20 +05:00
Andrey Rahmatullin 15edb3f656
Merge pull request #5095 from wRAR/scraper-slot-cleanup 
Cleanup of slot handling in Scraper.
 2021-04-13 19:46:26 +05:00
Andrey Rahmatullin 38010997a4
Merge pull request #5098 from storymode7/docs-update 
Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation
 2021-04-13 18:52:10 +05:00
Mayank Singhal a4415e4e6f Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation 2021-04-13 17:56:44 +05:30
Adrián Chaves 18502981f7
Merge pull request #5094 from elacuesta/fix-item-class-in-engine-tests 
Engine tests: fix item class spider, add minimal type hints
 2021-04-13 11:07:15 +02:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00
Andrey Rakhmatullin 4673f05dde Cleanup of slot handling in Scraper. 2021-04-09 23:42:24 +05:00
Eugenio Lacuesta b6f77806b0
Engine tests: fix item class spider, add minimal type hints 2021-04-09 13:09:47 -03:00
Adrián Chaves 5a75b14a5f docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17 2021-04-07 14:04:00 +02:00
Ricardo Amendoeira 8603f9d7a5
Apply changes to other examples in the same section. 2021-04-06 20:23:07 +01:00
Andrey Rakhmatullin e63188cbf7 Bump version: 2.4.1 → 2.5.0 2021-04-06 19:13:32 +05:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Mayank Singhal e7d51886ef Find bash from PATH instead of /bin/bash 2021-04-06 02:21:18 +05:30
Andrey Rakhmatullin 7dc857668f Also some typing for Scraper. 2021-04-04 16:15:33 +05:00
Andrey Rakhmatullin a0101361e2 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-04 16:01:47 +05:00
Andrey Rahmatullin 099fb6ead0
Merge pull request #5076 from wRAR/qualname 
Use __qualname__ in middleware handling.
 2021-04-04 15:52:32 +05:00
Andrey Rakhmatullin 414dd1119a Drop an unused import. 2021-04-03 17:54:55 +05:00
Andrey Rakhmatullin a9e96f9907 Add typing for middleware and coroutine related code. 2021-04-03 17:40:45 +05:00
Andrey Rakhmatullin 9e3b868dd8 Use __qualname__ in middleware handling. 2021-04-03 17:04:09 +05:00
Mikhail Korobov 8c5a3a5189
Merge pull request #5073 from elacuesta/fix-urlencode-doseq-arg 
Fix type for urlencode's doseq argument
 2021-04-02 00:51:15 +05:00
Eugenio Lacuesta 9e7cbc05ae
Fix type for urlencode's doseq argument 2021-04-01 15:22:51 -03:00
Mikhail Korobov 127a8586e1
Merge pull request #5063 from noviluni/update_usageError_message 
Update UsageError message
 2021-04-01 22:47:59 +05:00