RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,844 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

8844 Commits

Author SHA1 Message Date
Eugenio Lacuesta 479260dca0
Type hints for Request subclasses 2021-06-01 12:52:46 -03:00
Eugenio Lacuesta c9fecca010
More type hints 2021-06-01 12:41:37 -03:00
Eugenio Lacuesta 85f88a5710
More type hints for private functions used by FormRequest 2021-06-01 12:02:16 -03:00
Eugenio Lacuesta c594017e51
Type hints for private functions used by FormRequest 2021-06-01 11:27:21 -03:00
Eugenio Lacuesta 216dd37953
Type hints for FormRequest 2021-06-01 11:16:40 -03:00
Eugenio Lacuesta c92e8ad0a5
Merge branch 'master' into request-types 2021-06-01 11:01:23 -03:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Bhavesh ee682af3b0
[Fix] Change the truncation limit of Proxy TunnelError from 32 to 1000 (#5007) 
* [Fix] Change the truncation limit oof Proxy TunnelError from 32 to 64

* [Fix] Change the truncation limit for Proxy tunnel error

* [Fix] flake8 check

* [Fix] formatting issues

* [Remove] coverage report

* [Fix] truncation error issue

* [Fix] formatting issues

* [Remove] coverage report
 2021-05-12 01:23:02 +05:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Shinichi Takayanagi bd60c3f41f
More documentation for setting spider atributes 
* docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17

* Update feed-exports.rst

* Update feed-exports.rst

* Reflects the comments

* Remove redundant newline

* Update docs/topics/feed-exports.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Apply suggestions from code review

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 00:58:04 +05:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Renne Rocha 34b216289c
Update link for reasoning value of URLLENGTH_LIMIT (#5134) 2021-05-06 16:34:05 +02:00
Eugenio Lacuesta 19c7415aae
Request type hints 2021-05-01 16:34:46 -03:00
Andrey Rahmatullin e27eff47ac
Merge pull request #5122 from hrnciar/patch-1 
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources
 2021-04-28 17:28:04 +05:00
Tomáš Hrnčiar 4f500342c8
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources 2021-04-28 11:57:44 +02:00
Eugenio Lacuesta 02ae1deaf4
Deprecate unused squeues (#5117) 2021-04-27 09:41:44 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta 6837919798
Add peek method to queues (#5112) 2021-04-26 14:55:02 -03:00
Andrey Rahmatullin 7095df1558
Merge pull request #5082 from storymode7/shell-fix 
Find bash from PATH instead of /bin/bash
 2021-04-21 14:30:50 +05:00
Eugenio Lacuesta e779ed7d93
Dupefilter type hints (#5108) 2021-04-20 21:39:07 +02:00
Eugenio Lacuesta e3f81d8d5f
Engine: remove unnecessary parameter (#5106) 2021-04-20 11:46:43 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rahmatullin 5b78a64fca
Merge pull request #5070 from ric2b/patch-1 2021-04-15 14:50:59 +05:00
Andrey Rahmatullin 06f3d12c12
Merge pull request #5100 from wRAR/more-scraper-typing 
Additional typing for scraper and a small code change.
 2021-04-14 20:58:51 +05:00
Andrey Rakhmatullin 309a637f32 Small changes. 2021-04-14 20:26:37 +05:00
Andrey Rakhmatullin 77bff0db0a Additional typing for scraper and a small code change. 2021-04-14 19:10:13 +05:00
Andrey Rahmatullin 9bf9ab7291
Merge pull request #5077 from wRAR/deferred-typing 
Add typing for middleware and coroutine related code.
 2021-04-13 23:29:49 +05:00
Andrey Rakhmatullin 08e4eaf973 Import Deferred directly in scrapy/utils/defer.py. 2021-04-13 22:41:01 +05:00
Andrey Rakhmatullin cef0a8b3d6 Import Deferred directly in scrapy/middleware.py. 2021-04-13 21:08:06 +05:00
Andrey Rahmatullin a8de04c823
Update scrapy/utils/defer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:30 +05:00
Andrey Rahmatullin b0e7512574
Update scrapy/middleware.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:25 +05:00
Andrey Rahmatullin 335a256752
Update scrapy/core/spidermw.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:20 +05:00
Andrey Rakhmatullin 76fa2257ef Add typing also for return values, other small fixes. 2021-04-13 20:22:02 +05:00
Andrey Rakhmatullin 5b547f0808 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-13 19:47:20 +05:00
Andrey Rahmatullin 15edb3f656
Merge pull request #5095 from wRAR/scraper-slot-cleanup 
Cleanup of slot handling in Scraper.
 2021-04-13 19:46:26 +05:00
Andrey Rahmatullin 38010997a4
Merge pull request #5098 from storymode7/docs-update 
Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation
 2021-04-13 18:52:10 +05:00
Mayank Singhal a4415e4e6f Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation 2021-04-13 17:56:44 +05:30
Adrián Chaves 18502981f7
Merge pull request #5094 from elacuesta/fix-item-class-in-engine-tests 
Engine tests: fix item class spider, add minimal type hints
 2021-04-13 11:07:15 +02:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00
Andrey Rakhmatullin 4673f05dde Cleanup of slot handling in Scraper. 2021-04-09 23:42:24 +05:00
Eugenio Lacuesta b6f77806b0
Engine tests: fix item class spider, add minimal type hints 2021-04-09 13:09:47 -03:00
Adrián Chaves 5a75b14a5f docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17 2021-04-07 14:04:00 +02:00
Ricardo Amendoeira 8603f9d7a5
Apply changes to other examples in the same section. 2021-04-06 20:23:07 +01:00
Andrey Rakhmatullin e63188cbf7 Bump version: 2.4.1 → 2.5.0 2021-04-06 19:13:32 +05:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Mayank Singhal e7d51886ef Find bash from PATH instead of /bin/bash 2021-04-06 02:21:18 +05:30
Andrey Rakhmatullin 7dc857668f Also some typing for Scraper. 2021-04-04 16:15:33 +05:00
Andrey Rakhmatullin a0101361e2 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-04 16:01:47 +05:00
Andrey Rahmatullin 099fb6ead0
Merge pull request #5076 from wRAR/qualname 
Use __qualname__ in middleware handling.
 2021-04-04 15:52:32 +05:00
Andrey Rakhmatullin 414dd1119a Drop an unused import. 2021-04-03 17:54:55 +05:00