RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,890 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

8890 Commits

Author SHA1 Message Date
Adrián Chaves 73ff9ffd64
spiders.rst: indent warnings into class descriptions 2021-06-26 08:58:29 +02:00
Andrey Rahmatullin 016c7e92d1
Merge pull request #3579 from maramsumanth/duplicate_key 
[MRG+1] Issue #2919: Fix FormRequest.formdata with GET method duplicates same key in query string
 2021-06-15 13:36:53 +05:00
Andrey Rahmatullin af975a5b0a
Merge pull request #4649 from ajaymittur28/http-proxy-schema 
Support schema-less HTTP proxy (#4504)
 2021-06-15 13:36:17 +05:00
ajaymittur28 eb324fadad Merge branch 'http-proxy-schema' of https://github.com/ajaymittur28/scrapy into http-proxy-schema 2021-06-14 21:39:43 +05:30
ajaymittur28 7d653288e3 Update unittest 2021-06-14 21:39:18 +05:30
ajaymittur28 857e0e225a Merge branch 'master' into http-proxy-schema 2021-06-14 21:18:49 +05:30
Ajay Mittur 5044549c55 Update proxyScheme assignment 2021-06-14 14:58:19 +00:00
Andrey Rahmatullin 66e2004239
Fix a flake8 problem 2021-06-14 18:58:35 +05:00
Andrey Rahmatullin 16be658a31
Merge branch 'master' into duplicate_key 2021-06-14 18:45:33 +05:00
Andrey Rahmatullin 9f81de2a50
Merge pull request #5171 from elacuesta/request-types 
Type hints for Request and subclasses
 2021-06-11 17:11:19 +05:00
pdt1931 28858574d9
Add FAQ to code of Conduct (#5177) 
* added to CODE_OF_CONDUCT.md to include link to FAQ about the code of conduct

* added to CODE_OF_CONDUCT.md to include link to FAQ about the code of conduct
 2021-06-11 12:49:41 +05:00
Veniamin Gvozdikov e876d8e387
Rename scrapy-crawlera to scrapy-zyte-smartproxy (#5074) 2021-06-11 08:22:04 +02:00
Eugenio Lacuesta ce6447731a
Replace return type 2021-06-07 13:25:04 -03:00
Eugenio Lacuesta 479260dca0
Type hints for Request subclasses 2021-06-01 12:52:46 -03:00
Eugenio Lacuesta c9fecca010
More type hints 2021-06-01 12:41:37 -03:00
Eugenio Lacuesta 85f88a5710
More type hints for private functions used by FormRequest 2021-06-01 12:02:16 -03:00
Eugenio Lacuesta c594017e51
Type hints for private functions used by FormRequest 2021-06-01 11:27:21 -03:00
Eugenio Lacuesta 216dd37953
Type hints for FormRequest 2021-06-01 11:16:40 -03:00
Eugenio Lacuesta c92e8ad0a5
Merge branch 'master' into request-types 2021-06-01 11:01:23 -03:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Bhavesh ee682af3b0
[Fix] Change the truncation limit of Proxy TunnelError from 32 to 1000 (#5007) 
* [Fix] Change the truncation limit oof Proxy TunnelError from 32 to 64

* [Fix] Change the truncation limit for Proxy tunnel error

* [Fix] flake8 check

* [Fix] formatting issues

* [Remove] coverage report

* [Fix] truncation error issue

* [Fix] formatting issues

* [Remove] coverage report
 2021-05-12 01:23:02 +05:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Shinichi Takayanagi bd60c3f41f
More documentation for setting spider atributes 
* docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17

* Update feed-exports.rst

* Update feed-exports.rst

* Reflects the comments

* Remove redundant newline

* Update docs/topics/feed-exports.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Apply suggestions from code review

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 00:58:04 +05:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Renne Rocha 34b216289c
Update link for reasoning value of URLLENGTH_LIMIT (#5134) 2021-05-06 16:34:05 +02:00
Eugenio Lacuesta 19c7415aae
Request type hints 2021-05-01 16:34:46 -03:00
Andrey Rahmatullin e27eff47ac
Merge pull request #5122 from hrnciar/patch-1 
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources
 2021-04-28 17:28:04 +05:00
Tomáš Hrnčiar 4f500342c8
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources 2021-04-28 11:57:44 +02:00
Eugenio Lacuesta 02ae1deaf4
Deprecate unused squeues (#5117) 2021-04-27 09:41:44 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta 6837919798
Add peek method to queues (#5112) 2021-04-26 14:55:02 -03:00
Andrey Rahmatullin 7095df1558
Merge pull request #5082 from storymode7/shell-fix 
Find bash from PATH instead of /bin/bash
 2021-04-21 14:30:50 +05:00
Eugenio Lacuesta e779ed7d93
Dupefilter type hints (#5108) 2021-04-20 21:39:07 +02:00
Eugenio Lacuesta e3f81d8d5f
Engine: remove unnecessary parameter (#5106) 2021-04-20 11:46:43 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rahmatullin 5b78a64fca
Merge pull request #5070 from ric2b/patch-1 2021-04-15 14:50:59 +05:00
Andrey Rahmatullin 06f3d12c12
Merge pull request #5100 from wRAR/more-scraper-typing 
Additional typing for scraper and a small code change.
 2021-04-14 20:58:51 +05:00
Andrey Rakhmatullin 309a637f32 Small changes. 2021-04-14 20:26:37 +05:00
Andrey Rakhmatullin 77bff0db0a Additional typing for scraper and a small code change. 2021-04-14 19:10:13 +05:00
Andrey Rahmatullin 9bf9ab7291
Merge pull request #5077 from wRAR/deferred-typing 
Add typing for middleware and coroutine related code.
 2021-04-13 23:29:49 +05:00
Andrey Rakhmatullin 08e4eaf973 Import Deferred directly in scrapy/utils/defer.py. 2021-04-13 22:41:01 +05:00
Andrey Rakhmatullin cef0a8b3d6 Import Deferred directly in scrapy/middleware.py. 2021-04-13 21:08:06 +05:00
Andrey Rahmatullin a8de04c823
Update scrapy/utils/defer.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:30 +05:00
Andrey Rahmatullin b0e7512574
Update scrapy/middleware.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:25 +05:00
Andrey Rahmatullin 335a256752
Update scrapy/core/spidermw.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-13 21:05:20 +05:00
Andrey Rakhmatullin 76fa2257ef Add typing also for return values, other small fixes. 2021-04-13 20:22:02 +05:00
Andrey Rakhmatullin 5b547f0808 Merge remote-tracking branch 'origin/master' into deferred-typing 2021-04-13 19:47:20 +05:00
Andrey Rahmatullin 15edb3f656
Merge pull request #5095 from wRAR/scraper-slot-cleanup 
Cleanup of slot handling in Scraper.
 2021-04-13 19:46:26 +05:00
Andrey Rahmatullin 38010997a4
Merge pull request #5098 from storymode7/docs-update 
Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation
 2021-04-13 18:52:10 +05:00
Mayank Singhal a4415e4e6f Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation 2021-04-13 17:56:44 +05:30