RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,907 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

8907 Commits

Author SHA1 Message Date
Andrey Rahmatullin ee2df97bbd
Pin the libxml2 version in CI as a newer one breaks lxml (#5208) 2021-07-16 14:28:32 +02:00
Andrey Rahmatullin 89b654b82d
Make the pylint test pass (#5207) 
Co-authored-by: Vostretsov Nikita <whalebot.helmsman@gmail.com>
 2021-07-16 12:18:14 +02:00
Eugenio Lacuesta bcce066057
Update ItemFilter (#5203) 2021-07-14 17:56:07 +02:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Eugenio Lacuesta c062ed017a
[CI] fail-fast: false (#5200) 2021-07-12 18:34:22 +02:00
Andrey Rahmatullin 1c46d5aa93
Merge pull request #5191 from ivanprado/master 
CloseSpider can be raised on spider_idle signal handler to set the closing reason
 2021-07-12 18:46:40 +05:00
Eugenio Lacuesta cb08e3644b
Remove trailing whitespaces 2021-07-08 09:22:21 -03:00
Iván de Prado eca641aa3d
Update tests/test_engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:20 +01:00
Iván de Prado 6b8f694653
Update scrapy/core/engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:40:02 +01:00
Iván de Prado 7597d860c8
Update scrapy/core/engine.py 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-08 12:39:17 +01:00
Andrey Rahmatullin 15dfc8a76f
Merge pull request #5194 from scrapy/doc-spiders-warning-indentation-1 
spiders.rst: indent warnings into class descriptions
 2021-06-29 17:54:22 +05:00
Iván de Prado Alonso e94d3ac173 Expanded doc for idle signal 2021-06-29 13:40:43 +01:00
Iván de Prado Alonso f35970778b Test case for raising CloseSpider on spider idle signal handler 2021-06-29 13:21:38 +01:00
Adrián Chaves 73ff9ffd64
spiders.rst: indent warnings into class descriptions 2021-06-26 08:58:29 +02:00
Iván de Prado Alonso ce445f2046 Fix typing 2021-06-24 09:56:05 +01:00
Iván de Prado Alonso 812b4bb518 CloseSpider can be raised on spider_idle signal handler 2021-06-23 17:22:01 +01:00
Andrey Rahmatullin 016c7e92d1
Merge pull request #3579 from maramsumanth/duplicate_key 
[MRG+1] Issue #2919: Fix FormRequest.formdata with GET method duplicates same key in query string
 2021-06-15 13:36:53 +05:00
Andrey Rahmatullin af975a5b0a
Merge pull request #4649 from ajaymittur28/http-proxy-schema 
Support schema-less HTTP proxy (#4504)
 2021-06-15 13:36:17 +05:00
ajaymittur28 eb324fadad Merge branch 'http-proxy-schema' of https://github.com/ajaymittur28/scrapy into http-proxy-schema 2021-06-14 21:39:43 +05:30
ajaymittur28 7d653288e3 Update unittest 2021-06-14 21:39:18 +05:30
ajaymittur28 857e0e225a Merge branch 'master' into http-proxy-schema 2021-06-14 21:18:49 +05:30
Ajay Mittur 5044549c55 Update proxyScheme assignment 2021-06-14 14:58:19 +00:00
Andrey Rahmatullin 66e2004239
Fix a flake8 problem 2021-06-14 18:58:35 +05:00
Andrey Rahmatullin 16be658a31
Merge branch 'master' into duplicate_key 2021-06-14 18:45:33 +05:00
Andrey Rahmatullin 9f81de2a50
Merge pull request #5171 from elacuesta/request-types 
Type hints for Request and subclasses
 2021-06-11 17:11:19 +05:00
pdt1931 28858574d9
Add FAQ to code of Conduct (#5177) 
* added to CODE_OF_CONDUCT.md to include link to FAQ about the code of conduct

* added to CODE_OF_CONDUCT.md to include link to FAQ about the code of conduct
 2021-06-11 12:49:41 +05:00
Veniamin Gvozdikov e876d8e387
Rename scrapy-crawlera to scrapy-zyte-smartproxy (#5074) 2021-06-11 08:22:04 +02:00
Eugenio Lacuesta ce6447731a
Replace return type 2021-06-07 13:25:04 -03:00
Eugenio Lacuesta 479260dca0
Type hints for Request subclasses 2021-06-01 12:52:46 -03:00
Eugenio Lacuesta c9fecca010
More type hints 2021-06-01 12:41:37 -03:00
Eugenio Lacuesta 85f88a5710
More type hints for private functions used by FormRequest 2021-06-01 12:02:16 -03:00
Eugenio Lacuesta c594017e51
Type hints for private functions used by FormRequest 2021-06-01 11:27:21 -03:00
Eugenio Lacuesta 216dd37953
Type hints for FormRequest 2021-06-01 11:16:40 -03:00
Eugenio Lacuesta c92e8ad0a5
Merge branch 'master' into request-types 2021-06-01 11:01:23 -03:00
Vostretsov Nikita 23cfdb058e
Reducing amount of warnings during test run (#5162) 
* put flake8 options into separate file to remove pytest warnings

* remove ResourceLeaked warning in pypy

* suppress warnings from twisted

* ignore deprecation warnings here

* ignore deprecation warning in tests of deprecated methods

* ignore deprecation warnings here

* update test classes

* don`t use deprecated method call

* ignore deprecation warnings here

* proper warning class

* more selective ignoring

* Revert "don`t use deprecated method call"

This reverts commit 59216ab560.
 2021-05-28 14:45:06 +05:00
Bhavesh ee682af3b0
[Fix] Change the truncation limit of Proxy TunnelError from 32 to 1000 (#5007) 
* [Fix] Change the truncation limit oof Proxy TunnelError from 32 to 64

* [Fix] Change the truncation limit for Proxy tunnel error

* [Fix] flake8 check

* [Fix] formatting issues

* [Remove] coverage report

* [Fix] truncation error issue

* [Fix] formatting issues

* [Remove] coverage report
 2021-05-12 01:23:02 +05:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Shinichi Takayanagi bd60c3f41f
More documentation for setting spider atributes 
* docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17

* Update feed-exports.rst

* Update feed-exports.rst

* Reflects the comments

* Remove redundant newline

* Update docs/topics/feed-exports.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Apply suggestions from code review

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 00:58:04 +05:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Renne Rocha 34b216289c
Update link for reasoning value of URLLENGTH_LIMIT (#5134) 2021-05-06 16:34:05 +02:00
Eugenio Lacuesta 19c7415aae
Request type hints 2021-05-01 16:34:46 -03:00
Andrey Rahmatullin e27eff47ac
Merge pull request #5122 from hrnciar/patch-1 
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources
 2021-04-28 17:28:04 +05:00
Tomáš Hrnčiar 4f500342c8
Require setuptools, scrapy/cmdline.py, /setup.py and tests/test_webclient.py import pkg_resources 2021-04-28 11:57:44 +02:00
Eugenio Lacuesta 02ae1deaf4
Deprecate unused squeues (#5117) 2021-04-27 09:41:44 -03:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Eugenio Lacuesta 6837919798
Add peek method to queues (#5112) 2021-04-26 14:55:02 -03:00
Andrey Rahmatullin 7095df1558
Merge pull request #5082 from storymode7/shell-fix 
Find bash from PATH instead of /bin/bash
 2021-04-21 14:30:50 +05:00
Eugenio Lacuesta e779ed7d93
Dupefilter type hints (#5108) 2021-04-20 21:39:07 +02:00