RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,947 Commits 49 Branches 132 Tags 137 MiB
Commit Graph

7947 Commits

Author SHA1 Message Date
Adrián Chaves 94ee68695a Mock server: use 127.0.0.1 also for HTTPS 
Windows throws an error about 0.0.0.0 being external:
https://stackoverflow.com/a/23857995/939364
 2020-04-16 15:37:06 +02:00
Adrián Chaves 2f510fd47d Fix ShellTest.test_local_file on Windows 2020-04-15 21:10:05 +02:00
Adrián Chaves bdb28ac600 Merge remote-tracking branch 'origin/sybil' into azure-pipelines 2020-03-30 20:25:40 +02:00
Adrián Chaves 3fb0027138 Require sybil ≥ 1.3.0 for tests 2020-03-28 17:36:50 +01:00
Mikhail Korobov 8845773d44
Merge pull request #4456 from whalebot-helmsman/project_links 
Project URLs in machine-readable format for showing in pypi
 2020-03-27 14:11:03 +05:00
Daniel Graña 0699e6bb16 no need to install requirements.txt 2020-03-27 02:22:05 -03:00
Daniel Graña 02206e5ffe Run tox 2020-03-27 02:20:39 -03:00
Daniel Graña a175b6efc3 Set up CI with Azure Pipelines 
[skip ci]
 2020-03-27 02:10:10 -03:00
Vostretsov Nikita 16f2cb4a83 project URLs in machine-readable format for showing in pypi 2020-03-26 12:57:39 +00:00
Mikhail Korobov 0ee04e1e91
Merge pull request #4448 from adityaa30/fix-zope 
[fix] zope interface 5.0.0 unsupported
 2020-03-20 23:18:42 +05:00
Mikhail Korobov c26308dd45
Merge pull request #4438 from seregaxvm/master 
edit zsh completion
 2020-03-20 18:30:50 +05:00
Aditya 532cd1d93e [fix] zope interface 5.0.0 unsupported 2020-03-20 17:36:49 +05:30
Adrián Chaves 6c747953f9 Cover 2.0.1 in the release notes (#4437) 2020-03-18 19:17:16 +01:00
Matsievskiy S.V 9ab45325ff edit zsh completion 
- Fix bug introduced in https://github.com/scrapy/scrapy/pull/4291
- Enforce `[command] [options] [arguments]` syntax. Do not allow options after arguments
- Exclude already used option aliases from completion list
 2020-03-17 18:45:00 +03:00
sakshamb2113 533131a30f
Remove Guppy-specific code and documentation (#4343) 2020-03-17 10:12:49 +01:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
elacuesta e5711127b1
Remove deprecated ChunkedTransferMiddleware (#4431) 2020-03-16 19:43:02 +01:00
Andrey Rahmatullin ecd3a976d2
Merge pull request #4398 from nyov/remove-crawler-spiders 
Remove deprecated Crawler.spiders property
 2020-03-16 17:21:07 +05:00
Andrey Rahmatullin 1d862e0d35
Merge pull request #4404 from nyov/obsolete2 
Remove obsolete DEPRECATED_SETTINGS (deprecated.py)
 2020-03-16 17:18:48 +05:00
Aditya Kumar f9bf4b8d4d
Remove all top-level imports for twisted.internet.reactor (#4406) 2020-03-14 10:39:00 +01:00
Andrey Rahmatullin 0afe3da3dd
Merge pull request #4422 from Gallaecio/pin-working-pytest 
Restrict pytest to versions prior to 5.4
 2020-03-13 18:28:11 +05:00
Adrián Chaves 3f6cdcabce Restrict pytest to versions prior to 5.4 2020-03-13 13:25:53 +01:00
Lukas Anzinger ccc4d88779
Ignore a domain in allowed_domains with port and issue a warning (#4413) 2020-03-12 20:42:14 +01:00
nyov 3b0820d747
Deprecate Spider.make_requests_from_url, part 2 (#4412) 2020-03-12 20:15:49 +01:00
Mikhail Korobov db9198bc20
Merge pull request #4409 from nyov/remove-deprecation-warning 
Remove a 'twisted.test.proto_helpers' deprecation warning
 2020-03-12 18:46:47 +05:00
Mikhail Korobov ae5f59856c
Merge pull request #4420 from elacuesta/response_follow_all_empty_generator 
Response.follow_all: return empty generators for empty sequences
 2020-03-12 18:44:49 +05:00
Eugenio Lacuesta 8d30dc0888
Response.follow_all: return empty generators for empty sequences 2020-03-12 09:36:15 -03:00
Andrey Rahmatullin 388f23c30c
Merge pull request #3858 from elacuesta/multiple_feed_export_formats 
[MRG+1] Support for exporting to multiple feeds in a single crawl
 2020-03-12 14:31:57 +05:00
Eugenio Lacuesta c886a70eae
Use dict.setdefault in scrapy.utils.conf.feed_complete_default_values_from_settings 2020-03-11 21:06:51 -03:00
Eugenio Lacuesta f3bab819ab
Add tests for scrapy.utils.conf.feed_complete_default_values_from_settings 2020-03-11 20:57:31 -03:00
Eugenio Lacuesta 49156f2ecb
[doc] Feed exports: full local path as example 2020-03-11 20:45:54 -03:00
Andrey Rahmatullin 2eb990a252
Merge pull request #4410 from Lukas0907/fix-offsite-middleware-tests 
Fix handling of None in allowed_domains
 2020-03-11 23:11:35 +05:00
Lukas Anzinger 9d9dea0d69 Fix handling of None in allowed_domains. 
Nones in allowed_domains ought to be ignored and there are also tests
for that scenario. This commit fixes the handling of None and also the
accompanying tests which are now executed again.
 2020-03-07 20:34:18 +01:00
nyov 915e363db5 Remove a 'twisted.test.proto_helpers' deprecation warning 2020-03-07 18:03:25 +00:00
nyov c2c6ea376c Remove obsolete DEPRECATED_SETTINGS (deprecated.py) 2020-03-07 15:52:05 +00:00
Eugenio Lacuesta ada37c5409
Export to multiple formats in a single crawl 2020-03-05 09:33:57 -03:00
Daniel Graña c57512fa66
Merge pull request #4384 from nyov/nodeb 
Drop horribly outdated deb package build files
 2020-03-04 19:02:36 -03:00
nyov b1566a6962 Remove deprecated Crawler.spiders property 
Deprecated since 419026615 (2014, Scrapy 0.25)
 2020-03-04 21:23:42 +00:00
Mikhail Korobov ff8e826dd6
Merge pull request #4385 from nyov/obsolete 
Obsolete REDIRECT_MAX_METAREFRESH_DELAY, LOG_UNSERIALIZABLE_REQUESTS
 2020-03-05 00:04:29 +05:00
Mikhail Korobov 29f957d8cb
Merge pull request #4400 from nyov/obsolete-datatypes 
Remove deprecated SiteNode and MultiValueDict classes
 2020-03-04 23:56:36 +05:00
nyov 6c35baae25 Remove deprecated SiteNode and MultiValueDict classes 2020-03-04 00:40:11 +00:00
nyov 6400225555 Drop horribly outdated deb package build files 2020-03-03 14:03:44 +00:00
nyov 1b591ff061 Obsolete deprecated settings 
Obsolete REDIRECT_MAX_METAREFRESH_DELAY
  which has been deprecated since Scrapy 0.18

Obsolete LOG_UNSERIALIZABLE_REQUESTS
  which has been deprecated since Scrapy 1.2.0
  and is replaced by SCHEDULER_DEBUG
 2020-03-03 14:03:30 +00:00
Adrián Chaves a4dbb7754b Bump version: 1.8.0 → 2.0.0 2020-03-03 09:13:00 +01:00
Adrián Chaves 6aa0ba4553
Write release notes for Scrapy 2.0.0 (#4329) 2020-03-03 09:11:11 +01:00
MaliCN ef00f8eb8e
updated with new macOS name (#4308) (#4323) 
* changed for new name as "macOS"  (issue #4308)

* updated macOS name

* update macOS name

* updated macOS name

* update for new macOS name

* docs/intro/install.rst: fix macOS header symbols

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-29 00:42:07 +05:00
sakshamb2113 c411a51f42
Fix random failures from test_fixed_delay in some machines (#4372) 2020-02-28 13:17:02 +01:00
Andrey Rahmatullin 09d9e75c08
Merge pull request #4375 from Gallaecio/env-warn-whitelist 
Stop deprecation warnings on arbitrary SCRAPY-prefixed env vars
 2020-02-28 13:26:17 +05:00
Adrián Chaves 9aae4c0be7 Add tests for envvar setting warnings 2020-02-27 16:31:43 +01:00
Adrián Chaves 2acaa86231 Do not warn about valid environment variables 2020-02-27 15:46:21 +01:00