RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,612 Commits 49 Branches 132 Tags 163 MiB
Commit Graph

7612 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 855bbebc8b Move install_asyncio_reactor() from commands to CrawlerProcess. 2019-12-13 18:11:49 +05:00
Andrey Rakhmatullin 69cd2e247e Move a bunch of "from twisted.internet import reactor" inside functions. 2019-12-13 18:07:55 +05:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Andrey Rakhmatullin 352ddbb336 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-13 18:07:23 +05:00
Andrey Rahmatullin 02cdc53fb8
Add a test for a CrawlerProcess script. (#4218) 
* Add a test for a CrawlerProcess script.

* Add tests/CrawlerProcess to collect_ignore.

* Remove an extra line.

* Fix/improve conftest.py.
 2019-12-13 18:04:05 +05:00
Adrián Chaves 07b8cd28aa
Mark bandit’s 402 check as addressed by #4180 (#4181) 2019-12-05 14:48:31 +01:00
Andrey Rahmatullin 076f0764b7
Merge pull request #4121 from scrapy/remove-six-code 
Remove six-related code and __future__ imports
 2019-12-05 18:20:14 +05:00
Mikhail Korobov 250da28952
Merge pull request #4170 from mabelvj/4133-handle-start_url 
Raise error when start_url found instead of start_urls.
 2019-12-05 17:47:03 +05:00
Andrey Rahmatullin 7079d12c5e
Merge pull request #4212 from grammy-jiang/fix-imports 
Convert the relative imports to absolute imports
 2019-12-05 12:44:22 +05:00
Andrey Rahmatullin aaf94affee
Merge pull request #4213 from dqwerter/patch-1 
Update overview.rst | Fix an inconsistency
 2019-12-05 12:43:24 +05:00
Wang Qin af624ef414
Update overview.rst | Fix an inconsistency 
There exists an inconsistency between the code (line 37 - 38) and the output 'quotes.json' (line 56 - 68). 

Note that even though according to line 53 - 54  'quotes.json' is "reformatted here for better readability", it cannot explain why the "author" field precedes the "text" field. 

Intended output for the code BEFORE change: 
    [{
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d",
        "author": "Jane Austen"
    },
    {
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d",
        "author": "Groucho Marx"
    },
    {
        "text": "\u201cA day without sunshine is like, you know, night.\u201d",
        "author": "Steve Martin"
    },
    ...]

Intended output for the code After change (the inconsistency is fixed): 
    [{
        "author": "Jane Austen",
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"
    },
    {
        "author": "Groucho Marx",
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d"
    },
    {
        "author": "Steve Martin",
        "text": "\u201cA day without sunshine is like, you know, night.\u201d"
    },
    ...]
 2019-12-05 09:29:12 +08:00
Grammy Jiang 9b4b43f8ac
Convert the relative imports to absolute imports 
This commits converts the relative imports to absolute imports in the
entire package
 2019-12-05 11:25:19 +11:00
Eugenio Lacuesta 2c010152c3
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-12-04 15:43:02 -03:00
Andrey Rakhmatullin 0b9f29215f Update .travis.yml. 2019-12-04 22:06:35 +05:00
Andrey Rakhmatullin 97fb61cec8 Move an import to postpone another "import twisted.internet.reactor". 2019-12-04 21:53:07 +05:00
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin c079d5002b Run tests without asyncio support by default, add py35-asyncio and py38-asyncio envs. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 794cf71806 Fix or ignore flake8 problems. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 3ba25ccbd3 Don't use asyncio.iscoroutine, as it is True for generators. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin f41c2f3874 Add py38-no-asyncio to Travis. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin cc19ab5439 Add tests that check asyncio support. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 2fbe7d49dc Log asyncio support on spider start. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin b04b541372 Install the asyncio reactor only in scrapy.cmdline. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 8d8fbddbde Switch to the released version of pytest-twisted. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 63c3c62305 Add utils.deferred_from_coro. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 9777639533 Run tests using the asyncio reactor. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 1b437bbe9f Install the asyncio reactor on "import scrapy". 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 1b35260625 Add a test for downloader middlewares using Deferreds. 2019-12-04 21:32:56 +05:00
Grammy Jiang 74627033c4 Remove the used import and re-arrange the imports (#4208) 
This commit removes unused import and re-arrange the imports in cookies
module
 2019-12-04 14:24:14 +01:00
Grammy Jiang 702333478d Re-arrange the imports in httpcache module (#4209) 
This commit re-arrange the imports in httpcache module to follow pep8
 2019-12-04 14:23:28 +01:00
Grammy Jiang 5d8d4bb7d7 Re-arrange the imports in the httpproxy module (#4210) 
This commit re-arranges the imports in the httpproxy module to follow
pep8
 2019-12-04 14:22:10 +01:00
Mikhail Korobov 9b7452211a
Merge pull request #4099 from BurnzZ/itemloader-docs 
update docs of scrapy.loader.ItemLoader.item
 2019-12-03 13:14:45 +05:00
Grammy Jiang d1cdfb4701 Use pprint.pformat on overridden settings (#4199) 
Keeps consistency with scrapy.middleware
 2019-11-29 09:13:57 +01:00
Mikhail Korobov 3a5b86220f
Merge pull request #4194 from Gallaecio/intersphinx 
Use InterSphinx for coverage links
 2019-11-28 17:36:19 +05:00
Adrián Chaves b73fc99b60 Use InterSphinx for coverage links 2019-11-26 10:31:55 +01:00
Adrián Chaves 6d9ed6146d
Merge branch 'master' into remove-six-code 2019-11-25 10:34:21 +01:00
Andrey Rahmatullin 8a1c99676e
Merge pull request #3899 from elacuesta/py3_single_argument_processors 
[Py3] Item loaders: allow single argument functions as processors
 2019-11-25 13:47:58 +05:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 55cc5c9068
Skip pickle in bandit check 2019-11-22 12:41:31 -03:00
Eugenio Lacuesta 5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Mikhail Korobov 16e0636dcf
Merge pull request #4186 from Gallaecio/lgtm 
Remove unused imports
 2019-11-22 12:28:29 +05:00
Adrián Chaves 9b5053c564 Undo unintended tox.ini changes 2019-11-21 22:00:34 +01:00
Mikhail Korobov 0d416c6191
Merge pull request #4185 from Gallaecio/intersphinx 
Use InterSphinx for links to the pytest and tox documentation
 2019-11-21 23:27:57 +05:00
Mabel Villalba 070b3a4e84
Merge branch 'master' into 4133-handle-start_url 2019-11-21 17:10:31 +01:00
Mabel Villalba 1718e450ef [start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test. 2019-11-21 17:06:47 +01:00
Mikhail Korobov 82c01c261f
Merge pull request #4184 from Gallaecio/doctest 
Make debug doctests pass
 2019-11-21 20:53:56 +05:00
Eugenio Lacuesta bf503869e9
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-21 10:58:24 -03:00
Adrián Chaves b232881356 Restore intentional import of unused objects 2019-11-21 14:30:10 +01:00
Mikhail Korobov 91c9576675
Merge pull request #4183 from Gallaecio/sphinx-extensions-2 
Enable sphinx-hoverxref for all references
 2019-11-21 18:25:21 +05:00
Mikhail Korobov d2b73b8e21
Merge pull request #4114 from scrapy/remove-py2-tests 
Remove py2 tests
 2019-11-21 18:22:10 +05:00