RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Adrián Chaves b8cf522916
Merge branch 'master' into user-friendlier-tox 2019-12-17 16:53:08 +01:00
Andrey Rahmatullin 4be2bbfe75
Merge pull request #4239 from Apuyuseng/master 
Fix mail attachs tcmime *** (#4229)
 2019-12-17 19:43:30 +05:00
Andrey Rahmatullin bb3f164280
Merge pull request #4236 from wRAR/pipeline-tests 
Add simple tests for pipelines.
 2019-12-17 19:38:06 +05:00
Adrián Chaves 20719bac5c Fix import error 2019-12-17 15:09:43 +01:00
Adrián Chaves ab13221b0b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2019-12-17 14:06:14 +01:00
Marc Hernández 63cf5c75c8 Fix E502: backslash is redundant between brackets (#4238) 2019-12-17 13:53:15 +01:00
apu 7d0096da6e
Fix mail attachs tcmime *** (#4229) 
When the file name consists of alphanumeric characters, it is normal to receive the attachment name.
However，However, problems will occur if the file name is changed to Chinese.
This has nothing to do with the file type
 2019-12-17 09:47:01 +08:00
Andrey Rakhmatullin 5980d3bbff Add simple tests for pipelines. 2019-12-16 22:41:28 +05:00
Mikhail Korobov e9b24d62c6
Merge pull request #4231 from noviluni/docs_fix 
Docs fix
 2019-12-16 21:56:43 +05:00
Andrey Rakhmatullin 900de7c146 Fix the reactor_pytest fixture. 2019-12-16 21:11:58 +05:00
Andrey Rakhmatullin 039e6fe691 Refactor install_asyncio_reactor slightly. 2019-12-16 20:17:41 +05:00
Eugenio Lacuesta 451e7a616e
Scan callbacks/errbacks for return statements with values different than None 2019-12-16 11:30:40 -03:00
Andrey Rakhmatullin 2db7d45378 Enable skipping tests based on --reactor. 2019-12-16 19:25:49 +05:00
marc a59bb279d1 add year through code 2019-12-15 17:33:00 +01:00
marc 1aab20e1ce update copyright notice year 2019-12-14 10:37:31 +01:00
marc e3a3ad4aaf remove reference to old (Python 2.7) environment 2019-12-14 10:34:31 +01:00
Andrey Rakhmatullin a1605cade6 Hide utils.defer.isfuture(). 2019-12-13 19:35:09 +05:00
Andrey Rakhmatullin afc886e578 Simplify tox.ini asyncio entries. 2019-12-13 19:34:58 +05:00
Adrián Chaves a4ef9750f9 Fix Flake8-reported issues 2019-12-13 14:32:06 +01:00
Adrián Chaves b5c4c2cae8
Keep 2 spaces between code and inline comments (#4195) 2019-12-13 14:20:48 +01:00
Andrey Rakhmatullin bfb78b8dea Add CrawlerProcess tests for ASYNCIO_ENABLED. 2019-12-13 18:12:07 +05:00
Andrey Rakhmatullin 855bbebc8b Move install_asyncio_reactor() from commands to CrawlerProcess. 2019-12-13 18:11:49 +05:00
Andrey Rakhmatullin 69cd2e247e Move a bunch of "from twisted.internet import reactor" inside functions. 2019-12-13 18:07:55 +05:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Andrey Rakhmatullin 352ddbb336 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-13 18:07:23 +05:00
Andrey Rahmatullin 02cdc53fb8
Add a test for a CrawlerProcess script. (#4218) 
* Add a test for a CrawlerProcess script.

* Add tests/CrawlerProcess to collect_ignore.

* Remove an extra line.

* Fix/improve conftest.py.
 2019-12-13 18:04:05 +05:00
Eugenio Lacuesta 21a2f8c6f8
Merge branch 'master' into response_follow_all 2019-12-10 17:30:59 -03:00
Adrián Chaves 07b8cd28aa
Mark bandit’s 402 check as addressed by #4180 (#4181) 2019-12-05 14:48:31 +01:00
Adrián Chaves 1fc2b140c1
Merge branch 'master' into documentation-coverage 2019-12-05 14:43:36 +01:00
Andrey Rahmatullin 076f0764b7
Merge pull request #4121 from scrapy/remove-six-code 
Remove six-related code and __future__ imports
 2019-12-05 18:20:14 +05:00
Adrián Chaves d7b1c138f1
Merge branch 'master' into user-friendlier-tox 2019-12-05 14:02:24 +01:00
Adrián Chaves 83b8046fdc Do not indent doctests from the documentation unnecessarily 2019-12-05 13:52:31 +01:00
Mikhail Korobov 250da28952
Merge pull request #4170 from mabelvj/4133-handle-start_url 
Raise error when start_url found instead of start_urls.
 2019-12-05 17:47:03 +05:00
Adrián Chaves 57e36b5f0d
Merge branch 'master' into spider-name-collision 2019-12-05 13:03:43 +01:00
Andrey Rahmatullin 7079d12c5e
Merge pull request #4212 from grammy-jiang/fix-imports 
Convert the relative imports to absolute imports
 2019-12-05 12:44:22 +05:00
Andrey Rahmatullin aaf94affee
Merge pull request #4213 from dqwerter/patch-1 
Update overview.rst | Fix an inconsistency
 2019-12-05 12:43:24 +05:00
Wang Qin af624ef414
Update overview.rst | Fix an inconsistency 
There exists an inconsistency between the code (line 37 - 38) and the output 'quotes.json' (line 56 - 68). 

Note that even though according to line 53 - 54  'quotes.json' is "reformatted here for better readability", it cannot explain why the "author" field precedes the "text" field. 

Intended output for the code BEFORE change: 
    [{
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d",
        "author": "Jane Austen"
    },
    {
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d",
        "author": "Groucho Marx"
    },
    {
        "text": "\u201cA day without sunshine is like, you know, night.\u201d",
        "author": "Steve Martin"
    },
    ...]

Intended output for the code After change (the inconsistency is fixed): 
    [{
        "author": "Jane Austen",
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"
    },
    {
        "author": "Groucho Marx",
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d"
    },
    {
        "author": "Steve Martin",
        "text": "\u201cA day without sunshine is like, you know, night.\u201d"
    },
    ...]
 2019-12-05 09:29:12 +08:00
Grammy Jiang 9b4b43f8ac
Convert the relative imports to absolute imports 
This commits converts the relative imports to absolute imports in the
entire package
 2019-12-05 11:25:19 +11:00
Eugenio Lacuesta 2c010152c3
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-12-04 15:43:02 -03:00
Andrey Rakhmatullin 0b9f29215f Update .travis.yml. 2019-12-04 22:06:35 +05:00
Andrey Rakhmatullin 97fb61cec8 Move an import to postpone another "import twisted.internet.reactor". 2019-12-04 21:53:07 +05:00
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin c079d5002b Run tests without asyncio support by default, add py35-asyncio and py38-asyncio envs. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 794cf71806 Fix or ignore flake8 problems. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 3ba25ccbd3 Don't use asyncio.iscoroutine, as it is True for generators. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin f41c2f3874 Add py38-no-asyncio to Travis. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin cc19ab5439 Add tests that check asyncio support. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 2fbe7d49dc Log asyncio support on spider start. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin b04b541372 Install the asyncio reactor only in scrapy.cmdline. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 8d8fbddbde Switch to the released version of pytest-twisted. 2019-12-04 21:32:56 +05:00