RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,505 Commits 49 Branches 132 Tags 155 MiB
Commit Graph

5505 Commits

Author SHA1 Message Date
Paul Tremberth e0f48c486e Add link to CoC mardown file on Github 2016-01-27 13:04:08 +01:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
Paul Tremberth 6ed08d2332 Add note for DEPTH_PRIORITY 2016-01-27 11:53:29 +01:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 1c83108893 Clarify priority adjust settings docs 
Fixes #1593
 2016-01-26 19:24:11 +01:00
Daniel Graña 85f0596c43 Merge pull request #1725 from redapple/spiderstate-notconfigured 
[MRG+1] Disable SpiderState extension if no JOBDIR set
 2016-01-26 14:32:17 -03:00
Mikhail Korobov 6660175de9 Merge pull request #1726 from redapple/mwlist-logging-long 
Use long classes names for enabled middlewares in startup logs
 2016-01-26 21:29:44 +05:00
Paul Tremberth bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Paul Tremberth c22a4e3bb8 Use long classes names for enabled middlewares in startup logs 2016-01-26 16:41:16 +01:00
Paul Tremberth 29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Paul Tremberth 0349bbf9d3 Disable SpiderState extension if no JOBDIR set 2016-01-26 15:25:15 +01:00
Paul Tremberth 0d368c5d6f Merge pull request #1724 from scrapy/robotstxt-default 
[MRG+1] Enable robots.txt handling by default for new projects.
 2016-01-26 14:19:15 +01:00
Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Elias Dorneles 2246280bb6 Merge pull request #1723 from redapple/closespider-notconfigured 
[MRG+1] Disable CloseSpider extension if no CLOSPIDER_* setting set
 2016-01-26 10:35:59 -02:00
Paul Tremberth 52daa1b953 Merge pull request #1720 from scrapy/deprecations 
[MRG+1] deprecated unused and untested code in scrapy.utils.datatypes
 2016-01-26 13:25:21 +01:00
Mikhail Korobov 1e48574576 Merge pull request #1722 from redapple/mwlist-logging 
[MRG+1] Fix logging of enabled middlewares
 2016-01-26 17:21:08 +05:00
Paul Tremberth 6ee8d8650a Disable CloseSpider extension if no CLOSPIDER_* setting set 2016-01-26 13:08:42 +01:00
Paul Tremberth 7608da8868 Fix logging of enabled middlewares 
Wrong middlewares list was being pretty-printed
(introduced in #1263)
 2016-01-26 13:01:12 +01:00
Elias Dorneles de22b6f3ac Merge pull request #1721 from Digenis/offsitemw_subdomains 
[MRG+1] tests+doc for subdomains in offsite middleware
 2016-01-26 09:37:18 -02:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Paul Tremberth 713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Mikhail Korobov 9c2aa50ea2 deprecate unused and untested scrapy.utils.datatypes.MultiValueDict 2016-01-26 13:58:56 +05:00
Mikhail Korobov 7070dae48d deprecate unused and untested scrapy.utils.datatypes.SiteNode 2016-01-26 13:56:16 +05:00
Daniel Graña a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles 2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles 23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00
Paul Tremberth fb8ab2427b Move urlparsing statement in add_http_if_no_scheme() 2016-01-25 13:13:35 +01:00
Elias Dorneles 0c44fac2b5 added tests for feed export marshal and pickle 2016-01-24 19:17:42 -02:00
Elias Dorneles 4fef2c7628 Merge pull request #1717 from lopuhin/py3-test-spidermiddleware-httperror 
[MRG+1] py3: reviewed passing test_spidermiddleware_httperror.py
 2016-01-24 18:49:22 -02:00
Konstantin Lopuhin 1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Konstantin Lopuhin 4233b3cda4 py3: reviewed passing test_spidermiddleware_httperror.py 2016-01-24 23:10:03 +03:00
Konstantin Lopuhin 097082cffa reviewed py3 compat in pipelines/images.py and pipelines/files.py 2016-01-24 23:05:23 +03:00
Konstantin Lopuhin 333d4c91fb py3: add boto to py3 test requirements, test_pipeline_files and test_pipeline_images passing now 2016-01-24 22:55:21 +03:00
Mikhail Korobov 85ae273230 Merge pull request #1715 from lopuhin/py3-test-mail 
[MRG+1] py3: fix test_mail
 2016-01-24 17:47:57 +05:00
Konstantin Lopuhin 860353b0c0 py3: unskip test_mail and scrapy/mail.py 2016-01-24 13:27:41 +03:00
Konstantin Lopuhin 9704226ee4 py3: fix test_mail - get_payload returns bytes when decode is True 2016-01-24 13:25:14 +03:00
Elias Dorneles 9fbe6f3e81 added feedexport test for xml output 2016-01-23 17:17:40 -02:00
Elias Dorneles 935b1da8c2 uses ScrapyDeprecationWarning instead of silenced PendingDeprecationWarning 2016-01-23 16:13:42 -02:00
Elias Dorneles c75f1fe46a restore bytes instead of text, for easier reviewing 2016-01-23 16:09:57 -02:00
Paul Tremberth 5f09da60c1 Revert "Use pytest.mark.parametrize decorator" 
This reverts commit 1a30a7774b.
 2016-01-22 23:48:58 +01:00
Paul Tremberth 1a30a7774b Use pytest.mark.parametrize decorator 2016-01-22 18:22:19 +01:00
Paul Tremberth 61ca8d587a Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-22 18:16:36 +01:00
Paul Tremberth 7a51d370f3 Regex-based guess_scheme() + refactor tests 2016-01-22 17:16:27 +01:00
Paul Tremberth 60052b3c68 Remove unused re import 2016-01-22 13:18:08 +01:00
Paul Tremberth be239f339c Remove unused import 2016-01-22 13:13:46 +01:00
Elias Dorneles d0955fd083 add back test for latin-1 encoding 2016-01-22 10:07:55 -02:00
Paul Tremberth 6d73e057b5 Extract guess_scheme function and refactor tests 2016-01-22 13:07:42 +01:00
Mikhail Korobov c75f7ed20b Merge pull request #1711 from scrapy/py3-parse-command-tests 
PY3 enable tests for scrapy parse command
 2016-01-22 15:21:26 +05:00