RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,025 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

5025 Commits

Author SHA1 Message Date
Mikhail Korobov ff24cbbc47 PY3 depth, offsite and referer spider middlewares; Crawler 2015-08-28 04:11:30 +05:00
Mikhail Korobov d5984bbea9 PY3 port scrapy.spiders 2015-08-28 04:11:30 +05:00
Mikhail Korobov f2edbd05de PY3 port LinkExtractor 
* tests for other link extractors are moved to test_linkextractors_deprecated.py
* in Python 3 Link is converted to use native strings for urls
* minor cleanups
 2015-08-28 04:11:30 +05:00
Mikhail Korobov f46a450080 refactor test_linkextractors 
* rename LinkExtractorTestCase to BaseSgmlLinkExtractorTestCase
* add BaseLinkExtractorTestCase link extractor tests can inherit from
  and decouple it from SgmlLinkExtractor
* add an extra check for deny_extensions
* xfail test_restrict_xpaths_with_html_entities for LxmlLinkExtractor explicitly
 2015-08-28 04:11:30 +05:00
Mikhail Korobov 9bfab53075 Merge pull request #1462 from tagatac/master 
minor: scrapy.Spider docs grammar
 2015-08-28 02:42:37 +05:00
David Tagatac 08162a15d8 minor: scrapy.Spider docs grammar 2015-08-27 17:37:16 -04:00
Mikhail Korobov 8ecc4544b3 enable console tests in PY3 2015-08-27 21:37:24 +05:00
Mikhail Korobov 9616d91e4a Merge pull request #1444 from cyberplant/bpython_support 
[MRG +1] bpython support
 2015-08-27 21:28:05 +05:00
Rick aa88fd797a Small grammatical change 2015-08-26 15:01:50 -03:00
Daniel Graña 71bd79e70f Merge pull request #1212 from Digenis/test-fix-render_template_file 
fix utils.template.render_templatefile() bug +test
 2015-08-26 14:46:42 -03:00
Νικόλαος-Διγενής Καραγιάννης 56b3cf07ff fix string manipulation in render_templatefile() 2015-08-26 20:45:41 +03:00
Νικόλαος-Διγενής Καραγιάννης f5c54072ec Test utils.template.render_templatefile() 2015-08-26 20:44:33 +03:00
Daniel Graña 026a1caffb Merge pull request #1456 from scrapy/py3-fixes 
Small Python 3 fixes
 2015-08-26 11:46:00 -03:00
Mikhail Korobov 68a47ade46 PY3 port test_logformatter 2015-08-26 02:34:21 +05:00
Mikhail Korobov 642af00bb7 fix Referer logging 2015-08-26 02:19:33 +05:00
Mikhail Korobov 7da769feb2 enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
Mikhail Korobov 787b5af30e add coverage files to gitignore 2015-08-26 01:58:33 +05:00
Mikhail Korobov cfae62f9cc Merge pull request #1441 from aivarsk/fix-common-practices 
Make common practices sample code match the comments
 2015-08-23 17:36:09 +05:00
Daniel Graña 489c76b83e Merge pull request #1447 from jdemaeyer/fix/redirectmiddleware-respect-meta-attributes 
[MRG +1] Fix RedirectMiddleware not honouring handle_httpstatus meta keys
 2015-08-22 09:46:27 -03:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
Luar Roji ee66382f29 Fixed banner display in all 3 python shells 2015-08-21 01:12:58 +01:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Aivars Kalvāns b8b1e8e544 Make common practices sample code match the comments 2015-08-19 16:54:10 +03:00
Daniel Graña 280eab2416 Merge pull request #1352 from nyov/le-bogus-links 
[MRG+1] [LinkExtractors] Ignore bogus links (#907)
 2015-08-16 00:42:55 -03:00
Andrew Scorpil de15fcdf33 [LinkExtractors] Ignore bogus links 
(rebased the code for scrapy 1.0 and made a few code improvements --nyov)
 2015-08-15 00:16:39 +00:00
Mikhail Korobov 9adb5c31c0 Merge pull request #1433 from scrapy/codecov 
Coverage and reports at codecov.io and coveralls.io
 2015-08-14 05:21:10 +05:00
Daniel Graña 27077d2b6c Forward coverage envvars needed to cover subprocess 2015-08-13 21:07:45 -03:00
Daniel Graña 43fc2f23b8 Increase coverage of version command 2015-08-13 19:30:06 -03:00
Daniel Graña 54f9837804 Document how to get coverage report 2015-08-13 19:02:51 -03:00
Daniel Graña cb9577a6d4 Do not be verbose with coverage report by default 2015-08-13 19:02:36 -03:00
Daniel Graña 108c3e63a5 Add coverage report trough codecov.io 2015-08-13 13:56:24 -03:00
Daniel Graña 12bebb6172 Merge pull request #1431 from dangra/moved-to-parsel 
Replace usage of deprecated class by its parsel\'s counterpart
 2015-08-12 14:29:34 -03:00
Daniel Graña 6c2c9a4baf Merge pull request #1430 from dangra/openssl-version 
Add openssl version to version command
 2015-08-12 14:24:43 -03:00
Daniel Graña 7f634598b6 Add openssl version to version command 2015-08-12 14:24:24 -03:00
Daniel Graña 4ce0f53f7e Replace usage of deprecated class by its parsel\'s counterpart 2015-08-12 14:21:55 -03:00
Daniel Graña 68d2311298 Merge pull request #1427 from scrapy/python3-porting-status 
add Python 3 porting status badge to README
 2015-08-12 11:15:33 -03:00
Mikhail Korobov 1ed6a01c4b Add Python 3 porting status badge to the README 2015-08-12 19:13:54 +05:00
Daniel Graña 15c1300d35 Merge pull request #1409 from eliasdorneles/migrate-parsel 
[MRG+1] Migrating selectors to use parsel
 2015-08-11 16:22:58 -03:00
Elias Dorneles a5abd19e84 make Parsel's Selector more private, remove direct dependency of ParselSelectorList 2015-08-11 15:58:29 -03:00
Elias Dorneles 766c255152 upgrade parsel and add shim for deprecated selectorlist methods 2015-08-11 15:20:33 -03:00
Daniel Graña 457b97c13c Merge pull request #1388 from scrapy/dupefilter-persist 
Dupefilter persistence
 2015-08-11 14:16:34 -03:00
Daniel Graña e2f31f3018 explicit close file on file:// scheme handler 2015-08-11 14:09:49 -03:00
Elias Dorneles e50610bd3a set base_url in kwargs to be fully backward compatible 2015-08-11 14:09:34 -03:00
Elias Dorneles 8ef5aa2ffc using bytes for response body in tests 2015-08-11 14:09:34 -03:00
Elias Dorneles 3a03ef7c08 cleanup tests for selectors and translators 2015-08-11 14:09:34 -03:00
Elias Dorneles 12579b9afa warning when ambiguous root arguments and minor cleanups 2015-08-11 14:09:34 -03:00
Elias Dorneles 26ebccd37a upgrade parsel and use its function to instantiate root for finding form 2015-08-11 14:09:34 -03:00
Elias Dorneles 2fe6d128f5 upgrade parsel and using promoted root attribute 2015-08-11 14:09:16 -03:00
Elias Dorneles 67c98b185b avoid harcoded check for selector type 2015-08-11 14:09:00 -03:00
Elias Dorneles 94c3a345b7 remove deprecated module lxmldocument 2015-08-11 14:09:00 -03:00