RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,025 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

246 Commits

Author SHA1 Message Date
Mikhail Korobov ff24cbbc47 PY3 depth, offsite and referer spider middlewares; Crawler 2015-08-28 04:11:30 +05:00
Mikhail Korobov d5984bbea9 PY3 port scrapy.spiders 2015-08-28 04:11:30 +05:00
Mikhail Korobov f2edbd05de PY3 port LinkExtractor 
* tests for other link extractors are moved to test_linkextractors_deprecated.py
* in Python 3 Link is converted to use native strings for urls
* minor cleanups
 2015-08-28 04:11:30 +05:00
Mikhail Korobov f46a450080 refactor test_linkextractors 
* rename LinkExtractorTestCase to BaseSgmlLinkExtractorTestCase
* add BaseLinkExtractorTestCase link extractor tests can inherit from
  and decouple it from SgmlLinkExtractor
* add an extra check for deny_extensions
* xfail test_restrict_xpaths_with_html_entities for LxmlLinkExtractor explicitly
 2015-08-28 04:11:30 +05:00
Mikhail Korobov 8ecc4544b3 enable console tests in PY3 2015-08-27 21:37:24 +05:00
Mikhail Korobov 9616d91e4a Merge pull request #1444 from cyberplant/bpython_support 
[MRG +1] bpython support
 2015-08-27 21:28:05 +05:00
Daniel Graña 71bd79e70f Merge pull request #1212 from Digenis/test-fix-render_template_file 
fix utils.template.render_templatefile() bug +test
 2015-08-26 14:46:42 -03:00
Νικόλαος-Διγενής Καραγιάννης f5c54072ec Test utils.template.render_templatefile() 2015-08-26 20:44:33 +03:00
Mikhail Korobov 68a47ade46 PY3 port test_logformatter 2015-08-26 02:34:21 +05:00
Mikhail Korobov 7da769feb2 enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Andrew Scorpil de15fcdf33 [LinkExtractors] Ignore bogus links 
(rebased the code for scrapy 1.0 and made a few code improvements --nyov)
 2015-08-15 00:16:39 +00:00
Daniel Graña 27077d2b6c Forward coverage envvars needed to cover subprocess 2015-08-13 21:07:45 -03:00
Daniel Graña 43fc2f23b8 Increase coverage of version command 2015-08-13 19:30:06 -03:00
Daniel Graña 108c3e63a5 Add coverage report trough codecov.io 2015-08-13 13:56:24 -03:00
Daniel Graña 15c1300d35 Merge pull request #1409 from eliasdorneles/migrate-parsel 
[MRG+1] Migrating selectors to use parsel
 2015-08-11 16:22:58 -03:00
Elias Dorneles 766c255152 upgrade parsel and add shim for deprecated selectorlist methods 2015-08-11 15:20:33 -03:00
Daniel Graña 457b97c13c Merge pull request #1388 from scrapy/dupefilter-persist 
Dupefilter persistence
 2015-08-11 14:16:34 -03:00
Elias Dorneles e50610bd3a set base_url in kwargs to be fully backward compatible 2015-08-11 14:09:34 -03:00
Elias Dorneles 8ef5aa2ffc using bytes for response body in tests 2015-08-11 14:09:34 -03:00
Elias Dorneles 3a03ef7c08 cleanup tests for selectors and translators 2015-08-11 14:09:34 -03:00
Elias Dorneles 12579b9afa warning when ambiguous root arguments and minor cleanups 2015-08-11 14:09:34 -03:00
Elias Dorneles 2fe6d128f5 upgrade parsel and using promoted root attribute 2015-08-11 14:09:16 -03:00
Elias Dorneles 94c3a345b7 remove deprecated module lxmldocument 2015-08-11 14:09:00 -03:00
Elias Dorneles 17d7347a36 update minimal parsel version, add deprecated classes for csstranslator modules, fix test 2015-08-11 14:09:00 -03:00
Elias Dorneles 01d948f0fd remove selector support for LxmlDocument DOM cache and add deprecation warning for _root argument 2015-08-11 14:08:43 -03:00
Elias Dorneles ce21884a97 migrating scrapy Selector to use Parsel 2015-08-11 14:08:43 -03:00
Mikhail Korobov fa123b335b Merge pull request #1421 from dangra/nyov/lazyload-downloadhandlers 
lazy-load downloadhandlers (continuation of #1357)
 2015-08-11 18:59:47 +05:00
Mikhail Korobov fa3d84b050 Merge pull request #1420 from scrapy/py3-trackrefs 
PY3: port scrapy.utils.trackref
 2015-08-11 18:21:34 +05:00
nyov 8c7997083f lazy-loading for DownloadHandlers 2015-08-10 16:52:44 -03:00
Daniel Graña accd28cc71 PY3: port scrapy.utils.trackref 2015-08-10 15:13:29 -03:00
Daniel Graña 8d45b3c481 Do not fail representing non-http requests 2015-08-10 14:10:58 -03:00
Mikhail Korobov 56be610e6e TST a test for --profile option 2015-08-08 04:54:47 +05:00
nyov 17b5e9fb86 PY3 response bodies as bytes 2015-08-08 04:23:32 +05:00
nyov 45d441d444 PY3 fix test loader 2015-08-08 04:23:16 +05:00
nyov 34eced0ee8 PY3 fix tests pipelines images 2015-08-08 04:23:06 +05:00
nyov 991197003b PY3 fix tests pipelines files 2015-08-08 04:22:58 +05:00
nyov 6e762ce25c PY3 renames (six types) 2015-08-08 04:21:58 +05:00
nyov 3e6d6c43ac PY3 fix test cmdline 2015-08-08 04:19:55 +05:00
nyov 4d41cc0dc4 PY3 split requirements into files 2015-08-08 04:19:37 +05:00
Daniel Graña c5ffa1a59f Twisted 15.3.0 does not raises expected exception serializing lambda functions 2015-08-06 16:40:21 -03:00
Mikhail Korobov 91991e0bdc PY port scrapy.utils.reqser 2015-08-04 02:17:46 +05:00
Mikhail Korobov 95e6bd2f8d Merge pull request #1398 from scrapy/py3-cookies 
PY3 port http cookies handling
 2015-08-04 01:23:37 +05:00
Daniel Graña f4fc05c2a1 Do not propagate cookie log messages in tests so TopLevelFormatter does not rewrite them 2015-08-03 17:18:08 -03:00
Daniel Graña c6adf648dc PY3 port COOKIES_DEBUG and add tests 2015-08-03 16:28:29 -03:00
Daniel Graña dba7e39f61 Do not break cookie parsing on non-utf8 headers 2015-08-03 10:53:40 -03:00
Daniel Graña cbfb24dbeb Merge pull request #1353 from nyov/test-no-proxies 
[MRG+1] Unset environment proxies for tests
 2015-08-03 09:49:50 -03:00
Daniel Graña 5e6c492967 Merge pull request #1364 from jdemaeyer/enhancement/spider-handles-redirects 
[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list
 2015-08-02 23:00:00 -03:00
Daniel Graña 5f02ef82e8 PY3 port http cookies handling 2015-08-02 22:28:45 -03:00