RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,100 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

278 Commits

Author SHA1 Message Date
Daniel Graña 51b1249033 Revert "test suite requires recent queuelib to pass because it depends on queuelib test suite" 
This reverts commit a3390afc66.
 2015-09-18 02:27:07 -03:00
Daniel Graña a3390afc66 test suite requires recent queuelib to pass because it depends on queuelib test suite 2015-09-17 19:42:29 -03:00
Daniel Graña 3c596dcf46 Merge pull request #1467 from dacjames/master 
add support for a nested loaders
 2015-09-16 22:03:19 -03:00
Elias Dorneles 430e90f4b2 minor refactor on metarefresh redirect mware test 2015-09-13 12:29:41 -03:00
Elias Dorneles 81950f773d made encoding conversion more explicit, added test for header with utf-8 encoding replicating what browsers do 2015-09-11 18:51:48 -03:00
Elias Dorneles 179c4588ca adding test for latin1 location 2015-09-10 19:36:47 -03:00
Elias Dorneles defa899135 PY3 port redirect middleware 2015-09-10 16:31:59 -03:00
Daniel Graña 9a64d8ff97 fix scrapy squeue tests after recent changes to queuelib 2015-09-09 15:55:55 -03:00
Daniel Collins 311d5cd495 split nested_loader into seperate methods 2015-09-04 13:15:48 -07:00
Daniel Collins 425e35ee90 removed commented code and add test for error handling 2015-09-04 13:15:48 -07:00
Daniel Collins 1f95af3c07 add support for a nested loaders 2015-09-04 13:15:48 -07:00
Daniel Graña dd473145f2 Merge pull request #1473 from ArturGaspar/master 
[MRG+1] Support for returning deferreds in middlewares
 2015-09-02 13:04:26 -03:00
Artur Gaspar dd83f6123d Test for same response object in downloader middleware test. 2015-09-02 11:36:15 -03:00
Artur Gaspar 2748b38592 Test for not calling the download function when downloader middleware returns a response in process_request. 2015-09-02 11:15:37 -03:00
Artur Gaspar 1f4af9d81e Test for robots.txt middleware for logger not being called when the request for robots.txt is ignored. 2015-09-02 10:33:53 -03:00
Artur Gaspar 668e5fd257 Test for robots.txt middleware for processing a request for which the robots.txt parser is ready. 2015-09-02 10:23:24 -03:00
Artur Gaspar a6a629e707 Call actual error logger in robots.txt middleware tests. 2015-09-02 10:23:24 -03:00
Artur Gaspar 9ce9a293a6 Always check robots.txt before making another request in RobotsTxtMiddleware. 2015-09-02 10:23:24 -03:00
Daniel Graña 3919ad64c5 Merge pull request #1470 from scrapy/py3-middlewares 
[MRG] port some downloader middlewares to Python 3
 2015-09-02 00:17:37 -03:00
Olaf Dietsche 2c28b53cc0 Add tests with incomplete buttons 2015-09-01 12:35:49 +02:00
Mikhail Korobov c44cafe4f5 PY3 fix UserAgentMiddleware tests 2015-09-01 04:11:33 +05:00
Mikhail Korobov 3a9c73bc5d PY3 fix DownloaderStats middleware tests 2015-09-01 04:09:15 +05:00
Mikhail Korobov 78a4cd0f1c PY3 fix HttpAuthMiddleware tests 2015-09-01 04:00:26 +05:00
Mikhail Korobov 179a4409fb PY3 enable DownloadTimeoutMiddleware tests 2015-09-01 03:58:25 +05:00
Mikhail Korobov 3cf1911a92 PY3 fix DefaultHeadersMiddleware tests 2015-09-01 03:57:05 +05:00
Mikhail Korobov 45101829a5 PY3 fix AjacCrawlable middleware tests 2015-09-01 03:49:52 +05:00
Olaf Dietsche e5f26078fa Add test: submit form with button control 2015-08-31 19:57:45 +02:00
Mikhail Korobov 44bfcbcf0f TST split LinkExtractorTestCase.test_extraction into several methods; remove duplicated test 2015-08-31 00:49:38 +05:00
Mikhail Korobov f7052413e0 PY3 raise an exception if bytes are passed as url to Link constructor 2015-08-28 23:04:02 +05:00
Mikhail Korobov 9bfe6ece59 Merge branch 'master' into py3-linkextractors 
Conflicts:
	scrapy/linkextractors/lxmlhtml.py
	tests/test_linkextractors.py
 2015-08-28 04:53:32 +05:00
Mikhail Korobov ff24cbbc47 PY3 depth, offsite and referer spider middlewares; Crawler 2015-08-28 04:11:30 +05:00
Mikhail Korobov d5984bbea9 PY3 port scrapy.spiders 2015-08-28 04:11:30 +05:00
Mikhail Korobov f2edbd05de PY3 port LinkExtractor 
* tests for other link extractors are moved to test_linkextractors_deprecated.py
* in Python 3 Link is converted to use native strings for urls
* minor cleanups
 2015-08-28 04:11:30 +05:00
Mikhail Korobov f46a450080 refactor test_linkextractors 
* rename LinkExtractorTestCase to BaseSgmlLinkExtractorTestCase
* add BaseLinkExtractorTestCase link extractor tests can inherit from
  and decouple it from SgmlLinkExtractor
* add an extra check for deny_extensions
* xfail test_restrict_xpaths_with_html_entities for LxmlLinkExtractor explicitly
 2015-08-28 04:11:30 +05:00
Julia Medina aa31811cfd Merge pull request #1214 from rgtk/link-rel 
[MRG+1] Support link rel attribute with multiple values
 2015-08-27 19:57:54 -03:00
Rafał Gutkowski cb3007c066 support link rel attribute with multiple values 2015-08-27 20:13:47 +02:00
Mikhail Korobov 8ecc4544b3 enable console tests in PY3 2015-08-27 21:37:24 +05:00
Mikhail Korobov 9616d91e4a Merge pull request #1444 from cyberplant/bpython_support 
[MRG +1] bpython support
 2015-08-27 21:28:05 +05:00
Daniel Graña 71bd79e70f Merge pull request #1212 from Digenis/test-fix-render_template_file 
fix utils.template.render_templatefile() bug +test
 2015-08-26 14:46:42 -03:00
Νικόλαος-Διγενής Καραγιάννης f5c54072ec Test utils.template.render_templatefile() 2015-08-26 20:44:33 +03:00
Mikhail Korobov 68a47ade46 PY3 port test_logformatter 2015-08-26 02:34:21 +05:00
Mikhail Korobov 7da769feb2 enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Andrew Scorpil de15fcdf33 [LinkExtractors] Ignore bogus links 
(rebased the code for scrapy 1.0 and made a few code improvements --nyov)
 2015-08-15 00:16:39 +00:00
Daniel Graña 27077d2b6c Forward coverage envvars needed to cover subprocess 2015-08-13 21:07:45 -03:00
Daniel Graña 43fc2f23b8 Increase coverage of version command 2015-08-13 19:30:06 -03:00
Daniel Graña 108c3e63a5 Add coverage report trough codecov.io 2015-08-13 13:56:24 -03:00
Daniel Graña 15c1300d35 Merge pull request #1409 from eliasdorneles/migrate-parsel 
[MRG+1] Migrating selectors to use parsel
 2015-08-11 16:22:58 -03:00
Elias Dorneles 766c255152 upgrade parsel and add shim for deprecated selectorlist methods 2015-08-11 15:20:33 -03:00