RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,209 Commits 49 Branches 132 Tags 130 MiB
Commit Graph

2639 Commits

Author SHA1 Message Date
Mikhail Korobov 7da769feb2 enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
Jakob de Maeyer d164398a27 Fix RedirectMiddleware not honouring meta handle_httpstatus keys 2015-08-21 13:22:42 +02:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Andrew Scorpil de15fcdf33 [LinkExtractors] Ignore bogus links 
(rebased the code for scrapy 1.0 and made a few code improvements --nyov)
 2015-08-15 00:16:39 +00:00
nyov 1b4fd3a8df Support anonymous connections in S3DownloadHandler 
Also consider any unknown keyword args for S3DownloadHandler as
arguments to pass on to S3Connection (e.g. proxy settings).
 2015-08-14 22:12:12 +00:00
Daniel Graña 27077d2b6c Forward coverage envvars needed to cover subprocess 2015-08-13 21:07:45 -03:00
Daniel Graña 43fc2f23b8 Increase coverage of version command 2015-08-13 19:30:06 -03:00
Daniel Graña 108c3e63a5 Add coverage report trough codecov.io 2015-08-13 13:56:24 -03:00
Daniel Graña 15c1300d35 Merge pull request #1409 from eliasdorneles/migrate-parsel 
[MRG+1] Migrating selectors to use parsel
 2015-08-11 16:22:58 -03:00
Elias Dorneles 766c255152 upgrade parsel and add shim for deprecated selectorlist methods 2015-08-11 15:20:33 -03:00
Daniel Graña 457b97c13c Merge pull request #1388 from scrapy/dupefilter-persist 
Dupefilter persistence
 2015-08-11 14:16:34 -03:00
Elias Dorneles e50610bd3a set base_url in kwargs to be fully backward compatible 2015-08-11 14:09:34 -03:00
Elias Dorneles 8ef5aa2ffc using bytes for response body in tests 2015-08-11 14:09:34 -03:00
Elias Dorneles 3a03ef7c08 cleanup tests for selectors and translators 2015-08-11 14:09:34 -03:00
Elias Dorneles 12579b9afa warning when ambiguous root arguments and minor cleanups 2015-08-11 14:09:34 -03:00
Elias Dorneles 2fe6d128f5 upgrade parsel and using promoted root attribute 2015-08-11 14:09:16 -03:00
Elias Dorneles 94c3a345b7 remove deprecated module lxmldocument 2015-08-11 14:09:00 -03:00
Elias Dorneles 17d7347a36 update minimal parsel version, add deprecated classes for csstranslator modules, fix test 2015-08-11 14:09:00 -03:00
Elias Dorneles 01d948f0fd remove selector support for LxmlDocument DOM cache and add deprecation warning for _root argument 2015-08-11 14:08:43 -03:00
Elias Dorneles ce21884a97 migrating scrapy Selector to use Parsel 2015-08-11 14:08:43 -03:00
Mikhail Korobov fa123b335b Merge pull request #1421 from dangra/nyov/lazyload-downloadhandlers 
lazy-load downloadhandlers (continuation of #1357)
 2015-08-11 18:59:47 +05:00
Mikhail Korobov fa3d84b050 Merge pull request #1420 from scrapy/py3-trackrefs 
PY3: port scrapy.utils.trackref
 2015-08-11 18:21:34 +05:00
nyov 8c7997083f lazy-loading for DownloadHandlers 2015-08-10 16:52:44 -03:00
Daniel Graña accd28cc71 PY3: port scrapy.utils.trackref 2015-08-10 15:13:29 -03:00
Daniel Graña 8d45b3c481 Do not fail representing non-http requests 2015-08-10 14:10:58 -03:00
Mikhail Korobov 56be610e6e TST a test for --profile option 2015-08-08 04:54:47 +05:00
nyov 17b5e9fb86 PY3 response bodies as bytes 2015-08-08 04:23:32 +05:00
nyov 45d441d444 PY3 fix test loader 2015-08-08 04:23:16 +05:00
nyov 34eced0ee8 PY3 fix tests pipelines images 2015-08-08 04:23:06 +05:00
nyov 991197003b PY3 fix tests pipelines files 2015-08-08 04:22:58 +05:00
nyov 6e762ce25c PY3 renames (six types) 2015-08-08 04:21:58 +05:00
nyov 3e6d6c43ac PY3 fix test cmdline 2015-08-08 04:19:55 +05:00
nyov 4d41cc0dc4 PY3 split requirements into files 2015-08-08 04:19:37 +05:00
Daniel Graña c5ffa1a59f Twisted 15.3.0 does not raises expected exception serializing lambda functions 2015-08-06 16:40:21 -03:00
Mikhail Korobov 91991e0bdc PY port scrapy.utils.reqser 2015-08-04 02:17:46 +05:00
Mikhail Korobov 95e6bd2f8d Merge pull request #1398 from scrapy/py3-cookies 
PY3 port http cookies handling
 2015-08-04 01:23:37 +05:00
Daniel Graña f4fc05c2a1 Do not propagate cookie log messages in tests so TopLevelFormatter does not rewrite them 2015-08-03 17:18:08 -03:00
Daniel Graña c6adf648dc PY3 port COOKIES_DEBUG and add tests 2015-08-03 16:28:29 -03:00
Daniel Graña dba7e39f61 Do not break cookie parsing on non-utf8 headers 2015-08-03 10:53:40 -03:00
Daniel Graña cbfb24dbeb Merge pull request #1353 from nyov/test-no-proxies 
[MRG+1] Unset environment proxies for tests
 2015-08-03 09:49:50 -03:00
Daniel Graña 5e6c492967 Merge pull request #1364 from jdemaeyer/enhancement/spider-handles-redirects 
[MRG+1] Make RedirectMiddleware respect Spider.handle_httpstatus_list
 2015-08-02 23:00:00 -03:00
Daniel Graña 5f02ef82e8 PY3 port http cookies handling 2015-08-02 22:28:45 -03:00
nyov b2fd9bdb63 lazy-load s3 boto 2015-08-01 03:43:03 +00:00
Daniel Graña 71c6bd920e Merge pull request #1396 from scrapy/py3-utils-response 
PY3 port scrapy.utils.response
 2015-07-31 17:53:53 -03:00
Mikhail Korobov 9316b2317b remove scrapy.utils.testsite from PY3 ignores 2015-08-01 00:43:13 +05:00
Mikhail Korobov 7f927f68e1 PY3 port scrapy.utils.response 2015-08-01 00:35:43 +05:00
Mikhail Korobov d05cf6e0af Merge branch 'responsetypes' 2015-07-31 23:50:06 +05:00
Mikhail Korobov 02b5182608 small ResponseTypes cleanup 2015-07-31 23:48:49 +05:00
Daniel Graña de6e013b9a fix form requests tests on py3 2015-07-31 14:54:07 -03:00
Daniel Graña 786f62664b Merge pull request #1137 from DharmeshPandav/patch-1 
Update form.py to improve existing capability
 2015-07-31 11:00:57 -03:00
Gregory Vigo Torres 06b91da943 using bytes for body constant 2015-07-31 14:31:11 +02:00
Gregory Vigo Torres 5d75d44f2c removed test_responsetypes from py3ignores 2015-07-30 13:07:42 +02:00
nyov e044bfa60f PY3 fix test downloadermiddleware decompression 2015-07-29 13:51:01 +00:00
nyov ec8afbc060 PY3 fix test pipeline media 2015-07-29 13:51:01 +00:00
nyov 8ab1648a36 PY3 fix test middleware 2015-07-29 13:51:01 +00:00
Mikhail Korobov 757f983a97 PY3: enable RobotsTxtMiddleware tests 2015-07-29 18:37:04 +05:00
nyov 1a1bc2c6b5 PY3 port scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware 2015-07-29 04:21:57 +00:00
Mikhail Korobov c78e2636f9 TST a test to show that dupefilter persistence is not working 2015-07-28 14:31:42 +05:00
Daniel Graña 2c937f3554 PY3 port scrapy.dupefilters 2015-07-25 18:10:29 +02:00
Daniel Graña 776064aee1 PY3 port signals 2015-07-25 18:04:55 +02:00
Mikhail Korobov c53d6d7460 Merge pull request #1383 from zoliszeredi/py3/fix_test_utils_defer 
Py3/fix test utils defer
 2015-07-25 17:43:50 +02:00
Mikhail Korobov 7874bb9f13 PY3 port utils.request 2015-07-25 17:14:56 +02:00
Daniel Graña f750ee4c00 Remove ignored passing tests from py3 branch 2015-07-25 16:51:22 +02:00
Daniel Graña ea56067e40 ignore moved files from py3 tests 2015-07-25 16:39:02 +02:00
Mikhail Korobov e438df7028 PY3 enable Request and Response tests 2015-07-25 13:10:13 +02:00
Mikhail Korobov e853d9e910 partial port of Request and Response 2015-07-25 13:08:44 +02:00
Zoltán Szeredi 1bf7a98000 Remove test_utils_defer from ignores. 2015-07-25 13:05:50 +02:00
Zoltán Szeredi 9aaa8f8125 Improve python3 support for test_utils_defer. 2015-07-25 12:58:39 +02:00
Mikhail Korobov f576b3ffee [tmp] improve python 3 support for scrapy.utils.url 2015-07-25 00:40:45 +02:00
Mikhail Korobov a7b4a3e7f9 cleanup 
* run test_utils_python in Python 3;
* make tests for 'errors' argument more explicit
* add missing test_ prefix utf_16_strings_contain_null_bytes;
* cleanup test names.
 2015-07-23 20:03:33 +02:00
Mikhail Korobov 4073498652 port utils.python 
* stringify_dict is deprecated
* is_writable is deprecated
* setattr_default is deprecated
* get_spec is untested
* re_rsearch is untested
* retry_on_eintr is untested
 2015-07-23 18:33:56 +02:00
Mikhail Korobov 41bcae5d00 TST fix to_bytes and to_unicode tests in Python 3.x 2015-07-23 17:35:23 +02:00
Mikhail Korobov 61cd27e5c7 Rename unicode_to_str and str_to_unicode method. Fixes GH-778. 2015-07-23 15:00:06 +02:00
Jakob de Maeyer c153217bfb Fix RedirectMiddleware test 2015-07-17 16:11:53 +02:00
Jakob de Maeyer c908d31660 Make RedirectMiddleware respect Spider.handle_httpstatus_list 2015-07-16 12:50:26 +02:00
nyov 4f56c6c79b Unset environment proxies for tests 
because urllib doesn't handle $no_proxy correctly
and the unittest webserver is always local.
 2015-07-11 11:30:08 +00:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Julia Medina d3d61fde87 Fix a recent mock regression by not using it in a test 2015-07-10 19:58:38 -03:00
Demelziraptor 9a15fcf89a interpreting application/x-json as TextResponse 2015-07-02 19:56:30 +02:00
Daniel Graña 07f4f12e8b Merge pull request #1313 from eliasdorneles/support-empty-passwd-for-http-proxy 
[MRG+1] Support empty password for http_proxy config
 2015-06-30 19:58:16 -03:00
Mikhail Korobov ea4b175c2d Merge pull request #1254 from scrapy/bugfix-1232 
[MRG +1] Unquote request path before passing to FTPClient, it already escape paths
 2015-06-24 18:04:45 +05:00
Elias Dorneles eb8fed4ecd support empty password for http_proxy config 2015-06-23 00:43:15 -03:00
Berker Peksag 8a48d9c6a8 Ignore ScrapyDeprecationWarning warnings properly. 
Conflicts:

	tests/test_utils_deprecate.py
 2015-06-09 18:22:03 +03:00
Mikhail Korobov d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
Mikhail Korobov cc2f3e1b46 TST a test case to show custom_settings doesn't always work 2015-06-03 04:26:20 +05:00
Julia Medina ffc7b7fd6c Add helper to update deprecated class paths 2015-06-01 17:01:33 -03:00
Marven Sanchez bb3ebf13f9 Add tests for RFC2616 policy enhancements 
Add `scrapy/downloadermiddlewares/httpcache.py` to `tests/py3-ignores.txt
 2015-06-01 18:20:12 +08:00
Mikhail Korobov aa6a72707d fixed CrawlerProcess when settings are passed as dicts 
See https://github.com/scrapy/scrapy/pull/1156
 2015-05-30 06:59:15 +05:00
Daniel Graña ebe889a663 Unquote request path before passing to FTPClient, it already escape paths 2015-05-23 20:50:30 -03:00
Mikhail Korobov cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Daniel Graña ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Mikhail Korobov 7a5b5ec4d6 TST remove unnecessary lines from py3-ignores 
scrapy/contrib is already skipped - see https://github.com/scrapy/scrapy/pull/1165
 2015-05-19 00:57:39 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Mikhail Korobov 8d9e3b7e8d TST add a test for JSON lines item exporter 2015-05-18 16:43:23 +05:00
Julia Medina 6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Julia Medina 157cec7f64 Replace logger.exception calls for Python compatibility 2015-05-15 01:26:45 -03:00
Daniel Graña 5e5e44e91e Merge pull request #1228 from nramirezuy/item_fields_defined_on_fields 
item fields defined on fields class attribute fix
 2015-05-14 23:09:24 -03:00
nramirezuy 773ea5a5ef item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
bosnj 8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Victoria Terenina (torymur) 667864ae94 testing robotstxt error 2015-04-30 16:28:57 +03:00
Julia Medina b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina 54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Mikhail Korobov fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Julia Medina cc4c31e426 Relax SpiderLoader interface check 2015-04-23 15:08:04 -03:00
Julia Medina 6d1205063c Add a filter to replace '__name__' loggers with 'scrapy' 2015-04-22 17:24:41 -03:00
Julia Medina c174d78f12 Deprecate scrapy/log.py 2015-04-22 17:24:40 -03:00
Julia Medina 6acb3848fb Stdout redirect in configure_logging 2015-04-22 17:24:40 -03:00
Julia Medina 1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
Julia Medina ac40ef611a Custom handler to count log level occurrences in a crawler 2015-04-22 17:24:40 -03:00
Julia Medina b75556ef79 Add a logging filter to mimic Twisted's log.err formating for Failures 2015-04-22 17:24:40 -03:00
Julia Medina 6f9b423215 Restructure LogFormatter to comply with std logging calls 2015-04-22 17:24:40 -03:00
Julia Medina c2d716807a Use LogCapture in testfixtures package for tests 
This allows to remove `get_testlog` helper, `flushLoggedErrors` from
twisted.trial.unittest.TestCase and Twisted log observers created for
each test on conftest.py.
 2015-04-22 17:24:40 -03:00
Julia Medina 7a958f90be Replace scrapy.log calls for their equivalents in the logging std module 
Changes:
 - Each module takes 'scrapy' logger and logs through it
 - Lazy string evaluation in all log messages
 - Added missing log messages in scrapy/core/engine.py
 - Contextual data such as crawler or spider instances, and failures
 2015-04-22 17:24:39 -03:00
Daniel Graña 571bf68d7d Merge pull request #353 from nramirezuy/item-multi_inherit 
[MRG] Item multi inheritance fix
 2015-04-22 05:37:41 -03:00
nramirezuy 7871acd665 Item multi inherit fixed 2015-04-21 18:10:52 -03:00
Mikhail Korobov c3d3a94914 fixed backwards compatibility for SPIDER_MANAGER_CLASS option 2015-04-22 00:29:40 +05:00
Pablo Hoffman 0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Daniel Graña e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman 06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Nikolaos-Digenis Karagiannis 017fb25bee loader test with processors that use item's values 2015-04-20 11:04:31 +03:00
Julia Medina ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
Mikhail Korobov ad587ea792 rename CrawlerRunner.spiders to CrawlerRunner.spider_loader 2015-04-17 01:54:26 +05:00
Mikhail Korobov 403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
mrpandav 954c8fcecb changes `test_from_response_formname_notexists_fallback_formid` unit test for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 16:49:09 +05:30
mrpandav 020a32a3d9 Adding tests for pull request #1137 - addition of new shortcut for html form election by formid attribute 2015-04-15 11:23:25 +05:30
Julia Medina 526aa07f44 Merge pull request #1156 from kmike/crawlers-accept-dicts 
allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects
 2015-04-15 02:15:32 -03:00
Daniel Graña c013baa602 Merge pull request #1161 from scrapy/telnet-disable 
disable scrapy.telnet if twisted.conch is not available
 2015-04-15 00:32:42 -03:00
Mikhail Korobov 0b76487629 PY3 fix Python 3 syntax errors in ajaxcrawl.py 
Tests for _has_ajaxcrawlable_meta passin Python 3.
 2015-04-15 04:16:26 +05:00
Mikhail Korobov 378b6efc0e PY3 disable scrapy.telnet if twisted.conch is not available 2015-04-15 04:06:36 +05:00
Mikhail Korobov abeb8e3a01 TST skip a test in Python 3 because it imports some parts of Twisted which are not ported yet 2015-04-15 03:27:11 +05:00
Mikhail Korobov 647eeaea61 TST add a test for GH-1050. 2015-04-15 03:23:30 +05:00
Mikhail Korobov 1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
Mikhail Korobov 7644813286 allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects 2015-04-14 23:26:05 +05:00
Mikhail Korobov 973c31f735 TST cleanup: use assertIn instead of assert_ 2015-04-14 20:11:46 +05:00
Pablo Hoffman 71c0aface7 Merge pull request #1148 from Curita/verify-spidermanager-interface 
Verify SPIDER_MANAGER_CLASS interface while loading it in CrawlerRunner
 2015-04-13 14:25:32 -03:00
Jakob de Maeyer cf9d848f0f Fixed deprecated settings API in cmdline test 2015-04-12 14:18:47 +02:00
Nikolaos-Digenis Karagiannis bf301b7328 encode invalid xpath with unicode_escape under PY2 
The exception quotes an xpath string
which may be unicode.
 2015-04-10 22:23:40 +03:00
Julia Medina 24a07fd892 Verify SPIDER_MANAGER_CLASS interface in CrawlerRunner 2015-04-10 13:43:33 -03:00
nyov 6d48c19acd dissolve `scrapy.contrib_exp` 2015-04-07 21:20:48 +00:00
Pablo Hoffman 4b11501eb5 Merge pull request #963 from tpeng/fix-xmliter-lxml 
[MRG+1] support namespace prefix in xmliter_lxml
 2015-04-07 18:13:01 -03:00
Daniel Graña 9706119336 Merge pull request #1128 from Curita/move-spider-settings-population-to-crawler 
[MRG+1] Move spider settings population from CrawlerRunner to Crawler.__init__
 2015-04-04 18:08:05 -03:00
Julia Medina 6f9265b44b Move spider settings population to Crawler.__init__ 2015-04-01 23:29:10 -03:00
nyov dc88be75e2 more python3 compatibility changes for urllib 2015-04-01 19:50:56 +00:00
Daniel Graña ec4251afd7 Merge pull request #1089 from drack3800/master 
[MRG+1] Add test for webclient with POST method and no body given
 2015-03-31 17:30:08 -03:00
Pablo Hoffman bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
nyov 84edc2ebc7 Add Response.urljoin() testcase 
and add evaluation of base-url for HtmlResponse.
 2015-03-27 05:59:01 +00:00
drack3800 1b6d5a011a Added webclient test for checking Content-Length header in response for POST request with no given body 2015-03-21 04:02:51 +03:00
Mateusz Golewski f92bc09bf4 Add re_first() to SelectorList and iflatten() to utils.python 2015-03-18 21:11:08 -03:00
Mateusz Golewski 2742b4d8c2 Add tests to extract_first() 2015-03-18 21:11:08 -03:00
Nicolás Alejandro Ramírez Quiros ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Pablo Hoffman 934584a355 Merge pull request #1020 from jojje/gzip_http_cache 
[MRG+1] add gzip compression to filesystem http cache backend
 2015-03-17 14:32:06 -03:00
Pablo Hoffman f924567591 Merge pull request #983 from ArturGaspar/linkextractor_css 
[MRG+1] CSS support in link extractors
 2015-03-17 01:07:47 -03:00
nramirezuy c13e23641b httpcache dont_cache meta #19 #689 2015-03-16 11:50:04 -03:00
Mikhail Korobov 645366a0fd Merge pull request #1070 from berkerpeksag/items-equal 
assertItemsEqual was renamed to assertCountEqual in Python 3.
 2015-03-08 22:54:57 +05:00
Berker Peksag c86e1bebec assertItemsEqual was renamed to assertCountEqual in Python 3. 2015-03-08 15:18:49 +02:00
Berker Peksag 31e5f164d4 Import unittest.mock if available. 
mock is in the stdlib since Python 3.3.
 2015-03-08 15:03:04 +02:00
Sudhanshu Shekhar 839ffba971 Added the first version of SelectJmes 
Utilizes jmespath. Also, added tests and documentation for the same.
 2015-02-24 22:59:01 +05:30
Jonas Tingeborn bd5d99a2d2 add gzip compression to filesystem http cache backend 2015-01-21 20:18:11 +01:00
Julia Medina d68615a5af Test the parse command locally instead of against an external url 2015-01-19 10:28:25 -03:00
tpeng 82d138e87e support namespace prefix in xmliter_lxml 2014-12-15 16:17:06 +01:00
Artur Gaspar 403fc686b8 tests for CSS support in link extractors 2014-12-11 18:20:30 -02:00
immerrr 82b187f283 S3DownloadHandler: fix auth for requests with quoted paths/query params 2014-12-11 18:14:36 +03:00
Lev Berman fdb6bb07c0 #977 - test dropping requests 2014-11-28 10:53:33 +03:00
Lev Berman e04b0aff74 An attempt to resolve #977, add signal to be sent when request is dropped by the scheduler 2014-11-27 15:10:15 +03:00
Pablo Hoffman c31fb87335 Merge pull request #954 from kalessin/int-download-timeout 
Force to read DOWNLOAD_TIMEOUT as int (for example to pass using environment variable)
 2014-11-26 17:14:18 -02:00
Pablo Hoffman dedea72774 Merge pull request #946 from tpeng/limit-response-size 
avoid download large response
 2014-11-25 17:57:55 -02:00
tpeng cd19382754 attemp to fix travis fails 2014-11-25 14:20:25 +01:00
Daniel Graña 8d8e1b2c0c mitmproxy 0.10.1 needs netlib 0.10.1 too 2014-11-21 12:15:02 -02:00
Daniel Graña 314db3db8b pin mitmproxy 0.10.1 as >0.11 does not work with tests 2014-11-21 10:54:43 -02:00
Martin Olveyra 7910fa0172 Force to read DOWNLOAD_TIMEOUT as int (for example to pass using 
environment variable)
 2014-11-21 01:09:32 -02:00
tpeng a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Pablo Hoffman efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov 49645d4bf9 TST small cleanup of a cookie test 2014-09-21 05:31:34 +06:00
Mikhail Korobov c543fe6e4c Merge pull request #878 from andrewshir/master 
Fix bug for ".local" host name
 2014-09-21 05:24:54 +06:00
andrewshir e583c030db Test for local domains (without dots) added 2014-09-14 14:24:16 +06:00
Mikael Åhlén 22da1783bd added a test-case for wrong quotechar 2014-09-13 03:47:40 +02:00
Mikael Åhlén 47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Daniel Graña 5bcabfe9c9 SPIDER_MODULES can be set as a csv string 2014-09-10 23:25:57 -03:00
Daniel Graña ec93c0fdcc Add the tests changes for previous commit 2014-09-10 12:05:18 -03:00
Daniel Graña 99971dc8a8 Do not pop the crawler from the managed list 2014-09-09 20:59:07 +00:00
Julia Medina c2592b39fd Test verifying that CrawlerRunner populates spider class settings 2014-09-01 21:56:57 -03:00
Julia Medina 9ef3972cfb Per-spider settings tests 2014-09-01 21:56:57 -03:00
Daniel Graña ccde3317d7 Merge pull request #816 from Curita/api-cleanup 
GSoC API cleanup
 2014-09-01 21:55:36 -03:00
yakxxx e4689556f0 SgmlLinkExtractor - fix for parsing <area> tag with Unicode present 2014-08-28 18:55:58 +02:00
nramirezuy 2a540206a7 fix xmliter namespace on selected node 2014-08-19 15:44:49 -03:00
Daniel Graña d684ecad7b Merge pull request #846 from rocioar/master 
fix dont_merge_cookies bad behaviour when set to false on meta
 2014-08-18 13:54:11 -03:00
Daniel Graña a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Rocio Aramberri 51b0bd281d fix dont settings on meta behaviour, add docs and tests 2014-08-15 13:47:42 -07:00
Julia Medina 70f2010db1 Change error type when updating frozen settings 2014-08-14 11:59:25 -03:00
Julia Medina 419026615f Deprecate Crawler.spiders attribute 2014-08-14 09:19:41 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 900a487682 Support multiple simultaneous LogObservers listening different crawlers 2014-08-12 14:02:56 -03:00
Julia Medina 870438e5f4 Update tests utils, fixing get_crawler and removing docrawl 2014-08-12 14:02:56 -03:00
Julia Medina d7038b2a13 SpiderManager interface cleanup 2014-08-12 14:02:55 -03:00
Julia Medina 3ae971468f Add Settings.copy, freeze and frozencopy method 2014-08-12 14:02:55 -03:00
Julia Medina a995727117 Connect spider_closed signal after a crawler is bound to a Spider 2014-08-12 14:02:55 -03:00
Julia Medina eb0253e530 Update from_crawler method as well as set_crawler on CrawlSpider 2014-08-11 11:24:01 -03:00
Julia Medina 84fa004793 Add from_crawler class method to base Spider 2014-08-11 11:23:57 -03:00
Pablo Hoffman 69a665eb32 Merge pull request #832 from ivannotes/master 
Bugfix for leaking Proxy-Authorization header to targeted host
 2014-08-04 15:20:51 -03:00
Daniel Graña 511a26978f Merge pull request #817 from nramirezuy/fix-startproject-project-name 
[MRG] Fix startproject project name
 2014-08-04 11:55:35 -03:00
Mikhail Korobov e62bbf0766 PY3 top-level shortcuts work 2014-08-02 00:24:32 +06:00
Daniel Graña ee21eaa6f3 Merge pull request #835 from scrapy/modernize-asserts 
[MRG] modernize some of the asserts
 2014-08-01 15:19:36 -03:00
Mikhail Korobov 1f59a69a97 PY3 port scrapy.link 2014-08-02 00:16:01 +06:00
Mikhail Korobov 628a8d8a15 TST modernize some of the asserts 2014-08-01 23:44:23 +06:00
Nuno Maximiano 08224c92f4 add project name validation 2014-08-01 14:36:30 -03:00
Mikhail Korobov e2c1226838 PY3 port scrapy.contrib.feedexport 2014-08-01 23:28:28 +06:00
Mikhail Korobov 51bd9f7d64 TST enable doctests for Python 3 2014-08-01 23:12:52 +06:00
Mikhail Korobov 444052a2f9 PY3 port scrapy.utils.misc.arg_to_iter; fix scrapy.utils.misc.md5sum doctest 2014-08-01 22:56:03 +06:00
Mikhail Korobov 83432184c8 PY3 enable more tests 2014-08-01 16:53:18 +06:00
Mikhail Korobov 669b62c741 PY3 port scrapy.utils.spider 2014-08-01 16:43:39 +06:00
Mikhail Korobov 560c32d330 PY3 fix test_utils_sitemap. 
Sitemap class is only initialized with bytes by Scrapy;
broken tests was only a testing issue.
 2014-08-01 16:17:40 +06:00
Mikhail Korobov 330743411e PY3 enable more tests 2014-08-01 16:10:34 +06:00
Mikhail Korobov 7183a87c35 Merge pull request #828 from dangra/string-headers 
normalize header values to bytes (and PY3)
 2014-08-01 15:51:57 +06:00
Daniel Graña ccae49813a PY3: port scrapy/utils/httpobj.py 2014-08-01 02:38:29 -03:00
Daniel Graña c564334d00 PY3: port scrapy/utils/gz.py 2014-08-01 02:25:33 -03:00
Daniel Graña 910d0d36c1 PY3: port scrapy/utils/conf.py 2014-08-01 02:20:34 -03:00
Daniel Graña 521303e27f fix ftp test cases 2014-08-01 02:10:57 -03:00
Daniel Graña f42b44b50e normalize header values to bytes 2014-08-01 00:02:50 -03:00
ivannotes 484a0159d0 Add test case for tunneling proxy 2014-08-01 09:43:17 +08:00
Daniel Graña 7cd271eb8e remove more references to runtests.sh 2014-07-31 17:30:45 -03:00
Daniel Graña 70ee66ba0c Collect the list of packages not working on python3 2014-07-31 14:06:34 -03:00
Daniel Graña caab9f29f2 move tests requirements under tests dir 2014-07-30 19:45:08 -03:00
Daniel Graña 242c08553f move tests under root dir 2014-07-30 19:45:08 -03:00