RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,904 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

3375 Commits

Author SHA1 Message Date
Daniel Graña 786f62664b Merge pull request #1137 from DharmeshPandav/patch-1 
Update form.py to improve existing capability
 2015-07-31 11:00:57 -03:00
nyov e044bfa60f PY3 fix test downloadermiddleware decompression 2015-07-29 13:51:01 +00:00
nyov 683ef2a8d9 replace rfc822 with email.utils 2015-07-29 13:51:01 +00:00
nyov 1a1bc2c6b5 PY3 port scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware 2015-07-29 04:21:57 +00:00
Daniel Graña 776064aee1 PY3 port signals 2015-07-25 18:04:55 +02:00
Mikhail Korobov 82304c4fae PY3: fixed scrapy.utils.defer 2015-07-25 17:48:34 +02:00
Mikhail Korobov 7874bb9f13 PY3 port utils.request 2015-07-25 17:14:56 +02:00
Mikhail Korobov e853d9e910 partial port of Request and Response 2015-07-25 13:08:44 +02:00
Mikhail Korobov f576b3ffee [tmp] improve python 3 support for scrapy.utils.url 2015-07-25 00:40:45 +02:00
Mikhail Korobov 4073498652 port utils.python 
* stringify_dict is deprecated
* is_writable is deprecated
* setattr_default is deprecated
* get_spec is untested
* re_rsearch is untested
* retry_on_eintr is untested
 2015-07-23 18:33:56 +02:00
Mikhail Korobov 887936ebf6 PY3 port flatten and iflatten 2015-07-23 17:26:20 +02:00
Mikhail Korobov 61cd27e5c7 Rename unicode_to_str and str_to_unicode method. Fixes GH-778. 2015-07-23 15:00:06 +02:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Demelziraptor 9a15fcf89a interpreting application/x-json as TextResponse 2015-07-02 19:56:30 +02:00
Daniel Graña 3fc4e0b319 Merge pull request #1282 from otherchirps/memusage-check-interval 
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
 2015-07-02 13:50:55 -03:00
Daniel Graña 07f4f12e8b Merge pull request #1313 from eliasdorneles/support-empty-passwd-for-http-proxy 
[MRG+1] Support empty password for http_proxy config
 2015-06-30 19:58:16 -03:00
Mikhail Korobov d850238c22 add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs 2015-06-27 04:59:42 +05:00
Mikhail Korobov 1fef9f1132 AutoThrottle: respect download_delay=0 spider attribute 2015-06-26 19:09:26 +05:00
Mikhail Korobov 17cb51315b drop support for AUTOTHROTTLE_MIN_DOWNLOAD_DELAY 
it was deprecated for years
 2015-06-26 19:07:54 +05:00
Mikhail Korobov 584252e8f2 move AutoThrottle default options to default_settings.py 2015-06-26 18:58:29 +05:00
Mikhail Korobov e6272e5209 make AutoThrottle._adjust_delay easier to understand 2015-06-26 18:44:23 +05:00
Daniel Graña 5bcda9b7d1 Merge pull request #1290 from scrapy/crawler-runner-cleanup 
extract CrawlerRunner._crawl method which always expects Crawler
 2015-06-26 00:21:23 -03:00
Capi Etheriel 4cea1c1231 Fix SelectJmes documentation 2015-06-25 15:11:56 -03:00
Pablo Hoffman 8a140b6ba1 Merge pull request #1315 from scrapy/downloader-slots-cleanup 
Small downloader slots cleanup
 2015-06-24 16:25:13 -03:00
Mikhail Korobov ea4b175c2d Merge pull request #1254 from scrapy/bugfix-1232 
[MRG +1] Unquote request path before passing to FTPClient, it already escape paths
 2015-06-24 18:04:45 +05:00
Mikhail Korobov cb0445e8b8 Slot.__str__ and __repr__ methods; remove unused import 2015-06-24 03:22:23 +05:00
Mikhail Korobov 8da9836db3 don't access settings from downloader.Slot 2015-06-24 03:20:46 +05:00
Elias Dorneles eb8fed4ecd support empty password for http_proxy config 2015-06-23 00:43:15 -03:00
Julia Medina fa1c25c840 Merge pull request #1286 from scrapy/configure_logging 
configure_logging: change the meaning of settings=None
 2015-06-12 13:22:42 -03:00
Rolando Espinoza 877c25793a Do leveldb compactation for the httpcache on closing. 2015-06-11 13:27:49 -04:00
Daniel Graña 5bd0395be4 Merge pull request #1291 from scrapy/signalmanager-docstrings 
DOC SignalManager docstrings. See GH-713.
 2015-06-10 16:28:35 -03:00
Daniel Graña 6446652635 Merge pull request #1294 from berkerpeksag/ignore-warnings 
Ignore ScrapyDeprecationWarning warnings properly.
 2015-06-10 16:07:21 -03:00
Mikhail Korobov 6c9daf3a95 DOC remove unnecessary links; fix references in send_catch_log_deferred docstring 2015-06-10 01:44:19 +05:00
Mikhail Korobov a611f8dd2d DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring 2015-06-09 22:57:18 +05:00
Berker Peksag 8a48d9c6a8 Ignore ScrapyDeprecationWarning warnings properly. 
Conflicts:

	tests/test_utils_deprecate.py
 2015-06-09 18:22:03 +03:00
Daniel Graña 90aa5c075d Merge pull request #1284 from scrapy/crawler-cleanup 
CrawlerProcess cleanup
 2015-06-09 11:21:24 -03:00
Julia Medina 9ca87c5b6b Merge pull request #1293 from berkerpeksag/fix-typos 
Fix typos in scrapy/commands/setting.py
 2015-06-09 11:08:10 -03:00
Berker Peksag 66e5591eb9 Fix typos in scrapy/commands/setting.py 
* intepreted -> interpreted
* "a list" instead of "an float" in --getlist help
* "an float"-> "a float"

Also, backslashes were redundant. So I removed them.
 2015-06-09 04:29:24 +03:00
Mikhail Korobov 1740fcf1a6 DOC SignalManager docstrings. See GH-713. 
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
 2015-06-08 21:05:58 +05:00
Mikhail Korobov 9a787893e3 (backwards-incompatible) allow to pass settings=None to configure_logging 
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False
 2015-06-08 19:54:18 +05:00
Mikhail Korobov 3cbf8a0b2b extract CrawlerRunner._crawl method which always expects Crawler 
It provides an extension point where crawler instance is available;
it should make it easier to write alternative CrawlerRunner.crawl
implementations.

See also: https://github.com/scrapy/scrapy/pull/1256
 2015-06-08 18:35:44 +05:00
Chris Nilsson 61dec83f70 Moved default value of MEMUSAGE_CHECK_INTERVAL_SECONDS to default_settings 2015-06-06 11:19:29 +10:00
Mikhail Korobov d047665c02 make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
Mikhail Korobov 64399d18d8 Stop reactor on Ctrl-C regardless of 'stop_after_crawl'. Fixes GH-1279. 2015-06-06 02:53:36 +05:00
Mikhail Korobov 33d145e2f5 CrawlerProcess cleanup 
* remove unneeded lambda;
* extract _get_dns_resolver method and format code to pep8.
 2015-06-06 02:49:39 +05:00
Chris Nilsson eae25a04d9 Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. 
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
 2015-06-06 00:39:14 +10:00
Julia Medina 367ea81e71 Remove deprecated %z formatting from the default LOG_DATEFORMAT 2015-06-04 04:11:23 +08:00
Mikhail Korobov d42c420a6d fixed spider custom_settings 
https://github.com/scrapy/scrapy/pull/1128 moved spidercls.update_settings
call to a later stage; this commit moves it back.
 2015-06-03 04:29:10 +05:00
Julia Medina ffc7b7fd6c Add helper to update deprecated class paths 2015-06-01 17:01:33 -03:00
Jamey Sharp 1991550442 Allow client to bound max-age for revalidation. 
Unlike specifying "Cache-Control: no-cache", if the request specifies
"max-age=0", then the cached validators will be used if possible to
avoid re-fetching unchanged pages.

That said, it's still useful to be able to specify "no-cache" on the
request, in cases where the origin server may have changed page contents
without changing validators.
 2015-06-01 18:06:36 +08:00