d706310d8b
Merge pull request #1151 from marven/cache-control
...
[MRG+1] RFC2616 policy enhancements + tests
2015-07-11 08:06:20 -03:00
9a15fcf89a
interpreting application/x-json as TextResponse
2015-07-02 19:56:30 +02:00
3fc4e0b319
Merge pull request #1282 from otherchirps/memusage-check-interval
...
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
2015-07-02 13:50:55 -03:00
07f4f12e8b
Merge pull request #1313 from eliasdorneles/support-empty-passwd-for-http-proxy
...
[MRG+1] Support empty password for http_proxy config
2015-06-30 19:58:16 -03:00
d850238c22
add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs
2015-06-27 04:59:42 +05:00
1fef9f1132
AutoThrottle: respect download_delay=0 spider attribute
2015-06-26 19:09:26 +05:00
17cb51315b
drop support for AUTOTHROTTLE_MIN_DOWNLOAD_DELAY
...
it was deprecated for years
2015-06-26 19:07:54 +05:00
584252e8f2
move AutoThrottle default options to default_settings.py
2015-06-26 18:58:29 +05:00
e6272e5209
make AutoThrottle._adjust_delay easier to understand
2015-06-26 18:44:23 +05:00
5bcda9b7d1
Merge pull request #1290 from scrapy/crawler-runner-cleanup
...
extract CrawlerRunner._crawl method which always expects Crawler
2015-06-26 00:21:23 -03:00
4cea1c1231
Fix SelectJmes documentation
2015-06-25 15:11:56 -03:00
8a140b6ba1
Merge pull request #1315 from scrapy/downloader-slots-cleanup
...
Small downloader slots cleanup
2015-06-24 16:25:13 -03:00
ea4b175c2d
Merge pull request #1254 from scrapy/bugfix-1232
...
[MRG +1] Unquote request path before passing to FTPClient, it already escape paths
2015-06-24 18:04:45 +05:00
cb0445e8b8
Slot.__str__ and __repr__ methods; remove unused import
2015-06-24 03:22:23 +05:00
8da9836db3
don't access settings from downloader.Slot
2015-06-24 03:20:46 +05:00
eb8fed4ecd
support empty password for http_proxy config
2015-06-23 00:43:15 -03:00
fa1c25c840
Merge pull request #1286 from scrapy/configure_logging
...
configure_logging: change the meaning of settings=None
2015-06-12 13:22:42 -03:00
877c25793a
Do leveldb compactation for the httpcache on closing.
2015-06-11 13:27:49 -04:00
5bd0395be4
Merge pull request #1291 from scrapy/signalmanager-docstrings
...
DOC SignalManager docstrings. See GH-713.
2015-06-10 16:28:35 -03:00
6446652635
Merge pull request #1294 from berkerpeksag/ignore-warnings
...
Ignore ScrapyDeprecationWarning warnings properly.
2015-06-10 16:07:21 -03:00
6c9daf3a95
DOC remove unnecessary links; fix references in send_catch_log_deferred docstring
2015-06-10 01:44:19 +05:00
a611f8dd2d
DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring
2015-06-09 22:57:18 +05:00
8a48d9c6a8
Ignore ScrapyDeprecationWarning warnings properly.
...
Conflicts:
tests/test_utils_deprecate.py
2015-06-09 18:22:03 +03:00
90aa5c075d
Merge pull request #1284 from scrapy/crawler-cleanup
...
CrawlerProcess cleanup
2015-06-09 11:21:24 -03:00
9ca87c5b6b
Merge pull request #1293 from berkerpeksag/fix-typos
...
Fix typos in scrapy/commands/setting.py
2015-06-09 11:08:10 -03:00
66e5591eb9
Fix typos in scrapy/commands/setting.py
...
* intepreted -> interpreted
* "a list" instead of "an float" in --getlist help
* "an float"-> "a float"
Also, backslashes were redundant. So I removed them.
2015-06-09 04:29:24 +03:00
1740fcf1a6
DOC SignalManager docstrings. See GH-713.
...
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
2015-06-08 21:05:58 +05:00
9a787893e3
(backwards-incompatible) allow to pass settings=None to configure_logging
...
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False
2015-06-08 19:54:18 +05:00
3cbf8a0b2b
extract CrawlerRunner._crawl method which always expects Crawler
...
It provides an extension point where crawler instance is available;
it should make it easier to write alternative CrawlerRunner.crawl
implementations.
See also: https://github.com/scrapy/scrapy/pull/1256
2015-06-08 18:35:44 +05:00
61dec83f70
Moved default value of MEMUSAGE_CHECK_INTERVAL_SECONDS to default_settings
2015-06-06 11:19:29 +10:00
d047665c02
make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess
2015-06-06 03:23:13 +05:00
64399d18d8
Stop reactor on Ctrl-C regardless of 'stop_after_crawl'. Fixes GH-1279.
2015-06-06 02:53:36 +05:00
33d145e2f5
CrawlerProcess cleanup
...
* remove unneeded lambda;
* extract _get_dns_resolver method and format code to pep8.
2015-06-06 02:49:39 +05:00
eae25a04d9
Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
...
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
2015-06-06 00:39:14 +10:00
367ea81e71
Remove deprecated %z formatting from the default LOG_DATEFORMAT
2015-06-04 04:11:23 +08:00
d42c420a6d
fixed spider custom_settings
...
https://github.com/scrapy/scrapy/pull/1128 moved spidercls.update_settings
call to a later stage; this commit moves it back.
2015-06-03 04:29:10 +05:00
ffc7b7fd6c
Add helper to update deprecated class paths
2015-06-01 17:01:33 -03:00
1991550442
Allow client to bound max-age for revalidation.
...
Unlike specifying "Cache-Control: no-cache", if the request specifies
"max-age=0", then the cached validators will be used if possible to
avoid re-fetching unchanged pages.
That said, it's still useful to be able to specify "no-cache" on the
request, in cases where the origin server may have changed page contents
without changing validators.
2015-06-01 18:06:36 +08:00
c3b2cabf6c
Allow setting RFC2616Policy to cache unconditionally.
...
A spider may wish to have all responses available in the cache, for
future use with "Cache-Control: max-stale", for instance. The
DummyPolicy caches all responses but never revalidates them, and
sometimes a more nuanced policy is desirable.
This setting still respects "Cache-Control: no-store" directives in
responses. If you don't want that, filter "no-store" out of the
Cache-Control headers in responses you feed to the cache middleware.
2015-06-01 18:06:35 +08:00
e23a381337
Let spiders ignore bogus Cache-Control headers.
...
Sites often set "no-store", "no-cache", "must-revalidate", etc., but get
upset at the traffic a spider can generate if it respects those
directives.
Allow the spider's author to selectively ignore Cache-Control directives
that are known to be unimportant for the sites being crawled.
We assume that the spider will not issue Cache-Control directives in
requests unless it actually needs them, so directives in requests are
not filtered.
2015-06-01 18:06:35 +08:00
dd3a46295c
Support "Cache-Control: max-stale" in requests.
...
This allows spiders to be configured with the full RFC2616 cache policy,
but avoid revalidation on a request-by-request basis, while remaining
conformant with the HTTP spec.
2015-06-01 18:06:35 +08:00
4446baae33
Use cached responses if revalidation errors out.
2015-06-01 18:06:35 +08:00
aa6a72707d
fixed CrawlerProcess when settings are passed as dicts
...
See https://github.com/scrapy/scrapy/pull/1156
2015-05-30 06:59:15 +05:00
62a6eff218
Merge pull request #1259 from chekunkov/log-counter-handler-is-never-removed
...
[MRG +1] LogCounterHandler is never removed from root handlers list, fix that
2015-05-27 11:42:19 -03:00
b2765aabd8
LogCounterHandler is never removed from root handlers list, fix that
...
lambda is garbage collected and because receiver is added as weak reference by default - when signals.engine_stopped is fired logging.root.removeHandler is not executed. Fixed that by assigning lambda to a private argument and not by using connect(..., weak=False) because I belive this lambda function should be collected with crawler object
2015-05-27 13:52:47 +07:00
b0ea3e38d1
remove CrawlerProcess.stopping as it isn't used any more
2015-05-26 17:37:16 +07:00
ebe889a663
Unquote request path before passing to FTPClient, it already escape paths
2015-05-23 20:50:30 -03:00
3545468389
Merge branch 'deferdelay'
2015-05-23 18:09:20 -03:00
d439c26d76
update docstring and release notes
2015-05-22 20:00:58 -03:00
27ce3225bd
Makes scrapy more async by letting to reactor spend another couple of cycles to accomplish its needs.
2015-05-22 17:05:19 -03:00
Julia Medina
Demelziraptor
Daniel Graña
Mikhail Korobov
Capi Etheriel
Pablo Hoffman
Elias Dorneles
Rolando Espinoza
Berker Peksag
Chris Nilsson
Jamey Sharp
Alexander Chekunkov
Alexey Vishnevsky