877c25793a
Do leveldb compactation for the httpcache on closing.
2015-06-11 13:27:49 -04:00
5bd0395be4
Merge pull request #1291 from scrapy/signalmanager-docstrings
...
DOC SignalManager docstrings. See GH-713.
2015-06-10 16:28:35 -03:00
6446652635
Merge pull request #1294 from berkerpeksag/ignore-warnings
...
Ignore ScrapyDeprecationWarning warnings properly.
2015-06-10 16:07:21 -03:00
6c9daf3a95
DOC remove unnecessary links; fix references in send_catch_log_deferred docstring
2015-06-10 01:44:19 +05:00
a611f8dd2d
DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring
2015-06-09 22:57:18 +05:00
8a48d9c6a8
Ignore ScrapyDeprecationWarning warnings properly.
...
Conflicts:
tests/test_utils_deprecate.py
2015-06-09 18:22:03 +03:00
90aa5c075d
Merge pull request #1284 from scrapy/crawler-cleanup
...
CrawlerProcess cleanup
2015-06-09 11:21:24 -03:00
f91461c256
Merge pull request #1292 from scrapy/spidererror-docs
...
spider_error doesn't support deferreds
2015-06-09 11:19:10 -03:00
9ca87c5b6b
Merge pull request #1293 from berkerpeksag/fix-typos
...
Fix typos in scrapy/commands/setting.py
2015-06-09 11:08:10 -03:00
66e5591eb9
Fix typos in scrapy/commands/setting.py
...
* intepreted -> interpreted
* "a list" instead of "an float" in --getlist help
* "an float"-> "a float"
Also, backslashes were redundant. So I removed them.
2015-06-09 04:29:24 +03:00
790c67b643
DOC spider_error doesn't support deferreds
2015-06-09 02:20:10 +05:00
1740fcf1a6
DOC SignalManager docstrings. See GH-713.
...
This change is not 100% backwards compatible because of *args changes.
Their usage was not documented, so we're not breaking public interface.
2015-06-08 21:05:58 +05:00
9a787893e3
(backwards-incompatible) allow to pass settings=None to configure_logging
...
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False
2015-06-08 19:54:18 +05:00
3cbf8a0b2b
extract CrawlerRunner._crawl method which always expects Crawler
...
It provides an extension point where crawler instance is available;
it should make it easier to write alternative CrawlerRunner.crawl
implementations.
See also: https://github.com/scrapy/scrapy/pull/1256
2015-06-08 18:35:44 +05:00
87293965db
Merge pull request #1285 from scrapy/optional-settings-arguments
...
make it easier to use default settings
2015-06-07 20:29:24 -03:00
61dec83f70
Moved default value of MEMUSAGE_CHECK_INTERVAL_SECONDS to default_settings
2015-06-06 11:19:29 +10:00
0c532baf4c
Removed typo, and clarified time unit of setting
2015-06-06 11:18:13 +10:00
d047665c02
make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess
2015-06-06 03:23:13 +05:00
64399d18d8
Stop reactor on Ctrl-C regardless of 'stop_after_crawl'. Fixes GH-1279.
2015-06-06 02:53:36 +05:00
33d145e2f5
CrawlerProcess cleanup
...
* remove unneeded lambda;
* extract _get_dns_resolver method and format code to pep8.
2015-06-06 02:49:39 +05:00
24d8a85269
Update release notes for 1.0.0rc2
...
(cherry picked from commit 6e61d54168
2015-06-05 17:11:40 -03:00
eae25a04d9
Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
...
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
2015-06-06 00:39:14 +10:00
d9bcd48606
Merge pull request #1278 from Curita/remove-tz-aware-logformat
...
Remove deprecated %z formatting from the default LOG_DATEFORMAT
2015-06-04 13:39:01 -03:00
367ea81e71
Remove deprecated %z formatting from the default LOG_DATEFORMAT
2015-06-04 04:11:23 +08:00
f312ffcb54
Merge pull request #1276 from scrapy/fix-spider-settings
...
Fix Spider.custom_settings
2015-06-03 22:14:04 +05:00
d42c420a6d
fixed spider custom_settings
...
https://github.com/scrapy/scrapy/pull/1128 moved spidercls.update_settings
call to a later stage; this commit moves it back.
2015-06-03 04:29:10 +05:00
cc2f3e1b46
TST a test case to show custom_settings doesn't always work
2015-06-03 04:26:20 +05:00
d52cf8bb03
Merge pull request #1267 from Curita/fix-1265
...
Fix #1265
2015-06-01 20:31:46 -03:00
ffc7b7fd6c
Add helper to update deprecated class paths
2015-06-01 17:01:33 -03:00
bd2fe996aa
Spelling correction
...
incorrect use of "too" instead of "to"
2015-06-01 20:47:22 +05:00
9d1cf230ed
Merge pull request #1268 from scrapy/crawlerprocess-dict-settings
...
fixed CrawlerProcess when settings are passed as dicts
2015-06-01 12:35:54 -03:00
8771d1f79b
Update HTTPCache middleware docs
2015-06-01 18:20:59 +08:00
bb3ebf13f9
Add tests for RFC2616 policy enhancements
...
Add `scrapy/downloadermiddlewares/httpcache.py` to `tests/py3-ignores.txt
2015-06-01 18:20:12 +08:00
1991550442
Allow client to bound max-age for revalidation.
...
Unlike specifying "Cache-Control: no-cache", if the request specifies
"max-age=0", then the cached validators will be used if possible to
avoid re-fetching unchanged pages.
That said, it's still useful to be able to specify "no-cache" on the
request, in cases where the origin server may have changed page contents
without changing validators.
2015-06-01 18:06:36 +08:00
c3b2cabf6c
Allow setting RFC2616Policy to cache unconditionally.
...
A spider may wish to have all responses available in the cache, for
future use with "Cache-Control: max-stale", for instance. The
DummyPolicy caches all responses but never revalidates them, and
sometimes a more nuanced policy is desirable.
This setting still respects "Cache-Control: no-store" directives in
responses. If you don't want that, filter "no-store" out of the
Cache-Control headers in responses you feed to the cache middleware.
2015-06-01 18:06:35 +08:00
e23a381337
Let spiders ignore bogus Cache-Control headers.
...
Sites often set "no-store", "no-cache", "must-revalidate", etc., but get
upset at the traffic a spider can generate if it respects those
directives.
Allow the spider's author to selectively ignore Cache-Control directives
that are known to be unimportant for the sites being crawled.
We assume that the spider will not issue Cache-Control directives in
requests unless it actually needs them, so directives in requests are
not filtered.
2015-06-01 18:06:35 +08:00
dd3a46295c
Support "Cache-Control: max-stale" in requests.
...
This allows spiders to be configured with the full RFC2616 cache policy,
but avoid revalidation on a request-by-request basis, while remaining
conformant with the HTTP spec.
2015-06-01 18:06:35 +08:00
4446baae33
Use cached responses if revalidation errors out.
2015-06-01 18:06:35 +08:00
aa6a72707d
fixed CrawlerProcess when settings are passed as dicts
...
See https://github.com/scrapy/scrapy/pull/1156
2015-05-30 06:59:15 +05:00
342cb622f1
DOC fix non-working link (by removing it).
...
See https://github.com/scrapy/scrapy/pull/1260
2015-05-27 23:04:58 +05:00
343d20d791
Update 1.0 release notes
2015-05-27 11:53:54 -03:00
62a6eff218
Merge pull request #1259 from chekunkov/log-counter-handler-is-never-removed
...
[MRG +1] LogCounterHandler is never removed from root handlers list, fix that
2015-05-27 11:42:19 -03:00
26f50d3f43
Extend regex for tags that deploy to PyPI to support new release cycle
2015-05-27 09:17:18 -03:00
b2765aabd8
LogCounterHandler is never removed from root handlers list, fix that
...
lambda is garbage collected and because receiver is added as weak reference by default - when signals.engine_stopped is fired logging.root.removeHandler is not executed. Fixed that by assigning lambda to a private argument and not by using connect(..., weak=False) because I belive this lambda function should be collected with crawler object
2015-05-27 13:52:47 +07:00
5ee08865d6
Merge pull request #1258 from chekunkov/crawler-process-stopping-is-no-more
...
[MRG+1] Remove CrawlerProcess.stopping as it isn't used any more
2015-05-26 15:32:24 -03:00
b0ea3e38d1
remove CrawlerProcess.stopping as it isn't used any more
2015-05-26 17:37:16 +07:00
545c4224f9
update old crawlera link
2015-05-25 16:01:54 -03:00
ebe889a663
Unquote request path before passing to FTPClient, it already escape paths
2015-05-23 20:50:30 -03:00
3545468389
Merge branch 'deferdelay'
2015-05-23 18:09:20 -03:00
d439c26d76
update docstring and release notes
2015-05-22 20:00:58 -03:00
Rolando Espinoza
Daniel Graña
Mikhail Korobov
Berker Peksag
Julia Medina
Chris Nilsson
Ally Weir
Marven Sanchez
Jamey Sharp
Alexander Chekunkov
Pablo Hoffman