RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,783 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

4783 Commits

Author SHA1 Message Date
Mikhail Korobov f001641543 (backwards-incompatible) allow to pass settings=None to configure_logging 
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False

(cherry picked from commit 9a787893e3)
 2015-06-12 16:09:39 -03:00
Bryan Crowe 0e8f51beb1 Fix a couple typos 
(cherry picked from commit 6a4c475e87)
 2015-06-12 16:08:25 -03:00
Mikhail Korobov c34ec6694c Stop reactor on Ctrl-C regardless of 'stop_after_crawl'. Fixes GH-1279. 
(cherry picked from commit 64399d18d8)
 2015-06-12 16:07:24 -03:00
Mikhail Korobov 3b0391b8f2 CrawlerProcess cleanup 
* remove unneeded lambda;
* extract _get_dns_resolver method and format code to pep8.

(cherry picked from commit 33d145e2f5)
 2015-06-12 16:07:17 -03:00
Mikhail Korobov 2677f70fc3 DOC spider_error doesn't support deferreds 
(cherry picked from commit 790c67b643)
 2015-06-12 16:06:46 -03:00
Berker Peksag 423826c1be Fix typos in scrapy/commands/setting.py 
* intepreted -> interpreted
* "a list" instead of "an float" in --getlist help
* "an float"-> "a float"

Also, backslashes were redundant. So I removed them.

(cherry picked from commit 66e5591eb9)
 2015-06-12 16:06:15 -03:00
Mikhail Korobov 7a30e6acbe make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 
(cherry picked from commit d047665c02)
 2015-06-12 16:05:36 -03:00
Rolando Espinoza b285cebcbf Do leveldb compactation for the httpcache on closing. 2015-06-12 11:18:37 -03:00
Julia Medina a2dbde98d5 Bump version: 1.0.0rc1 → 1.0.0rc2 2015-06-05 17:12:56 -03:00
Julia Medina 6e61d54168 Update release notes for 1.0.0rc2 2015-06-05 17:10:18 -03:00
Julia Medina 5cd0296a16 Remove deprecated %z formatting from the default LOG_DATEFORMAT 2015-06-04 13:42:01 -03:00
Mikhail Korobov f6ae78f4b1 fixed spider custom_settings 
https://github.com/scrapy/scrapy/pull/1128 moved spidercls.update_settings
call to a later stage; this commit moves it back.
 2015-06-03 22:19:05 +05:00
Mikhail Korobov bd271d8b99 TST a test case to show custom_settings doesn't always work 2015-06-03 22:19:05 +05:00
Julia Medina 2d6450a4ad Add helper to update deprecated class paths 2015-06-01 20:39:29 -03:00
Pablo Hoffman 525b349932 update old crawlera link 2015-06-01 20:38:17 -03:00
Ally Weir 16218d6534 Spelling correction 
incorrect use of "too" instead of "to"
 2015-06-01 22:22:53 +05:00
Mikhail Korobov 19ac046343 fixed CrawlerProcess when settings are passed as dicts 
See https://github.com/scrapy/scrapy/pull/1156

(cherry picked from commit aa6a72707d)
 2015-06-01 12:50:41 -03:00
Mikhail Korobov 0b6c0eaa35 DOC fix non-working link (by removing it). 
See https://github.com/scrapy/scrapy/pull/1260
 2015-05-27 23:10:58 +05:00
Julia Medina e76359f2ae Update 1.0 release notes 
(cherry picked from commit 343d20d791)
 2015-05-27 12:02:56 -03:00
Alexander Chekunkov 6f6a251a9b LogCounterHandler is never removed from root handlers list, fix that 
lambda is garbage collected and because receiver is added as weak reference by default - when signals.engine_stopped is fired logging.root.removeHandler is not executed. Fixed that by assigning lambda to a private argument and not by using connect(..., weak=False) because I belive this lambda function should be collected with crawler object

(cherry picked from commit b2765aabd8)
 2015-05-27 12:02:45 -03:00
Alexander Chekunkov 6c264f0e22 remove CrawlerProcess.stopping as it isn't used any more 
(cherry picked from commit b0ea3e38d1)
 2015-05-27 12:02:34 -03:00
Pablo Hoffman 8d2531fb05 Merge pull request #1253 from scrapy/deferdelay 
Increase delay of defer calls
 2015-05-22 20:13:11 -03:00
Daniel Graña d439c26d76 update docstring and release notes 2015-05-22 20:00:58 -03:00
Alexey Vishnevsky 27ce3225bd Makes scrapy more async by letting to reactor spend another couple of cycles to accomplish its needs. 2015-05-22 17:05:19 -03:00
Julia Medina de6d232a02 Bump version: 0.25.1 → 1.0.0rc1 2015-05-22 13:24:27 -03:00
Julia Medina 29529e5e8e Merge pull request #1244 from Curita/1.0-release-notes 
1.0 release notes
 2015-05-22 13:21:17 -03:00
Julia Medina 600164594c New release cycle in .bumpversion.cfg 
1.0.0dev1 -> 1.0.0rc1 -> 1.0.0 -> 1.1.0dev1 -> ...
 2015-05-22 12:59:21 -03:00
Julia Medina afcf70cdc6 Add 1.0 release notes 2015-05-22 12:53:11 -03:00
Mikhail Korobov cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Daniel Graña 58717472f7 Merge pull request #1250 from chekunkov/scrapy-log-fix-incompatible-change 
[MRG+1] Keep level_names in scrapy.log for backwards compatibility
 2015-05-21 10:46:39 -03:00
Alexander Chekunkov 795ca3945f keep level_names in scrapy.log for backwards compatibility 2015-05-21 08:56:44 +00:00
Daniel Graña ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Daniel Graña 5beb9d251c Merge pull request #1243 from scrapy/remove-contrib-from-py3-ignores 
remove unnecessary lines from py3-ignores
 2015-05-19 12:20:51 -03:00
Mikhail Korobov 7a5b5ec4d6 TST remove unnecessary lines from py3-ignores 
scrapy/contrib is already skipped - see https://github.com/scrapy/scrapy/pull/1165
 2015-05-19 00:57:39 +05:00
Mikhail Korobov 21b17734c5 Merge pull request #1242 from Curita/exporters-single-module 
Move exporters/__init__.py to exporters.py
 2015-05-19 00:10:30 +05:00
Julia Medina 044f31cb8d Merge pull request #1240 from scrapy/fix-feedexport-logging 
MRG+1 fixed FeedExporter shutdown log messages
 2015-05-18 14:54:11 -03:00
Julia Medina af0c8f82f4 Move exporters/__init__.py to exporters.py 2015-05-18 14:46:23 -03:00
Mikhail Korobov 60e79db3ee fixed FeedExporter shutdown log messages 2015-05-18 19:28:37 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov c412034864 Fix GH-1223 2015-05-18 16:44:02 +05:00
Mikhail Korobov e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Mikhail Korobov 8d9e3b7e8d TST add a test for JSON lines item exporter 2015-05-18 16:43:23 +05:00
Daniel Graña 3a4d57b3f5 Merge pull request #1236 from Curita/fix-failure-tracebacks 
Replace FailureFormatter with direct failure to exc_info conversions in log calls
 2015-05-15 15:00:36 -03:00
Julia Medina e7574a809e Use method to get traceback of failures in failure_to_exc_info 2015-05-15 14:51:14 -03:00
Julia Medina 6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Daniel Graña 2d21677197 Merge pull request #1235 from Curita/remove-log-exception-calls 
Replace logger.exception calls to logger.error for Python compatibility
 2015-05-15 01:44:45 -03:00
Julia Medina 157cec7f64 Replace logger.exception calls for Python compatibility 2015-05-15 01:26:45 -03:00
Daniel Graña faba9bb095 Merge pull request #1206 from Curita/clean-test-collection 
Fix remaining warnings from pytest while collecting tests
 2015-05-14 23:19:47 -03:00
Daniel Graña 5e5e44e91e Merge pull request #1228 from nramirezuy/item_fields_defined_on_fields 
item fields defined on fields class attribute fix
 2015-05-14 23:09:24 -03:00