RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,828 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

3337 Commits

Author SHA1 Message Date
Alexander Sibiryakov 55f71043e1 nextcall repetitive calls (heartbeats). 2015-12-04 10:07:31 -03:00
Daniel Graña b262411683 Backport fix compatibility with Twisted 15.4.0 2015-09-29 23:00:30 -03:00
Daniel Graña 13c45aca0f Add openssl version to version command 2015-08-12 14:26:58 -03:00
Daniel Graña be321d1c9f Bump version: 1.0.2 → 1.0.3 2015-08-11 13:49:05 -03:00
Daniel Graña 7e9997b2c2 Bump version: 1.0.1 → 1.0.2 2015-08-06 18:28:18 -03:00
Daniel Graña b04dd7d1bf Twisted 15.3.0 does not raises expected exception serializing lambda functions 2015-08-06 17:57:17 -03:00
Daniel Graña 1f01385379 Bump version: 1.0.0 → 1.0.1 2015-07-01 01:45:25 -03:00
Capi Etheriel b8567bc4bc Fix SelectJmes documentation 2015-07-01 01:40:11 -03:00
Daniel Graña cc00ad20cf Unquote request path before passing to FTPClient, it already escape paths 2015-07-01 01:39:29 -03:00
Julia Medina 3017b1033f Bump version: 1.0.0rc3 → 1.0.0 2015-06-19 17:56:44 -03:00
Julia Medina 1e7eecc274 Bump version: 1.0.0rc2 → 1.0.0rc3 2015-06-12 16:41:50 -03:00
Mikhail Korobov 0f733e89e6 DOC remove FailureFormatter mentions, stop copy-pasting configure_logging docstring 
(cherry picked from commit a611f8dd2d)
 2015-06-12 16:09:46 -03:00
Mikhail Korobov f001641543 (backwards-incompatible) allow to pass settings=None to configure_logging 
* use explicit argument for disabling root handler;
* handle LOG_STDOUT even if install_root_handler is False

(cherry picked from commit 9a787893e3)
 2015-06-12 16:09:39 -03:00
Mikhail Korobov c34ec6694c Stop reactor on Ctrl-C regardless of 'stop_after_crawl'. Fixes GH-1279. 
(cherry picked from commit 64399d18d8)
 2015-06-12 16:07:24 -03:00
Mikhail Korobov 3b0391b8f2 CrawlerProcess cleanup 
* remove unneeded lambda;
* extract _get_dns_resolver method and format code to pep8.

(cherry picked from commit 33d145e2f5)
 2015-06-12 16:07:17 -03:00
Berker Peksag 423826c1be Fix typos in scrapy/commands/setting.py 
* intepreted -> interpreted
* "a list" instead of "an float" in --getlist help
* "an float"-> "a float"

Also, backslashes were redundant. So I removed them.

(cherry picked from commit 66e5591eb9)
 2015-06-12 16:06:15 -03:00
Mikhail Korobov 7a30e6acbe make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 
(cherry picked from commit d047665c02)
 2015-06-12 16:05:36 -03:00
Rolando Espinoza b285cebcbf Do leveldb compactation for the httpcache on closing. 2015-06-12 11:18:37 -03:00
Julia Medina a2dbde98d5 Bump version: 1.0.0rc1 → 1.0.0rc2 2015-06-05 17:12:56 -03:00
Julia Medina 5cd0296a16 Remove deprecated %z formatting from the default LOG_DATEFORMAT 2015-06-04 13:42:01 -03:00
Mikhail Korobov f6ae78f4b1 fixed spider custom_settings 
https://github.com/scrapy/scrapy/pull/1128 moved spidercls.update_settings
call to a later stage; this commit moves it back.
 2015-06-03 22:19:05 +05:00
Julia Medina 2d6450a4ad Add helper to update deprecated class paths 2015-06-01 20:39:29 -03:00
Mikhail Korobov 19ac046343 fixed CrawlerProcess when settings are passed as dicts 
See https://github.com/scrapy/scrapy/pull/1156

(cherry picked from commit aa6a72707d)
 2015-06-01 12:50:41 -03:00
Alexander Chekunkov 6f6a251a9b LogCounterHandler is never removed from root handlers list, fix that 
lambda is garbage collected and because receiver is added as weak reference by default - when signals.engine_stopped is fired logging.root.removeHandler is not executed. Fixed that by assigning lambda to a private argument and not by using connect(..., weak=False) because I belive this lambda function should be collected with crawler object

(cherry picked from commit b2765aabd8)
 2015-05-27 12:02:45 -03:00
Alexander Chekunkov 6c264f0e22 remove CrawlerProcess.stopping as it isn't used any more 
(cherry picked from commit b0ea3e38d1)
 2015-05-27 12:02:34 -03:00
Daniel Graña d439c26d76 update docstring and release notes 2015-05-22 20:00:58 -03:00
Alexey Vishnevsky 27ce3225bd Makes scrapy more async by letting to reactor spend another couple of cycles to accomplish its needs. 2015-05-22 17:05:19 -03:00
Julia Medina de6d232a02 Bump version: 0.25.1 → 1.0.0rc1 2015-05-22 13:24:27 -03:00
Mikhail Korobov cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Alexander Chekunkov 795ca3945f keep level_names in scrapy.log for backwards compatibility 2015-05-21 08:56:44 +00:00
Daniel Graña ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Mikhail Korobov 21b17734c5 Merge pull request #1242 from Curita/exporters-single-module 
Move exporters/__init__.py to exporters.py
 2015-05-19 00:10:30 +05:00
Julia Medina af0c8f82f4 Move exporters/__init__.py to exporters.py 2015-05-18 14:46:23 -03:00
Mikhail Korobov 60e79db3ee fixed FeedExporter shutdown log messages 2015-05-18 19:28:37 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov c412034864 Fix GH-1223 2015-05-18 16:44:02 +05:00
Julia Medina e7574a809e Use method to get traceback of failures in failure_to_exc_info 2015-05-15 14:51:14 -03:00
Julia Medina 6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Julia Medina 157cec7f64 Replace logger.exception calls for Python compatibility 2015-05-15 01:26:45 -03:00
Daniel Graña 5e5e44e91e Merge pull request #1228 from nramirezuy/item_fields_defined_on_fields 
item fields defined on fields class attribute fix
 2015-05-14 23:09:24 -03:00
nramirezuy 773ea5a5ef item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
Julia Medina 53fdaa3f70 scrapy/spider.py shim 2015-05-09 04:27:43 -03:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina d4926091ab Delete scrapy/spiders/__init__.py 2015-05-09 03:32:01 -03:00
Julia Medina 32945242ae scrapy/linkextractor.py shim 2015-05-09 03:28:39 -03:00
Julia Medina d72536688f Move scrapy/linkextractor.py to scrapy/linkextractors/__init__.py 2015-05-09 03:28:37 -03:00
Julia Medina 896b6dd481 Delete scrapy/linkextractors/__init__.py 2015-05-09 03:24:18 -03:00
Julia Medina cfd40ed57f scrapy/command.py shim 2015-05-09 03:20:48 -03:00
Julia Medina 616aec92c8 Move scrapy/command.py to scrapy/commands/__init__.py 2015-05-09 03:20:36 -03:00