RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,753 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

40 Commits

Author SHA1 Message Date
Dracony 33a39b368f added FEED_EXPORT_ENCODING setting to allow encoding specification 2016-07-12 16:20:17 +02:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
nanolab a583e4d531 Update httpcache.py 
It checks cache directory modification time, but have to check file modification time.
 2016-03-30 10:57:48 +02:00
Konstantin Lopuhin 3ada45a9bb S3FeedStorageTest: add botocore support, and organize boto/botocore checks 2016-02-15 19:59:47 +03:00
orangain 1cb841bfb3 PY3: Fix TypeError when outputting to stdout 
This will fix #1768.
 2016-02-07 10:06:42 +09:00
Paul Tremberth 0349bbf9d3 Disable SpiderState extension if no JOBDIR set 2016-01-26 15:25:15 +01:00
Paul Tremberth 6ee8d8650a Disable CloseSpider extension if no CLOSPIDER_* setting set 2016-01-26 13:08:42 +01:00
Konstantin Lopuhin 085fdd6283 py3 fix for ignoring cache controls - map is not a list 2016-01-15 15:40:45 +03:00
Konstantin Lopuhin b0648271d6 py3 fix for rfc1123_to_epoch - "except Exception" was hiding bytes/str error 2016-01-15 15:17:30 +03:00
Konstantin Lopuhin 87849780bc some py3 fixes for RFC2616Policy 2016-01-15 13:13:10 +03:00
Konstantin Lopuhin ea0471e33a py3: fix LeveldbCacheStorage - using bytes as keys and values in leveldb 2016-01-15 12:22:41 +03:00
Konstantin Lopuhin e7ed1fd70d py3 compat in httpcache - headers are bytes 2016-01-15 11:48:07 +03:00
Jakob de Maeyer 52ecee6a62 Replace BaseSettings._getcomposite() with public .getwithbase() method 2015-11-11 17:39:57 +01:00
Julia Medina 72eeead6db Merge pull request #1524 from Digenis/relocate_telnet_console 
[MRG+1] Relocate telnetconsole to extensions/
 2015-10-30 12:10:38 -03:00
Jakob de Maeyer f249b309ab Move scrapy.utils.conf.remove_none_values to s.u.python.without_none_values 2015-10-27 12:43:33 +01:00
Jakob de Maeyer 26586ef5a6 Deprecate _BASE settings, unify _BASE backwards-compatibility 2015-10-27 12:43:23 +01:00
Νικόλαος-Διγενής Καραγιάννης d523c75f5e Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00
Julia Medina d706310d8b Merge pull request #1151 from marven/cache-control 
[MRG+1] RFC2616 policy enhancements + tests
 2015-07-11 08:06:20 -03:00
Daniel Graña 3fc4e0b319 Merge pull request #1282 from otherchirps/memusage-check-interval 
[MRG+1] Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options.
 2015-07-02 13:50:55 -03:00
Mikhail Korobov d850238c22 add AUTOTHROTTLE_TARGET_CONCURRENCY option and expand AutoThrottle docs 2015-06-27 04:59:42 +05:00
Mikhail Korobov 1fef9f1132 AutoThrottle: respect download_delay=0 spider attribute 2015-06-26 19:09:26 +05:00
Mikhail Korobov 17cb51315b drop support for AUTOTHROTTLE_MIN_DOWNLOAD_DELAY 
it was deprecated for years
 2015-06-26 19:07:54 +05:00
Mikhail Korobov 584252e8f2 move AutoThrottle default options to default_settings.py 2015-06-26 18:58:29 +05:00
Mikhail Korobov e6272e5209 make AutoThrottle._adjust_delay easier to understand 2015-06-26 18:44:23 +05:00
Rolando Espinoza 877c25793a Do leveldb compactation for the httpcache on closing. 2015-06-11 13:27:49 -04:00
Chris Nilsson 61dec83f70 Moved default value of MEMUSAGE_CHECK_INTERVAL_SECONDS to default_settings 2015-06-06 11:19:29 +10:00
Chris Nilsson eae25a04d9 Added MEMUSAGE_CHECK_INTERVAL_SECONDS to Memory usage extension options. 
Kept the default as it was, at 60.0 seconds. But added a setting to
allow this to be changed as desired.
 2015-06-06 00:39:14 +10:00
Jamey Sharp 1991550442 Allow client to bound max-age for revalidation. 
Unlike specifying "Cache-Control: no-cache", if the request specifies
"max-age=0", then the cached validators will be used if possible to
avoid re-fetching unchanged pages.

That said, it's still useful to be able to specify "no-cache" on the
request, in cases where the origin server may have changed page contents
without changing validators.
 2015-06-01 18:06:36 +08:00
Jamey Sharp c3b2cabf6c Allow setting RFC2616Policy to cache unconditionally. 
A spider may wish to have all responses available in the cache, for
future use with "Cache-Control: max-stale", for instance. The
DummyPolicy caches all responses but never revalidates them, and
sometimes a more nuanced policy is desirable.

This setting still respects "Cache-Control: no-store" directives in
responses. If you don't want that, filter "no-store" out of the
Cache-Control headers in responses you feed to the cache middleware.
 2015-06-01 18:06:35 +08:00
Jamey Sharp e23a381337 Let spiders ignore bogus Cache-Control headers. 
Sites often set "no-store", "no-cache", "must-revalidate", etc., but get
upset at the traffic a spider can generate if it respects those
directives.

Allow the spider's author to selectively ignore Cache-Control directives
that are known to be unimportant for the sites being crawled.

We assume that the spider will not issue Cache-Control directives in
requests unless it actually needs them, so directives in requests are
not filtered.
 2015-06-01 18:06:35 +08:00
Jamey Sharp dd3a46295c Support "Cache-Control: max-stale" in requests. 
This allows spiders to be configured with the full RFC2616 cache policy,
but avoid revalidation on a request-by-request basis, while remaining
conformant with the HTTP spec.
 2015-06-01 18:06:35 +08:00
Jamey Sharp 4446baae33 Use cached responses if revalidation errors out. 2015-06-01 18:06:35 +08:00
Daniel Graña ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Mikhail Korobov 60e79db3ee fixed FeedExporter shutdown log messages 2015-05-18 19:28:37 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov c412034864 Fix GH-1223 2015-05-18 16:44:02 +05:00
Julia Medina 6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00