RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/scrapy
History
Jamey Sharp e23a381337 Let spiders ignore bogus Cache-Control headers. 
Sites often set "no-store", "no-cache", "must-revalidate", etc., but get
upset at the traffic a spider can generate if it respects those
directives.

Allow the spider's author to selectively ignore Cache-Control directives
that are known to be unimportant for the sites being crawled.

We assume that the spider will not issue Cache-Control directives in
requests unless it actually needs them, so directives in requests are
not filtered.
 		2015-06-01 18:06:35 +08:00
..
commands Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
contracts Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
contrib Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
contrib_exp Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
core Add backward compatibility to LogFormatter 2015-04-22 17:27:24 -03:00
downloadermiddlewares Use cached responses if revalidation errors out. 2015-06-01 18:06:35 +08:00
exporters Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
extensions Let spiders ignore bogus Cache-Control headers. 2015-06-01 18:06:35 +08:00
http more python3 compatibility changes for urllib 2015-04-01 19:50:56 +00:00
linkextractors Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
loader Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
pipelines Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
selector encode invalid xpath with unicode_escape under PY2 2015-04-10 22:23:40 +03:00
settings Let spiders ignore bogus Cache-Control headers. 2015-06-01 18:06:35 +08:00
spidermiddlewares Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
spiders Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
templates Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
utils Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
xlib pydispatch - pep8 2015-03-16 21:43:43 +00:00
VERSION Bump version: 0.25.0 → 0.25.1 2014-07-09 03:05:39 -03:00
__init__.py favoring web scraping over screen scraping in the descriptions 2015-04-25 11:20:20 -03:00
_monkeypatches.py cleanup toplevel namespace 2014-04-15 01:09:35 +06:00
cmdline.py PY3: use six.iterkeys, six.iteritems, and tempfile 2014-07-14 18:47:22 +08:00
command.py Drop support for ScrapyCommand.crawler property 2014-08-12 14:02:56 -03:00
conf.py better backwards compatibility for scrapy.conf.settings 2012-10-25 11:41:40 -02:00
crawler.py Merge pull request #1060 from Curita/python-logging 2015-04-29 23:20:34 +05:00
dupefilter.py Change 'scrapy' logger for '__name__' on every module 2015-04-22 17:24:41 -03:00
exceptions.py pylint cleanup: unused imports and old-style exceptions 2013-10-18 19:23:45 +06:00
extension.py Made ExtensionManager a subclass of MiddlewareManager 2010-08-22 05:33:08 -03:00
interfaces.py rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
item.py Item multi inherit fixed 2015-04-21 18:10:52 -03:00
link.py Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
linkextractor.py Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
log.py Fix backward compatible functions in scrapy.log 2015-04-22 17:27:24 -03:00
logformatter.py Restructure LogFormatter to comply with std logging calls 2015-04-22 17:24:40 -03:00
mail.py Change 'scrapy' logger for '__name__' on every module 2015-04-22 17:24:41 -03:00
middleware.py Change 'scrapy' logger for '__name__' on every module 2015-04-22 17:24:41 -03:00
mime.types moved module scrapy.core.downloader.responsetypes to scrapy.responsetypes 2011-08-07 02:49:57 -03:00
project.py Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
resolver.py Dns cache size and timeout options 2015-04-02 18:30:59 +02:00
responsetypes.py PY3 MimeTypes wants files in text mode. 2014-08-01 03:28:05 +06:00
shell.py Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
signalmanager.py remove debugging code 2013-02-27 03:52:55 -02:00
signals.py An attempt to resolve #977, add signal to be sent when request is dropped by the scheduler 2014-11-27 15:10:15 +03:00
spider.py Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
spiderloader.py rename CrawlerRunner.spiders to CrawlerRunner.spider_loader 2015-04-17 01:54:26 +05:00
spidermanager.py rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
squeue.py PY3: use six.moves.cPickle 2014-07-14 17:15:28 +08:00
stats.py Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
statscol.py Change 'scrapy' logger for '__name__' on every module 2015-04-22 17:24:41 -03:00
telnet.py Change 'scrapy' logger for '__name__' on every module 2015-04-22 17:24:41 -03:00