RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 131 MiB
Commit Graph

78 Commits

Author SHA1 Message Date
Paul Tremberth 163618c9b7 FAQ Rewrite note on Python 3 support on Windows 2017-03-29 12:02:44 +02:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Ally Weir bd2fe996aa Spelling correction 
incorrect use of "too" instead of "to"
 2015-06-01 20:47:22 +05:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Pablo Hoffman 1a12922b77 improve scrapy deploy documentation 2015-04-09 17:39:01 -03:00
Shadab Zafar 5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Daniel Graña 5b2faf61c3 recognize jl extension as jsonlines exporter and update docs 2014-06-25 13:55:15 -03:00
Julia Medina 80081054a2 Fix broken links in documentation 2014-04-09 18:57:52 -03:00
Capi Etheriel 72b6c96d9a Running lucasdemarchi/codespell to fix typos in docs 2014-03-06 12:40:55 -03:00
Daniel Graña 2df8156431 Drop Python 2.6 support 2013-10-29 13:44:00 -02:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
cacovsky 8007762890 Add FAQ entry referencing Request.meta usage 2013-05-27 13:02:17 -03:00
Mikhail Korobov b245d592aa Update faq.rst 
spider.DOWNLOAD_DELAY is deprecated
 2013-04-18 02:42:15 +06:00
Pablo Hoffman bb20907254 minor updated to faq 2013-03-14 16:43:00 -03:00
Pablo Hoffman 098ccff862 added FAQ about error: "cannot import name crawler" 2013-03-14 12:57:59 -03:00
Pablo Hoffman 6ab8afb992 improve documentation about removing namespaces 2013-01-18 12:35:30 -02:00
Pablo Hoffman 1e2ee76df2 add documentation topics: Broad Crawls & Common Practies 2012-12-26 14:02:13 -02:00
Pablo Hoffman 1f0d167037 doc: removed broken proxyhub link from FAQ 2012-11-22 15:10:26 -02:00
Pablo Hoffman fff2871828 added doc section (and FAQ) about spider arguments 2012-09-04 14:49:30 -03:00
Pablo Hoffman e1be9c01bc updated FAQ about bot bans 2012-06-08 18:33:53 -03:00
Pablo Hoffman e521da2e2f Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
Pablo Hoffman ade5efdc61 added -o option to scrapy crawl, a convenient shortcut for using feed exports 2011-10-22 20:53:49 -02:00
Pablo Hoffman 431441cb52 updated documentation to remove references to old issue tracker and mercurial repos 2011-09-25 13:06:24 -03:00
Pablo Hoffman ce03ccd4ec updated documentation about DEPTH_PRIORITY and DFO/BFO crawls 2011-09-23 13:22:25 -03:00
Pablo Hoffman 76af0cdd44 updated documentation and code to use -s instead of --set option 2011-09-01 14:35:37 -03:00
Pablo Hoffman 549725215e Initial support for a persistent scheduler, to support pausing and resuming 
crawls.

* requests are serialized (using marshal by default) and stored on disk, using
  one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
  through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
  SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
  are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
 2011-08-02 11:57:55 -03:00
Pablo Hoffman f354a49d0f added FAQ about preventing bots getting banned 2011-07-28 00:40:30 -03:00
Pablo Hoffman b6b0a54d9f removed FAQ entry 2011-07-20 01:31:36 -03:00
Pablo Hoffman e3f640c7bf added FAQ entry about scrapy deploy issue on Mac + Python 2.5 2011-07-19 19:53:32 -03:00
Pablo Hoffman 763f3dc628 minor update to doc 2011-07-12 19:56:39 -03:00
Pablo Hoffman 4fde1ef94d added CloseSpider exception, to manually close spiders 2011-07-12 14:24:10 -03:00
Pablo Hoffman bf73002428 removed googledir example, replaced by dirbot project on github. updated docs accordingly 2011-04-28 02:28:39 -03:00
Pablo Hoffman 3ee2c94e93 Improved cookies middleware by making COOKIES_DEBUG nicer and documenting it 2011-04-06 14:54:48 -03:00
Daniel Grana c55355642c fix FAQ typos reported by marlun_ at #scrapy IRC channel 2011-02-16 08:57:42 -02:00
Pablo Hoffman 16d9a33951 added FAQ entry about working with big data feeds 2011-02-15 07:24:52 -02:00
Pablo Hoffman ac007802d6 Simplified installation guide, including lxml as alternative dependency to libxml2. Closes #280 2010-11-17 21:32:23 -02:00