RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,239 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

86 Commits

Author SHA1 Message Date
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Mikhail Korobov a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests 
Document that the crawl order is BFO for small numbers of start requests
 2019-07-08 21:26:04 +05:00
Adrián Chaves 783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Konstantin Lopuhin bb1f311891 Add PyPy3 support to faq and install doc 2017-12-25 15:46:05 +03:00
Konstantin Lopuhin ea41114cf0 Mention PyPy support, add PyPy to install docs 2017-12-25 12:35:04 +03:00
Hugo 44623687ab Drop support for EOL Python 3.3 2017-12-19 17:59:05 +02:00
Aditya 9dd680d5c9 Use https for external links wherever possible in docs 2017-10-26 23:33:45 +05:30
Paul Tremberth 163618c9b7 FAQ Rewrite note on Python 3 support on Windows 2017-03-29 12:02:44 +02:00
Pawel Miech f98ffb53b6 add docs for some scheduler settings 2016-11-29 16:52:54 +01:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth d999e3f7a7 More explicit description of DEPTH_PRIORITY 2016-01-27 12:57:03 +01:00
David Chen 0025d5a943 Fixed minor grammar issues. 2015-11-16 07:30:17 +08:00
Ally Weir bd2fe996aa Spelling correction 
incorrect use of "too" instead of "to"
 2015-06-01 20:47:22 +05:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Pablo Hoffman 1a12922b77 improve scrapy deploy documentation 2015-04-09 17:39:01 -03:00
Shadab Zafar 5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Daniel Graña 5b2faf61c3 recognize jl extension as jsonlines exporter and update docs 2014-06-25 13:55:15 -03:00
Julia Medina 80081054a2 Fix broken links in documentation 2014-04-09 18:57:52 -03:00
Capi Etheriel 72b6c96d9a Running lucasdemarchi/codespell to fix typos in docs 2014-03-06 12:40:55 -03:00
Daniel Graña 2df8156431 Drop Python 2.6 support 2013-10-29 13:44:00 -02:00
Hart 0ad01c34d4 fixed typo to parallel fix on 0.16 branch 2013-08-03 17:06:10 -07:00
cacovsky 8007762890 Add FAQ entry referencing Request.meta usage 2013-05-27 13:02:17 -03:00
Mikhail Korobov b245d592aa Update faq.rst 
spider.DOWNLOAD_DELAY is deprecated
 2013-04-18 02:42:15 +06:00
Pablo Hoffman bb20907254 minor updated to faq 2013-03-14 16:43:00 -03:00
Pablo Hoffman 098ccff862 added FAQ about error: "cannot import name crawler" 2013-03-14 12:57:59 -03:00
Pablo Hoffman 6ab8afb992 improve documentation about removing namespaces 2013-01-18 12:35:30 -02:00
Pablo Hoffman 1e2ee76df2 add documentation topics: Broad Crawls & Common Practies 2012-12-26 14:02:13 -02:00
Pablo Hoffman 1f0d167037 doc: removed broken proxyhub link from FAQ 2012-11-22 15:10:26 -02:00
Pablo Hoffman fff2871828 added doc section (and FAQ) about spider arguments 2012-09-04 14:49:30 -03:00
Pablo Hoffman e1be9c01bc updated FAQ about bot bans 2012-06-08 18:33:53 -03:00
Pablo Hoffman e521da2e2f Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
Pablo Hoffman ade5efdc61 added -o option to scrapy crawl, a convenient shortcut for using feed exports 2011-10-22 20:53:49 -02:00
Pablo Hoffman 431441cb52 updated documentation to remove references to old issue tracker and mercurial repos 2011-09-25 13:06:24 -03:00
Pablo Hoffman ce03ccd4ec updated documentation about DEPTH_PRIORITY and DFO/BFO crawls 2011-09-23 13:22:25 -03:00
Pablo Hoffman 76af0cdd44 updated documentation and code to use -s instead of --set option 2011-09-01 14:35:37 -03:00
Pablo Hoffman 549725215e Initial support for a persistent scheduler, to support pausing and resuming 
crawls.

* requests are serialized (using marshal by default) and stored on disk, using
  one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
  through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
  SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
  are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
 2011-08-02 11:57:55 -03:00
Pablo Hoffman f354a49d0f added FAQ about preventing bots getting banned 2011-07-28 00:40:30 -03:00
Pablo Hoffman b6b0a54d9f removed FAQ entry 2011-07-20 01:31:36 -03:00