RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,816 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

1108 Commits

Author SHA1 Message Date
Julia Medina d6a06a769f Add CrawlerProcess to "Run Scrapy from a script" doc section 2015-04-29 23:09:20 -03:00
Julia Medina 3a71504d35 Extend CrawlerProcess documentation 2015-04-29 23:09:20 -03:00
Julia Medina a2ce78db63 Use autoclass for CrawlerRunner in docs/topics/api.rst 2015-04-29 23:09:20 -03:00
Julia Medina 353672d292 Add autodocs Sphinx extension 2015-04-29 23:09:20 -03:00
Julia Medina b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina 54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Wieland Hoffmann de6501ed1b Remove a duplicate word 2015-04-29 22:31:48 +02:00
Pablo Hoffman 3d2b74a6ff Merge pull request #1188 from eliasdorneles/favoring_web_scraping_over_screen_scraping 
[MRG+1] Favoring web scraping over screen scraping in the descriptions
 2015-04-29 16:49:43 -03:00
Elias Dorneles 3d3633f3d2 favoring web scraping over screen scraping in the descriptions 2015-04-25 11:20:20 -03:00
Julia Medina 7a92dae4c8 Change Scrapy log output through docs 2015-04-22 17:27:24 -03:00
Julia Medina 69a3d58110 Basic example on manually configuring log handlers 2015-04-22 17:24:41 -03:00
Julia Medina bd0b639b21 Fix logging usage across docs 2015-04-22 17:24:41 -03:00
Julia Medina 4811d16f1d Update `logger` attr and `log` method in the Spiders topic on docs 2015-04-22 17:24:41 -03:00
Julia Medina d47a7edc65 Update Logging topic on docs 2015-04-22 17:24:40 -03:00
Julia Medina ccdd8bfbcc Parametrize log formatting strings 2015-04-22 17:24:40 -03:00
Pablo Hoffman 0a5bbbaed3 Merge pull request #1159 from scrapy/feed-export-fields 
[MRG+1] FEED_EXPORT_FIELDS option
 2015-04-21 15:54:24 -03:00
Pablo Hoffman 4c4eb4f77d Merge pull request #1180 from eliasdorneles/tutorial-improvements 
Some improvements for Scrapy tutorial
 2015-04-21 15:49:39 -03:00
Daniel Graña e034947b54 Merge pull request #1166 from scrapy/spider-loader 
[MRG+1] rename SpiderManager to SpiderLoader
 2015-04-21 15:42:17 -03:00
Pablo Hoffman 06e1ca9ef2 Merge pull request #1177 from Curita/remove-djangoitem 
Remove djangoitem
 2015-04-21 15:35:47 -03:00
Pablo Hoffman e4122cdd79 Merge pull request #1150 from eliasdorneles/docs-files-pipeline 
Documenting Files Pipeline together with Images Pipeline
 2015-04-21 15:28:47 -03:00
Elias Dorneles d63567531d change data extraction in crawl example to be consistent with tutorial, removed statement implying mandatory usage of Item 2015-04-21 11:30:48 -03:00
Elias Dorneles f7da69d116 fixing example CSS expr 2015-04-21 11:19:10 -03:00
Elias Dorneles ff007afb9d expanded crawling primer with examples, and applied other suggestions from the review 2015-04-21 10:57:44 -03:00
Elias Dorneles 595146e158 some improvements for Scrapy tutorial 2015-04-20 21:10:07 -03:00
nyov 7ae37d61f9 remove scrapy deploy command 
(closes #1027, #1095, #1102)
 2015-04-20 20:35:06 -03:00
Julia Medina 6da091f162 Remove djangoitem topic from index 2015-04-20 20:19:28 -03:00
Daniel Graña 16a1a938c1 Add 0.24.6 release notes 2015-04-20 01:40:21 -03:00
Mikhail Korobov 1794a893f4 Merge pull request #1172 from bagratte/docs 
minor corrections in documentation.
 2015-04-19 21:41:29 +05:00
Julia Medina ffc60910aa Remove djangoitem since we moved it to scrapy/scrapy-djangoitem 2015-04-19 13:09:25 -03:00
bagratte 1312bcd007 minor corrections in documentation. 2015-04-19 18:58:15 +04:00
bagratte beea9267a1 minor corrections in documentation. 2015-04-18 19:48:25 +04:00
rajathkumarmp 02629b5f7b Added link to ipython in doc. 2015-04-18 13:00:34 +05:30
Pablo Hoffman 757309efea Merge pull request #1169 from bagratte/docs 
add some minor stylistic and grammar corrections to tutorial.rst.
 2015-04-17 19:52:11 -03:00
José Ricardo dffc4549d7 Fix small typo in the docs 2015-04-17 23:37:12 +05:00
bagratte 8d339da4e5 add some minor stylistic and grammar corrections to tutorial.rst. 2015-04-17 20:55:02 +04:00
Mikhail Korobov 403e7c7c70 rename scrapy.spidermanager.SpiderManager to scrapy.spiderloader.SpiderLoader 2015-04-16 20:07:53 +05:00
Elias Dorneles a1f3b3c7f0 images.rst -> media-pipeline.rst 2015-04-15 14:26:08 -03:00
Elias Dorneles fd1c688ad6 addressing issues from review at #1150 2015-04-15 14:21:48 -03:00
Pablo Hoffman fb85bd4b10 Merge pull request #1132 from sibiryakov/dns-options2 
[MRG+1] Dns cache size and timeout options
 2015-04-15 14:08:19 -03:00
Mikhail Korobov cb5280ab0d DOC tweak deployment docs 2015-04-15 19:25:22 +05:00
Nicolás Alejandro Ramírez Quiros e054b3c45f Merge pull request #1160 from scrapy/docs-theme 
Change local docs theme
 2015-04-15 10:34:19 -03:00
Mikhail Korobov a11bd3e89d DOC update docs readme file - sphinx >= 1.3 is now required. 2015-04-15 11:50:57 +05:00
Mikhail Korobov dd84f4bd9a DOC remove css file which isno longer needed 2015-04-15 03:44:49 +05:00
Mikhail Korobov 7dcd7f3ca7 DOC make local docs use the same theme as readthedocs.org 
sphinx_rtd_theme is builtin in recent Sphinx versions.
 2015-04-15 03:43:30 +05:00
Mikhail Korobov 1534e8540b FEED_EXPORT_FIELDS option 2015-04-15 02:48:26 +05:00
Mikhail Korobov 7644813286 allow Crawler, CrawlerRunner and CrawlerProcess to accept dicts instead of Setting objects 2015-04-14 23:26:05 +05:00
Daniel Graña e85679fa6b Merge pull request #1147 from Curita/pass-crawlers-to-runner-crawl-calls 
Allow passing Crawler instances directly to CrawlerRunner.crawl()
 2015-04-13 14:23:15 -03:00
Daniel Graña f71175a449 More replaces of references to old `sel` shortcut 2015-04-12 21:56:23 -03:00
Diana Huang 91a60d9f3d Remove references to the `sel` object in shell.rst 
The current documentation has references to the deprecated `sel` when interacting with the shell. I've removed them and replaced uses of `sel.xpath` with `response.xpath` instead.
 2015-04-12 13:44:32 -04:00
Elias Dorneles dce48b8654 documenting Files Pipeline together with Images Pipeline 2015-04-11 13:57:55 -03:00
Julia Medina 3dabde6706 Update docs for CrawlerRunner.crawl() new usage 2015-04-10 12:09:07 -03:00
Julia Medina 86b0951327 Delete `crawler_deferreds` doc in CrawlerRunner 
This attribute is now an internal one since it's main use-case was
covered by CrawlerRunner.stop().
 2015-04-10 12:06:27 -03:00
Pablo Hoffman 39fd2e62ef fix typo in deploy.rst 2015-04-09 18:01:11 -03:00
Pablo Hoffman 1a12922b77 improve scrapy deploy documentation 2015-04-09 17:39:01 -03:00
Pablo Hoffman 9ea309c354 rename deployment.rst -> deploy.rst (consist with others like debug.rst) 2015-04-09 16:56:35 -03:00
Pablo Hoffman d8184a7239 Merge pull request #1124 from rdowinton/deployment-doc 
Added deployment section covering scrapyd-deploy and shub
 2015-04-09 16:53:25 -03:00
Alexander Sibiryakov 85aa3c7596 Dns cache size and timeout options 2015-04-02 18:30:59 +02:00
nyov 92b574309e documentation build warning fixes 2015-04-01 19:46:21 +00:00
Daniel Graña 27591b55fc Merge pull request #1123 from sibiryakov/reactor-threadpool-size 
[MRG+1] Reactor threadpool max size setting
 2015-04-01 15:08:03 -03:00
Alexander Sibiryakov b794cdaf4b Broad crawls notes. 2015-04-01 12:07:03 +02:00
Alexander Sibiryakov e7b274edf3 Reformat to 80 characters per line. 2015-04-01 11:49:55 +02:00
Alexander Sibiryakov 94fceb4c15 Fixing underscore size. 2015-04-01 11:25:10 +02:00
Richard Dowinton 2d142d6401 Added deployment section covering scrapyd-deploy and shub 2015-03-31 12:17:31 +01:00
Alexander Sibiryakov 5864d291d4 Setting documentation. 2015-03-31 11:10:56 +02:00
Elias Dorneles 18c5bc75c2 some more minor grammar fixes 2015-03-27 21:00:21 -03:00
Elias Dorneles 7135c4e694 some minor grammar fixes 2015-03-27 17:56:56 -03:00
Pablo Hoffman bb4c922d85 Merge pull request #1081 from scrapy/dict-items 
Allow spiders to return dicts.
 2015-03-27 15:19:27 -03:00
Daniel Graña 55a23d102f Merge pull request #1086 from Curita/response-urljoin 
Add Response.urljoin() helper
 2015-03-27 15:17:54 -03:00
Julia Medina f4e241a018 Merge pull request #1106 from eliasdorneles/overview-page-improvements 
[MRG+1] some improvements to overview page
 2015-03-27 15:16:33 -03:00
Mikhail Korobov 39085ae18f Merge pull request #1098 from nyov/nyov/userconfig 
[+1 MRG]look in ~/.config/scrapy.cfg for user config
 2015-03-27 02:11:35 +05:00
nyov 1134a9cab0 config: look in ~/.config/scrapy.cfg as well 2015-03-26 20:36:14 +00:00
Peter Bronez 475766c73a Converted sel.xpath() calls to response.xpath() in Extracting the data 2015-03-26 15:34:30 -04:00
Elias Dorneles 4dcecc98f9 moved example data to a better place 2015-03-26 15:45:17 -03:00
Elias Dorneles 7402e27230 fix community link 2015-03-26 15:35:31 -03:00
Elias Dorneles 729861c864 fixing indentation 2015-03-26 15:31:42 -03:00
Elias Dorneles 13d0ecde77 addressing more review comments, to avoid ambiguity on desired reading flow 2015-03-26 15:26:16 -03:00
Elias Dorneles 76e3bf1250 addressing comments from the review plus further editing 2015-03-26 14:26:20 -03:00
Elias Dorneles 8f4a268f37 added bit about async requests, improved phrasing 2015-03-26 12:14:56 -03:00
Elias Dorneles 32423d4a33 some improvements to overview page 2015-03-25 19:27:52 -03:00
Ramiro Morales 933dbc6be6 Oops 2015-03-25 18:33:17 -03:00
Ramiro Morales ca2575001e Add missing callback arg in jobs topic example. 2015-03-25 18:32:20 -03:00
Mikhail Korobov 5ac91e4883 DOC remove Dynamic Creation of Item Classes section 
It was a hack, and dicts-as-items cover most use cases.

Dicts don't allow to attach metadata to fields,
but e.g. adding "_meta" key and removing it in a custom serializer
is no worse than creating classes dynamically.
 2015-03-23 18:11:35 +05:00
Julia Medina cda3922507 Add Response.urljoin() helper 2015-03-19 19:07:52 -03:00
Pablo Hoffman c81eefaf81 fix doc links 2015-03-19 17:42:48 -03:00
Mikhail Korobov 8ac397670f DOC move .. module: declaration to a proper place 2015-03-19 21:41:36 +05:00
Faisal Anees 643984e1b4 Updated architecture.rst 
Added http://krondo.com/blog/?page_id=1327 as a resource
 2015-03-18 23:55:22 -03:00
Mikhail Korobov f16a33f34e DOC change structure of spider docs: 
* start with scrapy.Spider, then mention spider arguments,
  then describe generic spiders;
* change wording regarding start_urls/start_requests;
* show an example of start_requests vs start_urls;
* show an example of dicts as items;
* as defining Item is an optional step now, docs for Items are
  moved below Spider docs.
 2015-03-19 05:25:15 +05:00
Mikhail Korobov 817dbc6cbd DOC mention dicts in documentation; explain better what are Items for 2015-03-19 05:16:14 +05:00
Julia Medina 959aaad205 Document `re_first` 2015-03-18 21:11:08 -03:00
Mateusz Golewski 127c6c694a Fix extract_first() docs 2015-03-18 21:11:08 -03:00