RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,290 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

66 Commits

Author SHA1 Message Date
Victor Torres 35ce92a419 fix typo (Response -> Request) 
check docs for more information b5c552d17f/docs/topics/spider-middleware.rst
 2019-04-05 14:31:15 -03:00
Daniel Graña 65d631329a Be consistent with domain used for links to documentation website 2019-01-31 01:28:53 -03:00
Mikhail Korobov 8c29be606c update spider templates to use .get 2018-09-12 17:57:27 +05:00
Aditya dae7b1cdd0 Migrate all subdomains on readthedocs.org to readthedocs.io 2017-10-28 16:53:32 +05:30
Aditya 9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Paul Tremberth aaaa4da7a4 Add template for a downloader middleware 2017-07-28 13:48:23 +02:00
Arvind Chembarpu e85f0db128 Use single quotes uniformly 
The default spider template mixes single and double quotes.
 2017-02-25 16:48:17 +05:30
mangogao 4274f0d4d4 Add omitted "self" arguments 
“self” argument is omitted in some methods.
 2017-02-25 15:44:20 +08:00
Raul Gallegos e7c7e055ff settings: fixing name of the pipeline template 2016-12-24 11:55:04 -05:00
muriloviana 09c401bf8e use crawler only to register signals in from_crawler() 2016-10-19 14:09:01 -02:00
muriloviana 32fd692810 define method spider_opened and update class instructions 2016-10-19 13:50:09 -02:00
muriloviana 38e292a132 add from_crawler method to template and turn the returns methods explicit 2016-10-19 12:47:23 -02:00
muriloviana 34f2014c55 change settings middleware name and updating middleware template 2016-10-19 00:37:31 -02:00
muriloviana d5bd44a5b9 add middleware to template project 2016-10-18 17:29:16 -02:00
Raul Gallegos 118b42ab59 making start_urls a list in basic genspider template 2016-10-11 22:08:05 -05:00
Raul Gallegos dd778892d0 genspider: removing www. from starturl templates 2016-10-05 12:16:45 -05:00
Valdir Stumm Junior 8987b17730 remove references to Item classes in templates 2016-07-06 15:10:48 -03:00
Mikhail Korobov f30758c246 Enable robots.txt handling by default for new projects. Fixes GH-1668. 
For backwards compatibility reasons the default value is not changed.
 2016-01-26 17:47:46 +05:00
Νικόλαος-Διγενής Καραγιάννης d523c75f5e Relocate telnetconsole to extensions/ 2015-10-05 21:16:04 +03:00
Mikhail Korobov 5a13022713 style fixes for settings.py created by scrapy startproject 2015-09-13 21:46:20 +05:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
nyov 7ae37d61f9 remove scrapy deploy command 
(closes #1027, #1095, #1102)
 2015-04-20 20:35:06 -03:00
Elias Dorneles d7cb2b9a91 making commented code indentation consistent 2015-03-19 15:41:43 -03:00
Elias Dorneles fd67fe273a using default values for settings that are off by default 2015-03-17 09:34:54 -03:00
Elias Dorneles aa56dd30ea updating new settings to template, as pointed in PR review 2015-03-16 17:36:46 -03:00
Elias Dorneles 05a88152af adding more settings to project template 2015-03-14 16:12:37 -03:00
Mikhail Korobov 6fb7093154 add utf8 encoding header to spider templates 2014-06-26 00:30:27 +06:00
Mikhail Korobov df4ba41c82 Merge pull request #688 from stillzhl/master 
add encoding utf-8 to the first line
 2014-06-26 00:26:12 +06:00
Daniel Graña a9ecef5662 promote LxmlLinkExtractor as default in docs 2014-06-25 14:34:30 -03:00
Alexey Bezhan 5d37e4dc5a Fix PEP8 warnings in spider templates 2014-04-24 17:30:35 +01:00
Alexey Bezhan 1d6cc257ea Fix PEP8 warnings in project template files 2014-04-24 17:30:13 +01:00
Daniel Graña 72f68dcc8e update spider templates 2014-04-23 23:39:58 -03:00
Mikhail Korobov 4ddad88d6f use "import scrapy" in templates 2014-04-15 01:09:35 +06:00
Irhine 86a60e2cfc support i18n by using utf-8 coding template files 2014-04-14 18:14:29 +08:00
Irhine 943b6f90e9 add encoding utf-8 to the first line 2014-04-14 17:05:42 +08:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
nyov b6a200d02a Update CrawlSpider Template with Selector changes 2013-12-06 20:48:25 +00:00
vkrest ac4c66f351 Fix broken doc links. 
All generated template files contain broken link to doc files.
 2012-11-12 22:43:42 +02:00
Pablo Hoffman 1c5294bee1 update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 2012-04-19 02:35:48 -03:00
Pablo Hoffman 27018fced7 changed default user agent to Scrapy/0.15 (+http://scrapy.org) and removed no longer needed BOT_VERSION setting 2012-03-23 13:45:21 -03:00
Pablo Hoffman 9f742fc97c removed unused import from 'crawl' spider template 2011-07-20 01:04:16 -03:00
Pablo Hoffman fbafb295e8 removed DEFAULT_ITEM_CLASS setting from settings in new project template 2011-06-23 11:34:28 -03:00
LucianU 0c5f605b0e The xmlfeed.tmpl file didn't use the naming convention specific of the XMLFeedSpider. Namely, it used parse_item (which has been deprecated) instead of parse_node and it didn't show the iterator and itertag attributes. 2011-01-11 17:23:46 -02:00
Pablo Hoffman d988ca1ec2 Some changes to scrapy deploy command: 
* changed deploy section names to [deploy:target]
* project is now passed through a -p|--project option
* version can now be set in the target configuration
* switched meaning of -l and -L options

* updated documentation accordingly
 2010-11-08 17:01:06 -02:00
Pablo Hoffman b6c2b55e5b Splitted settings classes from settings singleton. Closes #244 
--HG--
rename : scrapy/conf/__init__.py => scrapy/conf.py
rename : scrapy/conf/default_settings.py => scrapy/settings/default_settings.py
rename : scrapy/tests/test_conf.py => scrapy/tests/test_settings.py
 2010-09-22 15:47:33 -03:00
Pablo Hoffman 37776618a3 Changed format of scrapy.cfg file to contain a [settings] section and a 'default' key inside it, instead of the other way around 2010-09-02 21:22:17 -03:00
Pablo Hoffman 34554da201 Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. Closes #199. Also updated documenation accordingly and added convenient scrapy.bat script for running from Windows. 
--HG--
rename : debian/scrapy-ctl.1 => debian/scrapy.1
rename : docs/topics/scrapy-ctl.rst => docs/topics/cmdline.rst
 2010-08-18 19:48:32 -03:00
Pablo Hoffman d17695ee4b Added "scrapy" command with project settings auto-discovery. Refs #199 2010-08-17 00:59:24 -03:00