RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,673 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

3673 Commits

Author SHA1 Message Date
alexanderlukanin13 6b35166de6 Converted map() to list comprehension or generator for Python 3 compatibility 2013-10-22 13:18:26 +06:00
Pablo Hoffman 911c8082b0 simplified description of crawl command 2013-10-21 14:42:51 -02:00
Pablo Hoffman e8ee449a2a Merge pull request #432 from darkrho/crawl-url 
Removed URL reference in crawl command and .tld suffix in docs for spider names
 2013-10-21 09:40:58 -07:00
Rolando Espinoza La fuente 34543c2b2e DOCS removed .tld suffix for spider names for the sake of consistency. 2013-10-19 23:03:20 -04:00
Rolando Espinoza La fuente c2ced9a234 Removed reference to URL in crawl command as it's no longer supported. 2013-10-19 22:57:37 -04:00
Rolando Espinoza La fuente dc666f6e5d PEP8 improvements 
* Space lines around classes/functions.
* Space in inline comments.
* Remove redundant backslash between brackets.
 2013-10-19 15:40:42 -04:00
Mikhail Korobov 62fd5b3213 Merge pull request #430 from alexanderlukanin13/cleanup 
pylint cleanup: unused imports and old-style exceptions
 2013-10-18 16:20:04 -07:00
alexanderlukanin13 105e46205b restored import *, added comment 2013-10-19 00:48:22 +06:00
Daniel Graña 1f17230f17 add testing environment for development versions of w3lib and queuelib 2013-10-18 16:08:27 -02:00
alexanderlukanin13 ddcd571000 pylint cleanup: unused imports and old-style exceptions 2013-10-18 19:23:45 +06:00
Mikhail Korobov 2e8cc281b0 Merge pull request #429 from alexanderlukanin13/bugfix 
wrong variable name
 2013-10-18 02:28:06 -07:00
alexanderlukanin13 192c323a0c wrong variable name 2013-10-18 11:30:48 +06:00
Daniel Graña 757e262267 update requirement for cssselect >= 0.9 2013-10-17 09:33:15 -02:00
Daniel Graña 875b07aef8 fix references to old selector naming in docs 2013-10-17 09:33:15 -02:00
Mikhail Korobov 086b8a20d4 typo fix in TextResponse docs 2013-10-17 04:50:30 +06:00
Pablo Hoffman 951a9f3f4c Merge pull request #226 from scraperdragon/patch-1 
Parameters to Request() in wrong order
 2013-10-16 13:15:52 -07:00
Daniel Graña 289688e39e Merge pull request #426 from scrapy/selectors-unified 
[MRG] Selectors unified API
 2013-10-16 12:58:12 -07:00
Daniel Graña 1461363809 Replace `contenttype` references by `type` 
The type to choose from is the selector type, not the input type. A
content-type doesn't make sense in this context.
 2013-10-16 17:37:25 -02:00
Daniel Graña 155ea08ea1 use `sel` name for Selector's instances in docs, internals and shell 2013-10-15 15:58:42 -02:00
Daniel Graña 1abb1af0c6 fix typos and wording on selector's introduction 2013-10-15 10:13:43 -02:00
Dragon Dave a3b711bdea Move callback blob; mention errback 2013-10-15 12:19:42 +01:00
scraperdragon 0ba0d85685 Parameters to Request() in wrong order 
Implied that callback wasn't the first optional unnamed parameter.
 2013-10-15 11:50:43 +01:00
Daniel Graña 28999590fa update release notes 2013-10-14 16:41:04 -02:00
Daniel Graña ab9462a251 remove more references to libxml2 2013-10-14 16:37:14 -02:00
Daniel Graña 4645f9e03c Updates docs to reflect unified selectors api 2013-10-14 16:31:20 -02:00
Daniel Graña add3506928 remove internal references to old selector classes and api 2013-10-14 10:51:16 -02:00
Daniel Graña 4e94b38396 port tests to new Selector class 2013-10-14 10:35:02 -02:00
Daniel Graña e4d6e2eb31 default xpath selector was html 2013-10-11 19:53:32 -02:00
Daniel Graña a9eb0b74c8 port scrapy shell to unified selector api 2013-10-11 19:53:01 -02:00
Daniel Graña 681af6b258 Remove CSS*Selector classes and port its tests 2013-10-11 18:22:03 -02:00
Daniel Graña c3d28cc412 working implementaion of unified api 2013-10-11 18:06:27 -02:00
Daniel Graña bf37f78572 Drop libxml2 selectors backend 2013-10-11 18:02:35 -02:00
Daniel Graña 6d598f0d94 Update selectors docs 2013-10-10 18:24:00 -02:00
Paul Tremberth ea579bca04 Support only ::text and ::attr(name) + add more CSS selector tests 2013-10-10 18:23:15 -02:00
Paul Tremberth 52e2eb6b70 Adapt to latest cssselect API supporting pseudo-elements 2013-10-10 18:23:15 -02:00
Daniel Graña 4e6967b854 extend css selectors with ":text" and :attribute(<name>) #176 2013-10-10 18:23:15 -02:00
Daniel Graña b38ac27eee rename XPathSelectorList as SelectorList #176 2013-10-10 18:23:15 -02:00
Capi Etheriel bc17e9d412 Adds HtmlCSSSelector and XmlCSSSelector classes, cssselect as optional dependency. 
Ported .get() from _Element and .text_content() from HTMLMixin

Add CSS selectors to scrapy shell

Documenting CSS Selectors: Constructing selectors

Documenting CSS Selectors: Using Selectors

Make CSS Selectors a default feature.

Adds XPath powers to CSS Selectors and some syntactic sugar.

Removes methods copied over from lxml.html.HtmlMixin.

Updating docs to use new CSS Selector super powers.

Documenting CSS Selectors: Regular Expressions

Moving section after Nesting section, since it mentions it.

Documenting CSS Selectors: Nesting Selectors

Fix XPath specificity in lxml.selector.CSSSelectorMixin.text

Cleaning up unused stuff from cssel.py

Changing the behavior of lxml.selector.CSSSelectorMixin.text.

Concatenating all of the descendant text nodes is more useful
than returning it in pieces (there's xpath() if you need that).

Documenting CSS Selectors: CSS Selector objects

Documenting CSS Selectors: CSSSelectorList objects

Documenting CSS Selectors: HtmlCSSSelector objects

Documenting CSS Selectors: XmlCSSSelector objects

Fixing some documentations typos and errors

Enforcing the 80-char width lines

Tidying up CSS selectors and CSSSelectorMixin objects

Adding some missing references in documentation.

Fixing lxml.selector.CSSSelectorList.text
 2013-10-10 18:23:15 -02:00
Daniel Graña 8bf3284ebf Merge pull request #380 from dangra/dont-log-ignorerequest 
Dont log IgnoreRequest exception as download failure
 2013-10-10 11:53:08 -07:00
Pablo Hoffman 1a2db089ff Merge pull request #418 from nramirezuy/engine-multispider 
engine multispider support removed
 2013-10-10 10:31:13 -07:00
Daniel Graña 5b5dd679b0 Add 0.18.4 release notes 
Conflicts:
	docs/news.rst
 2013-10-10 01:04:35 -02:00
Daniel Graña aa6fb7daaa IPython refuses to update the namespace. fix #396 
IPython embedding code borrowed from https://github.com/mitsuhiko/werkzeug/pull/85
 2013-10-10 00:57:14 -02:00
Daniel Graña 7b1288ba54 Fix AlreadyCalledError replacing a request in shell command. closes #407 2013-10-09 23:50:35 -02:00
Daniel Graña 5eb429999e Fix start_requests lazyness and early hangs 
- Removes new public methods added by #330 to Crawler and CrawlerProcess
- Add test for start_requests lazy evaluation
- Fix and test hangs when start_requests erroed before returning the generator
- Add test when start_requests fails while generating requests
- Simplify Crawler and CrawlerProcess implementation taking in count
  that only one spider can be attached per Crawler. As required by SEP-019
- "scrapy settings" command do not require starting a Crawler anymore
 2013-10-09 22:58:10 -02:00
Pablo Hoffman 2bd5002fb4 Merge pull request #416 from redapple/attrnamespaces 
.remove_namespaces(): remove namespaces on elements' attributes
 2013-10-09 15:34:08 -07:00
Pablo Hoffman e1683ddf9b fix doc typo 2013-10-09 17:24:12 -02:00
nramirezuy a84e5f806c engine multispider support removed 2013-10-09 15:02:12 -02:00
Pablo Hoffman ce82d22c9e Merge pull request #423 from nramirezuy/defaultheaders-multispider 
defaultheaders multi spider support removed
 2013-10-09 09:03:33 -07:00
nramirezuy 332bf3b6b7 defaultheaders multi spider support removed 2013-10-09 13:50:15 -02:00
Pablo Hoffman 7ec01799d5 Merge pull request #419 from nramirezuy/robotstxt-multispider 
robotstxt mid multi spider support removed
 2013-10-09 07:43:11 -07:00