RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,809 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

6809 Commits

Author SHA1 Message Date
kasun Herath 71ef321b68 sort_keys while serializing to json 2018-12-12 11:12:48 +05:30
kasun Herath ecda69130e allow to send empty data values and docs changes 2018-12-10 22:34:49 +05:30
kasun Herath 3c981bf204 add documentation 2018-12-09 12:56:12 +05:30
kasun Herath c347acbff6 warning if body and data are provided 2018-12-09 11:27:09 +05:30
kasun Herath cd619c1d4f removed overriden replace method 2018-12-08 22:10:45 +05:30
kasun Herath 1b2b8b4bf0 fix tests under py3 2018-11-27 08:57:44 +05:30
kasun Herath 1ce6662a9d Implement Request subclass for json requests 2018-11-24 20:02:00 +05:30
Mikhail Korobov dc65e7527d
Merge pull request #3468 from ilhaoni/feature/update-docs-intro-tutorial 
[MRG+1] Update Scrapy Tutorial docs
 2018-11-06 03:33:09 +05:00
Immanuella Lim 6c98010f11 Remove 'Dive into Python3' reference 2018-11-04 16:04:45 +08:00
Immanuella Lim c9b5bd6ad7 Remove ad link Dive Into Python3 from tutorial docs 2018-10-18 02:22:32 +08:00
Daniel Graña 06f2db7fd1
Merge pull request #3445 from jfflisikowski/3097-item-urls-not-defined-in-example 
[MRG+1] Updating debug example file (#3097)
 2018-10-16 19:51:32 -03:00
jfflisikowski edaf74bfae Correct the unclear comments by adding <# < processing code not shown > 2018-10-02 19:48:48 +02:00
Daniel Graña bafd174a9f
Merge pull request #3390 from scrapy/parsel-1.5 
[MRG+1] update Scrapy to use parsel 1.5
 2018-09-18 13:07:41 -03:00
Mikhail Korobov ffbd33edac DOC mention gotcha with `foo::text` selector and empty `foo` elements 
also, move "Selecting attributes" reference closer to `a::atr(href)` example
 2018-09-18 05:03:35 +05:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments 
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
 2018-09-18 05:02:17 +05:00
Mikhail Korobov 9db21e5502 DOC fix remove_namespaces example 
See https://github.com/scrapy/parsel/pull/119
 2018-09-15 02:43:37 +05:00
Mikhail Korobov dc95ecbe25 DOC use autodocs for selectors; document more methods and attributes; suggest get/getall 2018-09-12 18:36:25 +05:00
Mikhail Korobov 7fdfdb7fa2 DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction 2018-09-12 17:57:27 +05:00
Mikhail Korobov bdcc045f62 DOC switch from .extract to get/getall API in docs 
Also, response.urljoin is added in a few places, for robustness.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov afce9716fa DOC mention .attrib in the tutorial 2018-09-12 17:57:27 +05:00
Mikhail Korobov 12e42bbe06 switch SgmlLinkExtractor to .getall 2018-09-12 17:57:27 +05:00
Mikhail Korobov 460f0f0451 [backwards incompatible] switch ItemLoader from .extract to .getall. 
This change is backwards incompatible if ItemLoader is used with a custom Selector
subclass which overrides .extract without overriding .getall.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov 8c29be606c update spider templates to use .get 2018-09-12 17:57:27 +05:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Mikhail Korobov 2c48d156db DOC cleanup references in tutorials: 
* remove unused link
* fix ReST syntax
* fix a link to regular expression docs
 2018-09-12 17:57:27 +05:00
Mikhail Korobov 09fd6c2a81 DOC unlink Firefox & Firebug sections from the tutorial for now. 
See https://github.com/scrapy/scrapy/issues/3373 and https://github.com/scrapy/scrapy/issues/3372 for motivation.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov d32c4deaa9 DOC update Scrapy selectors tutorial to match parsel's tutorial better 2018-09-12 17:57:27 +05:00
Mikhail Korobov ca27010d4f DOC .extract_first() -> .get() 2018-09-12 17:57:27 +05:00
Mikhail Korobov 395d9d033a add pytest temp files to gitignore 2018-09-12 17:57:27 +05:00
Mikhail Korobov 0ccead6681 DOC more Python 3 in examples 2018-09-12 17:57:27 +05:00
Mikhail Korobov 25ac4691b4 require parsel 1.5+ 2018-09-12 17:57:27 +05:00
Daniel Graña 7223978594
Merge pull request #3381 from StasDeep/fix/issue-3380 
[MRG+1] Use dont_filter=True for contracts requests
 2018-09-06 11:41:02 -03:00
Daniel Graña 2aae514f99
Merge branch 'master' into fix/issue-3380 2018-09-05 12:25:07 -03:00
Daniel Graña 5176765589
Merge pull request #3377 from StasDeep/feature/issue-3364 
Add handling of errors in contract methods
 2018-09-05 12:24:50 -03:00
Daniel Graña d7b98a3714
Merge branch 'master' into feature/issue-3364 2018-09-05 11:26:59 -03:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382 
[MRG+1] Add ability to use FormRequest in contracts
 2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Vostretsov Nikita c02cfa574c remove comma 2018-08-29 11:21:55 +00:00
Daniel Graña 0007cd032d
Merge pull request #3405 from stav/offsite-pep8 
[MRG+1] PEP8 ofsite middleware
 2018-08-28 14:51:39 -03:00
Steven Almeroth 6e9fa3a41f PEP8 ofsite middleware 2018-08-24 15:18:16 -04:00
Mikhail Korobov e45ef7dcd9
Merge pull request #3400 from testingcan/docs-developer-tools 
[MRG+1] Added general guide for developer tools

Fixes #3373 and #3372.
 2018-08-23 19:44:14 +05:00
Raphael Wuillemier 79de3d569a Removed obsolete firebug-images 2018-08-23 16:19:13 +02:00
testingcan e98e7f8506
Added missing curly brace 2018-08-23 14:50:49 +02:00
Raphael Wuillemier 4d3aaabbca Updated code, added code snippets and improved readability 2018-08-23 12:40:31 +02:00
testingcan 3a71e7dbce
Increased length of "=" 2018-08-22 16:57:51 +02:00
Raphael Wuillemier af555cab23 Added general guide for developer tools instead of Firefox and Firebug-sections 2018-08-22 14:15:53 +02:00
Stas Glubokiy 9ab85fe788 Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3382 2018-08-19 17:21:28 +03:00
Stas Glubokiy 57824600a8 Use six.get_unbound_function in test_same_url 2018-08-19 16:56:41 +03:00
Stas Glubokiy 0467737cf0 Fix mockserver usage 2018-08-18 15:43:46 +03:00