RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,814 Commits 42 Branches 132 Tags 165 MiB
Commit Graph

8814 Commits

Author SHA1 Message Date
kasun Herath 1ce6662a9d Implement Request subclass for json requests 2018-11-24 20:02:00 +05:30
Vostretsov Nikita a25cf5c82f function to get unique file queues for any type of base queue 2018-11-20 16:13:09 +00:00
Konstantin Lopuhin 886513c375
Merge pull request #3495 from toddrme2178/patch-1 
[MRG+1] Include additional files in sdists
 2018-11-20 11:10:32 +03:00
Frederik Elwert 127bf499f1
Add documentation to `scrapy shell` command. 
The special syntax required for local files (`./file.html`) is not documented as part of the `scrapy shell --help` output. This patch adds that.
 2018-11-16 22:15:03 +01:00
Todd 491929c212
Include additional files in sdists 
In particular this includes files needed for running the tests, as well as the changelog.
 2018-11-16 13:38:19 -05:00
Mikhail Korobov dc65e7527d
Merge pull request #3468 from ilhaoni/feature/update-docs-intro-tutorial 
[MRG+1] Update Scrapy Tutorial docs
 2018-11-06 03:33:09 +05:00
Immanuella Lim 6c98010f11 Remove 'Dive into Python3' reference 2018-11-04 16:04:45 +08:00
Eugenio Lacuesta f97e3e90f2 Use collections.deque instead of list to store methods 2018-10-29 12:40:20 -03:00
Immanuella Lim c9b5bd6ad7 Remove ad link Dive Into Python3 from tutorial docs 2018-10-18 02:22:32 +08:00
Daniel Graña 44f8e28b3c Fix headings' underlines 2018-10-16 19:53:20 -03:00
Daniel Graña 06f2db7fd1
Merge pull request #3445 from jfflisikowski/3097-item-urls-not-defined-in-example 
[MRG+1] Updating debug example file (#3097)
 2018-10-16 19:51:32 -03:00
Daniel Graña 553b3a1c12
Merge pull request #3465 from hcoura/telnet-auth 
Add Telnet console authentication docs
 2018-10-16 15:08:34 -03:00
Henrique Coura 92b7955d75 Add Telnet console authentication docs 2018-10-16 14:50:00 -03:00
Eugenio Lacuesta c602e69732 Force Travis build 2018-10-11 13:33:41 -03:00
Eugenio Lacuesta e0360e5223 Add tests for MutableChain 2018-10-11 11:55:13 -03:00
Eugenio Lacuesta 15f0a890ee Assign processing methods to a variable before iterating 2018-10-11 11:34:59 -03:00
Eugenio Lacuesta a05eaeed73 Simplify MutableChain 2018-10-11 11:31:51 -03:00
Eugenio Lacuesta 58f5565357 Move MutableChain to scrapy.utils.python 2018-10-11 11:23:12 -03:00
Eugenio Lacuesta 2396356a82 Merge branch 'master' into process_spider_exception_generator 2018-10-10 11:37:43 -03:00
jfflisikowski edaf74bfae Correct the unclear comments by adding <# < processing code not shown > 2018-10-02 19:48:48 +02:00
Daniel Graña d80f9ed725
Merge pull request #3429 from hcoura/telnet-auth 
Randomly generate telnet credentials by default
 2018-09-26 14:00:46 -03:00
Henrique Coura 441e1e750f Style changes 2018-09-26 13:28:34 -03:00
Henrique Coura 5f9931d2ad do not log username 2018-09-26 13:07:04 -03:00
Henrique Coura e57a629efc Generate only password, encode username/password only on login 2018-09-26 11:54:57 -03:00
Henrique Coura 37cfb49805 Randomly generate telnet credentials by default 2018-09-24 16:42:49 -03:00
Daniel Graña bafd174a9f
Merge pull request #3390 from scrapy/parsel-1.5 
[MRG+1] update Scrapy to use parsel 1.5
 2018-09-18 13:07:41 -03:00
Mikhail Korobov ffbd33edac DOC mention gotcha with `foo::text` selector and empty `foo` elements 
also, move "Selecting attributes" reference closer to `a::atr(href)` example
 2018-09-18 05:03:35 +05:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments 
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
 2018-09-18 05:02:17 +05:00
Mikhail Korobov 9db21e5502 DOC fix remove_namespaces example 
See https://github.com/scrapy/parsel/pull/119
 2018-09-15 02:43:37 +05:00
Mikhail Korobov dc95ecbe25 DOC use autodocs for selectors; document more methods and attributes; suggest get/getall 2018-09-12 18:36:25 +05:00
Mikhail Korobov 7fdfdb7fa2 DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction 2018-09-12 17:57:27 +05:00
Mikhail Korobov bdcc045f62 DOC switch from .extract to get/getall API in docs 
Also, response.urljoin is added in a few places, for robustness.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov afce9716fa DOC mention .attrib in the tutorial 2018-09-12 17:57:27 +05:00
Mikhail Korobov 12e42bbe06 switch SgmlLinkExtractor to .getall 2018-09-12 17:57:27 +05:00
Mikhail Korobov 460f0f0451 [backwards incompatible] switch ItemLoader from .extract to .getall. 
This change is backwards incompatible if ItemLoader is used with a custom Selector
subclass which overrides .extract without overriding .getall.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov 8c29be606c update spider templates to use .get 2018-09-12 17:57:27 +05:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Mikhail Korobov 2c48d156db DOC cleanup references in tutorials: 
* remove unused link
* fix ReST syntax
* fix a link to regular expression docs
 2018-09-12 17:57:27 +05:00
Mikhail Korobov 09fd6c2a81 DOC unlink Firefox & Firebug sections from the tutorial for now. 
See https://github.com/scrapy/scrapy/issues/3373 and https://github.com/scrapy/scrapy/issues/3372 for motivation.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov d32c4deaa9 DOC update Scrapy selectors tutorial to match parsel's tutorial better 2018-09-12 17:57:27 +05:00
Mikhail Korobov ca27010d4f DOC .extract_first() -> .get() 2018-09-12 17:57:27 +05:00
Mikhail Korobov 395d9d033a add pytest temp files to gitignore 2018-09-12 17:57:27 +05:00
Mikhail Korobov 0ccead6681 DOC more Python 3 in examples 2018-09-12 17:57:27 +05:00
Mikhail Korobov 25ac4691b4 require parsel 1.5+ 2018-09-12 17:57:27 +05:00
Daniel Graña 7223978594
Merge pull request #3381 from StasDeep/fix/issue-3380 
[MRG+1] Use dont_filter=True for contracts requests
 2018-09-06 11:41:02 -03:00
Daniel Graña 2aae514f99
Merge branch 'master' into fix/issue-3380 2018-09-05 12:25:07 -03:00
Daniel Graña 5176765589
Merge pull request #3377 from StasDeep/feature/issue-3364 
Add handling of errors in contract methods
 2018-09-05 12:24:50 -03:00
Daniel Graña d7b98a3714
Merge branch 'master' into feature/issue-3364 2018-09-05 11:26:59 -03:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382 
[MRG+1] Add ability to use FormRequest in contracts
 2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00