RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,828 Commits 42 Branches 132 Tags 161 MiB
Commit Graph

1086 Commits

Author SHA1 Message Date
Vostretsov Nikita 989bba6cb3 Revert "new signal" 
This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739.
remove tests
Revert "emit new signal"

This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf.
Revert "documentation for new signal"

This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 83eb537645 assorted cleanups: comments, docstrings, etc 
scheduler cleanup

Scheduler no longer converts requests to dicts; PriorityQueue
instances always work with Request instances; converting Requests
to dicts is now Priority Queue responsibility.
minor cleanup
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 3b1db71dac New signal 
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 338b78d796 Add documentation 
add section to broad-crawl topic
reword in accord with broad-crawl topic
add documentation for new priority queue
 2019-03-22 15:40:37 +03:00
Adrián Chaves 9c314800e4 Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2019-03-22 15:40:36 +03:00
jfflisikowski edaf74bfae Correct the unclear comments by adding <# < processing code not shown > 2018-10-02 19:48:48 +02:00
Mikhail Korobov ffbd33edac DOC mention gotcha with `foo::text` selector and empty `foo` elements 
also, move "Selecting attributes" reference closer to `a::atr(href)` example
 2018-09-18 05:03:35 +05:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments 
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
 2018-09-18 05:02:17 +05:00
Mikhail Korobov 9db21e5502 DOC fix remove_namespaces example 
See https://github.com/scrapy/parsel/pull/119
 2018-09-15 02:43:37 +05:00
Mikhail Korobov dc95ecbe25 DOC use autodocs for selectors; document more methods and attributes; suggest get/getall 2018-09-12 18:36:25 +05:00
Mikhail Korobov 7fdfdb7fa2 DOC reorganize selectors tutorial, port more topics from parsel docs, adjust wording in the introduction 2018-09-12 17:57:27 +05:00
Mikhail Korobov bdcc045f62 DOC switch from .extract to get/getall API in docs 
Also, response.urljoin is added in a few places, for robustness.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov 2c48d156db DOC cleanup references in tutorials: 
* remove unused link
* fix ReST syntax
* fix a link to regular expression docs
 2018-09-12 17:57:27 +05:00
Mikhail Korobov d32c4deaa9 DOC update Scrapy selectors tutorial to match parsel's tutorial better 2018-09-12 17:57:27 +05:00
Mikhail Korobov ca27010d4f DOC .extract_first() -> .get() 2018-09-12 17:57:27 +05:00
Mikhail Korobov 0ccead6681 DOC more Python 3 in examples 2018-09-12 17:57:27 +05:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382 
[MRG+1] Add ability to use FormRequest in contracts
 2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Vostretsov Nikita c02cfa574c remove comma 2018-08-29 11:21:55 +00:00
Raphael Wuillemier 79de3d569a Removed obsolete firebug-images 2018-08-23 16:19:13 +02:00
testingcan e98e7f8506
Added missing curly brace 2018-08-23 14:50:49 +02:00
Raphael Wuillemier 4d3aaabbca Updated code, added code snippets and improved readability 2018-08-23 12:40:31 +02:00
testingcan 3a71e7dbce
Increased length of "=" 2018-08-22 16:57:51 +02:00
Raphael Wuillemier af555cab23 Added general guide for developer tools instead of Firefox and Firebug-sections 2018-08-22 14:15:53 +02:00
Stas Glubokiy 9ab85fe788 Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3382 2018-08-19 17:21:28 +03:00
Vostretsov Nikita 597b8a97ad documentation for new signal 2018-08-17 14:39:42 +00:00
Stas Glubokiy b4b1e48343 Add ability to use FormRequest in contracts 2018-08-11 22:18:43 +03:00
Valdir Stumm Junior c61e8a617f
[doc] update default RETRY_HTTP_CODES 2018-07-13 11:55:16 -07:00
Daniel Graña f8f5f463e6
Merge pull request #3253 from rpkilby/depth-stats 
[MRG+1] Update depth middleware stats (fixes #3245)
 2018-07-11 12:00:08 -03:00
chainly 74ce156154 add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
Daniel Graña 594a441eb5
Merge pull request #3294 from rennerocha/feed-exporter-docs 
[MRG+1] Improve docs of S3 Storage to make Python version more explicit
 2018-06-27 11:16:07 -03:00
Renne Rocha e2bb218e9a Include Python version indication to each required library used in S3 storage 2018-06-13 18:11:43 -03:00
mugayoshi d4511667fb Update debugging memory leaks section in the docs 
Add Python3 tools description.
 2018-06-09 18:17:11 +09:00
Mikhail Korobov 847b50ce9c
Merge pull request #3201 from grammy-jiang/master 
[MRG+1] fix a mistake in topic spider-middleware.rst
 2018-06-02 01:09:59 +05:00
Mikhail Korobov 13b15dc9a8
Merge pull request #3279 from lewoudar/patch-1 
Update spiders.rst
 2018-06-01 22:20:41 +05:00
Kevin Tewouda ffa7bede17
Update spiders.rst 
I changed URLs to :class:`~scrapy.http.Request` in start_urls explanation of the default spider
 2018-05-30 06:33:18 +02:00
Ryan P Kilby 2dfc5d128b Update DEPTH_STATS refs to DEPTH_STATS_VERBOSE 2018-05-17 11:00:21 -04:00
Konstantin Lopuhin c4f096d3a5
Merge pull request #3224 from lucywang000/better-processors-doc 
[MRG+1] Improve document about functions as processors
 2018-04-26 09:36:39 +03:00
Konstantin Lopuhin f36e1b53d4
Merge pull request #3199 from rhoboro/gcs_acl 
[MRG+1] FilesPipeline supports ACL for Google Cloud Storage
 2018-04-25 19:42:19 +03:00
Lucy Wang 57b0e6b695 improve document about functions as processors 2018-04-19 13:35:46 +08:00
rhoboro 6ef6585b5a update docs 2018-04-13 19:06:29 +09:00
rhoboro 464973489e Using bucket's default object ACL 2018-04-13 12:06:39 +09:00
grammy-jiang cb76b88331 fix a mistake in topic spider-middleware.rst 2018-04-04 05:56:05 -04:00
rhoboro 74a9c65290 update docs for support gcs acl 2018-04-03 18:20:37 +09:00
Steven Almeroth c6d20bdd82 Doc: update wording for COOKIES_ENABLED 2018-03-27 16:21:07 -04:00
Daniel Graña 2dee191374
Merge branch 'master' into extending-s3-files-store 2017-12-31 16:44:38 -03:00
Raphael a1cc5a63d3 Add mention to dont_merge_cookies in CookiesMiddlewares docs (#2999) (#3030) 
Add mention to dont_merge_cookies in CookiesMiddlewares docs (#2999)
 2017-12-27 17:54:17 -03:00
Daniel Graña 3cf0332ec3
Merge pull request #2957 from ScrapingLab/add_meta_json_to_parse_command 
[MRG+1] Scrapy Command: add --meta/-m to the "parse" command to pass additional meta data into the request
 2017-11-29 16:26:48 -03:00
IAlwaysBeCoding 6af323d7c8
Fix spelling mistake on scrapy parse command docs 
Fixed spelling mistake from "will be pass" to "will be passed"
 2017-11-26 00:24:52 +01:00