RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,312 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

1859 Commits

Author SHA1 Message Date
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mikhail Korobov e5f12faf5f
Merge pull request #3703 from Gallaecio/ajax-docs 
Add a topic about reaching data that selectors cannot reach
 2019-06-25 23:11:59 +03:00
Adrián Chaves d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Adrián Chaves 8a3b15eb91 Document how to select dynamically-loaded content 2019-06-24 16:06:21 +02:00
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
Daniel Graña 8a022ac6ea
Merge pull request #3810 from Gallaecio/documentation-coverage 
Skip scrapy.contracts private APIs in the documentation coverage report
 2019-06-13 17:16:07 -03:00
Adrián Chaves c7ba72b5dc Skip scrapy.contracts private APIs in the documentation coverage report 2019-06-07 13:14:33 +02:00
Artem Kuchumov c81e15ed6e
Tutorial: scrapy shell example should say "text" not "title" (#3807) 
Tutorial: scrapy shell example should say "text" not "title"
 2019-06-05 13:15:23 +05:00
Mikhail Korobov f755507d68
Merge pull request #3567 from Gallaecio/documentation-coverage 
Add a Sphinx extension to generate documentation coverage information
 2019-06-04 18:47:24 +05:00
Adrián Chaves c7b5ad0e20 Add a Sphinx extension to generate documentation coverage information 2019-06-04 15:03:25 +02:00
Adrián Chaves ea209a0ea7 Fix module double indexing issues in the documentation 2019-06-04 14:35:27 +02:00
Mikhail Korobov 890e7fbd5b
Merge pull request #3791 from barraponto/patch-3 
[MRG+1] Fix documentation for spiderloader
 2019-05-28 19:16:12 +01:00
Mikhail Korobov 8d50384210
Merge pull request #3797 from mar-heaven/master 
[MRG+1] remove a "is"
 2019-05-28 19:12:04 +01:00
mar-heaven 18f01ea6ec
remove a "is" 
When I translated in Chinese, I found a needless "is"
 2019-05-27 17:15:30 +08:00
Anubhav Patel da82ede8a0
describe method as a command 2019-05-25 17:19:10 +05:30
Capi Etheriel 7d36fa7435
Fix documentation for spiderloader 2019-05-24 10:32:55 -03:00
Mikhail Korobov a3d38041e2
Merge pull request #3764 from Jeffallan/patch-1 
[MRG+1] Update telnetconsole.rst
 2019-05-09 20:59:30 +05:00
Mikhail Korobov 137a3d8146
Merge pull request #3762 from Vandenn/update-logging-docs 
[MRG+1] doc: update configure_logging docs to discourage use with CrawlerProcess
 2019-05-09 20:59:20 +05:00
Adrián Chaves 611249bb7f Implement the METAREFRESH_IGNORE_TAGS setting 2019-05-08 12:52:29 +02:00
Aditya 3a7850fa15
Update contributing.rst 2019-05-05 18:45:40 +05:30
Jeffallan 5814344adf
Update telnetconsole.rst 
Change spelling of bellow to below.
 2019-05-04 14:15:47 -05:00
Adrián Chaves bc1a929213
Improve the documentation about detecting check runs 2019-05-03 14:42:12 +02:00
Matthijs Vos 8bd207a2f6 Add documentation 2019-05-03 13:50:24 +02:00
Vandenn 122ca62119 doc: update configure_logging docs to discourage use with CrawlerProcess 2019-05-03 16:39:26 +08:00
Mikhail Korobov 6e49c379a8
Merge pull request #3748 from Gallaecio/docstring-conventions 
Cover PEP 257 in the documentation policies
 2019-04-17 13:22:11 +05:00
Mikhail Korobov 5b667b6156
Merge pull request #3707 from Gallaecio/pytest-xdist 
Use pytest-xdist
 2019-04-17 13:20:37 +05:00
Adrián Chaves 5a6fb3daa6 Use pytest-xdist 2019-04-17 09:02:34 +02:00
Adrián Chaves fbb42fe14e Cover PEP 257 in the documentation policies 2019-04-17 08:25:22 +02:00
Adrián Chaves 92fa657f9e
Merge pull request #3730 from ankostis/doc_logstats_delay 
[MRG+1] doc: document LOGSTATS_INTERVAL setting
 2019-04-11 17:34:45 +02:00
Mikhail Korobov 9280185054
Merge pull request #3520 from whalebot-helmsman/round-robin-scheduler-tested 
[MRG+1] Downloader-aware Priority Queue for Scrapy
 2019-04-07 00:06:08 +05:00
Kostis Anagnostopoulos a8f83ab967 doc: document LOGSTATS_INTERVAL setting 2019-04-06 14:58:32 +02:00
float13 a101d5fe5c text edit - delete 2 extra words 2019-04-05 01:12:20 -04:00
float13 77e3695686 grammar fix - add apostrophe-s to browser 2019-04-05 01:04:59 -04:00
float13 d711ecfc18 grammar fix - delete extra word "shell" 2019-04-05 00:56:51 -04:00
float13 6336e1d1f3 grammar fix - delete unneeded apostrophe in "lets" 2019-04-05 00:54:46 -04:00
Eugenio Lacuesta 7a38623cec [Docs] Clarify comment about meta dictionary 2019-04-01 17:09:49 -03:00
Eugenio Lacuesta 8ebbc731b2 [Docs] Rephrase Rule docs 2019-04-01 16:15:03 -03:00
Eugenio Lacuesta 07ff9248a5 [Docs] CrawlSpider: add note about link text 2019-04-01 12:31:26 -03:00
Konstantin Lopuhin b5c552d17f
Merge pull request #2061 from elacuesta/process_spider_exception_generator 
[MRG+1] process_spider_exception on generators
 2019-04-01 10:43:19 +03:00
Eugenio Lacuesta 294ef51bb2 parse command: update docs about passing callback keyword arguments 2019-03-29 16:12:55 -03:00
Eugenio Lacuesta ccb56a317e Update docs about cb_kwargs and meta 2019-03-29 14:13:50 -03:00
Vostretsov Nikita 1c6733454e added underlines 2019-03-29 10:44:55 +00:00
Vostretsov Nikita ef743983a9 change wording 2019-03-29 10:38:13 +00:00
Adrián Chaves 1ee99e1f42
Update docs/topics/settings.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:29:15 +00:00
Adrián Chaves bd228f1d96
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:29:04 +00:00
Adrián Chaves e3df6be360
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:28:52 +00:00
Adrián Chaves 46b9ab0c58
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:28:36 +00:00
Eugenio Lacuesta 8fb077694f Request.cb_kwargs: Update docs 2019-03-28 16:32:35 -03:00
Eugenio Lacuesta e8af6331b5 Add cb_kwargs option to the parse command 2019-03-28 15:03:40 -03:00
Eugenio Lacuesta 3efe3bea1c Update docs about cb_kwargs and meta 2019-03-28 14:16:03 -03:00
Mikhail Korobov ec719f55e7
Merge pull request #3682 from elacuesta/rule_process_request_response_parameter 
[MRG+1] Rule.process_request: access Response object
 2019-03-28 01:39:54 +05:00
Eugenio Lacuesta c43a23152b Merge remote-tracking branch 'upstream/master' into callback_kwargs 2019-03-27 14:42:41 -03:00
Eugenio Lacuesta 8528f5065f [Doc] Update cb_kwargs example 2019-03-27 14:42:26 -03:00
Anubhav Patel 9c9bca4e1c make suggested changes. 2019-03-27 18:29:48 +05:30
Mikhail Korobov 845bae6637
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-27 08:49:19 +00:00
Mikhail Korobov 820adb69c0
Merge pull request #3701 from Gallaecio/documentation-policy-api 
Update the documentation policies: Ask to use docstrings to document …
 2019-03-27 12:25:38 +05:00
Adrián Chaves ce837b0f74 Update the documentation policies: Ask to use docstrings to document API members 2019-03-27 07:09:18 +01:00
Maram Sumanth 2fd8b7c28c [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
Adrián Chaves 431f18a9a1 Document FilesPipeline.file_path and ImagesPipeline.file_path 2019-03-26 12:15:19 +01:00
Vostretsov Nikita 103fdc40c1 Merge remote-tracking branch 'upstream/master' into round-robin-scheduler-tested 2019-03-25 16:16:42 +03:00
Vostretsov Nikita 8afffb7234 Tests Cleanup 
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 989bba6cb3 Revert "new signal" 
This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739.
remove tests
Revert "emit new signal"

This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf.
Revert "documentation for new signal"

This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 83eb537645 assorted cleanups: comments, docstrings, etc 
scheduler cleanup

Scheduler no longer converts requests to dicts; PriorityQueue
instances always work with Request instances; converting Requests
to dicts is now Priority Queue responsibility.
minor cleanup
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 3b1db71dac New signal 
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
 2019-03-25 16:08:40 +03:00
Eugenio Lacuesta 174ba3cc56 Rule.process_request: update docs 2019-03-22 19:16:18 -03:00
Mikhail Korobov 8583c033ef
Merge pull request #3694 from noviluni/master 
[MRG+1] Fix numeration
 2019-03-23 00:23:06 +05:00
Mikhail Korobov 729580dae2
Merge pull request #3671 from Gallaecio/item-deepcopy 
Implement Item.deepcopy()
 2019-03-22 22:56:39 +05:00
Adrián Chaves 7c148fce5a Implement Item.deepcopy() 2019-03-22 17:07:10 +01:00
Konstantin Lopuhin af2b666595
Merge pull request #3505 from kasun/json_request 
[MRG+2] Request subclass for json requests #3504
 2019-03-22 18:25:54 +03:00
Vostretsov Nikita 338b78d796 Add documentation 
add section to broad-crawl topic
reword in accord with broad-crawl topic
add documentation for new priority queue
 2019-03-22 15:40:37 +03:00
Adrián Chaves 9c314800e4 Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2019-03-22 15:40:36 +03:00
Adrián Chaves afdb69ea6d Add a troubleshooting section to the installation instructions 
Its initial content covers the workaround for #2473.
 2019-03-22 15:40:36 +03:00
Marc Hernández 70aa5b1333
Fix numeration 2019-03-20 15:32:20 +01:00
Anubhav Patel 044318920a doc for creating custom cache storage backend. 2019-03-17 16:54:28 +05:30
Eugenio Lacuesta 6760bca74b Rename Request.kwargs to Request.cb_kwargs 2019-03-15 22:42:29 +00:00
Victor Torres fda1d04b81
Merge branch 'master' into feed-storage-s3-acl 2019-03-15 13:26:45 -03:00
Adrián Chaves d346b8cb0f
Merge pull request #3626 from Gallaecio/middlewares-from-crawler 
[MRG+1] Document that the main entry point of downloader and spider middlewar…
 2019-03-15 08:58:35 +01:00
Adrián Chaves 1f7413dc88
Merge pull request #3635 from matthieucham/feature-filteringlinkextractor-restrict-text 
[MRG+1] Feature filteringlinkextractor restrict text
 2019-03-15 07:58:28 +01:00
Adrián Chaves a2ff647aac Remove docs/topics/ubuntu.rst 2019-03-15 06:36:15 +01:00
Anubhav Patel 66a502db8c
Merge branch 'master' into logFormatter-doc-patch 2019-03-15 00:28:54 +05:30
Mikhail Korobov 5dc94db847
Merge branch 'master' into backward 2019-03-14 22:21:09 +05:00
Mikhail Korobov 54d106033f
Merge pull request #3649 from Gallaecio/inline-code-formatting 
[MRG+1] Switch from ` to `` where inline code formatting is desired
 2019-03-14 22:18:57 +05:00
Mikhail Korobov 7c54479a0e
Merge pull request #3662 from anubhavp28/link-patch 
Fix a link inside docs
 2019-03-14 22:18:04 +05:00
Mikhail Korobov c3b6755671
Merge pull request #3668 from Gallaecio/retry-complete-docs 
Remove the unexisting retry_complete signal from the documentation
 2019-03-14 22:16:28 +05:00
Mikhail Korobov f439553844
Merge pull request #3670 from Gallaecio/depth-first-doc 
Clarify the documentation of DEPTH_PRIORITY further
 2019-03-14 22:10:42 +05:00
Mikhail Korobov 9c2c312451
Merge pull request #3673 from Gallaecio/friendlier-documentation-1 
Use the description from README.rst on index.rst
 2019-03-14 21:53:44 +05:00
Eugenio Lacuesta b30ca379b6 Rule.process_request: docs 2019-03-13 11:02:51 +00:00
Anubhav Patel 82049e9c41 make suggested changes. 2019-03-10 20:17:07 +05:30
Adrián Chaves 91aec8b3bb
Update developer-tools.rst 
Fixes #3674
 2019-03-08 18:19:30 +01:00
Adrián Chaves b1063d9b2c Use the description from README.rst on index.rst 2019-03-08 17:22:49 +01:00
Adrián Chaves e108e3adbf Clarify the documentation of DEPTH_PRIORITY further 2019-03-08 15:13:11 +01:00
Adrián Chaves 4ef38d925e Remove the unexisting retry_complete signal from the documentation 2019-03-08 14:21:00 +01:00
Anubhav Patel 924b67437b move api docs to source code 2019-03-07 16:40:59 +05:30
Anubhav Patel 184def1060 fix a link inside docs 2019-03-07 00:09:10 +05:30
Anubhav Patel 82d239f3b1 docs for scrapy.logformatter 2019-03-06 12:08:09 +05:30
Adrián Chaves 75d6f56c8a Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
Adrián Chaves 858f5be747 backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
Matthieu Grandrie e3b15252c8 New constructor arg *restrict_text* for FilteringLinkExtractor. 
Same as allow and deny args, it holds a string, a regex or an iterable of. Links whose text don't match one of the regex are filtered out.
DOC restrict_text in LxmlLinkExtractor
 2019-02-28 17:21:17 +01:00
Adrián Chaves 50bf4c60c4 Document that the main entry point of downloader and spider middlewares is from_crawler() 2019-02-13 17:39:20 +01:00