RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,976 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

1144 Commits

Author SHA1 Message Date
Adrián Chaves e3df6be360
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:28:52 +00:00
Adrián Chaves 46b9ab0c58
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:28:36 +00:00
Mikhail Korobov 845bae6637
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-27 08:49:19 +00:00
Vostretsov Nikita 103fdc40c1 Merge remote-tracking branch 'upstream/master' into round-robin-scheduler-tested 2019-03-25 16:16:42 +03:00
Vostretsov Nikita 8afffb7234 Tests Cleanup 
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 989bba6cb3 Revert "new signal" 
This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739.
remove tests
Revert "emit new signal"

This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf.
Revert "documentation for new signal"

This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 83eb537645 assorted cleanups: comments, docstrings, etc 
scheduler cleanup

Scheduler no longer converts requests to dicts; PriorityQueue
instances always work with Request instances; converting Requests
to dicts is now Priority Queue responsibility.
minor cleanup
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 3b1db71dac New signal 
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 8583c033ef
Merge pull request #3694 from noviluni/master 
[MRG+1] Fix numeration
 2019-03-23 00:23:06 +05:00
Mikhail Korobov 729580dae2
Merge pull request #3671 from Gallaecio/item-deepcopy 
Implement Item.deepcopy()
 2019-03-22 22:56:39 +05:00
Adrián Chaves 7c148fce5a Implement Item.deepcopy() 2019-03-22 17:07:10 +01:00
Konstantin Lopuhin af2b666595
Merge pull request #3505 from kasun/json_request 
[MRG+2] Request subclass for json requests #3504
 2019-03-22 18:25:54 +03:00
Vostretsov Nikita 338b78d796 Add documentation 
add section to broad-crawl topic
reword in accord with broad-crawl topic
add documentation for new priority queue
 2019-03-22 15:40:37 +03:00
Adrián Chaves 9c314800e4 Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2019-03-22 15:40:36 +03:00
Marc Hernández 70aa5b1333
Fix numeration 2019-03-20 15:32:20 +01:00
Victor Torres fda1d04b81
Merge branch 'master' into feed-storage-s3-acl 2019-03-15 13:26:45 -03:00
Adrián Chaves d346b8cb0f
Merge pull request #3626 from Gallaecio/middlewares-from-crawler 
[MRG+1] Document that the main entry point of downloader and spider middlewar…
 2019-03-15 08:58:35 +01:00
Adrián Chaves 1f7413dc88
Merge pull request #3635 from matthieucham/feature-filteringlinkextractor-restrict-text 
[MRG+1] Feature filteringlinkextractor restrict text
 2019-03-15 07:58:28 +01:00
Mikhail Korobov 5dc94db847
Merge branch 'master' into backward 2019-03-14 22:21:09 +05:00
Mikhail Korobov 54d106033f
Merge pull request #3649 from Gallaecio/inline-code-formatting 
[MRG+1] Switch from ` to `` where inline code formatting is desired
 2019-03-14 22:18:57 +05:00
Mikhail Korobov 7c54479a0e
Merge pull request #3662 from anubhavp28/link-patch 
Fix a link inside docs
 2019-03-14 22:18:04 +05:00
Mikhail Korobov c3b6755671
Merge pull request #3668 from Gallaecio/retry-complete-docs 
Remove the unexisting retry_complete signal from the documentation
 2019-03-14 22:16:28 +05:00
Mikhail Korobov f439553844
Merge pull request #3670 from Gallaecio/depth-first-doc 
Clarify the documentation of DEPTH_PRIORITY further
 2019-03-14 22:10:42 +05:00
Adrián Chaves 91aec8b3bb
Update developer-tools.rst 
Fixes #3674
 2019-03-08 18:19:30 +01:00
Adrián Chaves e108e3adbf Clarify the documentation of DEPTH_PRIORITY further 2019-03-08 15:13:11 +01:00
Adrián Chaves 4ef38d925e Remove the unexisting retry_complete signal from the documentation 2019-03-08 14:21:00 +01:00
Anubhav Patel 184def1060 fix a link inside docs 2019-03-07 00:09:10 +05:30
Adrián Chaves 75d6f56c8a Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
Adrián Chaves 858f5be747 backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
Matthieu Grandrie e3b15252c8 New constructor arg *restrict_text* for FilteringLinkExtractor. 
Same as allow and deny args, it holds a string, a regex or an iterable of. Links whose text don't match one of the regex are filtered out.
DOC restrict_text in LxmlLinkExtractor
 2019-02-28 17:21:17 +01:00
Adrián Chaves 50bf4c60c4 Document that the main entry point of downloader and spider middlewares is from_crawler() 2019-02-13 17:39:20 +01:00
Victor Torres 984e706fd2 using blank string instead of None as default value as proposed by @kmike 2019-02-12 12:26:57 -02:00
Victor Torres e25b9a2323 calling it feeds instead of objects 2019-02-08 15:51:23 -02:00
Victor Torres 7b83ed7c5e remove typo 2019-02-08 15:51:23 -02:00
Victor Torres e0f34be383 update docs 2019-02-08 15:51:23 -02:00
Adrián Chaves 38af090f4d Indicate that users must implement their own authentication result check 
The example of form-based login could lead some users to think its authentication result
check was final. See https://stackoverflow.com/a/54410966/939364

This change should make it more obvious that users are expected to implement their
own logic to check whether authentication worked or not.
 2019-02-04 11:17:58 +01:00
Daniel Graña 65d631329a Be consistent with domain used for links to documentation website 2019-01-31 01:28:53 -03:00
Daniel Graña b5026b842c
Merge pull request #3544 from joaquingx/fix-item-pipeline-x 
Fix item-pipeline example
 2019-01-28 10:44:19 -03:00
Harry Moreno 8fca98616a fix grammar 2019-01-28 10:19:31 -03:00
Harry Moreno b828b5f8c8
fix grammar 2019-01-26 18:39:05 -05:00
kasun Herath f3813e376c Merge remote-tracking branch 'origin/master' into json_request 2019-01-14 23:03:21 +05:30
Joaquin Garmendia Cabrera e1f8b55ba0
Improve syntax for readability 2018-12-28 16:53:12 -05:00
Victor Torres 5a824c906c using shorter import version and moving datetime import to the beginning of the code snippet 2018-12-27 18:34:41 -03:00
Victor Torres bfbcf52e9d fix SitemapSpider import 2018-12-27 18:12:31 -03:00
Victor Torres b68308779a improving docs 2018-12-27 17:37:59 -03:00
Victor Torres e1597f7c42 improve readability 2018-12-27 11:40:14 -03:00
Victor Torres 10f46bca54 documenting sitemap entries as suggested by @kmike 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres 672385a371 using a method definition instead of a None attribute 2018-12-27 11:40:14 -03:00
Victor Torres a5e1b7bb47 add sitemap_filter attribute to SitemapSpider class 
it makes it possible to filter sitemap urls by any available attribute

for example, you can filter urls with lastmod greater than a given datetime

it can be helpful when the url loc itself does not aggregate that information
 2018-12-27 11:40:14 -03:00