RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

1904 Commits

Author SHA1 Message Date
Ricardo Amendoeira b247fa9982
Include loading settings in `Running multiple spiders in the same process` section 
The example in the documentation doesn't take into account the project settings
 2021-03-29 01:48:28 +01:00
Andrey Rakhmatullin 7b515a0485 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-03-25 22:13:49 +05:00
Andrey Rakhmatullin a97dc55c71 Add/improve docs. 2021-03-25 20:49:12 +05:00
Pratik Mahankal f0e1a33225
Sort the list of Request.meta alphabetically #5061 (#5065) 2021-03-23 22:16:50 +05:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
Mikhail Korobov ab037ab7c9
Merge pull request #5016 from deepang17/scrapy-4996 
DOCS: Cover scrapy-bench in the documentation
 2021-03-12 04:18:56 +05:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
deepang17 3d88ac605b FIX: Updated benchmarking.rst 2021-03-09 17:19:34 +05:30
deepang17 36f1dbf665 DOCS: Covered scrapy-bench 2021-03-02 22:12:44 +05:30
James McKinney 3c5668d0db
docs: Clarify there's one extension instance per spider 2021-03-01 22:00:33 -05:00
deepang17 b25616d107 DOCS: Cover scrapy-bench in the documentation 2021-02-28 16:26:46 +05:30
deepang17 7a54580679 DOCS:Cover scrapy-bench in the documentation 2021-02-28 15:33:09 +05:30
Adrián Chaves f689615e8d Close files in the PerYearXmlExportPipeline documentation example 2021-02-24 12:54:56 +01:00
Adrián Chaves 6ab990181c Document get_retry_requests 2021-02-22 14:48:03 +01:00
Andrey Rakhmatullin 58f8481301 Update docs. 2021-02-05 16:21:56 +05:00
Adrián Chaves 4c801551fa Document that the bytes_received signal is not yet implemented for HTTP/2 2021-02-03 21:11:46 +01:00
Adrián Chaves 1773eaf5dc Move lists closer to their introducing paragraph 2021-02-03 11:43:18 +01:00
Adrián Chaves 1a7bde0d8e Document that HTTP/2 server pushes are ignored 2021-02-03 10:55:11 +01:00
Adrián Chaves c8d8b180bf Merge remote-tracking branch 'upstream/master' into http2 2021-02-03 09:37:35 +01:00
Adrián Chaves 2ce8e0c742 Document the (hard-coded) maximum HTTP/2 frame size accepted from servers 2021-02-03 09:09:53 +01:00
Adrián Chaves f30f53b3cc Scrapinghub → Zyte 2021-02-02 15:03:20 +01:00
Eugenio Lacuesta e494a3f733
protocol attribute for h2 responses 2020-12-31 11:50:15 -03:00
Eugenio Lacuesta d698b5147b
Merge branch 'master' into http2 2020-12-31 11:13:25 -03:00
Adrián Chaves d0af008608
Merge pull request #4878 from elacuesta/response-protocol-attribute 
Response.protocol attribute
 2020-12-16 15:27:40 +01:00
etimoz 95d39d5cb4 removed wrong super argument in overriding serialize_fields code example 2020-11-29 13:24:04 +01:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
Eugenio Lacuesta 5e9a99e6a1
Reponse.protocol as string 2020-11-11 11:15:29 -03:00
Eugenio Lacuesta 2242412556
Docs: placeholder for versionadded directive 2020-11-11 10:50:54 -03:00
Eugenio Lacuesta 61d089485c
Docs: sort versionadded directives 2020-11-11 01:31:21 -03:00
Eugenio Lacuesta 587b4dd71f
Docs for the Response.protocol attribute 2020-11-11 01:22:33 -03:00
Eugenio Lacuesta 114229eb4a
Docs: add a note about asyncio.set_event_loop 2020-11-06 13:29:14 -03:00
Adrián Chaves 015c82b974
Scrapy 2.4 release notes (#4808) 2020-10-11 22:09:45 +02:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
OfirD1 1a597d5e3d
moved the sentence about processing pending requests when a spider is closed onto a generic note. 2020-10-06 21:54:42 +03:00
Eugenio Lacuesta 137c8ba6ee
Docs: mention limitation about Cookie header 2020-10-06 10:50:17 -03:00
P. Chen da3171d4f7
Using the `zstandard` package than `zstd` for supporting frames both with and without the content size info 
See also: https://github.com/sergey-dryabzhinsky/python-zstd/issues/53
 2020-10-05 23:18:58 +01:00
P. Chen c6c3f2ce66
Updating the doc entry for the HTTP compress downloader middleware on zstd 2020-10-05 21:10:40 +01:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Andrey Rahmatullin 5a38639359
Merge pull request #4809 from madeny/patch-1 
Correct some typos
 2020-09-28 18:26:43 +05:00
madeny eff96038c7
Correct some typos 
This won't be an issue if **your** spider doesn't rely on cookies.
 2020-09-26 22:50:38 +02:00
Mikhail Korobov 85e13aff74
Merge pull request #4782 from sripriyesha/patch-1 
add mention of FTP server storage in media storage intro
 2020-09-15 01:17:46 +05:00
Adrián Chaves 4d6359df2d Mark HTTP/2 as experimental 2020-09-11 13:51:05 +02:00
KAILASA's Sri Nithya Priyeshananda 7a83474cc5
add mention of FTP server storage in media storage intro 
At the beginning of this doc, in "Specifying where to store the media" feature details, FTP server storage mention was missing
 2020-09-08 17:16:31 +02:00
Ilia Sergunin d10464ca96
Update docs/topics/feed-exports.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-09-01 10:13:40 +03:00
Adrián Chaves 8a3ba34a75 Merge remote-tracking branch 'upstream/master' into http2 2020-08-31 16:43:32 +02:00
Jose Galdos a41c205928 Update httpstatus documentation. 2020-08-31 08:58:43 -05:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
maranqz 29725e4b58 Fix tabulation of rst and change documentation link. 2020-08-26 14:38:37 +03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
maranqz 560c335c07 Add errors parameter in documentation. 2020-08-26 14:00:51 +03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Aditya 26d344b762 Merge branch 'http2' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-24 14:58:23 +05:30
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Aditya a87ab71d10 refactor(http2): metadata for Stream 
- Add Note about HTTP/2 Cleartext not supported in settings.rst
 2020-08-18 05:52:40 +05:30
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
Aditya d97cf973dd Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-16 11:31:10 +05:30
Andrey Rahmatullin acb3b4433a
Merge pull request #4724 from Gallaecio/feed-uri-params 
Document FEED_URI_PARAMS
 2020-08-14 14:47:55 +05:00
Mikhail Korobov 61653418ff
Merge pull request #4090 from Gallaecio/documentation-build 
Fix references to Python types in parameter type fields
 2020-08-14 00:58:06 +05:00
Adrián Chaves 65e0abaea5 Document FEED_URI_PARAMS 2020-08-13 09:05:51 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Adrián Chaves b2f4df5cb7 Merge remote-tracking branch 'upstream/master' into documentation-build 2020-08-11 13:16:14 +02:00
Aditya c67d6dea31 fix: H2 docs, NotImplementedError for H2 Tunnel 2020-08-11 04:39:41 +05:30
Adrián Chaves 4ee538e44b
Update unicode references from Python 2 times in the documentation (#4703) 2020-08-04 20:34:11 +02:00
Adrián Chaves 3600582f56
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708) 2020-08-04 20:05:56 +02:00
Aditya d707f8b5d9 docs: mention H2DownloadHandler in settings.rst 2020-07-30 18:06:21 +05:30
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
BroodingKangaroo a6c1d79b7c pep8 tiny changes 2020-07-28 11:53:05 +03:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
BroodingKangaroo 41263f61c6 Change single quotes to double in example in docs 2020-07-16 18:41:45 +03:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
BroodingKangaroo 8bdcdb0a76 Add quotes to example in docs 2020-07-16 09:13:54 +03:00
Júlio César Batista 38496a00b7
Use the itemlaoders library (#4516) 2020-07-15 12:08:36 +02:00
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Eugenio Lacuesta 2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse 2020-07-08 14:00:54 -03:00
BroodingKangaroo 6454d456d2 Make check of placeholder less strict 2020-07-03 10:24:13 +03:00
BroodingKangaroo 1e245046ed Change setting name. Add leading zeroes to batch_id. Minor fixes. 2020-07-02 12:38:08 +03:00
Andrey Rahmatullin af55d23167
Update the OpenSSL cipher list format link 
OpenSSL `ciphers(1)` is now almost empty: https://www.openssl.org/docs/manmaster/man1/ciphers.html

Alternative would be linking to 1.1.1 docs specifically.
 2020-07-01 19:46:54 +05:00
BroodingKangaroo 7b1d3c35ea Minor updates 2020-07-01 12:24:31 +03:00
Eugenio Lacuesta 5b88c522ac
Simplify dataclass example in item loader docs 2020-06-30 12:18:21 -03:00
BroodingKangaroo 05c2587c6a Docs update and tiny fixes 2020-06-28 09:45:45 +03:00
BroodingKangaroo c3cee74fd4 Change default value of FEED_STORAGE_BATCH_ITEM_COUNT to 0 2020-06-26 18:45:21 +03:00
BroodingKangaroo 77333666f1 resolve conflicts 2020-06-26 14:55:46 +03:00
Adrián Chaves 6f4c964aa4
Cover Scrapy 2.2.0 in the release notes (#4630) 2020-06-24 12:26:38 +02:00
Adrián Chaves cfd039aeb6 Remove a duplicate GCS_PROJECT_ID reference target 2020-06-22 19:28:33 +02:00
Eugenio Lacuesta 73b6ce8cb5
Update docs about dataclass items and loaders 2020-06-22 14:13:37 -03:00
Eugenio Lacuesta 1335d9053e
Update docs/topics/loaders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-06-22 14:05:44 -03:00
Eugenio Lacuesta 3efea98e05
Docs: add note about dataclass items and loaders 2020-06-22 12:41:25 -03:00
Eugenio Lacuesta c7f1c7e3f3
Merge branch 'master' into feat-685 2020-06-19 21:30:54 -03:00
Adrián Chaves f9c63384c0
Merge branch 'master' into allow-customizing-export-column-names 2020-06-18 14:09:20 +02:00
Devi Sandeep 5d54173187
Update docs on accessing callback arguments in errback (#4634) 2020-06-18 12:01:38 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta b6c5289fb9
Close page in pyppeteer example, mention asyncio reactor 2020-06-10 12:11:49 -03:00
Eugenio Lacuesta 78aa1b2bfc
Fix typo 2020-06-08 11:19:15 -03:00
Eugenio Lacuesta bdabc500aa
Update headless browser docs 2020-06-06 16:47:15 -03:00
Mikhail Korobov 1d2b692666
Merge pull request #4596 from sharmarohit/remove-unneeded-escape-sequence 
remove unneeded escape sequence
 2020-06-02 02:30:33 +05:00
Bulat Khabibullin 5cef927944
Implement TextResponse.json() (#4574) 2020-06-01 06:57:23 +02:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin 7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header 
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
 2020-05-27 21:41:43 +05:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta 634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download 2020-05-27 10:27:45 -03:00
Rohit Sharma 0cc0e51ca3 remove uneeded escape sequence 
removed uneeded wscape sequences from method arguments in the docs
folder
 2020-05-24 13:38:44 -07:00
Adrián Chaves 08756cd87a Fix the description of the ScreenshotPipeline example item pipeline 2020-05-22 18:06:54 +02:00
Eugenio Lacuesta f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header 2020-05-21 11:07:41 -03:00
BroodingKangaroo 5f6c1dceb1 Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-21 12:43:04 +03:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Adrián Chaves f418c6d5f7 logging.rst: remove unused, misleading import 2020-05-19 20:04:47 +02:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
BroodingKangaroo 10ae1a284f Minor fixes 2020-05-15 22:50:54 +03:00
BroodingKangaroo 1cdcf8b08b Minor fixes 2020-05-15 20:14:07 +03:00
BroodingKangaroo 69c005f013 Documentation indent fix 2020-05-14 10:35:56 +03:00
BroodingKangaroo 111a58fe3d Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-14 09:53:14 +03:00
BroodingKangaroo 8662d3587d Documentation and code refactoring 2020-05-14 00:05:36 +03:00
BroodingKangaroo 2327ecead0 Rename FEED_STORAGE_BATCH_SIZE to FEED_STORAGE_BATCH_ITEM_COUNT 2020-05-13 22:50:04 +03:00
Adrián Chaves 7dac99bb12
Merge branch 'master' into deprecate-baseitem 2020-05-13 20:35:03 +02:00
Aditya Kumar 8d1269bcbc
Cover chompjs in documentation (#4562) 2020-05-12 20:42:28 +02:00
nsirletti cb8140a42a
Deprecate Response.body_as_unicode() (#4555) 
Co-authored-by: Nicolas Sirletti <n.sirletti@gmail.com>
 2020-05-11 20:20:31 +02:00
Eugenio Lacuesta fb7ba6964d
Merge branch 'master' into deprecate-baseitem 2020-05-11 10:21:56 -03:00
Adrián Chaves e07708e374 request-response: update the consequences of str(b'') 2020-05-09 15:54:31 +02:00
Adrián Chaves c6746f0e38 bytes array → bytes object 2020-05-09 15:51:11 +02:00
Eugenio Lacuesta 6160370927
Merge remote-tracking branch 'upstream/master' into deprecate-baseitem 2020-05-07 14:38:54 -03:00
Adrián Chaves bbd9d05dbf request-response.rst: review type references around body mentions 2020-05-07 11:44:43 +02:00
Eugenio Lacuesta 7fd2085725
Merge remote-tracking branch 'upstream/master' into bytes_received_signal 2020-05-06 22:41:16 -03:00
Adrián Chaves a859ea0c8b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-05-06 19:48:32 +02:00
Eugenio Lacuesta 8d1e3ee0dd
Remove deprecated BaseItem from the docs 2020-05-06 09:24:40 -03:00
Mikhail Korobov fe6154e4fa clarify DOWNLOADER_HTTPCLIENTFACTORY docs 2020-05-04 18:18:38 +05:00
Eugenio Lacuesta e1948b4923
Add example about bytes_received signal 2020-05-04 09:07:27 -03:00
Mikhail Korobov 83d7360bb7 Don't mention unsupported package versions in docs 2020-05-04 02:00:11 +05:00
BroodingKangaroo dad2ea7522 Change time_id to batch_time 2020-05-02 01:21:03 +03:00
Eugenio Lacuesta 4d625afda5
Merge branch 'master' into bytes_received_signal 2020-05-01 16:11:32 -03:00
BroodingKangaroo df81870f39 Merge branch 'master' of https://github.com/scrapy/scrapy into ISSUE-4250-add_batch_deliveries 2020-05-01 12:18:39 +03:00
Adrián Chaves 1006db1e10
Merge branch 'master' into documentation-build 2020-04-28 21:07:14 +02:00
Ashe c207dbf939
Remove the asyncio warning from coroutines page (#4513) 2020-04-27 19:45:19 +02:00
BroodingKangaroo a18621552f Merge branch 'master' of https://github.com/BroodingKangaroo/scrapy into ISSUE-4250-add_batch_deliveries 2020-04-27 10:34:36 +03:00
BroodingKangaroo 2eee6c8101 Documentation spelling fix 2020-04-27 09:58:14 +03:00
BroodingKangaroo f0f1be76d1 Using time_id instead of time as a timestamp 2020-04-27 09:56:57 +03:00
Adrián Chaves ffe576c4ed
Cover Scrapy 2.1 in the release notes (#4499) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-04-24 11:44:36 +02:00
Aditya Kumar e4750f2fbd
async/deferred signal handlers (#4390) 
* [docs] async/deferred signal handlers

* [docs] update deferred signals example

* [docs] add subsections for built-in signals

* docs(signals): update signal handler example

* docs(signals): update signal handler example
 2020-04-20 20:47:57 +05:00
sakshamb2113 bfeb2c8c13
Added warning to use double quotes in Windows for scrapy shell in shell.rst (#4450) 
* modified debugging memory leaks with guppy in leaks.rst

* modified leaks.rst(issue #4285)

* removed guppy from telnet.py

* Fix undefined name error

* removed hpy key from telnet_vars in telnet.py

* updated shell.rst

* Update docs/topics/shell.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-04-18 20:21:26 +05:00
Eugenio Lacuesta 04b6295a69
Docs: replace deprecated FEED_* settings 2020-04-17 20:50:17 -03:00
Eugenio Lacuesta 1f2e2a6006
Merge branch 'master' into response_ip_address 2020-04-16 11:32:37 -03:00
BroodingKangaroo cac1f3a6ad Update documentation 2020-04-16 10:06:56 +03:00
Mikhail Korobov 36abe9235c
Merge pull request #4445 from adityaa30/docs-redirect-links 
update redirect links to python3
 2020-04-15 23:13:55 +05:00
Mikhail Korobov 066c02d0c3
Merge pull request #4455 from aditi137/docs-curl2scrapy 
[Docs] mention curl2scrapy in Request.from_curl
 2020-04-15 23:12:33 +05:00
Eugenio Lacuesta ac869181fb
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-04-15 13:42:35 -03:00
ilias-ant 83a0cc6cdf Add status to files information 2020-04-12 23:22:17 +03:00
Eugenio Lacuesta 34e81d0d74
Docs: remove duplicated setting definitions 2020-04-10 17:29:02 -03:00
Eugenio Lacuesta c4a5e3f0da
Simplify bytes_received signal 
Remove "source" parameter
 2020-04-06 09:26:13 -03:00
Eugenio Lacuesta e3342669ae
Merge branch 'master' into bytes_received_signal 2020-04-06 09:17:35 -03:00
Aditya b1904729d5 [docs] change mod to doc redirect link 2020-03-27 04:37:26 +05:30
Aditi Dutta 010edfe85c [Docs] mention curl2scrapy in Request.from_curl 2020-03-25 14:38:22 -04:00
Aditya 80c69d68ad [docs] refactor python docs links using intersphinx 2020-03-24 18:35:42 +05:30
Aditya f37b1bdc56 [docs] update redirect links to python3 2020-03-20 05:31:41 +05:30
Aditya ca08e04198 [docs] update redirect links python2 -> python3 2020-03-20 02:31:35 +05:30
Eugenio Lacuesta 34d607194b
Merge branch 'master' into feat-685 2020-03-18 21:24:36 -03:00
sakshamb2113 533131a30f
Remove Guppy-specific code and documentation (#4343) 2020-03-17 10:12:49 +01:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Eugenio Lacuesta aa6075e08c
Merge branch 'master' into bytes_received_signal 2020-03-15 18:05:04 -03:00
Eugenio Lacuesta bde0384dfd
Merge branch 'master' into spider.parse 2020-03-14 19:53:36 -03:00
Aditya Kumar f9bf4b8d4d
Remove all top-level imports for twisted.internet.reactor (#4406) 2020-03-14 10:39:00 +01:00
Eugenio Lacuesta 21f1eddc90
Merge branch 'master' into bytes_received_signal 2020-03-11 21:16:35 -03:00
Eugenio Lacuesta 49156f2ecb
[doc] Feed exports: full local path as example 2020-03-11 20:45:54 -03:00
Eugenio Lacuesta ada37c5409
Export to multiple formats in a single crawl 2020-03-05 09:33:57 -03:00
Eugenio Lacuesta 3aa5eab993
Merge branch 'master' into response_ip_address 2020-03-03 13:53:43 -03:00
Adrián Chaves 6aa0ba4553
Write release notes for Scrapy 2.0.0 (#4329) 2020-03-03 09:11:11 +01:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
Eugenio Lacuesta a44942d27d
Merge branch 'master' into response_ip_address 2020-02-23 18:13:52 -03:00
elacuesta 9d983c1b99
Expose certificate for HTTPS responses (#4054) 
* Expose certificate for HTTPS responses

* Fix test (missing inlineCallbacks decorator)

* Note about Response.certificate

* Explicitly cover None as the default value of Response.certificate

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-22 17:20:31 +05:00
Andrey Rahmatullin 88179027de
Merge pull request #4331 from Gallaecio/response-cb-kwargs 
Implement Response.cb_kwargs
 2020-02-19 22:40:14 +05:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00
Adrián Chaves a04dd13cd0
ie. → i.e. 2020-02-14 22:31:30 +01:00
Adrián Chaves 5ae3e1678f
ie. → i.e. 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-14 22:30:36 +01:00
Adrián Chaves df937d8280 Implement Response.cb_kwargs 2020-02-13 22:33:36 +01:00
Adrián Chaves 037ae5b22e
Explicitly indicate None as ip_address’s default value 2020-02-10 19:54:47 +01:00
Adrián Chaves 96bb3b5144
Merge branch 'master' into feat-685 2020-02-10 19:45:09 +01:00
Adrián Chaves 7025c18b15
Clear line of spaces 2020-02-10 19:43:23 +01:00
Eugenio Lacuesta 13ba9bc629
Note about Response.ip_address 2020-02-10 12:29:39 -03:00
Eugenio Lacuesta 42b4e9b337
Reword signal docs 2020-02-10 11:23:38 -03:00
Adrián Chaves c2f484de88
Merge branch 'master' into response_ip_address 2020-02-07 21:49:38 +01:00
Adrián Chaves aae4935605
Merge branch 'master' into spider.parse 2020-02-07 21:17:56 +01:00
Mikhail Korobov c3b690a5b5
Merge pull request #4290 from dekimsey/patch-1 
FilesPipeline.file_path has optional arguments
 2020-02-07 23:41:31 +05:00
Mikhail Korobov afbaf9d430
Merge pull request #4303 from whalebot-helmsman/request_left_downloader_signal 
request_left_downloader signal
 2020-02-07 23:33:51 +05:00
Vostretsov Nikita 153b78e53f
Update docs/topics/signals.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-07 11:08:55 +05:00
Vostretsov Nikita 8817b9e8e9
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:53 +05:00
Vostretsov Nikita 2f83f3e2cb
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:43 +05:00
Vostretsov Nikita 84b55b7364
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:35 +05:00
Eugenio Lacuesta 6c333491a8
Merge branch 'master' into response_ip_address 2020-02-06 18:53:11 -03:00
Adrián Chaves 24bb9fd5f7
Merge branch 'master' into spider.parse 2020-02-06 21:39:09 +01:00
Mikhail Korobov 042e71e2b8
Merge pull request #4311 from Gallaecio/metarefresh-ignore-tags 
Make METAREFRESH_IGNORE_TAGS an empty list by default
 2020-02-06 23:40:45 +05:00
Mikhail Korobov 8d2705f23c
Merge pull request #4305 from Respawnz/patch-1 
fix a typo in devloper-tools.rst
 2020-02-06 23:17:28 +05:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Vostretsov Nikita 6733f4d976
Update docs/topics/signals.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-06 18:40:42 +05:00
Adrián Chaves 576663e5a7 Make METAREFRESH_IGNORE_TAGS an empty list by default 2020-02-06 10:43:20 +01:00
Respawnz c2cca36821
typo 2020-02-06 05:39:15 +08:00
Vostretsov Nikita aab39f6341 docummentation for new signal 2020-02-05 11:35:03 +00:00
Eugenio Lacuesta b9e3a6201b
Merge branch 'master' into response_ip_address 2020-02-03 04:07:44 -03:00
Eugenio Lacuesta 6f02a8dccb
Add source parameter to bytes_received signal 2020-01-29 15:51:25 -03:00
Daniel Kimsey 752e8f7018 FilesPipeline.file_path has optional arguments 
Documented signature doesn't match the actual interface in [files.py](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L520).

Specifically, it looks like it may be [called](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L422) without a response value.

I found this when I was implementing the pipeline with the signature `file_path(self, request, response, info)` and the following error was being return in my results :

    [(False, <twisted.python.failure.Failure builtins.TypeError: file_path() missing 1 required positional argument: 'response'>)]

Scrapy==1.8.0
 2020-01-28 15:21:15 -06:00
Eugenio Lacuesta 4ffd18fb11
[docs] Mention that signals.bytes_received could be fired multiple times 2020-01-27 13:29:33 -03:00
Eugenio Lacuesta dbe20a863f
bytes_received signal: send spider argument 2020-01-27 12:28:54 -03:00
Eugenio Lacuesta 72b8613ee9
bytes_received signal (no tests) 2020-01-27 12:06:48 -03:00
Eugenio Lacuesta 8529dff41d
Update docs regarding Response.ip_address and IPv6 2020-01-26 18:00:56 -03:00
Eugenio Lacuesta 80925ab845
Get server IP address for HTTP/1.1 responses 2020-01-26 16:57:04 -03:00
Peter Vandenabeele f72d4e93e6
[Docs] 2 typos + 1 clarification in docs 
Fixing 2 small typos and adding 1 word as clarification in the downloader-middlewares.

Also, I was confused with the entries like `ref:Reppy <reppy-parser>` and similar entries.
Are these supposed to be links to other parts of the doc, or is this the intended way of showing these references ?
 2020-01-26 10:48:28 +01:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
OmarFarrag fc98aa6b67
Merge branch 'master' into ftp_files#3928 2020-01-24 14:52:40 +02:00
Mikhail Korobov 6a98d660e5
Merge pull request #3551 from jpbalarini/change_scraper_slot 
[MRG+1] Add ability to change max_active_size by setting
 2020-01-23 23:40:05 +05:00
JP Balarini 0f2d871d88 Use PEP 515 style for SCRAPER_SLOT_MAX_ACTIVE_SIZE documentation 2020-01-20 11:28:28 -03:00
Juan Pablo Balarini eaa8ed02d0 Add ability to change max_active_size by settings 2020-01-20 11:27:58 -03:00
Eugenio Lacuesta 302d3f552b
[doc] DNS_RESOLVER setting 2020-01-18 01:46:24 -03:00
Eugenio Lacuesta 2e405d2d5c
Merge branch 'master' into response_follow_all 2020-01-05 00:33:19 -03:00
1um0s 14d4428e70 Rephrasing documentation for image and file pipelines (#4252) 
* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* Rebase master

* Add class to media pipeline docs

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>

Co-authored-by: elacuesta <elacuesta@users.noreply.github.com>
 2019-12-30 00:56:22 +05:00
Andrey Rakhmatullin dc1ee09481 Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly. 2019-12-27 21:56:28 +05:00
Eugenio Lacuesta 8a1dc26d46
[doc] Note about the 'parse' method for CrawlSpider/XMLFeedSpider 2019-12-26 15:14:47 -03:00
Eugenio Lacuesta 5982e3477c
Take keyword arguments in base parsing methods 2019-12-23 14:12:21 -03:00
Andrey Rakhmatullin 20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Mikhail Korobov fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Mikhail Korobov 6a32a36e09
Merge pull request #4143 from Gallaecio/undent-rst-doctests 
Do not indent doctests from the documentation unnecessarily
 2019-12-20 01:59:07 +05:00
Adrián Chaves b4a2d985a7
Merge branch 'master' into undent-rst-doctests 2019-12-19 12:39:17 +01:00
Marc Hernandez Cabot 16b363de31 Merge branch 'master' into fix_typos 2019-12-19 10:02:21 +01:00
Marc Hernandez Cabot 23a67cec27 fix first letter capitalization for Raring and Scrapy 2019-12-19 09:57:17 +01:00
Mikhail Korobov 8f7faaa63d
Merge pull request #4190 from Gallaecio/doctest 
Make developer-tools doctests pass
 2019-12-18 23:40:21 +05:00
Marc Hernandez Cabot c0d84f0962 fix typos 2019-12-18 19:39:21 +01:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Adrián Chaves ab13221b0b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2019-12-17 14:06:14 +01:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Eugenio Lacuesta 21a2f8c6f8
Merge branch 'master' into response_follow_all 2019-12-10 17:30:59 -03:00
Adrián Chaves 1fc2b140c1
Merge branch 'master' into documentation-coverage 2019-12-05 14:43:36 +01:00
Adrián Chaves 83b8046fdc Do not indent doctests from the documentation unnecessarily 2019-12-05 13:52:31 +01:00
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Mikhail Korobov 9b7452211a
Merge pull request #4099 from BurnzZ/itemloader-docs 
update docs of scrapy.loader.ItemLoader.item
 2019-12-03 13:14:45 +05:00
Adrián Chaves 54b056c4be Make developer-tools doctests pass 2019-11-25 12:32:35 +01:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Adrián Chaves f251dda268 Make debug doctests pass 2019-11-21 11:59:10 +01:00
Mikhail Korobov f1fd7ec318
Merge pull request #4171 from Gallaecio/doctest 
Make command doctests pass
 2019-11-20 19:47:22 +05:00
Eugenio Lacuesta 48223c107a
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-20 09:41:42 -03:00
Adrián Chaves e84cb18ca0 Use InterSphinx to link to the Twisted documentation 2019-11-18 15:50:45 +01:00
Adrián Chaves 74589df02f Make command doctests pass 2019-11-18 14:51:44 +01:00
Andrey Rakhmatullin dd367438fa Improve the dbm module ref. 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin a9c891399d Fix a duplicate ref name in docs. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 1d7c8cb0b1 Remove six.PY2 and six.PY3 conditionals. 2019-11-14 13:45:37 +05:00
Andrey Rahmatullin c911e80209
Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs 
docs: use __init__ method instead of constructor
 2019-11-12 21:48:16 +05:00
Andrey Rahmatullin d8098350f0
Merge pull request #3960 from thernstig/issue_2149 
Update documentation for logging manually
 2019-11-12 16:17:49 +05:00
Adrián Chaves aef98188fa Improve the details about request serialization requirements for JOBDIR 2019-11-07 18:06:55 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Adrián Chaves 94f060fcc8
Cover Scrapy 1.8.0 in the release notes (#3952) 2019-10-29 12:53:46 +01:00
Andrey Rahmatullin 18b808b2e9
Merge pull request #4092 from further-reading/master 
Add Python 3.8 official support
 2019-10-29 16:30:58 +05:00
Adrián Chaves 3d0df419c4 Mark the LevelDB storage backend as deprecated 2019-10-28 11:24:47 +01:00
elacuesta 7731814cc2 ItemLoader: improve handling of initial item (#4036) 2019-10-28 10:53:53 +01:00
Kevin Lloyd Bernal 84fe4011b0 update docs of scrapy.loader.ItemLoader.item 2019-10-23 20:39:53 +08:00
Adrián Chaves 5479e7ecc7 Indicate that lists of emails may be provided as a single string or as a list of strings 2019-10-22 15:24:44 +02:00
Ammar Najjar d21e1034f0
docs: correct point,comma and plural replacements 
Issue #4086
 2019-10-22 13:29:38 +02:00
WinterComes 5d75ed4cba Remove an old note about contracts (#4093) 2019-10-22 12:19:07 +02:00
Ammar Najjar da8cd9448d
docs: always surround __init__ with `` in docs 
Issue #4086
 2019-10-21 19:56:58 +02:00
Adrián Chaves ad6075440c Fix references to Python types in parameter type fields 2019-10-21 19:00:03 +02:00
Ammar Najjar 68a7d05ed8
docs: use __init__ method instead of constructor 
Issue #4086
 2019-10-21 15:42:24 +02:00
Adrián Chaves 0fbd1ff4a9 constructor → __init__ method 2019-10-21 14:06:45 +02:00
Mikhail Korobov 8cb53441b0
Merge pull request #4056 from Gallaecio/documentation-build 
Fix internal links in the tutorial and release notes
 2019-10-19 00:00:02 +02:00
Bulat 84be6a941e Refactor sentence. 2019-10-16 14:04:07 +03:00
Bulat c9614a5bdd Fixed BOT_NAME documentation 2019-10-16 12:07:19 +03:00
Adrián Chaves d72ed46fe0 Improve how extra Item API members are introduced in the documentation 2019-10-15 16:04:26 +02:00
Baron Hou 7b1e69dec4 reponse → response (#4079) 2019-10-15 14:51:15 +02:00
Eugenio Lacuesta 5f168cd459
Response.follow_all 2019-10-08 16:32:56 -03:00
Adrián Chaves a4aa5b8926 Fix internal links in the tutorial and release notes 2019-10-02 14:51:06 +02:00
elacuesta 6ad5a89cb0 [Doc] Use autoclass in topics/request-response.rst (#4055) 2019-10-02 12:18:36 +02:00
Mikhail Korobov 74b4a5c77c
Merge pull request #4006 from whalebot-helmsman/protego-default 
Use protego as a default robots.txt parser
 2019-10-02 00:03:50 +05:00
Kristobal Junta c232bbdc42 fix typo in docs/topics/spiders 2019-10-01 17:41:38 +03:00
OmarFarrag 175cd2ece5
Update docs/topics/media-pipeline.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-10-01 07:27:31 +02:00
Adrián Chaves 7f4f98fd38 Provide complete API documentation coverage of scrapy.linkextractors 2019-09-30 18:22:28 +02:00
OmarFarrag 28005b2872
Update media-pipeline.rst 2019-09-28 06:21:14 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
OmarFarrag b14c3cb612
Add media pipelines FTP documentation 2019-09-19 23:33:57 +02:00
Adrián Chaves c26a9015ad Clarify the effects of dont_merge_cookies 2019-09-19 11:08:06 +02:00
Adrián Chaves 9b65f9aa5b
Fix the item exporter example (#4022) 2019-09-19 09:17:23 +02:00
Vostretsov Nikita 2438ac529a
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:22 +05:00
Vostretsov Nikita d1d0bf8491
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:12 +05:00
Vostretsov Nikita 57e6f4c750 add link to performance comparison 2019-09-17 07:18:37 +00:00
Vostretsov Nikita d39ef77e6b add link to google description of lenght-based rule 2019-09-17 06:34:33 +00:00
Vostretsov Nikita b792dba528 remove periods 2019-09-17 06:28:33 +00:00
Vostretsov Nikita 50c85e3114 Merge branch 'protego-default' of github.com:whalebot-helmsman/scrapy into protego-default 2019-09-16 14:24:31 +00:00
Vostretsov Nikita 5197b39e01 fix capitalization, remove commas 2019-09-16 14:24:25 +00:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Vostretsov Nikita 66145b4eaf
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
 2019-09-12 18:51:00 +03:00
Vostretsov Nikita 171fa1cd10 documentation rework 2019-09-10 09:59:36 +00:00
Vostretsov Nikita 6bd88711f2 update documentation 2019-09-10 08:55:37 +00:00
Vostretsov Nikita 38828d3fd4
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:13 +03:00
Vostretsov Nikita 9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta 2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Adrián Chaves 2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov 93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Adrián Chaves 0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Tobias Hernstig 2b0de0606c Fix merge conflicts 2019-08-15 18:54:28 +02:00
Tobias Hernstig 50c4cafe0c Update documentation for logging manually 
Usage of basicConfig() together with crawlerRunner is not recommended.
Update documentation to highlight this fact.

Closes #2149, #2352, #3146
 2019-08-15 18:46:00 +02:00
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves 04bca6af7c
Merge pull request #3894 from KristobalJunta/fix_retry_docs 
fix default RETRY_HTTP_CODES value in docs
 2019-07-29 18:20:55 +02:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Kristobal Junta 43d5b5a524 fix default RETRY_HTTP_CODES value in docs 2019-07-22 10:24:14 +03:00
Andrey Rakhmatullin 42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Deschner, Magdalena c44d49b238 minor PEP8 style changes 2019-07-17 13:13:52 +02:00
Deschner, Magdalena 6660020ebb remove detailed description about individual settings 2019-07-17 11:30:02 +02:00
Deschner, Magdalena e892a484e8 add instructions about how to define output file when running scrapy from script instead of cmd 2019-07-16 13:53:56 +02:00
Adrián Chaves fa6a0d799b
Merge pull request #3684 from Gallaecio/remove-ubuntu-topic 
Remove docs/topics/ubuntu.rst
 2019-07-12 14:03:11 +02:00
Adrián Chaves ed487b7db3 broad-crawls.rst: Refactor the memory usage section 2019-07-11 11:14:22 +02:00
Andrey Rakhmatullin 98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Robin 5cdf277017 Update broad-crawls.rst 
Added section on how to treat memory consumption problems of broad crawls.
 2019-07-11 08:49:24 +02:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov 050a62a955
Merge pull request #3861 from Gallaecio/documentation-coverage 
Increase the documentation coverage of exceptions
 2019-07-10 10:18:34 +05:00
Eugenio Lacuesta 9fb0f8454e
Extend docs about Crawling Rules 2019-07-09 15:30:22 -03:00
Eugenio Lacuesta d04e84c11d Merge remote-tracking branch 'upstream/master' into docs_crawlspider_link_text 2019-07-09 15:24:31 -03:00
Adrián Chaves a0bbbe8ffa Add the API documentation of ContractFail to the contracts topic 2019-07-09 12:33:57 +02:00
Andrey Rakhmatullin 0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Mikhail Korobov 4d4bd0e823
Merge pull request #3609 from Gallaecio/2253 
Document FilesPipeline.file_path and ImagesPipeline.file_path
 2019-07-04 22:05:46 +05:00
Mikhail Korobov 4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00
Mikhail Korobov e5f12faf5f
Merge pull request #3703 from Gallaecio/ajax-docs 
Add a topic about reaching data that selectors cannot reach
 2019-06-25 23:11:59 +03:00
Adrián Chaves d7795484dc Document scrapy.crawler.Crawler.stop() 2019-06-25 17:29:49 +02:00
Adrián Chaves 8a3b15eb91 Document how to select dynamically-loaded content 2019-06-24 16:06:21 +02:00
Daniel Graña cdfb20aee8
Merge pull request #3739 from Matthijsy/feature/scrapy_check_env 
[MRG+1] Add SCRAPY_CHECK environment variable
 2019-06-13 19:43:01 -03:00
Adrián Chaves ea209a0ea7 Fix module double indexing issues in the documentation 2019-06-04 14:35:27 +02:00
Mikhail Korobov 890e7fbd5b
Merge pull request #3791 from barraponto/patch-3 
[MRG+1] Fix documentation for spiderloader
 2019-05-28 19:16:12 +01:00
Mikhail Korobov 8d50384210
Merge pull request #3797 from mar-heaven/master 
[MRG+1] remove a "is"
 2019-05-28 19:12:04 +01:00
mar-heaven 18f01ea6ec
remove a "is" 
When I translated in Chinese, I found a needless "is"
 2019-05-27 17:15:30 +08:00
Anubhav Patel da82ede8a0
describe method as a command 2019-05-25 17:19:10 +05:30
Capi Etheriel 7d36fa7435
Fix documentation for spiderloader 2019-05-24 10:32:55 -03:00
Mikhail Korobov a3d38041e2
Merge pull request #3764 from Jeffallan/patch-1 
[MRG+1] Update telnetconsole.rst
 2019-05-09 20:59:30 +05:00
Adrián Chaves 611249bb7f Implement the METAREFRESH_IGNORE_TAGS setting 2019-05-08 12:52:29 +02:00
Jeffallan 5814344adf
Update telnetconsole.rst 
Change spelling of bellow to below.
 2019-05-04 14:15:47 -05:00
Adrián Chaves bc1a929213
Improve the documentation about detecting check runs 2019-05-03 14:42:12 +02:00
Matthijs Vos 8bd207a2f6 Add documentation 2019-05-03 13:50:24 +02:00
Vandenn 122ca62119 doc: update configure_logging docs to discourage use with CrawlerProcess 2019-05-03 16:39:26 +08:00
Adrián Chaves 92fa657f9e
Merge pull request #3730 from ankostis/doc_logstats_delay 
[MRG+1] doc: document LOGSTATS_INTERVAL setting
 2019-04-11 17:34:45 +02:00
Mikhail Korobov 9280185054
Merge pull request #3520 from whalebot-helmsman/round-robin-scheduler-tested 
[MRG+1] Downloader-aware Priority Queue for Scrapy
 2019-04-07 00:06:08 +05:00
Kostis Anagnostopoulos a8f83ab967 doc: document LOGSTATS_INTERVAL setting 2019-04-06 14:58:32 +02:00
Eugenio Lacuesta 7a38623cec [Docs] Clarify comment about meta dictionary 2019-04-01 17:09:49 -03:00
Eugenio Lacuesta 8ebbc731b2 [Docs] Rephrase Rule docs 2019-04-01 16:15:03 -03:00
Eugenio Lacuesta 07ff9248a5 [Docs] CrawlSpider: add note about link text 2019-04-01 12:31:26 -03:00
Konstantin Lopuhin b5c552d17f
Merge pull request #2061 from elacuesta/process_spider_exception_generator 
[MRG+1] process_spider_exception on generators
 2019-04-01 10:43:19 +03:00
Eugenio Lacuesta 294ef51bb2 parse command: update docs about passing callback keyword arguments 2019-03-29 16:12:55 -03:00
Eugenio Lacuesta ccb56a317e Update docs about cb_kwargs and meta 2019-03-29 14:13:50 -03:00
Adrián Chaves 484bd0d22a Allow customizing export column names 2019-03-29 15:16:54 +01:00
Vostretsov Nikita 1c6733454e added underlines 2019-03-29 10:44:55 +00:00
Vostretsov Nikita ef743983a9 change wording 2019-03-29 10:38:13 +00:00
Adrián Chaves 1ee99e1f42
Update docs/topics/settings.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:29:15 +00:00
Adrián Chaves bd228f1d96
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:29:04 +00:00
Adrián Chaves e3df6be360
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:28:52 +00:00
Adrián Chaves 46b9ab0c58
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-29 10:28:36 +00:00
Eugenio Lacuesta 8fb077694f Request.cb_kwargs: Update docs 2019-03-28 16:32:35 -03:00
Eugenio Lacuesta e8af6331b5 Add cb_kwargs option to the parse command 2019-03-28 15:03:40 -03:00
Eugenio Lacuesta 3efe3bea1c Update docs about cb_kwargs and meta 2019-03-28 14:16:03 -03:00
Mikhail Korobov ec719f55e7
Merge pull request #3682 from elacuesta/rule_process_request_response_parameter 
[MRG+1] Rule.process_request: access Response object
 2019-03-28 01:39:54 +05:00
Eugenio Lacuesta c43a23152b Merge remote-tracking branch 'upstream/master' into callback_kwargs 2019-03-27 14:42:41 -03:00
Eugenio Lacuesta 8528f5065f [Doc] Update cb_kwargs example 2019-03-27 14:42:26 -03:00
Anubhav Patel 9c9bca4e1c make suggested changes. 2019-03-27 18:29:48 +05:30
Mikhail Korobov 845bae6637
Update docs/topics/broad-crawls.rst 
Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-27 08:49:19 +00:00
Maram Sumanth 2fd8b7c28c [MRG+1] redirect_reasons in Request.meta (#3687) 2019-03-27 00:15:53 +05:00
Adrián Chaves 431f18a9a1 Document FilesPipeline.file_path and ImagesPipeline.file_path 2019-03-26 12:15:19 +01:00
Adrián Chaves ba13de29e1
Merge branch 'master' into feat-685 2019-03-26 10:38:38 +01:00
Vostretsov Nikita 103fdc40c1 Merge remote-tracking branch 'upstream/master' into round-robin-scheduler-tested 2019-03-25 16:16:42 +03:00
Vostretsov Nikita 8afffb7234 Tests Cleanup 
add doctest for function
no need in this variables
move common assertion inside function
rename variable
rename variables
rename function
use function
this is not a method of public API
correct name for test
Update docs/topics/settings.rst

Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 989bba6cb3 Revert "new signal" 
This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739.
remove tests
Revert "emit new signal"

This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf.
Revert "documentation for new signal"

This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.
 2019-03-25 16:08:40 +03:00
Mikhail Korobov 83eb537645 assorted cleanups: comments, docstrings, etc 
scheduler cleanup

Scheduler no longer converts requests to dicts; PriorityQueue
instances always work with Request instances; converting Requests
to dicts is now Priority Queue responsibility.
minor cleanup
 2019-03-25 16:08:40 +03:00
Vostretsov Nikita 3b1db71dac New signal 
update signature
documentation for new signal
utilize new signal
correct signal handler signature
emit new signal
test another signal
new signal
rename test file
faster test
rename test case
tests for signal emitting in bad cases
 2019-03-25 16:08:40 +03:00
Eugenio Lacuesta 174ba3cc56 Rule.process_request: update docs 2019-03-22 19:16:18 -03:00
Mikhail Korobov 8583c033ef
Merge pull request #3694 from noviluni/master 
[MRG+1] Fix numeration
 2019-03-23 00:23:06 +05:00
Mikhail Korobov 729580dae2
Merge pull request #3671 from Gallaecio/item-deepcopy 
Implement Item.deepcopy()
 2019-03-22 22:56:39 +05:00
Adrián Chaves 7c148fce5a Implement Item.deepcopy() 2019-03-22 17:07:10 +01:00
Konstantin Lopuhin af2b666595
Merge pull request #3505 from kasun/json_request 
[MRG+2] Request subclass for json requests #3504
 2019-03-22 18:25:54 +03:00
Júlio César Batista 2cb4dc3205
Mentioning to use JSON API for ACLs 2019-03-22 09:50:11 -03:00
Vostretsov Nikita 338b78d796 Add documentation 
add section to broad-crawl topic
reword in accord with broad-crawl topic
add documentation for new priority queue
 2019-03-22 15:40:37 +03:00
Adrián Chaves 9c314800e4 Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2019-03-22 15:40:36 +03:00
Marc Hernández 70aa5b1333
Fix numeration 2019-03-20 15:32:20 +01:00