RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

2770 Commits

Author SHA1 Message Date
databender 1a8b98843a updated documentation for python version for reppy 2021-08-16 17:00:05 +05:30
databender 8bbaea9892 updated documentation for python version for reppy 2021-08-16 16:57:43 +05:30
Aaron Tan 954f303590
Update docs/topics/settings.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-09 22:23:23 +10:00
Aaron Tan 8e7d2ef133 Document JOBDIR option issue #5173 
Add JOBDIR setting to the settings page.
Add default JOBDIR setting to global defaults in scrapy.settings.default_settings module.
 2021-08-07 11:44:12 +10:00
Aaron Tan 880a4d9493
Update docs/topics/extensions.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-08-01 11:02:27 +10:00
Aaron Tan d55b6fcad6 Fix for duplicate object description error 2021-07-28 12:10:34 +10:00
Aaron Tan 714aa3970e
Merge branch 'scrapy:master' into docs-extension 2021-07-30 20:36:58 +10:00
Andrey Rahmatullin b406affd5b
Merge pull request #5210 from laggardkernel/feature/aws-session-token 2021-07-30 15:15:55 +05:00
Eugenio Lacuesta cc89f6be38
Response.attributes (#5218) 2021-07-29 22:12:44 +02:00
Eugenio Lacuesta 4b62ac6c3a
Update headless browser docs to mention playwright 2021-07-28 15:00:24 -03:00
Eugenio Lacuesta 9aef9b78eb
Merge remote-tracking branch 'upstream/master' into docs-headless-browser 2021-07-28 14:31:49 -03:00
Eugenio Lacuesta 0e3d50dd18
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:30:16 -03:00
Eugenio Lacuesta 494e0ad8ff
Update docs/topics/dynamic-content.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-28 14:29:50 -03:00
laggardkernel 8e7b96d8a2 Tweak doc for setting AWS_SESSION_TOKEN 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-07-27 19:40:02 +08:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Rob Banagale b22a004398
Document media pipeline file naming (#5152) 2021-07-26 20:51:32 +02:00
Aaron Tan 209c1fce02
Reference MailSender in StatsMailer 
Added a reference to MailSender in the StatsMailer extension description and included a link to the document detailing how to instantiate MailSender and using Scrapy settings objects.
 2021-07-24 14:50:48 +10:00
Adrián Chaves 8a2768561f
Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased 2021-07-16 14:32:15 +02:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Türkalp Burak KAYRANCIOĞLU fcc6becc58
S3FeedStorage: allow custom endpoint (#4998) 
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2021-07-14 10:00:43 +02:00
D R Siddhartha 4ddc9d6b55
Feeds: Item Filters (#5178) 2021-07-13 17:22:29 +02:00
Andrey Rahmatullin 1c46d5aa93
Merge pull request #5191 from ivanprado/master 
CloseSpider can be raised on spider_idle signal handler to set the closing reason
 2021-07-12 18:46:40 +05:00
Iván de Prado Alonso e94d3ac173 Expanded doc for idle signal 2021-06-29 13:40:43 +01:00
Adrián Chaves 73ff9ffd64
spiders.rst: indent warnings into class descriptions 2021-06-26 08:58:29 +02:00
Iván de Prado Alonso 812b4bb518 CloseSpider can be raised on spider_idle signal handler 2021-06-23 17:22:01 +01:00
Andrey Rakhmatullin de69d967f9 Fix async spider examples. 2021-06-11 16:12:25 +05:00
Andrey Rakhmatullin ecfc924ca8 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-06-04 18:55:32 +05:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Shinichi Takayanagi bd60c3f41f
More documentation for setting spider atributes 
* docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17

* Update feed-exports.rst

* Update feed-exports.rst

* Reflects the comments

* Remove redundant newline

* Update docs/topics/feed-exports.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Apply suggestions from code review

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 00:58:04 +05:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Renne Rocha 34b216289c
Update link for reasoning value of URLLENGTH_LIMIT (#5134) 2021-05-06 16:34:05 +02:00
Eugenio Lacuesta ddea6b7bfa
Scheduler: minimal interface, API docs (#3559) 2021-04-26 16:16:14 -03:00
Andrey Rahmatullin 7095df1558
Merge pull request #5082 from storymode7/shell-fix 
Find bash from PATH instead of /bin/bash
 2021-04-21 14:30:50 +05:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Andrey Rahmatullin 5b78a64fca
Merge pull request #5070 from ric2b/patch-1 2021-04-15 14:50:59 +05:00
Andrey Rakhmatullin ffc6f525ce Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-04-14 17:34:58 +05:00
Mayank Singhal a4415e4e6f Add DataURI download handler in DOWNLOAD_HANDLERS_BASE documentation 2021-04-13 17:56:44 +05:30
Adrián Chaves 5a75b14a5f docs: require sphinx-rtd-theme>=0.5.2 and the latest pip to prevent installing breaking docutils>=0.17 2021-04-07 14:04:00 +02:00
Ricardo Amendoeira 8603f9d7a5
Apply changes to other examples in the same section. 2021-04-06 20:23:07 +01:00
Adrián Chaves a71d6ef29d
2.5.0 release notes (#5028) 
Co-authored-by: Eugenio Lacuesta <1731933+elacuesta@users.noreply.github.com>
 2021-04-06 16:09:07 +02:00
Mayank Singhal e7d51886ef Find bash from PATH instead of /bin/bash 2021-04-06 02:21:18 +05:30
Andrey Rakhmatullin 849472535e Update docs. 2021-04-02 20:20:35 +05:00
Mikhail Korobov f0c8d31193
Merge pull request #4694 from Jgaldos/improve-httpstatus-all-meta 
Improve http status all on http error middleware
 2021-04-01 22:43:45 +05:00
Akshay Sharma cc095aa895
Windows pip installation guide (#4736) 
* added initial steps

* fixing link

* python3 -> python

* remaining steps

* steps updated

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* added link to Visual Studio

* removed 'install V'

* Update docs/intro/install.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:39:33 +05:00
anay2103 5492972d8a
added customized filtering examples in logging.rst (#4965) 
* added customized filtering examples in logging.rst

* Update logging.rst

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update logging.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2021-04-01 22:30:48 +05:00
Mikhail Korobov b09ff99d81
Merge pull request #4974 from Gallaecio/ul-margin 
Move lists closer to their introducing paragraph
 2021-04-01 22:27:59 +05:00
Mikhail Korobov e0a2d2b3c9
Merge pull request #5014 from jpmckinney/patch-1 
docs: Clarify there's one extension instance per crawler
 2021-04-01 22:26:20 +05:00
Mikhail Korobov d0e23486b4
Merge pull request #4902 from Gallaecio/retry-request 
Implement a retry request function
 2021-04-01 22:23:56 +05:00
Veniamin Gvozdikov 90fe494ba2 Rebranding, updated GA code 2021-04-01 11:11:28 +03:00
Ricardo Amendoeira b247fa9982
Include loading settings in `Running multiple spiders in the same process` section 
The example in the documentation doesn't take into account the project settings
 2021-03-29 01:48:28 +01:00
Andrey Rakhmatullin 7b515a0485 Merge remote-tracking branch 'origin/master' into asyncio-parse-asyncgen-proper-rebased 2021-03-25 22:13:49 +05:00
Andrey Rakhmatullin a97dc55c71 Add/improve docs. 2021-03-25 20:49:12 +05:00
Pratik Mahankal f0e1a33225
Sort the list of Request.meta alphabetically #5061 (#5065) 2021-03-23 22:16:50 +05:00
Adrián Chaves 0dad0fce72
Use pip<20.3 to fix ReadTheDocs builds (#5052) 2021-03-19 11:13:05 +01:00
Adrián Chaves 1d83697990
Merge pull request #4769 from scrapy/http2 
Implement experimental HTTP/2 support
 2021-03-18 22:18:12 +01:00
Mikhail Korobov ab037ab7c9
Merge pull request #5016 from deepang17/scrapy-4996 
DOCS: Cover scrapy-bench in the documentation
 2021-03-12 04:18:56 +05:00
Eugenio Lacuesta 0c16088230
headers_received signal (#4897) 2021-03-11 11:52:35 -03:00
Adrián Chaves 5b2d3e17a3
Merge branch 'master' into http2 2021-03-09 14:59:17 +01:00
deepang17 2d41c77e6a Merge branch 'master' of https://github.com/scrapy/scrapy into scrapy-4996 2021-03-09 17:33:05 +05:30
deepang17 3d88ac605b FIX: Updated benchmarking.rst 2021-03-09 17:19:34 +05:30
Andrey Rahmatullin 954c4b48e6
Merge pull request #3667 from Gallaecio/long-allowed-domains 
Add a FAQ entry on how to deal with long lists of allowed domains
 2021-03-05 20:16:37 +05:00
deepang17 36f1dbf665 DOCS: Covered scrapy-bench 2021-03-02 22:12:44 +05:30
James McKinney 3c5668d0db
docs: Clarify there's one extension instance per spider 2021-03-01 22:00:33 -05:00
deepang17 b25616d107 DOCS: Cover scrapy-bench in the documentation 2021-02-28 16:26:46 +05:30
deepang17 7a54580679 DOCS:Cover scrapy-bench in the documentation 2021-02-28 15:33:09 +05:30
Adrián Chaves f689615e8d Close files in the PerYearXmlExportPipeline documentation example 2021-02-24 12:54:56 +01:00
Wehzie a36f952198
fixed typo "an quotes.json" -> "a quotes.json" (#5005) 2021-02-24 08:15:44 +01:00
Adrián Chaves 6ab990181c Document get_retry_requests 2021-02-22 14:48:03 +01:00
Andrey Rakhmatullin 58f8481301 Update docs. 2021-02-05 16:21:56 +05:00
Adrián Chaves 4c801551fa Document that the bytes_received signal is not yet implemented for HTTP/2 2021-02-03 21:11:46 +01:00
Adrián Chaves 1773eaf5dc Move lists closer to their introducing paragraph 2021-02-03 11:43:18 +01:00
Adrián Chaves 1a7bde0d8e Document that HTTP/2 server pushes are ignored 2021-02-03 10:55:11 +01:00
Adrián Chaves c8d8b180bf Merge remote-tracking branch 'upstream/master' into http2 2021-02-03 09:37:35 +01:00
Adrián Chaves 2ce8e0c742 Document the (hard-coded) maximum HTTP/2 frame size accepted from servers 2021-02-03 09:09:53 +01:00
Adrián Chaves f30f53b3cc Scrapinghub → Zyte 2021-02-02 15:03:20 +01:00
M Ikram Ullah Khan 0a1e2fefab
Docs: fix typo in news.rst (#4942) 2021-01-04 10:30:23 -03:00
Eugenio Lacuesta e494a3f733
protocol attribute for h2 responses 2020-12-31 11:50:15 -03:00
Eugenio Lacuesta d698b5147b
Merge branch 'master' into http2 2020-12-31 11:13:25 -03:00
Adrián Chaves d0af008608
Merge pull request #4878 from elacuesta/response-protocol-attribute 
Response.protocol attribute
 2020-12-16 15:27:40 +01:00
gunadhya db10aaf9eb
Update links in installation guide (#4899) 2020-12-03 10:56:36 +01:00
etimoz 95d39d5cb4 removed wrong super argument in overriding serialize_fields code example 2020-11-29 13:24:04 +01:00
Adrián Chaves 07379cf9b7
Merge branch 'master' into allow-customizing-export-column-names 2020-11-20 20:06:18 +01:00
Adrián Chaves 63becd1bc8
Update news.rst 2020-11-17 21:58:08 +01:00
Adrián Chaves 15d301e968
Cover Scrapy 2.4.1 in the release notes (#4884) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-11-17 09:16:08 +01:00
Eugenio Lacuesta 5e9a99e6a1
Reponse.protocol as string 2020-11-11 11:15:29 -03:00
Eugenio Lacuesta 2242412556
Docs: placeholder for versionadded directive 2020-11-11 10:50:54 -03:00
Eugenio Lacuesta 61d089485c
Docs: sort versionadded directives 2020-11-11 01:31:21 -03:00
Eugenio Lacuesta 587b4dd71f
Docs for the Response.protocol attribute 2020-11-11 01:22:33 -03:00
Eugenio Lacuesta 114229eb4a
Docs: add a note about asyncio.set_event_loop 2020-11-06 13:29:14 -03:00
Adrián Chaves 47eac83740 Set a release date for Scrapy 2.4.0 2020-10-11 22:11:14 +02:00
Adrián Chaves 015c82b974
Scrapy 2.4 release notes (#4808) 2020-10-11 22:09:45 +02:00
Andrey Rahmatullin 9f8c3938cc
Merge pull request #4823 from elacuesta/cookies-revert-header 
Do not process cookies from headers
 2020-10-08 20:20:32 +05:00
Andrey Rahmatullin 45c06cfd80
Merge pull request #4831 from starrify/downloadermw-support-zstd 
Adding support for zstd in HttpCompressionMiddleware
 2020-10-08 14:08:18 +05:00
Adrián Chaves 269fe35d6d
Merge branch 'master' into http2 2020-10-06 21:05:10 +02:00
OfirD1 1a597d5e3d
moved the sentence about processing pending requests when a spider is closed onto a generic note. 2020-10-06 21:54:42 +03:00
Eugenio Lacuesta 137c8ba6ee
Docs: mention limitation about Cookie header 2020-10-06 10:50:17 -03:00
P. Chen da3171d4f7
Using the `zstandard` package than `zstd` for supporting frames both with and without the content size info 
See also: https://github.com/sergey-dryabzhinsky/python-zstd/issues/53
 2020-10-05 23:18:58 +01:00
P. Chen c6c3f2ce66
Updating the doc entry for the HTTP compress downloader middleware on zstd 2020-10-05 21:10:40 +01:00
Adrián Chaves 004b40a719
as soon as → as long as (#4825) 2020-10-02 19:53:55 -03:00
Mikhail Korobov 4f27c5f82b
Merge pull request #4768 from maranqz/feature/4606-exporter-from-FEEDS 
Pass info from FEEDS to ItemExporter
 2020-10-02 00:10:05 +05:00
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Andrey Rahmatullin 5a38639359
Merge pull request #4809 from madeny/patch-1 
Correct some typos
 2020-09-28 18:26:43 +05:00
madeny eff96038c7
Correct some typos 
This won't be an issue if **your** spider doesn't rely on cookies.
 2020-09-26 22:50:38 +02:00
Mirwaisse Djanbaz 3989f64baa
dependecies → dependencies (#4801) 2020-09-21 14:40:00 +02:00
Mirwaisse Djanbaz 7f1e74daa2
dependencides → dependencies (#4800) 2020-09-21 14:38:16 +02:00
Mikhail Korobov 85e13aff74
Merge pull request #4782 from sripriyesha/patch-1 
add mention of FTP server storage in media storage intro
 2020-09-15 01:17:46 +05:00
Adrián Chaves 4d6359df2d Mark HTTP/2 as experimental 2020-09-11 13:51:05 +02:00
KAILASA's Sri Nithya Priyeshananda 7a83474cc5
add mention of FTP server storage in media storage intro 
At the beginning of this doc, in "Specifying where to store the media" feature details, FTP server storage mention was missing
 2020-09-08 17:16:31 +02:00
Ilia Sergunin d10464ca96
Update docs/topics/feed-exports.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-09-01 10:13:40 +03:00
Adrián Chaves 8a3ba34a75 Merge remote-tracking branch 'upstream/master' into http2 2020-08-31 16:43:32 +02:00
Jose Galdos a41c205928 Update httpstatus documentation. 2020-08-31 08:58:43 -05:00
maranqz a8e895e684 kwargs for Item exporters classes 
test
docs
 2020-08-30 10:57:22 +03:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
maranqz 29725e4b58 Fix tabulation of rst and change documentation link. 2020-08-26 14:38:37 +03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
maranqz 560c335c07 Add errors parameter in documentation. 2020-08-26 14:00:51 +03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Aditya 26d344b762 Merge branch 'http2' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-24 14:58:23 +05:30
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Aditya a87ab71d10 refactor(http2): metadata for Stream 
- Add Note about HTTP/2 Cleartext not supported in settings.rst
 2020-08-18 05:52:40 +05:30
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
Aditya d97cf973dd Merge branch 'master' of https://github.com/scrapy/scrapy into h2-client-protocol 2020-08-16 11:31:10 +05:30
Andrey Rahmatullin acb3b4433a
Merge pull request #4724 from Gallaecio/feed-uri-params 
Document FEED_URI_PARAMS
 2020-08-14 14:47:55 +05:00
Mikhail Korobov 61653418ff
Merge pull request #4090 from Gallaecio/documentation-build 
Fix references to Python types in parameter type fields
 2020-08-14 00:58:06 +05:00
Mikhail Korobov cd0d3fd48d
Merge pull request #4705 from Gallaecio/deprecation-policy 
Cover our deprecation policy in the documentation
 2020-08-14 00:27:37 +05:00
Adrián Chaves 65e0abaea5 Document FEED_URI_PARAMS 2020-08-13 09:05:51 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves 5138f9a965 Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-08-11 13:42:05 +02:00
Adrián Chaves b2f4df5cb7 Merge remote-tracking branch 'upstream/master' into documentation-build 2020-08-11 13:16:14 +02:00
Aditya c67d6dea31 fix: H2 docs, NotImplementedError for H2 Tunnel 2020-08-11 04:39:41 +05:30
Adrián Chaves 4ee538e44b
Update unicode references from Python 2 times in the documentation (#4703) 2020-08-04 20:34:11 +02:00
Adrián Chaves 3600582f56
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708) 2020-08-04 20:05:56 +02:00
Aditya d707f8b5d9 docs: mention H2DownloadHandler in settings.rst 2020-07-30 18:06:21 +05:30
Adrián Chaves 6f4ccec567 Cover our deprecation policy in the documentation 2020-07-30 14:03:14 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
BroodingKangaroo a6c1d79b7c pep8 tiny changes 2020-07-28 11:53:05 +03:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Adrián Chaves d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) (#4504) 
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)

* Log a detailed error message to discover why MockServer is not working

* Go for all lines!

* Disable tests based on mitmproxy while running on PyPy

* Fix test_get_func_args for PyPy 3.6+

* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers

* Crossing fingers…

* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher

* Test the PyPy version currently documented as the minimum required version

* Fix the PYPY_VERSION tag

* Update the documentation about supported PyPy versions

* Also test the latest 3.5 Python version with PyPy

* Fix the PYPY_VERSION value for the latest 3.5 version

* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions

* Fix PyPy installation for the pypy3-pinned Tox environment

* Try installing Cython

* Maybe PyPy requires lxml 3.6.0?

* install.rst: minor clarification

* lxml 4.0.0 is required on PyPy

* Require setuptools 18.5+

* Revert "Require setuptools 18.5+"

This reverts commit 017ec33ac2.

* Maintain lxml as a dependency if setuptools < 18.5 is used
 2020-07-17 02:19:24 +05:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
BroodingKangaroo 41263f61c6 Change single quotes to double in example in docs 2020-07-16 18:41:45 +03:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
BroodingKangaroo 8bdcdb0a76 Add quotes to example in docs 2020-07-16 09:13:54 +03:00
Júlio César Batista 38496a00b7
Use the itemlaoders library (#4516) 2020-07-15 12:08:36 +02:00
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Eugenio Lacuesta 2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse 2020-07-08 14:00:54 -03:00
BroodingKangaroo 6454d456d2 Make check of placeholder less strict 2020-07-03 10:24:13 +03:00
BroodingKangaroo 1e245046ed Change setting name. Add leading zeroes to batch_id. Minor fixes. 2020-07-02 12:38:08 +03:00
Andrey Rahmatullin af55d23167
Update the OpenSSL cipher list format link 
OpenSSL `ciphers(1)` is now almost empty: https://www.openssl.org/docs/manmaster/man1/ciphers.html

Alternative would be linking to 1.1.1 docs specifically.
 2020-07-01 19:46:54 +05:00
BroodingKangaroo 7b1d3c35ea Minor updates 2020-07-01 12:24:31 +03:00
Eugenio Lacuesta 5b88c522ac
Simplify dataclass example in item loader docs 2020-06-30 12:18:21 -03:00
BroodingKangaroo 05c2587c6a Docs update and tiny fixes 2020-06-28 09:45:45 +03:00
BroodingKangaroo c3cee74fd4 Change default value of FEED_STORAGE_BATCH_ITEM_COUNT to 0 2020-06-26 18:45:21 +03:00
BroodingKangaroo 77333666f1 resolve conflicts 2020-06-26 14:55:46 +03:00
Adrián Chaves 6f4c964aa4
Cover Scrapy 2.2.0 in the release notes (#4630) 2020-06-24 12:26:38 +02:00
Adrián Chaves cfd039aeb6 Remove a duplicate GCS_PROJECT_ID reference target 2020-06-22 19:28:33 +02:00
Eugenio Lacuesta 73b6ce8cb5
Update docs about dataclass items and loaders 2020-06-22 14:13:37 -03:00
Eugenio Lacuesta 1335d9053e
Update docs/topics/loaders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-06-22 14:05:44 -03:00
Eugenio Lacuesta 3efea98e05
Docs: add note about dataclass items and loaders 2020-06-22 12:41:25 -03:00
Eugenio Lacuesta c7f1c7e3f3
Merge branch 'master' into feat-685 2020-06-19 21:30:54 -03:00
Adrián Chaves f9c63384c0
Merge branch 'master' into allow-customizing-export-column-names 2020-06-18 14:09:20 +02:00
Adrián Chaves 88ade19675
Merge branch 'master' into spider-name-collision 2020-06-18 14:01:02 +02:00
Devi Sandeep 5d54173187
Update docs on accessing callback arguments in errback (#4634) 2020-06-18 12:01:38 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta 8b549392f9
Bump minimum Python version to 3.5.2 (#4615) 2020-06-11 14:53:59 +02:00
Eugenio Lacuesta b6c5289fb9
Close page in pyppeteer example, mention asyncio reactor 2020-06-10 12:11:49 -03:00
Eugenio Lacuesta 78aa1b2bfc
Fix typo 2020-06-08 11:19:15 -03:00
Eugenio Lacuesta bdabc500aa
Update headless browser docs 2020-06-06 16:47:15 -03:00
Mikhail Korobov 1d2b692666
Merge pull request #4596 from sharmarohit/remove-unneeded-escape-sequence 
remove unneeded escape sequence
 2020-06-02 02:30:33 +05:00
Adrián Chaves c86a1035dd
Merge pull request #4563 from willbeaufoy/hoverxref 
Extend hoverxref_roles to custom crossrefs (#4495)
 2020-06-01 07:18:13 +02:00
Bulat Khabibullin 5cef927944
Implement TextResponse.json() (#4574) 2020-06-01 06:57:23 +02:00
Mikhail Korobov 13215cfb71
Merge pull request #4559 from elacuesta/cancel-request-download 
Stop response download from signal handler
 2020-05-27 21:46:00 +05:00
Andrey Rahmatullin 7bf375098e
Merge pull request #2400 from elacuesta/keep_cookie_header 
[MRG+1] CookiesMiddleware: keep cookies from 'Cookie' request header, fix encoding
 2020-05-27 21:41:43 +05:00
Eugenio Lacuesta 492197e477
StopDownload: store response in the exception instead of the failure 2020-05-27 10:45:07 -03:00
Eugenio Lacuesta 634ad5ebca
Merge remote-tracking branch 'upstream/master' into cancel-request-download 2020-05-27 10:27:45 -03:00
Will Beaufoy 898bdd3feb Update docs README with build instructions using tox 
None of the existing commands built the docs properly for me (I had to
revert the changes in 901892d to docs/conf.py to get them to build
properly, and even then no tooltips displayed).

Building them with tox worked for me, but other developers say they
can still use the original method, so the docs now contain both.
 2020-05-26 19:09:23 +01:00
Rohit Sharma 0cc0e51ca3 remove uneeded escape sequence 
removed uneeded wscape sequences from method arguments in the docs
folder
 2020-05-24 13:38:44 -07:00
Adrián Chaves 08756cd87a Fix the description of the ScreenshotPipeline example item pipeline 2020-05-22 18:06:54 +02:00
Eugenio Lacuesta f6ed5edc31
CookiesMiddleware: keep cookies from 'Cookie' request header 2020-05-21 11:07:41 -03:00
BroodingKangaroo 5f6c1dceb1 Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-21 12:43:04 +03:00
Eugenio Lacuesta 276721a5dc
Stop response download 2020-05-20 14:46:26 -03:00
Adrián Chaves f418c6d5f7 logging.rst: remove unused, misleading import 2020-05-19 20:04:47 +02:00
nsirletti c2a0cca0fe Suppress Sphinx warnings about MIME types other than .rst 2020-05-19 02:17:11 +02:00
Mikhail Korobov febe82a907
Merge pull request #4486 from ilias-ant/add-file-status-on-media-pipelines-file-info 
Add status (downloaded, uptodate) to files information
 2020-05-18 22:30:30 +05:00
Mikhail Korobov 14612fc3d3
Merge pull request #4534 from elacuesta/deprecate-baseitem 
Deprecate scrapy.item.BaseItem
 2020-05-16 03:24:06 +05:00
BroodingKangaroo 10ae1a284f Minor fixes 2020-05-15 22:50:54 +03:00
Adrián Chaves 36c3c9713e
Run tests with Python 3.5.0 (#4518) 
* Run tests with Python 3.5.0

* Use Ubuntu Trusty 14.04 to test Python 3.5.0

* Use mitmproxy<3.0.0 on Python 3.5.0

* Skip tests requiring mitmproxy in Python 3.5.0

* Change the minimum Python version from 3.5 to 3.5.1

* Do not prevent Scrapy from working with Python 3.5.0

* Force system Python 3.5.1

* Do not install a non-system Python in Python 3.5.1 jobs

* Switch to Trusty to be able to test Xenial’s Python version ¯\_(ツ)_/¯

* Add missing trusty

* Stop breaking old PyPy

* Allow installing Scrapy on Python 3.5.0
 2020-05-15 22:37:56 +05:00
BroodingKangaroo 1cdcf8b08b Minor fixes 2020-05-15 20:14:07 +03:00
BroodingKangaroo 69c005f013 Documentation indent fix 2020-05-14 10:35:56 +03:00
BroodingKangaroo 111a58fe3d Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries 2020-05-14 09:53:14 +03:00
BroodingKangaroo 8662d3587d Documentation and code refactoring 2020-05-14 00:05:36 +03:00
BroodingKangaroo 2327ecead0 Rename FEED_STORAGE_BATCH_SIZE to FEED_STORAGE_BATCH_ITEM_COUNT 2020-05-13 22:50:04 +03:00
Adrián Chaves 7dac99bb12
Merge branch 'master' into deprecate-baseitem 2020-05-13 20:35:03 +02:00
Aditya Kumar 8d1269bcbc
Cover chompjs in documentation (#4562) 2020-05-12 20:42:28 +02:00
nsirletti cb8140a42a
Deprecate Response.body_as_unicode() (#4555) 
Co-authored-by: Nicolas Sirletti <n.sirletti@gmail.com>
 2020-05-11 20:20:31 +02:00
Eugenio Lacuesta fb7ba6964d
Merge branch 'master' into deprecate-baseitem 2020-05-11 10:21:56 -03:00
Mikhail Korobov b183579564
Merge pull request #4205 from elacuesta/bytes_received_signal 
Add bytes_received signal
 2020-05-11 15:09:55 +05:00
Will Beaufoy 1bfbcc6180 Extend hoverxref_roles to custom crossrefs (#4495) 2020-05-10 13:49:13 +01:00
Adrián Chaves e07708e374 request-response: update the consequences of str(b'') 2020-05-09 15:54:31 +02:00
Adrián Chaves c6746f0e38 bytes array → bytes object 2020-05-09 15:51:11 +02:00
Eugenio Lacuesta 9485724d98
Merge branch 'master' into deprecate-baseitem 2020-05-08 19:32:47 -03:00
Mikhail Korobov 794ab19660
Merge pull request #3727 from Gallaecio/pylint 
Use pylint
 2020-05-09 02:22:54 +05:00
Antonio Gordillo Toledo cf09af787e
Remove Python 2 encoding header from files (#4553) 2020-05-08 15:45:19 +02:00
Eugenio Lacuesta 6160370927
Merge remote-tracking branch 'upstream/master' into deprecate-baseitem 2020-05-07 14:38:54 -03:00
Adrián Chaves bbd9d05dbf request-response.rst: review type references around body mentions 2020-05-07 11:44:43 +02:00
Eugenio Lacuesta 7fd2085725
Merge remote-tracking branch 'upstream/master' into bytes_received_signal 2020-05-06 22:41:16 -03:00
Adrián Chaves a19d15013a Merge remote-tracking branch 'upstream/master' into spider-name-collision 2020-05-06 20:52:36 +02:00
Adrián Chaves a859ea0c8b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2020-05-06 19:48:32 +02:00
Adrián Chaves a79cfce57b
Merge branch 'master' into pylint 2020-05-06 16:24:13 +02:00
Eugenio Lacuesta 8d1e3ee0dd
Remove deprecated BaseItem from the docs 2020-05-06 09:24:40 -03:00
Eugenio Lacuesta 7988c676a9
Update Item docstring, update BaseItem occurrences 2020-05-05 13:11:01 -03:00
Mikhail Korobov fe6154e4fa clarify DOWNLOADER_HTTPCLIENTFACTORY docs 2020-05-04 18:18:38 +05:00
Eugenio Lacuesta e1948b4923
Add example about bytes_received signal 2020-05-04 09:07:27 -03:00
Mikhail Korobov 83d7360bb7 Don't mention unsupported package versions in docs 2020-05-04 02:00:11 +05:00
BroodingKangaroo dad2ea7522 Change time_id to batch_time 2020-05-02 01:21:03 +03:00
Eugenio Lacuesta 4d625afda5
Merge branch 'master' into bytes_received_signal 2020-05-01 16:11:32 -03:00
BroodingKangaroo df81870f39 Merge branch 'master' of https://github.com/scrapy/scrapy into ISSUE-4250-add_batch_deliveries 2020-05-01 12:18:39 +03:00
Adrián Chaves ec0a0eb3ab
Merge branch 'master' into spider-name-collision 2020-04-28 21:13:08 +02:00
Adrián Chaves 1006db1e10
Merge branch 'master' into documentation-build 2020-04-28 21:07:14 +02:00
Ashe c207dbf939
Remove the asyncio warning from coroutines page (#4513) 2020-04-27 19:45:19 +02:00
BroodingKangaroo a18621552f Merge branch 'master' of https://github.com/BroodingKangaroo/scrapy into ISSUE-4250-add_batch_deliveries 2020-04-27 10:34:36 +03:00
BroodingKangaroo 2eee6c8101 Documentation spelling fix 2020-04-27 09:58:14 +03:00
BroodingKangaroo f0f1be76d1 Using time_id instead of time as a timestamp 2020-04-27 09:56:57 +03:00
Adrián Chaves ffe576c4ed
Cover Scrapy 2.1 in the release notes (#4499) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-04-24 11:44:36 +02:00
Aditya Kumar e4750f2fbd
async/deferred signal handlers (#4390) 
* [docs] async/deferred signal handlers

* [docs] update deferred signals example

* [docs] add subsections for built-in signals

* docs(signals): update signal handler example

* docs(signals): update signal handler example
 2020-04-20 20:47:57 +05:00
sakshamb2113 bfeb2c8c13
Added warning to use double quotes in Windows for scrapy shell in shell.rst (#4450) 
* modified debugging memory leaks with guppy in leaks.rst

* modified leaks.rst(issue #4285)

* removed guppy from telnet.py

* Fix undefined name error

* removed hpy key from telnet_vars in telnet.py

* updated shell.rst

* Update docs/topics/shell.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-04-18 20:21:26 +05:00
Eugenio Lacuesta 04b6295a69
Docs: replace deprecated FEED_* settings 2020-04-17 20:50:17 -03:00
Eugenio Lacuesta 1f2e2a6006
Merge branch 'master' into response_ip_address 2020-04-16 11:32:37 -03:00
Adrián Chaves 901892dab3 Fix the hoverxref configuration 2020-04-16 14:48:38 +02:00
BroodingKangaroo cac1f3a6ad Update documentation 2020-04-16 10:06:56 +03:00
Mikhail Korobov 36abe9235c
Merge pull request #4445 from adityaa30/docs-redirect-links 
update redirect links to python3
 2020-04-15 23:13:55 +05:00
Mikhail Korobov 066c02d0c3
Merge pull request #4455 from aditi137/docs-curl2scrapy 
[Docs] mention curl2scrapy in Request.from_curl
 2020-04-15 23:12:33 +05:00
Eugenio Lacuesta ac869181fb
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-04-15 13:42:35 -03:00
Eugenio Lacuesta ee4ee486b1
Revert unnecessary changes to docs/conf.py 2020-04-14 15:06:54 -03:00
Eugenio Lacuesta 0a4ef97fa3
Loose restrictions for docs requirements 2020-04-14 14:57:20 -03:00
ilias-ant 83a0cc6cdf Add status to files information 2020-04-12 23:22:17 +03:00
Eugenio Lacuesta 2205f04631
Docs: Add hoverxref_role_types setting 2020-04-10 18:08:04 -03:00
Eugenio Lacuesta 34e81d0d74
Docs: remove duplicated setting definitions 2020-04-10 17:29:02 -03:00
Eugenio Lacuesta 4383f45299
Replace os.path with pathlib in docs config 2020-04-10 16:49:14 -03:00
Eugenio Lacuesta 24a1d9acae
Get version in docs config 2020-04-10 16:48:42 -03:00
Eugenio Lacuesta f97fec5ebd
Pin Sphinx version, including extensions 2020-04-10 16:02:53 -03:00
Eugenio Lacuesta c4a5e3f0da
Simplify bytes_received signal 
Remove "source" parameter
 2020-04-06 09:26:13 -03:00
Eugenio Lacuesta e3342669ae
Merge branch 'master' into bytes_received_signal 2020-04-06 09:17:35 -03:00
Aditya b1904729d5 [docs] change mod to doc redirect link 2020-03-27 04:37:26 +05:30
Aditi Dutta 010edfe85c [Docs] mention curl2scrapy in Request.from_curl 2020-03-25 14:38:22 -04:00
Aditya 80c69d68ad [docs] refactor python docs links using intersphinx 2020-03-24 18:35:42 +05:30
Aditya f37b1bdc56 [docs] update redirect links to python3 2020-03-20 05:31:41 +05:30
Aditya ca08e04198 [docs] update redirect links python2 -> python3 2020-03-20 02:31:35 +05:30
Eugenio Lacuesta 34d607194b
Merge branch 'master' into feat-685 2020-03-18 21:24:36 -03:00
Adrián Chaves 6c747953f9 Cover 2.0.1 in the release notes (#4437) 2020-03-18 19:17:16 +01:00
sakshamb2113 533131a30f
Remove Guppy-specific code and documentation (#4343) 2020-03-17 10:12:49 +01:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Eugenio Lacuesta aa6075e08c
Merge branch 'master' into bytes_received_signal 2020-03-15 18:05:04 -03:00
Eugenio Lacuesta bde0384dfd
Merge branch 'master' into spider.parse 2020-03-14 19:53:36 -03:00
Aditya Kumar f9bf4b8d4d
Remove all top-level imports for twisted.internet.reactor (#4406) 2020-03-14 10:39:00 +01:00
Eugenio Lacuesta 21f1eddc90
Merge branch 'master' into bytes_received_signal 2020-03-11 21:16:35 -03:00
Eugenio Lacuesta 49156f2ecb
[doc] Feed exports: full local path as example 2020-03-11 20:45:54 -03:00
Eugenio Lacuesta ada37c5409
Export to multiple formats in a single crawl 2020-03-05 09:33:57 -03:00
Eugenio Lacuesta 3aa5eab993
Merge branch 'master' into response_ip_address 2020-03-03 13:53:43 -03:00
Adrián Chaves 6aa0ba4553
Write release notes for Scrapy 2.0.0 (#4329) 2020-03-03 09:11:11 +01:00
Adrián Chaves 431f6e7d90
Merge branch 'master' into patch-1 2020-03-01 18:17:18 +01:00
MaliCN ef00f8eb8e
updated with new macOS name (#4308) (#4323) 
* changed for new name as "macOS"  (issue #4308)

* updated macOS name

* update macOS name

* updated macOS name

* update for new macOS name

* docs/intro/install.rst: fix macOS header symbols

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-29 00:42:07 +05:00
Adrián Chaves 231c9ddef8
Update docs/intro/install.rst 2020-02-28 18:50:45 +01:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
Eugenio Lacuesta a44942d27d
Merge branch 'master' into response_ip_address 2020-02-23 18:13:52 -03:00
elacuesta 9d983c1b99
Expose certificate for HTTPS responses (#4054) 
* Expose certificate for HTTPS responses

* Fix test (missing inlineCallbacks decorator)

* Note about Response.certificate

* Explicitly cover None as the default value of Response.certificate

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-22 17:20:31 +05:00
Andrey Rahmatullin 88179027de
Merge pull request #4331 from Gallaecio/response-cb-kwargs 
Implement Response.cb_kwargs
 2020-02-19 22:40:14 +05:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00
leobalestri 6d6243afbb
Update install.rst 
Minor grammar and typo fixes
 2020-02-16 23:45:41 -08:00
Adrián Chaves a04dd13cd0
ie. → i.e. 2020-02-14 22:31:30 +01:00
Adrián Chaves 5ae3e1678f
ie. → i.e. 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-14 22:30:36 +01:00
Adrián Chaves df937d8280 Implement Response.cb_kwargs 2020-02-13 22:33:36 +01:00
Adrián Chaves 037ae5b22e
Explicitly indicate None as ip_address’s default value 2020-02-10 19:54:47 +01:00
Adrián Chaves 96bb3b5144
Merge branch 'master' into feat-685 2020-02-10 19:45:09 +01:00
Adrián Chaves 7025c18b15
Clear line of spaces 2020-02-10 19:43:23 +01:00
Eugenio Lacuesta 13ba9bc629
Note about Response.ip_address 2020-02-10 12:29:39 -03:00
Eugenio Lacuesta 42b4e9b337
Reword signal docs 2020-02-10 11:23:38 -03:00
Adrián Chaves c2f484de88
Merge branch 'master' into response_ip_address 2020-02-07 21:49:38 +01:00
Adrián Chaves aae4935605
Merge branch 'master' into spider.parse 2020-02-07 21:17:56 +01:00
Adrián Chaves 59653ebac6 Update installation instructions regarding Python 3 and virtual environments 2020-02-07 21:07:57 +01:00
Mikhail Korobov b0eaf114e5
Merge pull request #4197 from elacuesta/sphinx-twisted-api 
[Docs] Fix Twisted links
 2020-02-07 23:51:15 +05:00
Mikhail Korobov c3b690a5b5
Merge pull request #4290 from dekimsey/patch-1 
FilesPipeline.file_path has optional arguments
 2020-02-07 23:41:31 +05:00
Mikhail Korobov afbaf9d430
Merge pull request #4303 from whalebot-helmsman/request_left_downloader_signal 
request_left_downloader signal
 2020-02-07 23:33:51 +05:00
Vostretsov Nikita 153b78e53f
Update docs/topics/signals.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-07 11:08:55 +05:00
Vostretsov Nikita 8817b9e8e9
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:53 +05:00
Vostretsov Nikita 2f83f3e2cb
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:43 +05:00
Vostretsov Nikita 84b55b7364
Update docs/topics/signals.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2020-02-07 11:07:35 +05:00
Eugenio Lacuesta 6c333491a8
Merge branch 'master' into response_ip_address 2020-02-06 18:53:11 -03:00
Joy Bhalla 4f31c3ce01
Document a backward incompatibility that may affect custom schedulers (#4274) 2020-02-06 22:21:33 +01:00
Adrián Chaves 24bb9fd5f7
Merge branch 'master' into spider.parse 2020-02-06 21:39:09 +01:00
Mikhail Korobov 042e71e2b8
Merge pull request #4311 from Gallaecio/metarefresh-ignore-tags 
Make METAREFRESH_IGNORE_TAGS an empty list by default
 2020-02-06 23:40:45 +05:00
Mikhail Korobov 8d2705f23c
Merge pull request #4305 from Respawnz/patch-1 
fix a typo in devloper-tools.rst
 2020-02-06 23:17:28 +05:00
elacuesta 35dafef7f1
Specify Twisted reactor (TWISTED_REACTOR setting) (#4294) 
* Add the ability to install a specific reactor

* Add docs for the TWISTED_REACTOR setting

* Add tests for the TWISTED_REACTOR setting

* Update asyncio reactor test

* Ignore W503 globally

W503 is not PEP8-compliant:
c59c4376ad

* Line length adjustment

* Adjust asyncio reactor tests

* Merge ASYNCIO_ENABLED and TWISTED_REACTOR settings

* More docs about TWISTED_REACTOR

* Fix asyncio reactor test

* Docs: fix title

* Reword docs

* Check the TWISTED_REACTOR setting outside of the installing function

* Remove unrelated change

* Update scrapy/utils/log.py

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Update docs/topics/settings.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-02-06 22:42:34 +05:00
Vostretsov Nikita 6733f4d976
Update docs/topics/signals.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2020-02-06 18:40:42 +05:00
Adrián Chaves 576663e5a7 Make METAREFRESH_IGNORE_TAGS an empty list by default 2020-02-06 10:43:20 +01:00
Respawnz c2cca36821
typo 2020-02-06 05:39:15 +08:00
Vostretsov Nikita aab39f6341 docummentation for new signal 2020-02-05 11:35:03 +00:00
Eugenio Lacuesta b9e3a6201b
Merge branch 'master' into response_ip_address 2020-02-03 04:07:44 -03:00
Mikhail Korobov 47b9de93a9
Merge pull request #4288 from petervandenabeele/patch-1 
[Docs] 2 typos + 1 clarification in docs
 2020-01-30 03:15:18 +05:00
Eugenio Lacuesta 6f02a8dccb
Add source parameter to bytes_received signal 2020-01-29 15:51:25 -03:00
Daniel Kimsey 752e8f7018 FilesPipeline.file_path has optional arguments 
Documented signature doesn't match the actual interface in [files.py](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L520).

Specifically, it looks like it may be [called](https://github.com/scrapy/scrapy/blob/master/scrapy/pipelines/files.py#L422) without a response value.

I found this when I was implementing the pipeline with the signature `file_path(self, request, response, info)` and the following error was being return in my results :

    [(False, <twisted.python.failure.Failure builtins.TypeError: file_path() missing 1 required positional argument: 'response'>)]

Scrapy==1.8.0
 2020-01-28 15:21:15 -06:00
Eugenio Lacuesta 4ffd18fb11
[docs] Mention that signals.bytes_received could be fired multiple times 2020-01-27 13:29:33 -03:00
Eugenio Lacuesta dbe20a863f
bytes_received signal: send spider argument 2020-01-27 12:28:54 -03:00
Eugenio Lacuesta 72b8613ee9
bytes_received signal (no tests) 2020-01-27 12:06:48 -03:00
Eugenio Lacuesta 8529dff41d
Update docs regarding Response.ip_address and IPv6 2020-01-26 18:00:56 -03:00
Eugenio Lacuesta 80925ab845
Get server IP address for HTTP/1.1 responses 2020-01-26 16:57:04 -03:00
Peter Vandenabeele f72d4e93e6
[Docs] 2 typos + 1 clarification in docs 
Fixing 2 small typos and adding 1 word as clarification in the downloader-middlewares.

Also, I was confused with the entries like `ref:Reppy <reppy-parser>` and similar entries.
Are these supposed to be links to other parts of the doc, or is this the intended way of showing these references ?
 2020-01-26 10:48:28 +01:00
Peter Vandenabeele f3374a5047 Fix variable name `author_page_links` 
I did not test this code, but the change from `href` to this
author_page_links seems to have a typo ?
 2020-01-25 16:53:29 +01:00
Mikhail Korobov 8b8df31961
Merge pull request #4227 from elacuesta/name-resolver 
Name resolver with IPv6 support
 2020-01-25 05:03:13 +05:00
OmarFarrag fc98aa6b67
Merge branch 'master' into ftp_files#3928 2020-01-24 14:52:40 +02:00
Mikhail Korobov bd54f22fef
Merge pull request #4282 from petervandenabeele/patch-1 
fix logical documentation error with PER_DOMAIN or PER_DOMAIN
 2020-01-24 02:01:22 +05:00
Mikhail Korobov 6a98d660e5
Merge pull request #3551 from jpbalarini/change_scraper_slot 
[MRG+1] Add ability to change max_active_size by setting
 2020-01-23 23:40:05 +05:00
Peter Vandenabeele 7d5cebcf77
fix logical documentation error with PER_DOMAIN or PER_DOMAIN 2020-01-23 09:08:21 +01:00
Eugenio Lacuesta c75cf15b7a
Update CSS selectors in tutorial 2020-01-22 10:38:59 -03:00
JP Balarini 0f2d871d88 Use PEP 515 style for SCRAPER_SLOT_MAX_ACTIVE_SIZE documentation 2020-01-20 11:28:28 -03:00
Juan Pablo Balarini eaa8ed02d0 Add ability to change max_active_size by settings 2020-01-20 11:27:58 -03:00
Eugenio Lacuesta b471765d40
[doc] FAQ entry about the IPv6 and the DNS_RESOLVER setting 2020-01-18 01:52:29 -03:00
Eugenio Lacuesta 302d3f552b
[doc] DNS_RESOLVER setting 2020-01-18 01:46:24 -03:00
Eugenio Lacuesta 2e405d2d5c
Merge branch 'master' into response_follow_all 2020-01-05 00:33:19 -03:00
1um0s 14d4428e70 Rephrasing documentation for image and file pipelines (#4252) 
* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Clarify documentation for image and file pipelines

* scrapy#4034 Clarify documentation for file pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Simplify documentation for pipeline

* scrapy#4034 Revert image, file pipeline docs. Enhance custom media pipeline docs.

* scrapy#4034 rebase master

* Rebase master

* Add class to media pipeline docs

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>

Co-authored-by: elacuesta <elacuesta@users.noreply.github.com>
 2019-12-30 00:56:22 +05:00
Mikhail Korobov f0ae673452
Merge pull request #4258 from atul-g/patch-1 
Edited the link provided to homepage of lxml's website
 2019-12-30 00:55:15 +05:00
Atul Gopinathan 82861c73c8
Edited the link of the homepage of lxml website 
The link "https://lxml.de" is redirecting to a completely different and unintended website. I changed the link to the index page of lxml's official website. I thought of changing it to the PyPi page of lxml, but even they are providing the same "https://lxml.de" link which doesn't seem to be working now.
 2019-12-27 22:57:58 +05:30
Andrey Rakhmatullin dc1ee09481 Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly. 2019-12-27 21:56:28 +05:00
Eugenio Lacuesta 8a1dc26d46
[doc] Note about the 'parse' method for CrawlSpider/XMLFeedSpider 2019-12-26 15:14:47 -03:00
Eugenio Lacuesta 5982e3477c
Take keyword arguments in base parsing methods 2019-12-23 14:12:21 -03:00
Andrey Rakhmatullin 20289be810 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-20 19:31:20 +05:00
Mikhail Korobov fb3fb17c28
Merge pull request #4045 from Gallaecio/documentation-coverage 
Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors
 2019-12-20 02:01:45 +05:00
Mikhail Korobov 6a32a36e09
Merge pull request #4143 from Gallaecio/undent-rst-doctests 
Do not indent doctests from the documentation unnecessarily
 2019-12-20 01:59:07 +05:00
Adrián Chaves b4a2d985a7
Merge branch 'master' into undent-rst-doctests 2019-12-19 12:39:17 +01:00
Marc Hernandez Cabot 16b363de31 Merge branch 'master' into fix_typos 2019-12-19 10:02:21 +01:00
Marc Hernandez Cabot 23a67cec27 fix first letter capitalization for Raring and Scrapy 2019-12-19 09:57:17 +01:00
Mikhail Korobov 8f7faaa63d
Merge pull request #4190 from Gallaecio/doctest 
Make developer-tools doctests pass
 2019-12-18 23:40:21 +05:00
Marc Hernandez Cabot c0d84f0962 fix typos 2019-12-18 19:39:21 +01:00
elacuesta 916382e109 Add errback parameter to scrapy.spiders.crawl.Rule (#4000) 
* Add errback parameter to scrapy.spiders.crawl.Rule

* CrawlSpider: optimize by reducing iterations

* [test] Rule.errback

* [doc] Rule.errback

* [doc] Use autoclass in docs/topics/spiders.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

* Rule.process_links takes a list

* Fix aesthetic issue reported by Flake8
 2019-12-18 20:05:33 +05:00
Adrián Chaves 8c5a8a02ae
Merge branch 'master' into pylint 2019-12-18 15:43:40 +01:00
Mikhail Korobov 053319334c
Merge pull request #4179 from Gallaecio/user-friendlier-tox 
Make tox configuration more user friendly
 2019-12-18 18:35:19 +05:00
Vostretsov Nikita 012533924a remove requirements from here too 2019-12-18 11:13:36 +00:00
Adrián Chaves 607815dfc2
Merge branch 'master' into documentation-coverage 2019-12-17 16:54:27 +01:00
Adrián Chaves b8cf522916
Merge branch 'master' into user-friendlier-tox 2019-12-17 16:53:08 +01:00
Adrián Chaves ab13221b0b Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2019-12-17 14:06:14 +01:00
marc a59bb279d1 add year through code 2019-12-15 17:33:00 +01:00
marc 1aab20e1ce update copyright notice year 2019-12-14 10:37:31 +01:00
marc e3a3ad4aaf remove reference to old (Python 2.7) environment 2019-12-14 10:34:31 +01:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Andrey Rakhmatullin 352ddbb336 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-13 18:07:23 +05:00
Eugenio Lacuesta 21a2f8c6f8
Merge branch 'master' into response_follow_all 2019-12-10 17:30:59 -03:00
Adrián Chaves 1fc2b140c1
Merge branch 'master' into documentation-coverage 2019-12-05 14:43:36 +01:00
Adrián Chaves d7b1c138f1
Merge branch 'master' into user-friendlier-tox 2019-12-05 14:02:24 +01:00
Adrián Chaves 83b8046fdc Do not indent doctests from the documentation unnecessarily 2019-12-05 13:52:31 +01:00
Adrián Chaves 57e36b5f0d
Merge branch 'master' into spider-name-collision 2019-12-05 13:03:43 +01:00
Wang Qin af624ef414
Update overview.rst | Fix an inconsistency 
There exists an inconsistency between the code (line 37 - 38) and the output 'quotes.json' (line 56 - 68). 

Note that even though according to line 53 - 54  'quotes.json' is "reformatted here for better readability", it cannot explain why the "author" field precedes the "text" field. 

Intended output for the code BEFORE change: 
    [{
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d",
        "author": "Jane Austen"
    },
    {
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d",
        "author": "Groucho Marx"
    },
    {
        "text": "\u201cA day without sunshine is like, you know, night.\u201d",
        "author": "Steve Martin"
    },
    ...]

Intended output for the code After change (the inconsistency is fixed): 
    [{
        "author": "Jane Austen",
        "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"
    },
    {
        "author": "Groucho Marx",
        "text": "\u201cOutside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.\u201d"
    },
    {
        "author": "Steve Martin",
        "text": "\u201cA day without sunshine is like, you know, night.\u201d"
    },
    ...]
 2019-12-05 09:29:12 +08:00
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Mikhail Korobov 9b7452211a
Merge pull request #4099 from BurnzZ/itemloader-docs 
update docs of scrapy.loader.ItemLoader.item
 2019-12-03 13:14:45 +05:00
Adrián Chaves 5980b0f284 Don’t use follow_all where a single item is expected (#4) 2019-12-02 12:47:44 -03:00
Eugenio Lacuesta 048cd74ae5
Add separate mapping for Twisted API docs 2019-11-27 19:16:18 -03:00
Eugenio Lacuesta 17e6481823
[Docs] Fix Twisted links 2019-11-27 18:42:42 -03:00
Adrián Chaves b73fc99b60 Use InterSphinx for coverage links 2019-11-26 10:31:55 +01:00
Adrián Chaves dd12f5fdcd Use Response.follow_all in the documentation where appropiate 2019-11-25 11:59:59 -03:00
Adrián Chaves 54b056c4be Make developer-tools doctests pass 2019-11-25 12:32:35 +01:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Mikhail Korobov 0d416c6191
Merge pull request #4185 from Gallaecio/intersphinx 
Use InterSphinx for links to the pytest and tox documentation
 2019-11-21 23:27:57 +05:00
Mikhail Korobov 82c01c261f
Merge pull request #4184 from Gallaecio/doctest 
Make debug doctests pass
 2019-11-21 20:53:56 +05:00
Adrián Chaves fcfcabf1bd Use InterSphinx for links to the pytest and tox documentation 2019-11-21 12:15:13 +01:00
Adrián Chaves f251dda268 Make debug doctests pass 2019-11-21 11:59:10 +01:00
Adrián Chaves 4f80eff1e1 Enable sphinx-hoverxref for all references 2019-11-21 10:30:21 +01:00
Mikhail Korobov f1fd7ec318
Merge pull request #4171 from Gallaecio/doctest 
Make command doctests pass
 2019-11-20 19:47:22 +05:00
Mikhail Korobov d69e788fec
Merge pull request #4172 from Gallaecio/intersphinx 
Use InterSphinx to link to the Twisted documentation
 2019-11-20 19:38:50 +05:00
Mikhail Korobov 1f61b6b941
Merge pull request #4173 from Gallaecio/sphinx-extensions-2 
Add tooltips to documentation cross-references
 2019-11-20 19:38:05 +05:00
Eugenio Lacuesta 48223c107a
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-20 09:41:42 -03:00
Adrián Chaves fc3af54dbd Make tox configuration more user friendly 2019-11-20 08:04:17 +01:00
Adrián Chaves fed93515de Add tooltips to documentation cross-references 2019-11-18 16:11:03 +01:00
Adrián Chaves e84cb18ca0 Use InterSphinx to link to the Twisted documentation 2019-11-18 15:50:45 +01:00
Adrián Chaves 74589df02f Make command doctests pass 2019-11-18 14:51:44 +01:00
Adrián Chaves 393a2a1972 Include /requirements-py3.txt from /docs/requirements.txt 2019-11-18 09:15:48 +01:00
Mikhail Korobov a541c329a9
Merge pull request #4152 from Gallaecio/sphinx-extensions 
Install the sphinx-notfound-page Sphinx extension
 2019-11-15 00:36:56 +05:00
Mikhail Korobov 7b4637d083
Merge pull request #4146 from Gallaecio/doctest 
Make tutorial doctests pass
 2019-11-15 00:19:20 +05:00
Mikhail Korobov 49b5251e06
Merge pull request #4147 from Gallaecio/intersphinx 
Use intersphinx for links to the Sphinx documentation
 2019-11-15 00:16:39 +05:00
Mikhail Korobov cc0026f733
Merge pull request #4148 from Gallaecio/doc-view-code 
Allow opening the source code from the API documentation
 2019-11-14 23:43:44 +05:00
Andrey Rakhmatullin dd367438fa Improve the dbm module ref. 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin a9c891399d Fix a duplicate ref name in docs. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 1d7c8cb0b1 Remove six.PY2 and six.PY3 conditionals. 2019-11-14 13:45:37 +05:00
Adrián Chaves 76c31094df Install the sphinx-notfound-page Sphinx extension 2019-11-13 09:28:48 +01:00
Adrián Chaves b642a1fca2
Fix doctest skipping based on the running Python version 2019-11-13 09:14:20 +01:00
Adrián Chaves 414e6e2fd5 Skip a doctest in Python 3.5- because of dictionary changes 2019-11-13 08:40:31 +01:00
Andrey Rahmatullin c911e80209
Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs 
docs: use __init__ method instead of constructor
 2019-11-12 21:48:16 +05:00
Andrey Rahmatullin d8098350f0
Merge pull request #3960 from thernstig/issue_2149 
Update documentation for logging manually
 2019-11-12 16:17:49 +05:00
Adrián Chaves 8a6a063778 Allow opening the source code from the API documentation 2019-11-12 10:23:19 +01:00
Adrián Chaves 7b7bb028f4 Use intersphinx for links to the Sphinx documentation 2019-11-12 08:49:06 +01:00
Adrián Chaves 79d2f99995 Make tutorial doctests pass 2019-11-12 08:08:50 +01:00
Adrián Chaves ecc46ac741
Merge branch 'master' into pylint 2019-11-11 10:45:25 +01:00
Andrey Rahmatullin d267e183b4
Merge pull request #4139 from Gallaecio/jobdir-pickle-documentation 
Improve the details about request serialization requirements for JOBDIR
 2019-11-08 22:49:33 +05:00
Adrián Chaves 1df5755699 Set the bases for testing code examples from the documentation 2019-11-08 16:00:10 +01:00
Adrián Chaves 40ed184914
Merge branch 'master' into pylint 2019-11-07 18:40:58 +01:00
Adrián Chaves aef98188fa Improve the details about request serialization requirements for JOBDIR 2019-11-07 18:06:55 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Adrián Chaves 94f060fcc8
Cover Scrapy 1.8.0 in the release notes (#3952) 2019-10-29 12:53:46 +01:00
Andrey Rahmatullin 18b808b2e9
Merge pull request #4092 from further-reading/master 
Add Python 3.8 official support
 2019-10-29 16:30:58 +05:00
Adrián Chaves 3d0df419c4 Mark the LevelDB storage backend as deprecated 2019-10-28 11:24:47 +01:00
elacuesta 7731814cc2 ItemLoader: improve handling of initial item (#4036) 2019-10-28 10:53:53 +01:00
Adrián Chaves bb91f9c78c Cover Scrapy 1.7.4 in the release notes 2019-10-28 09:58:13 +01:00
Kevin Lloyd Bernal 84fe4011b0 update docs of scrapy.loader.ItemLoader.item 2019-10-23 20:39:53 +08:00
Adrián Chaves 3b6f7ac9f2 Use pylint 2019-10-22 19:44:11 +02:00
Adrián Chaves 5479e7ecc7 Indicate that lists of emails may be provided as a single string or as a list of strings 2019-10-22 15:24:44 +02:00
Ammar Najjar d21e1034f0
docs: correct point,comma and plural replacements 
Issue #4086
 2019-10-22 13:29:38 +02:00
WinterComes 5d75ed4cba Remove an old note about contracts (#4093) 2019-10-22 12:19:07 +02:00
illgitthat 07822935ec Updating link for miniconda (#4089) 2019-10-22 12:05:34 +02:00
Ammar Najjar da8cd9448d
docs: always surround __init__ with `` in docs 
Issue #4086
 2019-10-21 19:56:58 +02:00
Adrián Chaves ad6075440c Fix references to Python types in parameter type fields 2019-10-21 19:00:03 +02:00
Ammar Najjar 68a7d05ed8
docs: use __init__ method instead of constructor 
Issue #4086
 2019-10-21 15:42:24 +02:00
Adrián Chaves 0fbd1ff4a9 constructor → __init__ method 2019-10-21 14:06:45 +02:00
Mikhail Korobov 8cb53441b0
Merge pull request #4056 from Gallaecio/documentation-build 
Fix internal links in the tutorial and release notes
 2019-10-19 00:00:02 +02:00
Bulat 84be6a941e Refactor sentence. 2019-10-16 14:04:07 +03:00
Bulat c9614a5bdd Fixed BOT_NAME documentation 2019-10-16 12:07:19 +03:00
Adrián Chaves d72ed46fe0 Improve how extra Item API members are introduced in the documentation 2019-10-15 16:04:26 +02:00
Baron Hou 7b1e69dec4 reponse → response (#4079) 2019-10-15 14:51:15 +02:00
Eugenio Lacuesta 5f168cd459
Response.follow_all 2019-10-08 16:32:56 -03:00
John Bampton 39c9a3cc1c Fix case of GitHub. 2019-10-05 10:09:14 +10:00
Adrián Chaves a4aa5b8926 Fix internal links in the tutorial and release notes 2019-10-02 14:51:06 +02:00
elacuesta 6ad5a89cb0 [Doc] Use autoclass in topics/request-response.rst (#4055) 2019-10-02 12:18:36 +02:00
Mikhail Korobov 74b4a5c77c
Merge pull request #4006 from whalebot-helmsman/protego-default 
Use protego as a default robots.txt parser
 2019-10-02 00:03:50 +05:00
Kristobal Junta c232bbdc42 fix typo in docs/topics/spiders 2019-10-01 17:41:38 +03:00
OmarFarrag 175cd2ece5
Update docs/topics/media-pipeline.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-10-01 07:27:31 +02:00
Adrián Chaves 7f4f98fd38 Provide complete API documentation coverage of scrapy.linkextractors 2019-09-30 18:22:28 +02:00
OmarFarrag 28005b2872
Update media-pipeline.rst 2019-09-28 06:21:14 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
OmarFarrag b14c3cb612
Add media pipelines FTP documentation 2019-09-19 23:33:57 +02:00
Adrián Chaves c26a9015ad Clarify the effects of dont_merge_cookies 2019-09-19 11:08:06 +02:00
Adrián Chaves 9b65f9aa5b
Fix the item exporter example (#4022) 2019-09-19 09:17:23 +02:00
Vostretsov Nikita 2438ac529a
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:22 +05:00
Vostretsov Nikita d1d0bf8491
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-09-17 12:27:12 +05:00
Vostretsov Nikita 57e6f4c750 add link to performance comparison 2019-09-17 07:18:37 +00:00
Vostretsov Nikita d39ef77e6b add link to google description of lenght-based rule 2019-09-17 06:34:33 +00:00
Vostretsov Nikita b792dba528 remove periods 2019-09-17 06:28:33 +00:00
Vostretsov Nikita 50c85e3114 Merge branch 'protego-default' of github.com:whalebot-helmsman/scrapy into protego-default 2019-09-16 14:24:31 +00:00
Vostretsov Nikita 5197b39e01 fix capitalization, remove commas 2019-09-16 14:24:25 +00:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Vostretsov Nikita 66145b4eaf
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: Mikhail Korobov <kmike84@gmail.com>
 2019-09-12 18:51:00 +03:00
Vostretsov Nikita 171fa1cd10 documentation rework 2019-09-10 09:59:36 +00:00
Vostretsov Nikita 6bd88711f2 update documentation 2019-09-10 08:55:37 +00:00
Vostretsov Nikita 38828d3fd4
Update docs/topics/downloader-middleware.rst 
Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
 2019-09-09 17:04:13 +03:00
Vostretsov Nikita 9578f49099 use protego as a default robots.txt parser 2019-09-09 07:36:55 +00:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
Eugenio Lacuesta 2061f2a382
[doc] cb_kwargs contract 2019-08-31 02:10:18 -03:00
Adrián Chaves 2828cb769f Provide complete API documentation coverage of scrapy.extensions 2019-08-30 14:29:15 +02:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Mikhail Korobov 93d4b0b0d7
Merge pull request #3973 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.exporters
 2019-08-28 21:59:10 +05:00
Anubhav Patel 77c8ab2e62 makes suggested changes 2019-08-27 18:44:08 +05:30
Anubhav Patel ad824a264b fixes a link in doc 2019-08-27 18:30:11 +05:30
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Adrián Chaves 0fa384e80d Provide complete API documentation coverage of scrapy.exporters 2019-08-22 20:10:42 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Tobias Hernstig 2b0de0606c Fix merge conflicts 2019-08-15 18:54:28 +02:00
Tobias Hernstig 50c4cafe0c Update documentation for logging manually 
Usage of basicConfig() together with crawlerRunner is not recommended.
Update documentation to highlight this fact.

Closes #2149, #2352, #3146
 2019-08-15 18:46:00 +02:00
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin ce281d890d Documentation for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:53:19 +05:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Adrián Chaves a12e8251e0 Cover Scrapy 1.7.3 in the release notes 2019-08-01 17:10:31 +02:00
Adrián Chaves 04bca6af7c
Merge pull request #3894 from KristobalJunta/fix_retry_docs 
fix default RETRY_HTTP_CODES value in docs
 2019-07-29 18:20:55 +02:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Mikhail Korobov c679aefe85
Merge pull request #3660 from anubhavp28/logFormatter-doc-patch 
[MRG+1] docs for scrapy.logformatter
 2019-07-24 01:40:56 +05:00
Adrián Chaves 7843101f9a Cover Scrapy 1.7.2 in the release notes 2019-07-23 13:35:27 +02:00
Kristobal Junta 43d5b5a524 fix default RETRY_HTTP_CODES value in docs 2019-07-22 10:24:14 +03:00
Adrián Chaves b8a43011e7 Cover Scrapy 1.7.1 in the release notes 2019-07-19 18:17:24 +02:00
Andrey Rakhmatullin 42743fd9dd Move tls_verbose_logging extraction from __init__ to from_settings. 2019-07-18 20:49:25 +05:00
Adrián Chaves ae4eab9843 Cover the 1.7.1 PyPI repackaging in the release notes 2019-07-18 17:02:48 +02:00
Adrián Chaves a94b5bef3a Write the 1.7 release notes and cover dropping Python 2 support in the upcoming 2.0 2019-07-18 15:37:17 +02:00
Deschner, Magdalena c44d49b238 minor PEP8 style changes 2019-07-17 13:13:52 +02:00
Deschner, Magdalena 6660020ebb remove detailed description about individual settings 2019-07-17 11:30:02 +02:00
Deschner, Magdalena e892a484e8 add instructions about how to define output file when running scrapy from script instead of cmd 2019-07-16 13:53:56 +02:00
Adrián Chaves fa6a0d799b
Merge pull request #3684 from Gallaecio/remove-ubuntu-topic 
Remove docs/topics/ubuntu.rst
 2019-07-12 14:03:11 +02:00
Adrián Chaves ed487b7db3 broad-crawls.rst: Refactor the memory usage section 2019-07-11 11:14:22 +02:00
Andrey Rakhmatullin 98689b27a8 Improve the DOWNLOADER_CLIENTCONTEXTFACTORY doc. 2019-07-11 14:02:35 +05:00
Robin 5cdf277017 Update broad-crawls.rst 
Added section on how to treat memory consumption problems of broad crawls.
 2019-07-11 08:49:24 +02:00
Luiz Francisco Rodrigues da Silva df68c4b9b1 Add a setting to use active mode in FTPFeedStorage (#3829) 
Add a setting to use active mode in FTPFeedStorage

FTP servers can be configured in active and passive mode, by default
Python and Scrapy use passive mode and there was no way to use active.

This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed
to be exported to a FTP server configured in active mode.
 2019-07-10 14:25:52 +02:00
Mikhail Korobov 050a62a955
Merge pull request #3861 from Gallaecio/documentation-coverage 
Increase the documentation coverage of exceptions
 2019-07-10 10:18:34 +05:00
Eugenio Lacuesta 9fb0f8454e
Extend docs about Crawling Rules 2019-07-09 15:30:22 -03:00
Eugenio Lacuesta d04e84c11d Merge remote-tracking branch 'upstream/master' into docs_crawlspider_link_text 2019-07-09 15:24:31 -03:00
Adrián Chaves a0bbbe8ffa Add the API documentation of ContractFail to the contracts topic 2019-07-09 12:33:57 +02:00
Adrián Chaves 578bccf3bb Skip scrapy.exceptions.UsageError in the documentation coverage report 2019-07-09 12:33:07 +02:00
Mikhail Korobov a97a9b57bf
Merge pull request #3621 from Gallaecio/bfo-for-few-requests 
Document that the crawl order is BFO for small numbers of start requests
 2019-07-08 21:26:04 +05:00
Mikhail Korobov dda12b6537
Merge pull request #3672 from Gallaecio/split-items 
Add a FAQ entry about splitting items in item pipelines
 2019-07-08 20:15:23 +05:00
Adrián Chaves a0b09e0193 Skip scrapy.downloadermiddlewares private APIs in the documentation coverage report 2019-07-08 14:45:17 +02:00
Andrey Rakhmatullin 0b9dce3a6c Add DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING setting. 2019-07-08 17:40:56 +05:00
Adrián Chaves 783de8a8d8 Document how request concurrency settings impact request order 2019-07-08 14:23:21 +02:00
Adrián Chaves 6ea62cacc7 Add a FAQ entry about splitting items in item pipelines 2019-07-08 13:58:55 +02:00
Mikhail Korobov 4d4bd0e823
Merge pull request #3609 from Gallaecio/2253 
Document FilesPipeline.file_path and ImagesPipeline.file_path
 2019-07-04 22:05:46 +05:00
Mikhail Korobov 4c755f1ddc
Merge pull request #3692 from anubhavp28/doc-patch 
[MRG+1] doc for creating custom cache storage backend.
 2019-07-02 20:18:14 +05:00
Eugenio Lacuesta 428309ba1a Merge remote-tracking branch 'origin/master' into callback_kwargs 2019-06-26 11:03:31 -03:00
Mikhail Korobov c81d120bde
Merge pull request #3768 from Gallaecio/meta-ignore-tags 
Implement the METAREFRESH_IGNORE_TAGS setting
 2019-06-25 23:25:58 +03:00