RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,464 Commits 49 Branches 132 Tags 131 MiB
Commit Graph

2187 Commits

Author SHA1 Message Date
Habeeb Shopeju f47b120e2b
Documentation of link extractor usage (#4775) 
* Added description when using link extractor outside crawlspiders and created reference documentation for scrapy.link.Link class

* Added link.rst to toctree

* Corrected spelling errors, moved docs to Link doctstring to use autoclass

* Moved link docs to link_extractors

* Update docs/topics/link-extractors.rst

Co-authored-by: Adrián Chaves <adrian@chaves.io>

* Update link.py

Improvements to URL description

* Update link.py

* Update link.py

Fixed line length Flake issue

* Update link.py

Fixed trailing whitespace

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-10-01 23:50:11 +05:00
Andrey Rahmatullin 5a38639359
Merge pull request #4809 from madeny/patch-1 
Correct some typos
 2020-09-28 18:26:43 +05:00
madeny eff96038c7
Correct some typos 
This won't be an issue if **your** spider doesn't rely on cookies.
 2020-09-26 22:50:38 +02:00
Mirwaisse Djanbaz 3989f64baa
dependecies → dependencies (#4801) 2020-09-21 14:40:00 +02:00
Mirwaisse Djanbaz 7f1e74daa2
dependencides → dependencies (#4800) 2020-09-21 14:38:16 +02:00
Mikhail Korobov 85e13aff74
Merge pull request #4782 from sripriyesha/patch-1 
add mention of FTP server storage in media storage intro
 2020-09-15 01:17:46 +05:00
KAILASA's Sri Nithya Priyeshananda 7a83474cc5
add mention of FTP server storage in media storage intro 
At the beginning of this doc, in "Specifying where to store the media" feature details, FTP server storage mention was missing
 2020-09-08 17:16:31 +02:00
Ammar Najjar defeaacbc2 Merge branch 'master' into 4307-use-f-strings 
Conflicts resolved:
- delete tests/py36/_test_crawl.py
 2020-08-29 07:12:44 +02:00
Adrián Chaves 3f0a677c04
Cover version directive usage in the documentation policy (#4310) 
* Cover version directives in the documentation policy

* Remove version directives in preparation for Scrapy 2.0

* Update the policy based on the deprecation policy

* Only remove version directives after 3 years
 2020-08-27 23:56:58 +05:00
Adrián Chaves 195f738bba
Update Python version references after dropping support for 3.5 (#4742) 
* Update Python version references after dropping support for 3.5

* Remove outdated test

* Undo change affecting collect_asyncgen

* Undo change to be handled by #4743

* Remove unused import

* Remove unused import

* Update tests/requirements-py3.txt

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>

Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2020-08-27 15:43:43 +05:00
Mikhail Korobov a6b67cf4a1
Merge pull request #4755 from maranqz/csv-item-exporter-errors 
Add errors parameter for CsvItemExporter with tests
 2020-08-27 12:51:53 +05:00
Ammar Najjar 7597193dbe
Merge branch 'master' into 4307-use-f-strings 
Resolve Conflicts:
	tests/test_middleware.py
 2020-08-26 13:46:46 +02:00
maranqz 29725e4b58 Fix tabulation of rst and change documentation link. 2020-08-26 14:38:37 +03:00
nyov cf50561b86
Allow passing classes directly in Settings (#3873) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-08-26 13:08:14 +02:00
maranqz 560c335c07 Add errors parameter in documentation. 2020-08-26 14:00:51 +03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
sakshamb2113 42383cc267
Add a setting to customize the asyncio event loop (#4414) 2020-08-19 09:18:14 +02:00
Adrián Chaves e70975f0bb
Allow overwriting feeds (#4512) 
Co-authored-by: Yuval Hager <yhager@yhager.com>
 2020-08-17 15:10:08 +02:00
Eugenio Lacuesta 2aa4f3cbf9
Conditional request attribute binding for responses (#4632) 2020-08-17 10:39:59 +02:00
Andrey Rahmatullin acb3b4433a
Merge pull request #4724 from Gallaecio/feed-uri-params 
Document FEED_URI_PARAMS
 2020-08-14 14:47:55 +05:00
Mikhail Korobov 61653418ff
Merge pull request #4090 from Gallaecio/documentation-build 
Fix references to Python types in parameter type fields
 2020-08-14 00:58:06 +05:00
Mikhail Korobov cd0d3fd48d
Merge pull request #4705 from Gallaecio/deprecation-policy 
Cover our deprecation policy in the documentation
 2020-08-14 00:27:37 +05:00
Adrián Chaves 65e0abaea5 Document FEED_URI_PARAMS 2020-08-13 09:05:51 +02:00
Ajay Mittur 1c4b4cc6b0
Support defining file path based on item in media pipelines (#4686) 2020-08-11 14:12:44 +02:00
Adrián Chaves b2f4df5cb7 Merge remote-tracking branch 'upstream/master' into documentation-build 2020-08-11 13:16:14 +02:00
Adrián Chaves 4ee538e44b
Update unicode references from Python 2 times in the documentation (#4703) 2020-08-04 20:34:11 +02:00
Adrián Chaves 3600582f56
Cover Scrapy 2.2.1 and 2.3 in the release notes (#4708) 2020-08-04 20:05:56 +02:00
Adrián Chaves 6f4ccec567 Cover our deprecation policy in the documentation 2020-07-30 14:03:14 +02:00
Adrián Chaves 890b2138a6 Remove the u prefix from strings 2020-07-30 13:39:30 +02:00
Mikhail Korobov 5e2d1bd187
Merge pull request #4434 from BroodingKangaroo/ISSUE-4250-add_batch_deliveries 
Feed exports: add batch deliveries
 2020-07-29 23:03:28 +05:00
BroodingKangaroo a6c1d79b7c pep8 tiny changes 2020-07-28 11:53:05 +03:00
Andrey Rahmatullin f3372a3753
Merge pull request #4254 from elacuesta/spider.parse 
Change Scraper API to call internal `_parse` method
 2020-07-21 17:37:54 +05:00
Adrián Chaves d29bec60d7
Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest) (#4504) 
* Upgrade PyPy for CI, and test both 3.5 (oldest) and 3.6 (newest)

* Log a detailed error message to discover why MockServer is not working

* Go for all lines!

* Disable tests based on mitmproxy while running on PyPy

* Fix test_get_func_args for PyPy 3.6+

* Make testPayloadDefaultCiphers work regardless of OpenSSL default ciphers

* Crossing fingers…

* Rename: testPayloadDefaultCiphers → testPayloadDisabledCipher

* Test the PyPy version currently documented as the minimum required version

* Fix the PYPY_VERSION tag

* Update the documentation about supported PyPy versions

* Also test the latest 3.5 Python version with PyPy

* Fix the PYPY_VERSION value for the latest 3.5 version

* Use pinned dependencies for asyncio and PyPy tests against oldest supported Python versions

* Fix PyPy installation for the pypy3-pinned Tox environment

* Try installing Cython

* Maybe PyPy requires lxml 3.6.0?

* install.rst: minor clarification

* lxml 4.0.0 is required on PyPy

* Require setuptools 18.5+

* Revert "Require setuptools 18.5+"

This reverts commit 017ec33ac2.

* Maintain lxml as a dependency if setuptools < 18.5 is used
 2020-07-17 02:19:24 +05:00
Adrián Chaves c4f9250220
Merge branch 'master' into ISSUE-4250-add_batch_deliveries 2020-07-16 17:44:18 +02:00
BroodingKangaroo 41263f61c6 Change single quotes to double in example in docs 2020-07-16 18:41:45 +03:00
Mikhail Korobov 07470e1a3c
Merge pull request #3608 from ejulio/feat-685 
[MRG+1] Fix for #685 Add Google Cloud Storage Feed Export
 2020-07-16 18:11:19 +05:00
BroodingKangaroo 8bdcdb0a76 Add quotes to example in docs 2020-07-16 09:13:54 +03:00
Júlio César Batista 38496a00b7
Use the itemlaoders library (#4516) 2020-07-15 12:08:36 +02:00
Aditya Kumar 3f7e8635f4
Allow the parse command to write data to a file (#4377) 2020-07-11 08:48:24 +02:00
Eugenio Lacuesta 2be2bdd2df
Merge remote-tracking branch 'upstream/master' into spider.parse 2020-07-08 14:00:54 -03:00
BroodingKangaroo 6454d456d2 Make check of placeholder less strict 2020-07-03 10:24:13 +03:00
BroodingKangaroo 1e245046ed Change setting name. Add leading zeroes to batch_id. Minor fixes. 2020-07-02 12:38:08 +03:00
Andrey Rahmatullin af55d23167
Update the OpenSSL cipher list format link 
OpenSSL `ciphers(1)` is now almost empty: https://www.openssl.org/docs/manmaster/man1/ciphers.html

Alternative would be linking to 1.1.1 docs specifically.
 2020-07-01 19:46:54 +05:00
BroodingKangaroo 7b1d3c35ea Minor updates 2020-07-01 12:24:31 +03:00
Eugenio Lacuesta 5b88c522ac
Simplify dataclass example in item loader docs 2020-06-30 12:18:21 -03:00
BroodingKangaroo 05c2587c6a Docs update and tiny fixes 2020-06-28 09:45:45 +03:00
BroodingKangaroo c3cee74fd4 Change default value of FEED_STORAGE_BATCH_ITEM_COUNT to 0 2020-06-26 18:45:21 +03:00
BroodingKangaroo 77333666f1 resolve conflicts 2020-06-26 14:55:46 +03:00
Adrián Chaves 6f4c964aa4
Cover Scrapy 2.2.0 in the release notes (#4630) 2020-06-24 12:26:38 +02:00
Adrián Chaves cfd039aeb6 Remove a duplicate GCS_PROJECT_ID reference target 2020-06-22 19:28:33 +02:00