RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,139 Commits 49 Branches 132 Tags 131 MiB
Commit Graph

2386 Commits

Author SHA1 Message Date
Laerte Pereira 2e6721fd86
docs: Update minimal versions that Scrapy is tested against 2022-06-09 08:37:01 -03:00
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Mikhail Korobov 517cbc8d71
Merge pull request #5508 from mtabbasi/thumb_path-access-item 
pass on item to thumb_path function as additional argument
 2022-06-05 03:42:45 +05:00
Mikhail Korobov bafe874ecc
Merge pull request #5358 from burakcank/signal_coroutine 
Document coroutines for signals.
 2022-05-28 13:38:05 +05:00
Mohammadtaher Abbasi f39def4492 add docs 2022-05-25 23:57:38 +04:30
Yash 319e67f779 documentation update for multiple spiders 
i noticed passing settings to configure logging function made weird output go away. checked documentation and it says first parameter is settings file. Is this correct?
 2022-04-12 11:42:22 -05:00
Andrey Rahmatullin 8c3e5a2421
Merge pull request #5457 from d00399830/typo_fixes_sm 
Fix capitalization of JavaScript within the Developer Tools documentation
 2022-03-23 11:06:54 +05:00
Víctor Ruiz 2227be7af6
Fix a typo in the HTTP cache documentation (#5455) 2022-03-22 15:21:16 +01:00
D00399830 fcf3d8e0a0 Updated the documentation for developer tools to have JavaScript instead of Javascript, as JavaScript is the more correct way to write it 2022-03-21 14:09:31 -06:00
Eugene 9a28eb0bad
Suggest installing the brotli package instead of brotlipy (#4267) 2022-03-17 05:39:54 +01:00
Andrey Rahmatullin 2d6042b108
Merge pull request #5432 from Farsene1/master 
Recommend Common Crawl instead of Google Cache
 2022-03-11 20:05:43 +05:00
NaincyKumariKnoldus e264cc30d1
removed the pywin32 docs section (#5370) 2022-03-10 14:54:33 +01:00
Ali Rastegar d469214fe7
Update tutorial.rst (#5442) 
Fixed typo
 2022-03-08 10:29:22 +01:00
Florentin ccdbb795ff Recommend Common Crawl instead of Google Cache 2022-03-01 22:01:55 +01:00
Adrián Chaves fab3e90729 Cover 2.6.1 in the release notes 2022-03-01 13:41:20 +01:00
Adrián Chaves e865c4430e
Merge pull request from GHSA-mfjm-vh54-3f96 
* Ignore cookies with a public suffix as domain unless it matches the request domain

* Fix the merge of 1.8.2 release notes

* Re-apply removal of tldextract restriction
 2022-03-01 12:38:19 +01:00
Adrián Chaves 8ce01b3b76
Merge pull request from GHSA-cjvr-mfj7-j4j8 
* Do not carry over cookies to a different domain on redirect

* Cover the cookie-domain redirect fix in the release notes

* Cover 1.8.2 in the release notes

* Fix redirect Cookie handling when the cookie middleware is disabled

* Update the 1.8.2 release date
 2022-03-01 12:26:05 +01:00
Adrián Chaves aa0306a167
Cover 2.6.0 in the release notes (#5399) 2022-03-01 12:16:37 +01:00
Gowtham Chowdary 3b42ccfeba
Add a link to Discord (#5422) 2022-02-16 21:33:56 +01:00
Andrey Rahmatullin 0b0eea3601
Merge pull request #5419 from PendalF89/patch-2 
Update downloader-middleware.rst
 2022-02-15 10:57:35 +05:00
Abhishek K M 187b5c8876
Update the documentation link for robots.txt (#5415) 2022-02-14 19:16:53 +01:00
Boris Zabolotskikh bbb693d046
Update downloader-middleware.rst 
Added a link to the method
 2022-02-14 12:07:45 +03:00
Andrey Rahmatullin 840362496c
Merge pull request #5396 from peter-gy/master 
docs: use https scheme for each quotes.toscrape.com url occurrence
 2022-02-08 11:05:52 +05:00
Eugenio Lacuesta fca49cca92
Remove deprecated DictItem class 2022-02-06 18:31:55 -03:00
Péter Ferenc Gyarmati 38d2a154ec docs: use https scheme for each quotes.toscrape.com url occurrence 2022-02-06 18:52:15 +01:00
Sixuan (Cherie) Wu 30d5779ea9
Fix FEED_URI_PARAMS: custom params throws KeyError (#4966) 
* fix FEED_URI_PARAMS: custom params throws KeyError closes #4962

* another try FEED_URI_PARAMS

* add warning message and change default function

* Add tests for FEED_URI_PARAMS

* FEED_URI_PARAMS: warn if the params dict has been modified in-place

* [Doc] FEED_URI_PARAMS: modifying params in-place is deprecated

* Remove whileline

* Rename parameters for lambda function

* Type hints for FeedExporter._get_uri_params

Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2022-01-28 14:30:30 -03:00
Mikhail Korobov 9dd77b42b5
Merge pull request #1582 from starrify/add_doc_textresponse_urljoin 
added: Doc for `scrapy.http.TextResponse.urljoin`
 2022-01-23 21:40:21 +05:00
Andrey Rahmatullin 5059b8c493
Merge branch 'master' into doc-fix 2022-01-21 13:58:43 +05:00
Andrey Rahmatullin 1608b1a7fa
Merge branch 'master' into patch-1 2022-01-21 13:24:51 +05:00
Andrey Rakhmatullin 3577c00569 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2022-01-03 13:59:26 +05:00
Andrey Rahmatullin c5ab58056c
Set WindowsSelectorEventLoopPolicy on Windows (#5315) 2021-12-31 20:38:10 +01:00
Andrey Rakhmatullin 6eaceec735 Implement docs suggestions. 2021-12-31 20:14:24 +05:00
Andrey Rakhmatullin 60800ee5f2 Merge remote-tracking branch 'origin/master' into crawler-process-reactor-later 2021-12-31 20:12:30 +05:00
Andrey Rahmatullin 57dc58123b
Remove the experimental note about asyncio (#5332) 2021-12-31 13:15:08 +01:00
Andrey Oskin b81938684b
Docs: correct process repetition start step (#5356) 
The process repeats from step 3, the scheduler feeds request to the engine. Steps 1 and 2 are not parts of the loop as their incarnations steps 7 and 8 are parts of the loop.
 2021-12-31 11:49:18 +01:00
Burak Can Kahraman a9dfd85ea6 Document coroutines for signals. 2021-12-30 15:48:53 +03:00
Andrey Rakhmatullin a986792def Add more docs for TWISTED_REACTOR. 2021-12-24 19:43:14 +05:00
Andrey Rakhmatullin 940cc0776f Add docs about TWISTED_REACTOR and other per-process settings. 2021-12-24 17:12:50 +05:00
azzamsa 67994d1ddd fix: `CodeBlockParser` has been renamed to `PythonCodeBlockParser` 2021-10-27 22:19:07 +07:00
Andrey Rahmatullin 144d1eb834
Add Deferred-to-Future helpers (#5288) 2021-10-22 18:46:01 +02:00
Andrey Rahmatullin afa5881ada
Merge pull request #5279 from erikkemperman/option-logfile-truncate 
Add LOG_FILE_APPEND to settings
 2021-10-18 11:51:42 +05:00
raphaelts3 027ecd8686
Update docs/intro/overview.rst 
Co-authored-by: azzamsa <17734314+azzamsa@users.noreply.github.com>
 2021-10-16 10:52:54 -03:00
Raphael Tomé Santana aec7146e2f Add how Scrapy is pronounced to the docs 2021-10-15 20:38:53 -03:00
Erik Kemperman ca320feb2a
Add LOG_FILE_APPEND to settings 2021-10-15 16:36:41 +02:00
谭九鼎 3243aa2cd5
docs: fix typo 2021-10-14 10:18:26 +08:00
Jake Herbst d08199f631
Removing unnecessary line from docs to prevent test failure (#5274) 2021-10-12 19:20:09 +02:00
Andrey Rahmatullin 1ae53a9d7c
Merge pull request #5256 from rydwhelchel/documentation-changes 
docs: restructed phrasing for clarity
 2021-10-12 16:49:31 +05:00
Andrey Rahmatullin d90b2a2bfc
Merge pull request #5250 from iDeepverma/URLLENGTH_LIMIT-doc 
Documentation update for URLLENGTH_LIMIT
 2021-10-12 16:48:42 +05:00
Andrey Rahmatullin 8a743ae43a
Merge pull request #5266 from MarvinPetzoldt/patch-1 
Fixed documentation example
 2021-10-12 16:46:51 +05:00