RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,437 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

9437 Commits

Author SHA1 Message Date
Andrey Rahmatullin b9c8db13c2
Merge pull request #5724 from Laerte/remove-scrapy_-env-variables-override 
Remove support for override settings with `SCRAPY_` environment variables
 2022-11-22 22:19:02 +05:00
Laerte Pereira 1506479672
Remove deprecated test 2022-11-22 10:07:32 -03:00
Laerte Pereira 8e0025f53d
Remove support for override settings with `SCRAPY_` environment variables 2022-11-22 09:38:54 -03:00
Andrey Rahmatullin aac6103179
Merge pull request #5722 from JazzGlobal/5717-memusage-logging-extension 
Adds get_virtual_size to _check_limit Logging
 2022-11-22 16:08:20 +05:00
Andrey Rahmatullin 603e42bc26
Merge pull request #5720 from Laerte/remove-filteringlinkextractor 
Remove `FilteringLinkExtractor`
 2022-11-22 13:54:43 +05:00
Christopher Gambrell eb159c78f1 Use variable and convert to megabytes. 2022-11-22 03:36:00 -05:00
Christopher Gambrell 8f2adad7a7 Log self.get_virtual_size() on every call of _check_limit 2022-11-22 00:48:24 -05:00
Christopher Gambrell 24a18e9af1 Adds virtualsize property to _check_limit error log. 2022-11-21 17:41:06 -05:00
Andrey Rahmatullin a60c120192
Merge pull request #5721 from pablohoffman/debug-vscode 
doc: add instructions to debug Scrapy spiders in Visual Studio Code
 2022-11-22 00:36:13 +05:00
Pablo Hoffman c04ccbceb9 doc: add instructions to debug Scrapy spiders in Visual Studio Code 2022-11-21 15:49:33 +01:00
Laerte Pereira 1a6408c3fa
Remove `FilteringLinkExtractor` 2022-11-21 07:30:20 -03:00
Laerte Pereira d5b6c236a9
Remove deprecated code (#5719) 2022-11-21 09:32:26 +01:00
Andrey Rahmatullin 8f01a60c73
Merge pull request #5715 from islem-esi/main 
minor fix for readability
 2022-11-16 13:47:54 +05:00
islem-esi 1200a54543 minor fix for readability 2022-11-15 16:28:45 +01:00
Hassan Shoayb 29bf7f5a6c
broad-crawls.rst: fix a typo (#5714) 2022-11-14 09:45:00 +01:00
Laerte Pereira ae3fd01729
Fix stray “commands” (#5712) 2022-11-10 15:38:46 +01:00
Andrey Rahmatullin c56caaf0b7
Merge pull request #4753 from drs-11/3689-update 
Do not recode JPEG files in imagepipelines
 2022-11-07 20:36:22 +05:00
Andrey Rahmatullin bbe24d79a5
Fix test issues 2022-11-07 17:08:54 +05:00
Andrey Rahmatullin 6c0890ff54
Simplify the changes after the merge 2022-11-07 16:36:54 +05:00
Andrey Rahmatullin b03d84e9fb
Merge branch 'master' into 3689-update 2022-11-07 16:33:21 +05:00
Andrey Rahmatullin 77cd511a5e
Merge pull request #5489 from PluT00/deprecate-NoimagesDrop 
Deprecate scrapy.pipelines.images.NoimagesDrop
 2022-11-05 14:18:19 +05:00
Andrey Rahmatullin 8004075823
Merge pull request #5699 from elacuesta/crawlspider-callback-keyword-arguments 
CrawlSpider: pass cb_kwargs from process_request
 2022-11-02 17:35:03 +05:00
Andrey Rakhmatullin 6ded3cf4cd Bump version: 2.7.0 → 2.7.1 2022-11-02 17:00:47 +06:00
Andrey Rahmatullin 95880c5de1
Merge pull request #5701 from scrapy/relnotes-2.7.1 2022-11-02 15:42:52 +05:00
Andrey Rakhmatullin 5ec175b8bb Small relnotes fixes. 2022-11-02 13:54:00 +06:00
Andrey Rakhmatullin 940a73863b Release notes for 2.7.1. 2022-11-01 19:00:33 +06:00
Andrey Rahmatullin a95a338eea
Merge pull request #5599 from tonal/patch-1 
Partial in is_generator_with_return_value
 2022-10-31 20:08:16 +05:00
Eugenio Lacuesta b18560315b
Add tests 2022-10-30 18:28:16 -03:00
Eugenio Lacuesta 3259a42525
CrawlSpider: pass cb_kwargs from process_request 2022-10-30 13:17:37 -03:00
Andrey Rahmatullin 9077d0f9b4
Merge pull request #5698 from pankali/patch-1 
Update Code of Conduct to Contributor Covenant v2.1
 2022-10-28 12:51:38 +05:00
Andrey Rahmatullin 76c2cb070e
Merge pull request #5697 from iamkaushal/#5686_fix 
added typing.io and typing.re in pytest warning filter to ignore
 2022-10-28 11:38:55 +05:00
എതിരാളിക്കൊരു പോരാളി 9f45be439d
Update Code of Conduct to Contributor Covenant v2.1 2022-10-28 02:13:37 +02:00
iamkaushal bd9e482c2f added typing.io and typing.re in pytest warning filter to ignore 2022-10-27 23:21:55 +05:30
Magnus Offermanns fd692f3091
Prevent running the -O and -t command-line options together (#5605) 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2022-10-27 14:43:31 +02:00
Andrey Rakhmatullin b71d0292d5 Add a test for processing partial callbacks. 2022-10-27 18:13:47 +06:00
Godson 3a34fa8399
Get the event loop from event_loop_policy to avoid a deprecation warning (#5689) 2022-10-27 14:02:12 +02:00
Andrey Rahmatullin b654183084
Use Python 3.11 as the default in CI (#5696) 2022-10-27 14:00:36 +02:00
Andrey Rakhmatullin ca50af6453 Remove an unused import. 2022-10-27 17:12:26 +06:00
Andrey Rahmatullin 5780ccba55
Merge branch 'master' into deprecate-NoimagesDrop 2022-10-27 15:46:32 +05:00
Mikhail Korobov 9fbd819b0e
Merge pull request #5695 from jidicula/patch-2 
ci: Update macos runner
 2022-10-27 12:45:43 +05:00
Andrey Rahmatullin c20b76d98f
Merge branch 'master' into patch-1 2022-10-27 12:04:00 +05:00
Johanan Idicula a214147359
ci: Update macos runner 
The GitHub Actions macos-10.15 runner image is now deprecated, and GitHub
Actions has begun to temporarily fail jobs referencing it during brownout
periods. The image will be fully unsupported by 2022-12-01, which is just about
a month away.

This change updates the macOS runner image to the latest generally-available
version, to help reduce spurious CI failures during the brownout periods, and to
stay abreast of the sunsetting of the macos-10.15 image.

See also: actions/runner-images#5583
 2022-10-26 21:44:47 -04:00
Andrei Andrukhovich b394f2165a
Fix typo: [they] depends → depend (#5694) 2022-10-26 22:11:28 +02:00
Laerte Pereira 2464939b7e
Fixed deprecation warning in scrapy.core.engine (#5589) 
* Change `download` function logic

* Fix CI error in 3.7 checks

* Make `spider` parameter optional in `_download` function, assign spider value from self if `None`
 2022-10-26 23:58:20 +05:00
Mikhail Korobov a1075b8979
Merge pull request #5626 from Gallaecio/more-lenient-proxying 
Relax Proxy-Authorization restrictions
 2022-10-26 23:36:29 +05:00
Adrián Chaves 46dd152b3e Merge remote-tracking branch 'upstream/master' into more-lenient-proxying 2022-10-26 16:55:04 +02:00
Andrey Rahmatullin 92be5ba257
Merge pull request #5691 from Godson-Gnanaraj/5323-fix-unable-to-parse-decorated-method 
Replace indentation of source before parsing with ast.
 2022-10-26 12:47:08 +05:00
Godson-Gnanaraj b0ddffc47b Misc. changes: 
- compile regex
  - readability improvements
 2022-10-26 06:53:43 +05:30
Godson-Gnanaraj 830e1c5dd8 Add test for parsing decorated methods 2022-10-26 01:26:54 +05:30
Kaushal Sharma f4e2a10ed6
Image.ANTIALIAS → Image.Resampling.LANCZOS (#5692) 2022-10-25 11:45:46 +02:00