RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,139 Commits 49 Branches 132 Tags 133 MiB
Commit Graph

9139 Commits

Author SHA1 Message Date
Laerte Pereira 6770d1ec62
chore(tests): Remove validations for unsupported modules versions 2022-06-09 09:08:09 -03:00
Laerte Pereira 2e6721fd86
docs: Update minimal versions that Scrapy is tested against 2022-06-09 08:37:01 -03:00
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Mikhail Korobov 517cbc8d71
Merge pull request #5508 from mtabbasi/thumb_path-access-item 
pass on item to thumb_path function as additional argument
 2022-06-05 03:42:45 +05:00
Mikhail Korobov bafe874ecc
Merge pull request #5358 from burakcank/signal_coroutine 
Document coroutines for signals.
 2022-05-28 13:38:05 +05:00
Mikhail Korobov 99cddecd28
Merge pull request #5497 from AngelikiBoura/issue_3264/command_parser_unhandled_error 
Issue #3264, fix error handling when spider is not matched
 2022-05-28 13:26:23 +05:00
Mohammadtaher Abbasi 896f16f2de make thumb_path method backwards compatible 2022-05-25 23:59:25 +04:30
Mohammadtaher Abbasi 5c586d78f0 add tests 2022-05-25 23:58:09 +04:30
Mohammadtaher Abbasi f39def4492 add docs 2022-05-25 23:57:38 +04:30
Mohammadtaher Abbasi 8633190012 pass on item to thumb_path function as additional argument resolves #5504 2022-05-24 14:51:17 +04:30
Andreas Tziortziortziopoulos b5c15d87ff [issue3264] Separate test for not matched spider to a url 2022-05-22 12:19:20 +03:00
Maxime Nannan 078622cfb0
Fix file expiration issue with GCS (#5318) 2022-05-20 08:30:06 +02:00
Andrey Rahmatullin 965fde24a4 Pin mitmproxy to < 8 for now (#5459) 2022-05-20 07:17:09 +02:00
Andreas Tziortziortziopoulos 83c1939281 Issue #3264, fix error handling when spider is not matched 
Changes
Implementation:
- Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider
Testing:
- Add a test case with invalid url inside test_command_parse
  Test proves that non-matched Spider does not throw an AttributeError
 2022-05-06 03:59:30 +03:00
AngelikiBoura b2afcbfe2b
Fix typos in three files for Flake8 check (#5487) 
* Fix typos in extensions files

Made some fixes in files memusage.py and statsmailer.py
in order to pass the flake8 check.

* Fix typos in twisted_reactor_custom_settings_same.py

A small change was needed in order for flake8 check
to pass.
 2022-05-05 15:49:52 +02:00
Andrey Rahmatullin ded28f7bb9
Merge pull request #5461 from yash-fn/patch-1 
documentation update for multiple spiders
 2022-04-15 14:39:26 +05:00
Yash 319e67f779 documentation update for multiple spiders 
i noticed passing settings to configure logging function made weird output go away. checked documentation and it says first parameter is settings file. Is this correct?
 2022-04-12 11:42:22 -05:00
Andrey Rahmatullin 8c3e5a2421
Merge pull request #5457 from d00399830/typo_fixes_sm 
Fix capitalization of JavaScript within the Developer Tools documentation
 2022-03-23 11:06:54 +05:00
Víctor Ruiz 2227be7af6
Fix a typo in the HTTP cache documentation (#5455) 2022-03-22 15:21:16 +01:00
D00399830 fcf3d8e0a0 Updated the documentation for developer tools to have JavaScript instead of Javascript, as JavaScript is the more correct way to write it 2022-03-21 14:09:31 -06:00
Eugene 9a28eb0bad
Suggest installing the brotli package instead of brotlipy (#4267) 2022-03-17 05:39:54 +01:00
Andrey Rahmatullin 2d6042b108
Merge pull request #5432 from Farsene1/master 
Recommend Common Crawl instead of Google Cache
 2022-03-11 20:05:43 +05:00
NaincyKumariKnoldus e264cc30d1
removed the pywin32 docs section (#5370) 2022-03-10 14:54:33 +01:00
Ali Rastegar d469214fe7
Update tutorial.rst (#5442) 
Fixed typo
 2022-03-08 10:29:22 +01:00
Florentin ccdbb795ff Recommend Common Crawl instead of Google Cache 2022-03-01 22:01:55 +01:00
Adrián Chaves 50c8becbe0
Freeze and upgrade CI packages (#5429) 2022-03-01 17:29:08 +01:00
Adrián Chaves 23537a0f95 Bump version: 2.6.0 → 2.6.1 2022-03-01 13:48:40 +01:00
Adrián Chaves fab3e90729 Cover 2.6.1 in the release notes 2022-03-01 13:41:20 +01:00
Adrián Chaves d60636d0de Fix redirect handling regression 2022-03-01 13:06:58 +01:00
Adrián Chaves 84853c4fa6 bandit: allow-list B324 for the time being 2022-03-01 13:01:20 +01:00
Adrián Chaves 6b63e7c147 Bump version: 2.5.0 → 2.6.0 2022-03-01 12:43:11 +01:00
Adrián Chaves e865c4430e
Merge pull request from GHSA-mfjm-vh54-3f96 
* Ignore cookies with a public suffix as domain unless it matches the request domain

* Fix the merge of 1.8.2 release notes

* Re-apply removal of tldextract restriction
 2022-03-01 12:38:19 +01:00
Adrián Chaves 8ce01b3b76
Merge pull request from GHSA-cjvr-mfj7-j4j8 
* Do not carry over cookies to a different domain on redirect

* Cover the cookie-domain redirect fix in the release notes

* Cover 1.8.2 in the release notes

* Fix redirect Cookie handling when the cookie middleware is disabled

* Update the 1.8.2 release date
 2022-03-01 12:26:05 +01:00
Adrián Chaves aa0306a167
Cover 2.6.0 in the release notes (#5399) 2022-03-01 12:16:37 +01:00
Andrey Rahmatullin 08557e09db
Pin old markupsafe when we pin old mitmproxy (#5427) 2022-02-23 19:52:18 +01:00
Gowtham Chowdary 3b42ccfeba
Add a link to Discord (#5422) 2022-02-16 21:33:56 +01:00
Andrey Rahmatullin 8840403b70
Merge pull request #5412 from Laerte/master 2022-02-15 18:15:39 +05:00
Andrey Rahmatullin 0b0eea3601
Merge pull request #5419 from PendalF89/patch-2 
Update downloader-middleware.rst
 2022-02-15 10:57:35 +05:00
Abhishek K M 187b5c8876
Update the documentation link for robots.txt (#5415) 2022-02-14 19:16:53 +01:00
Boris Zabolotskikh bbb693d046
Update downloader-middleware.rst 
Added a link to the method
 2022-02-14 12:07:45 +03:00
Laerte Pereira befb6df119 Remove Python 2 code from WrappedRequest 2022-02-11 06:19:27 -03:00
Andrey Rahmatullin 5d7c0a5f86
Use toscrape.com instead of example.com in test_command_check. (#5407) 2022-02-10 10:50:12 +01:00
Alex e2e2ffd0d1
Move from optparse to argparse (#5374) 2022-02-09 20:52:07 +01:00
Andrey Rahmatullin 1356836e7e
Merge pull request #5405 from wRAR/tests-twisted-22.1-fix 
Fix running tests with Twisted 22.1.0
 2022-02-09 00:06:25 +05:00
Andrey Rakhmatullin 77547a1ab5 Revert "Temporarily pin Twisted to an older version in CI (#5401)" 
This reverts commit b282a7af01.
 2022-02-08 21:06:02 +05:00
Andrey Rakhmatullin 1e1cfc26db Copy resource classes from twisted.web.test.test_webclient. 2022-02-08 21:01:16 +05:00
Raihan Nismara fd55f62207
Update Logo in README.rst (#5258) 2022-02-08 15:36:25 +01:00
Laerte 4bda0976b2
Fix csviter call, add parse_rows test (#5394) 2022-02-08 14:57:19 +01:00
Andrey Rahmatullin 840362496c
Merge pull request #5396 from peter-gy/master 
docs: use https scheme for each quotes.toscrape.com url occurrence
 2022-02-08 11:05:52 +05:00