RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,338 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

334 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 393d715205
Drop PyPy 3.10 support, restore coverage for subprocesses (#7050) 2025-09-15 11:24:09 +02:00
Andrey Rakhmatullin 6b2997af90
Migrate to pytest-twisted (#6938) 
* Migrate to pytest-twisted (WIP)

* Some typing fixes.

* Make --reactor=asyncio the default again.

* Try installing the correct event loop policy in tests on Windows.

* Make reactor_pytest a normal fixture.

* Fix test warnings.

* Fix FTPDownloadHandler teardown.

* Cleanups, typing.

* More cleanup.

* Update only_asyncio/only_not_asyncio mark messages.
 2025-07-06 21:27:17 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 91b186cf18
Use new pytest for new Twisted. (#6893) 2025-06-23 17:42:11 +02:00
Andrey Rakhmatullin 92c18d15b4
Remove ProcessTest and SiteTest. (#6885) 
* Remove ProcessTest and SiteTest.

* Restore the support for Windows line endings in TestParseCommand.

* Add a test for running a scrapy command in a project subdir.

* Remove pywin32 from test deps.
 2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin ba10dcfd1a
Pin older Twisted in tests, update type hints. (#6882) 2025-06-09 09:53:42 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin 4899d416e7
Add PyPy 3.11 to CI. (#6697) 2025-05-06 12:31:28 +02:00
Suejung Shin e50914e0f5
Codecov: Add test analytics (#6741) 2025-03-21 12:28:47 +01:00
Andrey Rakhmatullin d0dabbc097
Enable AsyncioSelectorReactor by default. (#6713) 
* Enable AsyncioSelectorReactor by default.

* Improve get_crawler(), switch more tests to it.

* Fix the remaining default-reactor test failures.

* Address documentation feedback.

* Make pinned envs more consistent.
 2025-03-12 01:18:30 +05:00
Adrián Chaves 803b4f258d
tox: move to posargs pytest parameters that can be too noisy when running specific tests (#6724) 2025-03-11 14:53:42 +01:00
Adrián Chaves 0c9200094e
Extend BaseSettings with utils for add-ons (#6614) 2025-03-05 10:31:59 +01:00
Andrey Rakhmatullin ba5df629a2
Refactor downloader tests (#6647) 
* Make download handler test base classes abstract.

* Small cleanup.

* Don't run the full test suite for special HTTP cases.

* Don't run tests in imported base classes.

* Remove an obsolete service_identity check.

* Move FTP imports back to the top level.

* Simplify the H2DownloadHandler import.

* Forbig pytest 8.2.x.

* Revert "Simplify the H2DownloadHandler import."

This reverts commit ed187046ac.
 2025-02-03 20:11:47 +05:00
Andrey Rakhmatullin d4b152bbf6
Drop PyPy 3.9, add a pypy3-extra-deps CI job. (#6613) 2025-01-23 09:22:18 +01:00
Andrey Rakhmatullin 3154b08e90 Improve coverage speed on Python 3.12+. 2025-01-07 20:42:03 +05:00
Andrey Rakhmatullin 7dfbecd392 Fix tracking of coverage in subprocesses. 2025-01-07 20:41:48 +05:00
Andrey Rakhmatullin f7af7b282d Bump mypy and stubs. 2024-12-29 16:45:26 +05:00
Andrey Rakhmatullin 4a0c05749c Bump tool versions. 2024-12-29 14:29:27 +05:00
Swayam Gupta 8c23da943c
Integrating configs into pyproject.toml (#6547) 2024-11-19 19:21:15 +05:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Andrey Rakhmatullin d10c58ff38 Bump pyftpdlib to the version supporting Python 3.13 on Windows. 2024-10-22 19:07:21 +05:00
Andrey Rakhmatullin 7e07d48cc5 Small 3.7 and 3.8 cleanup. 2024-10-17 23:22:37 +05:00
Andrey Rakhmatullin f65e64a724
Misc typing improvements. (#6494) 2024-10-16 18:38:43 +02:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 87651fdf47 Don't use types-setuptools. 2024-10-08 16:04:26 +05:00
Andrey Rakhmatullin 5794071f96
Typing fixes and updates. (#6460) 2024-08-23 12:48:01 +02:00
Andrey Rakhmatullin 3c9c1a31bc Bump pylint. 2024-07-11 12:30:12 +05:00
Andrey Rakhmatullin 129dbfa0bf Bump tool versions. 2024-07-11 12:23:43 +05:00
Andrey Rakhmatullin 96d6519b25
Bump twine in twinecheck. (#6416) 2024-06-26 14:43:59 +02:00
Andrey Rakhmatullin a364560fad
Unpin markupsafe in extra-deps. (#6403) 2024-06-17 11:38:10 +02:00
Andrey Rakhmatullin de146ad7ce Bump typing deps. 2024-06-05 22:09:19 +05:00
Andrey Rakhmatullin 9ba4dd311d
Install typing stubs for boto3 and botocore. (#6370) 2024-05-28 10:27:49 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Andrey Rakhmatullin 93f0628530
Improve typing for Spider.parse(). (#6274) 2024-05-13 11:55:45 +02:00
Andrey Rakhmatullin 57acad3c38
Full typing for scrapy/extensions, part 3. (#6325) 2024-04-29 09:43:45 +02:00
Lucas Belo 4460d3ed96
Remove tests/requirements.txt and refactor extra deps (#6272) 
Co-authored-by: lucasbelo777 <lucas.belo@cialdnb.com>
 2024-03-13 07:22:48 +01:00
Yuri H. Galvao 4cd94aa668
Restore brotlipy support (#6261) 2024-03-01 11:07:38 +01:00
Andrey Rakhmatullin 4f9dd998dc Bump pylint, cleanup the ignored tags. 2024-02-28 14:31:21 +05:00
Andrey Rakhmatullin d2c05d9d96 Bump mypy and type stubs. 2024-02-28 14:31:21 +05:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Andrey Rakhmatullin 68fccb1d58 Fix and re-enable newer mitmproxy usage in tests. 2024-01-06 01:35:56 +04:00
Andrey Rakhmatullin 5e4fb0bc5f Re-enable uvloop tests on 3.12 (#6098) 2023-11-16 19:16:31 +04:00
Andrey Rakhmatullin 4cb2fc2c3b Update typing package versions. 2023-10-31 22:55:39 +04:00
Andrey Rakhmatullin 991121fa91
Re-enable uvloop tests on 3.12 (#6098) 2023-10-17 19:24:44 +02:00
Andrey Rakhmatullin da39fbd270
Update tool versions (#6046) 2023-09-12 20:07:10 +02:00
Andrey Rakhmatullin 2c7484d433 Merge remote-tracking branch 'origin/master' into typing-utils-2 2023-08-07 14:55:38 +04:00
Andrey Rakhmatullin 110d5fffb4
Update tool versions. (#6002) 2023-08-07 10:57:48 +02:00
Andrey Rakhmatullin f5f593e5f5 Remove a workaround for a w3lib typing bug. 2023-08-06 17:46:28 +04:00
Andrey Rakhmatullin 5e1582491b mypy --show-error-codes is the default now. 2023-07-22 23:19:25 +04:00
Andrey Rakhmatullin 187e8f9a2d Typing for scrapy/settings/__init__.py. 2023-07-17 00:01:36 +04:00
Andrey Rakhmatullin 93962ebefc Bump typing package versions. 2023-07-14 23:20:35 +04:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Andrey Rakhmatullin 075b89eab5 Bump lxml and cryptography to versions with 3.8 wheels available. 2023-06-18 19:08:41 +04:00
Andrey Rakhmatullin 2122278d4b Drop Python 3.7 support. 2023-06-18 18:37:50 +04:00
Andrey Rakhmatullin 0adbd210ac Fix extra-deps-pinned tests. 2023-06-13 19:34:26 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 048812ba35 Bump types-* versions. 2023-05-07 22:40:29 +04:00
Andrey Rakhmatullin 8c8fb67057
Update tool versions (#5908) 2023-04-24 09:34:34 +02:00
Jalil SA 5a37af146f
Update documentation expectations for Parsel 1.8.0 (#5902) 2023-04-21 09:29:57 +02:00
Andrey Rakhmatullin d911837389
Merge pull request #5881 from sbartlett97/LinkExtractor-Patch 
Patched LxmlParserLinkExtractor
 2023-04-12 12:28:21 +04:00
Adrián Chaves 3f0c2fae5e Skip test_skip_bad_links based on the w3lib version 2023-04-12 09:28:28 +02:00
Andrey Rakhmatullin c7730627a0
Typing improvements for some core components (#5889) 2023-04-11 13:47:41 +02:00
jazzthief a17d996da2
Change `boto3` version to 1.20.0 for `extra-deps-pinned` env 2023-03-16 17:20:22 +01:00
Adrián Chaves 590955fac8 Provide separate test environments for botocore and boto3 extras; add extra-deps-pinned and remote extras from pinned 2023-03-15 16:03:44 +01:00
Adrián Chaves 39dbfa1d82 Minimize test reliance on S3; do not install botocore on the default test environments 2023-03-15 15:31:39 +01:00
Andrey Rakhmatullin d7bf39ee78 Fix a merge problem. 2023-02-24 00:28:06 +04:00
Andrey Rakhmatullin 3a40c06ed9 Bump mypy and types-*. 2023-02-24 00:16:49 +04:00
Andrey Rakhmatullin 824641e349 Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-11 22:02:19 +04:00
Andrey Rakhmatullin b1f33a68ac Bump mypy and types-*. 2023-02-11 21:39:27 +04:00
Adrián Chaves 32a01e32f3 Update pylint exceptions, and run pylint by default with tox 2023-02-02 06:52:01 +01:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Adrián Chaves 78eaf0671b Remove typing-extensions from tox.ini 2023-01-30 14:33:11 +01:00
Andrey Rakhmatullin 69bb9a7859 Install types-Pillow and types-Pygments. 2023-01-25 23:52:46 +04:00
Andrey Rakhmatullin a4edff31b9 Replace lxml-stubs with types-lxml. 2023-01-25 23:30:21 +04:00
Adrián Chaves 9272c4af0c Merge remote-tracking branch 'scrapy/master' into no-callback 2023-01-25 17:41:41 +01:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Adrián Chaves 4242ae405d Restore Python 3.7 support 2023-01-19 20:37:24 +01:00
Adrián Chaves 5c1559f60e Address typing issues 2023-01-19 20:30:22 +01:00
Adrián Chaves 16546564f6 Minimum cryptography: 3.3 → 3.4.6 2023-01-06 20:16:54 +01:00
Andrey Rakhmatullin 3e03004b3c
Merge pull request #5781 from Gallaecio/ci-clang 
Remove Reppy from CI
 2023-01-02 20:13:45 +05:00
Adrián Chaves 016d1de64e Remove Reppy from CI 2023-01-02 14:13:04 +01:00
Adrián Chaves 6084e8f627 Do not use -U for pip in Tox 2023-01-02 14:00:37 +01:00
Adrián Chaves e47ada2c7c Add CC to tox.ini:[testenv]passenv 2023-01-02 13:49:25 +01:00
Laerte Pereira deaf1fb6cf
Stop using setup.py (#5776) 2022-12-31 20:11:45 +01:00
Emmanuel Rondan c48accf7ca adding pre-commit with config 2022-12-26 13:51:27 -03:00
Emmanuel Rondan 09dc4cf308 adding black to tox and defining line length on pyproject.toml 2022-12-22 16:11:21 -03:00
Andrey Rahmatullin 5bd27191a2
Bump mypy, flake8, and pylint (#5738) 2022-11-28 19:37:57 +01:00
Andrey Rakhmatullin 5bf4260679 Update flake8. 2022-10-10 11:14:20 +06:00
Andrey Rakhmatullin 5fa613b419 Run flake8 directly. 2022-10-10 11:04:00 +06:00
Andrey Rakhmatullin d1515cc075 Update pylint. 2022-10-07 15:30:44 +06:00
Andrey Rakhmatullin ccb6a8c098 Add a note about flake8. 2022-10-07 15:27:14 +06:00
Andrey Rakhmatullin 424849b275 Update mypy. 2022-10-07 15:17:55 +06:00
Andrey Rahmatullin 28d65c88b1
Merge pull request #5662 from gliptak/pyopenssl1 
Match pyOpenSSL and types-pyOpenSSL versions
 2022-10-04 12:54:30 +05:00
Abdul Rauf 69bf5c6625 CI: move twinecheck to tox env 2022-10-02 20:27:24 +05:00
Gábor Lipták 9f006e3aa5 Match pyOpenSSL and types-pyOpenSSL versions 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-30 22:27:57 -04:00
Gábor Lipták 385acd5598
Match pyOpenSSL and service_identity to Twisted 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-24 21:33:02 -04:00
Mikhail Korobov 56e2eeac10 fix typing issues by upgrading mypy 2022-07-27 09:41:12 +05:00
Adrián Chaves 5862f6b8e1
Update frozen CI packages (#5574) 2022-07-25 17:25:10 +02:00
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Maxime Nannan 078622cfb0
Fix file expiration issue with GCS (#5318) 2022-05-20 08:30:06 +02:00