RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,208 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

320 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 3154b08e90 Improve coverage speed on Python 3.12+. 2025-01-07 20:42:03 +05:00
Andrey Rakhmatullin 7dfbecd392 Fix tracking of coverage in subprocesses. 2025-01-07 20:41:48 +05:00
Andrey Rakhmatullin f7af7b282d Bump mypy and stubs. 2024-12-29 16:45:26 +05:00
Andrey Rakhmatullin 4a0c05749c Bump tool versions. 2024-12-29 14:29:27 +05:00
Swayam Gupta 8c23da943c
Integrating configs into pyproject.toml (#6547) 2024-11-19 19:21:15 +05:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Andrey Rakhmatullin d10c58ff38 Bump pyftpdlib to the version supporting Python 3.13 on Windows. 2024-10-22 19:07:21 +05:00
Andrey Rakhmatullin 7e07d48cc5 Small 3.7 and 3.8 cleanup. 2024-10-17 23:22:37 +05:00
Andrey Rakhmatullin f65e64a724
Misc typing improvements. (#6494) 2024-10-16 18:38:43 +02:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin 87651fdf47 Don't use types-setuptools. 2024-10-08 16:04:26 +05:00
Andrey Rakhmatullin 5794071f96
Typing fixes and updates. (#6460) 2024-08-23 12:48:01 +02:00
Andrey Rakhmatullin 3c9c1a31bc Bump pylint. 2024-07-11 12:30:12 +05:00
Andrey Rakhmatullin 129dbfa0bf Bump tool versions. 2024-07-11 12:23:43 +05:00
Andrey Rakhmatullin 96d6519b25
Bump twine in twinecheck. (#6416) 2024-06-26 14:43:59 +02:00
Andrey Rakhmatullin a364560fad
Unpin markupsafe in extra-deps. (#6403) 2024-06-17 11:38:10 +02:00
Andrey Rakhmatullin de146ad7ce Bump typing deps. 2024-06-05 22:09:19 +05:00
Andrey Rakhmatullin 9ba4dd311d
Install typing stubs for boto3 and botocore. (#6370) 2024-05-28 10:27:49 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Andrey Rakhmatullin 93f0628530
Improve typing for Spider.parse(). (#6274) 2024-05-13 11:55:45 +02:00
Andrey Rakhmatullin 57acad3c38
Full typing for scrapy/extensions, part 3. (#6325) 2024-04-29 09:43:45 +02:00
Lucas Belo 4460d3ed96
Remove tests/requirements.txt and refactor extra deps (#6272) 
Co-authored-by: lucasbelo777 <lucas.belo@cialdnb.com>
 2024-03-13 07:22:48 +01:00
Yuri H. Galvao 4cd94aa668
Restore brotlipy support (#6261) 2024-03-01 11:07:38 +01:00
Andrey Rakhmatullin 4f9dd998dc Bump pylint, cleanup the ignored tags. 2024-02-28 14:31:21 +05:00
Andrey Rakhmatullin d2c05d9d96 Bump mypy and type stubs. 2024-02-28 14:31:21 +05:00
Adrián Chaves 6bd45bb6d9 Merge remote-tracking branch 'scrapy/2.11' into 2.11-port 2024-02-15 06:39:50 +01:00
Andrey Rakhmatullin 68fccb1d58 Fix and re-enable newer mitmproxy usage in tests. 2024-01-06 01:35:56 +04:00
Andrey Rakhmatullin 5e4fb0bc5f Re-enable uvloop tests on 3.12 (#6098) 2023-11-16 19:16:31 +04:00
Andrey Rakhmatullin 4cb2fc2c3b Update typing package versions. 2023-10-31 22:55:39 +04:00
Andrey Rakhmatullin 991121fa91
Re-enable uvloop tests on 3.12 (#6098) 2023-10-17 19:24:44 +02:00
Andrey Rakhmatullin da39fbd270
Update tool versions (#6046) 2023-09-12 20:07:10 +02:00
Andrey Rakhmatullin 2c7484d433 Merge remote-tracking branch 'origin/master' into typing-utils-2 2023-08-07 14:55:38 +04:00
Andrey Rakhmatullin 110d5fffb4
Update tool versions. (#6002) 2023-08-07 10:57:48 +02:00
Andrey Rakhmatullin f5f593e5f5 Remove a workaround for a w3lib typing bug. 2023-08-06 17:46:28 +04:00
Andrey Rakhmatullin 5e1582491b mypy --show-error-codes is the default now. 2023-07-22 23:19:25 +04:00
Andrey Rakhmatullin 187e8f9a2d Typing for scrapy/settings/__init__.py. 2023-07-17 00:01:36 +04:00
Andrey Rakhmatullin 93962ebefc Bump typing package versions. 2023-07-14 23:20:35 +04:00
Andrey Rakhmatullin 712ee98848 Merge branch 'master' into typing-utils 2023-06-24 18:31:56 +04:00
Andrey Rakhmatullin 075b89eab5 Bump lxml and cryptography to versions with 3.8 wheels available. 2023-06-18 19:08:41 +04:00
Andrey Rakhmatullin 2122278d4b Drop Python 3.7 support. 2023-06-18 18:37:50 +04:00
Andrey Rakhmatullin 0adbd210ac Fix extra-deps-pinned tests. 2023-06-13 19:34:26 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 048812ba35 Bump types-* versions. 2023-05-07 22:40:29 +04:00
Andrey Rakhmatullin 8c8fb67057
Update tool versions (#5908) 2023-04-24 09:34:34 +02:00
Jalil SA 5a37af146f
Update documentation expectations for Parsel 1.8.0 (#5902) 2023-04-21 09:29:57 +02:00
Andrey Rakhmatullin d911837389
Merge pull request #5881 from sbartlett97/LinkExtractor-Patch 
Patched LxmlParserLinkExtractor
 2023-04-12 12:28:21 +04:00
Adrián Chaves 3f0c2fae5e Skip test_skip_bad_links based on the w3lib version 2023-04-12 09:28:28 +02:00
Andrey Rakhmatullin c7730627a0
Typing improvements for some core components (#5889) 2023-04-11 13:47:41 +02:00
jazzthief a17d996da2
Change `boto3` version to 1.20.0 for `extra-deps-pinned` env 2023-03-16 17:20:22 +01:00
Adrián Chaves 590955fac8 Provide separate test environments for botocore and boto3 extras; add extra-deps-pinned and remote extras from pinned 2023-03-15 16:03:44 +01:00
Adrián Chaves 39dbfa1d82 Minimize test reliance on S3; do not install botocore on the default test environments 2023-03-15 15:31:39 +01:00
Andrey Rakhmatullin d7bf39ee78 Fix a merge problem. 2023-02-24 00:28:06 +04:00
Andrey Rakhmatullin 3a40c06ed9 Bump mypy and types-*. 2023-02-24 00:16:49 +04:00
Andrey Rakhmatullin 824641e349 Merge remote-tracking branch 'origin/master' into mypy-imports 2023-02-11 22:02:19 +04:00
Andrey Rakhmatullin b1f33a68ac Bump mypy and types-*. 2023-02-11 21:39:27 +04:00
Adrián Chaves 32a01e32f3 Update pylint exceptions, and run pylint by default with tox 2023-02-02 06:52:01 +01:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Adrián Chaves 78eaf0671b Remove typing-extensions from tox.ini 2023-01-30 14:33:11 +01:00
Andrey Rakhmatullin 69bb9a7859 Install types-Pillow and types-Pygments. 2023-01-25 23:52:46 +04:00
Andrey Rakhmatullin a4edff31b9 Replace lxml-stubs with types-lxml. 2023-01-25 23:30:21 +04:00
Adrián Chaves 9272c4af0c Merge remote-tracking branch 'scrapy/master' into no-callback 2023-01-25 17:41:41 +01:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Adrián Chaves 4242ae405d Restore Python 3.7 support 2023-01-19 20:37:24 +01:00
Adrián Chaves 5c1559f60e Address typing issues 2023-01-19 20:30:22 +01:00
Adrián Chaves 16546564f6 Minimum cryptography: 3.3 → 3.4.6 2023-01-06 20:16:54 +01:00
Andrey Rakhmatullin 3e03004b3c
Merge pull request #5781 from Gallaecio/ci-clang 
Remove Reppy from CI
 2023-01-02 20:13:45 +05:00
Adrián Chaves 016d1de64e Remove Reppy from CI 2023-01-02 14:13:04 +01:00
Adrián Chaves 6084e8f627 Do not use -U for pip in Tox 2023-01-02 14:00:37 +01:00
Adrián Chaves e47ada2c7c Add CC to tox.ini:[testenv]passenv 2023-01-02 13:49:25 +01:00
Laerte Pereira deaf1fb6cf
Stop using setup.py (#5776) 2022-12-31 20:11:45 +01:00
Emmanuel Rondan c48accf7ca adding pre-commit with config 2022-12-26 13:51:27 -03:00
Emmanuel Rondan 09dc4cf308 adding black to tox and defining line length on pyproject.toml 2022-12-22 16:11:21 -03:00
Andrey Rahmatullin 5bd27191a2
Bump mypy, flake8, and pylint (#5738) 2022-11-28 19:37:57 +01:00
Andrey Rakhmatullin 5bf4260679 Update flake8. 2022-10-10 11:14:20 +06:00
Andrey Rakhmatullin 5fa613b419 Run flake8 directly. 2022-10-10 11:04:00 +06:00
Andrey Rakhmatullin d1515cc075 Update pylint. 2022-10-07 15:30:44 +06:00
Andrey Rakhmatullin ccb6a8c098 Add a note about flake8. 2022-10-07 15:27:14 +06:00
Andrey Rakhmatullin 424849b275 Update mypy. 2022-10-07 15:17:55 +06:00
Andrey Rahmatullin 28d65c88b1
Merge pull request #5662 from gliptak/pyopenssl1 
Match pyOpenSSL and types-pyOpenSSL versions
 2022-10-04 12:54:30 +05:00
Abdul Rauf 69bf5c6625 CI: move twinecheck to tox env 2022-10-02 20:27:24 +05:00
Gábor Lipták 9f006e3aa5 Match pyOpenSSL and types-pyOpenSSL versions 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-30 22:27:57 -04:00
Gábor Lipták 385acd5598
Match pyOpenSSL and service_identity to Twisted 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-24 21:33:02 -04:00
Mikhail Korobov 56e2eeac10 fix typing issues by upgrading mypy 2022-07-27 09:41:12 +05:00
Adrián Chaves 5862f6b8e1
Update frozen CI packages (#5574) 2022-07-25 17:25:10 +02:00
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Maxime Nannan 078622cfb0
Fix file expiration issue with GCS (#5318) 2022-05-20 08:30:06 +02:00
Andrey Rahmatullin 965fde24a4 Pin mitmproxy to < 8 for now (#5459) 2022-05-20 07:17:09 +02:00
Adrián Chaves 50c8becbe0
Freeze and upgrade CI packages (#5429) 2022-03-01 17:29:08 +01:00
Andrey Rahmatullin 08557e09db
Pin old markupsafe when we pin old mitmproxy (#5427) 2022-02-23 19:52:18 +01:00
Andrey Rakhmatullin 77547a1ab5 Revert "Temporarily pin Twisted to an older version in CI (#5401)" 
This reverts commit b282a7af01.
 2022-02-08 21:06:02 +05:00
Andrey Rahmatullin b282a7af01
Temporarily pin Twisted to an older version in CI (#5401) 2022-02-07 21:25:08 +01:00
Andrey Rakhmatullin 6ec66c96fb Fix and pin pylint. 2021-11-26 12:25:45 +05:00
azzamsa 51adf71b1b refactor: use `pytest` command as the recommended entry point 
`pytest` is recommended command since pytest 3.0.
There is a possibility for `py.test` to be deprecated or even removed.

https://github.com/pytest-dev/pytest/issues/1629
 2021-10-24 10:52:56 +07:00
Eugenio Lacuesta cfff79cee6
Make Python 3.10 support official (#5265) 2021-10-18 22:09:17 +02:00
Adrián Chaves 98ee3ddffe
Freeze flake8 2021-10-15 17:18:46 +02:00
Andrey Rakhmatullin b63369c148 Rename tests/requirements-py3.txt to tests/requirements.txt. 2021-08-11 20:02:45 +05:00
laggardkernel 7e4321f201 Add support for temporary security credential in AWS auth 2021-07-27 19:39:59 +08:00
Pascal Corpet 70dddfe2b2 Typing: switch to a newer version of MyPy to check types 2021-07-21 17:10:10 +02:00
Adrián Chaves c5b1ee8101
Make Twisted[http2] installation optional (#5113) 
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
 2021-05-11 09:04:53 +02:00
Eugenio Lacuesta d8d1dc5b50
Ignore typing warning in test 2021-04-12 10:43:02 -03:00