RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,795 Commits 42 Branches 132 Tags 165 MiB
Commit Graph

97 Commits

Author SHA1 Message Date
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin b70443f2d0 Split ruff and pylint ignores into two categories, some pylint cleanup. 2025-01-02 01:03:42 +05:00
Andrey Rakhmatullin c87354cd46 Enable SIM Ruff rules. 2025-01-02 01:03:36 +05:00
Andrey Rakhmatullin cde0845ab2
Ruff: migrate pyupgrade and bandit, enable some other rules (#6577) 2024-12-10 19:53:27 +01:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
Andrey Rakhmatullin e47110f9a5
Add parameteres to most Deferred instances. (#6414) 2024-06-26 10:01:43 +02:00
Andrey Rakhmatullin 4ed5c5ae91
Use ParamSpec for callables. (#6353) 2024-05-13 12:01:52 +02:00
Andrey Rakhmatullin 6ecc9e0a34
Add typing for scrapy/commands (#6268) 2024-03-06 13:21:08 +01:00
Adrián Chaves 2bfd9a2257 bandit: allow-list false positives 2024-02-29 11:11:42 +01:00
Andrey Rakhmatullin 282767f23b Bump black. 2024-02-28 14:30:38 +05:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Emmanuel Rondan 8ee4817471 reaplying black, fixing conflicts and ignoring bandit checks on test directory 2023-01-20 10:55:46 -03:00
Emmanuel Rondan c5cdd0d30c reaplying black again and removing line length on pre-commit black config 2022-12-29 12:48:20 -03:00
Emmanuel Rondan 303f0a70fc re applying black to the code with default line length 2022-12-29 11:26:06 -03:00
Emmanuel Rondan 12b556a352
Merge branch 'master' into add-black-formatter 2022-12-26 12:25:32 -03:00
Alex 8b09b0e0d7 set asyncio event loop at start of new thread 2022-12-13 20:41:48 -08:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Eugenio Lacuesta df8a1d1c01
Flake8: Remove E741 2020-05-14 10:40:50 -03:00
Eugenio Lacuesta 5256eae60d
Meta class to handle isinstance checks for BaseItem 2020-05-07 14:37:41 -03:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Mikhail Korobov c4ee4b6075
Merge pull request #4347 from noviluni/deprecate_sel_shortcut 
Remove deprecated `sel` shortcut in scrapy shell
 2020-02-20 02:56:34 +05:00
Adrián Chaves 6972a19707
Remove unused imports 2020-02-19 18:59:09 +01:00
Marc Hernandez Cabot eb21dae524 deprecare sel shortcut in scrapy shell 2020-02-19 17:49:42 +01:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00
Andrey Rakhmatullin 69cd2e247e Move a bunch of "from twisted.internet import reactor" inside functions. 2019-12-13 18:07:55 +05:00
Eugenio Lacuesta 415526d922
Remove __future__ imports 2019-11-02 23:26:38 -03:00
Eugenio Lacuesta 793b2376f8 Populate spider variable when using shell.inspect_response 2017-07-03 11:28:04 -03:00
Paul Tremberth 70a69d2199 Use built-in range() 2016-12-12 22:40:48 +01:00
Paul Tremberth 7d17836032 Update documentation about --no-redirect option 2016-12-08 17:27:25 +01:00
Paul Tremberth 778bed07bf Let framework handle only HTTP redirects by default for fetch and shell commands 2016-12-07 17:56:13 +01:00
Paul Tremberth 35b655d2f8 Handle redirects transparently by default in shell and fetch 
Adds --no-status-aware command line option to have previous behaviour
 2016-11-24 12:23:22 +01:00
Paul Tremberth effaab867e Update shell help with availability of scrapy module 2016-09-15 21:37:15 +02:00
Paul Tremberth 105163fece Make scrapy available in shell without explicit import statement 2016-09-15 19:26:53 +02:00
Mikhail Lyundin be7821a482 Optimization - avoid temporary list objects, unnecessary function call 2015-09-08 22:10:24 +03:00
Luar Roji ee66382f29 Fixed banner display in all 3 python shells 2015-08-21 01:12:58 +01:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 9cbbfd8b04 Adjust spiders' utils to new SpiderManager API 2014-08-12 14:02:56 -03:00
Felix Yan 4f24e724a3 PY3: use six.iterkeys, six.iteritems, and tempfile 2014-07-14 18:47:22 +08:00
Daniel Graña 1e7ddc8e52 Restore and deprecate "sel" shortcut 2014-04-24 12:47:45 -03:00
Daniel Graña d37308ad77 Remove "sel" shortcut from scrapy shell$ 
Use response.xpath(), response.css() or response.selector instead.
 2014-04-23 23:58:34 -03:00
Rolando Espinoza 4081ba238d PEP8 minor edits. 2014-01-23 11:13:54 -04:00
Rolando Espinoza 240fdde667 Expose current crawler in the scrapy shell. 2014-01-23 11:08:56 -04:00