RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,546 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

79 Commits

Author SHA1 Message Date
Alex 8b09b0e0d7 set asyncio event loop at start of new thread 2022-12-13 20:41:48 -08:00
Eugenio Lacuesta 7e23677b52
Engine: deprecations and type hints (#5090) 2021-04-20 08:45:28 -03:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta ee13c3e95f
Merge branch 'master' into deprecate-baseitem 2020-05-14 13:31:16 -03:00
Eugenio Lacuesta df8a1d1c01
Flake8: Remove E741 2020-05-14 10:40:50 -03:00
Eugenio Lacuesta 5256eae60d
Meta class to handle isinstance checks for BaseItem 2020-05-07 14:37:41 -03:00
elacuesta dfbe1d9507
Remove object base class (#4430) 2020-03-16 20:12:46 +01:00
Mikhail Korobov c4ee4b6075
Merge pull request #4347 from noviluni/deprecate_sel_shortcut 
Remove deprecated `sel` shortcut in scrapy shell
 2020-02-20 02:56:34 +05:00
Adrián Chaves 6972a19707
Remove unused imports 2020-02-19 18:59:09 +01:00
Marc Hernandez Cabot eb21dae524 deprecare sel shortcut in scrapy shell 2020-02-19 17:49:42 +01:00
Akshay Sharma 182445f9d9
Fix a spelling error: ie. → i.e. (#4338) 2020-02-18 17:58:31 +01:00
Andrey Rakhmatullin 69cd2e247e Move a bunch of "from twisted.internet import reactor" inside functions. 2019-12-13 18:07:55 +05:00
Eugenio Lacuesta 415526d922
Remove __future__ imports 2019-11-02 23:26:38 -03:00
Eugenio Lacuesta 793b2376f8 Populate spider variable when using shell.inspect_response 2017-07-03 11:28:04 -03:00
Paul Tremberth 70a69d2199 Use built-in range() 2016-12-12 22:40:48 +01:00
Paul Tremberth 7d17836032 Update documentation about --no-redirect option 2016-12-08 17:27:25 +01:00
Paul Tremberth 778bed07bf Let framework handle only HTTP redirects by default for fetch and shell commands 2016-12-07 17:56:13 +01:00
Paul Tremberth 35b655d2f8 Handle redirects transparently by default in shell and fetch 
Adds --no-status-aware command line option to have previous behaviour
 2016-11-24 12:23:22 +01:00
Paul Tremberth effaab867e Update shell help with availability of scrapy module 2016-09-15 21:37:15 +02:00
Paul Tremberth 105163fece Make scrapy available in shell without explicit import statement 2016-09-15 19:26:53 +02:00
Mikhail Lyundin be7821a482 Optimization - avoid temporary list objects, unnecessary function call 2015-09-08 22:10:24 +03:00
Luar Roji ee66382f29 Fixed banner display in all 3 python shells 2015-08-21 01:12:58 +01:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Julia Medina 9cbbfd8b04 Adjust spiders' utils to new SpiderManager API 2014-08-12 14:02:56 -03:00
Felix Yan 4f24e724a3 PY3: use six.iterkeys, six.iteritems, and tempfile 2014-07-14 18:47:22 +08:00
Daniel Graña 1e7ddc8e52 Restore and deprecate "sel" shortcut 2014-04-24 12:47:45 -03:00
Daniel Graña d37308ad77 Remove "sel" shortcut from scrapy shell$ 
Use response.xpath(), response.css() or response.selector instead.
 2014-04-23 23:58:34 -03:00
Rolando Espinoza 4081ba238d PEP8 minor edits. 2014-01-23 11:13:54 -04:00
Rolando Espinoza 240fdde667 Expose current crawler in the scrapy shell. 2014-01-23 11:08:56 -04:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
alexanderlukanin13 ae4a61a2d5 Added "from __future__ import print_function" in all relevant places except xlib 2013-10-20 11:55:07 +06:00
alexanderlukanin13 d381a35732 Python 3 compatible syntax: print, except, raise, octal numbers; removed Python 2.2 boolean compatibility code in xlib/pydispatch/dispatcher.py 2013-10-20 01:59:35 +06:00
Daniel Graña 155ea08ea1 use `sel` name for Selector's instances in docs, internals and shell 2013-10-15 15:58:42 -02:00
Daniel Graña a9eb0b74c8 port scrapy shell to unified selector api 2013-10-11 19:53:01 -02:00
Capi Etheriel bc17e9d412 Adds HtmlCSSSelector and XmlCSSSelector classes, cssselect as optional dependency. 
Ported .get() from _Element and .text_content() from HTMLMixin

Add CSS selectors to scrapy shell

Documenting CSS Selectors: Constructing selectors

Documenting CSS Selectors: Using Selectors

Make CSS Selectors a default feature.

Adds XPath powers to CSS Selectors and some syntactic sugar.

Removes methods copied over from lxml.html.HtmlMixin.

Updating docs to use new CSS Selector super powers.

Documenting CSS Selectors: Regular Expressions

Moving section after Nesting section, since it mentions it.

Documenting CSS Selectors: Nesting Selectors

Fix XPath specificity in lxml.selector.CSSSelectorMixin.text

Cleaning up unused stuff from cssel.py

Changing the behavior of lxml.selector.CSSSelectorMixin.text.

Concatenating all of the descendant text nodes is more useful
than returning it in pieces (there's xpath() if you need that).

Documenting CSS Selectors: CSS Selector objects

Documenting CSS Selectors: CSSSelectorList objects

Documenting CSS Selectors: HtmlCSSSelector objects

Documenting CSS Selectors: XmlCSSSelector objects

Fixing some documentations typos and errors

Enforcing the 80-char width lines

Tidying up CSS selectors and CSSSelectorMixin objects

Adding some missing references in documentation.

Fixing lxml.selector.CSSSelectorList.text
 2013-10-10 18:23:15 -02:00
Daniel Graña 7b1288ba54 Fix AlreadyCalledError replacing a request in shell command. closes #407 2013-10-09 23:50:35 -02:00
Steven Almeroth 81111dd39a fetch command should catch IgnoreRequest exception 2013-03-02 20:13:14 -06:00
Pablo Hoffman 8d77005047 scrapy shell: start shell in main thread and crawler in secondary thread, instead of the other way around. fixes #100 2012-05-22 19:15:54 -03:00
Pablo Hoffman 6d13de4366 fixed "No free spider slots" bug when calling fetch() from scrapy shell 2011-11-14 20:03:43 -02:00
Pablo Hoffman 84f518fc5e More core changes: 
* removed execution queue (replaced by newer spider queues)
* added real support for returning iterators in Spider.start_requests()
* removed support for passing urls to 'scrapy crawl' command
 2011-07-15 15:18:39 -03:00
Pablo Hoffman 5275343fa1 use handle_httpstatus_all=True in scrapy shell 2011-06-28 17:27:40 -03:00
Pablo Hoffman 03751749a8 Scheduler refactoring which introduces the following changes: 
* dropped deferred stored along with requests in scheduler queues, which will
  add the ability to support persistent schedulers in the future
* moved duplicates filter into the scheduler itself, using the same
  dupe fltering class as before (DUPEFILTER_CLASS setting)
* removed scheduler middleware component to simplify, as it was only used for
  duplicates filtering and that is now done in the scheduler itself
* adapted media pipeline to work with new scheduler
* cleanup old docstrings
 2011-06-06 03:16:56 -03:00
Pablo Hoffman 18d303b5f1 ported internal scrapy.utils imports to w3lib 2011-04-19 01:33:52 -03:00
Pablo Hoffman 09f084c220 simplified scrapy shell code after recent changes. refs #306 2011-01-13 13:11:39 -02:00
Pablo Hoffman 0aac226b42 Fixed bug in Scrapy shell's fetch() which wasn't updating local variables properly. Closes #306 2011-01-13 13:08:11 -02:00
Pablo Hoffman 1d726063d6 * Added tests for shell/fetch/version commands (closes #255) 
* Fixed bug causing Scrapy shell to fail if started without any argument (closes #294)
 2010-11-28 18:14:45 -02:00
Pablo Hoffman 08bbbc2f82 shell: properly refresh all vars when fetching a new request 2010-11-11 18:05:36 -02:00