Commit Graph

24 Commits

Author SHA1 Message Date
Adrian fe96c1f54b
Serialize dates and times as ISO 8601 in ScrapyJSONEncoder (#7918) 2026-08-10 14:07:42 +05:00
Andrey Rakhmatullin 61b4befc60
Remove code deprecated in 2.12.0 (#7126)
* Remove smaller things.

* Remove from_settings().

* Remove old MediaPipeline init code.

* Cleanup.

* Typo.
2025-10-27 17:14:38 +05:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Rohit Kumar Singh d2156696c4
Deprecate unused scrapy utils (#6519)
* Added deprecation warnings for unused Scrapy.utils

* Grammatical corrections

* Exceptions class connected

* Deprecation of ScrapyJSONDecoder

* request_authenticate function deprecation

* Making all warning similar

* Added ignore statements for deprecation warning in tests

* Missing stacklevel attr. added

* Added Deprecation message
2024-11-04 21:09:45 +05:00
Andrey Rakhmatullin 4596a58a13 Typing for smaller scrapy/utils/ modules. 2023-05-07 00:54:23 +04:00
Adrián Chaves b07d3f85a3 Upgrade CI tools 2023-02-02 06:37:40 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677)
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
2022-11-27 11:00:13 +01:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Marc Hernández 336f19f5cc
Change super syntax (#4707) 2020-08-04 20:42:01 +02:00
Eugenio Lacuesta ec98dabfab
Support for dataclass and attrs items (#3881) 2020-06-14 11:45:27 +02:00
Eugenio Lacuesta 5256eae60d
Meta class to handle isinstance checks for BaseItem 2020-05-07 14:37:41 -03:00
Diego Allen e17fdd7276 Add set serialization to ScrapyJSONEncoder 2016-07-22 17:20:03 -04:00
Daniel Graña a9292cfab7 jsonrpc webservice moved to https://github.com/scrapy/scrapy-jsonrpc repository 2014-08-15 23:28:13 -03:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
Pablo Hoffman 5fa5c4545b use ScrapyJSONEncoder JsonItemExporter & JsonLinesItemExporter, to support nested items properly 2012-08-22 13:46:50 -03:00
Pablo Hoffman 4f28ffcb2c removed no longer needed dependency on simplejson 2012-04-10 16:01:36 -03:00
Pablo Hoffman e521da2e2f Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
Pablo Hoffman 5c63b2307f Another step towards singleton removal: deprecated crawler singleton import (from scrapy.project import crawler) by a new class method that extensions can implement to receive the crawler 2011-08-08 11:42:44 -03:00
Pablo Hoffman ef7a097272 Replaced old manager references with crawler 2010-08-25 19:31:04 -03:00
Pablo Hoffman 8fc78c4d0a Refactoring of Crawler, Commands, Execution Queue and Spider Manager:
Commands changes:

* removed (somewhat hacky) --init argument from settings command
* added set_crawler method to Commands, and a ``crawler`` property that returns
  a configured crawler. This way, commands that don't require a crawler (such
  as startproject) won't need to configure one.

Execution Queue changes:

* changed SERVICE_QUEUE_FILE setting to SQLITE_DB
* removed SERVICE_QUEUE setting
* added QUEUE_CLASS setting for defining the class to use for the execution queue
* added SERVER_QUEUE_CLASS setting for defining the class to use for the
  execution queue in server mode (runserver command)

Spider Manager changes:

* simplified SpiderManager API by removing the load() method
* added from_settings classmethod to SpiderManager
* added spider_modules constructor argument to SpiderManager

Crawler changes:

* added install() method to Crawler (to install it in scrapy.project) and
  uninstall() to remove it
* use CrawlerProcess.install() in scrapy.cmdline
* use crawler.install() and crawler.uninstall() in tests that a crawler in
  scrapy.project
* make telnet console and webservice play nicer with twisted by stopping
  listening when then engine goes down
* refactored Scrapy engine tests - it no longer uses the crawler singleton.
  Closes #215.
2010-08-25 19:24:36 -03:00
Pablo Hoffman faf7a7da83 Moved scrapymanager singleton to scrapy.project module. Refs #189
Detail of changes:

* Moved scrapy.core.manager.ExecutionManager class to scrapy.crawler.Crawler
* Added scrapy.project.crawler singleton to reference a singleton instance of
  Crawler class (previously known as scrapymanager)
* Left an alias scrapy.core.manager.scrapymanager to scrapy.project.crawler for
  backwards compatibility (to be removed in Scrapy 0.11)
2010-08-22 02:10:53 -03:00
Pablo Hoffman 495f23dea2 utils.serialize: added support for encoding Deferreds, and to refer spiders by name using 'spider::name' 2010-06-11 18:16:09 -03:00
Pablo Hoffman 6a33d6c4d0 * Added Scrapy Web Service with documentation and tests.
* Marked Web Console as deprecated.
* Removed Web Console documentation to discourage its use.
2010-06-09 13:46:22 -03:00