RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,281 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

57 Commits

Author SHA1 Message Date
Adrian Chaves 16389bc1ed Merge remote-tracking branch 'origin/master' into proofreading 2026-07-30 09:00:14 +02:00
Adrian a54c438da1
Provide extras for optional dependencies (#7596) 2026-07-22 10:44:26 +02:00
Andrey Rakhmatullin 36bf1185e5
Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737) 2026-07-21 12:22:04 +02:00
Andrey Rakhmatullin 7e881ce2d7
Release notes for 2.15.0 (#7373) 
* Cover 2.15.0 in the release notes (up to 31bf7c3).

* Replace VERSION.

* Document unified download handler exceptions.

* Updates.

* Cover 2.15.0 in the release notes (up to 0c6ccf5).

* Cover 2.15.0 in the release notes (up to b68f267).

* Run linkcheck and fix issues.

* Cleanup unused link targets.

* Cleanup.
 2026-04-09 16:56:30 +05:00
Andrey Rakhmatullin b9be5ce053
Reactorless shell and other small shell improvements (#7395) 
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.

* defer.Deferred -> Deferred.

* Allow Shell with TWISTED_REACTOR_ENABLED=False.

* Shell workflow notes.

* Remove an unused argument.

* Shell.fetch_available.

* Shell._use_reactor.

* Some more comments.

* Simplify _schedule() and fix the spider shell var.

* More async def.

* Reactorless shell support.

* Add pragma: no cover.

* More notes.

* Remove a TODO.
 2026-04-08 18:24:51 +05:00
Andrey Rakhmatullin b2b2d0b015
Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394) 2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin 0c6ccf50b3
Docs for the reactorless mode (#7385) 
* Reactorless mode docs.

* Unset the ftp handler in the reactorless mode.

* Cleanup reactorless subprocess tests.

* Better handling of incompatible TWISTED_ENABLED cases.

* Improvements.

* Improve early error logging in test scripts.

* Address feedback.

* Address feedback.
 2026-04-01 20:40:34 +05:00
Adrián Chaves fe057bece0 Address typos and grammar issues in docs/ 2025-10-23 11:52:57 +02:00
Sebastien Dionne a02abdcf63
Fix typos in documentation (#7076) 2025-10-06 11:43:36 +02:00
Andrey Rakhmatullin daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts (#6888) 
* Remove the chdir fixture.

* Re-enable fancy pytest asserts.

* Remove doc files from pytest ignores.

* Restore docs/_ext in test collection ignores.

* Skip a doctest that fails on Windows.

* Fix tests that were writing to the current dir.
 2025-06-14 14:01:20 +05:00
pankaj1707k cc9eb3fa79
Format python console blocks using blacken-docs 
Prepend '.. code-block:: pycon' to make python console blocks detectable by blacken-docs
 2023-02-01 20:37:39 +05:30
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs' 
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
 2023-02-01 16:30:57 +05:30
Kian-Meng, Ang 3a263280ba Fix typos 2021-10-11 22:32:42 +08:00
Marlena Chatzigrigoriou d7deba7e89
Document all import paths and use the shortest in examples (#5099) 2021-07-14 10:34:28 +02:00
Adrián Chaves 1773eaf5dc Move lists closer to their introducing paragraph 2021-02-03 11:43:18 +01:00
sakshamb2113 bfeb2c8c13
Added warning to use double quotes in Windows for scrapy shell in shell.rst (#4450) 
* modified debugging memory leaks with guppy in leaks.rst

* modified leaks.rst(issue #4285)

* removed guppy from telnet.py

* Fix undefined name error

* removed hpy key from telnet_vars in telnet.py

* updated shell.rst

* Update docs/topics/shell.rst

Co-Authored-By: Adrián Chaves <adrian@chaves.io>

Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2020-04-18 20:21:26 +05:00
nyov a34c366fa4 DOC linkcheck run; https and 301 link updates. 
Closes #4359
 2020-02-25 16:51:33 +00:00
Mikhail Korobov 6a32a36e09
Merge pull request #4143 from Gallaecio/undent-rst-doctests 
Do not indent doctests from the documentation unnecessarily
 2019-12-20 01:59:07 +05:00
Marc Hernandez Cabot 23a67cec27 fix first letter capitalization for Raring and Scrapy 2019-12-19 09:57:17 +01:00
Adrián Chaves 83b8046fdc Do not indent doctests from the documentation unnecessarily 2019-12-05 13:52:31 +01:00
Mikhail Korobov bdcc045f62 DOC switch from .extract to get/getall API in docs 
Also, response.urljoin is added in a few places, for robustness.
 2018-09-12 17:57:27 +05:00
Mikhail Korobov ca27010d4f DOC .extract_first() -> .get() 2018-09-12 17:57:27 +05:00
Aditya 9d9d83a8c3 Use https links wherever possible 2017-10-28 16:24:40 +05:30
Aditya 9dd680d5c9 Use https for external links wherever possible in docs 2017-10-26 23:33:45 +05:30
Elias Dorneles d09ec3db68 Merge pull request #2410 from redapple/fetch-transparent-redirect 
[MRG+1] Transparently handle redirections in fetch and shell
 2016-12-21 09:49:15 -02:00
Mikhail Korobov 0fc73a9d55 DOC update examples with long longger names 2016-12-16 21:47:58 +05:00
Paul Tremberth 7d17836032 Update documentation about --no-redirect option 2016-12-08 17:27:25 +01:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth bb1f4013a3 Rewrite warning about shell with local files as note 2016-01-26 17:23:28 +01:00
Paul Tremberth 713e1eee9b Update docs about local files support for "scrapy shell" 2016-01-26 10:44:38 +01:00
Valdir Stumm Jr 2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
smirecki 8379bea7ed Typos corrections 
I've made a few small corrections, some spelling changes and typo fixes.
I've tried to respect regional spelling differences and avoided proposing hyphenating compound words.

 Please enter the commit message for your changes. Lines starting
 2015-10-02 23:48:27 -04:00
nyov 509cc8d41e Add support for bpython console. 
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.

config snippet:

cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF

(closes #270, #1100, #1301)
 2015-08-21 01:12:58 +01:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
Julia Medina 7a92dae4c8 Change Scrapy log output through docs 2015-04-22 17:27:24 -03:00
Diana Huang 91a60d9f3d Remove references to the `sel` object in shell.rst 
The current documentation has references to the deprecated `sel` when interacting with the shell. I've removed them and replaced uses of `sel.xpath` with `response.xpath` instead.
 2015-04-12 13:44:32 -04:00
Julia Medina c90977ca98 Drop support for scrapy.project.crawler (And scrapy.stats consequently) 2014-08-12 14:02:56 -03:00
Lisimia b6b44a33f7 Update shell.rst typo 
There was a typo. 
"start" was mistyped as "star"
 2014-07-28 12:26:13 -04:00
Julia Medina bdca06240c Fix settings repr on the logs of the shell and tutorial docs topics 2014-06-10 11:26:50 -03:00
Mikhail Korobov 2d3803672b DOC use top-level shortcuts in docs 2014-04-15 01:09:35 +06:00
Rolando Espinoza 9aab9224cb Updated shell docs with the crawler reference and fixed the actual shell output. 
Also updated the shell example with a reproducible code example.
 2014-01-23 18:04:57 -04:00
Mikhail Korobov a27d91f0a6 Rename BaseSpider to Spider. See GH-495. 2013-12-30 19:46:41 +06:00
Mikhail Korobov e713733edf minor fixes to scrapy shell docs 
* better IPython links;
* MDC link instead of w3schools;
* small formatting fixes;
* show quoted URL in example
 2013-12-30 10:27:39 +06:00
Daniel Graña 155ea08ea1 use `sel` name for Selector's instances in docs, internals and shell 2013-10-15 15:58:42 -02:00
Daniel Graña 4645f9e03c Updates docs to reflect unified selectors api 2013-10-14 16:31:20 -02:00
Pablo Hoffman ebdb733e95 Updated some old messages in Scrapy shell doc 2010-09-05 04:45:43 -03:00
Pablo Hoffman c276c48c91 Added settings to Scrapy shell variables 2010-08-21 05:10:06 -03:00
Pablo Hoffman 9aefa242d5 Applied documentation patch provided by Lucian Ursu (closes #207) 2010-08-21 01:26:35 -03:00
Pablo Hoffman 1d3b9e2ca8 Scrapy shell refactoring 2010-08-20 11:26:14 -03:00
Pablo Hoffman 7858244dca Scrapy shell: moved python console starting code to scrapy.utils.console and get rid of noisy console banners 2010-08-20 01:33:02 -03:00