RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,317 Commits 49 Branches 132 Tags 148 MiB
Commit Graph

2071 Commits

Author SHA1 Message Date
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian 482a02d30c
Add a cookies documentation page (#7947) 2026-08-09 22:41:32 +05:00
Adrian 786ab10494
Document how to write a custom item exporter (#7931) 2026-08-09 22:31:53 +05:00
Adrian 609f64c55d
Make brotli a hard dependency (#7929) 2026-08-09 19:24:14 +02:00
Adrian 5427080f48
Create a page on optimization (#7938) 2026-08-09 18:37:24 +02:00
Adrian a6c017c2ca
Advise setting an identifying user agent (#7890) 2026-08-09 16:59:22 +05:00
Adrian 18ed0c0f7c
Fall back to the response encoding in TextResponse.json() (#7897) 2026-08-09 16:58:09 +05:00
Adrian 050a8cf159
Improve the docs about delaying start request iteration (#7883) 2026-08-09 16:57:08 +05:00
Adrian 59ce27afdd
Send the bytes_received and headers_received signals over HTTP/2 (#7896) 2026-08-09 16:55:02 +05:00
Adrian 7c797968a3
Docs: clarify the handling of exceptions raised in errbacks (#7898) 2026-08-09 16:53:33 +05:00
Adrian a18d58d7b5
Document that process_spider_output receives a lazy result (#7939) 2026-08-09 12:26:15 +02:00
Adrian 9e84112221
Cover update_vars() in the shell docs (#7889) 2026-08-09 14:25:14 +05:00
Adrian 0b2d220197
Improve docs for multi-spider runs (#7907) 2026-08-09 14:19:50 +05:00
Adrian 0e324f3d4a
Let spiders change allowed_domains at run time (#7912) 2026-08-09 14:17:44 +05:00
Adrian 4a69e48f0f
Log the first depth-limited link only (#7916) 2026-08-09 14:08:49 +05:00
Adrian 81d12c6eb8
Document the Referer caveat of DEFAULT_REQUEST_HEADERS (#7917) 2026-08-09 14:08:06 +05:00
Adrian 5b4888a0b1
Document that signal handler order is undefined (#7941) 2026-08-09 14:07:23 +05:00
Adrian 91b70e4db4
Improve the docs about scrapy parse --pipelines (#7876) 2026-08-04 17:30:51 +02:00
Adrian 6e2081ca41
Ask custom download handlers not to use engine.download_async() (#7871) 2026-08-04 14:41:25 +05:00
Adrian 54da6c88aa
Deprecate the download_delay spider attribute, and fix the suggested replacement for max_concurrent_requests (#7833) 
* Deprecate the download_delay and max_concurrent_requests spider attributes

* Fix the deprecation entry of max_concurrent_requests
 2026-08-03 23:46:41 +05:00
Adrian a9f1770306
Docs: sort and compact the component-settings list (#7862) 2026-08-03 13:03:28 +02:00
Adrian e83c709574
Docs: a job directory belongs to one Scrapy version (#7861) 2026-08-03 12:44:48 +02:00
Adrian 14478e3f24
Support CONCURRENT_REQUESTS = 0 for unlimited concurrency (#7840) 2026-07-31 20:19:07 +05:00
Adrian 1f03fbc17e
Add scrapy.utils.asyncio.sleep() (#7843) 2026-07-31 18:59:38 +05:00
Adrian 6cefaa5434
Add a stats reference (#7814) 2026-07-31 15:43:40 +05:00
Adrian 746bc7548d
Clarify crawl vs runspider in help and docs (#7832) 2026-07-31 12:17:34 +05:00
Adrian 37661508db
Add RobotParser.crawl_delay() and a robots_parsed signal (#7830) 
* Add RobotParser.crawl_delay() and a robots_parsed signal

* Improve test coverage
 2026-07-31 11:59:52 +05:00
Adrian 434fd1154a
Improve pre-crawler setting docs (#7835) 2026-07-31 11:38:33 +05:00
Adrian f02a99fe71
Add doc sections for callbacks and errbacks (#7821) 2026-07-30 20:15:15 +05:00
Adrian 433603e6ca
Add AWS_MAX_POOL_CONNECTIONS (#7794) 2026-07-30 15:51:45 +02:00
Adrian 98696efa80
Export item fields in declaration order (#7824) 2026-07-30 15:46:45 +05:00
Laerte Pereira 5b4828a012
docs(practices): Remove scrapoxy mention (#7817) 2026-07-30 08:02:54 +02:00
Adrian 0cbb20e8e8
Treat broken cache records as cache misses (#7805) 2026-07-29 14:39:27 +05:00
Youssef Mohamed bc5b5fb1f6
Add Request.to_curl() (#7743) (#7802) 2026-07-29 13:47:15 +05:00
Adrian 58ed9fdccc
Document urlparse_cached (#7777) 2026-07-24 20:08:58 +05:00
Adrian 41bb09741a
Use autodoc for contracts (#7775) 2026-07-24 15:35:54 +02:00
Adrian 0b578c1cbf
trackref: use autodoc (#7771) 2026-07-24 16:24:54 +05:00
Adrian abbc024bbe
Use autodoc for stats.rst (#7769) 2026-07-23 19:48:59 +05:00
Adrian 67e5282684
Use autodoc for exceptions and improve their docs (#7767) 2026-07-23 16:02:41 +05:00
Andrey Rakhmatullin 56dee203e9
Use httpx2 as a preferred alternative to httpx. (#7762) 2026-07-23 10:29:22 +02:00
Adrian 1157b3e235
Add a middleware to detect response.meta copies (#7588) 2026-07-22 13:27:31 +02:00
Adrian a54c438da1
Provide extras for optional dependencies (#7596) 2026-07-22 10:44:26 +02:00
Andrey Rakhmatullin ca21306df7
Reactorless import hook improvements. (#7747) 
* Reactorless import hook improvements.

* Replace prints with logging.

* More tests.

* Uninstall the import hook earlier.

* Add pragma: no branch.
 2026-07-21 19:58:40 +05:00
Andrey Rakhmatullin 394c2797f3
Remove non-Twisted uses of zope.interface (#7731) 2026-07-21 13:30:21 +02:00
Andrey Rakhmatullin 36bf1185e5
Modernize code examples, drop docs for nonexistent `MEMDEBUG_NOTIFY` (#7737) 2026-07-21 12:22:04 +02:00
Andrey Rakhmatullin 1c5404dce5
Add docs about running Scrapy from apps and notebooks. (#7751) 2026-07-21 11:23:52 +02:00
Andrey Rakhmatullin a591d15c04
Deprecate Spider.log(). (#7739) 2026-07-20 14:59:55 +02:00
Andrey Rakhmatullin d8ba1571e7
Assorted docs fixes, part 2. (#7725) 
* Assorted docs fixes, part 2.

* Second pass.

* Address feedback.
 2026-07-14 00:49:44 +05:00
Adrian b3670369b8
Solve the feed Path issue (#7674) 
* Solve the feed Path issue

* Address additional, related issues
 2026-07-13 22:46:41 +05:00
Andrey Rakhmatullin bdf3067935
Assorted docs fixes, part 1. (#7710) 2026-07-07 18:39:34 +05:00