RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

1904 Commits

Author SHA1 Message Date
Georgiy Zatserklianyi 8b6a50a935 periodic_log: docs added 2023-08-04 11:12:07 +03:00
Adrián Chaves 72de48be6d asyncio: cover accidental bad reactor installation, sort sections, reword the Windows section 2023-08-04 09:56:30 +02:00
Andrey Rakhmatullin 7c6aaedda2
Merge pull request #5846 from jxlil/docs/spiders 
Added documentation for Spider.update_settings
 2023-08-02 21:09:44 +04:00
Andrey Rakhmatullin af1be835e4
Apply suggestions from code review 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-08-02 19:46:16 +04:00
Andrey Rakhmatullin 72462a53e2 Add more docs for update_settings(). 2023-08-02 12:32:53 +04:00
Andrey Rakhmatullin 06ebdee35d
Merge pull request #5950 from wRAR/addons 
Add-ons
 2023-08-01 22:18:57 +04:00
Andrey Rakhmatullin cf55eb05f5 Fix a typo. 2023-08-01 13:30:56 +04:00
Andrey Rakhmatullin 41a4a163e3 Handle NotConfigured in add-ons. 2023-07-31 21:09:18 +04:00
Andrey Rakhmatullin b67a81b81d Use the MyHandler class directly. 2023-07-31 20:25:56 +04:00
Andrey Rakhmatullin d8c5b41559 Add more addon tests. 2023-07-31 19:09:21 +04:00
Andrey Rakhmatullin bddbbc522a
Merge pull request #5979 from BurnzZ/closespider-timeout-no-item 2023-07-28 12:56:44 +04:00
Andrey Rakhmatullin 3ba2dc4d68 Fixes for addon docs. 2023-07-24 17:49:04 +04:00
Kevin Lloyd Bernal 9a1bf40c2f expose AWS_REGION_NAME in S3FeedStorage 2023-07-24 16:37:52 +08:00
Mikhail Korobov af2aa4b421
Merge pull request #5847 from namelessGonbai/feat/FixFeedExport 
Fix FeedExporter not to export empty file
 2023-07-22 23:18:36 +05:00
Andrey Rakhmatullin 005c8cc5f0 Unify the "add-on" spelling. 2023-07-19 13:15:35 +04:00
Andrey Rakhmatullin 0a25a300cf Fix docs. 2023-07-18 20:29:42 +04:00
Andrey Rakhmatullin 90dae3ee60 Doc fixes. 2023-07-18 19:55:46 +04:00
Kevin Lloyd Bernal 3f5bbe3a8f introduce CLOSESPIDER_TIMEOUT_NO_ITEM in CloseSpider 2023-07-18 18:30:21 +08:00
Andrey Rakhmatullin cdda8ad46d Add docs about fallbacks in addons. 2023-07-17 21:05:58 +04:00
Andrey Rakhmatullin a2264d3b8b Improve docs about setting settings in addons. 2023-07-14 18:57:27 +04:00
Andrey Rakhmatullin c92c9af075 Add create_instance support to addons. 2023-06-29 15:04:46 +04:00
Andrey Rakhmatullin 9612ae3e93 Remove more code. 2023-06-28 18:44:30 +04:00
Andrey Rakhmatullin e7124447f7 Remove unneeded code. 2023-06-26 16:57:46 +04:00
Andrey Rakhmatullin 4e21e3e59d Merge remote-tracking branch 'origin/master' into addons 2023-06-26 14:25:35 +04:00
Alex 04ee3303e4
Adding support for Windows of absolute pathlib.Path objects in FeedExporter (#5939) 2023-06-22 07:04:06 +02:00
Aaron Smith ee215a2970
Change redirect text from Response.request docs (#5937) 2023-06-21 20:05:39 +02:00
Laerte Pereira 0097b4c0bb cleanup: Remove `pkg_resources` usage 2023-06-20 16:40:38 -03:00
Serhii A 777a6ea412
Make the retry middleware exception list configurable (#5929) 2023-06-16 15:46:06 +02:00
Andrey Rakhmatullin f582246d7b More doc fixes. 2023-06-15 18:49:08 +04:00
Andrey Rakhmatullin 2f9ebb66c3 Remove some dead code/docs. 2023-06-15 17:00:38 +04:00
Andrey Rakhmatullin 27f5f35134 More quick doc fixes. 2023-06-14 18:30:33 +04:00
Andrey Rakhmatullin fdfab17438 Fix docs for the renamed ADDONS setting. 2023-06-14 18:00:18 +04:00
Andrey Rakhmatullin 22bd0d9a79 Fix docs build. 2023-06-14 17:58:11 +04:00
Andrey Rakhmatullin 7ebb8256f0 Some cleanup. 2023-06-14 17:52:11 +04:00
Andrey Rakhmatullin 075ad6f196 Test, linter etc. fixes. 2023-06-14 16:34:18 +04:00
Andrey Rakhmatullin 898e3045a1 Merge branch 'master' into addons 2023-06-14 15:37:08 +04:00
Andrey Rakhmatullin 85fe88f80f
Merge pull request #5833 from GodelTech/issue_960_s3_feed_export 
Fix #960: S3 Feed Export throws boto error
 2023-06-13 18:44:06 +04:00
Andrey Rakhmatullin 8a64f3e8de
Merge branch 'master' into feat/FixFeedExport 2023-06-13 18:21:06 +04:00
Andrey Rakhmatullin 33b418dc84
Merge pull request #5927 from bulatbulat48/master 
Add additional requests examples.
 2023-05-16 19:13:02 +04:00
Adrián Chaves d362699fa3 Move inline request examples to the coroutines documentation 2023-05-16 13:39:02 +02:00
Adrián Chaves e4cf8fc121
Update asyncio.rst 2023-05-15 18:51:58 +02:00
isabela_catanante 5adada5d19 Improve the overwrite feed option documentation 2023-05-12 12:55:24 +02:00
bulat 8de2064ba3 add import. 2023-05-11 13:22:33 +05:00
bulat 4878cc7ef0 Add proper imports. 2023-05-11 13:19:40 +05:00
bulat b62c1263de add import to the example. 2023-05-11 13:15:30 +05:00
bulat fc2d1b2171 make example reachable. 2023-05-11 12:56:29 +05:00
bulat 6a0bbad677 Merge branch 'master' of https://github.com/bulatbulat48/scrapy 2023-05-11 12:54:04 +05:00
bulat 6194db1335 Update title. 2023-05-11 12:54:01 +05:00
Bulat Khabibullin 85103b4932
add proper example 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-05-11 12:53:43 +05:00
Bulat Khabibullin 26374e21f8
Update docs/topics/asyncio.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-05-10 18:32:36 +05:00
Bulat Khabibullin 57f3140daa
Update docs/topics/asyncio.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-05-10 18:31:54 +05:00
bulat 87d10161cd Add selector as class. 2023-05-10 15:35:48 +05:00
bulat b1f4017788 Refactor batch sample. 2023-05-10 15:34:58 +05:00
bulat 99b0ece165 remove extra line. 2023-05-09 20:27:46 +05:00
bulat d32c678234 Update description. 2023-05-09 19:02:34 +05:00
bulat 2c6a772145 Merge branch 'master' of https://github.com/bulatbulat48/scrapy 2023-05-09 18:57:45 +05:00
bulat a75231a1ec fix underline. 2023-05-09 18:57:43 +05:00
Bulat Khabibullin 75cd4958fe
Merge branch 'scrapy:master' into master 2023-05-09 18:05:40 +05:00
bulat c327a92e97 add additional requests examples. 2023-05-09 18:04:18 +05:00
Andrey Rakhmatullin 5bb6dfbc25
Merge pull request #5909 from scrapy/relnotes-2.9 
Release notes for 2.9.0
 2023-05-08 14:27:21 +04:00
Andrey Rakhmatullin cba891a66c Enable doc tests for selectors.rst, fix issues. 2023-05-04 15:04:33 +04:00
Andrey Rakhmatullin ea15ff1d32
Merge pull request #5915 from jxlil/feature/support-parsel-jmespath 
Added support for the Parsel JMESPath feature
 2023-05-03 13:18:39 +04:00
Jalil SA 865c36bdbb update docs 2023-04-28 08:56:11 -06:00
guillermo-bondonno b50c032ee9
Add feed_slot_closed and feed_exporter_closed signals (#5876) 2023-04-26 08:20:37 +02:00
Andrey Rakhmatullin 67bfb304cd Release notes for the current master. 2023-04-21 20:49:51 +04:00
Andrey Rakhmatullin 1e20ba0a1b
Merge branch 'master' into feat/FixFeedExport 2023-04-11 19:48:19 +04:00
Mojtaba Dashtinejad 98571eb946
Add missing slash in selecting dynamically-loaded content documentation (#5890) 2023-04-10 14:14:49 +02:00
Andrey Rakhmatullin 96033ce5a7
Merge pull request #5883 from sdidier-dev/patch-1 
Add missing 'crawl' command in crawl examples ✏️
 2023-03-31 14:47:52 +04:00
Sébastien DIDIER 6d94aa061c
Add missing 'crawl' command in crawl examples 2023-03-31 11:05:20 +02:00
Jalil SA 44cdaa442b
Update docs/topics/spiders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-17 13:19:03 -06:00
Jalil SA 24f28c415c fix: docs update_settings() 2023-03-17 12:16:08 -06:00
Jalil SA a1fc37cbff
Update docs/topics/spiders.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-17 12:13:05 -06:00
Jalil SA 495372648c fix: docs update_settings() 2023-03-16 23:14:57 -06:00
jazzthief cb67bc17b7
Remove `botocore` from docs 2023-03-16 17:25:05 +01:00
namelessGonbai 6ab49e954f
Update docs/topics/feed-exports.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2023-03-16 18:03:06 +09:00
kenshi kikuchi 50801c7207 Fix Docs 2023-03-16 17:17:20 +09:00
Adrián Chaves dfbb63a2f1 Merge remote-tracking branch 'scrapy/master' into issue_960_s3_feed_export 2023-03-15 14:07:54 +01:00
Alex 101a0c32d7
Support genspider with HTTPS (#5808) 2023-03-14 08:13:44 +01:00
Andrey Rakhmatullin 3659a8c8d9
Merge pull request #5328 from GeorgeA92/per_slot_settings 
Per slot settings
 2023-03-09 22:31:24 +05:00
Jalil SA 96d51c3afa docs: update 2023-03-08 04:21:21 -06:00
Jalil SA 1d862d0831 fix: remove line breaks 2023-03-08 03:26:38 -06:00
Jalil SA 05893e1796 docs: Spider.update_settings 2023-03-08 02:59:47 -06:00
Georgiy Zatserklianyi 218829b1db per slot settings: typo fixed 2023-03-06 16:25:49 +02:00
Georgiy Zatserklianyi be52fe4f67 Revert "periodic log stats: typo in docs fixed" 
This reverts commit 68ba25cb69.
 2023-03-06 16:24:28 +02:00
Georgiy Zatserklianyi 68ba25cb69 periodic log stats: typo in docs fixed 2023-03-04 22:15:06 +02:00
Oleg 2b3a8f0d69
Fix a typo in the LOG_FORMAT description (#5839) 2023-03-02 10:19:41 +01:00
Georgiy Zatserklianyi 045092e8d7 per_slot_settings(docs): code style fix (from previous pre-commit check) 2023-02-16 17:55:59 +02:00
Georgiy Zatserklianyi 07e1429877 per_slot_settings(docs): docs `:setting` bindings added 2023-02-16 15:14:01 +02:00
Yegor 29c2477f0a
Document the need to install boto3 2023-02-15 16:40:05 +01:00
Andrey Rakhmatullin 76eba9977b
Skip or fix failing code block tests (#5826) 2023-02-14 09:40:38 +01:00
Georgiy Zatserklianyi 474087be6f per_slot_settings(docs): settings 3 2023-02-09 14:52:28 +02:00
Georgiy Zatserklianyi 5208d436ae per_slot_settings(docs): settings 2 2023-02-09 00:33:21 +02:00
Georgiy Zatserklianyi c3033a54b1 per_slot_settings(docs): settings 1 2023-02-08 23:55:07 +02:00
pankaj1707k 03f32c018f
Update black version and fix minor typos and punctuations 2023-02-03 11:52:20 +05:30
Pankaj Kumar 280cd6ce71
Merge branch 'scrapy:master' into blacken-docs 2023-02-03 11:38:26 +05:30
pankaj1707k 7fdbbd3ccb
Fix for python 3.11 (docs) check on pull request 2023-02-02 15:52:22 +05:30
Adrián Chaves 068af85722
Add release notes for Scrapy 2.8 (#5799) 2023-02-02 04:54:02 +01:00
pankaj1707k cc9eb3fa79
Format python console blocks using blacken-docs 
Prepend '.. code-block:: pycon' to make python console blocks detectable by blacken-docs
 2023-02-01 20:37:39 +05:30
pankaj1707k c1bbb299d7
Add and run pre-commit hook 'blacken-docs' 
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
 2023-02-01 16:30:57 +05:30
Adrián Chaves 9272c4af0c Merge remote-tracking branch 'scrapy/master' into no-callback 2023-01-25 17:41:41 +01:00
Laerte Pereira b611848029
fix(docs): Change `versionchanged` value 2023-01-20 08:35:17 -03:00
Laerte Pereira 973f0cf567
fix: line break 2023-01-20 08:23:05 -03:00
Laerte Pereira 8270df754d
Set `FEED_EXPORT_ENCODING='utf-8'` in the default template 2023-01-20 07:55:16 -03:00
Adrián Chaves a493464942 Update the screenshot pipeline code example 2023-01-19 19:53:53 +01:00
Adrián Chaves 50500a6b28 Implement a NO_CALLBACK value for Request.callback 2023-01-19 17:14:18 +01:00
Andrey Rakhmatullin f87d061069
Merge branch 'master' into download-delay-documentation 2023-01-13 00:22:48 +05:00
Josh Soref b62aacfee3 spelling: initializing 
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
 2022-12-18 21:33:21 -05:00
Yohanna Lisnichuk b63ca6f834 docs: use the right notes directive 2022-12-15 13:20:45 -03:00
Yohanna Lisnichuk 96e526aad8
docs: change DOWNLOAD_DELAY description format 
To be consistent with the other settings' descriptions
 2022-12-15 10:01:55 -03:00
Andrey Rahmatullin d19a216e10
Merge branch 'master' into pathlib 2022-11-25 18:20:25 +05:00
Pablo Hoffman c04ccbceb9 doc: add instructions to debug Scrapy spiders in Visual Studio Code 2022-11-21 15:49:33 +01:00
Hassan Shoayb 29bf7f5a6c
broad-crawls.rst: fix a typo (#5714) 2022-11-14 09:45:00 +01:00
Magnus Offermanns fd692f3091
Prevent running the -O and -t command-line options together (#5605) 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2022-10-27 14:43:31 +02:00
Mikhail Korobov 82f25bc44a
Merge pull request #5681 from mattkohl-flex/master 
Documentation: typo fixes
 2022-10-18 15:27:52 +05:00
Matthew Donoughe 40d9ca3bdd
use pathlib 2022-10-17 17:40:10 -04:00
Adrián Chaves 06c8f673af
2.7 release notes (#5680) 
* Fix the display name of documented fingerprinter class methods

* Initial draft for the Scrapy 2.7 release notes

* Update VERSION and PREVIOUS_VERSION references

* Clarify the restrictions lifted for item field output names

* Fix the description of the BOM bug fix

* Fix the note about changes in MIME sniffing

* Fix typo

* Extend highlights

* Fyx typo
 2022-10-17 18:04:29 +05:00
mattkohl-flex c49764ffd7 typo fixes 2022-10-17 11:15:17 +01:00
Andrey Rakhmatullin 62cc26e209 Change TWISTED_REACTOR in the default template. 2022-10-14 22:03:54 +06:00
Mikhail Korobov 4f296b61b0
Merge pull request #5608 from zaid-ismail031/master 
Changed incorrect info in documentation regarding the return type of parse method.
 2022-10-03 00:19:08 +05:00
Gábor Lipták 79a4bc3da0
Cleanup METHOD_SSLv3 
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
 2022-09-25 17:01:43 -04:00
Tim B 5f19420211
Documented how settings must be picklable (#5629) 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Laerte Pereira <5853172+Laerte@users.noreply.github.com>
 2022-09-21 08:27:27 +02:00
zaid-ismail031 c49b5aaf77 Changed incorrect information regarding the return type of parse/request callback method. 2022-08-26 21:17:31 +02:00
Andrey Rahmatullin 83ecdf1bca
Update docs/topics/components.rst 
Co-authored-by: Adrián Chaves <adrian@chaves.io>
 2022-07-27 23:12:31 +05:00
Mikhail Korobov 4dde2f2c36 Merge branch 'master' into asyncio-parse-fixes 2022-07-27 09:32:54 +05:00
Adrián Chaves 00ccb02f1f Merge remote-tracking branch 'upstream/2.6' 2022-07-25 14:13:22 +02:00
Andrey Rahmatullin a0681fb811
Merge branch 'master' into asyncio-parse-asyncgen-proper-rebased 2022-07-22 18:30:38 +05:00
Mikhail Korobov f60c7ae768
Fixed heading levels in downloader middleware docs (#5567) 2022-07-20 11:01:22 +02:00
Mikhail Korobov 77a3b02523
Merge pull request #5553 from Rotzbua/fix-doc-typo 
Fix doc: `scrapy.exporter` to `scrapy.exporters`
 2022-07-17 00:21:50 +05:00
Rotzbua 1c7ed4f2e5
[doc] Remove incompatible web service project 
* Abandoned since 2017
* Not compatible with Python3
 2022-07-12 13:09:20 +02:00
Rotzbua 09c3a4ad08
Fix doc: `scrapy.exporter` to `scrapy.exporters` 2022-07-12 12:45:15 +02:00
Mikhail Korobov 500dae82e2
Merge pull request #3696 from Gallaecio/allow-customizing-export-column-names 
Allow customizing export column names
 2022-06-27 00:26:05 +05:00
Adrián Chaves 34e4ed72ea Document how DOWNLOAD_DELAY affects per-domain concurrency 2022-06-21 12:46:54 +02:00
Adrián Chaves 4ef71829b2
If TWISTED_REACTOR is None, reuse any pre-installed reactor (#5528) 2022-06-17 10:37:27 +02:00
Michel Ace 6e878490e8
Support and prefer the .jsonl file extension (#4848) 2022-06-17 08:37:14 +02:00
Adrián Chaves 3e994bda45 Update for Python 3.7+ 2022-06-16 20:53:14 +02:00
Adrián Chaves e8503217fb Merge remote-tracking branch 'upstream/master' into allow-customizing-export-column-names 2022-06-16 20:19:52 +02:00
Laerte Pereira 407562b38b
Drop Python 3.6 support (#5514) 
* chore: Drop Python 3.6 support

* Attend PR comments

* Tweak versions

* Update dependencies version

* fix: Ubuntu workflow

* fix windows workflow

* chore: Remove comment

* update `install_requires` dependencies versions

* move lxml to main pinned requirements

* Attend code-review comments

* remove non-pinned 3.7 from windows workflow

* simplify condition

* lint

* remove paragraph

* refactor

* remove leftover
 2022-06-09 00:25:03 -03:00
Adrián Chaves c5627af15b
Centralize request fingerprints (#4524) 
Co-authored-by: Mikhail Korobov <kmike84@gmail.com>
 2022-06-07 18:44:54 +02:00
Mikhail Korobov 517cbc8d71
Merge pull request #5508 from mtabbasi/thumb_path-access-item 
pass on item to thumb_path function as additional argument
 2022-06-05 03:42:45 +05:00
Mikhail Korobov bafe874ecc
Merge pull request #5358 from burakcank/signal_coroutine 
Document coroutines for signals.
 2022-05-28 13:38:05 +05:00
Mohammadtaher Abbasi f39def4492 add docs 2022-05-25 23:57:38 +04:30
Yash 319e67f779 documentation update for multiple spiders 
i noticed passing settings to configure logging function made weird output go away. checked documentation and it says first parameter is settings file. Is this correct?
 2022-04-12 11:42:22 -05:00
Andrey Rahmatullin 8c3e5a2421
Merge pull request #5457 from d00399830/typo_fixes_sm 
Fix capitalization of JavaScript within the Developer Tools documentation
 2022-03-23 11:06:54 +05:00
Víctor Ruiz 2227be7af6
Fix a typo in the HTTP cache documentation (#5455) 2022-03-22 15:21:16 +01:00
D00399830 fcf3d8e0a0 Updated the documentation for developer tools to have JavaScript instead of Javascript, as JavaScript is the more correct way to write it 2022-03-21 14:09:31 -06:00
Adrián Chaves b95c634b86 Document how to enforce Scrapy versions on Scrapy components 2022-03-17 22:23:25 +01:00
Eugene 9a28eb0bad
Suggest installing the brotli package instead of brotlipy (#4267) 2022-03-17 05:39:54 +01:00
Adrián Chaves b78e6915c6 Clarify that without async-to-sync conversions items yielded before an exception are processed 2022-03-16 20:17:25 +01:00
Adrián Chaves fd08bb6cd9 Refactor the asynchronous process_spider_output documentation 2022-03-16 18:10:57 +01:00
Adrián Chaves cb3afcf0e7 Merge remote-tracking branch 'upstream/master' into wrar-asyncio-parse-asyncgen-proper-rebased 2022-03-16 18:08:47 +01:00