RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,596 Commits 49 Branches 132 Tags 175 MiB
Commit Graph

2286 Commits

Author SHA1 Message Date
Adrián Chaves 2a4b7fe0f8 dont_throttle → autothrottle_dont_adjust_delay 2024-11-07 16:17:16 +01:00
Adrián Chaves f57fc454be Replace Slot.throttle with Request.meta['dont_throttle'] 2024-11-05 19:44:30 +01:00
Andrey Rakhmatullin ce5a132f12
Run and fix linkcheck. (#6524) 2024-11-04 11:40:07 +01:00
Andrey Rakhmatullin d85c39f5bc
Deprecation removals. (#6500) 
* Deprecation removals.

* Clean up the default pytest filterwarnings.

* Remove test_get_images_old().

* Redo boto-requiring test filtering.

* Remove an unused function.

* Improve the Crawler.crawl() error message.

* Fix the test.
 2024-10-31 18:06:22 +05:00
Laerte Pereira d2bdbad8c8
Deprecate `scrapy.twisted_version` (#6512) 
* Deprecate scrapy.twisted_version

* fix: typing

* remove typing

* raise default exception if attribute is not found

* remove redudant ()

* add tests

* rollback exception raised

* add filterwarnings again

* change order

* lint
 2024-10-30 00:28:35 +05:00
Andrey Rakhmatullin fcb5ab6cff
Remove code for unsupported Twisted. (#6510) 2024-10-29 10:21:07 +01:00
Andrey Rakhmatullin fffacb9dac Merge remote-tracking branch 'origin/master' into py313 2024-10-22 19:07:10 +05:00
Andrey Rakhmatullin 04d0411bf7
Filter test-time warnings. (#6501) 2024-10-21 12:30:49 +02:00
Andrey Rakhmatullin 5759b3f0f2 Drop Reppy. 2024-10-17 23:41:23 +05:00
Andrey Rakhmatullin 7e07d48cc5 Small 3.7 and 3.8 cleanup. 2024-10-17 23:22:37 +05:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Andrey Rakhmatullin 845b1ffd44 Merge remote-tracking branch 'origin/master' into py313 2024-10-16 14:50:12 +05:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
mmoriniere 46cddc6ecf
Ignore SyntaxError as well when SPIDER_LOADER_WARN_ONLY is set to True (#6484) 2024-10-02 10:04:03 +02:00
Andrey Rakhmatullin e139d22db9 Fix expectations for get_func_args() on 3.13. 2024-09-20 19:28:28 +05:00
LucasSD f260f819e0 Remove debug log message from _schedule_request method 2024-09-09 20:26:02 +01:00
Laerte Pereira b85e5a66ed Add support for meta in Spider Contracts 2024-08-26 23:21:09 -03:00
Georgiy Zatserklianyi 6ce0342beb
Allow yielding items from start_requests (#6417) 
Co-authored-by: Georgiy Zatserklianyi <george.zatseklyany@gmail.com>
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2024-08-26 20:53:06 +02:00
guillermo-bondonno 9cdbcb4f63 add test_closespider_pagecount_no_item_with_pagecount 2024-07-11 14:02:24 -03:00
guillermo-bondonno 5f0fad16f5 improve test_closespider_pagecount_no_item and corresponding test spider 2024-07-11 13:26:22 -03:00
guillermo-bondonno a40d5281cf improve test_closespider_pagecount_no_item and MaxItemsSpider 2024-07-11 11:14:30 -03:00
guillermo-bondonno d6352f9f66 refactor changes on closespider.py and improve test 2024-07-10 11:03:01 -03:00
guillermo-bondonno a44818afea restore mitmproxy-dhparam 2024-07-09 16:07:55 -03:00
guillermo-bondonno 0b8604bb5d add CLOSESPIDER_PAGECOUNT_NO_ITEM to CloseSpider extension 2024-07-09 15:52:49 -03:00
Andrey Rakhmatullin ceedb026f8
Remove top-level imports that install the reactor from scrapy.extensions.telnet. (#6432) 2024-07-09 08:34:58 +02:00
Andrey Rakhmatullin d08f559600
Add flake8-type-checking. (#6413) 2024-06-25 10:20:59 +02:00
Andrey Rakhmatullin 3d8dbd5648 flake8-bugbear 2024-06-20 00:27:38 +05:00
Andrey Rakhmatullin 1ef9c337ca flake8-docstrings 2024-06-19 23:57:40 +05:00
Andrey Rakhmatullin 1c70d3e605 flake8-comprehensions 2024-06-19 23:36:36 +05:00
Andrey Rakhmatullin 1282ddf8f7
Add parameters to most Deferred in scrapy/core. (#6395) 2024-06-10 10:27:50 +02:00
Andrey Rakhmatullin ddc98fe91b
Deprecate scrapy.utils.defer.process_chain_both(). (#6397) 2024-06-10 10:16:26 +02:00
Andrey Rakhmatullin 5850b8f3e6 Merge branch 'master' into pyupgrade 2024-06-07 11:20:31 +05:00
Andrey Rakhmatullin feb0b8f7dc Add pyupgrade. 2024-06-05 22:57:18 +05:00
Andrey Rakhmatullin 2e214210f6 Add parameters to iterable generics, replace generators with iterables. 2024-06-05 21:44:04 +05:00
Suvan Banerjee 3f76853bd2
Handle AttributeError: 'NoneType' in contract parsing (#6388) 2024-06-05 06:34:46 +02:00
Andrey Rakhmatullin 492c3bce9d
Don't run callbacks of requests from get_media_requests(). (#6386) 2024-06-03 13:28:20 +02:00
Fabian Schneebauer 0d58af8697 Add support for multiple referer policy tokens. 2024-05-29 10:59:32 +02:00
Sanchay Kumar cadb0dd707
Fix overridable methods in MediaPipeline (#6368) 2024-05-28 10:42:58 +02:00
Laerte Pereira 2facdd4fb0 Add change reactor test to CrawlerRunner 2024-05-26 19:55:54 -03:00
Andrey Rakhmatullin 631fc65fad
Update expectations of cookies after redirects. (#6367) 2024-05-16 16:42:09 +02:00
Adrián Chaves 812fd2368f
Allow user-defined secure cookies (#6357) 2024-05-15 11:48:43 +02:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00
Bagaudin Magomedov b88f22c6c5
Add SpiderLoggerAdapter, change Spider.logger to return SpiderLoggerAdapter (#6324) 2024-05-13 14:33:24 +02:00
Sanchay Kumar c9ef520936
Add Downloader.get_slot_key() without a spider parameter (#6352) 2024-05-10 14:26:45 +02:00
aisha-partha 180bc9bad7 Closes #6342. Setting METAREFRESH_IGNORE_TAGS to [‘noscript’] by default 2024-05-08 22:06:46 +05:30
Andrey Rakhmatullin a5da77d01d
Merge pull request #6221 from jxlil/fix/LxmlLinkExtractor 
fix: LxmlLinkExtractor unique_list missing key
 2024-04-19 19:57:11 +05:00
kokobhara b1fe97dc6c
Fix test expectations (#6316) 2024-04-17 13:14:57 +02:00
Víctor Ruiz 48c5a8c98f
Fix WrappedRequest.get_header raising TypeError if default is None (#6310) 2024-04-08 11:47:53 +02:00
Andrey Rakhmatullin 5d31e89262
Merge pull request #6298 from lorenzoverardo/bug-6292/robots.txt-non-UTF-8 
Handle robots.txt files not UTF-8 encoded
 2024-04-05 13:49:10 +05:00
Lorenzo Verardo 7b37dcd80d Handle robots.txt files not UTF-8 encoded 2024-04-04 12:22:50 +02:00