RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,905 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

833 Commits

Author SHA1 Message Date
Adrián Chaves 70a0f1f2e4
Merge pull request #3610 from Gallaecio/spidercls-check 
[MRG+1] Check that spidercls arguments in scrapy.crawler classes are not spid…
 2019-03-15 08:25:16 +01:00
Adrián Chaves 1f7413dc88
Merge pull request #3635 from matthieucham/feature-filteringlinkextractor-restrict-text 
[MRG+1] Feature filteringlinkextractor restrict text
 2019-03-15 07:58:28 +01:00
Mikhail Korobov 5dc94db847
Merge branch 'master' into backward 2019-03-14 22:21:09 +05:00
Adrián Chaves 75d6f56c8a Switch from ` to `` where inline code formatting is desired 2019-03-01 16:56:58 +01:00
Adrián Chaves 858f5be747 backwards → backward (adj.) 2019-03-01 16:10:23 +01:00
Matthieu Grandrie e3b15252c8 New constructor arg *restrict_text* for FilteringLinkExtractor. 
Same as allow and deny args, it holds a string, a regex or an iterable of. Links whose text don't match one of the regex are filtered out.
DOC restrict_text in LxmlLinkExtractor
 2019-02-28 17:21:17 +01:00
Adrián Chaves 03e61b9908 Check that spidercls arguments in scrapy.crawler classes are not spider objects 2019-02-12 14:46:27 +01:00
Daniel Graña 65d631329a Be consistent with domain used for links to documentation website 2019-01-31 01:28:53 -03:00
Júlio César Batista 8eade7d864 Testing stats and log messages from RFPDupeFilter 2019-01-18 11:39:35 -02:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Daniel Graña 93cf38354d
Merge pull request #3394 from hcoura/dh-lazyloading-optional 
Make lazy loading Download Handlers optional
 2018-12-26 11:16:19 -03:00
Daniel Graña f6ce716372
Merge pull request #3476 from elacuesta/deque_appendleft 
[MRG+1] Use collections.deque instead of list to store MiddlewareManager's methods
 2018-12-26 11:13:52 -03:00
Mikhail Korobov dbfabf02e8 Merge branch 'master' into telnet-auth 2018-12-26 01:04:34 +05:00
Eugenio Lacuesta f97e3e90f2 Use collections.deque instead of list to store methods 2018-10-29 12:40:20 -03:00
Henrique Coura 441e1e750f Style changes 2018-09-26 13:28:34 -03:00
Henrique Coura e57a629efc Generate only password, encode username/password only on login 2018-09-26 11:54:57 -03:00
Henrique Coura 37cfb49805 Randomly generate telnet credentials by default 2018-09-24 16:42:49 -03:00
Mikhail Korobov 2c3b2158c9 DOC address @stummjr's review comments 
* fixed several small issues
* re-written "Creating Selectors" section
* fixed remaining .extract usage in tests
 2018-09-18 05:02:17 +05:00
Mikhail Korobov 53da56c8dc TST update tests to use get/getall/attrib instead of extract 2018-09-12 17:57:27 +05:00
Daniel Graña 7223978594
Merge pull request #3381 from StasDeep/fix/issue-3380 
[MRG+1] Use dont_filter=True for contracts requests
 2018-09-06 11:41:02 -03:00
Daniel Graña 2aae514f99
Merge branch 'master' into fix/issue-3380 2018-09-05 12:25:07 -03:00
Daniel Graña d7b98a3714
Merge branch 'master' into feature/issue-3364 2018-09-05 11:26:59 -03:00
Daniel Graña ae8a0dc77c
Merge pull request #3383 from StasDeep/feature/issue-3382 
[MRG+1] Add ability to use FormRequest in contracts
 2018-09-05 11:21:26 -03:00
Daniel Graña 4da0b59cd7
Merge pull request #3393 from whalebot-helmsman/singal-request-added-to-downloader-slot 
[MRG+1] New signal for reqeuests reached downloader
 2018-09-05 11:17:13 -03:00
Stas Glubokiy 8dbbbd1395 Use request_cls attribute in contract definition 2018-09-03 20:07:37 +03:00
Henrique Coura 167211ffb0 Default is lazy, load_object exception handling, code improvements 2018-08-20 15:54:04 -03:00
Stas Glubokiy 9ab85fe788 Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3382 2018-08-19 17:21:28 +03:00
Stas Glubokiy 57824600a8 Use six.get_unbound_function in test_same_url 2018-08-19 16:56:41 +03:00
Stas Glubokiy 0467737cf0 Fix mockserver usage 2018-08-18 15:43:46 +03:00
Stas Glubokiy 1311f6b536 Merge branch 'master' of github.com:StasDeep/scrapy into fix/issue-3380 2018-08-18 15:42:10 +03:00
Stas Glubokiy 11576f5c8f Merge branch 'master' of github.com:StasDeep/scrapy into feature/issue-3364 2018-08-18 15:24:07 +03:00
Henrique Coura 5bac436764 Make lazy loading Download Handlers optional 2018-08-17 15:07:37 -03:00
Vostretsov Nikita afb1458bd3 tests for new signal 2018-08-17 14:39:54 +00:00
Mikhail Korobov c7654f7cb1
Merge pull request #3379 from StasDeep/fix/issue-3378 
Use inspect.getmembers in tested_methods_from_spidercls
 2018-08-17 19:25:41 +05:00
Stas Glubokiy ddd69f4c10 Use MockServer in test_same_url 2018-08-15 20:39:43 +03:00
Stas Glubokiy 2cb4decb6a Move TestSameUrlSpider to test method 2018-08-15 20:36:10 +03:00
Stas Glubokiy 4de493efdd Add test_same_url 2018-08-15 20:24:00 +03:00
Daniel Graña 38608bc249 Use ignore_errors option from rmtree 2018-08-15 11:59:09 -03:00
Daniel Graña 4eaf8690b1 Twisted's unittest.Testcase assertRaiess can't be used as context manager 2018-08-15 08:54:18 -03:00
Daniel Grana a304d6b692 Workaround to pass tests/test_feedexporter.py under windows 2018-08-15 02:23:07 -07:00
Daniel Grana e7fe243c3e Fix test_crawler under windows 2018-08-15 01:09:23 -07:00
Daniel Grana 96517cb7de Fix test_command_parse under windows 2018-08-15 01:08:40 -07:00
Daniel Graña d93d960319 Fix test_utils_project under Windows 2018-08-15 01:53:20 -03:00
Jakob de Maeyer cb28175750 Fix csviter tests by explicitly using newline only 2018-08-15 01:35:01 -03:00
Daniel Graña ca53a8699a Fix presentation of template directory in startproject command 2018-08-15 01:35:01 -03:00
Daniel Graña 0e532e3dd8 Creating a connection to 0.0.0.0 fails on windows but not on linux nor mac 2018-08-15 01:35:01 -03:00
Daniel Graña a21abac743 fix ftp tests on windows 2018-08-15 01:35:01 -03:00