RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,894 Commits 49 Branches 132 Tags 136 MiB
Commit Graph

162 Commits

Author SHA1 Message Date
Paul Tremberth 5bf10cc8ff Fix tests on URL path encoding for links from latin1 document 
UTF-8 is to be used for path component.
Page encoding only affects query part of URLs.
 2016-04-07 22:43:47 +02:00
pawelmhm 8efd6d26ad [Backport][1.0] response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-04-06 20:21:26 +02:00
Elias Dorneles 9396e0ee3f Merge pull request #1669 from redapple/bogus-links 
[MRG+1] [1.0.x backport] [LinkExtractors] Ignore bogus links
 2016-01-19 13:48:00 -02:00
Andrew Scorpil 108195e780 [LinkExtractors] Ignore bogus links 
(rebased the code for scrapy 1.0 and made a few code improvements --nyov)
 2016-01-11 15:42:56 +01:00
Νικόλαος-Διγενής Καραγιάννης 4418fc3b86 test xml nodename with dots 2015-12-30 15:23:10 -03:00
Mikhail Korobov 97f2fb302d TST a test to show that dupefilter persistence is not working 2015-12-30 14:47:01 -03:00
Mikhail Korobov a6535c2ae5 pin pytest to 2.7.3 2015-09-29 03:36:16 +05:00
Daniel Graña 3365c01514 fix scrapy squeue tests after recent changes to queuelib 2015-09-09 16:02:37 -03:00
Julia Medina 04493d5799 Fix a recent mock regression by not using it in a test 2015-08-06 18:28:18 -03:00
Daniel Graña b04dd7d1bf Twisted 15.3.0 does not raises expected exception serializing lambda functions 2015-08-06 17:57:17 -03:00
Daniel Graña cc00ad20cf Unquote request path before passing to FTPClient, it already escape paths 2015-07-01 01:39:29 -03:00
Mikhail Korobov 7a30e6acbe make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 
(cherry picked from commit d047665c02)
 2015-06-12 16:05:36 -03:00
Mikhail Korobov bd271d8b99 TST a test case to show custom_settings doesn't always work 2015-06-03 22:19:05 +05:00
Julia Medina 2d6450a4ad Add helper to update deprecated class paths 2015-06-01 20:39:29 -03:00
Mikhail Korobov 19ac046343 fixed CrawlerProcess when settings are passed as dicts 
See https://github.com/scrapy/scrapy/pull/1156

(cherry picked from commit aa6a72707d)
 2015-06-01 12:50:41 -03:00
Mikhail Korobov cc2258b2bb Merge pull request #1145 from bosnj/master 
[MRG+1] default return value for extract_first
 2015-05-21 22:03:54 +05:00
Daniel Graña ee59112480 Merge pull request #1224 from scrapy/fix-empty-feed-export-fields 
[MRG] fixed FEED_EXPORT_FIELDS handling (see #1223)
 2015-05-19 16:36:05 -03:00
Mikhail Korobov 7a5b5ec4d6 TST remove unnecessary lines from py3-ignores 
scrapy/contrib is already skipped - see https://github.com/scrapy/scrapy/pull/1165
 2015-05-19 00:57:39 +05:00
Mikhail Korobov 9b0ca1b7a0 drop support for FEED_EXPORT_FIELD=[] meaning "no fields" 2015-05-18 17:13:25 +05:00
Mikhail Korobov 9fb318338b support FEED_EXPORT_FIELDS=[] 2015-05-18 16:44:02 +05:00
Mikhail Korobov e1efd19175 TST, DOC document that Scrapy only infers field names for CSV 2015-05-18 16:43:23 +05:00
Mikhail Korobov 8d9e3b7e8d TST add a test for JSON lines item exporter 2015-05-18 16:43:23 +05:00
Julia Medina 6dccb3a9b3 Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
Julia Medina 157cec7f64 Replace logger.exception calls for Python compatibility 2015-05-15 01:26:45 -03:00
Daniel Graña 5e5e44e91e Merge pull request #1228 from nramirezuy/item_fields_defined_on_fields 
item fields defined on fields class attribute fix
 2015-05-14 23:09:24 -03:00
nramirezuy 773ea5a5ef item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
Julia Medina d3f576a816 Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
bosnj 8ae05478be added docs and test case, fixed handling empty string vs None 2015-05-04 21:22:17 +02:00
Victoria Terenina (torymur) 667864ae94 testing robotstxt error 2015-04-30 16:28:57 +03:00
Julia Medina b827097c91 Rename scrapy/statscol.py to scrapy/statscollectors.py 2015-04-29 21:36:40 -03:00
Julia Medina 54a4ce069c Rename scrapy/dupefilter.py to scrapy/dupefilters.py 2015-04-29 21:27:19 -03:00
Julia Medina f5bdf64f9c Rename scrapy/squeue.py to scrapy/squeues.py 2015-04-29 21:27:19 -03:00
Julia Medina 645cdcbf9e Rename scrapy/loader/processor.py to scrapy/loader/processors.py 2015-04-29 21:27:19 -03:00
Julia Medina 9a3e3ba505 Move scrapy/contrib remaining top-level files to scrapy/extensions 2015-04-29 21:27:19 -03:00
Julia Medina fc346cba4d Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
Julia Medina 180272c092 Move scrapy/contrib/spidermiddleware to scrapy/spidermiddlewares 2015-04-29 21:26:35 -03:00
Julia Medina 8021df18d4 Move scrapy/contrib/pipeline to scrapy/pipelines 2015-04-29 21:26:35 -03:00
Julia Medina b47228ada8 Move scrapy/contrib/loader to scrapy/loader 2015-04-29 21:24:30 -03:00
Julia Medina cf064b1437 Move scrapy/contrib/linkextractors to scrapy/linkextractors 2015-04-29 21:24:30 -03:00
Julia Medina 7804b3d778 Move scrapy/contrib/exporter to scrapy/exporters 2015-04-29 21:24:30 -03:00
Julia Medina d7c444fefb Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares 2015-04-29 21:24:30 -03:00
Mikhail Korobov fbb1078f58 Merge pull request #1060 from Curita/python-logging 
[MRG+1] Python logging
 2015-04-29 23:20:34 +05:00
Julia Medina cc4c31e426 Relax SpiderLoader interface check 2015-04-23 15:08:04 -03:00
Julia Medina 6d1205063c Add a filter to replace '__name__' loggers with 'scrapy' 2015-04-22 17:24:41 -03:00
Julia Medina c174d78f12 Deprecate scrapy/log.py 2015-04-22 17:24:40 -03:00
Julia Medina 6acb3848fb Stdout redirect in configure_logging 2015-04-22 17:24:40 -03:00
Julia Medina 1c8708eb82 Create a logger for every Spider and adapt Spider.log to log through it 2015-04-22 17:24:40 -03:00
Julia Medina ac40ef611a Custom handler to count log level occurrences in a crawler 2015-04-22 17:24:40 -03:00
Julia Medina b75556ef79 Add a logging filter to mimic Twisted's log.err formating for Failures 2015-04-22 17:24:40 -03:00
Julia Medina 6f9b423215 Restructure LogFormatter to comply with std logging calls 2015-04-22 17:24:40 -03:00