RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,575 Commits 49 Branches 132 Tags 134 MiB
Commit Graph

456 Commits

Author SHA1 Message Date
Paul Tremberth a19f3475be Add idempotence tests for canonicalize_url 2016-04-27 13:55:07 +02:00
Paul Tremberth 7f94b57b3f Fix canonicalize_url() on Python 3 and re-enable tests 2016-04-27 13:54:36 +02:00
Paul Tremberth 08232a3f82 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-20 17:36:58 +02:00
Paul Tremberth 33a7caac90 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 17:25:25 +02:00
djunzu bb68cc120b Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-04-11 18:06:03 +02:00
djunzu 6627e05e92 Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-04-11 18:06:02 +02:00
Paul Tremberth a1a799249b Add link extractor test for non-ASCII characters in query part of URL 2016-04-11 16:55:56 +02:00
Paul Tremberth 6672883df8 Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-11 16:55:42 +02:00
Aron Bordin b2e914943d added FEED_TEMPDIR to settings 2016-04-04 13:59:04 +02:00
nyov 5ae8863d3d Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-31 11:56:00 +02:00
pawelmhm b181c10fc4 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-31 11:52:49 +02:00
Paul Tremberth a0c5c61855 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-03-01 15:00:44 +01:00
Paul Tremberth 6d4c51b266 Merge pull request #1809 from scrapy/backport-1.1-1787 
[backport][1.1] Better tracebacks
 2016-02-24 17:54:56 +01:00
Paul Tremberth e25052e4ef Add "Content-Length: 0" for body-less HTTP/1.1 POST requests 
GH-823 was fixed only for HTTP/1.0 (in GH-1089)
 2016-02-24 17:21:08 +01:00
Paul Tremberth 50f7b416c5 Minor change on quotes 
Trying to force Travis CI to build
 2016-02-24 17:14:48 +01:00
Mikhail Korobov 0d596c27a9 Preserve tracebacks better. Fixes GH-1760. 2016-02-24 17:14:48 +01:00
Mikhail Korobov 4459ee5d53 TST clean up RunSpiderCommandTest 2016-02-24 17:14:48 +01:00
Paul Tremberth a6bc758db1 Merge pull request #1792 from scrapy/backport-1.1-pr1761 
[backport][1.1.x] Py3 S3 botocore
 2016-02-19 11:17:20 +01:00
Paul Tremberth 3331cea788 More liberal Content-Disposition header parsing 
Fixes #1782
 2016-02-18 22:56:17 +01:00
Konstantin Lopuhin 25e481fd14 [backport][1.1.x] Py3 S3 botocore 
Originally from https://github.com/scrapy/scrapy/pull/1761
 2016-02-18 22:48:37 +01:00
Elias Dorneles f928014d91 add deprecation for pydispatch (thanks for the help @redapple) 2016-02-15 11:40:08 +01:00
Nicolas Pennequin 72229f92de MailSender.send: allow passing a charset. 
Resolves Issue #348
 2016-02-09 19:52:06 +01:00
orangain 281e9bd046 PY3: Implement some attributes of WrappedRequest required in Python 3 
This will fix #1770.
 2016-02-09 16:10:58 +01:00
orangain 522c71bb8a Fix SitemapSpider to extract sitemap urls from robots.txt properly 
This will fix #1766.
 2016-02-09 16:10:58 +01:00
Elias Dorneles a8a6f050e7 Merge pull request #1735 from ArturGaspar/master 
[MRG+1] Fix for KeyError in robots.txt middleware
 2016-02-03 13:30:06 -02:00
Mikhail Korobov 43a53aca12 Merge pull request #1746 from redapple/shell-settings-logging 
[MRG+1] Remove __str__ and __repr__ from settings, introduce copy_to_dict()
 2016-02-03 19:59:24 +05:00
stummjr bb2cf7c0d7 Fixed bug on XMLItemExporter with non-string fields in items 2016-01-30 10:00:06 -02:00
Paul Tremberth d843a0aae8 Amend "settings" command to output JSON for dict settings 2016-01-29 21:12:03 +01:00
Paul Tremberth aa78758bc7 Update tests for settings copy_to_dict() 2016-01-29 18:59:12 +01:00
Mikhail Korobov a35aec71e9 Merge pull request #1710 from redapple/1550-shell_file-cont 
[MRG+1] shell command's ability to open local files + tests
 2016-01-29 21:03:40 +05:00
Paul Tremberth 78f00401cd Remove unused import in tests 2016-01-29 16:56:05 +01:00
stummjr cf2ebb0687 Include tests for exporters: JSON, JSON-Lines, Pickle and Marshal. 2016-01-28 19:43:53 -02:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 481e251775 Move guess_scheme() tests to relevant test module 2016-01-28 13:51:50 +01:00
Paul Tremberth cae268402d Move guess_scheme() to scrapy.utils.url 2016-01-28 13:42:04 +01:00
stummjr c55ff110a3 Fix CSV exporter for non string Python types. 2016-01-27 15:43:17 -02:00
stummjr f1d971a5c0 fix PythonItemExporter for non-string types 2016-01-27 14:34:46 -02:00
Artur Gaspar dc8701ea42 Add test for already failed deferreds when downloading page in robots.txt middleware. 2016-01-27 12:56:42 -02:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Daniel Graña a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles 2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles 23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00
Elias Dorneles 0c44fac2b5 added tests for feed export marshal and pickle 2016-01-24 19:17:42 -02:00
Konstantin Lopuhin 1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Konstantin Lopuhin 4233b3cda4 py3: reviewed passing test_spidermiddleware_httperror.py 2016-01-24 23:10:03 +03:00
Konstantin Lopuhin 097082cffa reviewed py3 compat in pipelines/images.py and pipelines/files.py 2016-01-24 23:05:23 +03:00