RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/tests
History
Paul Tremberth db40852892 Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322) 
* Do not interpret non-ASCII bytes in "Location" and percent-encode them

Fixes GH-2321

The idea is to not guess the encoding of "Location" header value
and simply percent-encode non-ASCII bytes,
which should then be re-interpreted correctly by the remote website
in whatever encoding was used originally.

See https://tools.ietf.org/html/rfc3987#section-3.2

This is similar to the changes to safe_url_string in
https://github.com/scrapy/w3lib/pull/45

* Remove unused import
 		2016-10-19 23:26:12 -03:00
..
keys Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
mocks move tests under root dir 2014-07-30 19:45:08 -03:00
sample_data Modifying existing gzip read failure recovery mechanism to patch read for broken archives 2016-08-11 18:21:42 +05:30
test_cmdline Amend "settings" command to output JSON for dict settings 2016-01-29 21:12:03 +01:00
test_settings Update tests for settings copy_to_dict() 2016-01-29 18:59:12 +01:00
test_spiderloader TST improve CrawlerRunner tests 2015-10-30 20:24:37 +05:00
test_utils_misc move tests under root dir 2014-07-30 19:45:08 -03:00
__init__.py Forward coverage envvars needed to cover subprocess 2015-08-13 21:07:45 -03:00
mockserver.py Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
py3-ignores.txt py3: reviewed s3 downloader handlers 2016-02-15 19:59:48 +03:00
requirements-py3.txt TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-06 18:29:49 +05:00
requirements.txt TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-06 18:29:49 +05:00
spiders.py py3 compat: use range, fixes CrawlTestCase.test_start_requests_bug_yielding 2016-01-21 17:52:12 +03:00
test_closespider.py Use LogCapture in testfixtures package for tests 2015-04-22 17:24:40 -03:00
test_command_fetch.py py3: port fetch and shell commands, and review + enable already passing test_closespider.py and tests/test_utils_template.py 2016-01-20 23:01:29 +03:00
test_command_parse.py Moved parse command tests to its own file. Added some checks to check for logged errors. 2016-09-19 05:33:05 +02:00
test_command_shell.py Add shell test for using scrapy.Request() directly without importing scrapy 2016-09-15 19:25:20 +02:00
test_command_version.py Increase coverage of version command 2015-08-13 19:30:06 -03:00
test_commands.py Moved parse command tests to its own file. Added some checks to check for logged errors. 2016-09-19 05:33:05 +02:00
test_contracts.py Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
test_crawl.py py3: pass CrawlTestCase.test_referer_header, fixing Echo resource in mockserver and json decoding in test 2016-01-21 17:52:12 +03:00
test_crawler.py make "settings" argument optional for Crawler, CrawlerRunner and CrawlerProcess 2015-06-06 03:23:13 +05:00
test_dependencies.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_downloader_handlers.py Merge pull request #2094 from redapple/dns-invalid-id 2016-07-13 10:48:08 +00:00
test_downloadermiddleware.py py3: body to bytes in tests, unskip test file 2016-01-15 11:26:01 +03:00
test_downloadermiddleware_ajaxcrawlable.py PY3 fix AjacCrawlable middleware tests 2015-09-01 03:49:52 +05:00
test_downloadermiddleware_cookies.py Do not propagate cookie log messages in tests so TopLevelFormatter does not rewrite them 2015-08-03 17:18:08 -03:00
test_downloadermiddleware_decompression.py PY3 fix test downloadermiddleware decompression 2015-07-29 13:51:01 +00:00
test_downloadermiddleware_defaultheaders.py PY3 fix DefaultHeadersMiddleware tests 2015-09-01 03:57:05 +05:00
test_downloadermiddleware_downloadtimeout.py Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
test_downloadermiddleware_httpauth.py PY3 fix HttpAuthMiddleware tests 2015-09-01 04:00:26 +05:00
test_downloadermiddleware_httpcache.py fix error reporting in test: we can fail in process_request too, so result should always be defined 2016-01-15 15:16:12 +03:00
test_downloadermiddleware_httpcompression.py [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-06-20 16:37:00 +02:00
test_downloadermiddleware_httpproxy.py adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
test_downloadermiddleware_redirect.py Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322) 2016-10-19 23:26:12 -03:00
test_downloadermiddleware_retry.py py3 fix HttpProxy and Retry Middlewares 2016-01-21 09:35:04 -05:00
test_downloadermiddleware_robotstxt.py Add test for already failed deferreds when downloading page in robots.txt middleware. 2016-01-27 12:56:42 -02:00
test_downloadermiddleware_stats.py PY3 fix DownloaderStats middleware tests 2015-09-01 04:09:15 +05:00
test_downloadermiddleware_useragent.py PY3 fix UserAgentMiddleware tests 2015-09-01 04:11:33 +05:00
test_dupefilters.py Merge pull request #1388 from scrapy/dupefilter-persist 2015-08-11 14:16:34 -03:00
test_engine.py response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
test_exporters.py Fixed bug on XMLItemExporter with non-string fields in items 2016-01-30 10:00:06 -02:00
test_feedexport.py Feed exporter: start exporting only on first item 2016-09-17 01:36:56 +02:00
test_http_cookies.py PY3: Implement some attributes of WrappedRequest required in Python 3 2016-02-07 14:19:27 +09:00
test_http_headers.py normalize header values to bytes 2014-08-01 00:02:50 -03:00
test_http_request.py Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-12 00:33:25 +02:00
test_http_response.py response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
test_item.py item fields defined on fields class attribute fix 2015-05-13 15:51:23 -03:00
test_link.py PY3 raise an exception if bytes are passed as url to Link constructor 2015-08-28 23:04:02 +05:00
test_linkextractors.py TST split LinkExtractorTestCase.test_extraction into several methods; remove duplicated test 2015-08-31 00:49:38 +05:00
test_linkextractors_deprecated.py Add link extractor test for non-ASCII characters in query part of URL 2016-04-09 15:15:01 +02:00
test_loader.py split nested_loader into seperate methods 2015-09-04 13:15:48 -07:00
test_logformatter.py PY3 port test_logformatter 2015-08-26 02:34:21 +05:00
test_mail.py MailSender.send: allow passing a charset. 2016-02-04 19:33:44 +01:00
test_middleware.py PY3 fix test middleware 2015-07-29 13:51:01 +00:00
test_pipeline_files.py [image & file pipeline] loading setting for user classes 2016-09-15 09:39:16 +02:00
test_pipeline_images.py [image & file pipeline] loading setting for user classes 2016-09-15 09:39:16 +02:00
test_pipeline_media.py PY3 fix test pipeline media 2015-07-29 13:51:01 +00:00
test_proxy_connect.py Use LogCapture in testfixtures package for tests 2015-04-22 17:24:40 -03:00
test_pydispatch_deprecated.py add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
test_responsetypes.py Merge pull request #1503 from demelziraptor/amazon-json-response 2016-09-12 13:21:16 -03:00
test_selector.py Selector should not receive both response and text 2016-07-30 10:35:16 +03:00
test_selector_csstranslator.py cleanup tests for selectors and translators 2015-08-11 14:09:34 -03:00
test_spider.py Fix SitemapSpider to extract sitemap urls from robots.txt properly 2016-02-06 23:54:07 +09:00
test_spidermiddleware_depth.py Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
test_spidermiddleware_httperror.py Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
test_spidermiddleware_offsite.py tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
test_spidermiddleware_referer.py PY3 depth, offsite and referer spider middlewares; Crawler 2015-08-28 04:11:30 +05:00
test_spidermiddleware_urllength.py Move scrapy/spider.py to scrapy/spiders/__init__.py 2015-05-09 04:20:09 -03:00
test_spiderstate.py Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
test_squeues.py fix scrapy squeue tests after recent changes to queuelib 2015-09-09 15:55:55 -03:00
test_stats.py enable test_stats and test_utils_log tests in Python 3 2015-08-26 01:58:59 +05:00
test_toplevel.py drop deprecated "optional_features" set 2015-09-15 09:00:08 +00:00
test_urlparse_monkeypatches.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_utils_conf.py Add backwards compatibility for build_component_list 2015-11-11 17:39:56 +01:00
test_utils_console.py Add support for bpython console. 2015-08-21 01:12:58 +01:00
test_utils_datatypes.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_utils_defer.py Improve python3 support for test_utils_defer. 2015-07-25 12:58:39 +02:00
test_utils_deprecate.py Ignore ScrapyDeprecationWarning warnings properly. 2015-06-09 18:22:03 +03:00
test_utils_gz.py Modifying existing gzip read failure recovery mechanism to patch read for broken archives 2016-08-11 18:21:42 +05:30
test_utils_http.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_utils_httpobj.py move tests under root dir 2014-07-30 19:45:08 -03:00
test_utils_iterators.py Refactored test code 2016-01-13 12:32:28 +01:00
test_utils_log.py Replace FailureFormatter with direct exc_info conversions in log calls 2015-05-15 05:46:14 -03:00
test_utils_project.py test abs path outside project as well 2016-09-30 15:07:58 -03:00
test_utils_python.py Rename isbinarytext function to binary_is_text for clarity 2016-03-30 15:44:15 +00:00
test_utils_reqser.py PY3 response bodies as bytes 2015-08-08 04:23:32 +05:00
test_utils_request.py Do not fail representing non-http requests 2015-08-10 14:10:58 -03:00
test_utils_response.py response_status_message should not fail on non-standard HTTP codes 2016-03-12 14:16:40 +01:00
test_utils_serialize.py Add set serialization to ScrapyJSONEncoder 2016-07-22 17:20:03 -04:00
test_utils_signal.py PY3 port signals 2015-07-25 18:04:55 +02:00
test_utils_sitemap.py PY3 fix test_utils_sitemap. 2014-08-01 16:17:40 +06:00
test_utils_spider.py Move scrapy/contrib/spiders to scrapy/spiders 2015-04-29 21:27:19 -03:00
test_utils_template.py Test utils.template.render_templatefile() 2015-08-26 20:44:33 +03:00
test_utils_trackref.py PY3: port scrapy.utils.trackref 2015-08-10 15:13:29 -03:00
test_utils_url.py Use w3lib.url.canonicalize_url() from w3lib 1.15.0 2016-08-16 17:42:16 +05:30
test_webclient.py revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3 2016-01-18 15:23:01 +03:00