RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,619 Commits 49 Branches 132 Tags 152 MiB
Commit Graph

5619 Commits

Author SHA1 Message Date
Panayiotis Lipiridis 3f8be374c3 Github >> GitHub 2016-05-09 16:15:18 +02:00
Mikhail Korobov 064f05349f Merge pull request #1962 from redapple/w3lib-dep 
Bump w3lib version dependency in setup.py
 2016-04-29 16:55:10 +06:00
Paul Tremberth c8bff23751 Bump w3lib version dependency in setup.py 2016-04-29 10:29:37 +02:00
Paul Tremberth cb38f3896b Add notes on botocore support and JsonItemExporter opening/closing brackets 2016-04-28 22:34:17 +02:00
Paul Tremberth dc9de7524f Update release notes with recent features and bug fixes. 2016-04-28 22:34:07 +02:00
Mikhail Korobov dbef7e2b45 Merge pull request #1947 from scrapy/canonicalize-url 
[MRG+1] Fix canonicalize_url() on Python 3 and re-enable tests
 2016-04-27 01:32:42 +06:00
Paul Tremberth 0e11b3e6f0 Add idempotence tests for canonicalize_url 2016-04-26 20:03:17 +02:00
Paul Tremberth efbe75ea5e Use six.PY2 also for conditional imports 2016-04-26 16:14:15 +02:00
Paul Tremberth 25401fd322 Use six.PY2 instead of six.PY3 for Python version variations 
Also don't test passed encoding against 'utf8';
Just consider that if encoding failed, it must have been another encoding.
 2016-04-26 15:12:00 +02:00
Paul Tremberth cf716ea200 Merge pull request #1950 from patcon/patch-1 
[MRG+1] Place brackets on own lines with JsonItemExporter
 2016-04-25 15:21:21 +02:00
Patrick Connolly 19a4a0adf9 Place brackets on own lines with JsonItemExporter. 
Placing the opening and closing brackets on their own lines makes it slightly easier to sort lines after the `spider_closed` signal is fired.
 2016-04-24 14:55:08 -04:00
Paul Tremberth 8efa98794d Allow more pre-releases with bumpversion 2016-04-21 16:51:17 +02:00
Paul Tremberth 68dedf54cb Fix canonicalize_url() on Python 3 and re-enable tests 2016-04-21 14:50:59 +02:00
Mikhail Korobov 73a5571c60 Merge pull request #1923 from scrapy/request-new-safe_url_string 
[MRG+1] Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests
 2016-04-20 19:33:45 +06:00
Paul Tremberth 417279cceb Bump up w3lib requirement to v1.14.2 2016-04-20 15:12:19 +02:00
Mikhail Korobov 679a680638 Merge pull request #1933 from scrapy/cert-verif-ignore 
Ignore HTTPS certificate verification failures
 2016-04-20 18:57:33 +06:00
Paul Tremberth cd979ace40 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
Mikhail Korobov 3735eb8e5f Merge pull request #1912 from redapple/https-proxy-pool-key 
[MRG+1] Fix HTTP Pool key for HTTPS proxy tunneled connections (CONNECT method)
 2016-04-20 17:32:04 +06:00
Mikhail Korobov a8b49472b5 Merge pull request #1938 from redapple/https-proxy-connect-sni 
Set SNI properly when using CONNECT
 2016-04-20 17:30:13 +06:00
Paul Tremberth dcea11a70c Fall back to no-SNi context factory is Twisted<14 is used 2016-04-19 10:41:13 +02:00
Paul Tremberth d6760dbaac Set SNI properly when using CONNECT 2016-04-18 18:30:01 +02:00
Paul Tremberth 25ee023561 Catch VerificationError but keep the rest of ClientTLSOptions 2016-04-14 17:19:55 +02:00
Paul Tremberth a087d2593a Ignore HTTPS certificate verification failures 
Fixes #1930
 2016-04-14 15:59:41 +02:00
Mikhail Korobov ba6dbad1e0 Merge pull request #1926 from redapple/faq-mcve 
Reference StackOverflow's "minimal, complete, and verifiable example" guide
 2016-04-12 18:35:34 +06:00
Paul Tremberth 2849ebf4c6 Reference StackOverflow's "minimal, complete, and verifiable example" guide 2016-04-12 14:07:33 +02:00
Paul Tremberth 47bfac1669 Merge pull request #1924 from lopuhin/faq-fix-py3 
[MRG+1] Fix FAQ entry about python versions support (add Python 3.3+)
 2016-04-12 12:15:17 +02:00
Konstantin Lopuhin 1ec49c2ada Fix FAQ entry about python versions support 2016-04-12 11:48:57 +03:00
Paul Tremberth d42a98d3b5 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-12 00:33:25 +02:00
Paul Tremberth 86e4442d17 Fix HTTP Pool key for HTTPS proxy tunneled connections (CONNECT method) 2016-04-11 18:28:38 +02:00
Paul Tremberth 10d03ee419 Merge pull request #1916 from nblock/patch-1 
Fix spelling mistake
 2016-04-11 15:12:24 +02:00
nblock a3557dd34d Fix spelling mistake 2016-04-11 14:06:57 +02:00
Mikhail Korobov ff80e1c381 Merge pull request #1913 from redapple/link-extractor-new-w3lib 
Fix link extractor tests for non-ASCII characters from latin1 document
 2016-04-09 22:05:59 +06:00
Paul Tremberth 7b5243a263 Add link extractor test for non-ASCII characters in query part of URL 2016-04-09 15:15:01 +02:00
Paul Tremberth 1656fbcffa Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-08 23:25:50 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
Paul Tremberth cbb695d08c Merge pull request #1881 from nyov/dedupe 
[MRG+1] Remove duplicate code now handled by newer w3lib
 2016-04-06 15:47:06 +02:00
Paul Tremberth 642fedb3d6 Merge pull request #1902 from starrify/case-insensitive-robots-txt-for-sitemap 
[MRG+1] Added: Making it case-insensitive when extracting sitemap URLs from a robots.txt
 2016-04-04 15:23:03 +02:00
djunzu 6988e9cd4b Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-01 21:51:15 -03:00
Pengyu CHEN 103f6eaa88 Added: Making it case-insensitive when extracting sitemap URLs from a robots.txt 2016-04-02 02:04:50 +08:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
djunzu 537083524e Change ImagesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/images.py
 2016-03-31 19:20:43 -03:00
djunzu 8228a0c491 Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-03-31 19:20:39 -03:00
djunzu c7fc17866f Move default settings to settings/default_settings.py. 
modified:   scrapy/pipelines/files.py
	modified:   scrapy/pipelines/images.py
	modified:   scrapy/settings/default_settings.py
 2016-03-31 19:20:33 -03:00
djunzu e9d48f8a8e Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-03-31 19:19:49 -03:00
Paul Tremberth 9d8c368ce8 Merge pull request #1879 from scrapy/scrapy-arch-docs 
DOC improved Architecture overview
 2016-03-31 12:09:24 +02:00
Paul Tremberth 9ae4e46f32 Merge pull request #1883 from lopuhin/botocore-files-store-fix 
[MRG+1] Make FilesPipeline work with S3FilesStore using botocore
 2016-03-31 11:57:39 +02:00
Paul Tremberth 3ba5671fbc Merge pull request #1851 from nyov/binary_or_text 
[MRG+1] Rename isbinarytext function to binary_is_text for clarity
 2016-03-31 11:55:09 +02:00
Paul Tremberth 3a763f7ba7 Merge pull request #1857 from pawelmhm/fix_response_status_msg 
[MRG+1] response_status_message should not fail on non-standard HTTP codes
 2016-03-31 11:44:44 +02:00
nyov e8ca467572 Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-30 15:44:15 +00:00