RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,554 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

5554 Commits

Author SHA1 Message Date
Patrick Connolly 9c29215bef Place brackets on own lines with JsonItemExporter. 
Placing the opening and closing brackets on their own lines makes it slightly easier to sort lines after the `spider_closed` signal is fired.
 2016-04-25 15:28:41 +02:00
Paul Tremberth 19b585361e Allow more pre-releases with bumpversion 2016-04-21 16:53:17 +02:00
Paul Tremberth 11b22a35dd Merge pull request #1945 from redapple/backport-1.1-pr1923 
[backport][1.1] Use newer w3lib.url.safe_url_string() (PR #1923)
 2016-04-20 17:55:55 +02:00
Paul Tremberth 04977aa8d5 Bump up w3lib requirement to v1.14.2 2016-04-20 17:37:16 +02:00
Paul Tremberth 08232a3f82 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-20 17:36:58 +02:00
Paul Tremberth d61a71b723 Merge pull request #1944 from redapple/backport-1.1-pr1933 
[backport][1.1] Ignore HTTPS certificate verification failures (PR #1933)
 2016-04-20 17:35:21 +02:00
Paul Tremberth 33a7caac90 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 17:25:25 +02:00
Paul Tremberth ebbec3cd39 Catch VerificationError but keep the rest of ClientTLSOptions 2016-04-20 17:25:07 +02:00
Paul Tremberth 2f54f9680d Ignore HTTPS certificate verification failures 
Fixes #1930
 2016-04-20 17:24:49 +02:00
Paul Tremberth 6e4742755e Merge pull request #1943 from redapple/backport-1.1-pr1912 
[backport][1.1] Fix HTTP Pool key for HTTPS proxy tunneled connections (PR #1912)
 2016-04-20 17:19:39 +02:00
Paul Tremberth 7d3a529dbc Fix HTTP Pool key for HTTPS proxy tunneled connections (CONNECT method) 2016-04-20 17:02:47 +02:00
Paul Tremberth f70d538d10 Merge pull request #1942 from scrapy/backport-1.1-pr1938 
[backport][1.1] Set SNI properly when using CONNECT (PR #1938)
 2016-04-20 17:01:34 +02:00
Paul Tremberth 352d0d36ee Fall back to no-SNi context factory is Twisted<14 is used 2016-04-20 16:48:32 +02:00
Paul Tremberth 5cb071b466 Set SNI properly when using CONNECT 2016-04-20 16:48:20 +02:00
Paul Tremberth bd6b3c0608 Reference StackOverflow's "minimal, complete, and verifiable example" guide 2016-04-12 16:13:05 +02:00
Konstantin Lopuhin db61d54663 Fix FAQ entry about python versions support 2016-04-12 16:12:46 +02:00
Paul Tremberth f1020effc5 Merge pull request #1920 from scrapy/backport-1.1-pr1891 
[backport][1.1] Change Files/ImagesPipelines class attributes to instance attributes (PR #1891)
 2016-04-12 16:07:07 +02:00
Paul Tremberth 9f1d215b4a Merge pull request #1919 from scrapy/backport-1.1-pr1881 
[backport][1.1] Remove duplicate code now handled by newer w3lib (PR #1881)
 2016-04-12 16:06:35 +02:00
djunzu fd98e3c031 Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-11 18:06:03 +02:00
djunzu 6d56faf11f Change ImagesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/images.py
 2016-04-11 18:06:03 +02:00
djunzu bb68cc120b Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-04-11 18:06:03 +02:00
djunzu 6d8184953a Move default settings to settings/default_settings.py. 
modified:   scrapy/pipelines/files.py
	modified:   scrapy/pipelines/images.py
	modified:   scrapy/settings/default_settings.py
 2016-04-11 18:06:02 +02:00
djunzu 6627e05e92 Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-04-11 18:06:02 +02:00
nyov 1a62d87e35 Remove duplicate code now handled by newer w3lib 
see f3029a6a10
 2016-04-11 17:39:17 +02:00
nblock 00792557ec Fix spelling mistake 2016-04-11 17:36:31 +02:00
Paul Tremberth d30dfada11 Merge pull request #1921 from scrapy/backport-1.1-pr1913 
[backport][1.1] Fix link extractor tests for non-ASCII characters from latin1 document (PR #1913)
 2016-04-11 17:34:47 +02:00
Paul Tremberth a1a799249b Add link extractor test for non-ASCII characters in query part of URL 2016-04-11 16:55:56 +02:00
Paul Tremberth 6672883df8 Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-11 16:55:42 +02:00
Pengyu CHEN 160bcbacfc Added: Making it case-insensitive when extracting sitemap URLs from a robots.txt 2016-04-04 15:25:49 +02:00
Aron Bordin b2e914943d added FEED_TEMPDIR to settings 2016-04-04 13:59:04 +02:00
Mikhail Korobov 288f8c09f8 DOC clarify Architecture docs 2016-03-31 12:33:29 +02:00
Mikhail Korobov 1c70ece728 DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-31 12:33:29 +02:00
Paul Tremberth 2ea9bc81f6 Merge pull request #1898 from scrapy/backport-1.1-pr1883 
[Backport][1.1] Make FilesPipeline work with S3FilesStore using botocore
 2016-03-31 12:30:45 +02:00
Konstantin Lopuhin b01bf4728f Fix a race condition in the FilesPipeline 
Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
 2016-03-31 12:00:26 +02:00
Konstantin Lopuhin 84371ffb22 Fix handling of meta=None in S3FilesStore.persist_file 2016-03-31 12:00:26 +02:00
nyov 5ae8863d3d Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-31 11:56:00 +02:00
pawelmhm b181c10fc4 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-31 11:52:49 +02:00
Aron Bordin 7b1dddceb3 small doc style fixes 2016-03-30 18:36:34 +02:00
nyov 718d386ebd Update documentation links 2016-03-30 18:36:34 +02:00
djunzu 3d6abacc55 Add pps extension to IGNORED_EXTENSIONS 
modified:   scrapy/linkextractors/__init__.py
 2016-03-30 18:33:58 +02:00
Paul Tremberth d381f8d930 Merge pull request #1892 from scrapy/backport-1.1-pr1822 
[Backport][1.1] Allow core Scheduler priority queue customization
 2016-03-30 18:17:22 +02:00
nyov 1adc47ce93 Allow core Scheduler priority queue customization 2016-03-30 17:51:37 +02:00
Paul Tremberth 844f040bc4 Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 17:49:14 +02:00
nanolab e7a5e3717b Update httpcache.py 
It checks cache directory modification time, but have to check file modification time.
 2016-03-30 10:53:54 +02:00
Lele 15ffb076ca Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-30 10:53:54 +02:00
Paul Tremberth e025f374e8 Bump version: 1.1.0rc2 → 1.1.0rc3 2016-03-01 23:40:30 +01:00
Paul Tremberth 9a4ead8e5d Merge pull request #1831 from redapple/backport-1.1-pr1804 
[Backport][1.1] Re-enable HTTPS tests for download timeouts
 2016-03-01 15:22:31 +01:00
Paul Tremberth 4547a7e759 Explicitly call Twisted transport stopProducing() on HTTP/1.0 timeouts 2016-03-01 15:00:58 +01:00
Paul Tremberth a0c5c61855 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-03-01 15:00:44 +01:00
Mikhail Korobov 9a992cdfd7 declare Python 3 support in setup.py 2016-03-01 14:38:49 +01:00