RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,542 Commits 49 Branches 132 Tags 137 MiB
Commit Graph

5542 Commits

Author SHA1 Message Date
Paul Tremberth 352d0d36ee Fall back to no-SNi context factory is Twisted<14 is used 2016-04-20 16:48:32 +02:00
Paul Tremberth 5cb071b466 Set SNI properly when using CONNECT 2016-04-20 16:48:20 +02:00
Paul Tremberth bd6b3c0608 Reference StackOverflow's "minimal, complete, and verifiable example" guide 2016-04-12 16:13:05 +02:00
Konstantin Lopuhin db61d54663 Fix FAQ entry about python versions support 2016-04-12 16:12:46 +02:00
Paul Tremberth f1020effc5 Merge pull request #1920 from scrapy/backport-1.1-pr1891 
[backport][1.1] Change Files/ImagesPipelines class attributes to instance attributes (PR #1891)
 2016-04-12 16:07:07 +02:00
Paul Tremberth 9f1d215b4a Merge pull request #1919 from scrapy/backport-1.1-pr1881 
[backport][1.1] Remove duplicate code now handled by newer w3lib (PR #1881)
 2016-04-12 16:06:35 +02:00
djunzu fd98e3c031 Update docs. 
modified:   docs/topics/media-pipeline.rst
 2016-04-11 18:06:03 +02:00
djunzu 6d56faf11f Change ImagesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/images.py
 2016-04-11 18:06:03 +02:00
djunzu bb68cc120b Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-04-11 18:06:03 +02:00
djunzu 6d8184953a Move default settings to settings/default_settings.py. 
modified:   scrapy/pipelines/files.py
	modified:   scrapy/pipelines/images.py
	modified:   scrapy/settings/default_settings.py
 2016-04-11 18:06:02 +02:00
djunzu 6627e05e92 Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-04-11 18:06:02 +02:00
nyov 1a62d87e35 Remove duplicate code now handled by newer w3lib 
see f3029a6a10
 2016-04-11 17:39:17 +02:00
nblock 00792557ec Fix spelling mistake 2016-04-11 17:36:31 +02:00
Paul Tremberth d30dfada11 Merge pull request #1921 from scrapy/backport-1.1-pr1913 
[backport][1.1] Fix link extractor tests for non-ASCII characters from latin1 document (PR #1913)
 2016-04-11 17:34:47 +02:00
Paul Tremberth a1a799249b Add link extractor test for non-ASCII characters in query part of URL 2016-04-11 16:55:56 +02:00
Paul Tremberth 6672883df8 Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-11 16:55:42 +02:00
Pengyu CHEN 160bcbacfc Added: Making it case-insensitive when extracting sitemap URLs from a robots.txt 2016-04-04 15:25:49 +02:00
Aron Bordin b2e914943d added FEED_TEMPDIR to settings 2016-04-04 13:59:04 +02:00
Mikhail Korobov 288f8c09f8 DOC clarify Architecture docs 2016-03-31 12:33:29 +02:00
Mikhail Korobov 1c70ece728 DOC improved Architecture overview 
* spiders don't have to work on specific domains;
* explain what to use Downloader middleware for
  and what to use Spider middleware for;
* Engine no longer locates spiders based on domains;
* "Spider middleware output direction" step was missing.

See also: GH-1569.
 2016-03-31 12:33:29 +02:00
Paul Tremberth 2ea9bc81f6 Merge pull request #1898 from scrapy/backport-1.1-pr1883 
[Backport][1.1] Make FilesPipeline work with S3FilesStore using botocore
 2016-03-31 12:30:45 +02:00
Konstantin Lopuhin b01bf4728f Fix a race condition in the FilesPipeline 
Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
 2016-03-31 12:00:26 +02:00
Konstantin Lopuhin 84371ffb22 Fix handling of meta=None in S3FilesStore.persist_file 2016-03-31 12:00:26 +02:00
nyov 5ae8863d3d Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-31 11:56:00 +02:00
pawelmhm b181c10fc4 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-31 11:52:49 +02:00
Aron Bordin 7b1dddceb3 small doc style fixes 2016-03-30 18:36:34 +02:00
nyov 718d386ebd Update documentation links 2016-03-30 18:36:34 +02:00
djunzu 3d6abacc55 Add pps extension to IGNORED_EXTENSIONS 
modified:   scrapy/linkextractors/__init__.py
 2016-03-30 18:33:58 +02:00
Paul Tremberth d381f8d930 Merge pull request #1892 from scrapy/backport-1.1-pr1822 
[Backport][1.1] Allow core Scheduler priority queue customization
 2016-03-30 18:17:22 +02:00
nyov 1adc47ce93 Allow core Scheduler priority queue customization 2016-03-30 17:51:37 +02:00
Paul Tremberth 844f040bc4 Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 17:49:14 +02:00
nanolab e7a5e3717b Update httpcache.py 
It checks cache directory modification time, but have to check file modification time.
 2016-03-30 10:53:54 +02:00
Lele 15ffb076ca Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-30 10:53:54 +02:00
Paul Tremberth e025f374e8 Bump version: 1.1.0rc2 → 1.1.0rc3 2016-03-01 23:40:30 +01:00
Paul Tremberth 9a4ead8e5d Merge pull request #1831 from redapple/backport-1.1-pr1804 
[Backport][1.1] Re-enable HTTPS tests for download timeouts
 2016-03-01 15:22:31 +01:00
Paul Tremberth 4547a7e759 Explicitly call Twisted transport stopProducing() on HTTP/1.0 timeouts 2016-03-01 15:00:58 +01:00
Paul Tremberth a0c5c61855 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-03-01 15:00:44 +01:00
Mikhail Korobov 9a992cdfd7 declare Python 3 support in setup.py 2016-03-01 14:38:49 +01:00
Mikhail Korobov a83ea5d50b Extract a function to build CONNECT request; add tests for it. See GH-1701 and GH-1808. 2016-03-01 14:38:39 +01:00
Paul Tremberth fe3a2fcce9 Build universal wheels 2016-03-01 14:18:14 +01:00
Paul Tremberth ab5e2accda Bump version: 1.1.0rc1 → 1.1.0rc2 2016-02-29 13:15:54 +01:00
Paul Tremberth a4cc8cd2a2 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 13:15:50 +01:00
Paul Tremberth acfc915b7a Update release notes about HTTPS downloader 2016-02-27 15:13:30 +01:00
Paul Tremberth c1337d74ab Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-27 15:12:56 +01:00
Paul Tremberth ed3ef17397 Merge pull request #1821 from redapple/backport-1.1-pr1818 
[backport][1.1] Refactored SpiderLoader class constructor for easier subclassing
 2016-02-27 14:50:36 +01:00
Lucas Moauro 9766695550 Refactored SpiderLoader class constructor for easier subclassing 2016-02-27 14:47:20 +01:00
Paul Tremberth 75743c95fd Merge pull request #1816 from redapple/backport-1.1-pr1769 
[backport][1.1] PY3: Fix TypeError when outputting to stdout
 2016-02-25 00:46:21 +01:00
orangain 838bc08098 PY3: Fix TypeError when outputting to stdout 
This will fix #1768.
 2016-02-25 00:35:41 +01:00
Victor Mireyev bc170a2495 Fix typo in docstring 2016-02-25 00:28:54 +01:00
Paul Tremberth 394f42440b Merge pull request #1815 from redapple/backport-1.1-pr1794 
[backport][1.1] Use best practices for TLS connections when using Twisted>=14.0
 2016-02-24 22:59:54 +01:00