RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,889 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

5889 Commits

Author SHA1 Message Date
Paul Tremberth eb5d396527 Merge pull request #2344 from jaympatel/master 
Typo (through was misspelled)
 2016-10-20 22:24:14 +02:00
Jay Patel f357ccd0d7 Typo (through was misspelled) 2016-10-20 15:52:28 -04:00
Mikhail Korobov 871eec9827 Merge pull request #2327 from bopace/http-header-docs 
[MRG+1] Added documentation about accessing header values
 2016-10-20 09:05:14 +02:00
Mikhail Korobov 71c8278f57 Merge pull request #2329 from josericardo/scrapy-2262 
[MRG+1] Better explain middleware orders and processing directions
 2016-10-20 09:04:19 +02:00
Paul Tremberth db40852892 Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322) 
* Do not interpret non-ASCII bytes in "Location" and percent-encode them

Fixes GH-2321

The idea is to not guess the encoding of "Location" header value
and simply percent-encode non-ASCII bytes,
which should then be re-interpreted correctly by the remote website
in whatever encoding was used originally.

See https://tools.ietf.org/html/rfc3987#section-3.2

This is similar to the changes to safe_url_string in
https://github.com/scrapy/w3lib/pull/45

* Remove unused import
 2016-10-19 23:26:12 -03:00
bopace fd016ee71b Fixed wording of documentation 2016-10-18 09:37:45 -06:00
Paul Tremberth 6cc83c041e Merge pull request #2330 from lfmattossch/note-python2-name 
[MRG+1] Added a note about invalid spider names in python 2 (fixes #2251)
 2016-10-18 16:35:19 +02:00
Jose Ricardo e12e364a40 Add details to the spider middlewares docs 
Document the effects of the middleware order in a more detailed way.
 2016-10-18 12:29:30 -02:00
Mikhail Korobov a5f4450313 Merge pull request #2302 from Granitosaurus/pipeline_doc_fix 
[MRG+1] Fix JsonWriterPipeline example in docs
 2016-10-18 16:20:59 +02:00
Luiz Fernando Mattos Schlindwein 7c33e0cb55 added a note about invalid spider names in python 2 2016-10-18 11:55:51 -02:00
Jose Ricardo ea7bd39529 Make architecture overview references a little more clear on the docs 
Expliciting what actually happens by adding links to the respective methods
that are invoked in each processing phase.
 2016-10-18 11:50:51 -02:00
Jose Ricardo bebcd5081c Add downloader middleware ordering details to the docs 
Add more details, making it easier to understand what are the effects of
setting a downloader middleware order.
 2016-10-18 11:22:55 -02:00
Bo Pace bfe28ae707 Added documentation about accessing header values 2016-10-17 14:10:05 -06:00
Mikhail Korobov 7e20725eb7 Merge pull request #2314 from redapple/tls-ciphers 
[MRG+1] Use OpenSSL default ciphers
 2016-10-17 11:44:28 +02:00
Paul Tremberth dc1f9ad211 Merge pull request #2307 from eLRuLL/genspider-no-www-fix 
genspider: removing www. from starturl templates
 2016-10-12 15:32:18 +02:00
Raul Gallegos 118b42ab59 making start_urls a list in basic genspider template 2016-10-11 22:08:05 -05:00
Paul Tremberth c3411373e8 Use OpenSSL default ciphers 
Twisted default TLS options restricts the ciphers list a bit -- "a secure default"
e38cc25a67/src/twisted/internet/_sslverify.py (L1861)

We want to be a bit more permissive with Scrapy
(at least as permissive as Scrapy 1.0 was, and which used a default OpenSSL Context)

See https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER_STRINGS

OpenSSL's 'DEFAULT' seems to be reasonable enough.

Fixes #2311
 2016-10-07 12:28:16 +02:00
Raul Gallegos dd778892d0 genspider: removing www. from starturl templates 2016-10-05 12:16:45 -05:00
Bernardas dfba151f59 Remove unnecessary note for the JsonWriterPipeline example 2016-10-05 16:36:02 +00:00
Bernardas eb91cb8ea2 fix JsonWriterPipeline example 2016-10-03 20:31:41 +00:00
Paul Tremberth 3235bfeb1e Bump version: 1.2.0dev2 → 1.2.0 2016-10-03 13:04:11 +02:00
Paul Tremberth 95c6b9dffd Merge pull request #2280 from scrapy/release-notes-1.2 
Update release notes for upcoming 1.2.0
 2016-10-03 13:00:14 +02:00
Paul Tremberth e2137d77ce Add release date for scrapy 1.2 2016-10-03 12:41:18 +02:00
Paul Tremberth a433126301 Merge pull request #2294 from redapple/release-headers 
Change release section titles to have correct links in ToC
 2016-10-03 12:38:44 +02:00
Paul Tremberth 32c46453c3 Merge pull request #2300 from mineo/patch-3 
Remove duplicate colons from the feed export settings docs
 2016-10-03 12:34:28 +02:00
Wieland Hoffmann e8edc6c2bb Remove duplicate colons from the feed export settings docs 2016-10-02 16:09:29 +02:00
Daniel Graña 5d38a8e2d9 Merge pull request #2298 from scrapy/update-1.2-relnotes-data-dir-util 
Update 1.2 release notes with data_path changes from #1581
 2016-09-30 15:23:55 -03:00
Daniel Graña bca374d651 Merge pull request #1581 from scrapy/fix-util-function-to-work-outside-project-dir 
[MRG+1] Make data_path work when outside project (used by HttpCacheMiddleware and Deltafetch plugin)
 2016-09-30 15:23:34 -03:00
Elias Dorneles 2d932c173c test abs path outside project as well 2016-09-30 15:07:58 -03:00
Paul Tremberth fed53c1e28 Merge pull request #2267 from scrapy/deprecate-ubuntu-packages 
[MRG+1] Deprecate official Ubuntu packages and update installation instructions
 2016-09-30 17:35:29 +02:00
Elias Dorneles 559b4edaec update release notes with changes from #1581 2016-09-30 11:42:50 -03:00
Paul Tremberth ab3f27b502 Merge pull request #2296 from scrapy/architecture-overview-once-more-with-feeling 
[MRG+1] architecture docs: restore explanation loop to step 1 (see comment in 3ac3ac4)
 2016-09-30 16:30:37 +02:00
Elias Dorneles 8bafcf33a1 improve description of engine 2016-09-30 11:20:30 -03:00
Elias Dorneles 9c90d9515a update data_path dosctring 2016-09-29 19:11:17 -03:00
Elias Dorneles e107212043 restore explanation loop to step 1 (see comment in 3ac3ac4) 2016-09-29 19:02:57 -03:00
Elias Dorneles 25bd3b3fea add .scrapy when outside spider too, add tests 2016-09-29 18:30:42 -03:00
Elias Dorneles 8e4947ef0d make utils.project.data_path return path unmodified when outside project 2016-09-29 10:17:35 -03:00
Paul Tremberth fd0b6faecb Change release section titles to have correct links in ToC 2016-09-29 14:36:13 +02:00
Paul Tremberth 47ef202acd Merge pull request #2285 from redapple/relnotes-debian-jessie-baseline 
Mention new Debian Jessie baseline
 2016-09-29 14:25:53 +02:00
Paul Tremberth c670e3cdf6 Move mention 2016-09-29 14:23:52 +02:00
Elias Dorneles 86eb865598 Merge pull request #2287 from pawelmhm/docs-2230 
[MRG+1][docs/item_pipeline] process_item returning Deferred docs
 2016-09-29 09:15:09 -03:00
Elias Dorneles 60559369b5 Merge pull request #2292 from stummjr/fix-example-section 
[MRG+1] Update examples section in documentation with quotes.toscrape spiders
 2016-09-29 09:11:45 -03:00
Paul Tremberth 7ba16b7360 Merge pull request #2293 from scrapy/architecture-overview-fix-reqs-vs-urls 
docs: update data flow description and image (fixes: #2278)
 2016-09-29 10:27:36 +02:00
Paul Tremberth 33d04684e5 Fix typo 2016-09-29 10:25:21 +02:00
Elias Dorneles 3ac3ac4d92 docs: update data flow description and image (fixes: #2278) 
This fixes the explanation to use Requests instead of URLs,
which is what actually happens, and is also consistent with the
new tutorial, which already explains how URLs become Request objects.

I've also changed the "loop", jumping from 9 to step 2.
 2016-09-28 16:38:45 -03:00
Valdir Stumm Junior 5680c61063 Doc: update examples section with quotes.toscrape.com spiders 2016-09-28 15:45:01 -03:00
Paul Tremberth 23abf2d31b Fix title underline 2016-09-28 18:31:27 +02:00
Paul Tremberth 8d130b295a Update installation guide 2016-09-28 17:09:17 +02:00
Paul Tremberth c707c31345 Fix Debian number 2016-09-26 17:20:13 +02:00
pawelmhm 39dbd890b0 [docs/item_pipeline] process_item returning Deferred docs 
* quote url
* use hash of url as filename
 2016-09-24 08:36:09 +02:00