RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,896 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

5896 Commits

Author SHA1 Message Date
Mikhail Korobov 1be2447af9 Merge pull request #2346 from redapple/master-pr2313 
Fix tutorial AuthorSpider
 2016-10-21 11:08:17 +02:00
Randy Pen c7d245b90b update 
Thx for your advice.
 2016-10-21 10:52:50 +02:00
Randy Pen eacc5937e4 fix example code 
In the AuthorSpider, original css selector failed to get links of author pages
 2016-10-21 10:52:50 +02:00
Paul Tremberth 6df48d57e0 Bump version: 1.2.0 → 1.2.1 2016-10-21 10:37:45 +02:00
Paul Tremberth f0e935735f Merge pull request #2341 from redapple/release-notes-1.2.1 
Update release notes for upcoming 1.2.1
 2016-10-21 10:34:47 +02:00
Paul Tremberth c559a64c20 Set date for 1.2.1 release 2016-10-21 10:17:46 +02:00
Paul Tremberth 22772a61c8 Update release notes for upcoming 1.2.1 2016-10-21 10:17:04 +02:00
Paul Tremberth eb5d396527 Merge pull request #2344 from jaympatel/master 
Typo (through was misspelled)
 2016-10-20 22:24:14 +02:00
Jay Patel f357ccd0d7 Typo (through was misspelled) 2016-10-20 15:52:28 -04:00
Mikhail Korobov 871eec9827 Merge pull request #2327 from bopace/http-header-docs 
[MRG+1] Added documentation about accessing header values
 2016-10-20 09:05:14 +02:00
Mikhail Korobov 71c8278f57 Merge pull request #2329 from josericardo/scrapy-2262 
[MRG+1] Better explain middleware orders and processing directions
 2016-10-20 09:04:19 +02:00
Paul Tremberth db40852892 Do not interpret non-ASCII bytes in "Location" and percent-encode them (#2322) 
* Do not interpret non-ASCII bytes in "Location" and percent-encode them

Fixes GH-2321

The idea is to not guess the encoding of "Location" header value
and simply percent-encode non-ASCII bytes,
which should then be re-interpreted correctly by the remote website
in whatever encoding was used originally.

See https://tools.ietf.org/html/rfc3987#section-3.2

This is similar to the changes to safe_url_string in
https://github.com/scrapy/w3lib/pull/45

* Remove unused import
 2016-10-19 23:26:12 -03:00
bopace fd016ee71b Fixed wording of documentation 2016-10-18 09:37:45 -06:00
Paul Tremberth 6cc83c041e Merge pull request #2330 from lfmattossch/note-python2-name 
[MRG+1] Added a note about invalid spider names in python 2 (fixes #2251)
 2016-10-18 16:35:19 +02:00
Jose Ricardo e12e364a40 Add details to the spider middlewares docs 
Document the effects of the middleware order in a more detailed way.
 2016-10-18 12:29:30 -02:00
Mikhail Korobov a5f4450313 Merge pull request #2302 from Granitosaurus/pipeline_doc_fix 
[MRG+1] Fix JsonWriterPipeline example in docs
 2016-10-18 16:20:59 +02:00
Luiz Fernando Mattos Schlindwein 7c33e0cb55 added a note about invalid spider names in python 2 2016-10-18 11:55:51 -02:00
Jose Ricardo ea7bd39529 Make architecture overview references a little more clear on the docs 
Expliciting what actually happens by adding links to the respective methods
that are invoked in each processing phase.
 2016-10-18 11:50:51 -02:00
Jose Ricardo bebcd5081c Add downloader middleware ordering details to the docs 
Add more details, making it easier to understand what are the effects of
setting a downloader middleware order.
 2016-10-18 11:22:55 -02:00
Bo Pace bfe28ae707 Added documentation about accessing header values 2016-10-17 14:10:05 -06:00
Mikhail Korobov 7e20725eb7 Merge pull request #2314 from redapple/tls-ciphers 
[MRG+1] Use OpenSSL default ciphers
 2016-10-17 11:44:28 +02:00
Paul Tremberth dc1f9ad211 Merge pull request #2307 from eLRuLL/genspider-no-www-fix 
genspider: removing www. from starturl templates
 2016-10-12 15:32:18 +02:00
Raul Gallegos 118b42ab59 making start_urls a list in basic genspider template 2016-10-11 22:08:05 -05:00
Paul Tremberth c3411373e8 Use OpenSSL default ciphers 
Twisted default TLS options restricts the ciphers list a bit -- "a secure default"
e38cc25a67/src/twisted/internet/_sslverify.py (L1861)

We want to be a bit more permissive with Scrapy
(at least as permissive as Scrapy 1.0 was, and which used a default OpenSSL Context)

See https://www.openssl.org/docs/manmaster/apps/ciphers.html#CIPHER_STRINGS

OpenSSL's 'DEFAULT' seems to be reasonable enough.

Fixes #2311
 2016-10-07 12:28:16 +02:00
Raul Gallegos dd778892d0 genspider: removing www. from starturl templates 2016-10-05 12:16:45 -05:00
Bernardas dfba151f59 Remove unnecessary note for the JsonWriterPipeline example 2016-10-05 16:36:02 +00:00
Bernardas eb91cb8ea2 fix JsonWriterPipeline example 2016-10-03 20:31:41 +00:00
Paul Tremberth 3235bfeb1e Bump version: 1.2.0dev2 → 1.2.0 2016-10-03 13:04:11 +02:00
Paul Tremberth 95c6b9dffd Merge pull request #2280 from scrapy/release-notes-1.2 
Update release notes for upcoming 1.2.0
 2016-10-03 13:00:14 +02:00
Paul Tremberth e2137d77ce Add release date for scrapy 1.2 2016-10-03 12:41:18 +02:00
Paul Tremberth a433126301 Merge pull request #2294 from redapple/release-headers 
Change release section titles to have correct links in ToC
 2016-10-03 12:38:44 +02:00
Paul Tremberth 32c46453c3 Merge pull request #2300 from mineo/patch-3 
Remove duplicate colons from the feed export settings docs
 2016-10-03 12:34:28 +02:00
Wieland Hoffmann e8edc6c2bb Remove duplicate colons from the feed export settings docs 2016-10-02 16:09:29 +02:00
Daniel Graña 5d38a8e2d9 Merge pull request #2298 from scrapy/update-1.2-relnotes-data-dir-util 
Update 1.2 release notes with data_path changes from #1581
 2016-09-30 15:23:55 -03:00
Daniel Graña bca374d651 Merge pull request #1581 from scrapy/fix-util-function-to-work-outside-project-dir 
[MRG+1] Make data_path work when outside project (used by HttpCacheMiddleware and Deltafetch plugin)
 2016-09-30 15:23:34 -03:00
Elias Dorneles 2d932c173c test abs path outside project as well 2016-09-30 15:07:58 -03:00
Paul Tremberth fed53c1e28 Merge pull request #2267 from scrapy/deprecate-ubuntu-packages 
[MRG+1] Deprecate official Ubuntu packages and update installation instructions
 2016-09-30 17:35:29 +02:00
Elias Dorneles 559b4edaec update release notes with changes from #1581 2016-09-30 11:42:50 -03:00
Paul Tremberth ab3f27b502 Merge pull request #2296 from scrapy/architecture-overview-once-more-with-feeling 
[MRG+1] architecture docs: restore explanation loop to step 1 (see comment in 3ac3ac4)
 2016-09-30 16:30:37 +02:00
Elias Dorneles 8bafcf33a1 improve description of engine 2016-09-30 11:20:30 -03:00
Elias Dorneles 9c90d9515a update data_path dosctring 2016-09-29 19:11:17 -03:00
Elias Dorneles e107212043 restore explanation loop to step 1 (see comment in 3ac3ac4) 2016-09-29 19:02:57 -03:00
Elias Dorneles 25bd3b3fea add .scrapy when outside spider too, add tests 2016-09-29 18:30:42 -03:00
Elias Dorneles 8e4947ef0d make utils.project.data_path return path unmodified when outside project 2016-09-29 10:17:35 -03:00
Paul Tremberth fd0b6faecb Change release section titles to have correct links in ToC 2016-09-29 14:36:13 +02:00
Paul Tremberth 47ef202acd Merge pull request #2285 from redapple/relnotes-debian-jessie-baseline 
Mention new Debian Jessie baseline
 2016-09-29 14:25:53 +02:00
Paul Tremberth c670e3cdf6 Move mention 2016-09-29 14:23:52 +02:00
Elias Dorneles 86eb865598 Merge pull request #2287 from pawelmhm/docs-2230 
[MRG+1][docs/item_pipeline] process_item returning Deferred docs
 2016-09-29 09:15:09 -03:00
Elias Dorneles 60559369b5 Merge pull request #2292 from stummjr/fix-example-section 
[MRG+1] Update examples section in documentation with quotes.toscrape spiders
 2016-09-29 09:11:45 -03:00
Paul Tremberth 7ba16b7360 Merge pull request #2293 from scrapy/architecture-overview-fix-reqs-vs-urls 
docs: update data flow description and image (fixes: #2278)
 2016-09-29 10:27:36 +02:00