RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,222 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

6222 Commits

Author SHA1 Message Date
Mikhail Korobov f0c4e5dfd5 TST fixed ProjectUtilsTest on OS X 
Temp folder can be a symlink on OS X.
 2017-02-16 16:32:23 +05:00
Mikhail Korobov 71dda2700b DOC remove “Python 3 progress” badge 
Badge doesn’t work, and Scrapy already works fine in Python 3
for practical purposes.
 2017-02-16 04:22:19 +05:00
Mikhail Korobov 692975acb4 deprecate Spider.make_requests_from_url. Fixes #1495. 2017-02-16 03:39:34 +05:00
Mikhail Korobov d09eed7674 use w3lib.html.strip_html5_whitespace function; expand docs; strip consistently before calling process_value 2017-02-16 02:22:18 +05:00
Mikhail Korobov d079e15fe2 Strip leading/trailing whitespaces in link extractors. Fixes GH-838. 2017-02-16 02:22:17 +05:00
Mikhail Korobov ad36a4a6ae RegexLinkExtractor: add \x0c to whitespace characters, as per html5 standard 2017-02-16 02:22:17 +05:00
Mikhail Korobov 074caf434e FormRequest: handle whitespaces in action attribute properly 2017-02-16 02:13:39 +05:00
Mikhail Korobov fade5763af TST more response.follow tests 2017-02-16 02:02:50 +05:00
Mikhail Korobov 5b79c6a679 DOC document response.follow methods; expand the tutorial 2017-02-16 00:06:52 +05:00
Mikhail Korobov 160da6abab fixed tests in Python 2 2017-02-15 04:41:53 +05:00
Mikhail Korobov 2674f317df Response.follow 2017-02-15 04:39:47 +05:00
Mikhail Korobov 608c3f0c45 handle whitespace in response.follow; add tests 2017-02-15 04:17:41 +05:00
Mikhail Korobov 71dd5d0bf9 strip URL extracted from selectors (as per html5 standard) 2017-02-15 01:22:53 +05:00
Mikhail Korobov 877057fac0 initial response.follow implementation 2017-02-15 01:22:53 +05:00
Mikhail Korobov e1ceaf3b5f require w3lib 1.17+ 2017-02-15 00:32:44 +05:00
Mikhail Korobov 39df675f09 make retry middleware changes backwards compatible 2017-02-14 23:28:50 +05:00
Eugenio Lacuesta 922d3fec54 Doc: binary mode is required for exporters 2017-02-14 12:51:03 -03:00
Mikhail Korobov e285b1d6c2 retry stats 2017-02-14 20:27:49 +05:00
Mikhail Korobov afac3fd2c2 Merge pull request #2530 from elacuesta/proxy_credentials 
[MRG+1] Use credentials from request.meta['proxy']
 2017-02-14 20:58:39 +06:00
Eugenio Lacuesta ae0ea31abd Add HTTPPROXY_ENABLED setting (default True) 2017-02-14 11:33:01 -03:00
terut 5b31dfe3c9 Separate building request from _requests_to_follow in CrawlSpider 
You just overwrite buiding request if you can use another request class
because of something like splash-plugin.
 2017-02-13 23:51:43 -08:00
Daniel Graña 7dd7646e65 Bump version: 1.3.1 → 1.3.2 2017-02-13 14:57:55 -03:00
Daniel Graña 9315e944a2 Release notes for 1.3.2 2017-02-13 14:56:29 -03:00
Daniel Graña 45f19021a5 Merge pull request #2558 from scrapy/twisted-17-fix 
Fixed compatibility with twisted 17+
 2017-02-13 14:48:12 -03:00
Mikhail Korobov 1bc4d8b6b6 fixed tls in Twisted 17+ 2017-02-13 20:03:53 +05:00
Mikhail Korobov de65ad3fb1 TST replace Ubuntu 12.04 tox environment with 14.04 2017-02-13 18:44:39 +05:00
Mikhail Korobov 9956f198db add a couple more lines to gitignore 2017-02-13 18:40:53 +05:00
Mikhail Korobov 5ee595d695 Merge pull request #2551 from redapple/tutorial-author-selector 
Use consistent selectors for author field in tutorial
 2017-02-09 18:16:49 +05:00
Paul Tremberth 29e60213db Use consistent selectors for author field in tutorial 2017-02-09 10:41:21 +01:00
Paul Tremberth f32a229e3d Merge pull request #2510 from elacuesta/reqser_request_class 
[MRG+1] Preserve request class when converting to/from dicts (utils.reqser)
 2017-02-08 18:30:07 +01:00
Eugenio Lacuesta 9c0aae724e Use credentials from request.meta['proxy'] if present 2017-02-08 13:22:30 -03:00
Paul Tremberth af55a8713e Bump version: 1.3.0 → 1.3.1 2017-02-08 17:08:19 +01:00
Paul Tremberth ff8a564b1a Set date for 1.3.1 release 2017-02-08 17:05:06 +01:00
Mikhail Korobov 094937bc27 Merge pull request #2536 from scrapy/release-notes-1.3.1 
[MRG] Update changelog for upcoming 1.3.1 release
 2017-02-08 20:47:10 +05:00
Paul Tremberth 48c8c679de Update changelog for upcoming 1.3.1 release 2017-02-08 15:53:44 +01:00
Paul Tremberth d205206aaa Merge pull request #2345 from gustavodeandrade/master 
[MRG+1] Fix documentation about HTML entities decoding with selector extraction
 2017-02-08 13:21:15 +01:00
Paul Tremberth 7d0b89042f Merge pull request #2533 from djrobust/patch-1 
[MRG+1] Use yield with nested parsing of responses
 2017-02-08 13:02:50 +01:00
Paul Tremberth a8893190a1 Merge pull request #2464 from elacuesta/component_order_integer 
[MRG+2] Validate numeric values (components order)
 2017-02-08 13:01:02 +01:00
Paul Tremberth 3becb861af Merge pull request #2544 from scrapy/twisted-version-cleanup 
[MRG+1] Remove code required to support ancient twisted versions.
 2017-02-08 12:59:45 +01:00
Paul Tremberth e87f18447c Merge pull request #2538 from scrapy/linkextractor-fail-fast 
LinkExtractor: don’t check all regexes if one of them matches
 2017-02-08 12:58:41 +01:00
Paul Tremberth 5ad64204ac Merge pull request #2509 from rolando-contrib/py36-classcell-compat 
[MRG+1] BUG: Fix __classcell__ propagation required in Python 3.6.
 2017-02-08 12:15:15 +01:00
Rolando Espinoza 4e765acaed BUG: Fix __classcell__ propagation. 
Python 3.6 added simpler customization of class creation but this
requires to propagate correctly the __classcell__ attribute in custom
__new__ methods.

See https://docs.python.org/3.6/whatsnew/3.6.html#pep-487-simpler-
customization-of-class-creation
 2017-02-08 02:12:52 -04:00
Mikhail Korobov f2f9350c47 Merge pull request #2542 from scrapy/redirect-cleanup 
[MRG+1] Cleanup MetaRefreshMiddleware: remove redundant check
 2017-02-08 00:57:21 +05:00
Mikhail Korobov 04b2f79e7a Remove code required to support ancient twisted versions. See GH-1887. 2017-02-07 22:30:58 +05:00
Mikhail Korobov eaf62ab69c cleanup MetaRefreshMiddleware: remove redundant check 2017-02-07 17:56:43 +05:00
Eugenio Lacuesta 24e82bfe75 Validate values for components order 2017-02-07 09:13:57 -03:00
Pawel Miech fb4ef21a1d [httpcompression] minor style edits 2017-02-07 10:22:42 +01:00
Mikhail Korobov 85a124970a Enable memusage extension by default. Fixes GH-2187. 2017-02-07 03:32:54 +05:00
Mikhail Korobov f73eb715ac LinkExtractor: don’t check all regexes if one of them matches 2017-02-07 02:16:06 +05:00
Pawel Miech af802bad14 [httpcompression] add brotlipy for python 3 2017-02-06 15:45:21 +01:00