RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,137 Commits 49 Branches 132 Tags 141 MiB
Commit Graph

6137 Commits

Author SHA1 Message Date
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Daniel Graña b0388e49b4 Merge pull request #1728 from scrapy/deprecate-make-requests-from-url 
deprecate Spider.make_requests_from_url.
 2017-02-20 11:23:49 -03:00
Daniel Graña c68140e68a Merge pull request #2540 from scrapy/response-follow 
response.follow
 2017-02-20 11:21:21 -03:00
Daniel Graña 377db31a32 Merge pull request #2557 from scrapy/gitignore 
add a couple more lines to gitignore
 2017-02-20 11:18:48 -03:00
Daniel Graña e578be73b6 Merge pull request #2539 from scrapy/enable-memusage 
Enable memusage extension by default.
 2017-02-20 11:18:04 -03:00
Paul Tremberth 4a1b88fe13 Merge pull request #2567 from scrapy/no-py3-badge 
remove “Python 3 progress” badge
 2017-02-20 15:16:40 +01:00
Daniel Graña b15b4541d1 Merge pull request #2569 from scrapy/fix-project-util-test-osx 
TST fixed ProjectUtilsTest on OS X
 2017-02-20 11:15:21 -03:00
Daniel Graña fab168bbfb Merge pull request #2572 from advarisk/warning-formrequest-from-response 
[MRG+1] document issue with FormRequest.from_response due to bug in lxml
 2017-02-20 11:14:28 -03:00
Daniel Graña 85ef6a6229 Merge pull request #2564 from elacuesta/docs_exporters 
[MRG+1] Doc: binary mode is required for exporters
 2017-02-20 11:12:48 -03:00
Daniel Graña b2d66dc1f0 Merge pull request #2562 from terut/feature/build_request 
[MRG+1] Separate building request from _requests_to_follow in CrawlSpider
 2017-02-20 11:12:10 -03:00
Daniel Graña 322fd68e4c Merge pull request #2548 from scrapy/formrequest-whitespaces 
[MRG+1] FormRequest: handle whitespaces in action attribute properly
 2017-02-20 11:09:50 -03:00
Daniel Graña 4a93be4ad8 Merge pull request #2547 from scrapy/linkextractor-strip-whitespaces 
[MRG+1] LinkExtractors: strip whitespaces
 2017-02-20 11:08:32 -03:00
Daniel Graña 58a18e3007 Merge pull request #2535 from pawelmhm/brotli 
[MRG+1] [httpcompression] add support for br - brotli content encoding
 2017-02-20 11:04:55 -03:00
Mikhail Korobov 7ad5156287 Merge pull request #2570 from scrapy/tests-cleanup-resources 
[MRG+1] TST remove temp files and folders
 2017-02-20 20:00:32 +06:00
Ashish Kulkarni 165e2cb8c9 document issue with FormRequest.from_response due to bug in lxml 
This can make the spider fail due to incorrect values being posted
server-side, which is extremely hard to debug because it is easy
to miss leading/trailing whitespace, even with a logging proxy.

The fix was merged for lxml 3.8 in lxml/lxml#228 so document that
as well.
 2017-02-17 14:54:22 +05:30
Mikhail Korobov 5adacc4dde fixed method override check in Python 2 2017-02-17 02:35:02 +05:00
Mikhail Korobov a1e8a8525d fix make_requests_from_url deprcation implementation, add tests 2017-02-17 00:18:29 +05:00
Mikhail Korobov 78cb46a079 TST fix a weird OS X testing issue 
pytest intercepts temp file creation to provide readable file/folder
names; path is built from method name; in case of conflicts
pytests uses increasing numbers, but it seems it doesn’t account
for case-insensitive (but case preserving) OS X filesystem. There
are methods named test_encoding, pytest thinks test_Encoding is
different and fails to create a test folder
 2017-02-16 18:22:04 +05:00
Mikhail Korobov b9928558b2 TST remove temporary files and folders 2017-02-16 17:50:38 +05:00
Mikhail Korobov f0c4e5dfd5 TST fixed ProjectUtilsTest on OS X 
Temp folder can be a symlink on OS X.
 2017-02-16 16:32:23 +05:00
Mikhail Korobov 71dda2700b DOC remove “Python 3 progress” badge 
Badge doesn’t work, and Scrapy already works fine in Python 3
for practical purposes.
 2017-02-16 04:22:19 +05:00
Mikhail Korobov 692975acb4 deprecate Spider.make_requests_from_url. Fixes #1495. 2017-02-16 03:39:34 +05:00
Mikhail Korobov d09eed7674 use w3lib.html.strip_html5_whitespace function; expand docs; strip consistently before calling process_value 2017-02-16 02:22:18 +05:00
Mikhail Korobov d079e15fe2 Strip leading/trailing whitespaces in link extractors. Fixes GH-838. 2017-02-16 02:22:17 +05:00
Mikhail Korobov ad36a4a6ae RegexLinkExtractor: add \x0c to whitespace characters, as per html5 standard 2017-02-16 02:22:17 +05:00
Mikhail Korobov 074caf434e FormRequest: handle whitespaces in action attribute properly 2017-02-16 02:13:39 +05:00
Mikhail Korobov fade5763af TST more response.follow tests 2017-02-16 02:02:50 +05:00
Mikhail Korobov 5b79c6a679 DOC document response.follow methods; expand the tutorial 2017-02-16 00:06:52 +05:00
Mikhail Korobov 160da6abab fixed tests in Python 2 2017-02-15 04:41:53 +05:00
Mikhail Korobov 2674f317df Response.follow 2017-02-15 04:39:47 +05:00
Mikhail Korobov 608c3f0c45 handle whitespace in response.follow; add tests 2017-02-15 04:17:41 +05:00
Mikhail Korobov 71dd5d0bf9 strip URL extracted from selectors (as per html5 standard) 2017-02-15 01:22:53 +05:00
Mikhail Korobov 877057fac0 initial response.follow implementation 2017-02-15 01:22:53 +05:00
Mikhail Korobov e1ceaf3b5f require w3lib 1.17+ 2017-02-15 00:32:44 +05:00
Eugenio Lacuesta 922d3fec54 Doc: binary mode is required for exporters 2017-02-14 12:51:03 -03:00
Mikhail Korobov afac3fd2c2 Merge pull request #2530 from elacuesta/proxy_credentials 
[MRG+1] Use credentials from request.meta['proxy']
 2017-02-14 20:58:39 +06:00
Eugenio Lacuesta ae0ea31abd Add HTTPPROXY_ENABLED setting (default True) 2017-02-14 11:33:01 -03:00
terut 5b31dfe3c9 Separate building request from _requests_to_follow in CrawlSpider 
You just overwrite buiding request if you can use another request class
because of something like splash-plugin.
 2017-02-13 23:51:43 -08:00
Daniel Graña 7dd7646e65 Bump version: 1.3.1 → 1.3.2 2017-02-13 14:57:55 -03:00
Daniel Graña 9315e944a2 Release notes for 1.3.2 2017-02-13 14:56:29 -03:00
Daniel Graña 45f19021a5 Merge pull request #2558 from scrapy/twisted-17-fix 
Fixed compatibility with twisted 17+
 2017-02-13 14:48:12 -03:00
Mikhail Korobov 1bc4d8b6b6 fixed tls in Twisted 17+ 2017-02-13 20:03:53 +05:00
Mikhail Korobov de65ad3fb1 TST replace Ubuntu 12.04 tox environment with 14.04 2017-02-13 18:44:39 +05:00
Mikhail Korobov 9956f198db add a couple more lines to gitignore 2017-02-13 18:40:53 +05:00
Mikhail Korobov 5ee595d695 Merge pull request #2551 from redapple/tutorial-author-selector 
Use consistent selectors for author field in tutorial
 2017-02-09 18:16:49 +05:00
Paul Tremberth 29e60213db Use consistent selectors for author field in tutorial 2017-02-09 10:41:21 +01:00
Paul Tremberth f32a229e3d Merge pull request #2510 from elacuesta/reqser_request_class 
[MRG+1] Preserve request class when converting to/from dicts (utils.reqser)
 2017-02-08 18:30:07 +01:00
Eugenio Lacuesta 9c0aae724e Use credentials from request.meta['proxy'] if present 2017-02-08 13:22:30 -03:00
Paul Tremberth af55a8713e Bump version: 1.3.0 → 1.3.1 2017-02-08 17:08:19 +01:00
Paul Tremberth ff8a564b1a Set date for 1.3.1 release 2017-02-08 17:05:06 +01:00