RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
Eugenio Lacuesta bddfeaba4c Add Request.kwargs docs 2019-01-15 19:14:59 -03:00
kasun Herath f3813e376c Merge remote-tracking branch 'origin/master' into json_request 2019-01-14 23:03:21 +05:30
kasun Herath 3f914f6d8c made jsonrequest dump into private method 2019-01-14 23:03:14 +05:30
Maram Sumanth 023290dabc
Update test_http_request.py 2019-01-13 23:50:31 +05:30
Maram Sumanth 4abcdcb306
Update test_http_request.py 2019-01-13 23:22:53 +05:30
Maram Sumanth 9a4bbd6d02
Update form.py 2019-01-13 23:05:58 +05:30
Maram Sumanth 1bea5d3076
Fixed error 2019-01-13 22:35:16 +05:30
Maram Sumanth b5e454809e
Included test 2019-01-13 20:12:31 +05:30
Maram Sumanth ac111088c6
duplicate keys handled 2019-01-13 20:12:29 +05:30
Eugenio Lacuesta 57e7c76977 Test callback kwargs 2019-01-09 10:40:44 -03:00
Eugenio Lacuesta 770a501fb3 Test request kwargs (copy, serialization) 2019-01-09 10:40:03 -03:00
Eugenio Lacuesta a67f1ce512 Serialize Request kwargs 2019-01-03 18:19:12 -03:00
Eugenio Lacuesta 69a1ee79aa Copy request.kwargs 2019-01-03 17:38:29 -03:00
Eugenio Lacuesta a2b509a422 Pass callback kwargs with response.follow 2019-01-03 17:38:06 -03:00
Eugenio Lacuesta 50a0d87d1e Passing keyword arguments to callbacks 2019-01-03 17:20:08 -03:00
Eugenio Lacuesta 6c78b3d5ef Deques can't be sliced, use itertools.islice instead 2019-01-03 13:15:58 -03:00
Eugenio Lacuesta 9759112a3a Merge branch 'master' into process_spider_exception_generator 2019-01-03 11:34:07 -03:00
Joaquin Garmendia Cabrera e1f8b55ba0
Improve syntax for readability 2018-12-28 16:53:12 -05:00
Mikhail Korobov 094dde6fdb
Merge pull request #3512 from victor-torres/sitemap_filter 
[MRG+1] Add sitemap_filter function to SitemapSpider class
 2018-12-28 20:11:46 +05:00
Victor Torres 5a824c906c using shorter import version and moving datetime import to the beginning of the code snippet 2018-12-27 18:34:41 -03:00
Victor Torres bfbcf52e9d fix SitemapSpider import 2018-12-27 18:12:31 -03:00
Victor Torres b68308779a improving docs 2018-12-27 17:37:59 -03:00
Victor Torres e1597f7c42 improve readability 2018-12-27 11:40:14 -03:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 10f46bca54 documenting sitemap entries as suggested by @kmike 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Victor Torres 672385a371 using a method definition instead of a None attribute 2018-12-27 11:40:14 -03:00
Victor Torres a5e1b7bb47 add sitemap_filter attribute to SitemapSpider class 
it makes it possible to filter sitemap urls by any available attribute

for example, you can filter urls with lastmod greater than a given datetime

it can be helpful when the url loc itself does not aggregate that information
 2018-12-27 11:40:14 -03:00
Daniel Graña 93cf38354d
Merge pull request #3394 from hcoura/dh-lazyloading-optional 
Make lazy loading Download Handlers optional
 2018-12-26 11:16:19 -03:00
Daniel Graña f6ce716372
Merge pull request #3476 from elacuesta/deque_appendleft 
[MRG+1] Use collections.deque instead of list to store MiddlewareManager's methods
 2018-12-26 11:13:52 -03:00
Daniel Graña 76433feb4f
Merge pull request #3548 from scrapy/fix-aws-settings-docs 
fix docs for AWS_... settings. A follow-up to GH-2609.
 2018-12-26 11:10:57 -03:00
Daniel Graña 5cf26c9985
Merge pull request #3547 from scrapy/setup.py-3.7 
declare Python 3.7 support in setup.py
 2018-12-26 11:09:07 -03:00
Daniel Graña d473be2040
Merge pull request #3519 from fpghost/master 
[MRG+1] the strip() isnt needed after base64 encoding user pass
 2018-12-26 11:08:49 -03:00
Daniel Graña 5d02aab9f9
Merge pull request #3496 from frederik-elwert/patch-1 
[MRG+1] Add documentation to `scrapy shell` command.
 2018-12-26 11:04:30 -03:00
Daniel Graña 6a0ea0cf26
Merge pull request #3415 from scrapy/telnet-auth 
[MRG+1] Telnet console authentication
 2018-12-26 10:58:44 -03:00
Mikhail Korobov 71e47629b1 DOC fix docs for AWS_... settings. A follow-up to GH-2609. 2018-12-26 16:35:05 +05:00
Mikhail Korobov 4306886ac6
Merge pull request #3527 from hsiaoyi0504/patch-1 
unify the quote style
 2018-12-26 14:45:06 +05:00
Mikhail Korobov cdd04dfb1d declare Python 3.7 support in setup.py 2018-12-26 13:13:49 +05:00
Mikhail Korobov 7c26701012 DOC warn about telnet console being insecure 2018-12-26 01:33:58 +05:00
Mikhail Korobov dbfabf02e8 Merge branch 'master' into telnet-auth 2018-12-26 01:04:34 +05:00
Joaquin Garmendia Cabrera f85c915872
Update item-pipeline example 2018-12-23 00:26:58 -05:00
Daniel Graña 23bffff567
Merge pull request #3526 from lucywang000/update-travis 
remove "sudo: false" now that travis no longer supports it
 2018-12-20 19:54:34 -03:00
Daniel Graña d4fd1236b7
Merge pull request #3538 from scrapy/boto-import-error-under-jessie 
Fix boto import error under Jessie testing environment
 2018-12-20 19:49:55 -03:00
Daniel Graña 8ed6beb7f9 Needs to be installed within tox env 2018-12-20 19:39:29 -03:00
Daniel Graña f6dfc5f3dd Fix boto import error under Jessie testing environment 2018-12-20 19:23:23 -03:00
kasun Herath 24acc50d18 dumps_kwargs parameter in docs 2018-12-18 23:16:14 +05:30
kasun Herath 12ad06b7ac docs change 2018-12-17 23:17:13 +05:30
kasun Herath 8f1507a4a5 dumps_kwargs 2018-12-17 23:14:06 +05:30