RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 153 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Harry Moreno b828b5f8c8
fix grammar 2019-01-26 18:39:05 -05:00
kasun Herath d9aa539132 enabled sort keys only if not provided 2019-01-25 21:26:28 +05:30
Daniel Graña 71743a6546 Add release notes for v1.5.2 2019-01-22 18:56:47 -03:00
Maram Sumanth 722a30ac2b
Update test_http_request.py 2019-01-19 13:20:05 +05:30
Maram Sumanth 7dee841b8b
Update form.py 2019-01-19 13:20:01 +05:30
Júlio César Batista 8eade7d864 Testing stats and log messages from RFPDupeFilter 2019-01-18 11:39:35 -02:00
Júlio César Batista bdf12f7750 Logging the request referer when DUPEFILTER_DEBUG is active 2019-01-18 11:38:59 -02:00
Maram Sumanth a9f68acb6d
modified code 2019-01-17 23:51:09 +05:30
Maram Sumanth 6be73f06c3
Updated tests 2019-01-17 23:50:58 +05:30
Maram Sumanth 6f86c93f36
Increased test cases 2019-01-16 23:54:35 +05:30
Maram Sumanth 3e67fa8fc1
Improved for better user readability 2019-01-16 23:01:38 +05:30
Maram Sumanth 9f1f4df966
Update test_http_request.py 2019-01-16 22:59:41 +05:30
Eugenio Lacuesta bddfeaba4c Add Request.kwargs docs 2019-01-15 19:14:59 -03:00
kasun Herath f3813e376c Merge remote-tracking branch 'origin/master' into json_request 2019-01-14 23:03:21 +05:30
kasun Herath 3f914f6d8c made jsonrequest dump into private method 2019-01-14 23:03:14 +05:30
Maram Sumanth 023290dabc
Update test_http_request.py 2019-01-13 23:50:31 +05:30
Maram Sumanth 4abcdcb306
Update test_http_request.py 2019-01-13 23:22:53 +05:30
Maram Sumanth 9a4bbd6d02
Update form.py 2019-01-13 23:05:58 +05:30
Maram Sumanth 1bea5d3076
Fixed error 2019-01-13 22:35:16 +05:30
Maram Sumanth b5e454809e
Included test 2019-01-13 20:12:31 +05:30
Maram Sumanth ac111088c6
duplicate keys handled 2019-01-13 20:12:29 +05:30
Eugenio Lacuesta 57e7c76977 Test callback kwargs 2019-01-09 10:40:44 -03:00
Eugenio Lacuesta 770a501fb3 Test request kwargs (copy, serialization) 2019-01-09 10:40:03 -03:00
Eugenio Lacuesta a67f1ce512 Serialize Request kwargs 2019-01-03 18:19:12 -03:00
Eugenio Lacuesta 69a1ee79aa Copy request.kwargs 2019-01-03 17:38:29 -03:00
Eugenio Lacuesta a2b509a422 Pass callback kwargs with response.follow 2019-01-03 17:38:06 -03:00
Eugenio Lacuesta 50a0d87d1e Passing keyword arguments to callbacks 2019-01-03 17:20:08 -03:00
Eugenio Lacuesta 6c78b3d5ef Deques can't be sliced, use itertools.islice instead 2019-01-03 13:15:58 -03:00
Eugenio Lacuesta 9759112a3a Merge branch 'master' into process_spider_exception_generator 2019-01-03 11:34:07 -03:00
Joaquin Garmendia Cabrera e1f8b55ba0
Improve syntax for readability 2018-12-28 16:53:12 -05:00
Mikhail Korobov 094dde6fdb
Merge pull request #3512 from victor-torres/sitemap_filter 
[MRG+1] Add sitemap_filter function to SitemapSpider class
 2018-12-28 20:11:46 +05:00
Victor Torres 5a824c906c using shorter import version and moving datetime import to the beginning of the code snippet 2018-12-27 18:34:41 -03:00
Victor Torres bfbcf52e9d fix SitemapSpider import 2018-12-27 18:12:31 -03:00
Victor Torres b68308779a improving docs 2018-12-27 17:37:59 -03:00
Victor Torres e1597f7c42 improve readability 2018-12-27 11:40:14 -03:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 10f46bca54 documenting sitemap entries as suggested by @kmike 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Victor Torres 672385a371 using a method definition instead of a None attribute 2018-12-27 11:40:14 -03:00
Victor Torres a5e1b7bb47 add sitemap_filter attribute to SitemapSpider class 
it makes it possible to filter sitemap urls by any available attribute

for example, you can filter urls with lastmod greater than a given datetime

it can be helpful when the url loc itself does not aggregate that information
 2018-12-27 11:40:14 -03:00
Daniel Graña 93cf38354d
Merge pull request #3394 from hcoura/dh-lazyloading-optional 
Make lazy loading Download Handlers optional
 2018-12-26 11:16:19 -03:00
Daniel Graña f6ce716372
Merge pull request #3476 from elacuesta/deque_appendleft 
[MRG+1] Use collections.deque instead of list to store MiddlewareManager's methods
 2018-12-26 11:13:52 -03:00
Daniel Graña 76433feb4f
Merge pull request #3548 from scrapy/fix-aws-settings-docs 
fix docs for AWS_... settings. A follow-up to GH-2609.
 2018-12-26 11:10:57 -03:00
Daniel Graña 5cf26c9985
Merge pull request #3547 from scrapy/setup.py-3.7 
declare Python 3.7 support in setup.py
 2018-12-26 11:09:07 -03:00
Daniel Graña d473be2040
Merge pull request #3519 from fpghost/master 
[MRG+1] the strip() isnt needed after base64 encoding user pass
 2018-12-26 11:08:49 -03:00
Daniel Graña 5d02aab9f9
Merge pull request #3496 from frederik-elwert/patch-1 
[MRG+1] Add documentation to `scrapy shell` command.
 2018-12-26 11:04:30 -03:00
Daniel Graña 6a0ea0cf26
Merge pull request #3415 from scrapy/telnet-auth 
[MRG+1] Telnet console authentication
 2018-12-26 10:58:44 -03:00
Mikhail Korobov 71e47629b1 DOC fix docs for AWS_... settings. A follow-up to GH-2609. 2018-12-26 16:35:05 +05:00