RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,393 Commits 49 Branches 132 Tags 155 MiB
Commit Graph

10393 Commits

Author SHA1 Message Date
Joaquin Garmendia Cabrera e1f8b55ba0
Improve syntax for readability 2018-12-28 16:53:12 -05:00
Mikhail Korobov 094dde6fdb
Merge pull request #3512 from victor-torres/sitemap_filter 
[MRG+1] Add sitemap_filter function to SitemapSpider class
 2018-12-28 20:11:46 +05:00
Victor Torres 5a824c906c using shorter import version and moving datetime import to the beginning of the code snippet 2018-12-27 18:34:41 -03:00
Victor Torres bfbcf52e9d fix SitemapSpider import 2018-12-27 18:12:31 -03:00
Victor Torres b68308779a improving docs 2018-12-27 17:37:59 -03:00
Victor Torres e1597f7c42 improve readability 2018-12-27 11:40:14 -03:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 10f46bca54 documenting sitemap entries as suggested by @kmike 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Victor Torres 672385a371 using a method definition instead of a None attribute 2018-12-27 11:40:14 -03:00
Victor Torres a5e1b7bb47 add sitemap_filter attribute to SitemapSpider class 
it makes it possible to filter sitemap urls by any available attribute

for example, you can filter urls with lastmod greater than a given datetime

it can be helpful when the url loc itself does not aggregate that information
 2018-12-27 11:40:14 -03:00
Daniel Graña 93cf38354d
Merge pull request #3394 from hcoura/dh-lazyloading-optional 
Make lazy loading Download Handlers optional
 2018-12-26 11:16:19 -03:00
Daniel Graña f6ce716372
Merge pull request #3476 from elacuesta/deque_appendleft 
[MRG+1] Use collections.deque instead of list to store MiddlewareManager's methods
 2018-12-26 11:13:52 -03:00
Daniel Graña 76433feb4f
Merge pull request #3548 from scrapy/fix-aws-settings-docs 
fix docs for AWS_... settings. A follow-up to GH-2609.
 2018-12-26 11:10:57 -03:00
Daniel Graña 5cf26c9985
Merge pull request #3547 from scrapy/setup.py-3.7 
declare Python 3.7 support in setup.py
 2018-12-26 11:09:07 -03:00
Daniel Graña d473be2040
Merge pull request #3519 from fpghost/master 
[MRG+1] the strip() isnt needed after base64 encoding user pass
 2018-12-26 11:08:49 -03:00
Daniel Graña 5d02aab9f9
Merge pull request #3496 from frederik-elwert/patch-1 
[MRG+1] Add documentation to `scrapy shell` command.
 2018-12-26 11:04:30 -03:00
Daniel Graña 6a0ea0cf26
Merge pull request #3415 from scrapy/telnet-auth 
[MRG+1] Telnet console authentication
 2018-12-26 10:58:44 -03:00
Mikhail Korobov 71e47629b1 DOC fix docs for AWS_... settings. A follow-up to GH-2609. 2018-12-26 16:35:05 +05:00
Mikhail Korobov 4306886ac6
Merge pull request #3527 from hsiaoyi0504/patch-1 
unify the quote style
 2018-12-26 14:45:06 +05:00
Mikhail Korobov cdd04dfb1d declare Python 3.7 support in setup.py 2018-12-26 13:13:49 +05:00
Mikhail Korobov 7c26701012 DOC warn about telnet console being insecure 2018-12-26 01:33:58 +05:00
Mikhail Korobov dbfabf02e8 Merge branch 'master' into telnet-auth 2018-12-26 01:04:34 +05:00
Joaquin Garmendia Cabrera f85c915872
Update item-pipeline example 2018-12-23 00:26:58 -05:00
Daniel Graña 23bffff567
Merge pull request #3526 from lucywang000/update-travis 
remove "sudo: false" now that travis no longer supports it
 2018-12-20 19:54:34 -03:00
Daniel Graña d4fd1236b7
Merge pull request #3538 from scrapy/boto-import-error-under-jessie 
Fix boto import error under Jessie testing environment
 2018-12-20 19:49:55 -03:00
Daniel Graña 8ed6beb7f9 Needs to be installed within tox env 2018-12-20 19:39:29 -03:00
Daniel Graña f6dfc5f3dd Fix boto import error under Jessie testing environment 2018-12-20 19:23:23 -03:00
kasun Herath 24acc50d18 dumps_kwargs parameter in docs 2018-12-18 23:16:14 +05:30
kasun Herath 12ad06b7ac docs change 2018-12-17 23:17:13 +05:30
kasun Herath 8f1507a4a5 dumps_kwargs 2018-12-17 23:14:06 +05:30
Daniel Graña 665c04b0c1
Merge pull request #3518 from Gallaecio/scrapy-project-doc 
Document the SCRAPY_PROJECT environment variable
 2018-12-12 11:56:47 -03:00
kasun Herath 71ef321b68 sort_keys while serializing to json 2018-12-12 11:12:48 +05:30
hsiao yi cd9d8e28cd
unify the quote style 2018-12-11 19:21:07 +08:00
kasun Herath ecda69130e allow to send empty data values and docs changes 2018-12-10 22:34:49 +05:30
Lucy Wang 4d48759978 remove "sudo: false" now that travis no longer supports it 
https://changelog.travis-ci.com/deprecation-container-based-linux-build-environment-82037
 2018-12-10 14:44:17 +08:00
kasun Herath 3c981bf204 add documentation 2018-12-09 12:56:12 +05:30
kasun Herath c347acbff6 warning if body and data are provided 2018-12-09 11:27:09 +05:30
kasun Herath cd619c1d4f removed overriden replace method 2018-12-08 22:10:45 +05:30
Mikhail Korobov e766bde3a3
Merge pull request #3517 from Gallaecio/install-troubleshooting 
[MRG+1] Add a troubleshooting section to the installation instructions
 2018-12-06 18:07:43 +05:00
fpghost d7c8eee2fc the strip() isnt needed 2018-12-04 10:57:51 +01:00
Adrián Chaves 62f3349c1a Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2018-12-03 17:14:10 +01:00
Adrián Chaves 274b65dff4 Add a troubleshooting section to the installation instructions 
Its initial content covers the workaround for #2473.
 2018-12-03 16:36:05 +01:00
kasun Herath 1b2b8b4bf0 fix tests under py3 2018-11-27 08:57:44 +05:30
kasun Herath 1ce6662a9d Implement Request subclass for json requests 2018-11-24 20:02:00 +05:30
Vostretsov Nikita a25cf5c82f function to get unique file queues for any type of base queue 2018-11-20 16:13:09 +00:00
Konstantin Lopuhin 886513c375
Merge pull request #3495 from toddrme2178/patch-1 
[MRG+1] Include additional files in sdists
 2018-11-20 11:10:32 +03:00
Frederik Elwert 127bf499f1
Add documentation to `scrapy shell` command. 
The special syntax required for local files (`./file.html`) is not documented as part of the `scrapy shell --help` output. This patch adds that.
 2018-11-16 22:15:03 +01:00