RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,858 Commits 49 Branches 132 Tags 134 MiB
Commit Graph

6858 Commits

Author SHA1 Message Date
Eugenio Lacuesta 770a501fb3 Test request kwargs (copy, serialization) 2019-01-09 10:40:03 -03:00
Eugenio Lacuesta a67f1ce512 Serialize Request kwargs 2019-01-03 18:19:12 -03:00
Eugenio Lacuesta 69a1ee79aa Copy request.kwargs 2019-01-03 17:38:29 -03:00
Eugenio Lacuesta a2b509a422 Pass callback kwargs with response.follow 2019-01-03 17:38:06 -03:00
Eugenio Lacuesta 50a0d87d1e Passing keyword arguments to callbacks 2019-01-03 17:20:08 -03:00
Mikhail Korobov 094dde6fdb
Merge pull request #3512 from victor-torres/sitemap_filter 
[MRG+1] Add sitemap_filter function to SitemapSpider class
 2018-12-28 20:11:46 +05:00
Victor Torres 5a824c906c using shorter import version and moving datetime import to the beginning of the code snippet 2018-12-27 18:34:41 -03:00
Victor Torres bfbcf52e9d fix SitemapSpider import 2018-12-27 18:12:31 -03:00
Victor Torres b68308779a improving docs 2018-12-27 17:37:59 -03:00
Victor Torres e1597f7c42 improve readability 2018-12-27 11:40:14 -03:00
Victor Torres fe283bcd05 add test case for sitemap filter with alternate links 2018-12-27 11:40:14 -03:00
Victor Torres 10f46bca54 documenting sitemap entries as suggested by @kmike 2018-12-27 11:40:14 -03:00
Victor Torres 5e7ecf9dc1 add tests for sitemapindex 2018-12-27 11:40:14 -03:00
Victor Torres 657f0663b3 rename param from urls to entries 2018-12-27 11:40:14 -03:00
Victor Torres d7d5917ff1 add tests for the sitemap_filter method in the SitemapSpider class 2018-12-27 11:40:14 -03:00
Victor Torres 672385a371 using a method definition instead of a None attribute 2018-12-27 11:40:14 -03:00
Victor Torres a5e1b7bb47 add sitemap_filter attribute to SitemapSpider class 
it makes it possible to filter sitemap urls by any available attribute

for example, you can filter urls with lastmod greater than a given datetime

it can be helpful when the url loc itself does not aggregate that information
 2018-12-27 11:40:14 -03:00
Daniel Graña 93cf38354d
Merge pull request #3394 from hcoura/dh-lazyloading-optional 
Make lazy loading Download Handlers optional
 2018-12-26 11:16:19 -03:00
Daniel Graña f6ce716372
Merge pull request #3476 from elacuesta/deque_appendleft 
[MRG+1] Use collections.deque instead of list to store MiddlewareManager's methods
 2018-12-26 11:13:52 -03:00
Daniel Graña 76433feb4f
Merge pull request #3548 from scrapy/fix-aws-settings-docs 
fix docs for AWS_... settings. A follow-up to GH-2609.
 2018-12-26 11:10:57 -03:00
Daniel Graña 5cf26c9985
Merge pull request #3547 from scrapy/setup.py-3.7 
declare Python 3.7 support in setup.py
 2018-12-26 11:09:07 -03:00
Daniel Graña d473be2040
Merge pull request #3519 from fpghost/master 
[MRG+1] the strip() isnt needed after base64 encoding user pass
 2018-12-26 11:08:49 -03:00
Daniel Graña 5d02aab9f9
Merge pull request #3496 from frederik-elwert/patch-1 
[MRG+1] Add documentation to `scrapy shell` command.
 2018-12-26 11:04:30 -03:00
Daniel Graña 6a0ea0cf26
Merge pull request #3415 from scrapy/telnet-auth 
[MRG+1] Telnet console authentication
 2018-12-26 10:58:44 -03:00
Mikhail Korobov 71e47629b1 DOC fix docs for AWS_... settings. A follow-up to GH-2609. 2018-12-26 16:35:05 +05:00
Mikhail Korobov 4306886ac6
Merge pull request #3527 from hsiaoyi0504/patch-1 
unify the quote style
 2018-12-26 14:45:06 +05:00
Mikhail Korobov cdd04dfb1d declare Python 3.7 support in setup.py 2018-12-26 13:13:49 +05:00
Mikhail Korobov 7c26701012 DOC warn about telnet console being insecure 2018-12-26 01:33:58 +05:00
Mikhail Korobov dbfabf02e8 Merge branch 'master' into telnet-auth 2018-12-26 01:04:34 +05:00
Daniel Graña 23bffff567
Merge pull request #3526 from lucywang000/update-travis 
remove "sudo: false" now that travis no longer supports it
 2018-12-20 19:54:34 -03:00
Daniel Graña d4fd1236b7
Merge pull request #3538 from scrapy/boto-import-error-under-jessie 
Fix boto import error under Jessie testing environment
 2018-12-20 19:49:55 -03:00
Daniel Graña 8ed6beb7f9 Needs to be installed within tox env 2018-12-20 19:39:29 -03:00
Daniel Graña f6dfc5f3dd Fix boto import error under Jessie testing environment 2018-12-20 19:23:23 -03:00
Daniel Graña 665c04b0c1
Merge pull request #3518 from Gallaecio/scrapy-project-doc 
Document the SCRAPY_PROJECT environment variable
 2018-12-12 11:56:47 -03:00
hsiao yi cd9d8e28cd
unify the quote style 2018-12-11 19:21:07 +08:00
Lucy Wang 4d48759978 remove "sudo: false" now that travis no longer supports it 
https://changelog.travis-ci.com/deprecation-container-based-linux-build-environment-82037
 2018-12-10 14:44:17 +08:00
Mikhail Korobov e766bde3a3
Merge pull request #3517 from Gallaecio/install-troubleshooting 
[MRG+1] Add a troubleshooting section to the installation instructions
 2018-12-06 18:07:43 +05:00
fpghost d7c8eee2fc the strip() isnt needed 2018-12-04 10:57:51 +01:00
Adrián Chaves 62f3349c1a Document the SCRAPY_PROJECT environment variable 
Fixes #1109
 2018-12-03 17:14:10 +01:00
Adrián Chaves 274b65dff4 Add a troubleshooting section to the installation instructions 
Its initial content covers the workaround for #2473.
 2018-12-03 16:36:05 +01:00
Konstantin Lopuhin 886513c375
Merge pull request #3495 from toddrme2178/patch-1 
[MRG+1] Include additional files in sdists
 2018-11-20 11:10:32 +03:00
Frederik Elwert 127bf499f1
Add documentation to `scrapy shell` command. 
The special syntax required for local files (`./file.html`) is not documented as part of the `scrapy shell --help` output. This patch adds that.
 2018-11-16 22:15:03 +01:00
Todd 491929c212
Include additional files in sdists 
In particular this includes files needed for running the tests, as well as the changelog.
 2018-11-16 13:38:19 -05:00
Mikhail Korobov dc65e7527d
Merge pull request #3468 from ilhaoni/feature/update-docs-intro-tutorial 
[MRG+1] Update Scrapy Tutorial docs
 2018-11-06 03:33:09 +05:00
Immanuella Lim 6c98010f11 Remove 'Dive into Python3' reference 2018-11-04 16:04:45 +08:00
Eugenio Lacuesta f97e3e90f2 Use collections.deque instead of list to store methods 2018-10-29 12:40:20 -03:00
Immanuella Lim c9b5bd6ad7 Remove ad link Dive Into Python3 from tutorial docs 2018-10-18 02:22:32 +08:00
Daniel Graña 44f8e28b3c Fix headings' underlines 2018-10-16 19:53:20 -03:00
Daniel Graña 06f2db7fd1
Merge pull request #3445 from jfflisikowski/3097-item-urls-not-defined-in-example 
[MRG+1] Updating debug example file (#3097)
 2018-10-16 19:51:32 -03:00
Daniel Graña 553b3a1c12
Merge pull request #3465 from hcoura/telnet-auth 
Add Telnet console authentication docs
 2018-10-16 15:08:34 -03:00