RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,710 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

10710 Commits

Author SHA1 Message Date
pawelmhm 7d88209543 [image & file pipeline] loading setting for user classes 
if user has some custom subclass of Image pipeline and no setting for
this pipeline, he should get default settings defined for Image Pipeline.

Fixes #2198
 2016-09-15 09:39:16 +02:00
Elias Dorneles a9a96bed8f updated tutorial as per review comments 2016-09-14 18:09:39 -03:00
Valdir Stumm Junior bc67cd9edd fix indentation issue 2016-09-14 12:39:29 -03:00
Paul Tremberth 498a3725d1 Add flush() method to StreamLogger 
Fixes GH-2125
 2016-09-14 12:19:50 +02:00
Valdir Stumm Junior 10f8c52f5d changed tutorial examples from dmoz to quotes.toscrape.com 2016-09-13 14:05:26 -03:00
Elias Dorneles 129421c7e3 Merge pull request #1503 from demelziraptor/amazon-json-response 
[MRG+1] interpreting json-amazonui-streaming as TextResponse
 2016-09-12 13:21:16 -03:00
Paul Tremberth fbb5559299 Add tests for crawl command non-default cases 2016-09-12 13:35:14 +02:00
Andrew Hlynskyi 80260824c6 Fix completion in `scrapy shell` for new imports 2016-09-12 01:13:23 +03:00
Pengyu CHEN 114437c169 added: Doc for `scrapy.http.TextResponse.urljoin` 2016-09-10 10:38:36 +12:00
Joakim Uddholm 743a0aa422 Two fixes for when using the parse command and the '-r' flag (rules). 
1. Use default "parse" as callback when the matching rule has no callback.
2. Log error and return when no rule matches the parsed url.
 2016-09-08 21:52:14 +02:00
Matti Remes 0ef570f6f0 Update exceptions.rst 
Added the missing dot. (+1 squashed commit)
Squashed commits:
[2198972] Update exceptions.rst

There are namely no constructors in classes in Python but an ``__init__`` method instead.
 2016-09-08 19:38:17 +05:00
Paul Tremberth ec4ab126b6 Merge pull request #2220 from scrapy/comment-typo-fix 
typo fix in HttpProxyMiddleware
 2016-09-07 10:15:12 +02:00
Mikhail Korobov 960b1bc8f0 typo fix in HttpProxyMiddleware 2016-09-07 04:54:32 +05:00
Paul Tremberth 9e6a72cc4b Merge pull request #2217 from stummjr/add-analytics-to-docs 
[MRG+1] Add Segment Analytics to Documentation
 2016-09-05 15:59:58 +02:00
Valdir Stumm Junior 9cea6f0730 Add Segment Analytics to Documentation 2016-09-02 14:51:07 -03:00
Paul Tremberth b188f61b95 Update release notes for upcoming 1.2.0 version 2016-09-01 17:38:38 +02:00
Paul Tremberth 58cd7bf895 Remove "precise" test env from Travis-CI config 2016-09-01 11:17:53 +02:00
Paul Tremberth 2b2bfcea88 Add "jessie" build to Travis-CI config 2016-09-01 10:20:49 +02:00
Paul Tremberth 22e870e955 Add Debian Jessie test env 2016-09-01 10:19:49 +02:00
Paul Tremberth eedb6ce774 Merge pull request #2190 from stummjr/fix-docs 
[MRG+1] Fix RANDOMIZE_DOWNLOAD_DELAY description in the docs
 2016-08-31 11:51:47 +02:00
Elias Dorneles 1e95bf59ef Merge pull request #2197 from thomdixon/improve-response-documentation 
[MRG+1] Correct documentation about Response parameters
 2016-08-29 11:08:30 -03:00
Mikhail Korobov 495d322691 DOC move Data Flow below the picture; add links to components 2016-08-26 20:16:22 +05:00
Thom Dixon f68dc3026d Fix indentation 2016-08-24 09:11:27 -07:00
Thom Dixon 633abfbea1 Correct documentation about Response parameters 
This fixes issue #2196
 2016-08-24 08:47:52 -07:00
Valdir Stumm Junior d61650d843 fix RANDOMIZE_DOWNLOAD_DELAY description in the docs 2016-08-19 18:24:32 -03:00
Paul Tremberth cacd038b10 Merge pull request #2188 from scrapy/release-notes-1.1.2-master 
Add release notes for 1.1.2 version
 2016-08-19 17:04:28 +02:00
Paul Tremberth f18c3e5ce5 Add release notes for 1.1.2 version 2016-08-19 17:01:57 +02:00
Paul Tremberth 9de6f1ca75 Merge pull request #1905 from rootAvish/duplication-fix 
[MRG+1] Modified read failure recovery in utils/gz.py to read only the last f.extrasize bytes of f.extrabuf[ ]
 2016-08-17 14:51:30 +02:00
Mikhail Korobov 241bd00e76 Merge pull request #2168 from advarisk/w3lib-canonicalize-url 
[MRG+1] Use w3lib.url.canonicalize_url() from w3lib 1.15.0
 2016-08-16 20:59:17 +06:00
Ashish Kulkarni bb3b806467 Use w3lib.url.canonicalize_url() from w3lib 1.15.0 
Also remove code/imports which are now unused due to this change.

fixes #2157
 2016-08-16 17:42:16 +05:30
Paul Tremberth 9a734e6759 Merge pull request #2058 from dalleng/serialize_set 
[MRG+1] Add set serialization to ScrapyJSONEncoder
 2016-08-12 18:28:34 +02:00
rootavish d9437fd3d9 Modifying existing gzip read failure recovery mechanism to patch read for broken archives 2016-08-11 18:21:42 +05:30
Paul Tremberth 1ec210068f Merge pull request #2169 from Tethik/parse_command_callback_typo 
[MRG+1] Typo fix for error in parse command
 2016-08-11 11:52:59 +02:00
Joakim Uddholm 625c69fdc7 Fixed typo in error message when selecting a callback method for the parse command. 2016-08-08 14:32:53 +02:00
Mikhail Korobov 414857a593 Merge pull request #2140 from jesuslosada/images-expires 
[MRG+1] Fix IMAGES_EXPIRES default value
 2016-08-05 21:52:27 -04:00
Mikhail Korobov fa78849e33 Merge pull request #2165 from loreguerra/master 
[MRG+1] Updated architecture graph for organization/clarity
 2016-08-05 21:46:28 -04:00
Lorena 7d432872bf text updates to match graphic 2016-08-04 11:01:14 -07:00
Lorena 04f93e096c updated graph for organization/clarity 2016-08-04 10:04:47 -07:00
Paul Tremberth 27d4cea6a5 Merge pull request #2161 from redapple/release-notes-1.1.1-master 
Release notes for 1.1.1
 2016-08-01 20:50:14 +02:00
Paul Tremberth 5b1d98b8c8 Update 1.1.1 release date 2016-08-01 20:21:12 +02:00
Paul Tremberth 928e93f8f3 Update notes with latest 1.1 commits 2016-08-01 20:21:12 +02:00
Paul Tremberth e1d118d5ca Update release notes for upcoming 1.1.1 release 2016-08-01 20:21:12 +02:00
Paul Tremberth 2a0a96aef0 Merge pull request #2160 from stummjr/patch-1 
[MRG+1] Remove README download count badge
 2016-08-01 17:57:57 +02:00
Valdir Stumm Jr 63876fc690 Remove download stats badge 2016-08-01 12:16:50 -03:00
Elias Dorneles 4eec05351f remove bumpversion prerelease configuration 
I propose to remove the prerelease configuration from bumpversion,
because I think its behavior is just too confusing.

The rational for this is that making the release procedure predictable
is more important than facilitating making pre-releases, which are sort
of the exception in the workflow.

The current configuration makes most common cases confusing:

* bug fix releases require you have to remember to use `--serialize "{major}.{minor}.{patch}"`
* to start a pre-release cycle, you actually use `minor` or `patch`
* to do the actual minor or patch release, you use `prerel`

Also, `prerel` breaks if you run it on a branch with a final release,
because it can't parse the prerelease information.

Therefore, I propose keeping the bumpversion defaults, and do the
prereleases (dev1, dev2, rc1, etc) manually (with `--new-version`),
which makes for a more predictable and intuitive behavior.

* `bumpversion minor` and `bumpversion patch` will work as expected
* pre-releases will be manually handled, but this seems a small overhead
  than remembering the details I mention above.

If you're happy with this, I'll also update [the wiki][1] with new
instructions.

[1]: https://github.com/scrapy/scrapy/wiki/Scrapy-release-procedure
 2016-08-01 11:37:51 -03:00
Mikhail Korobov 2c9a38d1f5 Merge pull request #2153 from Digenis/Selector_bad_args 
[MRG+1] Selector should not receive both response and text
 2016-07-31 21:28:38 -04:00
Νικόλαος-Διγενής Καραγιάννης 643dbeffcf Selector should not receive both response and text 2016-07-30 10:35:16 +03:00
Elias Dorneles 34e7dadf38 Merge pull request #1610 from darshanime/scheduler_debug 
[MGR+1] Change, document `LOG_UNSERIALIZABLE_REQUESTS`
 2016-07-29 10:12:52 -03:00
darshanime d8e62e660d update log demo print 2016-07-26 20:46:12 +05:30
Paul Tremberth fe68a45c82 Merge pull request #2138 from jesuslosada/master 
[MRG+1] Apply the FILES_STORE_S3_ACL setting in ImagesPipeline
 2016-07-26 12:50:25 +02:00