RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,222 Commits 42 Branches 132 Tags 165 MiB
Commit Graph

6222 Commits

Author SHA1 Message Date
Peter Bronez 475766c73a Converted sel.xpath() calls to response.xpath() in Extracting the data 2015-03-26 15:34:30 -04:00
Elias Dorneles 4dcecc98f9 moved example data to a better place 2015-03-26 15:45:17 -03:00
Elias Dorneles 7402e27230 fix community link 2015-03-26 15:35:31 -03:00
Elias Dorneles 729861c864 fixing indentation 2015-03-26 15:31:42 -03:00
Elias Dorneles 13d0ecde77 addressing more review comments, to avoid ambiguity on desired reading flow 2015-03-26 15:26:16 -03:00
Elias Dorneles 76e3bf1250 addressing comments from the review plus further editing 2015-03-26 14:26:20 -03:00
Elias Dorneles 8f4a268f37 added bit about async requests, improved phrasing 2015-03-26 12:14:56 -03:00
Elias Dorneles 32423d4a33 some improvements to overview page 2015-03-25 19:27:52 -03:00
Mikhail Korobov e3e60da6c2 Merge pull request #1105 from ramiro/patch-1 
Add missing callback arg in jobs topic example.
 2015-03-26 03:19:39 +05:00
Ramiro Morales 933dbc6be6 Oops 2015-03-25 18:33:17 -03:00
Ramiro Morales ca2575001e Add missing callback arg in jobs topic example. 2015-03-25 18:32:20 -03:00
drack3800 6afd1a78bd style fix 2015-03-25 20:43:07 +03:00
Daniel Graña ad36de4e62 Merge pull request #1101 from nyov/nyov/tls-sni 
handle TLS SNI
 2015-03-24 18:21:10 -03:00
nyov aaeb837db4 handle TLS SNI if we have twisted>=14.0 
(closes #981, #1101)
 2015-03-24 10:52:29 +00:00
Mikhail Korobov 5ac91e4883 DOC remove Dynamic Creation of Item Classes section 
It was a hack, and dicts-as-items cover most use cases.

Dicts don't allow to attach metadata to fields,
but e.g. adding "_meta" key and removing it in a custom serializer
is no worse than creating classes dynamically.
 2015-03-23 18:11:35 +05:00
drack3800 549882590b Fixed bug with no specified Content-Length header by ScrapyHTTPClientFactory for POST request with no given body 2015-03-22 19:31:29 +03:00
drack3800 deb5bb530c Fixed bug with no specified Content-Length header by ScrapyHTTPClientFactory for POST request with no given body 2015-03-22 19:25:08 +03:00
drack3800 1b6d5a011a Added webclient test for checking Content-Length header in response for POST request with no given body 2015-03-21 04:02:51 +03:00
Julia Medina cda3922507 Add Response.urljoin() helper 2015-03-19 19:07:52 -03:00
Pablo Hoffman c81eefaf81 fix doc links 2015-03-19 17:42:48 -03:00
Elias Dorneles d7cb2b9a91 making commented code indentation consistent 2015-03-19 15:41:43 -03:00
Mikhail Korobov 8ac397670f DOC move .. module: declaration to a proper place 2015-03-19 21:41:36 +05:00
Julia Medina 67a85c77f2 Merge branch 'coder46-master' from pull request #647 2015-03-18 23:55:42 -03:00
Faisal Anees 643984e1b4 Updated architecture.rst 
Added http://krondo.com/blog/?page_id=1327 as a resource
 2015-03-18 23:55:22 -03:00
Julia Medina ff64584876 Merge branch 'ananana-selectorlist-extract-first' from pull request #624 2015-03-18 21:29:49 -03:00
Mikhail Korobov f16a33f34e DOC change structure of spider docs: 
* start with scrapy.Spider, then mention spider arguments,
  then describe generic spiders;
* change wording regarding start_urls/start_requests;
* show an example of start_requests vs start_urls;
* show an example of dicts as items;
* as defining Item is an optional step now, docs for Items are
  moved below Spider docs.
 2015-03-19 05:25:15 +05:00
Mikhail Korobov 817dbc6cbd DOC mention dicts in documentation; explain better what are Items for 2015-03-19 05:16:14 +05:00
Julia Medina 959aaad205 Document `re_first` 2015-03-18 21:11:08 -03:00
Julia Medina 0dade7315b Use generator sintax in re_first 2015-03-18 21:11:08 -03:00
Mateusz Golewski f92bc09bf4 Add re_first() to SelectorList and iflatten() to utils.python 2015-03-18 21:11:08 -03:00
Mateusz Golewski 127c6c694a Fix extract_first() docs 2015-03-18 21:11:08 -03:00
Mateusz Golewski 012211accd Add docs for extract_first() 2015-03-18 21:11:08 -03:00
Mateusz Golewski 2742b4d8c2 Add tests to extract_first() 2015-03-18 21:11:08 -03:00
Mateusz Golewski bd126be356 Optimize extract_first() 2015-03-18 21:11:08 -03:00
Mateusz Golewski 6c7bd54fc3 Add extract_first() method to SelectorList 2015-03-18 21:11:08 -03:00
Daniel Graña 12eedd90d1 fix truncated 0.24.5 release notes. closes #1084 2015-03-18 20:46:28 -03:00
Julia Medina e17f97e637 Merge branch 'dufferzafar-fix-linkcheck-606' from pull request #1041 2015-03-18 20:05:50 -03:00
Julia Medina 4fb818a250 Run linkfix over current docs 2015-03-18 20:04:14 -03:00
Julia Medina ff987fb5a5 Add linkfix rule to docs Makefile 2015-03-18 20:00:03 -03:00
Shadab Zafar c05f5f175e Added linkfix script to docs/utils 
https://github.com/scrapy/scrapy/pull/1041#issuecomment-78143576
 2015-03-18 19:41:26 -03:00
Shadab Zafar 5a58d64131 Fix some redirection links in documentation 
Fixes #606
 2015-03-18 19:41:26 -03:00
Mikhail Korobov 4c11201de6 Merge pull request #1082 from iKevinY/readme-badges 
Use Shields.io SVGs for README badges
 2015-03-18 23:33:10 +05:00
Nicolás Alejandro Ramírez Quiros ee82fe0e24 Merge pull request #1016 from SudShekhar/jsonProcessor 
[MRG+1] Added JmesSelect
 2015-03-18 08:11:25 -03:00
Kevin Yap 776616bdec Use Shields.io SVGs for README badges 
- Use SVGs for badges (more friendly to retina displays).
- Add alt text to PyPI version and build status badges.
 2015-03-17 22:26:06 -07:00
Pablo Hoffman 5846d6154c emphasize web crawling over screen scraping on scrapy description. closes #586 2015-03-18 00:24:16 -03:00
Mikhail Korobov 39635e5f55 Allow spiders to return dicts. See GH-1064. 2015-03-18 07:26:56 +05:00
Mikhail Korobov da90449edf typo fix in scrapy.contrib.pipeline.media 2015-03-18 07:24:15 +05:00
Mikhail Korobov d14c972ed9 Merge pull request #1079 from kmike/feed-iterate-spider-output 
CSVFeedSpider cleanup: use iterate_spider_output
 2015-03-18 04:44:50 +05:00
Pablo Hoffman 3f15b6df8b Merge pull request #1078 from kmike/iter-spider-output 
remove unnecessary check from scrapy.utils.spider.iter_spider_output
 2015-03-17 16:36:43 -03:00
Mikhail Korobov 6974902323 CSVFeedSpider cleanup: use iterate_spider_output 
A similar fix was made for XMLFeedSpider in 95fde0a498
 2015-03-17 23:31:29 +05:00