RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,602 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

71 Commits

Author SHA1 Message Date
Adrián Chaves e1699479f6
Fix typo: download middleware → downloader middleware 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2023-01-30 11:54:31 +01:00
Andrey Rakhmatullin ccd1385e11
Merge branch 'master' into no-callback 2023-01-30 15:43:05 +05:00
Adrián Chaves f03b47db05 Make NO_CALLBACK a callable 2023-01-27 17:35:32 +01:00
Jalil SA a5c1ef8276 sort imports with isort 2023-01-25 14:25:15 -06:00
Adrián Chaves 1f3e42897a _NoCallback → NoCallbackType 2023-01-25 18:30:29 +01:00
Adrián Chaves 9d07be61b4 Merge remote-tracking branch 'Gallaecio/no-callback' into no-callback 2023-01-25 17:49:36 +01:00
Adrián Chaves c883a13006 Make the _set_xback condition more readable 2023-01-25 17:43:10 +01:00
Adrián Chaves 9272c4af0c Merge remote-tracking branch 'scrapy/master' into no-callback 2023-01-25 17:41:41 +01:00
Adrián Chaves 818d69fa00
Fix typo: it → is 
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
 2023-01-20 12:38:07 +01:00
Adrián Chaves 4242ae405d Restore Python 3.7 support 2023-01-19 20:37:24 +01:00
Adrián Chaves 5c1559f60e Address typing issues 2023-01-19 20:30:22 +01:00
Adrián Chaves 50500a6b28 Implement a NO_CALLBACK value for Request.callback 2023-01-19 17:14:18 +01:00
Emmanuel Rondan e211ec0aa2 adding black formatter to all the code 2022-11-29 11:30:46 -03:00
Mark Mayo f9a29f03d9
Address some issues reported by Pylint (#5677) 
Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com>
Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>
 2022-11-27 11:00:13 +01:00
Andrey Rahmatullin c3b1700774
Restore lost typing 2022-11-23 12:17:30 +05:00
Adrián Chaves e7d3fad00e
Merge branch 'master' into master 2021-07-14 11:24:56 +02:00
Eugenio Lacuesta ce6447731a
Replace return type 2021-06-07 13:25:04 -03:00
Eugenio Lacuesta c92e8ad0a5
Merge branch 'master' into request-types 2021-06-01 11:01:23 -03:00
Gustavo Bordin 5bacc4822e changing dunder-str to dunder-repr 2021-05-16 18:40:30 -03:00
Eugenio Lacuesta cec36a9284
Refactor request to/from dict (#5130) 2021-05-10 18:00:08 +02:00
Eugenio Lacuesta 19c7415aae
Request type hints 2021-05-01 16:34:46 -03:00
Adrián Chaves e40788153c Do not consider about: URLs invalid 2020-10-06 19:13:29 +02:00
Ammar Najjar 58ca8bbf6d Use f-strings (#4307) 2020-08-26 07:11:05 +02:00
Eugenio Lacuesta efb6f13deb
Remove assertions from production code (#4440) 2020-04-23 12:40:10 +02:00
Aditi Dutta 010edfe85c [Docs] mention curl2scrapy in Request.from_curl 2020-03-25 14:38:22 -04:00
Mikhail Korobov f80c7776ae
Merge pull request #4008 from elacuesta/docs_request_errback 
Request: remove restriction about errback without callback
 2020-01-23 23:12:44 +05:00
Eugenio Lacuesta 62778cf23f
Request: remove restriction about errback without callback 2019-12-04 08:53:38 -03:00
Eugenio Lacuesta 05785c1c17
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-19 11:01:34 -03:00
Eugenio Lacuesta 54a786b102
Remove six imports 2019-11-03 00:58:47 -03:00
Eugenio Lacuesta ce8e515fa8
Remove six type wrappers 2019-11-03 00:36:25 -03:00
Victor Torres f701f5b0db fix #2552 by improving request schema check on its initialization 2019-10-22 10:48:02 -03:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
Eugenio Lacuesta 6760bca74b Rename Request.kwargs to Request.cb_kwargs 2019-03-15 22:42:29 +00:00
Eugenio Lacuesta 770a501fb3 Test request kwargs (copy, serialization) 2019-01-09 10:40:03 -03:00
Eugenio Lacuesta 69a1ee79aa Copy request.kwargs 2019-01-03 17:38:29 -03:00
Eugenio Lacuesta 50a0d87d1e Passing keyword arguments to callbacks 2019-01-03 17:20:08 -03:00
Eugenio Lacuesta 6a38fc39f8 Include flags when copying requests 2018-07-19 11:56:23 -03:00
Valdir Stumm Junior bb0bd691d9 Improve error message when callback is not callable 2017-07-24 11:12:09 -03:00
Valdir Stumm Junior 60727dedf6 verify if Request callback is callable 2017-05-31 15:00:38 -03:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Paul Tremberth d42a98d3b5 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-12 00:33:25 +02:00
Mikhail Korobov e853d9e910 partial port of Request and Response 2015-07-25 13:08:44 +02:00
Daniel Graña 4161a8c82c lint scrapy/http/request/*.py 2014-08-01 02:38:33 -03:00
Daniel Graña ccae49813a PY3: port scrapy/utils/httpobj.py 2014-08-01 02:38:29 -03:00
Pablo Hoffman 19ff9ac4f9 url/body attributes of Request/Response objects are now immutable 2013-08-23 12:43:22 -03:00
Daniel Graña acf69dac44 ajax crawling wasn't expanding for unicode urls 2012-02-07 14:44:06 -02:00
Pablo Hoffman 37ad4f8791 added support for ajax crawleable urls 2011-10-28 16:33:12 -02:00
Pablo Hoffman 75284015b5 persistent scheduler: use pickle (instead of marshal) as the default serialization format, to support serializing more objects out of the box. also removed __slots__ from Request/Response objects to make them serializable by default. 2011-09-01 14:27:29 -03:00
Pablo Hoffman 549725215e Initial support for a persistent scheduler, to support pausing and resuming 
crawls.

* requests are serialized (using marshal by default) and stored on disk, using
  one queue per priority
* request priorities must be integers now
* breadh-first and depth-first crawling orders can now be configured
  through a new DEPTH_PRIORITY setting (see doc). backwards compatilibty with
  SCHEDULER_ORDER was kept.
* requests that can't be serialized (for example, non serializable callbacks)
  are always kept in memory queues
* adapted crawl spider to work with persitent scheduler
 2011-08-02 11:57:55 -03:00
Pablo Hoffman c4a607fc78 Raise ValueError if url has no scheme in Request constructor 2011-06-10 18:22:36 -03:00