RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Allow comma-separated values in the rel tag 

Comma-separated `rel` values are often seen in the wild, because Google allows it (see https://developers.google.com/search/docs/advanced/guidelines/qualify-outbound-links).
This commit is contained in:
Michel Ace 2021-08-10 17:09:37 +02:00 committed by GitHub
parent 94baa4b272
commit 1ba0f68483
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
scrapy/utils/misc.py
View File

@ -138,7 +138,7 @@ def md5sum(file):
def rel_has_nofollow(rel):
"""Return True if link rel attribute has nofollow type"""
return rel is not None and 'nofollow' in rel.split()
return rel is not None and 'nofollow' in rel.replace(',', ' ').split()
def create_instance(objcls, settings, crawler, *args, **kwargs):