git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-22 17:31:03 +01:00
Code Issues Packages Projects Releases Wiki Activity

Update spider.py

Browse Source 
修复“//” 开头的url报错   ValueError('Missing scheme in request url: %s' % self._url)
This commit is contained in:
Seven2Nine
2020-07-08 11:06:52 +08:00
committed by GitHub
parent d75a993121
commit 142367a5f0
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
backend/template/scrapy/config_spider/spiders/spider.py
View File

@@ -9,7 +9,7 @@ def get_real_url(response, url):
return url
elif re.search(r'^\/\/', url):
u = urlparse(response.url)
return u.scheme + url
return u.scheme + ":" + url
return urljoin(response.url, url)
class ConfigSpider(scrapy.Spider):