Merge pull request #49 from tikazyq/develop

Develop
This commit is contained in:
Marvin Zhang
2019-05-30 21:16:59 +08:00
committed by GitHub
55 changed files with 577 additions and 41 deletions

View File

@@ -5,14 +5,15 @@
<img src="https://img.shields.io/badge/License-BSD-blue.svg">
</a>
中文 | [English](https://github.com/tikazyq/crawlab/blob/master/README.md)
基于Celery的爬虫分布式爬虫管理平台支持多种编程语言以及多种爬虫框架.
[查看演示 Demo](http://114.67.75.98:8080)
[English Documentation](https://github.com/tikazyq/crawlab/blob/master/README.md)
[查看演示 Demo](http://114.67.75.98:8080) | [文档](https://tikazyq.github.io/crawlab)
## 要求
- Python3
- Python 3.6+
- Node.js 8.12+
- MongoDB
- Redis
@@ -177,6 +178,8 @@ Crawlab使用起来很方便也很通用可以适用于几乎任何主流
- [ ] 登录和用户管理
- [ ] 全局搜索
## 社区 & 赞助
如果您觉得Crawlab对您的日常开发或公司有帮助请加作者微信 tikazyq1 并注明"Crawlab",作者会将你拉入群。或者,您可以扫下方支付宝二维码给作者打赏去升级团队协作软件或买一杯咖啡。
<p align="center">

View File

@@ -5,14 +5,16 @@
<img src="https://img.shields.io/badge/License-BSD-blue.svg">
</a>
[中文](https://github.com/tikazyq/crawlab/blob/master/README-zh.md) | English
Celery-based web crawler admin platform for managing distributed web spiders regardless of languages and frameworks.
[Demo](http://114.67.75.98:8080)
[Demo](http://114.67.75.98:8080) | [Documentation](https://tikazyq.github.io/crawlab)
[中文文档](https://github.com/tikazyq/crawlab/blob/master/README-zh.md)
## Pre-requisite
- Python3
- Python 3.6+
- Node.js 8.12+
- MongoDB
- Redis
@@ -166,12 +168,14 @@ Crawlab is easy to use, general enough to adapt spiders in any language and any
- [ ] More spider examples
##### Frontend
- [ ] Task Stats/Analytics
- [ ] Table Filters
- [x] Task Stats/Analytics
- [x] Table Filters
- [x] Multi-Language Support (中文)
- [ ] Login & User Management
- [ ] General Search
## Community & Sponsorship
If you feel Crawlab could benefit your daily work or your company, please add the author's Wechat account noting "Crawlab" to enter the discussion group. Or you scan the Alipay QR code below to give us a reward to upgrade our teamwork software or buy a coffee.
<p align="center">

View File

@@ -397,7 +397,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"App","level":"1.4.2","depth":2,"next":{"title":"Examples","level":"1.5","depth":1,"path":"Examples/README.md","ref":"Examples/README.md","articles":[{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},{"title":"与Puppeteer集成","level":"1.5.2","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]}]},"previous":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/App.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"App","level":"1.4.2","depth":2,"next":{"title":"Examples","level":"1.5","depth":1,"path":"Examples/README.md","ref":"Examples/README.md","articles":[{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},{"title":"与Puppeteer集成","level":"1.5.2","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]}]},"previous":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/App.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -397,7 +397,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"Celery","level":"1.4.1","depth":2,"next":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"previous":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/Celery.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"Celery","level":"1.4.1","depth":2,"next":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"previous":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/Celery.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -397,7 +397,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"架构","level":"1.4","depth":1,"next":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"previous":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"架构","level":"1.4","depth":1,"next":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"previous":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -399,7 +399,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"部署","level":"1.3.4","depth":2,"next":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"previous":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Deploy.md","mtime":"2019-03-28T12:06:24.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"部署","level":"1.3.4","depth":2,"next":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"previous":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Deploy.md","mtime":"2019-03-28T12:06:24.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -398,7 +398,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"节点","level":"1.3.1","depth":2,"next":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"previous":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Node.md","mtime":"2019-03-28T12:02:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"节点","level":"1.3.1","depth":2,"next":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"previous":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Node.md","mtime":"2019-03-28T12:02:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -403,7 +403,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"爬虫","level":"1.3.2","depth":2,"next":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"previous":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Spider.md","mtime":"2019-03-28T12:03:08.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"爬虫","level":"1.3.2","depth":2,"next":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"previous":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Spider.md","mtime":"2019-03-28T12:03:08.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -398,7 +398,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"任务","level":"1.3.3","depth":2,"next":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"previous":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Task.md","mtime":"2019-03-28T12:04:12.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"任务","level":"1.3.3","depth":2,"next":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"previous":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Task.md","mtime":"2019-03-28T12:04:12.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -397,7 +397,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"概念","level":"1.3","depth":1,"next":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"previous":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/README.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"概念","level":"1.3","depth":1,"next":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"previous":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/README.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -397,7 +397,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"Examples","level":"1.5","depth":1,"next":{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},"previous":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Examples/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"Examples","level":"1.5","depth":1,"next":{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},"previous":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Examples/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -0,0 +1,61 @@
# 功能列表
类别 | 功能名称 | 已统计 | 备注
--- | --- | --- | ---
全局 | 打开页面 | Y | _trackPageview
全局 | 切换中英文 | Y
全局 | 允许/禁止统计 | Y
节点 | 刷新 | Y
节点 | 查看 | Y
节点 | 删除 | Y
节点详情 | 保存 | Y
节点详情 | 切换节点 | Y
爬虫 | 部署所有爬虫 | Y
爬虫 | 导入爬虫 | Y
爬虫 | 添加爬虫-可配置爬虫 | Y
爬虫 | 添加爬虫-自定义爬虫 | Y
爬虫 | 刷新 | Y
爬虫 | 查看 | Y
爬虫 | 删除 | Y
爬虫 | 部署 | Y
爬虫 | 运行 | Y
爬虫 | 搜索网站 | Y
爬虫详情 | 切换爬虫 | Y
爬虫详情 | 切换标签 | Y
爬虫详情-概览 | 保存 | Y
爬虫详情-概览 | 部署 | Y
爬虫详情-概览 | 运行 | Y
爬虫详情-环境 | 添加 | Y
爬虫详情-环境 | 删除 | Y
爬虫详情-环境 | 保存 | Y
爬虫详情-配置 | 保存 | Y
爬虫详情-配置 | 预览 | Y
爬虫详情-配置 | 提取字段 | Y
爬虫详情-配置 | 运行 | Y
爬虫详情-配置 | 添加字段 | Y
爬虫详情-配置 | 更改字段 | Y
爬虫详情-配置 | 删除字段 | Y
爬虫详情-配置 | 设置详情页URL | Y
任务 | 选择节点 | Y
任务 | 选择爬虫 | Y
任务 | 点击爬虫详情 | Y
任务 | 点击节点详情 | Y
任务 | 搜索 | Y
任务 | 查看 | Y
任务 | 删除 | Y
任务详情 | 切换标签 | Y
任务详情-概览 | 点击爬虫详情 | Y
任务详情-概览 | 点击节点详情 | Y
任务详情-结果 | 下载CSV | Y
定时任务 | 添加 | Y
定时任务 | 修改 | Y
定时任务 | 删除 | Y
定时任务 | 提交 | Y
部署 | 刷新 | Y
网站 | 搜索 | Y
网站 | 选择主类别 | Y
网站 | 选择类别 | Y
网站 | 点击域名 | Y
网站 | 点击爬虫数 | Y
网站 | 点击Robots协议 | N

View File

@@ -408,7 +408,7 @@ npm install
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"安装","level":"1.2.1","depth":2,"next":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"previous":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Installation.md","mtime":"2019-03-28T11:55:48.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"安装","level":"1.2.1","depth":2,"next":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"previous":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Installation.md","mtime":"2019-03-28T11:55:48.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -440,7 +440,7 @@ npm run serve
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"运行","level":"1.2.2","depth":2,"next":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"previous":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Run.md","mtime":"2019-03-28T12:00:11.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"运行","level":"1.2.2","depth":2,"next":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"previous":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Run.md","mtime":"2019-03-28T12:00:11.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -401,7 +401,7 @@
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"快速开始","level":"1.2","depth":1,"next":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"previous":{"title":"简介","level":"1.1","depth":1,"path":"README.md","ref":"README.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/README.md","mtime":"2019-03-28T12:02:02.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"快速开始","level":"1.2","depth":1,"next":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"previous":{"title":"简介","level":"1.1","depth":1,"path":"README.md","ref":"README.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/README.md","mtime":"2019-03-28T12:02:02.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
});
</script>
</div>

View File

@@ -548,7 +548,7 @@ MONGO_DB = <span class="hljs-string">&apos;crawlab_test&apos;</span>
<script>
var gitbook = gitbook || [];
gitbook.push(function() {
gitbook.page.hasChanged({"page":{"title":"简介","level":"1.1","depth":1,"next":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"README.md","mtime":"2019-03-28T11:44:15.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":".","book":{"language":""}});
gitbook.page.hasChanged({"page":{"title":"简介","level":"1.1","depth":1,"next":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"README.md","mtime":"2019-03-28T11:44:15.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":".","book":{"language":""}});
});
</script>
</div>

View File

@@ -29,9 +29,11 @@ export default {
localStorage.setItem('useStats', value)
document.querySelector('.el-message__closeBtn').click()
if (value === 1) {
window._hmt.push(['_trackPageview', '/allow_stats'])
this.$st.sendPv('/allow_stats')
this.$st.sendEv('全局', '允许/禁止统计', 'value', 'allow')
} else {
window._hmt.push(['_trackPageview', '/disallow_stats'])
this.$st.sendPv('/disallow_stats')
this.$st.sendEv('全局', '允许/禁止统计', 'value', 'disallow')
}
}

View File

@@ -159,6 +159,7 @@ export default {
this.spiderForm.crawl_type = value
},
onSave () {
this.$st.sendEv('爬虫详情-配置', '保存')
return new Promise((resolve, reject) => {
this.saveLoading = true
this.$store.dispatch('spider/updateSpiderFields')
@@ -203,6 +204,7 @@ export default {
.finally(() => {
this.previewLoading = false
})
this.$st.sendEv('爬虫详情-配置', '预览')
})
},
onCrawl () {
@@ -215,6 +217,7 @@ export default {
.then(() => {
this.$message.success(this.$t(`Spider task has been scheduled`))
})
this.$st.sendEv('爬虫详情-配置', '运行')
})
},
onExtractFields () {
@@ -235,6 +238,7 @@ export default {
.finally(() => {
this.extractFieldsLoading = false
})
this.$st.sendEv('爬虫详情-配置', '提取字段')
})
}
},

View File

@@ -49,10 +49,11 @@ export default {
name: '',
value: ''
})
console.log(this.spiderForm)
this.$st.sendEv('爬虫详情-环境', '添加')
},
deleteEnv (index) {
this.spiderForm.envs.splice(index, 1)
this.$st.sendEv('爬虫详情-环境', '删除')
},
save () {
this.$store.dispatch('spider/updateSpiderEnvs')
@@ -62,6 +63,7 @@ export default {
.catch(error => {
this.$message.error(error)
})
this.$st.sendEv('爬虫详情-环境', '保存')
}
}
}

View File

@@ -55,6 +55,7 @@ export default {
})
}
})
this.$st.sendEv('节点详情', '保存')
}
}
}

View File

@@ -127,6 +127,7 @@ export default {
.then(() => {
this.$message.success(this.$t(`Spider task has been scheduled`))
})
this.$st.sendEv('爬虫详情-概览', '运行')
})
}
})
@@ -149,6 +150,7 @@ export default {
.then(() => {
this.$message.success(this.$t(`Spider has been deployed`))
})
this.$st.sendEv('爬虫详情-概览', '部署')
})
}
})
@@ -165,6 +167,7 @@ export default {
})
}
})
this.$st.sendEv('爬虫详情-概览', '保存')
},
fetchSiteSuggestions (keyword, callback) {
this.$request.get('/sites', {

View File

@@ -52,9 +52,11 @@ export default {
methods: {
onClickNodeTitle () {
this.$router.push(`/nodes/${this.nodeForm._id}`)
this.$st.sendEv('任务详情-概览', '点击节点详情')
},
onClickSpiderTitle () {
this.$router.push(`/spiders/${this.spiderForm._id}`)
this.$st.sendEv('任务详情-概览', '点击爬虫详情')
}
},
created () {

View File

@@ -109,14 +109,17 @@ export default {
type: 'css',
extract_type: 'text'
})
this.$st.sendEv('爬虫详情-配置', '添加字段')
},
deleteField (index) {
this.fields.splice(index, 1)
this.$st.sendEv('爬虫详情-配置', '删除字段')
},
onNameChange (row) {
if (this.fields.filter(d => d.name === row.name).length > 1) {
this.$message.error(this.$t(`Duplicated field names for ${row.name}`))
}
this.$st.sendEv('爬虫详情-配置', '更改字段')
},
onCheck (row) {
this.fields.forEach(d => {
@@ -124,6 +127,7 @@ export default {
this.$set(d, 'is_detail', false)
}
})
this.$st.sendEv('爬虫详情-配置', '设置详情页URL')
}
}
}

View File

@@ -12,8 +12,6 @@ import 'font-awesome/scss/font-awesome.scss'// FontAwesome
import 'codemirror/lib/codemirror.css'
// import ba from 'vue-ba'
import App from './App'
import store from './store'
import router from './router'
@@ -23,6 +21,7 @@ import '@/permission' // permission control
import request from './api/request'
import i18n from './i18n'
import utils from './utils'
Vue.use(ElementUI, { locale })
@@ -45,6 +44,12 @@ window._hmt = window._hmt || [];
// inject request api
Vue.prototype.$request = request
// inject utils
Vue.prototype.$utils = utils
// inject stats
Vue.prototype.$st = utils.stats
const app = new Vue({
el: '#app',
i18n,

View File

@@ -1,6 +1,8 @@
import Vue from 'vue'
import Router from 'vue-router'
import stats from '../utils/stats'
/* Layout */
import Layout from '../views/layout/Layout'
@@ -224,9 +226,7 @@ router.beforeEach((to, from, next) => {
router.afterEach((to, from, next) => {
if (to.path) {
if (localStorage.getItem('useStats') !== '0') {
window._hmt.push(['_trackPageview', to.path])
}
stats.sendPv(to.path)
}
})

View File

@@ -0,0 +1,5 @@
import stats from './stats'
export default {
stats
}

View File

@@ -0,0 +1,12 @@
export default {
sendPv (page) {
if (localStorage.getItem('useStats') !== '0') {
window._hmt.push(['_trackPageview', page])
}
},
sendEv (category, eventName, optLabel, optValue) {
if (localStorage.getItem('useStats') !== '0') {
window._hmt.push(['_trackEvent', category, eventName, optLabel, optValue])
}
}
}

View File

@@ -134,6 +134,7 @@ export default {
},
onRefresh () {
this.$store.dispatch('deploy/getDeployList')
this.$st.sendEv('部署', '刷新')
},
onView (row) {
this.$router.push(`/deploys/${row._id}`)

View File

@@ -57,6 +57,8 @@ export default {
window.localStorage.setItem('lang', lang)
this.$i18n.locale = lang
this.$store.commit('lang/SET_LANG', lang)
this.$st.sendEv('全局', '切换中英文', 'lang', lang)
}
}
}

View File

@@ -47,6 +47,7 @@ export default {
},
onNodeChange (id) {
this.$router.push(`/nodes/${id}`)
this.$st.sendEv('节点详情', '切换节点')
}
},
created () {

View File

@@ -123,8 +123,7 @@ export default {
}
},
methods: {
onSearch (value) {
console.log(value)
onSearch () {
},
onAdd () {
this.$store.commit('node/SET_NODE_FORM', [])
@@ -133,6 +132,7 @@ export default {
},
onRefresh () {
this.$store.dispatch('node/getNodeList')
this.$st.sendEv('节点', '刷新')
},
onSubmit () {
const vm = this
@@ -159,7 +159,6 @@ export default {
this.dialogVisible = false
},
onEdit (row) {
console.log(row)
this.isEditMode = true
this.$store.commit('node/SET_NODE_FORM', row)
this.dialogVisible = true
@@ -177,10 +176,13 @@ export default {
message: 'Deleted successfully'
})
})
this.$st.sendEv('节点', '删除', 'id', row._id)
})
},
onView (row) {
this.$router.push(`/nodes/${row._id}`)
this.$st.sendEv('节点', '查看', 'id', row._id)
},
onPageChange () {
this.$store.dispatch('node/getNodeList')

View File

@@ -160,6 +160,7 @@ export default {
this.isEdit = false
this.dialogVisible = true
this.$store.commit('schedule/SET_SCHEDULE_FORM', {})
this.$st.sendEv('定时任务', '添加')
},
onAddSubmit () {
this.$refs.scheduleForm.validate(res => {
@@ -179,6 +180,7 @@ export default {
})
}
})
this.$st.sendEv('定时任务', '提交')
},
isShowRun () {
},
@@ -186,6 +188,7 @@ export default {
this.$store.commit('schedule/SET_SCHEDULE_FORM', row)
this.dialogVisible = true
this.isEdit = true
this.$st.sendEv('定时任务', '修改', 'id', row._id)
},
onRemove (row) {
this.$store.dispatch('schedule/removeSchedule', row._id)
@@ -195,6 +198,7 @@ export default {
this.$message.success(`Schedule "${row.name}" has been removed`)
}, 100)
})
this.$st.sendEv('定时任务', '删除', 'id', row._id)
},
onCrawl () {
}

View File

@@ -8,11 +8,11 @@
v-model="keyword">
</el-input>
<el-select v-model="filter.mainCategory" class="filter-category" :placeholder="$t('Select Main Category')"
clearable filterable>
clearable filterable @change="onSelectMainCategory">
<el-option v-for="op in mainCategoryList" :key="op" :value="op" :label="op"></el-option>
</el-select>
<el-select v-model="filter.category" class="filter-category" :placeholder="$t('Select Category')"
clearable filterable>
clearable filterable @change="onSelectCategory">
<el-option v-for="op in categoryList" :key="op" :value="op" :label="op"></el-option>
</el-select>
<el-button type="success"
@@ -53,7 +53,8 @@
:width="col.width"
:align="col.align">
<template slot-scope="scope">
<a class="domain" :href="'http://' + scope.row[col.name]" target="_blank">
<a class="domain" :href="'http://' + scope.row[col.name]" target="_blank"
@click="onClickDomain(scope.row._id)">
{{scope.row[col.name]}}
</a>
</template>
@@ -84,7 +85,7 @@
<template slot-scope="scope">
<div>
<template v-if="scope.row[col.name]">
<a :href="`http://${scope.row._id}/robots.txt`" target="_blank">
<a :href="`http://${scope.row._id}/robots.txt`" target="_blank" @click="onClickRobots(scope.row._id)">
Y
</a>
</template>
@@ -242,6 +243,16 @@ export default {
setTimeout(() => {
this.$store.dispatch('site/getSiteList')
}, 0)
this.$st.sendEv('网站', '搜索')
},
onSelectMainCategory () {
this.$st.sendEv('网站', '选择主类别')
},
onSelectCategory () {
this.$st.sendEv('网站', '选择类别')
},
onClickDomain (domain) {
this.$st.sendEv('网站', '点击域名', 'domain', domain)
},
onPageChange () {
setTimeout(() => {
@@ -302,6 +313,10 @@ export default {
},
goToSpiders (domain) {
this.$router.push({ name: 'SpiderList', params: { domain } })
this.$st.sendEv('网站', '点击爬虫数', 'domain', domain)
},
onClickRobots (domain) {
this.$st.sendEv('网站', '点击Robots协议', 'domain', domain)
}
},
created () {

View File

@@ -72,15 +72,17 @@ export default {
}
},
methods: {
onTabClick () {
onTabClick (tab) {
if (this.activeTabName === 'analytics') {
setTimeout(() => {
this.$refs['spider-stats'].update()
}, 0)
}
this.$st.sendEv('爬虫详情', '切换标签', 'tabName', tab.name)
},
onSpiderChange (id) {
this.$router.push(`/spiders/${id}`)
this.$st.sendEv('爬虫详情', '切换爬虫')
}
},
created () {

View File

@@ -292,13 +292,16 @@ export default {
this.$store.commit('spider/SET_SPIDER_FORM', {})
this.addDialogVisible = false
this.addConfigurableDialogVisible = true
this.$st.sendEv('爬虫', '添加爬虫-可配置爬虫')
},
onAddCustomized () {
this.addDialogVisible = false
this.addCustomizedDialogVisible = true
this.$st.sendEv('爬虫', '添加爬虫-自定义爬虫')
},
onRefresh () {
this.$store.dispatch('spider/getSpiderList')
this.$st.sendEv('爬虫', '刷新')
},
onSubmit () {
const vm = this
@@ -354,6 +357,7 @@ export default {
message: 'Deleted successfully'
})
})
this.$st.sendEv('爬虫', '删除')
})
},
onDeploy (row) {
@@ -369,6 +373,7 @@ export default {
message: 'Deployed successfully'
})
})
this.$st.sendEv('爬虫', '部署')
})
},
onCrawl (row) {
@@ -381,10 +386,12 @@ export default {
.then(() => {
this.$message.success(this.$t(`Spider task has been scheduled`))
})
this.$st.sendEv('爬虫', '运行')
})
},
onView (row) {
this.$router.push(`/spiders/${row._id}`)
this.$st.sendEv('爬虫', '查看')
},
onPageChange () {
this.$store.dispatch('spider/getSpiderList')
@@ -408,6 +415,7 @@ export default {
})
}
})
this.$st.sendEv('爬虫', '导入爬虫')
},
openImportDialog () {
this.dialogVisible = true
@@ -423,6 +431,7 @@ export default {
.then(() => {
this.$message.success(this.$t('Deployed all spiders successfully'))
})
this.$st.sendEv('爬虫', '部署所有爬虫')
})
},
isShowRun (row) {
@@ -455,6 +464,7 @@ export default {
},
onSiteSelect (item) {
this.$store.commit('spider/SET_FILTER_SITE', item._id)
this.$st.sendEv('爬虫', '搜索网站')
},
onAddConfigurableSiteSelect (item) {
this.spiderForm.site = item._id

View File

@@ -80,7 +80,8 @@ export default {
}
},
methods: {
onTabClick () {
onTabClick (tab) {
this.$st.sendEv('任务详情', '切换标签', 'tabName', tab.name)
},
onSpiderChange (id) {
this.$router.push(`/spiders/${id}`)
@@ -93,6 +94,7 @@ export default {
},
downloadCSV () {
window.location.href = this.$request.baseUrl + '/tasks/' + this.$route.params.id + '/download_results'
this.$st.sendEv('任务详情-结果', '下载CSV')
}
},
created () {

View File

@@ -3,10 +3,20 @@
<!--filter-->
<div class="filter">
<div class="left">
<el-select class="filter-select" v-model="filter.node_id" :placeholder="$t('Node')" filterable clearable>
<el-select class="filter-select"
v-model="filter.node_id"
:placeholder="$t('Node')"
filterable
clearable
@change="onSelectNode">
<el-option v-for="op in nodeList" :key="op._id" :value="op._id" :label="op.name"></el-option>
</el-select>
<el-select class="filter-select" v-model="filter.spider_id" :placeholder="$t('Spider')" filterable clearable>
<el-select class="filter-select"
v-model="filter.spider_id"
:placeholder="$t('Spider')"
filterable
clearable
@change="onSelectSpider">
<el-option v-for="op in spiderList" :key="op._id" :value="op._id" :label="op.name"></el-option>
</el-select>
<el-button type="success"
@@ -183,6 +193,13 @@ export default {
},
onRefresh () {
this.$store.dispatch('task/getTaskList')
this.$st.sendEv('任务', '搜索')
},
onSelectNode () {
this.$st.sendEv('任务', '选择节点')
},
onSelectSpider () {
this.$st.sendEv('任务', '选择爬虫')
},
onRemove (row) {
this.$confirm(this.$t('Are you sure to delete this task?'), this.$t('Notification'), {
@@ -197,16 +214,20 @@ export default {
message: 'Deleted successfully'
})
})
this.$st.sendEv('任务', '删除', 'id', row._id)
})
},
onView (row) {
this.$router.push(`/tasks/${row._id}`)
this.$st.sendEv('任务', '搜索', 'id', row._id)
},
onClickSpider (row) {
this.$router.push(`/spiders/${row.spider_id}`)
this.$st.sendEv('任务', '点击爬虫详情', 'id', row.spider_id)
},
onClickNode (row) {
this.$router.push(`/nodes/${row.node_id}`)
this.$st.sendEv('任务', '点击节点详情', 'id', row.node_id)
},
onPageChange () {
setTimeout(() => {

View File

@@ -0,0 +1,2 @@
# App

View File

@@ -0,0 +1,2 @@
# Celery

View File

@@ -0,0 +1,2 @@
# 架构

View File

@@ -0,0 +1,6 @@
# 部署
所有爬虫在运行前需要被部署当相应当节点中。
部署时,爬虫会被打包到相应的目录中,方便环境隔离,开发环境的爬虫和生产环境的爬虫需要打包部署来实现隔离。

3
gitbook/Concept/Node.md Normal file
View File

@@ -0,0 +1,3 @@
# 节点
节点其实就是Celery中的Worker。一个节点运行时会连接到一个任务队列例如Redis来接收和运行任务。所有爬虫需要在运行时被部署到节点上用户在部署前需要定义节点的IP地址和端口。

View File

@@ -0,0 +1,2 @@
# 概念

15
gitbook/Concept/Spider.md Normal file
View File

@@ -0,0 +1,15 @@
# 爬虫
## 自动发现
`config.py`文件中,修改变量`PROJECT_SOURCE_FILE_FOLDER`作为爬虫项目所在的目录。Crawlab后台程序会自动发现这些爬虫项目并储存到数据库中。是不是很方便
## 部署爬虫
所有爬虫需要在抓取前被部署当相应当节点中。在"爬虫详情"页面点击"Deploy"按钮,爬虫将被部署到所有有效到节点中。
## 运行爬虫
部署爬虫之后,你可以在"爬虫详情"页面点击"Run"按钮来启动爬虫。一个爬虫任务将被触发,你可以在任务列表页面中看到这个任务。

3
gitbook/Concept/Task.md Normal file
View File

@@ -0,0 +1,3 @@
# 任务
任务被触发并被节点执行。用户可以在任务详情页面中看到任务到状态、日志和抓取结果。

View File

@@ -0,0 +1,2 @@
# Examples

View File

@@ -0,0 +1,61 @@
# 功能列表
类别 | 功能名称 | 已统计 | 备注
--- | --- | --- | ---
全局 | 打开页面 | Y | _trackPageview
全局 | 切换中英文 | Y
全局 | 允许/禁止统计 | Y
节点 | 刷新 | Y
节点 | 查看 | Y
节点 | 删除 | Y
节点详情 | 保存 | Y
节点详情 | 切换节点 | Y
爬虫 | 部署所有爬虫 | Y
爬虫 | 导入爬虫 | Y
爬虫 | 添加爬虫-可配置爬虫 | Y
爬虫 | 添加爬虫-自定义爬虫 | Y
爬虫 | 刷新 | Y
爬虫 | 查看 | Y
爬虫 | 删除 | Y
爬虫 | 部署 | Y
爬虫 | 运行 | Y
爬虫 | 搜索网站 | Y
爬虫详情 | 切换爬虫 | Y
爬虫详情 | 切换标签 | Y
爬虫详情-概览 | 保存 | Y
爬虫详情-概览 | 部署 | Y
爬虫详情-概览 | 运行 | Y
爬虫详情-环境 | 添加 | Y
爬虫详情-环境 | 删除 | Y
爬虫详情-环境 | 保存 | Y
爬虫详情-配置 | 保存 | Y
爬虫详情-配置 | 预览 | Y
爬虫详情-配置 | 提取字段 | Y
爬虫详情-配置 | 运行 | Y
爬虫详情-配置 | 添加字段 | Y
爬虫详情-配置 | 更改字段 | Y
爬虫详情-配置 | 删除字段 | Y
爬虫详情-配置 | 设置详情页URL | Y
任务 | 选择节点 | Y
任务 | 选择爬虫 | Y
任务 | 点击爬虫详情 | Y
任务 | 点击节点详情 | Y
任务 | 搜索 | Y
任务 | 查看 | Y
任务 | 删除 | Y
任务详情 | 切换标签 | Y
任务详情-概览 | 点击爬虫详情 | Y
任务详情-概览 | 点击节点详情 | Y
任务详情-结果 | 下载CSV | Y
定时任务 | 添加 | Y
定时任务 | 修改 | Y
定时任务 | 删除 | Y
定时任务 | 提交 | Y
部署 | 刷新 | Y
网站 | 搜索 | Y
网站 | 选择主类别 | Y
网站 | 选择类别 | Y
网站 | 点击域名 | Y
网站 | 点击爬虫数 | Y
网站 | 点击Robots协议 | N

View File

@@ -0,0 +1,22 @@
# 安装
最快安装Crawlab的方式是克隆一份代码到本地
```bash
git clone https://github.com/tikazyq/crawlab
```
安装类库
```bash
# 安装后台类库
pip install -r requirements.txt
```
```bash
# 安装前台类库
cd frontend
npm install
```

View File

@@ -0,0 +1,4 @@
# 快速开始
- [安装](Installation.md)
- [运行](Run.md)

59
gitbook/QuickStart/Run.md Normal file
View File

@@ -0,0 +1,59 @@
# 运行
在运行之前需要对Crawlab进行一些配置配置文件为`config.py`
```python
# project variables
PROJECT_SOURCE_FILE_FOLDER = '/Users/yeqing/projects/crawlab/spiders' # 爬虫源码根目录
PROJECT_DEPLOY_FILE_FOLDER = '/var/crawlab' # 爬虫部署根目录
PROJECT_LOGS_FOLDER = '/var/logs/crawlab' # 日志目录
PROJECT_TMP_FOLDER = '/tmp' # 临时文件目录
# celery variables
BROKER_URL = 'redis://192.168.99.100:6379/0' # 中间者URL连接redis
CELERY_RESULT_BACKEND = 'mongodb://192.168.99.100:27017/' # CELERY后台URL
CELERY_MONGODB_BACKEND_SETTINGS = {
'database': 'crawlab_test',
'taskmeta_collection': 'tasks_celery',
}
CELERY_TIMEZONE = 'Asia/Shanghai'
CELERY_ENABLE_UTC = True
# flower variables
FLOWER_API_ENDPOINT = 'http://localhost:5555/api' # Flower服务地址
# database variables
MONGO_HOST = '192.168.99.100'
MONGO_PORT = 27017
MONGO_DB = 'crawlab_test'
# flask variables
DEBUG = True
FLASK_HOST = '127.0.0.1'
FLASK_PORT = 8000
```
启动后端API也就是一个Flask App可以直接启动或者用gunicorn代替。
```bash
python app.py
```
启动Flower服务抱歉目前集成Flower到App服务中必须单独启动来获取节点信息后面的版本不需要这个操作
```bash
python ./bin/run_flower.py
```
启动本地Worker。在其他节点中如果想只是想执行任务的话只需要启动这一个服务就可以了。
```bash
python ./bin/run_worker.py
```
启动前端服务器。
```bash
cd ../frontend
npm run serve
```

167
gitbook/README.md Normal file
View File

@@ -0,0 +1,167 @@
# Crawlab
基于Celery的爬虫分布式爬虫管理平台支持多种编程语言以及多种爬虫框架.
[查看演示 Demo](http://139.129.230.98:8080)
[English Documentation](https://github.com/tikazyq/crawlab/blob/master/README.md)
## 要求
- Python3
- MongoDB
- Redis
## 安装
```bash
# 安装后台类库
pip install -r requirements.txt
```
```bash
# 安装前台类库
cd frontend
npm install
```
## 配置
请更改配置文件`config.py`配置API和数据库连接.
## 快速开始
```bash
# 启动后端API
python app.py
# 启动Flower服务
python ./bin/run_flower.py
# 启动worker
python ./bin/run_worker.py
```
```bash
# 运行前端
cd frontend
npm run serve
```
## 截图
#### 首页
![home](./img/screenshot-home.png)
#### 爬虫列表
![spider-list](./img/screenshot-spiders.png)
#### 爬虫详情 - 概览
![spider-list](./img/screenshot-spider-detail-overview.png)
#### 任务详情 - 抓取结果
![spider-list](./img/screenshot-task-detail-results.png)
## 架构
Crawlab的架构跟Celery非常相似但是加入了包括前端、爬虫、Flower在内的额外模块以支持爬虫管理的功能。
![crawlab-architecture](./img/crawlab-architecture.png)
### 节点
节点其实就是Celery中的Worker。一个节点运行时会连接到一个任务队列例如Redis来接收和运行任务。所有爬虫需要在运行时被部署到节点上用户在部署前需要定义节点的IP地址和端口。
### 爬虫
##### 自动发现
`config.py`文件中,修改变量`PROJECT_SOURCE_FILE_FOLDER`作为爬虫项目所在的目录。Crawlab后台程序会自动发现这些爬虫项目并储存到数据库中。是不是很方便
##### 部署爬虫
所有爬虫需要在抓取前被部署当相应当节点中。在"爬虫详情"页面点击"Deploy"按钮,爬虫将被部署到所有有效到节点中。
##### 运行爬虫
部署爬虫之后,你可以在"爬虫详情"页面点击"Run"按钮来启动爬虫。一个爬虫任务将被触发,你可以在任务列表页面中看到这个任务。
### 任务
任务被触发并被节点执行。用户可以在任务详情页面中看到任务到状态、日志和抓取结果。
### 后台应用
这是一个Flask应用提供了必要的API来支持常规操作例如CRUD、爬虫部署以及任务运行。每一个节点需要启动Flask应用来支持爬虫部署。运行`python manage.py app``python ./bin/run_app.py`来启动应用。
### 中间者
中间者跟Celery中定义的一样作为运行异步任务的队列。
### 前端
前端其实就是一个基于[Vue-Element-Admin](https://github.com/PanJiaChen/vue-element-admin)的单页应用。其中重用了很多Element-UI的控件来支持相应的展示。
## 与其他框架的集成
任务是利用python的`subprocess`模块中的`Popen`来实现的。任务ID将以环境变量`CRAWLAB_TASK_ID`的形式存在于爬虫任务运行的进程中,并以此来关联抓取数据。
在你的爬虫程序中,你需要将`CRAWLAB_TASK_ID`的值以`task_id`作为可以存入数据库中。这样Crawlab就直到如何将爬虫任务与抓取数据关联起来了。当前Crawlab只支持MongoDB。
### Scrapy
以下是Crawlab跟Scrapy集成的例子利用了Crawlab传过来的task_id和collection_name。
```python
import os
from pymongo import MongoClient
MONGO_HOST = '192.168.99.100'
MONGO_PORT = 27017
MONGO_DB = 'crawlab_test'
# scrapy example in the pipeline
class JuejinPipeline(object):
mongo = MongoClient(host=MONGO_HOST, port=MONGO_PORT)
db = mongo[MONGO_DB]
col_name = os.environ.get('CRAWLAB_COLLECTION')
if not col_name:
col_name = 'test'
col = db[col_name]
def process_item(self, item, spider):
item['task_id'] = os.environ.get('CRAWLAB_TASK_ID')
self.col.save(item)
return item
```
## 与其他框架比较
限制以及有一些爬虫管理框架了因此为啥还要用Crawlab
因为很多现有当平台都依赖于Scrapyd限制了爬虫的编程语言以及框架爬虫工程师只能用scrapy和python。当然scrapy是非常优秀的爬虫框架但是它不能做一切事情。
Crawlab使用起来很方便也很通用可以适用于几乎任何主流语言和框架。它还有一个精美的前端界面让用户可以方便的管理和运行爬虫。
|框架 | 类型 | 分布式 | 前端 | 依赖于Scrapyd |
|:---:|:---:|:---:|:---:|:---:|
| [Crawlab](https://github.com/tikazyq/crawlab) | 管理平台 | Y | Y | N
| [Gerapy](https://github.com/Gerapy/Gerapy) | 管理平台 | Y | Y | Y
| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | 管理平台 | Y | Y | Y
| [ScrapydWeb](https://github.com/my8100/scrapydweb) | 管理平台 | Y | Y | Y
| [Scrapyd](https://github.com/scrapy/scrapyd) | 网络服务 | Y | N | N/A
## TODOs
##### 后端
- [ ] 文件管理
- [ ] MySQL数据库支持
- [ ] 重跑任务
- [ ] 节点监控
- [ ] 更多爬虫例子
##### 前端
- [ ] 任务数据统计
- [ ] 表格过滤
- [x] 多语言支持 (中文)
- [ ] 登录和用户管理
- [ ] 全局搜索

18
gitbook/SUMMARY.md Normal file
View File

@@ -0,0 +1,18 @@
# Summary
* [简介](README.md)
* [快速开始](QuickStart/README.md)
* [安装](QuickStart/Installation.md)
* [运行](QuickStart/Run.md)
* [概念](Concept/README.md)
* [节点](Concept/Node.md)
* [爬虫](Concept/Spider.md)
* [任务](Concept/Task.md)
* [部署](Concept/Deploy.md)
* [架构](Architecture/README.md)
* [Celery](Architecture/Celery.md)
* [App](Architecture/App.md)
* [Examples](Examples/README.md)
* [与Scrapy集成](Examples/README.md)
* [与Puppeteer集成](Examples/README.md)

Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

BIN
gitbook/img/payment.jpg Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 78 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB