mirror of
https://github.com/crawlab-team/crawlab.git
synced 2026-01-22 17:31:03 +01:00
11
README-zh.md
11
README-zh.md
@@ -5,14 +5,15 @@
|
||||
<img src="https://img.shields.io/badge/License-BSD-blue.svg">
|
||||
</a>
|
||||
|
||||
中文 | [English](https://github.com/tikazyq/crawlab/blob/master/README.md)
|
||||
|
||||
基于Celery的爬虫分布式爬虫管理平台,支持多种编程语言以及多种爬虫框架.
|
||||
|
||||
[查看演示 Demo](http://114.67.75.98:8080)
|
||||
|
||||
[English Documentation](https://github.com/tikazyq/crawlab/blob/master/README.md)
|
||||
[查看演示 Demo](http://114.67.75.98:8080) | [文档](https://tikazyq.github.io/crawlab)
|
||||
|
||||
## 要求
|
||||
- Python3
|
||||
- Python 3.6+
|
||||
- Node.js 8.12+
|
||||
- MongoDB
|
||||
- Redis
|
||||
|
||||
@@ -177,6 +178,8 @@ Crawlab使用起来很方便,也很通用,可以适用于几乎任何主流
|
||||
- [ ] 登录和用户管理
|
||||
- [ ] 全局搜索
|
||||
|
||||
## 社区 & 赞助
|
||||
|
||||
如果您觉得Crawlab对您的日常开发或公司有帮助,请加作者微信 tikazyq1 并注明"Crawlab",作者会将你拉入群。或者,您可以扫下方支付宝二维码给作者打赏去升级团队协作软件或买一杯咖啡。
|
||||
|
||||
<p align="center">
|
||||
|
||||
14
README.md
14
README.md
@@ -5,14 +5,16 @@
|
||||
<img src="https://img.shields.io/badge/License-BSD-blue.svg">
|
||||
</a>
|
||||
|
||||
[中文](https://github.com/tikazyq/crawlab/blob/master/README-zh.md) | English
|
||||
|
||||
Celery-based web crawler admin platform for managing distributed web spiders regardless of languages and frameworks.
|
||||
|
||||
[Demo](http://114.67.75.98:8080)
|
||||
[Demo](http://114.67.75.98:8080) | [Documentation](https://tikazyq.github.io/crawlab)
|
||||
|
||||
[中文文档](https://github.com/tikazyq/crawlab/blob/master/README-zh.md)
|
||||
|
||||
## Pre-requisite
|
||||
- Python3
|
||||
- Python 3.6+
|
||||
- Node.js 8.12+
|
||||
- MongoDB
|
||||
- Redis
|
||||
|
||||
@@ -166,12 +168,14 @@ Crawlab is easy to use, general enough to adapt spiders in any language and any
|
||||
- [ ] More spider examples
|
||||
|
||||
##### Frontend
|
||||
- [ ] Task Stats/Analytics
|
||||
- [ ] Table Filters
|
||||
- [x] Task Stats/Analytics
|
||||
- [x] Table Filters
|
||||
- [x] Multi-Language Support (中文)
|
||||
- [ ] Login & User Management
|
||||
- [ ] General Search
|
||||
|
||||
## Community & Sponsorship
|
||||
|
||||
If you feel Crawlab could benefit your daily work or your company, please add the author's Wechat account noting "Crawlab" to enter the discussion group. Or you scan the Alipay QR code below to give us a reward to upgrade our teamwork software or buy a coffee.
|
||||
|
||||
<p align="center">
|
||||
|
||||
@@ -397,7 +397,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"App","level":"1.4.2","depth":2,"next":{"title":"Examples","level":"1.5","depth":1,"path":"Examples/README.md","ref":"Examples/README.md","articles":[{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},{"title":"与Puppeteer集成","level":"1.5.2","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]}]},"previous":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/App.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"App","level":"1.4.2","depth":2,"next":{"title":"Examples","level":"1.5","depth":1,"path":"Examples/README.md","ref":"Examples/README.md","articles":[{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},{"title":"与Puppeteer集成","level":"1.5.2","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]}]},"previous":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/App.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -397,7 +397,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"Celery","level":"1.4.1","depth":2,"next":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"previous":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/Celery.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"Celery","level":"1.4.1","depth":2,"next":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"previous":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/Celery.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -397,7 +397,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"架构","level":"1.4","depth":1,"next":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"previous":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"架构","level":"1.4","depth":1,"next":{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},"previous":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Architecture/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -399,7 +399,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"部署","level":"1.3.4","depth":2,"next":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"previous":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Deploy.md","mtime":"2019-03-28T12:06:24.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"部署","level":"1.3.4","depth":2,"next":{"title":"架构","level":"1.4","depth":1,"path":"Architecture/README.md","ref":"Architecture/README.md","articles":[{"title":"Celery","level":"1.4.1","depth":2,"path":"Architecture/Celery.md","ref":"Architecture/Celery.md","articles":[]},{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]}]},"previous":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Deploy.md","mtime":"2019-03-28T12:06:24.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -398,7 +398,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"节点","level":"1.3.1","depth":2,"next":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"previous":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Node.md","mtime":"2019-03-28T12:02:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"节点","level":"1.3.1","depth":2,"next":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"previous":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Node.md","mtime":"2019-03-28T12:02:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -403,7 +403,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"爬虫","level":"1.3.2","depth":2,"next":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"previous":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Spider.md","mtime":"2019-03-28T12:03:08.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"爬虫","level":"1.3.2","depth":2,"next":{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},"previous":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Spider.md","mtime":"2019-03-28T12:03:08.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -398,7 +398,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"任务","level":"1.3.3","depth":2,"next":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"previous":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Task.md","mtime":"2019-03-28T12:04:12.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"任务","level":"1.3.3","depth":2,"next":{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]},"previous":{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/Task.md","mtime":"2019-03-28T12:04:12.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -397,7 +397,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"概念","level":"1.3","depth":1,"next":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"previous":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/README.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"概念","level":"1.3","depth":1,"next":{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},"previous":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Concept/README.md","mtime":"2019-03-28T11:49:43.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -397,7 +397,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"Examples","level":"1.5","depth":1,"next":{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},"previous":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Examples/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"Examples","level":"1.5","depth":1,"next":{"title":"与Scrapy集成","level":"1.5.1","depth":2,"path":"Examples/README.md","ref":"Examples/README.md","articles":[]},"previous":{"title":"App","level":"1.4.2","depth":2,"path":"Architecture/App.md","ref":"Architecture/App.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"Examples/README.md","mtime":"2019-03-28T11:41:28.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
61
docs/Functions/FunctionList.md
Normal file
61
docs/Functions/FunctionList.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# 功能列表
|
||||
|
||||
类别 | 功能名称 | 已统计 | 备注
|
||||
--- | --- | --- | ---
|
||||
全局 | 打开页面 | Y | _trackPageview
|
||||
全局 | 切换中英文 | Y
|
||||
全局 | 允许/禁止统计 | Y
|
||||
节点 | 刷新 | Y
|
||||
节点 | 查看 | Y
|
||||
节点 | 删除 | Y
|
||||
节点详情 | 保存 | Y
|
||||
节点详情 | 切换节点 | Y
|
||||
爬虫 | 部署所有爬虫 | Y
|
||||
爬虫 | 导入爬虫 | Y
|
||||
爬虫 | 添加爬虫-可配置爬虫 | Y
|
||||
爬虫 | 添加爬虫-自定义爬虫 | Y
|
||||
爬虫 | 刷新 | Y
|
||||
爬虫 | 查看 | Y
|
||||
爬虫 | 删除 | Y
|
||||
爬虫 | 部署 | Y
|
||||
爬虫 | 运行 | Y
|
||||
爬虫 | 搜索网站 | Y
|
||||
爬虫详情 | 切换爬虫 | Y
|
||||
爬虫详情 | 切换标签 | Y
|
||||
爬虫详情-概览 | 保存 | Y
|
||||
爬虫详情-概览 | 部署 | Y
|
||||
爬虫详情-概览 | 运行 | Y
|
||||
爬虫详情-环境 | 添加 | Y
|
||||
爬虫详情-环境 | 删除 | Y
|
||||
爬虫详情-环境 | 保存 | Y
|
||||
爬虫详情-配置 | 保存 | Y
|
||||
爬虫详情-配置 | 预览 | Y
|
||||
爬虫详情-配置 | 提取字段 | Y
|
||||
爬虫详情-配置 | 运行 | Y
|
||||
爬虫详情-配置 | 添加字段 | Y
|
||||
爬虫详情-配置 | 更改字段 | Y
|
||||
爬虫详情-配置 | 删除字段 | Y
|
||||
爬虫详情-配置 | 设置详情页URL | Y
|
||||
任务 | 选择节点 | Y
|
||||
任务 | 选择爬虫 | Y
|
||||
任务 | 点击爬虫详情 | Y
|
||||
任务 | 点击节点详情 | Y
|
||||
任务 | 搜索 | Y
|
||||
任务 | 查看 | Y
|
||||
任务 | 删除 | Y
|
||||
任务详情 | 切换标签 | Y
|
||||
任务详情-概览 | 点击爬虫详情 | Y
|
||||
任务详情-概览 | 点击节点详情 | Y
|
||||
任务详情-结果 | 下载CSV | Y
|
||||
定时任务 | 添加 | Y
|
||||
定时任务 | 修改 | Y
|
||||
定时任务 | 删除 | Y
|
||||
定时任务 | 提交 | Y
|
||||
部署 | 刷新 | Y
|
||||
网站 | 搜索 | Y
|
||||
网站 | 选择主类别 | Y
|
||||
网站 | 选择类别 | Y
|
||||
网站 | 点击域名 | Y
|
||||
网站 | 点击爬虫数 | Y
|
||||
网站 | 点击Robots协议 | N
|
||||
|
||||
@@ -408,7 +408,7 @@ npm install
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"安装","level":"1.2.1","depth":2,"next":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"previous":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Installation.md","mtime":"2019-03-28T11:55:48.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"安装","level":"1.2.1","depth":2,"next":{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]},"previous":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Installation.md","mtime":"2019-03-28T11:55:48.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -440,7 +440,7 @@ npm run serve
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"运行","level":"1.2.2","depth":2,"next":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"previous":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Run.md","mtime":"2019-03-28T12:00:11.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"运行","level":"1.2.2","depth":2,"next":{"title":"概念","level":"1.3","depth":1,"path":"Concept/README.md","ref":"Concept/README.md","articles":[{"title":"节点","level":"1.3.1","depth":2,"path":"Concept/Node.md","ref":"Concept/Node.md","articles":[]},{"title":"爬虫","level":"1.3.2","depth":2,"path":"Concept/Spider.md","ref":"Concept/Spider.md","articles":[]},{"title":"任务","level":"1.3.3","depth":2,"path":"Concept/Task.md","ref":"Concept/Task.md","articles":[]},{"title":"部署","level":"1.3.4","depth":2,"path":"Concept/Deploy.md","ref":"Concept/Deploy.md","articles":[]}]},"previous":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/Run.md","mtime":"2019-03-28T12:00:11.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -401,7 +401,7 @@
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"快速开始","level":"1.2","depth":1,"next":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"previous":{"title":"简介","level":"1.1","depth":1,"path":"README.md","ref":"README.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/README.md","mtime":"2019-03-28T12:02:02.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":"..","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"快速开始","level":"1.2","depth":1,"next":{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},"previous":{"title":"简介","level":"1.1","depth":1,"path":"README.md","ref":"README.md","articles":[]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"QuickStart/README.md","mtime":"2019-03-28T12:02:02.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":"..","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -548,7 +548,7 @@ MONGO_DB = <span class="hljs-string">'crawlab_test'</span>
|
||||
<script>
|
||||
var gitbook = gitbook || [];
|
||||
gitbook.push(function() {
|
||||
gitbook.page.hasChanged({"page":{"title":"简介","level":"1.1","depth":1,"next":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"README.md","mtime":"2019-03-28T11:44:15.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T11:09:00.453Z"},"basePath":".","book":{"language":""}});
|
||||
gitbook.page.hasChanged({"page":{"title":"简介","level":"1.1","depth":1,"next":{"title":"快速开始","level":"1.2","depth":1,"path":"QuickStart/README.md","ref":"QuickStart/README.md","articles":[{"title":"安装","level":"1.2.1","depth":2,"path":"QuickStart/Installation.md","ref":"QuickStart/Installation.md","articles":[]},{"title":"运行","level":"1.2.2","depth":2,"path":"QuickStart/Run.md","ref":"QuickStart/Run.md","articles":[]}]},"dir":"ltr"},"config":{"gitbook":"*","theme":"default","variables":{},"plugins":[],"pluginsConfig":{"highlight":{},"search":{},"lunr":{"maxIndexSize":1000000,"ignoreSpecialCharacters":false},"sharing":{"facebook":true,"twitter":true,"google":false,"weibo":false,"instapaper":false,"vk":false,"all":["facebook","google","twitter","weibo","instapaper"]},"fontsettings":{"theme":"white","family":"sans","size":2},"theme-default":{"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"},"showLevel":false}},"structure":{"langs":"LANGS.md","readme":"README.md","glossary":"GLOSSARY.md","summary":"SUMMARY.md"},"pdf":{"pageNumbers":true,"fontSize":12,"fontFamily":"Arial","paperSize":"a4","chapterMark":"pagebreak","pageBreaksBefore":"/","margin":{"right":62,"left":62,"top":56,"bottom":56}},"styles":{"website":"styles/website.css","pdf":"styles/pdf.css","epub":"styles/epub.css","mobi":"styles/mobi.css","ebook":"styles/ebook.css","print":"styles/print.css"}},"file":{"path":"README.md","mtime":"2019-03-28T11:44:15.000Z","type":"markdown"},"gitbook":{"version":"3.2.3","time":"2019-05-30T13:15:04.840Z"},"basePath":".","book":{"language":""}});
|
||||
});
|
||||
</script>
|
||||
</div>
|
||||
|
||||
@@ -29,9 +29,11 @@ export default {
|
||||
localStorage.setItem('useStats', value)
|
||||
document.querySelector('.el-message__closeBtn').click()
|
||||
if (value === 1) {
|
||||
window._hmt.push(['_trackPageview', '/allow_stats'])
|
||||
this.$st.sendPv('/allow_stats')
|
||||
this.$st.sendEv('全局', '允许/禁止统计', 'value', 'allow')
|
||||
} else {
|
||||
window._hmt.push(['_trackPageview', '/disallow_stats'])
|
||||
this.$st.sendPv('/disallow_stats')
|
||||
this.$st.sendEv('全局', '允许/禁止统计', 'value', 'disallow')
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -159,6 +159,7 @@ export default {
|
||||
this.spiderForm.crawl_type = value
|
||||
},
|
||||
onSave () {
|
||||
this.$st.sendEv('爬虫详情-配置', '保存')
|
||||
return new Promise((resolve, reject) => {
|
||||
this.saveLoading = true
|
||||
this.$store.dispatch('spider/updateSpiderFields')
|
||||
@@ -203,6 +204,7 @@ export default {
|
||||
.finally(() => {
|
||||
this.previewLoading = false
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-配置', '预览')
|
||||
})
|
||||
},
|
||||
onCrawl () {
|
||||
@@ -215,6 +217,7 @@ export default {
|
||||
.then(() => {
|
||||
this.$message.success(this.$t(`Spider task has been scheduled`))
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-配置', '运行')
|
||||
})
|
||||
},
|
||||
onExtractFields () {
|
||||
@@ -235,6 +238,7 @@ export default {
|
||||
.finally(() => {
|
||||
this.extractFieldsLoading = false
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-配置', '提取字段')
|
||||
})
|
||||
}
|
||||
},
|
||||
|
||||
@@ -49,10 +49,11 @@ export default {
|
||||
name: '',
|
||||
value: ''
|
||||
})
|
||||
console.log(this.spiderForm)
|
||||
this.$st.sendEv('爬虫详情-环境', '添加')
|
||||
},
|
||||
deleteEnv (index) {
|
||||
this.spiderForm.envs.splice(index, 1)
|
||||
this.$st.sendEv('爬虫详情-环境', '删除')
|
||||
},
|
||||
save () {
|
||||
this.$store.dispatch('spider/updateSpiderEnvs')
|
||||
@@ -62,6 +63,7 @@ export default {
|
||||
.catch(error => {
|
||||
this.$message.error(error)
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-环境', '保存')
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -55,6 +55,7 @@ export default {
|
||||
})
|
||||
}
|
||||
})
|
||||
this.$st.sendEv('节点详情', '保存')
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -127,6 +127,7 @@ export default {
|
||||
.then(() => {
|
||||
this.$message.success(this.$t(`Spider task has been scheduled`))
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-概览', '运行')
|
||||
})
|
||||
}
|
||||
})
|
||||
@@ -149,6 +150,7 @@ export default {
|
||||
.then(() => {
|
||||
this.$message.success(this.$t(`Spider has been deployed`))
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-概览', '部署')
|
||||
})
|
||||
}
|
||||
})
|
||||
@@ -165,6 +167,7 @@ export default {
|
||||
})
|
||||
}
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-概览', '保存')
|
||||
},
|
||||
fetchSiteSuggestions (keyword, callback) {
|
||||
this.$request.get('/sites', {
|
||||
|
||||
@@ -52,9 +52,11 @@ export default {
|
||||
methods: {
|
||||
onClickNodeTitle () {
|
||||
this.$router.push(`/nodes/${this.nodeForm._id}`)
|
||||
this.$st.sendEv('任务详情-概览', '点击节点详情')
|
||||
},
|
||||
onClickSpiderTitle () {
|
||||
this.$router.push(`/spiders/${this.spiderForm._id}`)
|
||||
this.$st.sendEv('任务详情-概览', '点击爬虫详情')
|
||||
}
|
||||
},
|
||||
created () {
|
||||
|
||||
@@ -109,14 +109,17 @@ export default {
|
||||
type: 'css',
|
||||
extract_type: 'text'
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-配置', '添加字段')
|
||||
},
|
||||
deleteField (index) {
|
||||
this.fields.splice(index, 1)
|
||||
this.$st.sendEv('爬虫详情-配置', '删除字段')
|
||||
},
|
||||
onNameChange (row) {
|
||||
if (this.fields.filter(d => d.name === row.name).length > 1) {
|
||||
this.$message.error(this.$t(`Duplicated field names for ${row.name}`))
|
||||
}
|
||||
this.$st.sendEv('爬虫详情-配置', '更改字段')
|
||||
},
|
||||
onCheck (row) {
|
||||
this.fields.forEach(d => {
|
||||
@@ -124,6 +127,7 @@ export default {
|
||||
this.$set(d, 'is_detail', false)
|
||||
}
|
||||
})
|
||||
this.$st.sendEv('爬虫详情-配置', '设置详情页URL')
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,8 +12,6 @@ import 'font-awesome/scss/font-awesome.scss'// FontAwesome
|
||||
|
||||
import 'codemirror/lib/codemirror.css'
|
||||
|
||||
// import ba from 'vue-ba'
|
||||
|
||||
import App from './App'
|
||||
import store from './store'
|
||||
import router from './router'
|
||||
@@ -23,6 +21,7 @@ import '@/permission' // permission control
|
||||
|
||||
import request from './api/request'
|
||||
import i18n from './i18n'
|
||||
import utils from './utils'
|
||||
|
||||
Vue.use(ElementUI, { locale })
|
||||
|
||||
@@ -45,6 +44,12 @@ window._hmt = window._hmt || [];
|
||||
// inject request api
|
||||
Vue.prototype.$request = request
|
||||
|
||||
// inject utils
|
||||
Vue.prototype.$utils = utils
|
||||
|
||||
// inject stats
|
||||
Vue.prototype.$st = utils.stats
|
||||
|
||||
const app = new Vue({
|
||||
el: '#app',
|
||||
i18n,
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import Vue from 'vue'
|
||||
import Router from 'vue-router'
|
||||
|
||||
import stats from '../utils/stats'
|
||||
|
||||
/* Layout */
|
||||
import Layout from '../views/layout/Layout'
|
||||
|
||||
@@ -224,9 +226,7 @@ router.beforeEach((to, from, next) => {
|
||||
|
||||
router.afterEach((to, from, next) => {
|
||||
if (to.path) {
|
||||
if (localStorage.getItem('useStats') !== '0') {
|
||||
window._hmt.push(['_trackPageview', to.path])
|
||||
}
|
||||
stats.sendPv(to.path)
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
import stats from './stats'
|
||||
|
||||
export default {
|
||||
stats
|
||||
}
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
export default {
|
||||
sendPv (page) {
|
||||
if (localStorage.getItem('useStats') !== '0') {
|
||||
window._hmt.push(['_trackPageview', page])
|
||||
}
|
||||
},
|
||||
sendEv (category, eventName, optLabel, optValue) {
|
||||
if (localStorage.getItem('useStats') !== '0') {
|
||||
window._hmt.push(['_trackEvent', category, eventName, optLabel, optValue])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -134,6 +134,7 @@ export default {
|
||||
},
|
||||
onRefresh () {
|
||||
this.$store.dispatch('deploy/getDeployList')
|
||||
this.$st.sendEv('部署', '刷新')
|
||||
},
|
||||
onView (row) {
|
||||
this.$router.push(`/deploys/${row._id}`)
|
||||
|
||||
@@ -57,6 +57,8 @@ export default {
|
||||
window.localStorage.setItem('lang', lang)
|
||||
this.$i18n.locale = lang
|
||||
this.$store.commit('lang/SET_LANG', lang)
|
||||
|
||||
this.$st.sendEv('全局', '切换中英文', 'lang', lang)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -47,6 +47,7 @@ export default {
|
||||
},
|
||||
onNodeChange (id) {
|
||||
this.$router.push(`/nodes/${id}`)
|
||||
this.$st.sendEv('节点详情', '切换节点')
|
||||
}
|
||||
},
|
||||
created () {
|
||||
|
||||
@@ -123,8 +123,7 @@ export default {
|
||||
}
|
||||
},
|
||||
methods: {
|
||||
onSearch (value) {
|
||||
console.log(value)
|
||||
onSearch () {
|
||||
},
|
||||
onAdd () {
|
||||
this.$store.commit('node/SET_NODE_FORM', [])
|
||||
@@ -133,6 +132,7 @@ export default {
|
||||
},
|
||||
onRefresh () {
|
||||
this.$store.dispatch('node/getNodeList')
|
||||
this.$st.sendEv('节点', '刷新')
|
||||
},
|
||||
onSubmit () {
|
||||
const vm = this
|
||||
@@ -159,7 +159,6 @@ export default {
|
||||
this.dialogVisible = false
|
||||
},
|
||||
onEdit (row) {
|
||||
console.log(row)
|
||||
this.isEditMode = true
|
||||
this.$store.commit('node/SET_NODE_FORM', row)
|
||||
this.dialogVisible = true
|
||||
@@ -177,10 +176,13 @@ export default {
|
||||
message: 'Deleted successfully'
|
||||
})
|
||||
})
|
||||
this.$st.sendEv('节点', '删除', 'id', row._id)
|
||||
})
|
||||
},
|
||||
onView (row) {
|
||||
this.$router.push(`/nodes/${row._id}`)
|
||||
|
||||
this.$st.sendEv('节点', '查看', 'id', row._id)
|
||||
},
|
||||
onPageChange () {
|
||||
this.$store.dispatch('node/getNodeList')
|
||||
|
||||
@@ -160,6 +160,7 @@ export default {
|
||||
this.isEdit = false
|
||||
this.dialogVisible = true
|
||||
this.$store.commit('schedule/SET_SCHEDULE_FORM', {})
|
||||
this.$st.sendEv('定时任务', '添加')
|
||||
},
|
||||
onAddSubmit () {
|
||||
this.$refs.scheduleForm.validate(res => {
|
||||
@@ -179,6 +180,7 @@ export default {
|
||||
})
|
||||
}
|
||||
})
|
||||
this.$st.sendEv('定时任务', '提交')
|
||||
},
|
||||
isShowRun () {
|
||||
},
|
||||
@@ -186,6 +188,7 @@ export default {
|
||||
this.$store.commit('schedule/SET_SCHEDULE_FORM', row)
|
||||
this.dialogVisible = true
|
||||
this.isEdit = true
|
||||
this.$st.sendEv('定时任务', '修改', 'id', row._id)
|
||||
},
|
||||
onRemove (row) {
|
||||
this.$store.dispatch('schedule/removeSchedule', row._id)
|
||||
@@ -195,6 +198,7 @@ export default {
|
||||
this.$message.success(`Schedule "${row.name}" has been removed`)
|
||||
}, 100)
|
||||
})
|
||||
this.$st.sendEv('定时任务', '删除', 'id', row._id)
|
||||
},
|
||||
onCrawl () {
|
||||
}
|
||||
|
||||
@@ -8,11 +8,11 @@
|
||||
v-model="keyword">
|
||||
</el-input>
|
||||
<el-select v-model="filter.mainCategory" class="filter-category" :placeholder="$t('Select Main Category')"
|
||||
clearable filterable>
|
||||
clearable filterable @change="onSelectMainCategory">
|
||||
<el-option v-for="op in mainCategoryList" :key="op" :value="op" :label="op"></el-option>
|
||||
</el-select>
|
||||
<el-select v-model="filter.category" class="filter-category" :placeholder="$t('Select Category')"
|
||||
clearable filterable>
|
||||
clearable filterable @change="onSelectCategory">
|
||||
<el-option v-for="op in categoryList" :key="op" :value="op" :label="op"></el-option>
|
||||
</el-select>
|
||||
<el-button type="success"
|
||||
@@ -53,7 +53,8 @@
|
||||
:width="col.width"
|
||||
:align="col.align">
|
||||
<template slot-scope="scope">
|
||||
<a class="domain" :href="'http://' + scope.row[col.name]" target="_blank">
|
||||
<a class="domain" :href="'http://' + scope.row[col.name]" target="_blank"
|
||||
@click="onClickDomain(scope.row._id)">
|
||||
{{scope.row[col.name]}}
|
||||
</a>
|
||||
</template>
|
||||
@@ -84,7 +85,7 @@
|
||||
<template slot-scope="scope">
|
||||
<div>
|
||||
<template v-if="scope.row[col.name]">
|
||||
<a :href="`http://${scope.row._id}/robots.txt`" target="_blank">
|
||||
<a :href="`http://${scope.row._id}/robots.txt`" target="_blank" @click="onClickRobots(scope.row._id)">
|
||||
Y
|
||||
</a>
|
||||
</template>
|
||||
@@ -242,6 +243,16 @@ export default {
|
||||
setTimeout(() => {
|
||||
this.$store.dispatch('site/getSiteList')
|
||||
}, 0)
|
||||
this.$st.sendEv('网站', '搜索')
|
||||
},
|
||||
onSelectMainCategory () {
|
||||
this.$st.sendEv('网站', '选择主类别')
|
||||
},
|
||||
onSelectCategory () {
|
||||
this.$st.sendEv('网站', '选择类别')
|
||||
},
|
||||
onClickDomain (domain) {
|
||||
this.$st.sendEv('网站', '点击域名', 'domain', domain)
|
||||
},
|
||||
onPageChange () {
|
||||
setTimeout(() => {
|
||||
@@ -302,6 +313,10 @@ export default {
|
||||
},
|
||||
goToSpiders (domain) {
|
||||
this.$router.push({ name: 'SpiderList', params: { domain } })
|
||||
this.$st.sendEv('网站', '点击爬虫数', 'domain', domain)
|
||||
},
|
||||
onClickRobots (domain) {
|
||||
this.$st.sendEv('网站', '点击Robots协议', 'domain', domain)
|
||||
}
|
||||
},
|
||||
created () {
|
||||
|
||||
@@ -72,15 +72,17 @@ export default {
|
||||
}
|
||||
},
|
||||
methods: {
|
||||
onTabClick () {
|
||||
onTabClick (tab) {
|
||||
if (this.activeTabName === 'analytics') {
|
||||
setTimeout(() => {
|
||||
this.$refs['spider-stats'].update()
|
||||
}, 0)
|
||||
}
|
||||
this.$st.sendEv('爬虫详情', '切换标签', 'tabName', tab.name)
|
||||
},
|
||||
onSpiderChange (id) {
|
||||
this.$router.push(`/spiders/${id}`)
|
||||
this.$st.sendEv('爬虫详情', '切换爬虫')
|
||||
}
|
||||
},
|
||||
created () {
|
||||
|
||||
@@ -292,13 +292,16 @@ export default {
|
||||
this.$store.commit('spider/SET_SPIDER_FORM', {})
|
||||
this.addDialogVisible = false
|
||||
this.addConfigurableDialogVisible = true
|
||||
this.$st.sendEv('爬虫', '添加爬虫-可配置爬虫')
|
||||
},
|
||||
onAddCustomized () {
|
||||
this.addDialogVisible = false
|
||||
this.addCustomizedDialogVisible = true
|
||||
this.$st.sendEv('爬虫', '添加爬虫-自定义爬虫')
|
||||
},
|
||||
onRefresh () {
|
||||
this.$store.dispatch('spider/getSpiderList')
|
||||
this.$st.sendEv('爬虫', '刷新')
|
||||
},
|
||||
onSubmit () {
|
||||
const vm = this
|
||||
@@ -354,6 +357,7 @@ export default {
|
||||
message: 'Deleted successfully'
|
||||
})
|
||||
})
|
||||
this.$st.sendEv('爬虫', '删除')
|
||||
})
|
||||
},
|
||||
onDeploy (row) {
|
||||
@@ -369,6 +373,7 @@ export default {
|
||||
message: 'Deployed successfully'
|
||||
})
|
||||
})
|
||||
this.$st.sendEv('爬虫', '部署')
|
||||
})
|
||||
},
|
||||
onCrawl (row) {
|
||||
@@ -381,10 +386,12 @@ export default {
|
||||
.then(() => {
|
||||
this.$message.success(this.$t(`Spider task has been scheduled`))
|
||||
})
|
||||
this.$st.sendEv('爬虫', '运行')
|
||||
})
|
||||
},
|
||||
onView (row) {
|
||||
this.$router.push(`/spiders/${row._id}`)
|
||||
this.$st.sendEv('爬虫', '查看')
|
||||
},
|
||||
onPageChange () {
|
||||
this.$store.dispatch('spider/getSpiderList')
|
||||
@@ -408,6 +415,7 @@ export default {
|
||||
})
|
||||
}
|
||||
})
|
||||
this.$st.sendEv('爬虫', '导入爬虫')
|
||||
},
|
||||
openImportDialog () {
|
||||
this.dialogVisible = true
|
||||
@@ -423,6 +431,7 @@ export default {
|
||||
.then(() => {
|
||||
this.$message.success(this.$t('Deployed all spiders successfully'))
|
||||
})
|
||||
this.$st.sendEv('爬虫', '部署所有爬虫')
|
||||
})
|
||||
},
|
||||
isShowRun (row) {
|
||||
@@ -455,6 +464,7 @@ export default {
|
||||
},
|
||||
onSiteSelect (item) {
|
||||
this.$store.commit('spider/SET_FILTER_SITE', item._id)
|
||||
this.$st.sendEv('爬虫', '搜索网站')
|
||||
},
|
||||
onAddConfigurableSiteSelect (item) {
|
||||
this.spiderForm.site = item._id
|
||||
|
||||
@@ -80,7 +80,8 @@ export default {
|
||||
}
|
||||
},
|
||||
methods: {
|
||||
onTabClick () {
|
||||
onTabClick (tab) {
|
||||
this.$st.sendEv('任务详情', '切换标签', 'tabName', tab.name)
|
||||
},
|
||||
onSpiderChange (id) {
|
||||
this.$router.push(`/spiders/${id}`)
|
||||
@@ -93,6 +94,7 @@ export default {
|
||||
},
|
||||
downloadCSV () {
|
||||
window.location.href = this.$request.baseUrl + '/tasks/' + this.$route.params.id + '/download_results'
|
||||
this.$st.sendEv('任务详情-结果', '下载CSV')
|
||||
}
|
||||
},
|
||||
created () {
|
||||
|
||||
@@ -3,10 +3,20 @@
|
||||
<!--filter-->
|
||||
<div class="filter">
|
||||
<div class="left">
|
||||
<el-select class="filter-select" v-model="filter.node_id" :placeholder="$t('Node')" filterable clearable>
|
||||
<el-select class="filter-select"
|
||||
v-model="filter.node_id"
|
||||
:placeholder="$t('Node')"
|
||||
filterable
|
||||
clearable
|
||||
@change="onSelectNode">
|
||||
<el-option v-for="op in nodeList" :key="op._id" :value="op._id" :label="op.name"></el-option>
|
||||
</el-select>
|
||||
<el-select class="filter-select" v-model="filter.spider_id" :placeholder="$t('Spider')" filterable clearable>
|
||||
<el-select class="filter-select"
|
||||
v-model="filter.spider_id"
|
||||
:placeholder="$t('Spider')"
|
||||
filterable
|
||||
clearable
|
||||
@change="onSelectSpider">
|
||||
<el-option v-for="op in spiderList" :key="op._id" :value="op._id" :label="op.name"></el-option>
|
||||
</el-select>
|
||||
<el-button type="success"
|
||||
@@ -183,6 +193,13 @@ export default {
|
||||
},
|
||||
onRefresh () {
|
||||
this.$store.dispatch('task/getTaskList')
|
||||
this.$st.sendEv('任务', '搜索')
|
||||
},
|
||||
onSelectNode () {
|
||||
this.$st.sendEv('任务', '选择节点')
|
||||
},
|
||||
onSelectSpider () {
|
||||
this.$st.sendEv('任务', '选择爬虫')
|
||||
},
|
||||
onRemove (row) {
|
||||
this.$confirm(this.$t('Are you sure to delete this task?'), this.$t('Notification'), {
|
||||
@@ -197,16 +214,20 @@ export default {
|
||||
message: 'Deleted successfully'
|
||||
})
|
||||
})
|
||||
this.$st.sendEv('任务', '删除', 'id', row._id)
|
||||
})
|
||||
},
|
||||
onView (row) {
|
||||
this.$router.push(`/tasks/${row._id}`)
|
||||
this.$st.sendEv('任务', '搜索', 'id', row._id)
|
||||
},
|
||||
onClickSpider (row) {
|
||||
this.$router.push(`/spiders/${row.spider_id}`)
|
||||
this.$st.sendEv('任务', '点击爬虫详情', 'id', row.spider_id)
|
||||
},
|
||||
onClickNode (row) {
|
||||
this.$router.push(`/nodes/${row.node_id}`)
|
||||
this.$st.sendEv('任务', '点击节点详情', 'id', row.node_id)
|
||||
},
|
||||
onPageChange () {
|
||||
setTimeout(() => {
|
||||
|
||||
2
gitbook/Architecture/App.md
Normal file
2
gitbook/Architecture/App.md
Normal file
@@ -0,0 +1,2 @@
|
||||
# App
|
||||
|
||||
2
gitbook/Architecture/Celery.md
Normal file
2
gitbook/Architecture/Celery.md
Normal file
@@ -0,0 +1,2 @@
|
||||
# Celery
|
||||
|
||||
2
gitbook/Architecture/README.md
Normal file
2
gitbook/Architecture/README.md
Normal file
@@ -0,0 +1,2 @@
|
||||
# 架构
|
||||
|
||||
6
gitbook/Concept/Deploy.md
Normal file
6
gitbook/Concept/Deploy.md
Normal file
@@ -0,0 +1,6 @@
|
||||
# 部署
|
||||
|
||||
所有爬虫在运行前需要被部署当相应当节点中。
|
||||
|
||||
部署时,爬虫会被打包到相应的目录中,方便环境隔离,开发环境的爬虫和生产环境的爬虫需要打包部署来实现隔离。
|
||||
|
||||
3
gitbook/Concept/Node.md
Normal file
3
gitbook/Concept/Node.md
Normal file
@@ -0,0 +1,3 @@
|
||||
# 节点
|
||||
|
||||
节点其实就是Celery中的Worker。一个节点运行时会连接到一个任务队列(例如Redis)来接收和运行任务。所有爬虫需要在运行时被部署到节点上,用户在部署前需要定义节点的IP地址和端口。
|
||||
2
gitbook/Concept/README.md
Normal file
2
gitbook/Concept/README.md
Normal file
@@ -0,0 +1,2 @@
|
||||
# 概念
|
||||
|
||||
15
gitbook/Concept/Spider.md
Normal file
15
gitbook/Concept/Spider.md
Normal file
@@ -0,0 +1,15 @@
|
||||
# 爬虫
|
||||
|
||||
## 自动发现
|
||||
|
||||
在`config.py`文件中,修改变量`PROJECT_SOURCE_FILE_FOLDER`作为爬虫项目所在的目录。Crawlab后台程序会自动发现这些爬虫项目并储存到数据库中。是不是很方便?
|
||||
|
||||
## 部署爬虫
|
||||
|
||||
所有爬虫需要在抓取前被部署当相应当节点中。在"爬虫详情"页面点击"Deploy"按钮,爬虫将被部署到所有有效到节点中。
|
||||
|
||||
## 运行爬虫
|
||||
|
||||
部署爬虫之后,你可以在"爬虫详情"页面点击"Run"按钮来启动爬虫。一个爬虫任务将被触发,你可以在任务列表页面中看到这个任务。
|
||||
|
||||
|
||||
3
gitbook/Concept/Task.md
Normal file
3
gitbook/Concept/Task.md
Normal file
@@ -0,0 +1,3 @@
|
||||
# 任务
|
||||
|
||||
任务被触发并被节点执行。用户可以在任务详情页面中看到任务到状态、日志和抓取结果。
|
||||
2
gitbook/Examples/README.md
Normal file
2
gitbook/Examples/README.md
Normal file
@@ -0,0 +1,2 @@
|
||||
# Examples
|
||||
|
||||
61
gitbook/Functions/FunctionList.md
Normal file
61
gitbook/Functions/FunctionList.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# 功能列表
|
||||
|
||||
类别 | 功能名称 | 已统计 | 备注
|
||||
--- | --- | --- | ---
|
||||
全局 | 打开页面 | Y | _trackPageview
|
||||
全局 | 切换中英文 | Y
|
||||
全局 | 允许/禁止统计 | Y
|
||||
节点 | 刷新 | Y
|
||||
节点 | 查看 | Y
|
||||
节点 | 删除 | Y
|
||||
节点详情 | 保存 | Y
|
||||
节点详情 | 切换节点 | Y
|
||||
爬虫 | 部署所有爬虫 | Y
|
||||
爬虫 | 导入爬虫 | Y
|
||||
爬虫 | 添加爬虫-可配置爬虫 | Y
|
||||
爬虫 | 添加爬虫-自定义爬虫 | Y
|
||||
爬虫 | 刷新 | Y
|
||||
爬虫 | 查看 | Y
|
||||
爬虫 | 删除 | Y
|
||||
爬虫 | 部署 | Y
|
||||
爬虫 | 运行 | Y
|
||||
爬虫 | 搜索网站 | Y
|
||||
爬虫详情 | 切换爬虫 | Y
|
||||
爬虫详情 | 切换标签 | Y
|
||||
爬虫详情-概览 | 保存 | Y
|
||||
爬虫详情-概览 | 部署 | Y
|
||||
爬虫详情-概览 | 运行 | Y
|
||||
爬虫详情-环境 | 添加 | Y
|
||||
爬虫详情-环境 | 删除 | Y
|
||||
爬虫详情-环境 | 保存 | Y
|
||||
爬虫详情-配置 | 保存 | Y
|
||||
爬虫详情-配置 | 预览 | Y
|
||||
爬虫详情-配置 | 提取字段 | Y
|
||||
爬虫详情-配置 | 运行 | Y
|
||||
爬虫详情-配置 | 添加字段 | Y
|
||||
爬虫详情-配置 | 更改字段 | Y
|
||||
爬虫详情-配置 | 删除字段 | Y
|
||||
爬虫详情-配置 | 设置详情页URL | Y
|
||||
任务 | 选择节点 | Y
|
||||
任务 | 选择爬虫 | Y
|
||||
任务 | 点击爬虫详情 | Y
|
||||
任务 | 点击节点详情 | Y
|
||||
任务 | 搜索 | Y
|
||||
任务 | 查看 | Y
|
||||
任务 | 删除 | Y
|
||||
任务详情 | 切换标签 | Y
|
||||
任务详情-概览 | 点击爬虫详情 | Y
|
||||
任务详情-概览 | 点击节点详情 | Y
|
||||
任务详情-结果 | 下载CSV | Y
|
||||
定时任务 | 添加 | Y
|
||||
定时任务 | 修改 | Y
|
||||
定时任务 | 删除 | Y
|
||||
定时任务 | 提交 | Y
|
||||
部署 | 刷新 | Y
|
||||
网站 | 搜索 | Y
|
||||
网站 | 选择主类别 | Y
|
||||
网站 | 选择类别 | Y
|
||||
网站 | 点击域名 | Y
|
||||
网站 | 点击爬虫数 | Y
|
||||
网站 | 点击Robots协议 | N
|
||||
|
||||
22
gitbook/QuickStart/Installation.md
Normal file
22
gitbook/QuickStart/Installation.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# 安装
|
||||
|
||||
最快安装Crawlab的方式是克隆一份代码到本地
|
||||
|
||||
```bash
|
||||
git clone https://github.com/tikazyq/crawlab
|
||||
```
|
||||
|
||||
安装类库
|
||||
|
||||
```bash
|
||||
# 安装后台类库
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
```bash
|
||||
# 安装前台类库
|
||||
cd frontend
|
||||
npm install
|
||||
```
|
||||
|
||||
|
||||
4
gitbook/QuickStart/README.md
Normal file
4
gitbook/QuickStart/README.md
Normal file
@@ -0,0 +1,4 @@
|
||||
# 快速开始
|
||||
|
||||
- [安装](Installation.md)
|
||||
- [运行](Run.md)
|
||||
59
gitbook/QuickStart/Run.md
Normal file
59
gitbook/QuickStart/Run.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# 运行
|
||||
|
||||
在运行之前需要对Crawlab进行一些配置,配置文件为`config.py`。
|
||||
|
||||
```python
|
||||
# project variables
|
||||
PROJECT_SOURCE_FILE_FOLDER = '/Users/yeqing/projects/crawlab/spiders' # 爬虫源码根目录
|
||||
PROJECT_DEPLOY_FILE_FOLDER = '/var/crawlab' # 爬虫部署根目录
|
||||
PROJECT_LOGS_FOLDER = '/var/logs/crawlab' # 日志目录
|
||||
PROJECT_TMP_FOLDER = '/tmp' # 临时文件目录
|
||||
|
||||
# celery variables
|
||||
BROKER_URL = 'redis://192.168.99.100:6379/0' # 中间者URL,连接redis
|
||||
CELERY_RESULT_BACKEND = 'mongodb://192.168.99.100:27017/' # CELERY后台URL
|
||||
CELERY_MONGODB_BACKEND_SETTINGS = {
|
||||
'database': 'crawlab_test',
|
||||
'taskmeta_collection': 'tasks_celery',
|
||||
}
|
||||
CELERY_TIMEZONE = 'Asia/Shanghai'
|
||||
CELERY_ENABLE_UTC = True
|
||||
|
||||
# flower variables
|
||||
FLOWER_API_ENDPOINT = 'http://localhost:5555/api' # Flower服务地址
|
||||
|
||||
# database variables
|
||||
MONGO_HOST = '192.168.99.100'
|
||||
MONGO_PORT = 27017
|
||||
MONGO_DB = 'crawlab_test'
|
||||
|
||||
# flask variables
|
||||
DEBUG = True
|
||||
FLASK_HOST = '127.0.0.1'
|
||||
FLASK_PORT = 8000
|
||||
```
|
||||
|
||||
启动后端API,也就是一个Flask App,可以直接启动,或者用gunicorn代替。
|
||||
|
||||
```bash
|
||||
python app.py
|
||||
```
|
||||
|
||||
启动Flower服务(抱歉目前集成Flower到App服务中,必须单独启动来获取节点信息,后面的版本不需要这个操作)。
|
||||
|
||||
```bash
|
||||
python ./bin/run_flower.py
|
||||
```
|
||||
|
||||
启动本地Worker。在其他节点中如果想只是想执行任务的话,只需要启动这一个服务就可以了。
|
||||
|
||||
```bash
|
||||
python ./bin/run_worker.py
|
||||
```
|
||||
|
||||
启动前端服务器。
|
||||
|
||||
```bash
|
||||
cd ../frontend
|
||||
npm run serve
|
||||
```
|
||||
167
gitbook/README.md
Normal file
167
gitbook/README.md
Normal file
@@ -0,0 +1,167 @@
|
||||
# Crawlab
|
||||
基于Celery的爬虫分布式爬虫管理平台,支持多种编程语言以及多种爬虫框架.
|
||||
|
||||
[查看演示 Demo](http://139.129.230.98:8080)
|
||||
|
||||
[English Documentation](https://github.com/tikazyq/crawlab/blob/master/README.md)
|
||||
|
||||
## 要求
|
||||
- Python3
|
||||
- MongoDB
|
||||
- Redis
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
# 安装后台类库
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
```bash
|
||||
# 安装前台类库
|
||||
cd frontend
|
||||
npm install
|
||||
```
|
||||
|
||||
## 配置
|
||||
|
||||
请更改配置文件`config.py`,配置API和数据库连接.
|
||||
|
||||
## 快速开始
|
||||
```bash
|
||||
# 启动后端API
|
||||
python app.py
|
||||
|
||||
# 启动Flower服务
|
||||
python ./bin/run_flower.py
|
||||
|
||||
# 启动worker
|
||||
python ./bin/run_worker.py
|
||||
```
|
||||
|
||||
```bash
|
||||
# 运行前端
|
||||
cd frontend
|
||||
npm run serve
|
||||
```
|
||||
|
||||
## 截图
|
||||
|
||||
#### 首页
|
||||

|
||||
|
||||
#### 爬虫列表
|
||||
|
||||

|
||||
|
||||
#### 爬虫详情 - 概览
|
||||
|
||||

|
||||
|
||||
#### 任务详情 - 抓取结果
|
||||
|
||||

|
||||
|
||||
## 架构
|
||||
|
||||
Crawlab的架构跟Celery非常相似,但是加入了包括前端、爬虫、Flower在内的额外模块,以支持爬虫管理的功能。
|
||||
|
||||

|
||||
|
||||
### 节点
|
||||
|
||||
节点其实就是Celery中的Worker。一个节点运行时会连接到一个任务队列(例如Redis)来接收和运行任务。所有爬虫需要在运行时被部署到节点上,用户在部署前需要定义节点的IP地址和端口。
|
||||
|
||||
### 爬虫
|
||||
|
||||
##### 自动发现
|
||||
|
||||
在`config.py`文件中,修改变量`PROJECT_SOURCE_FILE_FOLDER`作为爬虫项目所在的目录。Crawlab后台程序会自动发现这些爬虫项目并储存到数据库中。是不是很方便?
|
||||
|
||||
##### 部署爬虫
|
||||
|
||||
所有爬虫需要在抓取前被部署当相应当节点中。在"爬虫详情"页面点击"Deploy"按钮,爬虫将被部署到所有有效到节点中。
|
||||
|
||||
##### 运行爬虫
|
||||
|
||||
部署爬虫之后,你可以在"爬虫详情"页面点击"Run"按钮来启动爬虫。一个爬虫任务将被触发,你可以在任务列表页面中看到这个任务。
|
||||
|
||||
### 任务
|
||||
|
||||
任务被触发并被节点执行。用户可以在任务详情页面中看到任务到状态、日志和抓取结果。
|
||||
|
||||
### 后台应用
|
||||
|
||||
这是一个Flask应用,提供了必要的API来支持常规操作,例如CRUD、爬虫部署以及任务运行。每一个节点需要启动Flask应用来支持爬虫部署。运行`python manage.py app`或`python ./bin/run_app.py`来启动应用。
|
||||
|
||||
### 中间者
|
||||
|
||||
中间者跟Celery中定义的一样,作为运行异步任务的队列。
|
||||
|
||||
### 前端
|
||||
|
||||
前端其实就是一个基于[Vue-Element-Admin](https://github.com/PanJiaChen/vue-element-admin)的单页应用。其中重用了很多Element-UI的控件来支持相应的展示。
|
||||
|
||||
## 与其他框架的集成
|
||||
|
||||
任务是利用python的`subprocess`模块中的`Popen`来实现的。任务ID将以环境变量`CRAWLAB_TASK_ID`的形式存在于爬虫任务运行的进程中,并以此来关联抓取数据。
|
||||
|
||||
在你的爬虫程序中,你需要将`CRAWLAB_TASK_ID`的值以`task_id`作为可以存入数据库中。这样Crawlab就直到如何将爬虫任务与抓取数据关联起来了。当前,Crawlab只支持MongoDB。
|
||||
|
||||
### Scrapy
|
||||
|
||||
以下是Crawlab跟Scrapy集成的例子,利用了Crawlab传过来的task_id和collection_name。
|
||||
|
||||
```python
|
||||
import os
|
||||
from pymongo import MongoClient
|
||||
|
||||
MONGO_HOST = '192.168.99.100'
|
||||
MONGO_PORT = 27017
|
||||
MONGO_DB = 'crawlab_test'
|
||||
|
||||
# scrapy example in the pipeline
|
||||
class JuejinPipeline(object):
|
||||
mongo = MongoClient(host=MONGO_HOST, port=MONGO_PORT)
|
||||
db = mongo[MONGO_DB]
|
||||
col_name = os.environ.get('CRAWLAB_COLLECTION')
|
||||
if not col_name:
|
||||
col_name = 'test'
|
||||
col = db[col_name]
|
||||
|
||||
def process_item(self, item, spider):
|
||||
item['task_id'] = os.environ.get('CRAWLAB_TASK_ID')
|
||||
self.col.save(item)
|
||||
return item
|
||||
```
|
||||
|
||||
## 与其他框架比较
|
||||
|
||||
限制以及有一些爬虫管理框架了,因此为啥还要用Crawlab?
|
||||
|
||||
因为很多现有当平台都依赖于Scrapyd,限制了爬虫的编程语言以及框架,爬虫工程师只能用scrapy和python。当然,scrapy是非常优秀的爬虫框架,但是它不能做一切事情。
|
||||
|
||||
Crawlab使用起来很方便,也很通用,可以适用于几乎任何主流语言和框架。它还有一个精美的前端界面,让用户可以方便的管理和运行爬虫。
|
||||
|
||||
|框架 | 类型 | 分布式 | 前端 | 依赖于Scrapyd |
|
||||
|:---:|:---:|:---:|:---:|:---:|
|
||||
| [Crawlab](https://github.com/tikazyq/crawlab) | 管理平台 | Y | Y | N
|
||||
| [Gerapy](https://github.com/Gerapy/Gerapy) | 管理平台 | Y | Y | Y
|
||||
| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | 管理平台 | Y | Y | Y
|
||||
| [ScrapydWeb](https://github.com/my8100/scrapydweb) | 管理平台 | Y | Y | Y
|
||||
| [Scrapyd](https://github.com/scrapy/scrapyd) | 网络服务 | Y | N | N/A
|
||||
|
||||
## TODOs
|
||||
##### 后端
|
||||
- [ ] 文件管理
|
||||
- [ ] MySQL数据库支持
|
||||
- [ ] 重跑任务
|
||||
- [ ] 节点监控
|
||||
- [ ] 更多爬虫例子
|
||||
|
||||
##### 前端
|
||||
- [ ] 任务数据统计
|
||||
- [ ] 表格过滤
|
||||
- [x] 多语言支持 (中文)
|
||||
- [ ] 登录和用户管理
|
||||
- [ ] 全局搜索
|
||||
18
gitbook/SUMMARY.md
Normal file
18
gitbook/SUMMARY.md
Normal file
@@ -0,0 +1,18 @@
|
||||
# Summary
|
||||
|
||||
* [简介](README.md)
|
||||
* [快速开始](QuickStart/README.md)
|
||||
* [安装](QuickStart/Installation.md)
|
||||
* [运行](QuickStart/Run.md)
|
||||
* [概念](Concept/README.md)
|
||||
* [节点](Concept/Node.md)
|
||||
* [爬虫](Concept/Spider.md)
|
||||
* [任务](Concept/Task.md)
|
||||
* [部署](Concept/Deploy.md)
|
||||
* [架构](Architecture/README.md)
|
||||
* [Celery](Architecture/Celery.md)
|
||||
* [App](Architecture/App.md)
|
||||
* [Examples](Examples/README.md)
|
||||
* [与Scrapy集成](Examples/README.md)
|
||||
* [与Puppeteer集成](Examples/README.md)
|
||||
|
||||
BIN
gitbook/img/crawlab-architecture.png
Normal file
BIN
gitbook/img/crawlab-architecture.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
BIN
gitbook/img/payment.jpg
Normal file
BIN
gitbook/img/payment.jpg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 78 KiB |
BIN
gitbook/img/用户使用流程图.png
Normal file
BIN
gitbook/img/用户使用流程图.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 89 KiB |
Reference in New Issue
Block a user