git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-22 17:31:03 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #107 from tikazyq/develop

Browse Source 
changed image urls
This commit is contained in:
Marvin Zhang
2019-08-03 12:55:31 +08:00
committed by GitHub
parent 868a52c14a bd727b73b1
commit c448682a59
2 changed files with 22 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

22
README-zh.md
View File

@@ -97,49 +97,49 @@ Docker部署的详情，请见[相关文档](https://tikazyq.github.io/crawlab/I
#### 登录
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/login.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/login.png)
#### 首页
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/home.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/home.png)
#### 节点列表
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-list.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/node-list.png)
#### 节点拓扑图
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-network.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/node-network.png)
#### 爬虫列表
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-list.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-list.png)
#### 爬虫概览
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-overview.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-overview.png)
#### 爬虫分析
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-analytics.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-analytics.png)
#### 爬虫文件
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-file.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-file.png)
#### 任务详情 - 抓取结果
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0_1">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/task-results.png)
#### 定时任务
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/schedule.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/schedule.png)
## 架构
Crawlab的架构包括了一个主节点（Master Node）和多个工作节点（Worker Node），以及负责通信和数据储存的Redis和MongoDB数据库。
![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/architecture.png)
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/architecture.png)
前端应用向主节点请求数据，主节点通过MongoDB和Redis来执行任务派发调度以及部署，工作节点收到任务之后，开始执行爬虫任务，并将任务结果储存到MongoDB。架构相对于`v0.3.0`之前的Celery版本有所精简，去除了不必要的节点监控模块Flower，节点监控主要由Redis完成。

22
README.md
View File

@@ -95,49 +95,49 @@ For Docker Deployment details, please refer to [relevant documentation](https://
#### Login
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/login.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/login.png)
#### Home Page
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/home.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/home.png)
#### Node List
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-list.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/node-list.png)
#### Node Network
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-network.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/node-network.png)
#### Spider List
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-list.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-list.png)
#### Spider Overview
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-overview.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-overview.png)
#### Spider Analytics
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-analytics.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-analytics.png)
#### Spider Files
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-file.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/spider-file.png)
#### Task Results
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0_1">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/task-results.png)
#### Cron Job
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/schedule.png?v0.3.0">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/schedule.png)
## Architecture
The architecture of Crawlab is consisted of the Master Node and multiple Worker Nodes, and Redis and MongoDB databases which are mainly for nodes communication and data storage.
<img src="https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/architecture.png">
![](https://raw.githubusercontent.com/tikazyq/crawlab-docs/master/images/architecture.png)
The frontend app makes requests to the Master Node, which assigns tasks and deploys spiders through MongoDB and Redis. When a Worker Node receives a task, it begins to execute the crawling task, and stores the results to MongoDB. The architecture is much more concise compared with versions before `v0.3.0`. It has removed unnecessary Flower module which offers node monitoring services. They are now done by Redis.