Merge pull request #16 from crawlab-team/develop

Develop
2026-01-22 17:31:03 +01:00 · 2019-12-25 09:04:39 +08:00
parent 232c299949 f96f623fc1
commit b337dacdd8
59 changed files with 1058 additions and 440 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,33 @@
+# 0.4.2 (unknown)
+### Features / Enhancement
+- **Disclaimer**. Added page for Disclaimer.
+- **Call API to fetch version**. [#371](https://github.com/crawlab-team/crawlab/issues/371)
+- **Configure to allow user registration**. [#346](https://github.com/crawlab-team/crawlab/issues/346)
+- **Allow adding new users**.
+
+### Bug Fixes
+- **"mongodb no reachable" error**. [#373](https://github.com/crawlab-team/crawlab/issues/373)
+
+# 0.4.1 (2019-12-13)
+### Features / Enhancement
+- **Spiderfile Optimization**. Stages changed from dictionary to array. [#358](https://github.com/crawlab-team/crawlab/issues/358)
+- **Baidu Tongji Update**.
+
+### Bug Fixes
+- **Unable to display schedule tasks**. [#353](https://github.com/crawlab-team/crawlab/issues/353)
+- **Duplicate node registration**. [#334](https://github.com/crawlab-team/crawlab/issues/334)
+
+# 0.4.0 (2019-12-06)
+### Features / Enhancement
+- **Configurable Spider**. Allow users to add spiders using *Spiderfile* to configure crawling rules.
+- **Execution Mode**. Allow users to select 3 modes for task execution: *All Nodes*, *Selected Nodes* and *Random*.
+
+### Bug Fixes
+- **Task accidentally killed**. [#306](https://github.com/crawlab-team/crawlab/issues/306)
+- **Documentation fix**. [#301](https://github.com/crawlab-team/crawlab/issues/258) [#301](https://github.com/crawlab-team/crawlab/issues/258)
+- **Direct deploy incompatible with Windows**. [#288](https://github.com/crawlab-team/crawlab/issues/288)
+- **Log files lost**. [#269](https://github.com/crawlab-team/crawlab/issues/269)
+
 # 0.3.5 (2019-10-28)
 ### Features / Enhancement
 - **Graceful Showdown**. [detail](https://github.com/crawlab-team/crawlab/commit/63fab3917b5a29fd9770f9f51f1572b9f0420385)
--- a/DISCLAIMER-zh.md
+++ b/DISCLAIMER-zh.md
@@ -0,0 +1,12 @@
+# 免责声明
+
+本免责及隐私保护声明(以下简称“免责声明”或“本声明”)适用于 Crawlab 开发组 (以下简称“开发组”)研发的系列软件(以下简称"Crawlab") 在您阅读本声明后若不同意此声明中的任何条款，或对本声明存在质疑，请立刻停止使用我们的软件。若您已经开始或正在使用 Crawlab，则表示您已阅读并同意本声明的所有条款之约定。
+
+1. 总则：您通过安装 Crawlab 并使用 Crawlab 提供的服务与功能即表示您已经同意与开发组立本协议。开发组可随时执行全权决定更改“条款”。经修订的“条款”一经在 Github 免责声明页面上公布后，立即自动生效。
+2. 本产品是基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。
+3. 一切因使用 Crawlab 而引致之任何意外、疏忽、合约毁坏、诽谤、版权或知识产权侵犯及其所造成的损失(包括在非官方站点下载 Crawlab 而感染电脑病毒)，Crawlab 开发组概不负责，亦不承担任何法律责任。
+4. 用户对使用 Crawlab 自行承担风险，我们不做任何形式的保证， 因网络状况、通讯线路等任何技术原因而导致用户不能正常升级更新，我们也不承担任何法律责任。
+5. 用户使用 Crawlab 对目标网站进行抓取时需遵从《网络安全法》等与爬虫相关的法律法规，切勿擅自采集公民个人信息、用 DDoS 等方式造成目标网站瘫痪、不遵从目标网站的 robots.txt 协议等非法手段。
+6. Crawlab 尊重并保护所有用户的个人隐私权，不会窃取任何用户计算机中的信息。
+7. 系统的版权：Crawlab 开发组对所有开发的或合作开发的产品拥有知识产权，著作权，版权和使用权，这些产品受到适用的知识产权、版权、商标、服务商标、专利或其他法律的保护。
+8. 传播:任何公司或个人在网络上发布，传播我们软件的行为都是允许的，但因公司或个人传播软件可能造成的任何法律和刑事事件 Crawlab 开发组不负任何责任。
--- a/DISCLAIMER.md
+++ b/DISCLAIMER.md
@@ -0,0 +1,12 @@
+# Disclaimer
+
+This Disclaimer and privacy protection statement (hereinafter referred to as "disclaimer statement" or "this statement") is applicable to the series of software (hereinafter referred to as "crawlab") developed by crawlab development group (hereinafter referred to as "development group") after you read this statement, if you do not agree with any terms in this statement or have doubts about this statement, please stop using our software immediately. If you have started or are using crawlab, you have read and agree to all terms of this statement.
+
+1. General: by installing crawlab and using the services and functions provided by crawlab, you have agreed to establish this agreement with the development team. The developer group may at any time change the terms at its sole discretion. The amended "terms" shall take effect automatically as soon as they are published on the GitHub disclaimer page.
+2. This product is a distributed crawler management platform based on golang, supporting python, nodejs, go, Java, PHP and other programming languages as well as a variety of crawler frameworks.
+3. The development team of crawlab shall not be responsible for any accident, negligence, contract damage, defamation, copyright or intellectual property infringement caused by the use of crawlab and any loss caused by it (including computer virus infection caused by downloading crawlab on the unofficial site), and shall not bear any legal responsibility.
+4. The user shall bear the risk of using crawlab by himself, we do not make any form of guarantee, and we will not bear any legal responsibility for the user's failure to upgrade and update normally due to any technical reasons such as network condition and communication line.
+5. When users use crawlab to grab the target website, they need to comply with the laws and regulations related to crawlers, such as the network security law. Do not collect personal information of citizens without authorization, cause the target website to be paralyzed by DDoS, or fail to comply with the robots.txt protocol and other illegal means of the target website.
+6. Crawlab respects and protects the personal privacy of all users and will not steal any information from users' computers.
+7. Copyright of the system: the crawleb development team owns the intellectual property rights, copyrights, copyrights and use rights for all developed or jointly developed products, which are protected by applicable intellectual property rights, copyrights, trademarks, service trademarks, patents or other laws.
+8. Communication: any company or individual who publishes or disseminates our software on the Internet is allowed, but the crawlab development team shall not be responsible for any legal and criminal events that may be caused by the company or individual disseminating the software.
--- a/1
+++ b/1
@@ -37,7 +37,6 @@ RUN apt-get update \
 RUN pip install scrapy pymongo bs4 requests -i https://pypi.tuna.tsinghua.edu.cn/simple

 # copy backend files
-COPY --from=backend-build /go/src/app .
 COPY --from=backend-build /go/bin/crawlab /usr/local/bin

 # install nginx
--- a/Dockerfile.local
+++ b/Dockerfile.local
@@ -4,16 +4,17 @@ WORKDIR /go/src/app
 COPY ./backend .

 ENV GO111MODULE on
-ENV GOPROXY https://mirrors.aliyun.com/goproxy/
+ENV GOPROXY https://goproxy.io

 RUN go install -v ./...

-FROM node:8.16.0 AS frontend-build
+FROM node:8.16.0-alpine AS frontend-build

 ADD ./frontend /app
 WORKDIR /app

 # install frontend
+RUN npm config set unsafe-perm true
 RUN npm install -g yarn && yarn install --registry=https://registry.npm.taobao.org

 RUN npm run build:prod
@@ -36,7 +37,6 @@ RUN apt-get update \
 RUN pip install scrapy pymongo bs4 requests -i https://pypi.tuna.tsinghua.edu.cn/simple

 # copy backend files
-COPY --from=backend-build /go/src/app .
 COPY --from=backend-build /go/bin/crawlab /usr/local/bin

 # install nginx
@@ -56,4 +56,4 @@ EXPOSE 8080
 EXPOSE 8000

 # start backend
-CMD ["/bin/sh", "/app/docker_init.sh"]
+CMD ["/bin/sh", "/app/docker_init.sh"]
--- a/README-zh.md
+++ b/README-zh.md
@@ -10,7 +10,7 @@

 中文 | [English](https://github.com/crawlab-team/crawlab)

-[安装](#安装) | [运行](#运行) | [截图](#截图) | [架构](#架构) | [集成](#与其他框架的集成) | [比较](#与其他框架比较) | [相关文章](#相关文章) | [社区&赞助](#社区--赞助)
+[安装](#安装) | [运行](#运行) | [截图](#截图) | [架构](#架构) | [集成](#与其他框架的集成) | [比较](#与其他框架比较) | [相关文章](#相关文章) | [社区&赞助](#社区--赞助) | [免责声明](https://github.com/crawlab-team/crawlab/blob/master/DISCLAIMER-zh.md)

 基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。

--- a/README.md
+++ b/README.md
@@ -10,7 +10,7 @@

 [中文](https://github.com/crawlab-team/crawlab/blob/master/README-zh.md) | English

-[Installation](#installation) | [Run](#run) | [Screenshot](#screenshot) | [Architecture](#architecture) | [Integration](#integration-with-other-frameworks) | [Compare](#comparison-with-other-frameworks) | [Community & Sponsorship](#community--sponsorship)
+[Installation](#installation) | [Run](#run) | [Screenshot](#screenshot) | [Architecture](#architecture) | [Integration](#integration-with-other-frameworks) | [Compare](#comparison-with-other-frameworks) | [Community & Sponsorship](#community--sponsorship) | [Disclaimer](https://github.com/crawlab-team/crawlab/blob/master/DISCLAIMER.md)

 Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium.

--- a/backend/conf/config.yml
+++ b/backend/conf/config.yml
@@ -32,3 +32,6 @@ task:
  workers: 4
 other:
  tmppath: "/tmp"
+version: 0.4.1
+setting:
+  allowRegister: "N"
--- a/backend/database/mongo.go
+++ b/backend/database/mongo.go
@@ -61,10 +61,36 @@ func InitMongo() error {
 			dialInfo.Password = mongoPassword
 			dialInfo.Source = mongoAuth
 		}
-		sess, err := mgo.DialWithInfo(&dialInfo)
-		if err != nil {
-			return err
+
+		// mongo session
+		var sess *mgo.Session
+
+		// 错误次数
+		errNum := 0
+
+		// 重复尝试连接mongo
+		for {
+			var err error
+
+			// 连接mongo
+			sess, err = mgo.DialWithInfo(&dialInfo)
+
+			if err != nil {
+				// 如果连接错误，休息1秒，错误次数+1
+				time.Sleep(1 * time.Second)
+				errNum++
+
+				// 如果错误次数超过30，返回错误
+				if errNum >= 30 {
+					return err
+				}
+			} else {
+				// 如果没有错误，退出循环
+				break
+			}
 		}
+
+		// 赋值给全局mongo session
 		Session = sess
 	}
 	return nil
--- a/backend/entity/config_spider.go
+++ b/backend/entity/config_spider.go
@@ -5,7 +5,7 @@ type ConfigSpiderData struct {
 	Engine     string            `yaml:"engine" json:"engine"`
 	StartUrl   string            `yaml:"start_url" json:"start_url"`
 	StartStage string            `yaml:"start_stage" json:"start_stage"`
-	Stages     map[string]Stage  `yaml:"stages" json:"stages"`
+	Stages     []Stage           `yaml:"stages" json:"stages"`
 	Settings   map[string]string `yaml:"settings" json:"settings"`
 }

--- a/backend/main.go
+++ b/backend/main.go
@@ -114,9 +114,9 @@ func main() {
 		app.Use(middlewares.CORSMiddleware())
 		anonymousGroup := app.Group("/")
 		{
-			anonymousGroup.POST("/login", routes.Login)  // 用户登录
-			anonymousGroup.PUT("/users", routes.PutUser) // 添加用户
-
+			anonymousGroup.POST("/login", routes.Login)       // 用户登录
+			anonymousGroup.PUT("/users", routes.PutUser)      // 添加用户
+			anonymousGroup.GET("/setting", routes.GetSetting) // 获取配置信息
 		}
 		authGroup := app.Group("/", middlewares.AuthorizationMiddleware())
 		{
@@ -176,6 +176,8 @@ func main() {
 			authGroup.POST("/users/:id", routes.PostUser)     // 更改用户
 			authGroup.DELETE("/users/:id", routes.DeleteUser) // 删除用户
 			authGroup.GET("/me", routes.GetMe)                // 获取自己账户
+			// release版本
+			authGroup.GET("/version", routes.GetVersion) // 获取发布的版本
 		}

 	}
--- a/backend/mock/schedule.go
+++ b/backend/mock/schedule.go
@@ -10,12 +10,15 @@ import (
 	"time"
 )

+var NodeIdss = []bson.ObjectId{bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
+	bson.ObjectIdHex("5d429e6c19f7abede924fee1")}
+
 var scheduleList = []model.Schedule{
 	{
 		Id:       bson.ObjectId("5d429e6c19f7abede924fee2"),
 		Name:     "test schedule",
 		SpiderId: "123",
-		NodeId:   bson.ObjectId("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
@@ -29,7 +32,7 @@ var scheduleList = []model.Schedule{
 		Id:       bson.ObjectId("xx429e6c19f7abede924fee2"),
 		Name:     "test schedule2",
 		SpiderId: "234",
-		NodeId:   bson.ObjectId("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
@@ -100,8 +103,10 @@ func PutSchedule(c *gin.Context) {
 	}

 	// 如果node_id为空，则置为空ObjectId
-	if item.NodeId == "" {
-		item.NodeId = bson.ObjectIdHex(constants.ObjectIdNull)
+	for _, NodeId := range item.NodeIds {
+		if NodeId == "" {
+			NodeId = bson.ObjectIdHex(constants.ObjectIdNull)
+		}
 	}

 	c.JSON(http.StatusOK, Response{
--- a/backend/mock/schedule_test.go
+++ b/backend/mock/schedule_test.go
@@ -75,7 +75,7 @@ func TestPostSchedule(t *testing.T) {
 		Id:       bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
 		Name:     "test schedule",
 		SpiderId: bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
-		NodeId:   bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
@@ -112,7 +112,7 @@ func TestPutSchedule(t *testing.T) {
 		Id:       bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
 		Name:     "test schedule",
 		SpiderId: bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
-		NodeId:   bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
--- a/backend/model/config_spider/common.go
+++ b/backend/model/config_spider/common.go
@@ -15,16 +15,12 @@ func GetAllFields(data entity.ConfigSpiderData) []entity.Field {
 func GetStartStageName(data entity.ConfigSpiderData) string {
 	// 如果 start_stage 设置了且在 stages 里，则返回
 	if data.StartStage != "" {
-		for stageName := range data.Stages {
-			if stageName == data.StartStage {
-				return data.StartStage
-			}
-		}
+		return data.StartStage
 	}

 	// 否则返回第一个 stage
-	for stageName := range data.Stages {
-		return stageName
+	for _, stage := range data.Stages {
+		return stage.Name
 	}
 	return ""
 }
--- a/backend/model/config_spider/scrapy.go
+++ b/backend/model/config_spider/scrapy.go
@@ -83,7 +83,8 @@ func (g ScrapyGenerator) ProcessSpider() error {

 	// 替换 parsers
 	strParser := ""
-	for stageName, stage := range g.ConfigData.Stages {
+	for _, stage := range g.ConfigData.Stages {
+		stageName := stage.Name
 		stageStr := g.GetParserString(stageName, stage)
 		strParser += stageStr
 	}
--- a/backend/model/node.go
+++ b/backend/model/node.go
@@ -55,7 +55,7 @@ func GetCurrentNode() (Node, error) {
 	for {
 		// 如果错误次数超过10次
 		if errNum >= 10 {
-			panic("cannot get current node")
+			return node, errors.New("cannot get current node")
 		}

 		// 尝试获取节点
--- a/backend/model/schedule.go
+++ b/backend/model/schedule.go
@@ -12,15 +12,18 @@ import (
 )

 type Schedule struct {
-	Id          bson.ObjectId `json:"_id" bson:"_id"`
-	Name        string        `json:"name" bson:"name"`
-	Description string        `json:"description" bson:"description"`
-	SpiderId    bson.ObjectId `json:"spider_id" bson:"spider_id"`
-	NodeId      bson.ObjectId `json:"node_id" bson:"node_id"`
-	NodeKey     string        `json:"node_key" bson:"node_key"`
-	Cron        string        `json:"cron" bson:"cron"`
-	EntryId     cron.EntryID  `json:"entry_id" bson:"entry_id"`
-	Param       string        `json:"param" bson:"param"`
+	Id          bson.ObjectId   `json:"_id" bson:"_id"`
+	Name        string          `json:"name" bson:"name"`
+	Description string          `json:"description" bson:"description"`
+	SpiderId    bson.ObjectId   `json:"spider_id" bson:"spider_id"`
+	//NodeId      bson.ObjectId   `json:"node_id" bson:"node_id"`
+	//NodeKey     string          `json:"node_key" bson:"node_key"`
+	Cron        string          `json:"cron" bson:"cron"`
+	EntryId     cron.EntryID    `json:"entry_id" bson:"entry_id"`
+	Param       string          `json:"param" bson:"param"`
+	RunType     string          `json:"run_type" bson:"run_type"`
+	NodeIds     []bson.ObjectId `json:"node_ids" bson:"node_ids"`
+
 	// 状态
 	Status string `json:"status" bson:"status"`

@@ -49,26 +52,26 @@ func (sch *Schedule) Delete() error {
 	return c.RemoveId(sch.Id)
 }

-func (sch *Schedule) SyncNodeIdAndSpiderId(node Node, spider Spider) {
-	sch.syncNodeId(node)
-	sch.syncSpiderId(spider)
-}
+//func (sch *Schedule) SyncNodeIdAndSpiderId(node Node, spider Spider) {
+//	sch.syncNodeId(node)
+//	sch.syncSpiderId(spider)
+//}

-func (sch *Schedule) syncNodeId(node Node) {
-	if node.Id.Hex() == sch.NodeId.Hex() {
-		return
-	}
-	sch.NodeId = node.Id
-	_ = sch.Save()
-}
+//func (sch *Schedule) syncNodeId(node Node) {
+//	if node.Id.Hex() == sch.NodeId.Hex() {
+//		return
+//	}
+//	sch.NodeId = node.Id
+//	_ = sch.Save()
+//}

-func (sch *Schedule) syncSpiderId(spider Spider) {
-	if spider.Id.Hex() == sch.SpiderId.Hex() {
-		return
-	}
-	sch.SpiderId = spider.Id
-	_ = sch.Save()
-}
+//func (sch *Schedule) syncSpiderId(spider Spider) {
+//	if spider.Id.Hex() == sch.SpiderId.Hex() {
+//		return
+//	}
+//	sch.SpiderId = spider.Id
+//	_ = sch.Save()
+//}

 func GetScheduleList(filter interface{}) ([]Schedule, error) {
 	s, c := database.GetCol("schedules")
@@ -81,20 +84,20 @@ func GetScheduleList(filter interface{}) ([]Schedule, error) {

 	var schs []Schedule
 	for _, schedule := range schedules {
-		// 获取节点名称
-		if schedule.NodeId == bson.ObjectIdHex(constants.ObjectIdNull) {
-			// 选择所有节点
-			schedule.NodeName = "All Nodes"
-		} else {
-			// 选择单一节点
-			node, err := GetNode(schedule.NodeId)
-			if err != nil {
-				schedule.Status = constants.ScheduleStatusError
-				schedule.Message = constants.ScheduleStatusErrorNotFoundNode
-			} else {
-				schedule.NodeName = node.Name
-			}
-		}
+		// TODO: 获取节点名称
+		//if schedule.NodeId == bson.ObjectIdHex(constants.ObjectIdNull) {
+		//	// 选择所有节点
+		//	schedule.NodeName = "All Nodes"
+		//} else {
+		//	// 选择单一节点
+		//	node, err := GetNode(schedule.NodeId)
+		//	if err != nil {
+		//		schedule.Status = constants.ScheduleStatusError
+		//		schedule.Message = constants.ScheduleStatusErrorNotFoundNode
+		//	} else {
+		//		schedule.NodeName = node.Name
+		//	}
+		//}

 		// 获取爬虫名称
 		spider, err := GetSpider(schedule.SpiderId)
@@ -130,12 +133,13 @@ func UpdateSchedule(id bson.ObjectId, item Schedule) error {
 	if err := c.FindId(id).One(&result); err != nil {
 		return err
 	}
-	node, err := GetNode(item.NodeId)
-	if err != nil {
-		return err
-	}
+	//node, err := GetNode(item.NodeId)
+	//if err != nil {
+	//	return err
+	//}

-	item.NodeKey = node.Key
+	item.UpdateTs = time.Now()
+	//item.NodeKey = node.Key
 	if err := item.Save(); err != nil {
 		return err
 	}
@@ -146,15 +150,15 @@ func AddSchedule(item Schedule) error {
 	s, c := database.GetCol("schedules")
 	defer s.Close()

-	node, err := GetNode(item.NodeId)
-	if err != nil {
-		return err
-	}
+	//node, err := GetNode(item.NodeId)
+	//if err != nil {
+	//	return err
+	//}

 	item.Id = bson.NewObjectId()
 	item.CreateTs = time.Now()
 	item.UpdateTs = time.Now()
-	item.NodeKey = node.Key
+	//item.NodeKey = node.Key

 	if err := c.Insert(&item); err != nil {
 		debug.PrintStack()
--- a/backend/model/spider.go
+++ b/backend/model/spider.go
@@ -319,11 +319,5 @@ func GetConfigSpiderData(spider Spider) (entity.ConfigSpiderData, error) {
 		return configData, err
 	}

-	// 赋值 stage_name
-	for stageName, stage := range configData.Stages {
-		stage.Name = stageName
-		configData.Stages[stageName] = stage
-	}
-
 	return configData, nil
 }
--- a/backend/routes/setting.go
+++ b/backend/routes/setting.go
@@ -0,0 +1,33 @@
+package routes
+
+import (
+	"github.com/gin-gonic/gin"
+	"github.com/spf13/viper"
+	"net/http"
+)
+
+type SettingBody struct {
+	AllowRegister string `json:"allow_register"`
+}
+
+func GetVersion(c *gin.Context) {
+	version := viper.GetString("version")
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    version,
+	})
+}
+
+func GetSetting(c *gin.Context) {
+	allowRegister := viper.GetString("setting.allowRegister")
+
+	body := SettingBody{AllowRegister: allowRegister}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    body,
+	})
+}
--- a/backend/routes/task.go
+++ b/backend/routes/task.go
@@ -119,7 +119,6 @@ func PutTask(c *gin.Context) {
 				return
 			}
 		}
-
 	} else if reqBody.RunType == constants.RunTypeRandom {
 		// 随机
 		t := model.Task{
@@ -130,7 +129,6 @@ func PutTask(c *gin.Context) {
 			HandleError(http.StatusInternalServerError, c, err)
 			return
 		}
-
 	} else if reqBody.RunType == constants.RunTypeSelectedNodes {
 		// 指定节点
 		for _, nodeId := range reqBody.NodeIds {
@@ -145,7 +143,6 @@ func PutTask(c *gin.Context) {
 				return
 			}
 		}
-
 	} else {
 		HandleErrorF(http.StatusInternalServerError, c, "invalid run_type")
 		return
--- a/backend/routes/user.go
+++ b/backend/routes/user.go
@@ -21,6 +21,7 @@ type UserListRequestData struct {
 type UserRequestData struct {
 	Username string `json:"username"`
 	Password string `json:"password"`
+	Role     string `json:"role"`
 }

 func GetUser(c *gin.Context) {
@@ -88,11 +89,16 @@ func PutUser(c *gin.Context) {
 		return
 	}

+	// 默认为正常用户
+	if reqData.Role == "" {
+		reqData.Role = constants.RoleNormal
+	}
+
 	// 添加用户
 	user := model.User{
 		Username: strings.ToLower(reqData.Username),
 		Password: utils.EncryptPassword(reqData.Password),
-		Role:     constants.RoleNormal,
+		Role:     reqData.Role,
 	}
 	if err := user.Add(); err != nil {
 		HandleError(http.StatusInternalServerError, c, err)
--- a/backend/services/config_spider.go
+++ b/backend/services/config_spider.go
@@ -61,7 +61,9 @@ func ValidateSpiderfile(configData entity.ConfigSpiderData) error {

 	// 校验stages
 	dict := map[string]int{}
-	for stageName, stage := range configData.Stages {
+	for _, stage := range configData.Stages {
+		stageName := stage.Name
+
 		// stage 名称不能为空
 		if stageName == "" {
 			return errors.New("spiderfile invalid: stage name is empty")
@@ -152,12 +154,6 @@ func IsUniqueConfigSpiderFields(fields []entity.Field) bool {
 func ProcessSpiderFilesFromConfigData(spider model.Spider, configData entity.ConfigSpiderData) error {
 	spiderDir := spider.Src

-	// 赋值 stage_name
-	for stageName, stage := range configData.Stages {
-		stage.Name = stageName
-		configData.Stages[stageName] = stage
-	}
-
 	// 删除已有的爬虫文件
 	for _, fInfo := range utils.ListDir(spiderDir) {
 		// 不删除Spiderfile
--- a/backend/services/node.go
+++ b/backend/services/node.go
@@ -167,27 +167,34 @@ func UpdateNodeData() {
 		debug.PrintStack()
 		return
 	}
-	// 构造节点数据
-	data := Data{
-		Key:          key,
-		Mac:          mac,
-		Ip:           ip,
-		Master:       model.IsMaster(),
-		UpdateTs:     time.Now(),
-		UpdateTsUnix: time.Now().Unix(),
+
+	//先获取所有Redis的nodekey
+	list, _ := database.RedisClient.HKeys("nodes")
+
+	if i := utils.Contains(list, key); i == false {
+		// 构造节点数据
+		data := Data{
+			Key:          key,
+			Mac:          mac,
+			Ip:           ip,
+			Master:       model.IsMaster(),
+			UpdateTs:     time.Now(),
+			UpdateTsUnix: time.Now().Unix(),
+		}
+
+		// 注册节点到Redis
+		dataBytes, err := json.Marshal(&data)
+		if err != nil {
+			log.Errorf(err.Error())
+			debug.PrintStack()
+			return
+		}
+		if err := database.RedisClient.HSet("nodes", key, utils.BytesToString(dataBytes)); err != nil {
+			log.Errorf(err.Error())
+			return
+		}
 	}

-	// 注册节点到Redis
-	dataBytes, err := json.Marshal(&data)
-	if err != nil {
-		log.Errorf(err.Error())
-		debug.PrintStack()
-		return
-	}
-	if err := database.RedisClient.HSet("nodes", key, utils.BytesToString(dataBytes)); err != nil {
-		log.Errorf(err.Error())
-		return
-	}
 }

 func MasterNodeCallback(message redis.Message) (err error) {
--- a/backend/services/schedule.go
+++ b/backend/services/schedule.go
@@ -7,7 +7,7 @@ import (
 	"errors"
 	"github.com/apex/log"
 	"github.com/globalsign/mgo/bson"
-	"github.com/satori/go.uuid"
+	uuid "github.com/satori/go.uuid"
 	"runtime/debug"
 )

@@ -19,48 +19,87 @@ type Scheduler struct {

 func AddScheduleTask(s model.Schedule) func() {
 	return func() {
-		node, err := model.GetNodeByKey(s.NodeKey)
-		if err != nil || node.Id.Hex() == "" {
-			log.Errorf("get node by key error: %s", err.Error())
-			debug.PrintStack()
-			return
-		}
-
-		spider := model.GetSpiderByName(s.SpiderName)
-		if spider == nil || spider.Id.Hex() == "" {
-			log.Errorf("get spider by name error: %s", err.Error())
-			debug.PrintStack()
-			return
-		}
-
-		// 同步ID到定时任务
-		s.SyncNodeIdAndSpiderId(node, *spider)
-
 		// 生成任务ID
 		id := uuid.NewV4()

-		// 生成任务模型
-		t := model.Task{
-			Id:       id.String(),
-			SpiderId: spider.Id,
-			NodeId:   node.Id,
-			Status:   constants.StatusPending,
-			Param:    s.Param,
-		}
+		if s.RunType == constants.RunTypeAllNodes {
+			// 所有节点
+			nodes, err := model.GetNodeList(nil)
+			if err != nil {
+				return
+			}
+			for _, node := range nodes {
+				t := model.Task{
+					Id:       id.String(),
+					SpiderId: s.SpiderId,
+					NodeId:   node.Id,
+					Param:    s.Param,
+				}

-		// 将任务存入数据库
-		if err := model.AddTask(t); err != nil {
-			log.Errorf(err.Error())
-			debug.PrintStack()
+				if err := AddTask(t); err != nil {
+					return
+				}
+				if err := AssignTask(t); err != nil {
+					log.Errorf(err.Error())
+					debug.PrintStack()
+					return
+				}
+			}
+		} else if s.RunType == constants.RunTypeRandom {
+			// 随机
+			t := model.Task{
+				Id:       id.String(),
+				SpiderId: s.SpiderId,
+				Param:    s.Param,
+			}
+			if err := AddTask(t); err != nil {
+				return
+			}
+			if err := AssignTask(t); err != nil {
+				log.Errorf(err.Error())
+				debug.PrintStack()
+				return
+			}
+		} else if s.RunType == constants.RunTypeSelectedNodes {
+			// 指定节点
+			for _, nodeId := range s.NodeIds {
+				t := model.Task{
+					Id:       id.String(),
+					SpiderId: s.SpiderId,
+					NodeId:   nodeId,
+					Param:    s.Param,
+				}
+
+				if err := AddTask(t); err != nil {
+					return
+				}
+
+				if err := AssignTask(t); err != nil {
+					log.Errorf(err.Error())
+					debug.PrintStack()
+					return
+				}
+			}
+		} else {
 			return
 		}

-		// 加入任务队列
-		if err := AssignTask(t); err != nil {
-			log.Errorf(err.Error())
-			debug.PrintStack()
-			return
-		}
+		//node, err := model.GetNodeByKey(s.NodeKey)
+		//if err != nil || node.Id.Hex() == "" {
+		//	log.Errorf("get node by key error: %s", err.Error())
+		//	debug.PrintStack()
+		//	return
+		//}
+		//
+		//spider := model.GetSpiderByName(s.SpiderName)
+		//if spider == nil || spider.Id.Hex() == "" {
+		//	log.Errorf("get spider by name error: %s", err.Error())
+		//	debug.PrintStack()
+		//	return
+		//}
+		//
+		//// 同步ID到定时任务
+		//s.SyncNodeIdAndSpiderId(node, *spider)
 	}
 }

--- a/backend/template/spiderfile/Spiderfile.163_news
+++ b/backend/template/spiderfile/Spiderfile.163_news
@@ -4,17 +4,17 @@ start_url: "http://news.163.com/special/0001386F/rank_news.html"
 start_stage: "list"
 engine: "scrapy"
 stages:
-  list:
-    is_list: true
-    list_css: "table tr:not(:first-child)"
-    fields:
-      - name: "title"
-        css: "td:nth-child(1) > a"
-      - name: "url"
-        css: "td:nth-child(1) > a"
-        attr: "href"
-      - name: "clicks"
-        css: "td.cBlue"
+- name: list
+  is_list: true
+  list_css: "table tr:not(:first-child)"
+  fields:
+  - name: "title"
+    css: "td:nth-child(1) > a"
+  - name: "url"
+    css: "td:nth-child(1) > a"
+    attr: "href"
+  - name: "clicks"
+    css: "td.cBlue"
 settings:
  ROBOTSTXT_OBEY: false
  USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
--- a/backend/template/spiderfile/Spiderfile.baidu
+++ b/backend/template/spiderfile/Spiderfile.baidu
@@ -4,19 +4,19 @@ start_url: http://www.baidu.com/s?wd=crawlab
 start_stage: list
 engine: scrapy
 stages:
-  list:
-    is_list: true
-    list_xpath: //*[contains(@class, "c-container")]
-    page_xpath: //*[@id="page"]//a[@class="n"][last()]
-    page_attr: href
-    fields:
-      - name: title
-        xpath: .//h3/a
-      - name: url
-        xpath: .//h3/a
-        attr: href
-      - name: abstract
-        xpath: .//*[@class="c-abstract"]
+- name: list
+  is_list: true
+  list_xpath: //*[contains(@class, "c-container")]
+  page_xpath: //*[@id="page"]//a[@class="n"][last()]
+  page_attr: href
+  fields:
+  - name: title
+    xpath: .//h3/a
+  - name: url
+    xpath: .//h3/a
+    attr: href
+  - name: abstract
+    xpath: .//*[@class="c-abstract"]
 settings:
  ROBOTSTXT_OBEY: false
  USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
--- a/backend/template/spiderfile/Spiderfile.toscrapy_books
+++ b/backend/template/spiderfile/Spiderfile.toscrapy_books
@@ -4,25 +4,25 @@ start_url: "http://books.toscrape.com"
 start_stage: "list"
 engine: "scrapy"
 stages:
-  list:
-    is_list: true
-    list_css: "section article.product_pod"
-    page_css: "ul.pager li.next a"
-    page_attr: "href"
-    fields:
-      - name: "title"
-        css: "h3 > a"
-      - name: "url"
-        css: "h3 > a"
-        attr: "href"
-        next_stage: "detail"
-      - name: "price"
-        css: ".product_price > .price_color"
-  detail:
-    is_list: false
-    fields:
-      - name: "description"
-        css: "#product_description + p"
+- name: list
+  is_list: true
+  list_css: "section article.product_pod"
+  page_css: "ul.pager li.next a"
+  page_attr: "href"
+  fields:
+  - name: "title"
+    css: "h3 > a"
+  - name: "url"
+    css: "h3 > a"
+    attr: "href"
+    next_stage: "detail"
+  - name: "price"
+    css: ".product_price > .price_color"
+- name: detail
+  is_list: false
+  fields:
+  - name: "description"
+    css: "#product_description + p"
 settings:
  ROBOTSTXT_OBEY: true
  AUTOTHROTTLE_ENABLED: true
--- a/backend/utils/helpers.go
+++ b/backend/utils/helpers.go
@@ -6,6 +6,7 @@ import (
 	"github.com/apex/log"
 	"github.com/gomodule/redigo/redis"
 	"io"
+	"reflect"
 	"runtime/debug"
 	"unsafe"
 )
@@ -40,3 +41,20 @@ func Close(c io.Closer) {
 		//log.WithError(err).Error("关闭资源文件失败。")
 	}
 }
+
+func Contains(array interface{}, val interface{}) (fla bool) {
+	fla = false
+	switch reflect.TypeOf(array).Kind() {
+	case reflect.Slice:
+		{
+			s := reflect.ValueOf(array)
+			for i := 0; i < s.Len(); i++ {
+				if reflect.DeepEqual(val, s.Index(i).Interface()) {
+					fla = true
+					return
+				}
+			}
+		}
+	}
+	return
+}
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -4,37 +4,37 @@ services:
    image: tikazyq/crawlab:latest
    container_name: master
    environment:
-      CRAWLAB_API_ADDRESS: "http://localhost:8000"
-      CRAWLAB_SERVER_MASTER: "Y"
-      CRAWLAB_MONGO_HOST: "mongo"
-      CRAWLAB_REDIS_ADDRESS: "redis"
+      CRAWLAB_API_ADDRESS: "http://localhost:8000"  # backend API address 后端 API 地址，设置为 http://<宿主机IP>:<端口>，端口为映射出来的端口
+      CRAWLAB_SERVER_MASTER: "Y"  # whether to be master node 是否为主节点，主节点为 Y，工作节点为 N
+      CRAWLAB_MONGO_HOST: "mongo1"  # MongoDB host address MongoDB 的地址，在 docker compose 网络中，直接引用服务名称
+      CRAWLAB_REDIS_ADDRESS: "redis"  # Redis host address Redis 的地址，在 docker compose 网络中，直接引用服务名称
    ports:    
-      - "8080:8080" # frontend
-      - "8000:8000" # backend
+      - "8080:8080" # frontend port mapping 前端端口映射
+      - "8000:8000" # backend port mapping 后端端口映射 
    depends_on:
-      - mongo
+      - mongo1
      - redis
  worker:
    image: tikazyq/crawlab:latest
    container_name: worker
    environment:
      CRAWLAB_SERVER_MASTER: "N"
-      CRAWLAB_MONGO_HOST: "mongo"
+      CRAWLAB_MONGO_HOST: "mongo1"
      CRAWLAB_REDIS_ADDRESS: "redis"
    depends_on:
-      - mongo
+      - mongo1
      - redis
-  mongo:
+  mongo1:
    image: mongo:latest
    restart: always
-    volumes:
-      - "/opt/crawlab/mongo/data/db:/data/db"
-    ports:
-      - "27017:27017"
+    # volumes:
+    #   - "/opt/crawlab/mongo/data/db:/data/db"  # make data persistent 持久化
+    # ports:
+    #   - "27017:27017"  # expose port to host machine 暴露接口到宿主机
  redis:
    image: redis:latest
    restart: always
-    volumes:
-      - "/opt/crawlab/redis/data:/data"
-    ports:
-      - "6379:6379"
+    # volumes:
+    #   - "/opt/crawlab/redis/data:/data"  # make data persistent 持久化
+    # ports:
+    #   - "6379:6379"  # expose port to host machine 暴露接口到宿主机
--- a/docker_init.sh
+++ b/docker_init.sh
@@ -22,4 +22,5 @@ fi
 # start nginx
 service nginx start

+# start backend
 crawlab
--- a/frontend/package.json
+++ b/frontend/package.json
@@ -1,6 +1,6 @@
 {
  "name": "crawlab",
-  "version": "0.3.5",
+  "version": "0.4.1",
  "private": true,
  "scripts": {
    "serve": "vue-cli-service serve --ip=0.0.0.0 --mode=development",
@@ -29,6 +29,7 @@
    "normalize.css": "7.0.0",
    "nprogress": "0.2.0",
    "path": "^0.12.7",
+    "showdown": "^1.9.1",
    "vcrontab": "^0.3.3",
    "vue": "^2.5.22",
    "vue-ba": "^1.2.5",
--- a/frontend/src/App.vue
+++ b/frontend/src/App.vue
@@ -6,6 +6,9 @@
 </template>

 <script>
+import {
+  mapState
+} from 'vuex'
 import DialogView from './components/Common/DialogView'

 export default {
@@ -19,21 +22,25 @@ export default {
    DialogView
  },
  computed: {
+    ...mapState('setting', ['setting']),
    useStats () {
      return localStorage.getItem('useStats')
    }
  },
  methods: {},
+  created () {
+    this.$store.dispatch('setting/getSetting')
+  },
  mounted () {
    window.setUseStats = (value) => {
      localStorage.setItem('useStats', value)
      document.querySelector('.el-message__closeBtn').click()
      if (value === 1) {
        this.$st.sendPv('/allow_stats')
-        this.$st.sendEv('全局', '允许/禁止统计', 'value', 'allow')
+        this.$st.sendEv('全局', '允许/禁止统计', '允许')
      } else {
        this.$st.sendPv('/disallow_stats')
-        this.$st.sendEv('全局', '允许/禁止统计', 'value', 'disallow')
+        this.$st.sendEv('全局', '允许/禁止统计', '禁止')
      }
    }

--- a/frontend/src/components/Common/CrawlConfirmDialog.vue
+++ b/frontend/src/components/Common/CrawlConfirmDialog.vue
@@ -29,10 +29,14 @@
      <el-form-item :label="$t('Parameters')" prop="param" inline-message>
        <el-input v-model="form.param" :placeholder="$t('Parameters')"></el-input>
      </el-form-item>
+      <el-form-item class="disclaimer-wrapper">
+        <el-checkbox v-model="isAllowDisclaimer"/>
+        <span style="margin-left: 5px">我已阅读并同意 <a href="javascript:" @click="onClickDisclaimer">《免责声明》</a> 所有内容</span>
+      </el-form-item>
    </el-form>
    <template slot="footer">
      <el-button type="plain" size="small" @click="$emit('close')">{{$t('Cancel')}}</el-button>
-      <el-button type="primary" size="small" @click="onConfirm">{{$t('Confirm')}}</el-button>
+      <el-button type="primary" size="small" @click="onConfirm" :disabled="!isAllowDisclaimer">{{$t('Confirm')}}</el-button>
    </template>
  </el-dialog>
 </template>
@@ -59,7 +63,8 @@ export default {
        nodeIds: undefined,
        param: '',
        nodeList: []
-      }
+      },
+      isAllowDisclaimer: true
    }
  },
  methods: {
@@ -80,8 +85,11 @@ export default {
            this.$message.success(this.$t('A task has been scheduled successfully'))
          })
        this.$emit('close')
-        this.$st.sendEv('爬虫', '运行')
+        this.$st.sendEv('爬虫确认', '确认运行', this.form.runType)
      })
+    },
+    onClickDisclaimer () {
+      this.$router.push('/disclaimer')
    }
  },
  created () {
@@ -105,4 +113,8 @@ export default {
  .crawl-confirm-dialog >>> .el-form .el-form-item {
    margin-bottom: 20px;
  }
+
+  .crawl-confirm-dialog >>> .disclaimer-wrapper a {
+    color: #409eff;
+  }
 </style>
--- a/frontend/src/components/Config/ConfigList.vue
+++ b/frontend/src/components/Config/ConfigList.vue
@@ -68,9 +68,10 @@
                  v-model="spiderForm.config.start_stage"
                  :placeholder="$t('Start Stage')"
                  :class="startStageClass"
+                  @change="$st.sendEv('爬虫详情', '配置', '改变起始阶段')"
                >
                  <el-option
-                    v-for="n in Object.keys(spiderForm.config.stages)"
+                    v-for="n in spiderForm.config.stages.map(s => s.name)"
                    :key="n"
                    :value="n"
                    :label="n"
@@ -133,9 +134,9 @@
          :value="activeNames"
        >
          <el-collapse-item
-            v-for="(stage, stageName) in spiderForm.config.stages"
-            :key="stageName"
-            :name="stageName"
+            v-for="(stage, index) in spiderForm.config.stages"
+            :key="index"
+            :name="stage.name"
          >
            <template slot="title">
              <ul class="stage-list">
@@ -271,7 +272,7 @@
              title="List Page Fields"
              :fields="stage.fields"
              :stage="stage"
-              :stage-names="Object.keys(spiderForm.config.stages)"
+              :stage-names="spiderForm.config.stages.map(s => s.name)"
            />
          </el-collapse-item>
        </el-collapse>
@@ -397,7 +398,7 @@ export default {
    },
    isCss () {
      let i = 0
-      Object.values(this.spiderForm.config.stages).forEach(stage => {
+      this.spiderForm.config.stages.forEach(stage => {
        stage.fields.forEach(field => {
          if (!field.css) i++
        })
@@ -406,7 +407,7 @@ export default {
    },
    isXpath () {
      let i = 0
-      Object.values(this.spiderForm.config.stages).forEach(stage => {
+      this.spiderForm.config.stages.forEach(stage => {
        stage.fields.forEach(field => {
          if (!field.xpath) i++
        })
@@ -414,7 +415,7 @@ export default {
      return i === 0
    },
    activeNames () {
-      return Object.values(this.spiderForm.config.stages).map(d => d.name)
+      return this.spiderForm.config.stages.map(d => d.name)
    },
    startUrlClass () {
      if (!this.spiderForm.config.start_url) {
@@ -464,7 +465,7 @@ export default {

      // 加入剩余阶段
      i = 0
-      const restStages = Object.values(this.spiderForm.config.stages)
+      const restStages = this.spiderForm.config.stages
        .filter(stage => !allStageNames.has(stage.name))
      restStages.forEach(stage => {
        // 加入节点信息
@@ -479,17 +480,10 @@ export default {
      })

      return nodes
-      // const stages = Object.values(this.spiderForm.config.stages)
-      // return stages.map((stage, i) => {
-      //   return {
-      //     name: stage.name,
-      //     ...stage
-      //   }
-      // })
    },
    stageEdges () {
      const edges = []
-      const stages = Object.values(this.spiderForm.config.stages)
+      const stages = this.spiderForm.config.stages
      stages.forEach(stage => {
        for (let i = 0; i < stage.fields.length; i++) {
          const field = stage.fields[i]
@@ -509,15 +503,20 @@ export default {
      this.spiderForm.crawl_type = value
    },
    async onSave () {
-      this.$st.sendEv('爬虫详情-配置', '保存')
+      this.$st.sendEv('爬虫详情', '配置', '保存')
      this.saveLoading = true
      try {
-        await this.$store.dispatch('spider/postConfigSpiderConfig')
-        this.$message.success(this.$t('Spider info has been saved successfully'))
+        const res = await this.$store.dispatch('spider/postConfigSpiderConfig')
+        if (!res.data.error) {
+          this.$message.success(this.$t('Spider info has been saved successfully'))
+          return true
+        }
+        return false
      } catch (e) {
-        this.$message.error(this.$t('Something wrong happened'))
+        return false
+      } finally {
+        this.saveLoading = false
      }
-      this.saveLoading = false
    },
    onDialogClose () {
      this.dialogVisible = false
@@ -544,15 +543,17 @@ export default {
                .finally(() => {
                  this.previewLoading = false
                })
-              this.$st.sendEv('爬虫详情-配置', '预览')
+              this.$st.sendEv('爬虫详情', '配置', '预览')
            })
        }
      })
    },
    async onCrawl () {
-      await this.onSave()
-      this.crawlConfirmDialogVisible = true
-      this.$st.sendEv('爬虫详情-配置', '点击运行')
+      this.$st.sendEv('爬虫详情', '配置', '点击运行')
+      const res = await this.onSave()
+      if (res) {
+        this.crawlConfirmDialogVisible = true
+      }
    },
    onExtractFields () {
      this.$refs['form'].validate(res => {
@@ -580,7 +581,7 @@ export default {
                .finally(() => {
                  this.extractFieldsLoading = false
                })
-              this.$st.sendEv('爬虫详情-配置', '提取字段')
+              this.$st.sendEv('爬虫详情', '配置', '提取字段')
            })
        }
      })
@@ -595,7 +596,8 @@ export default {
      return value
    },
    onClickSelectorType (selectorType) {
-      Object.values(this.spiderForm.config.stages).forEach(stage => {
+      this.$st.sendEv('爬虫详情', '配置', `点击阶段选择器类别-${selectorType}`)
+      this.spiderForm.config.stages.forEach(stage => {
        // 列表
        if (selectorType === 'css') {
          if (stage.list_xpath) stage.list_xpath = ''
@@ -627,40 +629,29 @@ export default {
      })
    },
    onStageNameFocus (ev) {
-      console.log(ev)
      ev.stopPropagation()
-      // ev.preventDefault()
    },
    onEditStage (stage) {
+      this.$st.sendEv('爬虫详情', '配置', '更改阶段名称')
      this.$set(stage, 'isEdit', !stage.isEdit)
      setTimeout(() => {
        this.$refs[`stage-name-${stage.name}`][0].focus()
      }, 0)
    },
    onCopyStage (stage) {
+      this.$st.sendEv('爬虫详情', '配置', '复制阶段')
+      const stages = this.spiderForm.config.stages
      const ts = Math.floor(new Date().getTime()).toString()
      const newStage = JSON.parse(JSON.stringify(stage))
-      newStage.name = `stage_${ts}`
-      this.$set(this.spiderForm.config.stages, newStage.name, newStage)
-    },
-    onRemoveStage (stage) {
-      const stages = JSON.parse(JSON.stringify(this.spiderForm.config.stages))
-      delete stages[stage.name]
-      this.$set(this.spiderForm.config, 'stages', stages)
-      if (Object.keys(stages).length === 0) {
-        this.onAddStage()
+      newStage.name = `${stage.name}_copy_${ts}`
+      for (let i = 0; i < stages.length; i++) {
+        if (stage.name === stages[i].name) {
+          stages.splice(i + 1, 0, newStage)
+        }
      }
-      // 重置next_stage被设置为该stage的field
-      Object.values(this.spiderForm.config.stages).forEach(_stage => {
-        _stage.fields.forEach(field => {
-          if (field.next_stage === stage.name) {
-            this.$set(field, 'next_stage', '')
-          }
-        })
-      })
    },
-    onAddStage (stage) {
-      const stages = JSON.parse(JSON.stringify(this.spiderForm.config.stages))
+    addStage (index) {
+      const stages = this.spiderForm.config.stages
      const ts = Math.floor(new Date().getTime()).toString()
      const newStageName = `stage_${ts}`
      const newField = { name: `field_${ts}`, next_stage: '' }
@@ -671,15 +662,46 @@ export default {
      } else {
        newField['xpath'] = '//body'
      }
-      stages[newStageName] = {
+      stages.splice(index + 1, 0, {
        name: newStageName,
        list_css: this.isCss ? 'body' : '',
        list_xpath: this.isXpath ? '//body' : '',
        page_css: '',
        page_xpath: '',
        fields: [newField]
+      })
+    },
+    onRemoveStage (stage) {
+      this.$st.sendEv('爬虫详情', '配置', '删除阶段')
+      const stages = this.spiderForm.config.stages
+      for (let i = 0; i < stages.length; i++) {
+        if (stage.name === stages[i].name) {
+          stages.splice(i, 1)
+          break
+        }
+      }
+      // 如果只剩一个stage，加入新的stage
+      if (stages.length === 0) {
+        this.addStage(0)
+      }
+      // 重置next_stage被设置为该stage的field
+      stages.forEach(_stage => {
+        _stage.fields.forEach(field => {
+          if (field.next_stage === stage.name) {
+            this.$set(field, 'next_stage', '')
+          }
+        })
+      })
+    },
+    onAddStage (stage) {
+      this.$st.sendEv('爬虫详情', '配置', '添加阶段')
+      const stages = this.spiderForm.config.stages
+      for (let i = 0; i < stages.length; i++) {
+        if (stage.name === stages[i].name) {
+          this.addStage(i)
+          break
+        }
      }
-      this.$set(this.spiderForm.config, 'stages', stages)
    },
    renderProcessChart () {
      const option = {
@@ -797,10 +819,12 @@ ${f.css || f.xpath} ${f.attr ? ('(' + f.attr + ')') : ''} ${f.next_stage ? (' --
    },
    onCheckIsList (value, stage) {
      if (value) {
+        this.$st.sendEv('爬虫详情', '配置', '勾选列表页')
        if (!stage.list_css && !stage.list_xpath) {
          stage.list_xpath = '//body'
        }
      } else {
+        this.$st.sendEv('爬虫详情', '配置', '取消勾选列表页')
        stage.list_css = ''
        stage.list_xpath = ''
      }
@@ -822,10 +846,12 @@ ${f.css || f.xpath} ${f.attr ? ('(' + f.attr + ')') : ''} ${f.next_stage ? (' --
    },
    onCheckIsPage (value, stage) {
      if (value) {
+        this.$st.sendEv('爬虫详情', '配置', '勾选分页')
        if (!stage.page_css && !stage.page_xpath) {
          stage.page_xpath = '//body'
        }
      } else {
+        this.$st.sendEv('爬虫详情', '配置', '取消勾选分页')
        stage.page_css = ''
        stage.page_xpath = ''
      }
@@ -852,7 +878,7 @@ ${f.css || f.xpath} ${f.attr ? ('(' + f.attr + ')') : ''} ${f.next_stage ? (' --
    }
  },
  mounted () {
-    this.activeNames = Object.keys(this.spiderForm.config.stages)
+    this.activeNames = this.spiderForm.config.stages.map(stage => stage.name)
  }
 }
 </script>
--- a/frontend/src/components/Environment/EnvironmentList.vue
+++ b/frontend/src/components/Environment/EnvironmentList.vue
@@ -49,11 +49,11 @@ export default {
        name: '',
        value: ''
      })
-      this.$st.sendEv('爬虫详情-环境', '添加')
+      this.$st.sendEv('爬虫详情', '环境', '添加')
    },
    deleteEnv (index) {
      this.spiderForm.envs.splice(index, 1)
-      this.$st.sendEv('爬虫详情-环境', '删除')
+      this.$st.sendEv('爬虫详情', '环境', '删除')
    },
    save () {
      this.$store.dispatch('spider/editSpider')
@@ -63,7 +63,7 @@ export default {
        .catch(error => {
          this.$message.error(error)
        })
-      this.$st.sendEv('爬虫详情-环境', '保存')
+      this.$st.sendEv('爬虫详情', '环境', '保存')
    }
  }
 }
--- a/frontend/src/components/InfoView/SpiderInfoView.vue
+++ b/frontend/src/components/InfoView/SpiderInfoView.vue
@@ -112,7 +112,7 @@ export default {
  methods: {
    onCrawl () {
      this.crawlConfirmDialogVisible = true
-      this.$st.sendEv('爬虫详情-概览', '点击运行')
+      this.$st.sendEv('爬虫详情', '概览', '点击运行')
    },
    onSave () {
      this.$refs['spiderForm'].validate(res => {
@@ -126,7 +126,7 @@ export default {
            })
        }
      })
-      this.$st.sendEv('爬虫详情-概览', '保存')
+      this.$st.sendEv('爬虫详情', '概览', '保存')
    },
    fetchSiteSuggestions (keyword, callback) {
      this.$request.get('/sites', {
--- a/frontend/src/components/Overview/TaskOverview.vue
+++ b/frontend/src/components/Overview/TaskOverview.vue
@@ -52,11 +52,11 @@ export default {
  methods: {
    onClickNodeTitle () {
      this.$router.push(`/nodes/${this.nodeForm._id}`)
-      this.$st.sendEv('任务详情-概览', '点击节点详情')
+      this.$st.sendEv('任务详情', '概览', '点击节点详情')
    },
    onClickSpiderTitle () {
      this.$router.push(`/spiders/${this.spiderForm._id}`)
-      this.$st.sendEv('任务详情-概览', '点击爬虫详情')
+      this.$st.sendEv('任务详情', '概览', '点击爬虫详情')
    }
  },
  created () {
--- a/frontend/src/components/TableView/FieldsTableView.vue
+++ b/frontend/src/components/TableView/FieldsTableView.vue
@@ -1,11 +1,5 @@
 <template>
  <div class="fields-table-view">
-    <!--    <el-row class="button-group-container">-->
-    <!--      <label class="title">{{$t(this.title)}}</label>-->
-    <!--      <div class="button-group">-->
-    <!--        <el-button type="primary" size="small" @click="addField" icon="el-icon-plus">{{$t('Add Field')}}</el-button>-->
-    <!--      </div>-->
-    <!--    </el-row>-->
    <el-row>
      <el-table :data="fields"
                class="table edit"
@@ -171,32 +165,14 @@ export default {
    }
  },
  methods: {
-    addField () {
-      this.fields.push({
-        type: 'css',
-        extract_type: 'text'
-      })
-      this.$st.sendEv('爬虫详情-配置', '添加字段')
-    },
-    deleteField (index) {
-      this.fields.splice(index, 1)
-      this.$st.sendEv('爬虫详情-配置', '删除字段')
-    },
    onNameChange (row) {
      if (this.fields.filter(d => d.name === row.name).length > 1) {
        this.$message.error(this.$t(`Duplicated field names for ${row.name}`))
      }
-      this.$st.sendEv('爬虫详情-配置', '更改字段')
-    },
-    onCheck (row) {
-      this.fields.forEach(d => {
-        if (row.name !== d.name) {
-          this.$set(d, 'is_detail', false)
-        }
-      })
-      this.$st.sendEv('爬虫详情-配置', '设置详情页URL')
+      this.$st.sendEv('爬虫详情', '配置', '更改字段')
    },
    onClickSelectorType (row, selectorType) {
+      this.$st.sendEv('爬虫详情', '配置', `点击字段选择器类别-${selectorType}`)
      if (selectorType === 'css') {
        if (row.xpath) this.$set(row, 'xpath', '')
        if (!row.css) this.$set(row, 'css', 'body')
@@ -206,6 +182,7 @@ export default {
      }
    },
    onClickIsAttribute (row, isAttribute) {
+      this.$st.sendEv('爬虫详情', '配置', '设置字段属性')
      if (!isAttribute) {
        // 文本
        if (row.attr) this.$set(row, 'attr', '')
@@ -224,6 +201,7 @@ export default {
      }
    },
    onRemoveField (row) {
+      this.$st.sendEv('爬虫详情', '配置', '删除字段')
      for (let i = 0; i < this.fields.length; i++) {
        if (row.name === this.fields[i].name) {
          this.fields.splice(i, 1)
@@ -238,6 +216,7 @@ export default {
      }
    },
    onAddField (row) {
+      this.$st.sendEv('爬虫详情', '配置', '添加字段')
      for (let i = 0; i < this.fields.length; i++) {
        if (row.name === this.fields[i].name) {
          this.fields.splice(i + 1, 0, {
--- a/frontend/src/components/TableView/SettingFieldsTableView.vue
+++ b/frontend/src/components/TableView/SettingFieldsTableView.vue
@@ -77,51 +77,14 @@ export default {
    }
  },
  methods: {
-    addField () {
-      this.list.push({
-        type: 'css',
-        extract_type: 'text'
-      })
-      this.$st.sendEv('爬虫详情-配置', '添加字段')
-    },
-    deleteField (index) {
-      this.list.splice(index, 1)
-      this.$st.sendEv('爬虫详情-配置', '删除字段')
-    },
    onChange (row) {
      if (this.list.filter(d => d.name === row.name).length > 1) {
        this.$message.error(this.$t(`Duplicated field names for ${row.name}`))
      }
      this.$store.commit('spider/SET_SPIDER_FORM_CONFIG_SETTINGS', this.list)
-      this.$st.sendEv('爬虫详情-配置', '更改字段')
-    },
-    onCheck (row) {
-      this.list.forEach(d => {
-        if (row.name !== d.name) {
-          this.$set(d, 'is_detail', false)
-        }
-      })
-      this.$st.sendEv('爬虫详情-配置', '设置详情页URL')
-    },
-    onClickSelectorType (row, selectorType) {
-      if (selectorType === 'css') {
-        if (row.xpath) this.$set(row, 'xpath', '')
-        if (!row.css) this.$set(row, 'css', 'body')
-      } else {
-        if (row.css) this.$set(row, 'css', '')
-        if (!row.xpath) this.$set(row, 'xpath', '//body')
-      }
-    },
-    onClickIsAttribute (row, isAttribute) {
-      if (!isAttribute) {
-        // 文本
-        if (row.attr) this.$set(row, 'attr', '')
-      } else {
-        // 属性
-        if (!row.attr) this.$set(row, 'attr', 'href')
-      }
    },
    onRemoveField (row) {
+      this.$st.sendEv('爬虫详情', '配置', '删除设置')
      const list = JSON.parse(JSON.stringify(this.list))
      for (let i = 0; i < list.length; i++) {
        if (row.name === list[i].name) {
@@ -137,6 +100,7 @@ export default {
      this.$store.commit('spider/SET_SPIDER_FORM_CONFIG_SETTINGS', list)
    },
    onAddField (row) {
+      this.$st.sendEv('爬虫详情', '配置', '添加设置')
      const list = JSON.parse(JSON.stringify(this.list))
      for (let i = 0; i < list.length; i++) {
        if (row.name === list[i].name) {
--- a/frontend/src/i18n/zh.js
+++ b/frontend/src/i18n/zh.js
@@ -259,7 +259,7 @@ export default {
  'Executables': '执行文件',

  // 弹出框
-  Notification: '提示',
+  'Notification': '提示',
  'Are you sure to delete this node?': '你确定要删除该节点?',
  'Are you sure to run this spider?': '你确定要运行该爬虫?',
  'Node info has been saved successfully': '节点信息已成功保存',
@@ -270,7 +270,7 @@ export default {
  'Do you allow us to collect some statistics to improve Crawlab?': '您允许我们收集统计数据以更好地优化Crawlab？',
  'Saved file successfully': '成功保存文件',
  'An error happened when fetching the data': '请求数据时出错',
-  'Error when logging in (Please check username and password)': '登录时出错（请检查用户名密码）',
+  'Error when logging in (Please read documentation Q&A)': '登录时出错（请查看文档 Q&A）',
  'Please enter the correct username': '请输入正确用户名',
  'Password length should be no shorter than 5': '密码长度不能小于5',
  'Two passwords must be the same': '两个密码必须要一致',
@@ -279,6 +279,9 @@ export default {
  'Saved successfully': '成功保存',
  'Please zip your spider files from the root directory': '爬虫文件请从根目录下开始压缩。',
  'English': 'English',
+  'Are you sure to delete the schedule task?': '确定删除定时任务?',
+  'Disclaimer': '免责声明',
+
  // 登录
  'Sign in': '登录',
  'Sign-in': '登录',
--- a/frontend/src/router/index.js
+++ b/frontend/src/router/index.js
@@ -191,6 +191,25 @@ export const constantRouterMap = [
      }
    ]
  },
+  {
+    path: '/disclaimer',
+    component: Layout,
+    meta: {
+      title: 'Disclaimer',
+      icon: 'fa fa-exclamation-triangle'
+    },
+    children: [
+      {
+        path: '',
+        name: 'Disclaimer',
+        component: () => import('../views/doc/Disclaimer'),
+        meta: {
+          title: 'Disclaimer',
+          icon: 'fa fa-exclamation-triangle'
+        }
+      }
+    ]
+  },

  { path: '*', redirect: '/404', hidden: true }
 ]
--- a/frontend/src/store/index.js
+++ b/frontend/src/store/index.js
@@ -13,6 +13,7 @@ import schedule from './modules/schedule'
 import lang from './modules/lang'
 import site from './modules/site'
 import stats from './modules/stats'
+import setting from './modules/setting'
 import getters from './getters'

 Vue.use(Vuex)
@@ -31,6 +32,7 @@ const store = new Vuex.Store({
    schedule,
    lang,
    site,
+    setting,
    // 百度统计
    stats
  },
--- a/frontend/src/store/modules/schedule.js
+++ b/frontend/src/store/modules/schedule.js
@@ -1,7 +1,9 @@
 import request from '../../api/request'
 const state = {
  scheduleList: [],
-  scheduleForm: {}
+  scheduleForm: {
+    node_ids: []
+  }
 }

 const getters = {}
--- a/frontend/src/store/modules/setting.js
+++ b/frontend/src/store/modules/setting.js
@@ -0,0 +1,28 @@
+import request from '../../api/request'
+
+const state = {
+  setting: {}
+}
+
+const getters = {}
+
+const mutations = {
+  SET_SETTING (state, value) {
+    state.setting = value
+  }
+}
+
+const actions = {
+  async getSetting ({ commit }) {
+    const res = await request.get('/setting')
+    commit('SET_SETTING', res.data.data)
+  }
+}
+
+export default {
+  namespaced: true,
+  state,
+  getters,
+  mutations,
+  actions
+}
--- a/frontend/src/store/modules/user.js
+++ b/frontend/src/store/modules/user.js
@@ -138,6 +138,11 @@ const user = {
    // 编辑用户
    editUser ({ state }) {
      return request.post(`/users/${state.userForm._id}`, state.userForm)
+    },
+
+    // 添加用户
+    addUser ({ dispatch, commit, state }) {
+      return request.put('/users', state.userForm)
    }
  }
 }
--- a/frontend/src/views/doc/Disclaimer.vue
+++ b/frontend/src/views/doc/Disclaimer.vue
@@ -0,0 +1,71 @@
+<template>
+  <div class="app-container disclaimer">
+    <el-card>
+      <div v-html="text"></div>
+    </el-card>
+  </div>
+</template>
+
+<script>
+import {
+  mapState
+} from 'vuex'
+import showdown from 'showdown'
+
+export default {
+  name: 'Disclaimer',
+  computed: {
+    ...mapState('lang', [
+      'lang'
+    ]),
+    text () {
+      if (!this.converter) return ''
+      if (this.lang === 'zh') {
+        return this.converter.makeHtml(this.textZh)
+      } else {
+        return this.converter.makeHtml(this.textEn)
+      }
+    }
+  },
+  data () {
+    const converter = new showdown.Converter()
+    return {
+      converter,
+      textEn: `
+# Disclaimer
+
+This Disclaimer and privacy protection statement (hereinafter referred to as "disclaimer statement" or "this statement") is applicable to the series of software (hereinafter referred to as "crawlab") developed by crawlab development group (hereinafter referred to as "development group") after you read this statement, if you do not agree with any terms in this statement or have doubts about this statement, please stop using our software immediately. If you have started or are using crawlab, you have read and agree to all terms of this statement.
+
+1. General: by installing crawlab and using the services and functions provided by crawlab, you have agreed to establish this agreement with the development team. The developer group may at any time change the terms at its sole discretion. The amended "terms" shall take effect automatically as soon as they are published on the GitHub disclaimer page.
+2. This product is a distributed crawler management platform based on golang, supporting python, nodejs, go, Java, PHP and other programming languages as well as a variety of crawler frameworks.
+3. The development team of crawlab shall not be responsible for any accident, negligence, contract damage, defamation, copyright or intellectual property infringement caused by the use of crawlab and any loss caused by it (including computer virus infection caused by downloading crawlab on the unofficial site), and shall not bear any legal responsibility.
+4. The user shall bear the risk of using crawlab by himself, we do not make any form of guarantee, and we will not bear any legal responsibility for the user's failure to upgrade and update normally due to any technical reasons such as network condition and communication line.
+5. When users use crawlab to grab the target website, they need to comply with the laws and regulations related to crawlers, such as the network security law. Do not collect personal information of citizens without authorization, cause the target website to be paralyzed by DDoS, or fail to comply with the robots.txt protocol and other illegal means of the target website.
+6. Crawlab respects and protects the personal privacy of all users and will not steal any information from users' computers.
+7. Copyright of the system: the crawleb development team owns the intellectual property rights, copyrights, copyrights and use rights for all developed or jointly developed products, which are protected by applicable intellectual property rights, copyrights, trademarks, service trademarks, patents or other laws.
+8. Communication: any company or individual who publishes or disseminates our software on the Internet is allowed, but the crawlab development team shall not be responsible for any legal and criminal events that may be caused by the company or individual disseminating the software.
+      `,
+      textZh: `
+# 免责声明
+
+本免责及隐私保护声明(下简称“免责声明”或“本声明”)适用于 Crawlab 开发组 (以下简称“开发组”)研发的系列软件(以下简称"Crawlab") 在您阅读本声明后若不同意此声明中的任何条款，或对本声明存在质疑，请立刻停止使用我们的软件。若您已经开始或正在使用 Crawlab，则表示您已阅读并同意本声明的所有条款之约定。
+
+1. 总则：您通过安装 Crawlab 并使用 Crawlab 提供的服务与功能即表示您已经同意与开发组立本协议。开发组可随时执行全权决定更改“条款”。经修订的“条款”一经在 Github 免责声明页面上公布后，立即自动生效。
+2. 本产品是基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。
+3. 一切因使用 Crawlab 而引致之任何意外、疏忽、合约毁坏、诽谤、版权或知识产权侵犯及其所造成的损失(包括在非官方站点下载 Crawlab 而感染电脑病毒)，Crawlab 开发组概不负责，亦不承担任何法律责任。
+4. 用户对使用 Crawlab 自行承担风险，我们不做任何形式的保证， 因网络状况、通讯线路等任何技术原因而导致用户不能正常升级更新，我们也不承担任何法律责任。
+5. 用户使用 Crawlab 对目标网站进行抓取时需遵从《网络安全法》等与爬虫相关的法律法规，切勿擅自采集公民个人信息、用 DDoS 等方式造成目标网站瘫痪、不遵从目标网站的 robots.txt 协议等非法手段。
+6. Crawlab 尊重并保护所有用户的个人隐私权，不会窃取任何用户计算机中的信息。
+7. 系统的版权：Crawlab 开发组对所有开发的或合作开发的产品拥有知识产权，著作权，版权和使用权，这些产品受到适用的知识产权、版权、商标、服务商标、专利或其他法律的保护。
+8. 传播:任何公司或个人在网络上发布，传播我们软件的行为都是允许的，但因公司或个人传播软件可能造成的任何法律和刑事事件 Crawlab 开发组不负任何责任。
+      `
+    }
+  }
+}
+</script>
+
+<style scoped>
+  .disclaimer >>> ol li {
+    margin: 10px 0;
+  }
+</style>
--- a/frontend/src/views/layout/components/Navbar.vue
+++ b/frontend/src/views/layout/components/Navbar.vue
@@ -8,9 +8,9 @@
        <i class="el-icon-arrow-down el-icon--right"></i>
      </span>
      <el-dropdown-menu slot="dropdown" class="user-dropdown">
-        <el-dropdown-item>
-          <span style="display:block;">v0.3.5</span>
-        </el-dropdown-item>
+<!--        <el-dropdown-item>-->
+<!--          <span style="display:block;">v0.4.1</span>-->
+<!--        </el-dropdown-item>-->
        <el-dropdown-item>
          <span style="display:block;" @click="logout">{{$t('Logout')}}</span>
        </el-dropdown-item>
@@ -31,7 +31,7 @@
      </el-dropdown-menu>
    </el-dropdown>
    <el-dropdown class="documentation right">
-      <a href="https://tikazyq.github.io/crawlab-docs" target="_blank">
+      <a href="https://docs.crawlab.cn" target="_blank">
        <font-awesome-icon :icon="['far', 'question-circle']"/>
        <span style="margin-left: 5px;">{{$t('Documentation')}}</span>
      </a>
@@ -74,7 +74,7 @@ export default {
      this.$i18n.locale = lang
      this.$store.commit('lang/SET_LANG', lang)

-      this.$st.sendEv('全局', '切换中英文', 'lang', lang)
+      this.$st.sendEv('全局', '切换中英文', lang)
    }
  }
 }
--- a/frontend/src/views/layout/components/Sidebar/index.vue
+++ b/frontend/src/views/layout/components/Sidebar/index.vue
@@ -1,7 +1,7 @@
 <template>
  <el-scrollbar wrap-class="scrollbar-wrapper">
    <div class="sidebar-logo" :class="isCollapse ? 'collapsed' : ''">
-      <span>C</span><span v-show="!isCollapse">rawlab</span>
+      <span>C</span><span v-show="!isCollapse">rawlab<span class="version">v{{version}}</span></span>
    </div>
    <el-menu
      :show-timeout="200"
@@ -48,6 +48,15 @@ export default {
    isCollapse () {
      return !this.sidebar.opened
    }
+  },
+  data () {
+    return {
+      version: ''
+    }
+  },
+  async created () {
+    const res = await this.$request.get('/version')
+    this.version = res.data.data
  }
 }
 </script>
@@ -73,4 +82,10 @@ export default {
  .sidebar-container .sidebar-logo.collapsed {
    padding-left: 8px;
  }
+
+  .sidebar-container .sidebar-logo .version {
+    margin-left: 5px;
+    font-weight: normal;
+    font-size: 12px;
+  }
 </style>
--- a/frontend/src/views/login/index.vue
+++ b/frontend/src/views/login/index.vue
@@ -48,7 +48,7 @@
        <div class="left">
          <span v-if="!isSignUp" class="forgot-password">{{$t('Forgot Password')}}</span>
        </div>
-        <div class="right">
+        <div class="right" v-if="setting.allow_register === 'Y'">
          <span v-if="isSignUp">{{$t('Has Account')}}, </span>
          <span v-if="isSignUp" class="sign-in" @click="$router.push('/login')">{{$t('Sign-in')}} ></span>
          <span v-if="!isSignUp">{{$t('New to Crawlab')}}, </span>
@@ -57,8 +57,8 @@
      </div>
      <div class="tips">
        <span>{{$t('Initial Username/Password')}}: admin/admin</span>
-        <a href="https://github.com/tikazyq/crawlab" target="_blank" style="float:right">
-          <img src="https://img.shields.io/badge/github-crawlab-blue">
+        <a href="https://github.com/crawlab-team/crawlab" target="_blank" style="float:right">
+          <img src="https://img.shields.io/github/stars/crawlab-team/crawlab?logo=github">
        </a>
      </div>
      <div class="lang">
@@ -116,6 +116,9 @@ export default {
    }
  },
  computed: {
+    ...mapState('setting', [
+      'setting'
+    ]),
    ...mapState('lang', [
      'lang'
    ]),
@@ -136,7 +139,7 @@ export default {
            this.$router.push({ path: this.redirect || '/' })
            this.$store.dispatch('user/getInfo')
          }).catch(() => {
-            this.$message.error(this.$t('Error when logging in (Please check username and password)'))
+            this.$message.error(this.$t('Error when logging in (Please read documentation Q&A)'))
            this.loading = false
          })
        }
--- a/frontend/src/views/node/NodeList.vue
+++ b/frontend/src/views/node/NodeList.vue
@@ -202,7 +202,7 @@ export default {
    },
    onRefresh () {
      this.$store.dispatch('node/getNodeList')
-      this.$st.sendEv('节点', '刷新')
+      this.$st.sendEv('节点列表', '刷新')
    },
    onSubmit () {
      const vm = this
@@ -246,13 +246,13 @@ export default {
              message: 'Deleted successfully'
            })
          })
-        this.$st.sendEv('节点', '删除', 'id', row._id)
+        this.$st.sendEv('节点列表', '删除节点')
      })
    },
    onView (row) {
      this.$router.push(`/nodes/${row._id}`)

-      this.$st.sendEv('节点', '查看', 'id', row._id)
+      this.$st.sendEv('节点列表', '查看节点')
    },
    onPageChange () {
      this.$store.dispatch('node/getNodeList')
--- a/frontend/src/views/schedule/ScheduleList.vue
+++ b/frontend/src/views/schedule/ScheduleList.vue
@@ -14,9 +14,15 @@
        <el-form-item :label="$t('Schedule Name')" prop="name" required>
          <el-input v-model="scheduleForm.name" :placeholder="$t('Schedule Name')"></el-input>
        </el-form-item>
-        <el-form-item :label="$t('Node')" prop="node_id" required>
-          <el-select v-model="scheduleForm.node_id">
-            <!--<el-option :label="$t('All Nodes')" value="000000000000000000000000"></el-option>-->
+        <el-form-item :label="$t('Run Type')" prop="run_type" required>
+          <el-select v-model="scheduleForm.run_type" :placeholder="$t('Run Type')">
+            <el-option value="all-nodes" :label="$t('All Nodes')"/>
+            <el-option value="selected-nodes" :label="$t('Selected Nodes')"/>
+            <el-option value="random" :label="$t('Random')"/>
+          </el-select>
+        </el-form-item>
+        <el-form-item v-if="scheduleForm.run_type === 'selected-nodes'" :label="$t('Nodes')" prop="node_ids" required>
+          <el-select v-model="scheduleForm.node_ids" :placeholder="$t('Nodes')" multiple filterable>
            <el-option
              v-for="op in nodeList"
              :key="op._id"
@@ -27,30 +33,19 @@
          </el-select>
        </el-form-item>
        <el-form-item :label="$t('Spider')" prop="spider_id" required>
-          <el-select v-model="scheduleForm.spider_id" filterable>
+          <el-select v-model="scheduleForm.spider_id" :placeholder="$t('Spider')" filterable>
            <el-option
              v-for="op in spiderList"
              :key="op._id"
              :value="op._id"
-              :label="op.name"
-              :disabled="!op.cmd"
+              :label="`${op.display_name} (${op.name})`"
+              :disabled="isDisabledSpider(op)"
            >
            </el-option>
          </el-select>
        </el-form-item>
-        <!--:rules="cronRules"-->
-        <el-form-item :label="$t('schedules.cron')" prop="cron"  required>
-          <!--<template slot="label">-->
-            <!--<el-tooltip :content="$t('schedules.cron_format')"-->
-                        <!--placement="top">-->
-              <!--<span>-->
-                <!--{{$t('schedules.cron')}}-->
-                <!--<i class="fa fa-exclamation-circle"></i>-->
-              <!--</span>-->
-            <!--</el-tooltip>-->
-          <!--</template>-->
-          <el-input style="padding-right:10px"
-                    v-model="scheduleForm.cron"
+        <el-form-item :label="$t('Cron')" prop="cron" required>
+          <el-input v-model="scheduleForm.cron"
                    :placeholder="$t('schedules.cron')">
          </el-input>
          <!--<el-button size="small" style="width:100px" type="primary" @click="onShowCronDialog">{{$t('schedules.add_cron')}}</el-button>-->
@@ -79,7 +74,7 @@

    <!--cron generation popup-->
    <!--<el-dialog title="生成 Cron" :visible.sync="showCron">-->
-      <!--<vcrontab @hide="showCron=false" @fill="onCrontabFill" :expression="expression"></vcrontab>-->
+    <!--<vcrontab @hide="showCron=false" @fill="onCrontabFill" :expression="expression"></vcrontab>-->
    <!--</el-dialog>-->

    <el-card style="border-radius: 0">
@@ -120,6 +115,13 @@
              </el-tag>
            </template>
          </el-table-column>
+          <el-table-column v-else-if="col.name === 'run_type'" :key="col.name" :label="$t(col.label)">
+            <template slot-scope="scope">
+              <template v-if="scope.row.run_type === 'all-nodes'">{{$t('All Nodes')}}</template>
+              <template v-else-if="scope.row.run_type === 'selected-nodes'">{{$t('Selected Nodes')}}</template>
+              <template v-else-if="scope.row.run_type === 'random'">{{$t('Random')}}</template>
+            </template>
+          </el-table-column>
          <el-table-column v-else :key="col.name"
                           :property="col.name"
                           :label="$t(col.label)"
@@ -141,7 +143,7 @@
            <el-tooltip :content="$t('Remove')" placement="top">
              <el-button type="danger" icon="el-icon-delete" size="mini" @click="onRemove(scope.row)"></el-button>
            </el-tooltip>
-            <el-tooltip content="暂停/运行" placement="top">
+            <el-tooltip v-if="false" :content="$t(getStatusTooltip(scope.row))" placement="top">
              <el-button type="success" icon="fa fa-bug" size="mini" @click="onCrawl(scope.row)"></el-button>
            </el-tooltip>
          </template>
@@ -165,7 +167,8 @@ export default {
    return {
      columns: [
        { name: 'name', label: 'Name', width: '180' },
-        { name: 'cron', label: 'schedules.cron', width: '120' },
+        { name: 'cron', label: 'Cron', width: '120' },
+        { name: 'run_type', label: 'Run Type', width: '150' },
        { name: 'node_name', label: 'Node', width: '150' },
        { name: 'spider_name', label: 'Spider', width: '150' },
        { name: 'param', label: 'Parameters', width: '150' },
@@ -208,8 +211,8 @@ export default {
    onAdd () {
      this.isEdit = false
      this.dialogVisible = true
-      this.$store.commit('schedule/SET_SCHEDULE_FORM', {})
-      this.$st.sendEv('定时任务', '添加')
+      this.$store.commit('schedule/SET_SCHEDULE_FORM', { node_ids: [] })
+      this.$st.sendEv('定时任务', '添加定时任务')
    },
    onAddSubmit () {
      this.$refs.scheduleForm.validate(res => {
@@ -235,7 +238,7 @@ export default {
          }
        }
      })
-      this.$st.sendEv('定时任务', '提交')
+      this.$st.sendEv('定时任务', '提交定时任务')
    },
    isShowRun (row) {
    },
@@ -243,12 +246,12 @@ export default {
      this.$store.commit('schedule/SET_SCHEDULE_FORM', row)
      this.dialogVisible = true
      this.isEdit = true
-      this.$st.sendEv('定时任务', '修改', 'id', row._id)
+      this.$st.sendEv('定时任务', '修改定时任务')
    },
    onRemove (row) {
-      this.$confirm('确定删除定时任务?', '提示', {
-        confirmButtonText: '确定',
-        cancelButtonText: '取消',
+      this.$confirm(this.$t('Are you sure to delete the schedule task?'), this.$t('Notification'), {
+        confirmButtonText: this.$t('Confirm'),
+        cancelButtonText: this.$t('Cancel'),
        type: 'warning'
      }).then(() => {
        this.$store.dispatch('schedule/removeSchedule', row._id)
@@ -258,15 +261,16 @@ export default {
              this.$message.success(`Schedule "${row.name}" has been removed`)
            }, 100)
          })
-      }).catch(() => {})
-      this.$st.sendEv('定时任务', '删除', 'id', row._id)
+      }).catch(() => {
+      })
+      this.$st.sendEv('定时任务', '删除定时任务')
    },
    onCrawl (row) {
      // 停止定时任务
      if (!row.status || row.status === 'running') {
-        this.$confirm('确定停止定时任务?', '提示', {
-          confirmButtonText: '确定',
-          cancelButtonText: '取消',
+        this.$confirm(this.$t('Are you sure to delete the schedule task?'), this.$t('Notification'), {
+          confirmButtonText: this.$t('Confirm'),
+          cancelButtonText: this.$t('Cancel'),
          type: 'warning'
        }).then(() => {
          this.$store.dispatch('schedule/stopSchedule', row._id)
@@ -280,13 +284,14 @@ export default {
                message: resp.data.error
              })
            })
-        }).catch(() => {})
+        }).catch(() => {
+        })
      }
      // 运行定时任务
      if (row.status === 'stop') {
-        this.$confirm('确定运行定时任务?', '提示', {
-          confirmButtonText: '确定',
-          cancelButtonText: '取消',
+        this.$confirm(this.$t('Are you sure to delete the schedule task?'), this.$t('Notification'), {
+          confirmButtonText: this.$t('Confirm'),
+          cancelButtonText: this.$t('Cancel'),
          type: 'warning'
        }).then(() => {
          this.$store.dispatch('schedule/runSchedule', row._id)
@@ -300,7 +305,24 @@ export default {
                message: resp.data.error
              })
            })
-        }).catch(() => {})
+        }).catch(() => {
+        })
+      }
+    },
+    isDisabledSpider (spider) {
+      if (spider.type === 'customized') {
+        return !spider.cmd
+      } else {
+        return false
+      }
+    },
+    getStatusTooltip (row) {
+      if (row.status === 'stop') {
+        return 'Start'
+      } else if (row.status === 'running') {
+        return 'Stop'
+      } else if (row.status === 'error') {
+        return 'Start'
      }
    }
  },
@@ -338,6 +360,7 @@ export default {
    min-height: 360px;
    margin-top: 10px;
  }
+
  .status-tag {
    cursor: pointer;
  }
--- a/frontend/src/views/spider/SpiderList.vue
+++ b/frontend/src/views/spider/SpiderList.vue
@@ -370,17 +370,17 @@ export default {
        }
        await this.$store.dispatch('spider/getSpiderList')
        this.$router.push(`/spiders/${res2.data.data._id}`)
-        this.$st.sendEv('爬虫', '添加爬虫-可配置爬虫')
+        this.$st.sendEv('爬虫列表', '添加爬虫', '可配置爬虫')
      })
    },
    onAddCustomized () {
      this.addDialogVisible = false
      this.addCustomizedDialogVisible = true
-      this.$st.sendEv('爬虫', '添加爬虫-自定义爬虫')
+      this.$st.sendEv('爬虫列表', '添加爬虫', '自定义爬虫')
    },
    onRefresh () {
      this.getList()
-      this.$st.sendEv('爬虫', '刷新')
+      this.$st.sendEv('爬虫列表', '刷新')
    },
    onSubmit () {
      const vm = this
@@ -434,19 +434,19 @@ export default {
              message: 'Deleted successfully'
            })
          })
-        this.$st.sendEv('爬虫', '删除')
+        this.$st.sendEv('爬虫列表', '删除爬虫')
      })
    },
    onCrawl (row, ev) {
      ev.stopPropagation()
      this.crawlConfirmDialogVisible = true
      this.activeSpiderId = row._id
-      this.$st.sendEv('爬虫', '点击运行')
+      this.$st.sendEv('爬虫列表', '点击运行')
    },
    onView (row, ev) {
      ev.stopPropagation()
      this.$router.push('/spiders/' + row._id)
-      this.$st.sendEv('爬虫', '查看')
+      this.$st.sendEv('爬虫列表', '查看爬虫')
    },
    onImport () {
      this.$refs.importForm.validate(valid => {
@@ -467,7 +467,7 @@ export default {
            })
        }
      })
-      this.$st.sendEv('爬虫', '导入爬虫')
+      this.$st.sendEv('爬虫列表', '导入爬虫')
    },
    openImportDialog () {
      this.dialogVisible = true
@@ -495,10 +495,6 @@ export default {
        callback(data)
      })
    },
-    onSiteSelect (item) {
-      this.$store.commit('spider/SET_FILTER_SITE', item._id)
-      this.$st.sendEv('爬虫', '搜索网站')
-    },
    onAddConfigurableSiteSelect (item) {
      this.spiderForm.site = item._id
    },
--- a/frontend/src/views/task/TaskDetail.vue
+++ b/frontend/src/views/task/TaskDetail.vue
@@ -97,7 +97,7 @@ export default {
    },
    downloadCSV () {
      this.$store.dispatch('task/getTaskResultExcel', this.$route.params.id)
-      this.$st.sendEv('任务详情-结果', '下载CSV')
+      this.$st.sendEv('任务详情', '结果', '下载CSV')
    },
    getTaskLog () {
      if (this.$route.params.id) {
--- a/frontend/src/views/task/TaskList.vue
+++ b/frontend/src/views/task/TaskList.vue
@@ -221,7 +221,7 @@ export default {
    },
    onRefresh () {
      this.$store.dispatch('task/getTaskList')
-      this.$st.sendEv('任务', '搜索')
+      this.$st.sendEv('任务列表', '搜索')
    },
    onRemoveMultipleTask () {
      if (this.multipleSelection.length === 0) {
@@ -267,20 +267,20 @@ export default {
              message: 'Deleted successfully'
            })
          })
-        this.$st.sendEv('任务', '删除', 'id', row._id)
+        this.$st.sendEv('任务列表', '删除任务')
      })
    },
    onView (row) {
      this.$router.push(`/tasks/${row._id}`)
-      this.$st.sendEv('任务', '搜索', 'id', row._id)
+      this.$st.sendEv('任务列表', '查看任务')
    },
    onClickSpider (row) {
      this.$router.push(`/spiders/${row.spider_id}`)
-      this.$st.sendEv('任务', '点击爬虫详情', 'id', row.spider_id)
+      this.$st.sendEv('任务列表', '点击爬虫详情')
    },
    onClickNode (row) {
      this.$router.push(`/nodes/${row.node_id}`)
-      this.$st.sendEv('任务', '点击节点详情', 'id', row.node_id)
+      this.$st.sendEv('任务列表', '点击节点详情')
    },
    onPageChange () {
      setTimeout(() => {
--- a/frontend/src/views/user/UserList.vue
+++ b/frontend/src/views/user/UserList.vue
@@ -3,14 +3,14 @@
    <!--dialog-->
    <el-dialog :visible.sync="dialogVisible" :title="$t('Edit User')">
      <el-form ref="form" :model="userForm" label-width="80px" :rules="rules" inline-message>
-        <el-form-item :label="$t('Username')">
-          <el-input v-model="userForm.username" disabled></el-input>
+        <el-form-item prop="username" :label="$t('Username')" required>
+          <el-input v-model="userForm.username" :placeholder="$t('Username')" :disabled="!isAdd"></el-input>
        </el-form-item>
-        <el-form-item prop="password" :label="$t('Password')">
+        <el-form-item prop="password" :label="$t('Password')" required>
          <el-input type="password" v-model="userForm.password" :placeholder="$t('Password')"></el-input>
        </el-form-item>
-        <el-form-item :label="$t('Role')">
-          <el-select v-model="userForm.role">
+        <el-form-item prop="role" :label="$t('Role')" required>
+          <el-select v-model="userForm.role" :placeholder="$t('Role')">
            <el-option value="admin" :label="$t('admin')"></el-option>
            <el-option value="normal" :label="$t('normal')"></el-option>
          </el-select>
@@ -27,7 +27,7 @@
      <div class="filter">
        <div class="left"></div>
        <div class="right">
-          <!--<el-button type="primary" size="small">新增用户</el-button>-->
+          <el-button type="success" icon="el-icon-plus" size="small" @click="onClickAddUser">添加用户</el-button>
        </div>
      </div>
      <!--table-->
@@ -109,6 +109,7 @@ export default {
    }
    return {
      dialogVisible: false,
+      isAdd: false,
      rules: {
        password: [{ validator: validatePass }]
      }
@@ -145,6 +146,7 @@ export default {
      return dayjs(ts).format('YYYY-MM-DD HH:mm:ss')
    },
    onEdit (row) {
+      this.isAdd = false
      this.$store.commit('user/SET_USER_FORM', row)
      this.dialogVisible = true
    },
@@ -161,24 +163,48 @@ export default {
              message: this.$t('Deleted successfully')
            })
          })
-        this.$st.sendEv('用户', '删除', 'id', row._id)
+          .then(() => {
+            this.$store.dispatch('user/getUserList')
+          })
+        this.$st.sendEv('用户列表', '删除用户')
      })
      // this.$store.commit('user/SET_USER_FORM', row)
    },
    onConfirm () {
-      this.dialogVisible = false
      this.$refs.form.validate(valid => {
-        if (valid) {
+        if (!valid) return
+        if (this.isAdd) {
+          // 添加用户
+          this.$store.dispatch('user/addUser')
+            .then(() => {
+              this.$message({
+                type: 'success',
+                message: this.$t('Saved successfully')
+              })
+              this.dialogVisible = false
+              this.$st.sendEv('用户列表', '添加用户')
+            })
+            .then(() => {
+              this.$store.dispatch('user/getUserList')
+            })
+        } else {
+          // 编辑用户
          this.$store.dispatch('user/editUser')
            .then(() => {
              this.$message({
                type: 'success',
                message: this.$t('Saved successfully')
              })
+              this.dialogVisible = false
+              this.$st.sendEv('用户列表', '编辑用户')
            })
        }
      })
-      this.$st.sendEv('用户', '编辑')
+    },
+    onClickAddUser () {
+      this.isAdd = true
+      this.$store.commit('user/SET_USER_FORM', {})
+      this.dialogVisible = true
    }
  },
  created () {
@@ -192,15 +218,18 @@ export default {
    display: flex;
    justify-content: space-between;
    margin-bottom: 8px;
-    .filter-search {
-      width: 240px;
-    }

-    .right {
-      .btn {
-        margin-left: 10px;
-      }
-    }
+  .filter-search {
+    width: 240px;
+  }
+
+  .right {
+
+  .btn {
+    margin-left: 10px;
+  }
+
+  }
  }

  .el-table {
--- a/frontend/yarn.lock
+++ b/frontend/yarn.lock
@@ -1158,6 +1158,11 @@ ansi-regex@^4.0.0:
  version "4.0.0"
  resolved "http://registry.npm.taobao.org/ansi-regex/download/ansi-regex-4.0.0.tgz#70de791edf021404c3fd615aa89118ae0432e5a9"

+ansi-regex@^4.1.0:
+  version "4.1.0"
+  resolved "https://registry.npm.taobao.org/ansi-regex/download/ansi-regex-4.1.0.tgz?cache=0&sync_timestamp=1570188570027&other_urls=https%3A%2F%2Fregistry.npm.taobao.org%2Fansi-regex%2Fdownload%2Fansi-regex-4.1.0.tgz#8b9f8f08cf1acb843756a839ca8c7e3168c51997"
+  integrity sha1-i5+PCM8ay4Q3Vqg5yox+MWjFGZc=
+
 ansi-styles@^2.2.1:
  version "2.2.1"
  resolved "http://registry.npm.taobao.org/ansi-styles/download/ansi-styles-2.2.1.tgz#b432dd3358b634cf75e1e4664368240533c1ddbe"
@@ -2086,6 +2091,15 @@ cliui@^4.0.0, cliui@^4.1.0:
    strip-ansi "^4.0.0"
    wrap-ansi "^2.0.0"

+cliui@^5.0.0:
+  version "5.0.0"
+  resolved "https://registry.npm.taobao.org/cliui/download/cliui-5.0.0.tgz#deefcfdb2e800784aa34f46fa08e06851c7bbbc5"
+  integrity sha1-3u/P2y6AB4SqNPRvoI4GhRx7u8U=
+  dependencies:
+    string-width "^3.1.0"
+    strip-ansi "^5.2.0"
+    wrap-ansi "^5.1.0"
+
 clone-deep@^2.0.1:
  version "2.0.2"
  resolved "http://registry.npm.taobao.org/clone-deep/download/clone-deep-2.0.2.tgz#00db3a1e173656730d1188c3d6aced6d7ea97713"
@@ -3884,6 +3898,11 @@ get-caller-file@^1.0.1:
  version "1.0.3"
  resolved "http://registry.npm.taobao.org/get-caller-file/download/get-caller-file-1.0.3.tgz#f978fa4c90d1dfe7ff2d6beda2a515e713bdcf4a"

+get-caller-file@^2.0.1:
+  version "2.0.5"
+  resolved "https://registry.npm.taobao.org/get-caller-file/download/get-caller-file-2.0.5.tgz#4f94412a82db32f36e3b0b9741f8a97feb031f7e"
+  integrity sha1-T5RBKoLbMvNuOwuXQfipf+sDH34=
+
 get-stdin@^4.0.1:
  version "4.0.1"
  resolved "http://registry.npm.taobao.org/get-stdin/download/get-stdin-4.0.1.tgz#b968c6b0a04384324902e8bf1a5df32579a450fe"
@@ -7271,6 +7290,11 @@ require-main-filename@^1.0.1:
  version "1.0.1"
  resolved "http://registry.npm.taobao.org/require-main-filename/download/require-main-filename-1.0.1.tgz#97f717b69d48784f5f526a6c5aa8ffdda055a4d1"

+require-main-filename@^2.0.0:
+  version "2.0.0"
+  resolved "https://registry.npm.taobao.org/require-main-filename/download/require-main-filename-2.0.0.tgz#d0b329ecc7cc0f61649f62215be69af54aa8989b"
+  integrity sha1-0LMp7MfMD2Fkn2IhW+aa9UqomJs=
+
 require-uncached@^1.0.3:
  version "1.0.3"
  resolved "http://registry.npm.taobao.org/require-uncached/download/require-uncached-1.0.3.tgz#4e0d56d6c9662fd31e43011c4b95aa49955421d3"
@@ -7586,6 +7610,13 @@ shellwords@^0.1.1:
  version "0.1.1"
  resolved "http://registry.npm.taobao.org/shellwords/download/shellwords-0.1.1.tgz#d6b9181c1a48d397324c84871efbcfc73fc0654b"

+showdown@^1.9.1:
+  version "1.9.1"
+  resolved "https://registry.npm.taobao.org/showdown/download/showdown-1.9.1.tgz#134e148e75cd4623e09c21b0511977d79b5ad0ef"
+  integrity sha1-E04UjnXNRiPgnCGwURl315ta0O8=
+  dependencies:
+    yargs "^14.2"
+
 sigmund@^1.0.1:
  version "1.0.1"
  resolved "http://registry.npm.taobao.org/sigmund/download/sigmund-1.0.1.tgz#3ff21f198cad2175f9f3b781853fd94d0d19b590"
@@ -7890,6 +7921,15 @@ string-width@^3.0.0:
    is-fullwidth-code-point "^2.0.0"
    strip-ansi "^5.0.0"

+string-width@^3.1.0:
+  version "3.1.0"
+  resolved "https://registry.npm.taobao.org/string-width/download/string-width-3.1.0.tgz?cache=0&other_urls=https%3A%2F%2Fregistry.npm.taobao.org%2Fstring-width%2Fdownload%2Fstring-width-3.1.0.tgz#22767be21b62af1081574306f69ac51b62203961"
+  integrity sha1-InZ74htirxCBV0MG9prFG2IgOWE=
+  dependencies:
+    emoji-regex "^7.0.1"
+    is-fullwidth-code-point "^2.0.0"
+    strip-ansi "^5.1.0"
+
 string.prototype.padend@^3.0.0:
  version "3.0.0"
  resolved "http://registry.npm.taobao.org/string.prototype.padend/download/string.prototype.padend-3.0.0.tgz#f3aaef7c1719f170c5eab1c32bf780d96e21f2f0"
@@ -7940,6 +7980,13 @@ strip-ansi@^5.0.0:
  dependencies:
    ansi-regex "^4.0.0"

+strip-ansi@^5.1.0, strip-ansi@^5.2.0:
+  version "5.2.0"
+  resolved "https://registry.npm.taobao.org/strip-ansi/download/strip-ansi-5.2.0.tgz#8c9a536feb6afc962bdfa5b104a5091c1ad9c0ae"
+  integrity sha1-jJpTb+tq/JYr36WxBKUJHBrZwK4=
+  dependencies:
+    ansi-regex "^4.1.0"
+
 strip-bom@3.0.0, strip-bom@^3.0.0:
  version "3.0.0"
  resolved "http://registry.npm.taobao.org/strip-bom/download/strip-bom-3.0.0.tgz#2334c18e9c759f7bdd56fdef7e9ae3d588e68ed3"
@@ -8834,6 +8881,15 @@ wrap-ansi@^2.0.0:
    string-width "^1.0.1"
    strip-ansi "^3.0.1"

+wrap-ansi@^5.1.0:
+  version "5.1.0"
+  resolved "https://registry.npm.taobao.org/wrap-ansi/download/wrap-ansi-5.1.0.tgz#1fd1f67235d5b6d0fee781056001bfb694c03b09"
+  integrity sha1-H9H2cjXVttD+54EFYAG/tpTAOwk=
+  dependencies:
+    ansi-styles "^3.2.0"
+    string-width "^3.0.0"
+    strip-ansi "^5.0.0"
+
 wrappy@1:
  version "1.0.2"
  resolved "http://registry.npm.taobao.org/wrappy/download/wrappy-1.0.2.tgz#b5243d8f3ec1aa35f1364605bc0d1036e30ab69f"
@@ -8911,6 +8967,14 @@ yargs-parser@^11.1.1:
    camelcase "^5.0.0"
    decamelize "^1.2.0"

+yargs-parser@^15.0.0:
+  version "15.0.0"
+  resolved "https://registry.npm.taobao.org/yargs-parser/download/yargs-parser-15.0.0.tgz#cdd7a97490ec836195f59f3f4dbe5ea9e8f75f08"
+  integrity sha1-zdepdJDsg2GV9Z8/Tb5eqej3Xwg=
+  dependencies:
+    camelcase "^5.0.0"
+    decamelize "^1.2.0"
+
 yargs-parser@^5.0.0:
  version "5.0.0"
  resolved "http://registry.npm.taobao.org/yargs-parser/download/yargs-parser-5.0.0.tgz#275ecf0d7ffe05c77e64e7c86e4cd94bf0e1228a"
@@ -8974,6 +9038,23 @@ yargs@^12.0.5:
    y18n "^3.2.1 || ^4.0.0"
    yargs-parser "^11.1.1"

+yargs@^14.2:
+  version "14.2.2"
+  resolved "https://registry.npm.taobao.org/yargs/download/yargs-14.2.2.tgz?cache=0&sync_timestamp=1574137859196&other_urls=https%3A%2F%2Fregistry.npm.taobao.org%2Fyargs%2Fdownload%2Fyargs-14.2.2.tgz#2769564379009ff8597cdd38fba09da9b493c4b5"
+  integrity sha1-J2lWQ3kAn/hZfN04+6CdqbSTxLU=
+  dependencies:
+    cliui "^5.0.0"
+    decamelize "^1.2.0"
+    find-up "^3.0.0"
+    get-caller-file "^2.0.1"
+    require-directory "^2.1.1"
+    require-main-filename "^2.0.0"
+    set-blocking "^2.0.0"
+    string-width "^3.0.0"
+    which-module "^2.0.0"
+    y18n "^4.0.0"
+    yargs-parser "^15.0.0"
+
 yargs@^7.0.0:
  version "7.1.0"
  resolved "http://registry.npm.taobao.org/yargs/download/yargs-7.1.0.tgz#6ba318eb16961727f5d284f8ea003e8d6154d0c8"
--- a/jenkins/develop/docker-compose.yaml
+++ b/jenkins/develop/docker-compose.yaml
@@ -27,10 +27,6 @@ services:
  mongo:
    image: mongo:latest
    restart: always
-    ports:
-      - "27027:27017"
  redis:
    image: redis:latest
    restart: always
-    ports:
-      - "6389:6379"
--- a/jenkins/master/docker-compose.yaml
+++ b/jenkins/master/docker-compose.yaml
@@ -29,12 +29,9 @@ services:
    restart: always
    volumes:
      - "/opt/crawlab/mongo/data/db:/data/db"
-    ports:
-      - "27017:27017"
+      - "/opt/crawlab/mongo/tmp:/tmp"
  redis:
    image: redis:latest
    restart: always
    volumes:
      - "/opt/crawlab/redis/data:/data"
-    ports:
-      - "6379:6379"
--- a/wait-for-it.sh
+++ b/wait-for-it.sh
@@ -0,0 +1,178 @@
+#!/usr/bin/env bash
+#   Use this script to test if a given TCP host/port are available
+
+WAITFORIT_cmdname=${0##*/}
+
+echoerr() { if [[ $WAITFORIT_QUIET -ne 1 ]]; then echo "$@" 1>&2; fi }
+
+usage()
+{
+    cat << USAGE >&2
+Usage:
+    $WAITFORIT_cmdname host:port [-s] [-t timeout] [-- command args]
+    -h HOST | --host=HOST       Host or IP under test
+    -p PORT | --port=PORT       TCP port under test
+                                Alternatively, you specify the host and port as host:port
+    -s | --strict               Only execute subcommand if the test succeeds
+    -q | --quiet                Don't output any status messages
+    -t TIMEOUT | --timeout=TIMEOUT
+                                Timeout in seconds, zero for no timeout
+    -- COMMAND ARGS             Execute command with args after the test finishes
+USAGE
+    exit 1
+}
+
+wait_for()
+{
+    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
+        echoerr "$WAITFORIT_cmdname: waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
+    else
+        echoerr "$WAITFORIT_cmdname: waiting for $WAITFORIT_HOST:$WAITFORIT_PORT without a timeout"
+    fi
+    WAITFORIT_start_ts=$(date +%s)
+    while :
+    do
+        if [[ $WAITFORIT_ISBUSY -eq 1 ]]; then
+            nc -z $WAITFORIT_HOST $WAITFORIT_PORT
+            WAITFORIT_result=$?
+        else
+            (echo > /dev/tcp/$WAITFORIT_HOST/$WAITFORIT_PORT) >/dev/null 2>&1
+            WAITFORIT_result=$?
+        fi
+        if [[ $WAITFORIT_result -eq 0 ]]; then
+            WAITFORIT_end_ts=$(date +%s)
+            echoerr "$WAITFORIT_cmdname: $WAITFORIT_HOST:$WAITFORIT_PORT is available after $((WAITFORIT_end_ts - WAITFORIT_start_ts)) seconds"
+            break
+        fi
+        sleep 1
+    done
+    return $WAITFORIT_result
+}
+
+wait_for_wrapper()
+{
+    # In order to support SIGINT during timeout: http://unix.stackexchange.com/a/57692
+    if [[ $WAITFORIT_QUIET -eq 1 ]]; then
+        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --quiet --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
+    else
+        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
+    fi
+    WAITFORIT_PID=$!
+    trap "kill -INT -$WAITFORIT_PID" INT
+    wait $WAITFORIT_PID
+    WAITFORIT_RESULT=$?
+    if [[ $WAITFORIT_RESULT -ne 0 ]]; then
+        echoerr "$WAITFORIT_cmdname: timeout occurred after waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
+    fi
+    return $WAITFORIT_RESULT
+}
+
+# process arguments
+while [[ $# -gt 0 ]]
+do
+    case "$1" in
+        *:* )
+        WAITFORIT_hostport=(${1//:/ })
+        WAITFORIT_HOST=${WAITFORIT_hostport[0]}
+        WAITFORIT_PORT=${WAITFORIT_hostport[1]}
+        shift 1
+        ;;
+        --child)
+        WAITFORIT_CHILD=1
+        shift 1
+        ;;
+        -q | --quiet)
+        WAITFORIT_QUIET=1
+        shift 1
+        ;;
+        -s | --strict)
+        WAITFORIT_STRICT=1
+        shift 1
+        ;;
+        -h)
+        WAITFORIT_HOST="$2"
+        if [[ $WAITFORIT_HOST == "" ]]; then break; fi
+        shift 2
+        ;;
+        --host=*)
+        WAITFORIT_HOST="${1#*=}"
+        shift 1
+        ;;
+        -p)
+        WAITFORIT_PORT="$2"
+        if [[ $WAITFORIT_PORT == "" ]]; then break; fi
+        shift 2
+        ;;
+        --port=*)
+        WAITFORIT_PORT="${1#*=}"
+        shift 1
+        ;;
+        -t)
+        WAITFORIT_TIMEOUT="$2"
+        if [[ $WAITFORIT_TIMEOUT == "" ]]; then break; fi
+        shift 2
+        ;;
+        --timeout=*)
+        WAITFORIT_TIMEOUT="${1#*=}"
+        shift 1
+        ;;
+        --)
+        shift
+        WAITFORIT_CLI=("$@")
+        break
+        ;;
+        --help)
+        usage
+        ;;
+        *)
+        echoerr "Unknown argument: $1"
+        usage
+        ;;
+    esac
+done
+
+if [[ "$WAITFORIT_HOST" == "" || "$WAITFORIT_PORT" == "" ]]; then
+    echoerr "Error: you need to provide a host and port to test."
+    usage
+fi
+
+WAITFORIT_TIMEOUT=${WAITFORIT_TIMEOUT:-15}
+WAITFORIT_STRICT=${WAITFORIT_STRICT:-0}
+WAITFORIT_CHILD=${WAITFORIT_CHILD:-0}
+WAITFORIT_QUIET=${WAITFORIT_QUIET:-0}
+
+# check to see if timeout is from busybox?
+WAITFORIT_TIMEOUT_PATH=$(type -p timeout)
+WAITFORIT_TIMEOUT_PATH=$(realpath $WAITFORIT_TIMEOUT_PATH 2>/dev/null || readlink -f $WAITFORIT_TIMEOUT_PATH)
+if [[ $WAITFORIT_TIMEOUT_PATH =~ "busybox" ]]; then
+        WAITFORIT_ISBUSY=1
+        WAITFORIT_BUSYTIMEFLAG="-t"
+
+else
+        WAITFORIT_ISBUSY=0
+        WAITFORIT_BUSYTIMEFLAG=""
+fi
+
+if [[ $WAITFORIT_CHILD -gt 0 ]]; then
+    wait_for
+    WAITFORIT_RESULT=$?
+    exit $WAITFORIT_RESULT
+else
+    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
+        wait_for_wrapper
+        WAITFORIT_RESULT=$?
+    else
+        wait_for
+        WAITFORIT_RESULT=$?
+    fi
+fi
+
+if [[ $WAITFORIT_CLI != "" ]]; then
+    if [[ $WAITFORIT_RESULT -ne 0 && $WAITFORIT_STRICT -eq 1 ]]; then
+        echoerr "$WAITFORIT_cmdname: strict mode, refusing to execute subprocess"
+        exit $WAITFORIT_RESULT
+    fi
+    exec "${WAITFORIT_CLI[@]}"
+else
+    exit $WAITFORIT_RESULT
+fi