Merge pull request #17 from crawlab-team/develop

Develop
2026-01-24 17:41:03 +01:00 · 2020-01-01 14:56:42 +08:00
parent 352e4c886a 6bf6212a22
commit aab8980772
259 changed files with 26866 additions and 7107 deletions
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.
--- a/.github/ISSUE_TEMPLATE/feature_request.md
+++ b/.github/ISSUE_TEMPLATE/feature_request.md
@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,37 @@
+# 0.4.2 (2019-12-26)
+### Features / Enhancement
+- **Disclaimer**. Added page for Disclaimer.
+- **Call API to fetch version**. [#371](https://github.com/crawlab-team/crawlab/issues/371)
+- **Configure to allow user registration**. [#346](https://github.com/crawlab-team/crawlab/issues/346)
+- **Allow adding new users**.
+- **More Advanced File Management**. Allow users to add / edit / rename / delete files. [#286](https://github.com/crawlab-team/crawlab/issues/286)
+- **Optimized Spider Creation Process**. Allow users to create an empty customized spider before uploading the zip file.
+- **Better Task Management**. Allow users to filter tasks by selecting through certian criterions. [#341](https://github.com/crawlab-team/crawlab/issues/341)
+
+### Bug Fixes
+- **Duplicated nodes**. [#391](https://github.com/crawlab-team/crawlab/issues/391)
+- **"mongodb no reachable" error**. [#373](https://github.com/crawlab-team/crawlab/issues/373)
+
+# 0.4.1 (2019-12-13)
+### Features / Enhancement
+- **Spiderfile Optimization**. Stages changed from dictionary to array. [#358](https://github.com/crawlab-team/crawlab/issues/358)
+- **Baidu Tongji Update**.
+
+### Bug Fixes
+- **Unable to display schedule tasks**. [#353](https://github.com/crawlab-team/crawlab/issues/353)
+- **Duplicate node registration**. [#334](https://github.com/crawlab-team/crawlab/issues/334)
+
+# 0.4.0 (2019-12-06)
+### Features / Enhancement
+- **Configurable Spider**. Allow users to add spiders using *Spiderfile* to configure crawling rules.
+- **Execution Mode**. Allow users to select 3 modes for task execution: *All Nodes*, *Selected Nodes* and *Random*.
+
+### Bug Fixes
+- **Task accidentally killed**. [#306](https://github.com/crawlab-team/crawlab/issues/306)
+- **Documentation fix**. [#301](https://github.com/crawlab-team/crawlab/issues/258) [#301](https://github.com/crawlab-team/crawlab/issues/258)
+- **Direct deploy incompatible with Windows**. [#288](https://github.com/crawlab-team/crawlab/issues/288)
+- **Log files lost**. [#269](https://github.com/crawlab-team/crawlab/issues/269)
+
 # 0.3.5 (2019-10-28)
 ### Features / Enhancement
 - **Graceful Showdown**. [detail](https://github.com/crawlab-team/crawlab/commit/63fab3917b5a29fd9770f9f51f1572b9f0420385)
--- a/DISCLAIMER-zh.md
+++ b/DISCLAIMER-zh.md
@@ -0,0 +1,12 @@
+# 免责声明
+
+本免责及隐私保护声明(以下简称“免责声明”或“本声明”)适用于 Crawlab 开发组 (以下简称“开发组”)研发的系列软件(以下简称"Crawlab") 在您阅读本声明后若不同意此声明中的任何条款，或对本声明存在质疑，请立刻停止使用我们的软件。若您已经开始或正在使用 Crawlab，则表示您已阅读并同意本声明的所有条款之约定。
+
+1. 总则：您通过安装 Crawlab 并使用 Crawlab 提供的服务与功能即表示您已经同意与开发组立本协议。开发组可随时执行全权决定更改“条款”。经修订的“条款”一经在 Github 免责声明页面上公布后，立即自动生效。
+2. 本产品是基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。
+3. 一切因使用 Crawlab 而引致之任何意外、疏忽、合约毁坏、诽谤、版权或知识产权侵犯及其所造成的损失(包括在非官方站点下载 Crawlab 而感染电脑病毒)，Crawlab 开发组概不负责，亦不承担任何法律责任。
+4. 用户对使用 Crawlab 自行承担风险，我们不做任何形式的保证， 因网络状况、通讯线路等任何技术原因而导致用户不能正常升级更新，我们也不承担任何法律责任。
+5. 用户使用 Crawlab 对目标网站进行抓取时需遵从《网络安全法》等与爬虫相关的法律法规，切勿擅自采集公民个人信息、用 DDoS 等方式造成目标网站瘫痪、不遵从目标网站的 robots.txt 协议等非法手段。
+6. Crawlab 尊重并保护所有用户的个人隐私权，不会窃取任何用户计算机中的信息。
+7. 系统的版权：Crawlab 开发组对所有开发的或合作开发的产品拥有知识产权，著作权，版权和使用权，这些产品受到适用的知识产权、版权、商标、服务商标、专利或其他法律的保护。
+8. 传播:任何公司或个人在网络上发布，传播我们软件的行为都是允许的，但因公司或个人传播软件可能造成的任何法律和刑事事件 Crawlab 开发组不负任何责任。
--- a/DISCLAIMER.md
+++ b/DISCLAIMER.md
@@ -0,0 +1,12 @@
+# Disclaimer
+
+This Disclaimer and privacy protection statement (hereinafter referred to as "disclaimer statement" or "this statement") is applicable to the series of software (hereinafter referred to as "crawlab") developed by crawlab development group (hereinafter referred to as "development group") after you read this statement, if you do not agree with any terms in this statement or have doubts about this statement, please stop using our software immediately. If you have started or are using crawlab, you have read and agree to all terms of this statement.
+
+1. General: by installing crawlab and using the services and functions provided by crawlab, you have agreed to establish this agreement with the development team. The developer group may at any time change the terms at its sole discretion. The amended "terms" shall take effect automatically as soon as they are published on the GitHub disclaimer page.
+2. This product is a distributed crawler management platform based on golang, supporting python, nodejs, go, Java, PHP and other programming languages as well as a variety of crawler frameworks.
+3. The development team of crawlab shall not be responsible for any accident, negligence, contract damage, defamation, copyright or intellectual property infringement caused by the use of crawlab and any loss caused by it (including computer virus infection caused by downloading crawlab on the unofficial site), and shall not bear any legal responsibility.
+4. The user shall bear the risk of using crawlab by himself, we do not make any form of guarantee, and we will not bear any legal responsibility for the user's failure to upgrade and update normally due to any technical reasons such as network condition and communication line.
+5. When users use crawlab to grab the target website, they need to comply with the laws and regulations related to crawlers, such as the network security law. Do not collect personal information of citizens without authorization, cause the target website to be paralyzed by DDoS, or fail to comply with the robots.txt protocol and other illegal means of the target website.
+6. Crawlab respects and protects the personal privacy of all users and will not steal any information from users' computers.
+7. Copyright of the system: the crawleb development team owns the intellectual property rights, copyrights, copyrights and use rights for all developed or jointly developed products, which are protected by applicable intellectual property rights, copyrights, trademarks, service trademarks, patents or other laws.
+8. Communication: any company or individual who publishes or disseminates our software on the Internet is allowed, but the crawlab development team shall not be responsible for any legal and criminal events that may be caused by the company or individual disseminating the software.
--- a/6
+++ b/6
@@ -15,7 +15,7 @@ WORKDIR /app

 # install frontend
 RUN npm config set unsafe-perm true
-RUN npm install -g yarn && yarn install --registry=https://registry.npm.taobao.org
+RUN npm install -g yarn && yarn install --registry=https://registry.npm.taobao.org # --sass_binary_site=https://npm.taobao.org/mirrors/node-sass/

 RUN npm run build:prod

@@ -27,6 +27,9 @@ ADD . /app
 # set as non-interactive
 ENV DEBIAN_FRONTEND noninteractive

+# set CRAWLAB_IS_DOCKER
+ENV CRAWLAB_IS_DOCKER Y
+
 # install packages
 RUN apt-get update \
 	&& apt-get install -y curl git net-tools iputils-ping ntp ntpdate python3 python3-pip \
@@ -37,7 +40,6 @@ RUN apt-get update \
 RUN pip install scrapy pymongo bs4 requests -i https://pypi.tuna.tsinghua.edu.cn/simple

 # copy backend files
-COPY --from=backend-build /go/src/app .
 COPY --from=backend-build /go/bin/crawlab /usr/local/bin

 # install nginx
--- a/Dockerfile.local
+++ b/Dockerfile.local
@@ -4,17 +4,18 @@ WORKDIR /go/src/app
 COPY ./backend .

 ENV GO111MODULE on
-ENV GOPROXY https://mirrors.aliyun.com/goproxy/
+ENV GOPROXY https://goproxy.io

 RUN go install -v ./...

-FROM node:8.16.0 AS frontend-build
+FROM node:8.16.0-alpine AS frontend-build

 ADD ./frontend /app
 WORKDIR /app

 # install frontend
-RUN npm install -g yarn && yarn install --registry=https://registry.npm.taobao.org
+RUN npm config set unsafe-perm true
+RUN npm install -g yarn && yarn install --registry=https://registry.npm.taobao.org # --sass_binary_site=https://npm.taobao.org/mirrors/node-sass/

 RUN npm run build:prod

@@ -27,7 +28,8 @@ ADD . /app
 ENV DEBIAN_FRONTEND noninteractive

 # install packages
-RUN apt-get update \
+RUN chmod 777 /tmp \
+	&& apt-get update \
 	&& apt-get install -y curl git net-tools iputils-ping ntp ntpdate python3 python3-pip \
 	&& ln -s /usr/bin/pip3 /usr/local/bin/pip \
 	&& ln -s /usr/bin/python3 /usr/local/bin/python
@@ -36,7 +38,6 @@ RUN apt-get update \
 RUN pip install scrapy pymongo bs4 requests -i https://pypi.tuna.tsinghua.edu.cn/simple

 # copy backend files
-COPY --from=backend-build /go/src/app .
 COPY --from=backend-build /go/bin/crawlab /usr/local/bin

 # install nginx
@@ -56,4 +57,4 @@ EXPOSE 8080
 EXPOSE 8000

 # start backend
-CMD ["/bin/sh", "/app/docker_init.sh"]
+CMD ["/bin/sh", "/app/docker_init.sh"]
--- a/14
+++ b/14
@@ -16,15 +16,6 @@ pipeline {
                    } else if (env.GIT_BRANCH == 'master') {
                        env.TAG = 'master'
                        env.DOCKERFILE = 'Dockerfile.local'
-                    } else if (env.GIT_BRANCH == 'frontend') {
-                        env.TAG = 'frontend-alpine'
-                        env.DOCKERFILE = 'docker/Dockerfile.frontend.alpine'
-                    } else if (env.GIT_BRANCH == 'backend-master') {
-                        env.TAG = 'master-alpine'
-                        env.DOCKERFILE = 'docker/Dockerfile.master.alpine'
-                    } else if (env.GIT_BRANCH == 'backend-worker') {
-                        env.TAG = 'worker-alpine'
-                        env.DOCKERFILE = 'docker/Dockerfile.worker.alpine'
                    } 
                }
            }
@@ -48,10 +39,7 @@ pipeline {
                sh """
                # 重启docker compose
                cd ./jenkins/${ENV:GIT_BRANCH}
-                docker-compose stop master | true
-                docker-compose rm -f master | true
-                docker-compose stop worker | true
-                docker-compose rm -f worker | true
+                docker-compose down | true
                docker-compose up -d | true
                """
            }
--- a/README-zh.md
+++ b/README-zh.md
@@ -10,11 +10,11 @@

 中文 | [English](https://github.com/crawlab-team/crawlab)

-[安装](#安装) | [运行](#运行) | [截图](#截图) | [架构](#架构) | [集成](#与其他框架的集成) | [比较](#与其他框架比较) | [相关文章](#相关文章) | [社区&赞助](#社区--赞助)
+[安装](#安装) | [运行](#运行) | [截图](#截图) | [架构](#架构) | [集成](#与其他框架的集成) | [比较](#与其他框架比较) | [相关文章](#相关文章) | [社区&赞助](#社区--赞助) | [免责声明](https://github.com/crawlab-team/crawlab/blob/master/DISCLAIMER-zh.md)

 基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。

-[查看演示 Demo](http://crawlab.cn/demo) | [文档](https://tikazyq.github.io/crawlab-docs)
+[查看演示 Demo](http://crawlab.cn/demo) | [文档](http://docs.crawlab.cn)

 ## 安装

@@ -47,7 +47,7 @@ services:
    image: tikazyq/crawlab:latest
    container_name: master
    environment:
-      CRAWLAB_API_ADDRESS: "localhost:8000"
+      CRAWLAB_API_ADDRESS: "http://localhost:8000"
      CRAWLAB_SERVER_MASTER: "Y"
      CRAWLAB_MONGO_HOST: "mongo"
      CRAWLAB_REDIS_ADDRESS: "redis"
@@ -254,6 +254,9 @@ Crawlab使用起来很方便，也很通用，可以适用于几乎任何主流
 <a href="https://github.com/hantmac">
  <img src="https://avatars2.githubusercontent.com/u/7600925?s=460&v=4" height="80">
 </a>
+<a href="https://github.com/duanbin0414">
+  <img src="https://avatars3.githubusercontent.com/u/50389867?s=460&v=4" height="80">
+</a>

 ## 社区 & 赞助

--- a/README.md
+++ b/README.md
@@ -10,11 +10,11 @@

 [中文](https://github.com/crawlab-team/crawlab/blob/master/README-zh.md) | English

-[Installation](#installation) | [Run](#run) | [Screenshot](#screenshot) | [Architecture](#architecture) | [Integration](#integration-with-other-frameworks) | [Compare](#comparison-with-other-frameworks) | [Community & Sponsorship](#community--sponsorship)
+[Installation](#installation) | [Run](#run) | [Screenshot](#screenshot) | [Architecture](#architecture) | [Integration](#integration-with-other-frameworks) | [Compare](#comparison-with-other-frameworks) | [Community & Sponsorship](#community--sponsorship) | [Disclaimer](https://github.com/crawlab-team/crawlab/blob/master/DISCLAIMER.md)

 Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium.

-[Demo](http://crawlab.cn/demo) | [Documentation](https://tikazyq.github.io/crawlab-docs)
+[Demo](http://crawlab.cn/demo) | [Documentation](http://docs.crawlab.cn)

 ## Installation

@@ -48,7 +48,7 @@ services:
    image: tikazyq/crawlab:latest
    container_name: master
    environment:
-      CRAWLAB_API_ADDRESS: "localhost:8000"
+      CRAWLAB_API_ADDRESS: "http://localhost:8000"
      CRAWLAB_SERVER_MASTER: "Y"
      CRAWLAB_MONGO_HOST: "mongo"
      CRAWLAB_REDIS_ADDRESS: "redis"
@@ -219,6 +219,9 @@ Crawlab is easy to use, general enough to adapt spiders in any language and any
 <a href="https://github.com/hantmac">
  <img src="https://avatars2.githubusercontent.com/u/7600925?s=460&v=4" height="80">
 </a>
+<a href="https://github.com/duanbin0414">
+  <img src="https://avatars3.githubusercontent.com/u/50389867?s=460&v=4" height="80">
+</a>

 ## Community & Sponsorship

--- a/backend/conf/config.yml
+++ b/backend/conf/config.yml
@@ -15,12 +15,12 @@ redis:
 log:
  level: info
  path: "/var/logs/crawlab"
-  isDeletePeriodically: "Y"
+  isDeletePeriodically: "N"
  deleteFrequency: "@hourly"
 server:
  host: 0.0.0.0
  port: 8000
-  master: "N"
+  master: "Y"
  secret: "crawlab"
  register:
    # mac地址 或者 ip地址，如果是ip，则需要手动指定IP
@@ -32,3 +32,6 @@ task:
  workers: 4
 other:
  tmppath: "/tmp"
+version: 0.4.2
+setting:
+  allowRegister: "N"
--- a/backend/config/config.go
+++ b/backend/config/config.go
@@ -28,7 +28,7 @@ func (c *Config) Init() error {
 	}
 	viper.SetConfigType("yaml")   // 设置配置文件格式为YAML
 	viper.AutomaticEnv()          // 读取匹配的环境变量
-	viper.SetEnvPrefix("CRAWLAB") // 读取环境变量的前缀为APISERVER
+	viper.SetEnvPrefix("CRAWLAB") // 读取环境变量的前缀为CRAWLAB
 	replacer := strings.NewReplacer(".", "_")
 	viper.SetEnvKeyReplacer(replacer)
 	if err := viper.ReadInConfig(); err != nil { // viper解析配置文件
--- a/backend/constants/anchor.go
+++ b/backend/constants/anchor.go
@@ -0,0 +1,8 @@
+package constants
+
+const (
+	AnchorStartStage = "START_STAGE"
+	AnchorStartUrl   = "START_URL"
+	AnchorItems      = "ITEMS"
+	AnchorParsers    = "PARSERS"
+)
--- a/backend/constants/config_spider.go
+++ b/backend/constants/config_spider.go
@@ -0,0 +1,6 @@
+package constants
+
+const (
+	EngineScrapy = "scrapy"
+	EngineColly  = "colly"
+)
--- a/backend/constants/schedule.go
+++ b/backend/constants/schedule.go
@@ -0,0 +1,10 @@
+package constants
+
+const (
+	ScheduleStatusStop    = "stop"
+	ScheduleStatusRunning = "running"
+	ScheduleStatusError   = "error"
+
+	ScheduleStatusErrorNotFoundNode   = "Not Found Node"
+	ScheduleStatusErrorNotFoundSpider = "Not Found Spider"
+)
--- a/backend/constants/scrapy.go
+++ b/backend/constants/scrapy.go
@@ -0,0 +1,5 @@
+package constants
+
+const ScrapyProtectedStageNames = ""
+
+const ScrapyProtectedFieldNames = "_id,task_id,ts"
--- a/backend/constants/spider.go
+++ b/backend/constants/spider.go
@@ -3,4 +3,5 @@ package constants
 const (
 	Customized   = "customized"
 	Configurable = "configurable"
+	Plugin       = "plugin"
 )
--- a/backend/constants/system.go
+++ b/backend/constants/system.go
@@ -5,3 +5,9 @@ const (
 	Linux   = "linux"
 	Darwin  = "darwin"
 )
+
+const (
+	Python = "python"
+	NodeJS = "node"
+	Java   = "java"
+)
--- a/backend/constants/task.go
+++ b/backend/constants/task.go
@@ -19,3 +19,9 @@ const (
 	TaskFinish string = "finish"
 	TaskCancel string = "cancel"
 )
+
+const (
+	RunTypeAllNodes      string = "all-nodes"
+	RunTypeRandom        string = "random"
+	RunTypeSelectedNodes string = "selected-nodes"
+)
--- a/backend/database/mongo.go
+++ b/backend/database/mongo.go
@@ -61,10 +61,36 @@ func InitMongo() error {
 			dialInfo.Password = mongoPassword
 			dialInfo.Source = mongoAuth
 		}
-		sess, err := mgo.DialWithInfo(&dialInfo)
-		if err != nil {
-			return err
+
+		// mongo session
+		var sess *mgo.Session
+
+		// 错误次数
+		errNum := 0
+
+		// 重复尝试连接mongo
+		for {
+			var err error
+
+			// 连接mongo
+			sess, err = mgo.DialWithInfo(&dialInfo)
+
+			if err != nil {
+				// 如果连接错误，休息1秒，错误次数+1
+				time.Sleep(1 * time.Second)
+				errNum++
+
+				// 如果错误次数超过30，返回错误
+				if errNum >= 30 {
+					return err
+				}
+			} else {
+				// 如果没有错误，退出循环
+				break
+			}
 		}
+
+		// 赋值给全局mongo session
 		Session = sess
 	}
 	return nil
--- a/backend/database/pubsub.go
+++ b/backend/database/pubsub.go
@@ -58,9 +58,9 @@ func (r *Redis) subscribe(ctx context.Context, consume ConsumeFunc, channel ...s
 			}
 			done <- nil
 		case <-tick.C:
-			//fmt.Printf("ping message  \n")
 			if err := psc.Ping(""); err != nil {
-				done <- err
+				fmt.Printf("ping message error: %s \n", err)
+				//done <- err
 			}
 		case err := <-done:
 			close(done)
--- a/backend/database/redis.go
+++ b/backend/database/redis.go
@@ -4,10 +4,12 @@ import (
 	"context"
 	"crawlab/entity"
 	"crawlab/utils"
+	"errors"
 	"github.com/apex/log"
 	"github.com/gomodule/redigo/redis"
 	"github.com/spf13/viper"
 	"runtime/debug"
+	"strings"
 	"time"
 )

@@ -17,9 +19,18 @@ type Redis struct {
 	pool *redis.Pool
 }

+type Mutex struct {
+	Name   string
+	expiry time.Duration
+	tries  int
+	delay  time.Duration
+	value  string
+}
+
 func NewRedisClient() *Redis {
 	return &Redis{pool: NewRedisPool()}
 }
+
 func (r *Redis) RPush(collection string, value interface{}) error {
 	c := r.pool.Get()
 	defer utils.Close(c)
@@ -102,7 +113,7 @@ func NewRedisPool() *redis.Pool {
 			return redis.DialURL(url,
 				redis.DialConnectTimeout(time.Second*10),
 				redis.DialReadTimeout(time.Second*10),
-				redis.DialWriteTimeout(time.Second*10),
+				redis.DialWriteTimeout(time.Second*15),
 			)
 		},
 		TestOnBorrow: func(c redis.Conn, t time.Time) error {
@@ -143,3 +154,59 @@ func Sub(channel string, consume ConsumeFunc) error {
 	}
 	return nil
 }
+
+// 构建同步锁key
+func (r *Redis) getLockKey(lockKey string) string {
+	lockKey = strings.ReplaceAll(lockKey, ":", "-")
+	return "nodes:lock:" + lockKey
+}
+
+// 获得锁
+func (r *Redis) Lock(lockKey string) (int64, error) {
+	c := r.pool.Get()
+	defer utils.Close(c)
+	lockKey = r.getLockKey(lockKey)
+
+	ts := time.Now().Unix()
+	ok, err := c.Do("SET", lockKey, ts, "NX", "PX", 30000)
+	if err != nil {
+		log.Errorf("get lock fail with error: %s", err.Error())
+		debug.PrintStack()
+		return 0, err
+	}
+	if err == nil && ok == nil {
+		log.Errorf("the lockKey is locked: key=%s", lockKey)
+		return 0, errors.New("the lockKey is locked")
+	}
+	return ts, nil
+}
+
+func (r *Redis) UnLock(lockKey string, value int64) {
+	c := r.pool.Get()
+	defer utils.Close(c)
+	lockKey = r.getLockKey(lockKey)
+
+	getValue, err := redis.Int64(c.Do("GET", lockKey))
+	if err != nil {
+		log.Errorf("get lockKey error: %s", err.Error())
+		debug.PrintStack()
+		return
+	}
+
+	if getValue != value {
+		log.Errorf("the lockKey value diff: %d, %d", value, getValue)
+		return
+	}
+
+	v, err := redis.Int64(c.Do("DEL", lockKey))
+	if err != nil {
+		log.Errorf("unlock failed, error: %s", err.Error())
+		debug.PrintStack()
+		return
+	}
+
+	if v == 0 {
+		log.Errorf("unlock failed: key=%s", lockKey)
+		return
+	}
+}
--- a/backend/entity/common.go
+++ b/backend/entity/common.go
@@ -3,15 +3,15 @@ package entity
 import "strconv"

 type Page struct {
-	Skip int
-	Limit int
-	PageNum int
+	Skip     int
+	Limit    int
+	PageNum  int
 	PageSize int
 }

-func (p *Page)GetPage(pageNum string, pageSize string) {
+func (p *Page) GetPage(pageNum string, pageSize string) {
 	p.PageNum, _ = strconv.Atoi(pageNum)
 	p.PageSize, _ = strconv.Atoi(pageSize)
 	p.Skip = p.PageSize * (p.PageNum - 1)
 	p.Limit = p.PageSize
-}
+}
--- a/backend/entity/config_spider.go
+++ b/backend/entity/config_spider.go
@@ -0,0 +1,30 @@
+package entity
+
+type ConfigSpiderData struct {
+	Version    string            `yaml:"version" json:"version"`
+	Engine     string            `yaml:"engine" json:"engine"`
+	StartUrl   string            `yaml:"start_url" json:"start_url"`
+	StartStage string            `yaml:"start_stage" json:"start_stage"`
+	Stages     []Stage           `yaml:"stages" json:"stages"`
+	Settings   map[string]string `yaml:"settings" json:"settings"`
+}
+
+type Stage struct {
+	Name      string  `yaml:"name" json:"name"`
+	IsList    bool    `yaml:"is_list" json:"is_list"`
+	ListCss   string  `yaml:"list_css" json:"list_css"`
+	ListXpath string  `yaml:"list_xpath" json:"list_xpath"`
+	PageCss   string  `yaml:"page_css" json:"page_css"`
+	PageXpath string  `yaml:"page_xpath" json:"page_xpath"`
+	PageAttr  string  `yaml:"page_attr" json:"page_attr"`
+	Fields    []Field `yaml:"fields" json:"fields"`
+}
+
+type Field struct {
+	Name      string `yaml:"name" json:"name"`
+	Css       string `yaml:"css" json:"css"`
+	Xpath     string `yaml:"xpath" json:"xpath"`
+	Attr      string `yaml:"attr" json:"attr"`
+	NextStage string `yaml:"next_stage" json:"next_stage"`
+	Remark    string `yaml:"remark" json:"remark"`
+}
--- a/backend/entity/system.go
+++ b/backend/entity/system.go
@@ -13,3 +13,18 @@ type Executable struct {
 	FileName    string `json:"file_name"`
 	DisplayName string `json:"display_name"`
 }
+
+type Lang struct {
+	Name              string `json:"name"`
+	ExecutableName    string `json:"executable_name"`
+	ExecutablePath    string `json:"executable_path"`
+	DepExecutablePath string `json:"dep_executable_path"`
+	Installed         bool   `json:"installed"`
+}
+
+type Dependency struct {
+	Name        string `json:"name"`
+	Version     string `json:"version"`
+	Description string `json:"description"`
+	Installed   bool   `json:"installed"`
+}
--- a/backend/go.mod
+++ b/backend/go.mod
@@ -11,10 +11,12 @@ require (
 	github.com/go-playground/locales v0.12.1 // indirect
 	github.com/go-playground/universal-translator v0.16.0 // indirect
 	github.com/gomodule/redigo v2.0.0+incompatible
+	github.com/imroc/req v0.2.4
 	github.com/leodido/go-urn v1.1.0 // indirect
 	github.com/pkg/errors v0.8.1
 	github.com/satori/go.uuid v1.2.0
 	github.com/smartystreets/goconvey v0.0.0-20190731233626-505e41936337
 	github.com/spf13/viper v1.4.0
 	gopkg.in/go-playground/validator.v9 v9.29.1
+	gopkg.in/yaml.v2 v2.2.2
 )
--- a/backend/go.sum
+++ b/backend/go.sum
@@ -66,6 +66,8 @@ github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t
 github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
+github.com/imroc/req v0.2.4 h1:8XbvaQpERLAJV6as/cB186DtH5f0m5zAOtHEaTQ4ac0=
+github.com/imroc/req v0.2.4/go.mod h1:J9FsaNHDTIVyW/b5r6/Df5qKEEEq2WzZKIgKSajd1AE=
 github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=
--- a/backend/main.go
+++ b/backend/main.go
@@ -31,22 +31,24 @@ func main() {
 		log.Error("init config error:" + err.Error())
 		panic(err)
 	}
-	log.Info("初始化配置成功")
+	log.Info("initialized config successfully")

 	// 初始化日志设置
 	logLevel := viper.GetString("log.level")
 	if logLevel != "" {
 		log.SetLevelFromString(logLevel)
 	}
-	log.Info("初始化日志设置成功")
+	log.Info("initialized log config successfully")

 	if viper.GetString("log.isDeletePeriodically") == "Y" {
 		err := services.InitDeleteLogPeriodically()
 		if err != nil {
-			log.Error("Init DeletePeriodically Failed")
+			log.Error("init DeletePeriodically failed")
 			panic(err)
 		}
-		log.Info("初始化定期清理日志配置成功")
+		log.Info("initialized periodically cleaning log successfully")
+	} else {
+		log.Info("periodically cleaning log is switched off")
 	}

 	// 初始化Mongodb数据库
@@ -55,7 +57,7 @@ func main() {
 		debug.PrintStack()
 		panic(err)
 	}
-	log.Info("初始化Mongodb数据库成功")
+	log.Info("initialized MongoDB successfully")

 	// 初始化Redis数据库
 	if err := database.InitRedis(); err != nil {
@@ -63,7 +65,7 @@ func main() {
 		debug.PrintStack()
 		panic(err)
 	}
-	log.Info("初始化Redis数据库成功")
+	log.Info("initialized Redis successfully")

 	if model.IsMaster() {
 		// 初始化定时任务
@@ -72,8 +74,8 @@ func main() {
 			debug.PrintStack()
 			panic(err)
 		}
-		log.Info("初始化定时任务成功")
 	}
+	log.Info("initialized schedule successfully")

 	// 初始化任务执行器
 	if err := services.InitTaskExecutor(); err != nil {
@@ -81,14 +83,14 @@ func main() {
 		debug.PrintStack()
 		panic(err)
 	}
-	log.Info("初始化任务执行器成功")
+	log.Info("initialized task executor successfully")

 	// 初始化节点服务
 	if err := services.InitNodeService(); err != nil {
 		log.Error("init node service error:" + err.Error())
 		panic(err)
 	}
-	log.Info("初始化节点配置成功")
+	log.Info("initialized node service successfully")

 	// 初始化爬虫服务
 	if err := services.InitSpiderService(); err != nil {
@@ -96,7 +98,7 @@ func main() {
 		debug.PrintStack()
 		panic(err)
 	}
-	log.Info("初始化爬虫服务成功")
+	log.Info("initialized spider service successfully")

 	// 初始化用户服务
 	if err := services.InitUserService(); err != nil {
@@ -104,57 +106,85 @@ func main() {
 		debug.PrintStack()
 		panic(err)
 	}
-	log.Info("初始化用户服务成功")
+	log.Info("initialized user service successfully")
+
+	// 初始化依赖服务
+	if err := services.InitDepsFetcher(); err != nil {
+		log.Error("init user service error:" + err.Error())
+		debug.PrintStack()
+		panic(err)
+	}
+	log.Info("initialized dependency fetcher successfully")

 	// 以下为主节点服务
 	if model.IsMaster() {
 		// 中间件
 		app.Use(middlewares.CORSMiddleware())
-		//app.Use(middlewares.AuthorizationMiddleware())
 		anonymousGroup := app.Group("/")
 		{
-			anonymousGroup.POST("/login", routes.Login)  // 用户登录
-			anonymousGroup.PUT("/users", routes.PutUser) // 添加用户
-
+			anonymousGroup.POST("/login", routes.Login)       // 用户登录
+			anonymousGroup.PUT("/users", routes.PutUser)      // 添加用户
+			anonymousGroup.GET("/setting", routes.GetSetting) // 获取配置信息
 		}
 		authGroup := app.Group("/", middlewares.AuthorizationMiddleware())
 		{
 			// 路由
 			// 节点
-			authGroup.GET("/nodes", routes.GetNodeList)               // 节点列表
-			authGroup.GET("/nodes/:id", routes.GetNode)               // 节点详情
-			authGroup.POST("/nodes/:id", routes.PostNode)             // 修改节点
-			authGroup.GET("/nodes/:id/tasks", routes.GetNodeTaskList) // 节点任务列表
-			authGroup.GET("/nodes/:id/system", routes.GetSystemInfo)  // 节点任务列表
-			authGroup.DELETE("/nodes/:id", routes.DeleteNode)         // 删除节点
+			authGroup.GET("/nodes", routes.GetNodeList)                            // 节点列表
+			authGroup.GET("/nodes/:id", routes.GetNode)                            // 节点详情
+			authGroup.POST("/nodes/:id", routes.PostNode)                          // 修改节点
+			authGroup.GET("/nodes/:id/tasks", routes.GetNodeTaskList)              // 节点任务列表
+			authGroup.GET("/nodes/:id/system", routes.GetSystemInfo)               // 节点任务列表
+			authGroup.DELETE("/nodes/:id", routes.DeleteNode)                      // 删除节点
+			authGroup.GET("/nodes/:id/langs", routes.GetLangList)                  // 节点语言环境列表
+			authGroup.GET("/nodes/:id/deps", routes.GetDepList)                    // 节点第三方依赖列表
+			authGroup.GET("/nodes/:id/deps/installed", routes.GetInstalledDepList) // 节点已安装第三方依赖列表
 			// 爬虫
-			authGroup.GET("/spiders", routes.GetSpiderList)              // 爬虫列表
-			authGroup.GET("/spiders/:id", routes.GetSpider)              // 爬虫详情
-			authGroup.POST("/spiders", routes.PutSpider)                 // 上传爬虫
-			authGroup.POST("/spiders/:id", routes.PostSpider)            // 修改爬虫
-			authGroup.POST("/spiders/:id/publish", routes.PublishSpider) // 发布爬虫
-			authGroup.DELETE("/spiders/:id", routes.DeleteSpider)        // 删除爬虫
-			authGroup.GET("/spiders/:id/tasks", routes.GetSpiderTasks)   // 爬虫任务列表
-			authGroup.GET("/spiders/:id/file", routes.GetSpiderFile)     // 爬虫文件读取
-			authGroup.POST("/spiders/:id/file", routes.PostSpiderFile)   // 爬虫目录写入
-			authGroup.GET("/spiders/:id/dir", routes.GetSpiderDir)       // 爬虫目录
-			authGroup.GET("/spiders/:id/stats", routes.GetSpiderStats)   // 爬虫统计数据
-			authGroup.GET("/spider/types", routes.GetSpiderTypes)        // 爬虫类型
+			authGroup.GET("/spiders", routes.GetSpiderList)                     // 爬虫列表
+			authGroup.GET("/spiders/:id", routes.GetSpider)                     // 爬虫详情
+			authGroup.PUT("/spiders", routes.PutSpider)                         // 添加爬虫
+			authGroup.POST("/spiders", routes.UploadSpider)                     // 上传爬虫
+			authGroup.POST("/spiders/:id", routes.PostSpider)                   // 修改爬虫
+			authGroup.POST("/spiders/:id/publish", routes.PublishSpider)        // 发布爬虫
+			authGroup.POST("/spiders/:id/upload", routes.UploadSpiderFromId)    // 上传爬虫（ID）
+			authGroup.DELETE("/spiders/:id", routes.DeleteSpider)               // 删除爬虫
+			authGroup.GET("/spiders/:id/tasks", routes.GetSpiderTasks)          // 爬虫任务列表
+			authGroup.GET("/spiders/:id/file", routes.GetSpiderFile)            // 爬虫文件读取
+			authGroup.POST("/spiders/:id/file", routes.PostSpiderFile)          // 爬虫文件更改
+			authGroup.PUT("/spiders/:id/file", routes.PutSpiderFile)            // 爬虫文件创建
+			authGroup.PUT("/spiders/:id/dir", routes.PutSpiderDir)              // 爬虫目录创建
+			authGroup.DELETE("/spiders/:id/file", routes.DeleteSpiderFile)      // 爬虫文件删除
+			authGroup.POST("/spiders/:id/file/rename", routes.RenameSpiderFile) // 爬虫文件重命名
+			authGroup.GET("/spiders/:id/dir", routes.GetSpiderDir)              // 爬虫目录
+			authGroup.GET("/spiders/:id/stats", routes.GetSpiderStats)          // 爬虫统计数据
+			authGroup.GET("/spider/types", routes.GetSpiderTypes)               // 爬虫类型
+			// 可配置爬虫
+			authGroup.GET("/config_spiders/:id/config", routes.GetConfigSpiderConfig)           // 获取可配置爬虫配置
+			authGroup.POST("/config_spiders/:id/config", routes.PostConfigSpiderConfig)         // 更改可配置爬虫配置
+			authGroup.PUT("/config_spiders", routes.PutConfigSpider)                            // 添加可配置爬虫
+			authGroup.POST("/config_spiders/:id", routes.PostConfigSpider)                      // 修改可配置爬虫
+			authGroup.POST("/config_spiders/:id/upload", routes.UploadConfigSpider)             // 上传可配置爬虫
+			authGroup.POST("/config_spiders/:id/spiderfile", routes.PostConfigSpiderSpiderfile) // 上传可配置爬虫
+			authGroup.GET("/config_spiders_templates", routes.GetConfigSpiderTemplateList)      // 获取可配置爬虫模版列表
 			// 任务
 			authGroup.GET("/tasks", routes.GetTaskList)                                 // 任务列表
 			authGroup.GET("/tasks/:id", routes.GetTask)                                 // 任务详情
 			authGroup.PUT("/tasks", routes.PutTask)                                     // 派发任务
 			authGroup.DELETE("/tasks/:id", routes.DeleteTask)                           // 删除任务
+			authGroup.DELETE("/tasks_multiple", routes.DeleteMultipleTask)              // 删除多个任务
+			authGroup.DELETE("/tasks_by_status", routes.DeleteTaskByStatus)             //删除指定状态的任务
 			authGroup.POST("/tasks/:id/cancel", routes.CancelTask)                      // 取消任务
 			authGroup.GET("/tasks/:id/log", routes.GetTaskLog)                          // 任务日志
 			authGroup.GET("/tasks/:id/results", routes.GetTaskResults)                  // 任务结果
 			authGroup.GET("/tasks/:id/results/download", routes.DownloadTaskResultsCsv) // 下载任务结果
 			// 定时任务
-			authGroup.GET("/schedules", routes.GetScheduleList)       // 定时任务列表
-			authGroup.GET("/schedules/:id", routes.GetSchedule)       // 定时任务详情
-			authGroup.PUT("/schedules", routes.PutSchedule)           // 创建定时任务
-			authGroup.POST("/schedules/:id", routes.PostSchedule)     // 修改定时任务
-			authGroup.DELETE("/schedules/:id", routes.DeleteSchedule) // 删除定时任务
+			authGroup.GET("/schedules", routes.GetScheduleList)        // 定时任务列表
+			authGroup.GET("/schedules/:id", routes.GetSchedule)        // 定时任务详情
+			authGroup.PUT("/schedules", routes.PutSchedule)            // 创建定时任务
+			authGroup.POST("/schedules/:id", routes.PostSchedule)      // 修改定时任务
+			authGroup.DELETE("/schedules/:id", routes.DeleteSchedule)  // 删除定时任务
+			authGroup.POST("/schedules/:id/stop", routes.StopSchedule) // 停止定时任务
+			authGroup.POST("/schedules/:id/run", routes.RunSchedule)   // 运行定时任务
 			// 统计数据
 			authGroup.GET("/stats/home", routes.GetHomeStats) // 首页统计数据
 			// 用户
@@ -163,6 +193,10 @@ func main() {
 			authGroup.POST("/users/:id", routes.PostUser)     // 更改用户
 			authGroup.DELETE("/users/:id", routes.DeleteUser) // 删除用户
 			authGroup.GET("/me", routes.GetMe)                // 获取自己账户
+			// release版本
+			authGroup.GET("/version", routes.GetVersion) // 获取发布的版本
+			// 系统
+			authGroup.GET("/system/deps", routes.GetAllDepList) // 节点所有第三方依赖列表
 		}

 	}
--- a/backend/mock/node_test.go
+++ b/backend/mock/node_test.go
@@ -42,12 +42,12 @@ func init() {
 	app.DELETE("/tasks/:id", DeleteTask)                           // 删除任务
 	app.GET("/tasks/:id/results", GetTaskResults)                  // 任务结果
 	app.GET("/tasks/:id/results/download", DownloadTaskResultsCsv) // 下载任务结果
-	app.GET("/spiders", GetSpiderList)              // 爬虫列表
-	app.GET("/spiders/:id", GetSpider)              // 爬虫详情
-	app.POST("/spiders/:id", PostSpider)            // 修改爬虫
-	app.DELETE("/spiders/:id",DeleteSpider)        // 删除爬虫
-	app.GET("/spiders/:id/tasks",GetSpiderTasks)   // 爬虫任务列表
-	app.GET("/spiders/:id/dir",GetSpiderDir)       // 爬虫目录
+	app.GET("/spiders", GetSpiderList)                             // 爬虫列表
+	app.GET("/spiders/:id", GetSpider)                             // 爬虫详情
+	app.POST("/spiders/:id", PostSpider)                           // 修改爬虫
+	app.DELETE("/spiders/:id", DeleteSpider)                       // 删除爬虫
+	app.GET("/spiders/:id/tasks", GetSpiderTasks)                  // 爬虫任务列表
+	app.GET("/spiders/:id/dir", GetSpiderDir)                      // 爬虫目录
 }

 //mock test, test data in ./mock
--- a/backend/mock/schedule.go
+++ b/backend/mock/schedule.go
@@ -10,12 +10,15 @@ import (
 	"time"
 )

+var NodeIdss = []bson.ObjectId{bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
+	bson.ObjectIdHex("5d429e6c19f7abede924fee1")}
+
 var scheduleList = []model.Schedule{
 	{
 		Id:       bson.ObjectId("5d429e6c19f7abede924fee2"),
 		Name:     "test schedule",
 		SpiderId: "123",
-		NodeId:   bson.ObjectId("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
@@ -29,7 +32,7 @@ var scheduleList = []model.Schedule{
 		Id:       bson.ObjectId("xx429e6c19f7abede924fee2"),
 		Name:     "test schedule2",
 		SpiderId: "234",
-		NodeId:   bson.ObjectId("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
@@ -100,8 +103,10 @@ func PutSchedule(c *gin.Context) {
 	}

 	// 如果node_id为空，则置为空ObjectId
-	if item.NodeId == "" {
-		item.NodeId = bson.ObjectIdHex(constants.ObjectIdNull)
+	for _, NodeId := range item.NodeIds {
+		if NodeId == "" {
+			NodeId = bson.ObjectIdHex(constants.ObjectIdNull)
+		}
 	}

 	c.JSON(http.StatusOK, Response{
--- a/backend/mock/schedule_test.go
+++ b/backend/mock/schedule_test.go
@@ -75,7 +75,7 @@ func TestPostSchedule(t *testing.T) {
 		Id:       bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
 		Name:     "test schedule",
 		SpiderId: bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
-		NodeId:   bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
@@ -112,7 +112,7 @@ func TestPutSchedule(t *testing.T) {
 		Id:       bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
 		Name:     "test schedule",
 		SpiderId: bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
-		NodeId:   bson.ObjectIdHex("5d429e6c19f7abede924fee2"),
+		NodeIds:  NodeIdss,
 		Cron:     "***1*",
 		EntryId:  10,
 		// 前端展示
--- a/backend/mock/stats.go
+++ b/backend/mock/stats.go
@@ -6,8 +6,6 @@ import (
 	"net/http"
 )

-
-
 var taskDailyItems = []model.TaskDailyItem{
 	{
 		Date:               "2019/08/19",
--- a/backend/mock/system.go
+++ b/backend/mock/system.go
@@ -1 +1 @@
-package mock
+package mock
--- a/backend/mock/user.go
+++ b/backend/mock/user.go
@@ -1 +1 @@
-package mock
+package mock
--- a/backend/model/config_spider/common.go
+++ b/backend/model/config_spider/common.go
@@ -0,0 +1,26 @@
+package config_spider
+
+import "crawlab/entity"
+
+func GetAllFields(data entity.ConfigSpiderData) []entity.Field {
+	var fields []entity.Field
+	for _, stage := range data.Stages {
+		for _, field := range stage.Fields {
+			fields = append(fields, field)
+		}
+	}
+	return fields
+}
+
+func GetStartStageName(data entity.ConfigSpiderData) string {
+	// 如果 start_stage 设置了且在 stages 里，则返回
+	if data.StartStage != "" {
+		return data.StartStage
+	}
+
+	// 否则返回第一个 stage
+	for _, stage := range data.Stages {
+		return stage.Name
+	}
+	return ""
+}
--- a/backend/model/config_spider/scrapy.go
+++ b/backend/model/config_spider/scrapy.go
@@ -0,0 +1,259 @@
+package config_spider
+
+import (
+	"crawlab/constants"
+	"crawlab/entity"
+	"crawlab/model"
+	"crawlab/utils"
+	"errors"
+	"fmt"
+	"path/filepath"
+)
+
+type ScrapyGenerator struct {
+	Spider     model.Spider
+	ConfigData entity.ConfigSpiderData
+}
+
+// 生成爬虫文件
+func (g ScrapyGenerator) Generate() error {
+	// 生成 items.py
+	if err := g.ProcessItems(); err != nil {
+		return err
+	}
+
+	// 生成 spider.py
+	if err := g.ProcessSpider(); err != nil {
+		return err
+	}
+	return nil
+}
+
+// 生成 items.py
+func (g ScrapyGenerator) ProcessItems() error {
+	// 待处理文件名
+	src := g.Spider.Src
+	filePath := filepath.Join(src, "config_spider", "items.py")
+
+	// 获取所有字段
+	fields := g.GetAllFields()
+
+	// 字段名列表（包含默认字段名）
+	fieldNames := []string{
+		"_id",
+		"task_id",
+		"ts",
+	}
+
+	// 加入字段
+	for _, field := range fields {
+		fieldNames = append(fieldNames, field.Name)
+	}
+
+	// 将字段名转化为python代码
+	str := ""
+	for _, fieldName := range fieldNames {
+		line := g.PadCode(fmt.Sprintf("%s = scrapy.Field()", fieldName), 1)
+		str += line
+	}
+
+	// 将占位符替换为代码
+	if err := utils.SetFileVariable(filePath, constants.AnchorItems, str); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// 生成 spider.py
+func (g ScrapyGenerator) ProcessSpider() error {
+	// 待处理文件名
+	src := g.Spider.Src
+	filePath := filepath.Join(src, "config_spider", "spiders", "spider.py")
+
+	// 替换 start_stage
+	if err := utils.SetFileVariable(filePath, constants.AnchorStartStage, "parse_"+GetStartStageName(g.ConfigData)); err != nil {
+		return err
+	}
+
+	// 替换 start_url
+	if err := utils.SetFileVariable(filePath, constants.AnchorStartUrl, g.ConfigData.StartUrl); err != nil {
+		return err
+	}
+
+	// 替换 parsers
+	strParser := ""
+	for _, stage := range g.ConfigData.Stages {
+		stageName := stage.Name
+		stageStr := g.GetParserString(stageName, stage)
+		strParser += stageStr
+	}
+	if err := utils.SetFileVariable(filePath, constants.AnchorParsers, strParser); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func (g ScrapyGenerator) GetParserString(stageName string, stage entity.Stage) string {
+	// 构造函数定义行
+	strDef := g.PadCode(fmt.Sprintf("def parse_%s(self, response):", stageName), 1)
+
+	strParse := ""
+	if stage.IsList {
+		// 列表逻辑
+		strParse = g.GetListParserString(stageName, stage)
+	} else {
+		// 非列表逻辑
+		strParse = g.GetNonListParserString(stageName, stage)
+	}
+
+	// 构造
+	str := fmt.Sprintf(`%s%s`, strDef, strParse)
+
+	return str
+}
+
+func (g ScrapyGenerator) PadCode(str string, num int) string {
+	res := ""
+	for i := 0; i < num; i++ {
+		res += "    "
+	}
+	res += str
+	res += "\n"
+	return res
+}
+
+func (g ScrapyGenerator) GetNonListParserString(stageName string, stage entity.Stage) string {
+	str := ""
+
+	// 获取或构造item
+	str += g.PadCode("item = Item() if response.meta.get('item') is None else response.meta.get('item')", 2)
+
+	// 遍历字段列表
+	for _, f := range stage.Fields {
+		line := fmt.Sprintf(`item['%s'] = response.%s.extract_first()`, f.Name, g.GetExtractStringFromField(f))
+		line = g.PadCode(line, 2)
+		str += line
+	}
+
+	// next stage 字段
+	if f, err := g.GetNextStageField(stage); err == nil {
+		// 如果找到 next stage 字段，进行下一个回调
+		str += g.PadCode(fmt.Sprintf(`yield scrapy.Request(url="get_real_url(response, item['%s'])", callback=self.parse_%s, meta={'item': item})`, f.Name, f.NextStage), 2)
+	} else {
+		// 如果没找到 next stage 字段，返回 item
+		str += g.PadCode(fmt.Sprintf(`yield item`), 2)
+	}
+
+	// 加入末尾换行
+	str += g.PadCode("", 0)
+
+	return str
+}
+
+func (g ScrapyGenerator) GetListParserString(stageName string, stage entity.Stage) string {
+	str := ""
+
+	// 获取前一个 stage 的 item
+	str += g.PadCode(`prev_item = response.meta.get('item')`, 2)
+
+	// for 循环遍历列表
+	str += g.PadCode(fmt.Sprintf(`for elem in response.%s:`, g.GetListString(stage)), 2)
+
+	// 构造item
+	str += g.PadCode(`item = Item()`, 3)
+
+	// 遍历字段列表
+	for _, f := range stage.Fields {
+		line := fmt.Sprintf(`item['%s'] = elem.%s.extract_first()`, f.Name, g.GetExtractStringFromField(f))
+		line = g.PadCode(line, 3)
+		str += line
+	}
+
+	// 把前一个 stage 的 item 值赋给当前 item
+	str += g.PadCode(`if prev_item is not None:`, 3)
+	str += g.PadCode(`for key, value in prev_item.items():`, 4)
+	str += g.PadCode(`item[key] = value`, 5)
+
+	// next stage 字段
+	if f, err := g.GetNextStageField(stage); err == nil {
+		// 如果找到 next stage 字段，进行下一个回调
+		str += g.PadCode(fmt.Sprintf(`yield scrapy.Request(url=get_real_url(response, item['%s']), callback=self.parse_%s, meta={'item': item})`, f.Name, f.NextStage), 3)
+	} else {
+		// 如果没找到 next stage 字段，返回 item
+		str += g.PadCode(fmt.Sprintf(`yield item`), 3)
+	}
+
+	// 分页
+	if stage.PageCss != "" || stage.PageXpath != "" {
+		str += g.PadCode(fmt.Sprintf(`next_url = response.%s.extract_first()`, g.GetExtractStringFromStage(stage)), 2)
+		str += g.PadCode(fmt.Sprintf(`yield scrapy.Request(url=get_real_url(response, next_url), callback=self.parse_%s, meta={'item': prev_item})`, stageName), 2)
+	}
+
+	// 加入末尾换行
+	str += g.PadCode("", 0)
+
+	return str
+}
+
+// 获取所有字段
+func (g ScrapyGenerator) GetAllFields() []entity.Field {
+	return GetAllFields(g.ConfigData)
+}
+
+// 获取包含 next stage 的字段
+func (g ScrapyGenerator) GetNextStageField(stage entity.Stage) (entity.Field, error) {
+	for _, field := range stage.Fields {
+		if field.NextStage != "" {
+			return field, nil
+		}
+	}
+	return entity.Field{}, errors.New("cannot find next stage field")
+}
+
+func (g ScrapyGenerator) GetExtractStringFromField(f entity.Field) string {
+	if f.Css != "" {
+		// 如果为CSS
+		if f.Attr == "" {
+			// 文本
+			return fmt.Sprintf(`css('%s::text')`, f.Css)
+		} else {
+			// 属性
+			return fmt.Sprintf(`css('%s::attr("%s")')`, f.Css, f.Attr)
+		}
+	} else {
+		// 如果为XPath
+		if f.Attr == "" {
+			// 文本
+			return fmt.Sprintf(`xpath('string(%s)')`, f.Xpath)
+		} else {
+			// 属性
+			return fmt.Sprintf(`xpath('%s/@%s')`, f.Xpath, f.Attr)
+		}
+	}
+}
+
+func (g ScrapyGenerator) GetExtractStringFromStage(stage entity.Stage) string {
+	// 分页元素属性，默认为 href
+	pageAttr := "href"
+	if stage.PageAttr != "" {
+		pageAttr = stage.PageAttr
+	}
+
+	if stage.PageCss != "" {
+		// 如果为CSS
+		return fmt.Sprintf(`css('%s::attr("%s")')`, stage.PageCss, pageAttr)
+	} else {
+		// 如果为XPath
+		return fmt.Sprintf(`xpath('%s/@%s')`, stage.PageXpath, pageAttr)
+	}
+}
+
+func (g ScrapyGenerator) GetListString(stage entity.Stage) string {
+	if stage.ListCss != "" {
+		return fmt.Sprintf(`css('%s')`, stage.ListCss)
+	} else {
+		return fmt.Sprintf(`xpath('%s')`, stage.ListXpath)
+	}
+}
--- a/backend/model/node.go
+++ b/backend/model/node.go
@@ -55,7 +55,7 @@ func GetCurrentNode() (Node, error) {
 	for {
 		// 如果错误次数超过10次
 		if errNum >= 10 {
-			panic("cannot get current node")
+			return node, errors.New("cannot get current node")
 		}

 		// 尝试获取节点
@@ -63,7 +63,9 @@ func GetCurrentNode() (Node, error) {
 		// 如果获取失败
 		if err != nil {
 			// 如果为主节点，表示为第一次注册，插入节点信息
-			if IsMaster() {
+			// update: 增加具体错误过滤。防止加入多个master节点，后续需要职责拆分，
+			//只在master节点运行的时候才检测master节点的信息是否存在
+			if IsMaster() && err == mgo.ErrNotFound {
 				// 获取本机信息
 				ip, mac, key, err := GetNodeBaseInfo()
 				if err != nil {
@@ -143,6 +145,7 @@ func (n *Node) GetTasks() ([]Task, error) {
 	return tasks, nil
 }

+// 节点列表
 func GetNodeList(filter interface{}) ([]Node, error) {
 	s, c := database.GetCol("nodes")
 	defer s.Close()
@@ -156,6 +159,7 @@ func GetNodeList(filter interface{}) ([]Node, error) {
 	return results, nil
 }

+// 节点信息
 func GetNode(id bson.ObjectId) (Node, error) {
 	var node Node

@@ -169,13 +173,14 @@ func GetNode(id bson.ObjectId) (Node, error) {
 	defer s.Close()

 	if err := c.FindId(id).One(&node); err != nil {
-		log.Errorf(err.Error())
+		log.Errorf("get node error: %s, id: %s", err.Error(), id.Hex())
 		debug.PrintStack()
 		return node, err
 	}
 	return node, nil
 }

+// 节点信息
 func GetNodeByKey(key string) (Node, error) {
 	s, c := database.GetCol("nodes")
 	defer s.Close()
@@ -191,6 +196,7 @@ func GetNodeByKey(key string) (Node, error) {
 	return node, nil
 }

+// 更新节点
 func UpdateNode(id bson.ObjectId, item Node) error {
 	s, c := database.GetCol("nodes")
 	defer s.Close()
@@ -206,6 +212,7 @@ func UpdateNode(id bson.ObjectId, item Node) error {
 	return nil
 }

+// 任务列表
 func GetNodeTaskList(id bson.ObjectId) ([]Task, error) {
 	node, err := GetNode(id)
 	if err != nil {
@@ -218,6 +225,7 @@ func GetNodeTaskList(id bson.ObjectId) ([]Task, error) {
 	return tasks, nil
 }

+// 节点数
 func GetNodeCount(query interface{}) (int, error) {
 	s, c := database.GetCol("nodes")
 	defer s.Close()
--- a/backend/model/schedule.go
+++ b/backend/model/schedule.go
@@ -12,19 +12,25 @@ import (
 )

 type Schedule struct {
-	Id          bson.ObjectId `json:"_id" bson:"_id"`
-	Name        string        `json:"name" bson:"name"`
-	Description string        `json:"description" bson:"description"`
-	SpiderId    bson.ObjectId `json:"spider_id" bson:"spider_id"`
-	NodeId      bson.ObjectId `json:"node_id" bson:"node_id"`
-	NodeKey     string        `json:"node_key" bson:"node_key"`
-	Cron        string        `json:"cron" bson:"cron"`
-	EntryId     cron.EntryID  `json:"entry_id" bson:"entry_id"`
-	Param       string        `json:"param" bson:"param"`
+	Id          bson.ObjectId   `json:"_id" bson:"_id"`
+	Name        string          `json:"name" bson:"name"`
+	Description string          `json:"description" bson:"description"`
+	SpiderId    bson.ObjectId   `json:"spider_id" bson:"spider_id"`
+	//NodeId      bson.ObjectId   `json:"node_id" bson:"node_id"`
+	//NodeKey     string          `json:"node_key" bson:"node_key"`
+	Cron        string          `json:"cron" bson:"cron"`
+	EntryId     cron.EntryID    `json:"entry_id" bson:"entry_id"`
+	Param       string          `json:"param" bson:"param"`
+	RunType     string          `json:"run_type" bson:"run_type"`
+	NodeIds     []bson.ObjectId `json:"node_ids" bson:"node_ids"`
+
+	// 状态
+	Status string `json:"status" bson:"status"`

 	// 前端展示
 	SpiderName string `json:"spider_name" bson:"spider_name"`
 	NodeName   string `json:"node_name" bson:"node_name"`
+	Message    string `json:"message" bson:"message"`

 	CreateTs time.Time `json:"create_ts" bson:"create_ts"`
 	UpdateTs time.Time `json:"update_ts" bson:"update_ts"`
@@ -46,26 +52,26 @@ func (sch *Schedule) Delete() error {
 	return c.RemoveId(sch.Id)
 }

-func (sch *Schedule) SyncNodeIdAndSpiderId(node Node, spider Spider) {
-	sch.syncNodeId(node)
-	sch.syncSpiderId(spider)
-}
+//func (sch *Schedule) SyncNodeIdAndSpiderId(node Node, spider Spider) {
+//	sch.syncNodeId(node)
+//	sch.syncSpiderId(spider)
+//}

-func (sch *Schedule) syncNodeId(node Node) {
-	if node.Id.Hex() == sch.NodeId.Hex() {
-		return
-	}
-	sch.NodeId = node.Id
-	_ = sch.Save()
-}
+//func (sch *Schedule) syncNodeId(node Node) {
+//	if node.Id.Hex() == sch.NodeId.Hex() {
+//		return
+//	}
+//	sch.NodeId = node.Id
+//	_ = sch.Save()
+//}

-func (sch *Schedule) syncSpiderId(spider Spider) {
-	if spider.Id.Hex() == sch.SpiderId.Hex() {
-		return
-	}
-	sch.SpiderId = spider.Id
-	_ = sch.Save()
-}
+//func (sch *Schedule) syncSpiderId(spider Spider) {
+//	if spider.Id.Hex() == sch.SpiderId.Hex() {
+//		return
+//	}
+//	sch.SpiderId = spider.Id
+//	_ = sch.Save()
+//}

 func GetScheduleList(filter interface{}) ([]Schedule, error) {
 	s, c := database.GetCol("schedules")
@@ -78,29 +84,31 @@ func GetScheduleList(filter interface{}) ([]Schedule, error) {

 	var schs []Schedule
 	for _, schedule := range schedules {
-		// 获取节点名称
-		if schedule.NodeId == bson.ObjectIdHex(constants.ObjectIdNull) {
-			// 选择所有节点
-			schedule.NodeName = "All Nodes"
-		} else {
-			// 选择单一节点
-			node, err := GetNode(schedule.NodeId)
-			if err != nil {
-				log.Errorf(err.Error())
-				continue
-			}
-			schedule.NodeName = node.Name
-		}
+		// TODO: 获取节点名称
+		//if schedule.NodeId == bson.ObjectIdHex(constants.ObjectIdNull) {
+		//	// 选择所有节点
+		//	schedule.NodeName = "All Nodes"
+		//} else {
+		//	// 选择单一节点
+		//	node, err := GetNode(schedule.NodeId)
+		//	if err != nil {
+		//		schedule.Status = constants.ScheduleStatusError
+		//		schedule.Message = constants.ScheduleStatusErrorNotFoundNode
+		//	} else {
+		//		schedule.NodeName = node.Name
+		//	}
+		//}

 		// 获取爬虫名称
 		spider, err := GetSpider(schedule.SpiderId)
 		if err != nil && err == mgo.ErrNotFound {
 			log.Errorf("get spider by id: %s, error: %s", schedule.SpiderId.Hex(), err.Error())
-			debug.PrintStack()
-			_ = schedule.Delete()
-			continue
+			schedule.Status = constants.ScheduleStatusError
+			schedule.Message = constants.ScheduleStatusErrorNotFoundSpider
+		} else {
+			schedule.SpiderName = spider.Name
 		}
-		schedule.SpiderName = spider.Name
+
 		schs = append(schs, schedule)
 	}
 	return schs, nil
@@ -125,12 +133,13 @@ func UpdateSchedule(id bson.ObjectId, item Schedule) error {
 	if err := c.FindId(id).One(&result); err != nil {
 		return err
 	}
-	node, err := GetNode(item.NodeId)
-	if err != nil {
-		return err
-	}
+	//node, err := GetNode(item.NodeId)
+	//if err != nil {
+	//	return err
+	//}

-	item.NodeKey = node.Key
+	item.UpdateTs = time.Now()
+	//item.NodeKey = node.Key
 	if err := item.Save(); err != nil {
 		return err
 	}
@@ -141,15 +150,15 @@ func AddSchedule(item Schedule) error {
 	s, c := database.GetCol("schedules")
 	defer s.Close()

-	node, err := GetNode(item.NodeId)
-	if err != nil {
-		return err
-	}
+	//node, err := GetNode(item.NodeId)
+	//if err != nil {
+	//	return err
+	//}

 	item.Id = bson.NewObjectId()
 	item.CreateTs = time.Now()
 	item.UpdateTs = time.Now()
-	item.NodeKey = node.Key
+	//item.NodeKey = node.Key

 	if err := c.Insert(&item); err != nil {
 		debug.PrintStack()
--- a/backend/model/spider.go
+++ b/backend/model/spider.go
@@ -1,11 +1,17 @@
 package model

 import (
+	"crawlab/constants"
 	"crawlab/database"
 	"crawlab/entity"
+	"crawlab/utils"
+	"errors"
 	"github.com/apex/log"
 	"github.com/globalsign/mgo"
 	"github.com/globalsign/mgo/bson"
+	"gopkg.in/yaml.v2"
+	"io/ioutil"
+	"path/filepath"
 	"runtime/debug"
 	"time"
 )
@@ -25,25 +31,20 @@ type Spider struct {
 	Site        string        `json:"site" bson:"site"`                 // 爬虫网站
 	Envs        []Env         `json:"envs" bson:"envs"`                 // 环境变量
 	Remark      string        `json:"remark" bson:"remark"`             // 备注
+	Src         string        `json:"src" bson:"src"`                   // 源码位置
+
 	// 自定义爬虫
-	Src string `json:"src" bson:"src"` // 源码位置
 	Cmd string `json:"cmd" bson:"cmd"` // 执行命令

+	// 可配置爬虫
+	Template string `json:"template" bson:"template"` // Spiderfile模版
+
 	// 前端展示
-	LastRunTs  time.Time `json:"last_run_ts"` // 最后一次执行时间
-	LastStatus string    `json:"last_status"` // 最后执行状态
-
-	// TODO: 可配置爬虫
-	//Fields                 []interface{} `json:"fields"`
-	//DetailFields           []interface{} `json:"detail_fields"`
-	//CrawlType              string        `json:"crawl_type"`
-	//StartUrl               string        `json:"start_url"`
-	//UrlPattern             string        `json:"url_pattern"`
-	//ItemSelector           string        `json:"item_selector"`
-	//ItemSelectorType       string        `json:"item_selector_type"`
-	//PaginationSelector     string        `json:"pagination_selector"`
-	//PaginationSelectorType string        `json:"pagination_selector_type"`
+	LastRunTs  time.Time               `json:"last_run_ts"` // 最后一次执行时间
+	LastStatus string                  `json:"last_status"` // 最后执行状态
+	Config     entity.ConfigSpiderData `json:"config"`      // 可配置爬虫配置

+	// 时间
 	CreateTs time.Time `json:"create_ts" bson:"create_ts"`
 	UpdateTs time.Time `json:"update_ts" bson:"update_ts"`
 }
@@ -98,13 +99,14 @@ func (spider *Spider) GetLastTask() (Task, error) {
 	return tasks[0], nil
 }

+// 删除爬虫
 func (spider *Spider) Delete() error {
 	s, c := database.GetCol("spiders")
 	defer s.Close()
 	return c.RemoveId(spider.Id)
 }

-// 爬虫列表
+// 获取爬虫列表
 func GetSpiderList(filter interface{}, skip int, limit int) ([]Spider, int, error) {
 	s, c := database.GetCol("spiders")
 	defer s.Close()
@@ -116,6 +118,10 @@ func GetSpiderList(filter interface{}, skip int, limit int) ([]Spider, int, erro
 		return spiders, 0, err
 	}

+	if spiders == nil {
+		spiders = []Spider{}
+	}
+
 	// 遍历爬虫列表
 	for i, spider := range spiders {
 		// 获取最后一次任务
@@ -136,7 +142,7 @@ func GetSpiderList(filter interface{}, skip int, limit int) ([]Spider, int, erro
 	return spiders, count, nil
 }

-// 获取爬虫
+// 获取爬虫(根据FileId)
 func GetSpiderByFileId(fileId bson.ObjectId) *Spider {
 	s, c := database.GetCol("spiders")
 	defer s.Close()
@@ -150,34 +156,44 @@ func GetSpiderByFileId(fileId bson.ObjectId) *Spider {
 	return result
 }

-// 获取爬虫
-func GetSpiderByName(name string) *Spider {
-	s, c := database.GetCol("spiders")
-	defer s.Close()
-
-	var result *Spider
-	if err := c.Find(bson.M{"name": name}).One(&result); err != nil {
-		log.Errorf("get spider error: %s, spider_name: %s", err.Error(), name)
-		debug.PrintStack()
-		return nil
-	}
-	return result
-}
-
-// 获取爬虫
-func GetSpider(id bson.ObjectId) (Spider, error) {
+// 获取爬虫(根据名称)
+func GetSpiderByName(name string) Spider {
 	s, c := database.GetCol("spiders")
 	defer s.Close()

 	var result Spider
-	if err := c.FindId(id).One(&result); err != nil {
+	if err := c.Find(bson.M{"name": name}).One(&result); err != nil {
+		log.Errorf("get spider error: %s, spider_name: %s", err.Error(), name)
+		//debug.PrintStack()
+		return result
+	}
+	return result
+}
+
+// 获取爬虫(根据ID)
+func GetSpider(id bson.ObjectId) (Spider, error) {
+	s, c := database.GetCol("spiders")
+	defer s.Close()
+
+	// 获取爬虫
+	var spider Spider
+	if err := c.FindId(id).One(&spider); err != nil {
 		if err != mgo.ErrNotFound {
 			log.Errorf("get spider error: %s, id: %id", err.Error(), id.Hex())
 			debug.PrintStack()
 		}
-		return result, err
+		return spider, err
 	}
-	return result, nil
+
+	// 如果为可配置爬虫，获取爬虫配置
+	if spider.Type == constants.Configurable && utils.Exists(filepath.Join(spider.Src, "Spiderfile")) {
+		config, err := GetConfigSpiderData(spider)
+		if err != nil {
+			return spider, err
+		}
+		spider.Config = config
+	}
+	return spider, nil
 }

 // 更新爬虫
@@ -217,10 +233,12 @@ func RemoveSpider(id bson.ObjectId) error {
 	s, gf := database.GetGridFs("files")
 	defer s.Close()

-	if err := gf.RemoveId(result.FileId); err != nil {
-		log.Error("remove file error, id:" + result.FileId.Hex())
-		debug.PrintStack()
-		return err
+	if result.FileId.Hex() != constants.ObjectIdNull {
+		if err := gf.RemoveId(result.FileId); err != nil {
+			log.Error("remove file error, id:" + result.FileId.Hex())
+			debug.PrintStack()
+			return err
+		}
 	}

 	return nil
@@ -245,7 +263,7 @@ func RemoveAllSpider() error {
 	return nil
 }

-// 爬虫总数
+// 获取爬虫总数
 func GetSpiderCount() (int, error) {
 	s, c := database.GetCol("spiders")
 	defer s.Close()
@@ -257,7 +275,7 @@ func GetSpiderCount() (int, error) {
 	return count, nil
 }

-// 爬虫类型
+// 获取爬虫类型
 func GetSpiderTypes() ([]*entity.SpiderType, error) {
 	s, c := database.GetCol("spiders")
 	defer s.Close()
@@ -277,3 +295,29 @@ func GetSpiderTypes() ([]*entity.SpiderType, error) {

 	return types, nil
 }
+
+func GetConfigSpiderData(spider Spider) (entity.ConfigSpiderData, error) {
+	// 构造配置数据
+	configData := entity.ConfigSpiderData{}
+
+	// 校验爬虫类别
+	if spider.Type != constants.Configurable {
+		return configData, errors.New("not a configurable spider")
+	}
+
+	// Spiderfile 目录
+	sfPath := filepath.Join(spider.Src, "Spiderfile")
+
+	// 读取YAML文件
+	yamlFile, err := ioutil.ReadFile(sfPath)
+	if err != nil {
+		return configData, err
+	}
+
+	// 反序列化
+	if err := yaml.Unmarshal(yamlFile, &configData); err != nil {
+		return configData, err
+	}
+
+	return configData, nil
+}
--- a/backend/model/task.go
+++ b/backend/model/task.go
@@ -61,6 +61,7 @@ func (t *Task) Save() error {
 	defer s.Close()
 	t.UpdateTs = time.Now()
 	if err := c.UpdateId(t.Id, t); err != nil {
+		log.Errorf("update task error: %s", err.Error())
 		debug.PrintStack()
 		return err
 	}
@@ -152,6 +153,7 @@ func GetTask(id string) (Task, error) {

 	var task Task
 	if err := c.FindId(id).One(&task); err != nil {
+		log.Infof("get task error: %s, id: %s", err.Error(), id)
 		debug.PrintStack()
 		return task, err
 	}
@@ -187,6 +189,20 @@ func RemoveTask(id string) error {
 	return nil
 }

+func RemoveTaskByStatus(status string) error {
+	tasks, err := GetTaskList(bson.M{"status": status}, 0, constants.Infinite, "-create_ts")
+	if err != nil {
+		log.Error("get tasks error:" + err.Error())
+	}
+	for _, task := range tasks {
+		if err := RemoveTask(task.Id); err != nil {
+			log.Error("remove task error:" + err.Error())
+			continue
+		}
+	}
+	return nil
+}
+
 // 删除task by spider_id
 func RemoveTaskBySpiderId(id bson.ObjectId) error {
 	tasks, err := GetTaskList(bson.M{"spider_id": id}, 0, constants.Infinite, "-create_ts")
--- a/backend/routes/config_spider.go
+++ b/backend/routes/config_spider.go
@@ -0,0 +1,316 @@
+package routes
+
+import (
+	"crawlab/constants"
+	"crawlab/entity"
+	"crawlab/model"
+	"crawlab/services"
+	"crawlab/utils"
+	"fmt"
+	"github.com/gin-gonic/gin"
+	"github.com/globalsign/mgo/bson"
+	"github.com/spf13/viper"
+	"gopkg.in/yaml.v2"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+// 添加可配置爬虫
+func PutConfigSpider(c *gin.Context) {
+	var spider model.Spider
+	if err := c.ShouldBindJSON(&spider); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+
+	// 爬虫名称不能为空
+	if spider.Name == "" {
+		HandleErrorF(http.StatusBadRequest, c, "spider name should not be empty")
+		return
+	}
+
+	// 模版名不能为空
+	if spider.Template == "" {
+		HandleErrorF(http.StatusBadRequest, c, "spider template should not be empty")
+		return
+	}
+
+	// 判断爬虫是否存在
+	if spider := model.GetSpiderByName(spider.Name); spider.Name != "" {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("spider for '%s' already exists", spider.Name))
+		return
+	}
+
+	// 设置爬虫类别
+	spider.Type = constants.Configurable
+
+	// 将FileId置空
+	spider.FileId = bson.ObjectIdHex(constants.ObjectIdNull)
+
+	// 创建爬虫目录
+	spiderDir := filepath.Join(viper.GetString("spider.path"), spider.Name)
+	if utils.Exists(spiderDir) {
+		if err := os.RemoveAll(spiderDir); err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+	}
+	if err := os.MkdirAll(spiderDir, 0777); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	spider.Src = spiderDir
+
+	// 复制Spiderfile模版
+	contentByte, err := ioutil.ReadFile("./template/spiderfile/Spiderfile." + spider.Template)
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	f, err := os.Create(filepath.Join(spider.Src, "Spiderfile"))
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	defer f.Close()
+	if _, err := f.Write(contentByte); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 添加爬虫到数据库
+	if err := spider.Add(); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    spider,
+	})
+}
+
+// 更改可配置爬虫
+func PostConfigSpider(c *gin.Context) {
+	PostSpider(c)
+}
+
+// 上传可配置爬虫Spiderfile
+func UploadConfigSpider(c *gin.Context) {
+	id := c.Param("id")
+
+	// 获取爬虫
+	var spider model.Spider
+	spider, err := model.GetSpider(bson.ObjectIdHex(id))
+	if err != nil {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("cannot find spider (id: %s)", id))
+	}
+
+	// 获取上传文件
+	file, header, err := c.Request.FormFile("file")
+	if err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+
+	// 文件名称必须为Spiderfile
+	filename := header.Filename
+	if filename != "Spiderfile" && filename != "Spiderfile.yaml" && filename != "Spiderfile.yml" {
+		HandleErrorF(http.StatusBadRequest, c, "filename must be 'Spiderfile(.yaml|.yml)'")
+		return
+	}
+
+	// 爬虫目录
+	spiderDir := filepath.Join(viper.GetString("spider.path"), spider.Name)
+
+	// 爬虫Spiderfile文件路径
+	sfPath := filepath.Join(spiderDir, filename)
+
+	// 创建（如果不存在）或打开Spiderfile（如果存在）
+	var f *os.File
+	if utils.Exists(sfPath) {
+		f, err = os.OpenFile(sfPath, os.O_WRONLY, 0777)
+		if err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+		}
+	} else {
+		f, err = os.Create(sfPath)
+		if err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+		}
+	}
+
+	// 将上传的文件拷贝到爬虫Spiderfile文件
+	_, err = io.Copy(f, file)
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 关闭Spiderfile文件
+	_ = f.Close()
+
+	// 构造配置数据
+	configData := entity.ConfigSpiderData{}
+
+	// 读取YAML文件
+	yamlFile, err := ioutil.ReadFile(sfPath)
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 反序列化
+	if err := yaml.Unmarshal(yamlFile, &configData); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 根据序列化后的数据处理爬虫文件
+	if err := services.ProcessSpiderFilesFromConfigData(spider, configData); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
+func PostConfigSpiderSpiderfile(c *gin.Context) {
+	type Body struct {
+		Content string `json:"content"`
+	}
+
+	id := c.Param("id")
+
+	// 文件内容
+	var reqBody Body
+	if err := c.ShouldBindJSON(&reqBody); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+	content := reqBody.Content
+
+	// 获取爬虫
+	var spider model.Spider
+	spider, err := model.GetSpider(bson.ObjectIdHex(id))
+	if err != nil {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("cannot find spider (id: %s)", id))
+		return
+	}
+
+	// 反序列化
+	var configData entity.ConfigSpiderData
+	if err := yaml.Unmarshal([]byte(content), &configData); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+
+	// 校验configData
+	if err := services.ValidateSpiderfile(configData); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 写文件
+	if err := ioutil.WriteFile(filepath.Join(spider.Src, "Spiderfile"), []byte(content), os.ModePerm); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 根据序列化后的数据处理爬虫文件
+	if err := services.ProcessSpiderFilesFromConfigData(spider, configData); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
+func PostConfigSpiderConfig(c *gin.Context) {
+	id := c.Param("id")
+
+	// 获取爬虫
+	var spider model.Spider
+	spider, err := model.GetSpider(bson.ObjectIdHex(id))
+	if err != nil {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("cannot find spider (id: %s)", id))
+		return
+	}
+
+	// 反序列化配置数据
+	var configData entity.ConfigSpiderData
+	if err := c.ShouldBindJSON(&configData); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+
+	// 校验configData
+	if err := services.ValidateSpiderfile(configData); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 替换Spiderfile文件
+	if err := services.GenerateSpiderfileFromConfigData(spider, configData); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 根据序列化后的数据处理爬虫文件
+	if err := services.ProcessSpiderFilesFromConfigData(spider, configData); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
+func GetConfigSpiderConfig(c *gin.Context) {
+	id := c.Param("id")
+
+	// 校验ID
+	if !bson.IsObjectIdHex(id) {
+		HandleErrorF(http.StatusBadRequest, c, "invalid id")
+	}
+
+	// 获取爬虫
+	spider, err := model.GetSpider(bson.ObjectIdHex(id))
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    spider.Config,
+	})
+}
+
+// 获取模版名称列表
+func GetConfigSpiderTemplateList(c *gin.Context) {
+	var data []string
+	for _, fInfo := range utils.ListDir("./template/spiderfile") {
+		templateName := strings.Replace(fInfo.Name(), "Spiderfile.", "", -1)
+		data = append(data, templateName)
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    data,
+	})
+}
--- a/backend/routes/schedule.go
+++ b/backend/routes/schedule.go
@@ -14,11 +14,7 @@ func GetScheduleList(c *gin.Context) {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-		Data:    results,
-	})
+	HandleSuccessData(c, results)
 }

 func GetSchedule(c *gin.Context) {
@@ -29,11 +25,8 @@ func GetSchedule(c *gin.Context) {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-		Data:    result,
-	})
+
+	HandleSuccessData(c, result)
 }

 func PostSchedule(c *gin.Context) {
@@ -48,7 +41,7 @@ func PostSchedule(c *gin.Context) {

 	// 验证cron表达式
 	if err := services.ParserCron(newItem.Cron); err != nil {
-		HandleError(http.StatusOK, c, err)
+		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}

@@ -65,10 +58,7 @@ func PostSchedule(c *gin.Context) {
 		return
 	}

-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-	})
+	HandleSuccess(c)
 }

 func PutSchedule(c *gin.Context) {
@@ -82,7 +72,7 @@ func PutSchedule(c *gin.Context) {

 	// 验证cron表达式
 	if err := services.ParserCron(item.Cron); err != nil {
-		HandleError(http.StatusOK, c, err)
+		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}

@@ -98,10 +88,7 @@ func PutSchedule(c *gin.Context) {
 		return
 	}

-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-	})
+	HandleSuccess(c)
 }

 func DeleteSchedule(c *gin.Context) {
@@ -119,8 +106,25 @@ func DeleteSchedule(c *gin.Context) {
 		return
 	}

-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-	})
+	HandleSuccess(c)
+}
+
+// 停止定时任务
+func StopSchedule(c *gin.Context) {
+	id := c.Param("id")
+	if err := services.Sched.Stop(bson.ObjectIdHex(id)); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	HandleSuccess(c)
+}
+
+// 运行定时任务
+func RunSchedule(c *gin.Context) {
+	id := c.Param("id")
+	if err := services.Sched.Run(bson.ObjectIdHex(id)); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	HandleSuccess(c)
 }
--- a/backend/routes/setting.go
+++ b/backend/routes/setting.go
@@ -0,0 +1,33 @@
+package routes
+
+import (
+	"github.com/gin-gonic/gin"
+	"github.com/spf13/viper"
+	"net/http"
+)
+
+type SettingBody struct {
+	AllowRegister string `json:"allow_register"`
+}
+
+func GetVersion(c *gin.Context) {
+	version := viper.GetString("version")
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    version,
+	})
+}
+
+func GetSetting(c *gin.Context) {
+	allowRegister := viper.GetString("setting.allowRegister")
+
+	body := SettingBody{AllowRegister: allowRegister}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    body,
+	})
+}
--- a/backend/routes/spider.go
+++ b/backend/routes/spider.go
@@ -7,6 +7,7 @@ import (
 	"crawlab/model"
 	"crawlab/services"
 	"crawlab/utils"
+	"fmt"
 	"github.com/apex/log"
 	"github.com/gin-gonic/gin"
 	"github.com/globalsign/mgo"
@@ -17,6 +18,7 @@ import (
 	"io/ioutil"
 	"net/http"
 	"os"
+	"path"
 	"path/filepath"
 	"runtime/debug"
 	"strconv"
@@ -34,7 +36,7 @@ func GetSpiderList(c *gin.Context) {
 		"name": bson.M{"$regex": bson.RegEx{Pattern: keyword, Options: "im"}},
 	}

-	if t != "" {
+	if t != "" && t != "all" {
 		filter["type"] = t
 	}

@@ -117,6 +119,64 @@ func PublishSpider(c *gin.Context) {
 }

 func PutSpider(c *gin.Context) {
+	var spider model.Spider
+	if err := c.ShouldBindJSON(&spider); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+
+	// 爬虫名称不能为空
+	if spider.Name == "" {
+		HandleErrorF(http.StatusBadRequest, c, "spider name should not be empty")
+		return
+	}
+
+	// 判断爬虫是否存在
+	if spider := model.GetSpiderByName(spider.Name); spider.Name != "" {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("spider for '%s' already exists", spider.Name))
+		return
+	}
+
+	// 设置爬虫类别
+	spider.Type = constants.Customized
+
+	// 将FileId置空
+	spider.FileId = bson.ObjectIdHex(constants.ObjectIdNull)
+
+	// 创建爬虫目录
+	spiderDir := filepath.Join(viper.GetString("spider.path"), spider.Name)
+	if utils.Exists(spiderDir) {
+		if err := os.RemoveAll(spiderDir); err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+	}
+	if err := os.MkdirAll(spiderDir, 0777); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	spider.Src = spiderDir
+
+	// 添加爬虫到数据库
+	if err := spider.Add(); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 同步到GridFS
+	if err := services.UploadSpiderToGridFsFromMaster(spider); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    spider,
+	})
+}
+
+func UploadSpider(c *gin.Context) {
 	// 从body中获取文件
 	uploadFile, err := c.FormFile("file")
 	if err != nil {
@@ -125,6 +185,144 @@ func PutSpider(c *gin.Context) {
 		return
 	}

+	// 获取参数
+	name := c.PostForm("name")
+	displayName := c.PostForm("display_name")
+	col := c.PostForm("col")
+	cmd := c.PostForm("cmd")
+
+	// 如果不为zip文件，返回错误
+	if !strings.HasSuffix(uploadFile.Filename, ".zip") {
+		HandleError(http.StatusBadRequest, c, errors.New("not a valid zip file"))
+		return
+	}
+
+	// 以防tmp目录不存在
+	tmpPath := viper.GetString("other.tmppath")
+	if !utils.Exists(tmpPath) {
+		if err := os.MkdirAll(tmpPath, os.ModePerm); err != nil {
+			log.Error("mkdir other.tmppath dir error:" + err.Error())
+			debug.PrintStack()
+			HandleError(http.StatusBadRequest, c, errors.New("mkdir other.tmppath dir error"))
+			return
+		}
+	}
+
+	// 保存到本地临时文件
+	randomId := uuid.NewV4()
+	tmpFilePath := filepath.Join(tmpPath, randomId.String()+".zip")
+	if err := c.SaveUploadedFile(uploadFile, tmpFilePath); err != nil {
+		log.Error("save upload file error: " + err.Error())
+		debug.PrintStack()
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 获取 GridFS 实例
+	s, gf := database.GetGridFs("files")
+	defer s.Close()
+
+	// 判断文件是否已经存在
+	var gfFile model.GridFs
+	if err := gf.Find(bson.M{"filename": uploadFile.Filename}).One(&gfFile); err == nil {
+		// 已经存在文件，则删除
+		_ = gf.RemoveId(gfFile.Id)
+	}
+
+	// 上传到GridFs
+	fid, err := services.UploadToGridFs(uploadFile.Filename, tmpFilePath)
+	if err != nil {
+		log.Errorf("upload to grid fs error: %s", err.Error())
+		debug.PrintStack()
+		return
+	}
+
+	idx := strings.LastIndex(uploadFile.Filename, "/")
+	targetFilename := uploadFile.Filename[idx+1:]
+
+	// 判断爬虫是否存在
+	spiderName := strings.Replace(targetFilename, ".zip", "", 1)
+	if name != "" {
+		spiderName = name
+	}
+	spider := model.GetSpiderByName(spiderName)
+	if spider.Name == "" {
+		// 保存爬虫信息
+		srcPath := viper.GetString("spider.path")
+		spider := model.Spider{
+			Name:        spiderName,
+			DisplayName: spiderName,
+			Type:        constants.Customized,
+			Src:         filepath.Join(srcPath, spiderName),
+			FileId:      fid,
+		}
+		if name != "" {
+			spider.Name = name
+		}
+		if displayName != "" {
+			spider.DisplayName = displayName
+		}
+		if col != "" {
+			spider.Col = col
+		}
+		if cmd != "" {
+			spider.Cmd = cmd
+		}
+		_ = spider.Add()
+	} else {
+		if name != "" {
+			spider.Name = name
+		}
+		if displayName != "" {
+			spider.DisplayName = displayName
+		}
+		if col != "" {
+			spider.Col = col
+		}
+		if cmd != "" {
+			spider.Cmd = cmd
+		}
+		// 更新file_id
+		spider.FileId = fid
+		_ = spider.Save()
+	}
+
+	// 发起同步
+	services.PublishAllSpiders()
+
+	// 获取爬虫
+	spider = model.GetSpiderByName(spiderName)
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    spider,
+	})
+}
+
+func UploadSpiderFromId(c *gin.Context) {
+	// TODO: 与 UploadSpider 部分逻辑重复，需要优化代码
+	// 爬虫ID
+	spiderId := c.Param("id")
+
+	// 获取爬虫
+	spider, err := model.GetSpider(bson.ObjectIdHex(spiderId))
+	if err != nil {
+		if err == mgo.ErrNotFound {
+			HandleErrorF(http.StatusNotFound, c, "cannot find spider")
+		} else {
+			HandleError(http.StatusInternalServerError, c, err)
+		}
+		return
+	}
+
+	// 从body中获取文件
+	uploadFile, err := c.FormFile("file")
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
 	// 如果不为zip文件，返回错误
 	if !strings.HasSuffix(uploadFile.Filename, ".zip") {
 		debug.PrintStack()
@@ -153,6 +351,7 @@ func PutSpider(c *gin.Context) {
 		return
 	}

+	// 获取 GridFS 实例
 	s, gf := database.GetGridFs("files")
 	defer s.Close()

@@ -171,28 +370,12 @@ func PutSpider(c *gin.Context) {
 		return
 	}

-	idx := strings.LastIndex(uploadFile.Filename, "/")
-	targetFilename := uploadFile.Filename[idx+1:]
+	// 更新file_id
+	spider.FileId = fid
+	_ = spider.Save()

-	// 判断爬虫是否存在
-	spiderName := strings.Replace(targetFilename, ".zip", "", 1)
-	spider := model.GetSpiderByName(spiderName)
-	if spider == nil {
-		// 保存爬虫信息
-		srcPath := viper.GetString("spider.path")
-		spider := model.Spider{
-			Name:        spiderName,
-			DisplayName: spiderName,
-			Type:        constants.Customized,
-			Src:         filepath.Join(srcPath, spiderName),
-			FileId:      fid,
-		}
-		_ = spider.Add()
-	} else {
-		// 更新file_id
-		spider.FileId = fid
-		_ = spider.Save()
-	}
+	// 发起同步
+	services.PublishSpider(spider)

 	c.JSON(http.StatusOK, Response{
 		Status:  "ok",
@@ -282,6 +465,14 @@ func GetSpiderDir(c *gin.Context) {
 	})
 }

+// 爬虫文件管理
+
+type SpiderFileReqBody struct {
+	Path    string `json:"path"`
+	Content string `json:"content"`
+	NewPath string `json:"new_path"`
+}
+
 func GetSpiderFile(c *gin.Context) {
 	// 爬虫ID
 	id := c.Param("id")
@@ -310,11 +501,6 @@ func GetSpiderFile(c *gin.Context) {
 	})
 }

-type SpiderFileReqBody struct {
-	Path    string `json:"path"`
-	Content string `json:"content"`
-}
-
 func PostSpiderFile(c *gin.Context) {
 	// 爬虫ID
 	id := c.Param("id")
@@ -339,6 +525,12 @@ func PostSpiderFile(c *gin.Context) {
 		return
 	}

+	// 同步到GridFS
+	if err := services.UploadSpiderToGridFsFromMaster(spider); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
 	// 返回结果
 	c.JSON(http.StatusOK, Response{
 		Status:  "ok",
@@ -346,6 +538,161 @@ func PostSpiderFile(c *gin.Context) {
 	})
 }

+func PutSpiderFile(c *gin.Context) {
+	spiderId := c.Param("id")
+	var reqBody SpiderFileReqBody
+	if err := c.ShouldBindJSON(&reqBody); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+	spider, err := model.GetSpider(bson.ObjectIdHex(spiderId))
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 文件路径
+	filePath := path.Join(spider.Src, reqBody.Path)
+
+	// 如果文件已存在，则报错
+	if utils.Exists(filePath) {
+		HandleErrorF(http.StatusInternalServerError, c, fmt.Sprintf(`%s already exists`, filePath))
+		return
+	}
+
+	// 写入文件
+	if err := ioutil.WriteFile(filePath, []byte(reqBody.Content), 0777); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 同步到GridFS
+	if err := services.UploadSpiderToGridFsFromMaster(spider); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
+func PutSpiderDir(c *gin.Context) {
+	spiderId := c.Param("id")
+	var reqBody SpiderFileReqBody
+	if err := c.ShouldBindJSON(&reqBody); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+	spider, err := model.GetSpider(bson.ObjectIdHex(spiderId))
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 文件路径
+	filePath := path.Join(spider.Src, reqBody.Path)
+
+	// 如果文件已存在，则报错
+	if utils.Exists(filePath) {
+		HandleErrorF(http.StatusInternalServerError, c, fmt.Sprintf(`%s already exists`, filePath))
+		return
+	}
+
+	// 创建文件夹
+	if err := os.MkdirAll(filePath, 0777); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 同步到GridFS
+	if err := services.UploadSpiderToGridFsFromMaster(spider); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
+func DeleteSpiderFile(c *gin.Context) {
+	spiderId := c.Param("id")
+	var reqBody SpiderFileReqBody
+	if err := c.ShouldBindJSON(&reqBody); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+		return
+	}
+	spider, err := model.GetSpider(bson.ObjectIdHex(spiderId))
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	filePath := path.Join(spider.Src, reqBody.Path)
+	if err := os.RemoveAll(filePath); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 同步到GridFS
+	if err := services.UploadSpiderToGridFsFromMaster(spider); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
+func RenameSpiderFile(c *gin.Context) {
+	spiderId := c.Param("id")
+	var reqBody SpiderFileReqBody
+	if err := c.ShouldBindJSON(&reqBody); err != nil {
+		HandleError(http.StatusBadRequest, c, err)
+	}
+	spider, err := model.GetSpider(bson.ObjectIdHex(spiderId))
+	if err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 原文件路径
+	filePath := path.Join(spider.Src, reqBody.Path)
+	newFilePath := path.Join(spider.Src, reqBody.NewPath)
+
+	// 如果新文件已存在，则报错
+	if utils.Exists(newFilePath) {
+		HandleErrorF(http.StatusInternalServerError, c, fmt.Sprintf(`%s already exists`, newFilePath))
+		return
+	}
+
+	// 重命名
+	if err := os.Rename(filePath, newFilePath); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	// 删除原文件
+	if err := os.RemoveAll(filePath); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+	}
+
+	// 同步到GridFS
+	if err := services.UploadSpiderToGridFsFromMaster(spider); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
 // 爬虫类型
 func GetSpiderTypes(c *gin.Context) {
 	types, err := model.GetSpiderTypes()
--- a/backend/routes/system.go
+++ b/backend/routes/system.go
@@ -0,0 +1,110 @@
+package routes
+
+import (
+	"crawlab/constants"
+	"crawlab/entity"
+	"crawlab/services"
+	"fmt"
+	"github.com/gin-gonic/gin"
+	"net/http"
+	"strings"
+)
+
+func GetLangList(c *gin.Context) {
+	nodeId := c.Param("id")
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    services.GetLangList(nodeId),
+	})
+}
+
+func GetDepList(c *gin.Context) {
+	nodeId := c.Param("id")
+	lang := c.Query("lang")
+	depName := c.Query("dep_name")
+
+	var depList []entity.Dependency
+	if lang == constants.Python {
+		list, err := services.GetPythonDepList(nodeId, depName)
+		if err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+		depList = list
+	} else {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("%s is not implemented", lang))
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    depList,
+	})
+}
+
+func GetInstalledDepList(c *gin.Context) {
+	nodeId := c.Param("id")
+	lang := c.Query("lang")
+	var depList []entity.Dependency
+	if lang == constants.Python {
+		list, err := services.GetPythonInstalledDepList(nodeId)
+		if err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+		depList = list
+	} else {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("%s is not implemented", lang))
+		return
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    depList,
+	})
+}
+
+func GetAllDepList(c *gin.Context) {
+	lang := c.Query("lang")
+	depName := c.Query("dep_name")
+
+	// 获取所有依赖列表
+	var list []string
+	if lang == constants.Python {
+		_list, err := services.GetPythonDepListFromRedis()
+		if err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+		list = _list
+	} else {
+		HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("%s is not implemented", lang))
+		return
+	}
+
+	// 过滤依赖列表
+	var depList []string
+	for _, name := range list {
+		if strings.HasPrefix(strings.ToLower(name), strings.ToLower(depName)) {
+			depList = append(depList, name)
+		}
+	}
+
+	// 只取前20
+	var returnList []string
+	for i, name := range depList {
+		if i >= 10 {
+			break
+		}
+		returnList = append(returnList, name)
+	}
+
+	c.JSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    returnList,
+	})
+}
--- a/backend/routes/task.go
+++ b/backend/routes/task.go
@@ -9,7 +9,6 @@ import (
 	"encoding/csv"
 	"github.com/gin-gonic/gin"
 	"github.com/globalsign/mgo/bson"
-	uuid "github.com/satori/go.uuid"
 	"net/http"
 )

@@ -18,6 +17,7 @@ type TaskListRequestData struct {
 	PageSize int    `form:"page_size"`
 	NodeId   string `form:"node_id"`
 	SpiderId string `form:"spider_id"`
+	Status   string `form:"status"`
 }

 type TaskResultsRequestData struct {
@@ -29,14 +29,14 @@ func GetTaskList(c *gin.Context) {
 	// 绑定数据
 	data := TaskListRequestData{}
 	if err := c.ShouldBindQuery(&data); err != nil {
-		HandleError(http.StatusBadRequest, c, err)
+		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
 	if data.PageNum == 0 {
 		data.PageNum = 1
 	}
 	if data.PageSize == 0 {
-		data.PageNum = 10
+		data.PageSize = 10
 	}

 	// 过滤条件
@@ -47,6 +47,10 @@ func GetTaskList(c *gin.Context) {
 	if data.SpiderId != "" {
 		query["spider_id"] = bson.ObjectIdHex(data.SpiderId)
 	}
+	//新增根据任务状态获取task列表
+	if data.Status != "" {
+		query["status"] = data.Status
+	}

 	// 获取任务列表
 	tasks, err := model.GetTaskList(query, (data.PageNum-1)*data.PageSize, data.PageSize, "-create_ts")
@@ -78,49 +82,114 @@ func GetTask(c *gin.Context) {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-		Data:    result,
-	})
+	HandleSuccessData(c, result)
 }

 func PutTask(c *gin.Context) {
-	// 生成任务ID
-	id := uuid.NewV4()
+	type TaskRequestBody struct {
+		SpiderId bson.ObjectId   `json:"spider_id"`
+		RunType  string          `json:"run_type"`
+		NodeIds  []bson.ObjectId `json:"node_ids"`
+		Param    string          `json:"param"`
+	}

 	// 绑定数据
-	var t model.Task
-	if err := c.ShouldBindJSON(&t); err != nil {
-		HandleError(http.StatusBadRequest, c, err)
-		return
-	}
-	t.Id = id.String()
-	t.Status = constants.StatusPending
-
-	// 如果没有传入node_id，则置为null
-	if t.NodeId.Hex() == "" {
-		t.NodeId = bson.ObjectIdHex(constants.ObjectIdNull)
-	}
-
-	// 将任务存入数据库
-	if err := model.AddTask(t); err != nil {
+	var reqBody TaskRequestBody
+	if err := c.ShouldBindJSON(&reqBody); err != nil {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}

-	// 加入任务队列
-	if err := services.AssignTask(t); err != nil {
-		HandleError(http.StatusInternalServerError, c, err)
+	if reqBody.RunType == constants.RunTypeAllNodes {
+		// 所有节点
+		nodes, err := model.GetNodeList(nil)
+		if err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+		for _, node := range nodes {
+			t := model.Task{
+				SpiderId: reqBody.SpiderId,
+				NodeId:   node.Id,
+				Param:    reqBody.Param,
+			}
+
+			if err := services.AddTask(t); err != nil {
+				HandleError(http.StatusInternalServerError, c, err)
+				return
+			}
+		}
+	} else if reqBody.RunType == constants.RunTypeRandom {
+		// 随机
+		t := model.Task{
+			SpiderId: reqBody.SpiderId,
+			Param:    reqBody.Param,
+		}
+		if err := services.AddTask(t); err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+	} else if reqBody.RunType == constants.RunTypeSelectedNodes {
+		// 指定节点
+		for _, nodeId := range reqBody.NodeIds {
+			t := model.Task{
+				SpiderId: reqBody.SpiderId,
+				NodeId:   nodeId,
+				Param:    reqBody.Param,
+			}
+
+			if err := services.AddTask(t); err != nil {
+				HandleError(http.StatusInternalServerError, c, err)
+				return
+			}
+		}
+	} else {
+		HandleErrorF(http.StatusInternalServerError, c, "invalid run_type")
 		return
 	}
-
-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-	})
+	HandleSuccess(c)
 }

+func DeleteTaskByStatus(c *gin.Context) {
+	status := c.Query("status")
+
+	//删除相应的日志文件
+	if err := services.RemoveLogByTaskStatus(status); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	//删除该状态下的task
+	if err := model.RemoveTaskByStatus(status); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+
+	HandleSuccess(c)
+}
+
+// 删除多个任务
+func DeleteMultipleTask(c *gin.Context) {
+	ids := make(map[string][]string)
+	if err := c.ShouldBindJSON(&ids); err != nil {
+		HandleError(http.StatusInternalServerError, c, err)
+		return
+	}
+	list := ids["ids"]
+	for _, id := range list {
+		if err := services.RemoveLogByTaskId(id); err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+		if err := model.RemoveTask(id); err != nil {
+			HandleError(http.StatusInternalServerError, c, err)
+			return
+		}
+	}
+	HandleSuccess(c)
+}
+
+// 删除单个任务
 func DeleteTask(c *gin.Context) {
 	id := c.Param("id")

@@ -129,33 +198,22 @@ func DeleteTask(c *gin.Context) {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
-
 	// 删除task
 	if err := model.RemoveTask(id); err != nil {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
-
-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-	})
+	HandleSuccess(c)
 }

 func GetTaskLog(c *gin.Context) {
 	id := c.Param("id")
-
 	logStr, err := services.GetTaskLog(id)
 	if err != nil {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
-
-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-		Data:    logStr,
-	})
+	HandleSuccessData(c, logStr)
 }

 func GetTaskResults(c *gin.Context) {
@@ -164,7 +222,7 @@ func GetTaskResults(c *gin.Context) {
 	// 绑定数据
 	data := TaskResultsRequestData{}
 	if err := c.ShouldBindQuery(&data); err != nil {
-		HandleError(http.StatusBadRequest, c, err)
+		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}

@@ -266,9 +324,5 @@ func CancelTask(c *gin.Context) {
 		HandleError(http.StatusInternalServerError, c, err)
 		return
 	}
-
-	c.JSON(http.StatusOK, Response{
-		Status:  "ok",
-		Message: "success",
-	})
+	HandleSuccess(c)
 }
--- a/backend/routes/user.go
+++ b/backend/routes/user.go
@@ -21,6 +21,7 @@ type UserListRequestData struct {
 type UserRequestData struct {
 	Username string `json:"username"`
 	Password string `json:"password"`
+	Role     string `json:"role"`
 }

 func GetUser(c *gin.Context) {
@@ -88,11 +89,16 @@ func PutUser(c *gin.Context) {
 		return
 	}

+	// 默认为正常用户
+	if reqData.Role == "" {
+		reqData.Role = constants.RoleNormal
+	}
+
 	// 添加用户
 	user := model.User{
 		Username: strings.ToLower(reqData.Username),
 		Password: utils.EncryptPassword(reqData.Password),
-		Role:     constants.RoleNormal,
+		Role:     reqData.Role,
 	}
 	if err := user.Add(); err != nil {
 		HandleError(http.StatusInternalServerError, c, err)
--- a/backend/routes/utils.go
+++ b/backend/routes/utils.go
@@ -1,17 +1,15 @@
 package routes

 import (
-	"github.com/apex/log"
 	"github.com/gin-gonic/gin"
+	"net/http"
 	"runtime/debug"
 )

 func HandleError(statusCode int, c *gin.Context, err error) {
-	log.Errorf("handle error:" + err.Error())
-	debug.PrintStack()
 	c.AbortWithStatusJSON(statusCode, Response{
-		Status:  "ok",
-		Message: "error",
+		Status:  "error",
+		Message: "failure",
 		Error:   err.Error(),
 	})
 }
@@ -24,3 +22,18 @@ func HandleErrorF(statusCode int, c *gin.Context, err string) {
 		Error:   err,
 	})
 }
+
+func HandleSuccess(c *gin.Context) {
+	c.AbortWithStatusJSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+	})
+}
+
+func HandleSuccessData(c *gin.Context, data interface{}) {
+	c.AbortWithStatusJSON(http.StatusOK, Response{
+		Status:  "ok",
+		Message: "success",
+		Data:    data,
+	})
+}
--- a/backend/services/config_spider.go
+++ b/backend/services/config_spider.go
@@ -0,0 +1,261 @@
+package services
+
+import (
+	"crawlab/constants"
+	"crawlab/database"
+	"crawlab/entity"
+	"crawlab/model"
+	"crawlab/model/config_spider"
+	"crawlab/utils"
+	"errors"
+	"fmt"
+	"github.com/apex/log"
+	"github.com/globalsign/mgo/bson"
+	uuid "github.com/satori/go.uuid"
+	"github.com/spf13/viper"
+	"gopkg.in/yaml.v2"
+	"os"
+	"path/filepath"
+	"strings"
+)
+
+func GenerateConfigSpiderFiles(spider model.Spider, configData entity.ConfigSpiderData) error {
+	// 校验Spiderfile正确性
+	if err := ValidateSpiderfile(configData); err != nil {
+		return err
+	}
+
+	// 构造代码生成器
+	generator := config_spider.ScrapyGenerator{
+		Spider:     spider,
+		ConfigData: configData,
+	}
+
+	// 生成代码
+	if err := generator.Generate(); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// 验证Spiderfile
+func ValidateSpiderfile(configData entity.ConfigSpiderData) error {
+	// 获取所有字段
+	fields := config_spider.GetAllFields(configData)
+
+	// 校验是否存在 start_url
+	if configData.StartUrl == "" {
+		return errors.New("spiderfile invalid: start_url is empty")
+	}
+
+	// 校验是否存在 start_stage
+	if configData.StartStage == "" {
+		return errors.New("spiderfile invalid: start_stage is empty")
+	}
+
+	// 校验是否存在 stages
+	if len(configData.Stages) == 0 {
+		return errors.New("spiderfile invalid: stages is empty")
+	}
+
+	// 校验stages
+	dict := map[string]int{}
+	for _, stage := range configData.Stages {
+		stageName := stage.Name
+
+		// stage 名称不能为空
+		if stageName == "" {
+			return errors.New("spiderfile invalid: stage name is empty")
+		}
+
+		// stage 名称不能为保留字符串
+		// NOTE: 如果有其他Engine，可以扩展，默认为Scrapy
+		if configData.Engine == "" || configData.Engine == constants.EngineScrapy {
+			if strings.Contains(constants.ScrapyProtectedStageNames, stageName) {
+				return errors.New(fmt.Sprintf("spiderfile invalid: stage name '%s' is protected", stageName))
+			}
+		} else {
+			return errors.New(fmt.Sprintf("spiderfile invalid: engine '%s' is not implemented", configData.Engine))
+		}
+
+		// stage 名称不能重复
+		if dict[stageName] == 1 {
+			return errors.New(fmt.Sprintf("spiderfile invalid: stage name '%s' is duplicated", stageName))
+		}
+		dict[stageName] = 1
+
+		// stage 字段不能为空
+		if len(stage.Fields) == 0 {
+			return errors.New(fmt.Sprintf("spiderfile invalid: stage '%s' has no fields", stageName))
+		}
+
+		// 是否包含 next_stage
+		hasNextStage := false
+
+		// 遍历字段列表
+		for _, field := range stage.Fields {
+			// stage 的 next stage 只能有一个
+			if field.NextStage != "" {
+				if hasNextStage {
+					return errors.New(fmt.Sprintf("spiderfile invalid: stage '%s' has more than 1 next_stage", stageName))
+				}
+				hasNextStage = true
+			}
+
+			// 字段里 css 和 xpath 只能包含一个
+			if field.Css != "" && field.Xpath != "" {
+				return errors.New(fmt.Sprintf("spiderfile invalid: field '%s' in stage '%s' has both css and xpath set which is prohibited", field.Name, stageName))
+			}
+		}
+
+		// stage 里 page_css 和 page_xpath 只能包含一个
+		if stage.PageCss != "" && stage.PageXpath != "" {
+			return errors.New(fmt.Sprintf("spiderfile invalid: stage '%s' has both page_css and page_xpath set which is prohibited", stageName))
+		}
+
+		// stage 里 list_css 和 list_xpath 只能包含一个
+		if stage.ListCss != "" && stage.ListXpath != "" {
+			return errors.New(fmt.Sprintf("spiderfile invalid: stage '%s' has both list_css and list_xpath set which is prohibited", stageName))
+		}
+
+		// 如果 stage 的 is_list 为 true 但 list_css 为空，报错
+		if stage.IsList && (stage.ListCss == "" && stage.ListXpath == "") {
+			return errors.New("spiderfile invalid: stage with is_list = true should have either list_css or list_xpath being set")
+		}
+	}
+
+	// 校验字段唯一性
+	if !IsUniqueConfigSpiderFields(fields) {
+		return errors.New("spiderfile invalid: fields not unique")
+	}
+
+	// 字段名称不能为保留字符串
+	for _, field := range fields {
+		if strings.Contains(constants.ScrapyProtectedFieldNames, field.Name) {
+			return errors.New(fmt.Sprintf("spiderfile invalid: field name '%s' is protected", field.Name))
+		}
+	}
+
+	return nil
+}
+
+func IsUniqueConfigSpiderFields(fields []entity.Field) bool {
+	dict := map[string]int{}
+	for _, field := range fields {
+		if dict[field.Name] == 1 {
+			return false
+		}
+		dict[field.Name] = 1
+	}
+	return true
+}
+
+func ProcessSpiderFilesFromConfigData(spider model.Spider, configData entity.ConfigSpiderData) error {
+	spiderDir := spider.Src
+
+	// 删除已有的爬虫文件
+	for _, fInfo := range utils.ListDir(spiderDir) {
+		// 不删除Spiderfile
+		if fInfo.Name() == "Spiderfile" {
+			continue
+		}
+
+		// 删除其他文件
+		if err := os.RemoveAll(filepath.Join(spiderDir, fInfo.Name())); err != nil {
+			return err
+		}
+	}
+
+	// 拷贝爬虫文件
+	tplDir := "./template/scrapy"
+	for _, fInfo := range utils.ListDir(tplDir) {
+		// 跳过Spiderfile
+		if fInfo.Name() == "Spiderfile" {
+			continue
+		}
+
+		srcPath := filepath.Join(tplDir, fInfo.Name())
+		if fInfo.IsDir() {
+			dirPath := filepath.Join(spiderDir, fInfo.Name())
+			if err := utils.CopyDir(srcPath, dirPath); err != nil {
+				return err
+			}
+		} else {
+			if err := utils.CopyFile(srcPath, filepath.Join(spiderDir, fInfo.Name())); err != nil {
+				return err
+			}
+		}
+	}
+
+	// 更改爬虫文件
+	if err := GenerateConfigSpiderFiles(spider, configData); err != nil {
+		return err
+	}
+
+	// 打包为 zip 文件
+	files, err := utils.GetFilesFromDir(spiderDir)
+	if err != nil {
+		return err
+	}
+	randomId := uuid.NewV4()
+	tmpFilePath := filepath.Join(viper.GetString("other.tmppath"), spider.Name+"."+randomId.String()+".zip")
+	spiderZipFileName := spider.Name + ".zip"
+	if err := utils.Compress(files, tmpFilePath); err != nil {
+		return err
+	}
+
+	// 获取 GridFS 实例
+	s, gf := database.GetGridFs("files")
+	defer s.Close()
+
+	// 判断文件是否已经存在
+	var gfFile model.GridFs
+	if err := gf.Find(bson.M{"filename": spiderZipFileName}).One(&gfFile); err == nil {
+		// 已经存在文件，则删除
+		_ = gf.RemoveId(gfFile.Id)
+	}
+
+	// 上传到GridFs
+	fid, err := UploadToGridFs(spiderZipFileName, tmpFilePath)
+	if err != nil {
+		log.Errorf("upload to grid fs error: %s", err.Error())
+		return err
+	}
+
+	// 保存爬虫 FileId
+	spider.FileId = fid
+	_ = spider.Save()
+
+	return nil
+}
+
+func GenerateSpiderfileFromConfigData(spider model.Spider, configData entity.ConfigSpiderData) error {
+	// Spiderfile 路径
+	sfPath := filepath.Join(spider.Src, "Spiderfile")
+
+	// 生成Yaml内容
+	sfContentByte, err := yaml.Marshal(configData)
+	if err != nil {
+		return err
+	}
+
+	// 打开文件
+	var f *os.File
+	if utils.Exists(sfPath) {
+		f, err = os.OpenFile(sfPath, os.O_WRONLY|os.O_TRUNC, 0777)
+	} else {
+		f, err = os.OpenFile(sfPath, os.O_CREATE, 0777)
+	}
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+
+	// 写入内容
+	if _, err := f.Write(sfContentByte); err != nil {
+		return err
+	}
+
+	return nil
+}
--- a/backend/services/log.go
+++ b/backend/services/log.go
@@ -49,10 +49,8 @@ func GetRemoteLog(task model.Task) (logStr string, err error) {
 	select {
 	case logStr = <-ch:
 		log.Infof("get remote log")
-		break
 	case <-time.After(30 * time.Second):
 		logStr = "get remote log timeout"
-		break
 	}

 	return logStr, nil
@@ -119,6 +117,18 @@ func RemoveLogByTaskId(id string) error {
 	return nil
 }

+func RemoveLogByTaskStatus(status string) error {
+	tasks, err := model.GetTaskList(bson.M{"status": status}, 0, constants.Infinite, "-create_ts")
+	if err != nil {
+		log.Error("get tasks error:" + err.Error())
+		return err
+	}
+	for _, task := range tasks {
+		RemoveLogByTaskId(task.Id)
+	}
+	return nil
+}
+
 func removeLog(t model.Task) {
 	if err := RemoveLocalLog(t.LogPath); err != nil {
 		log.Errorf("remove local log error: %s", err.Error())
--- a/backend/services/node.go
+++ b/backend/services/node.go
@@ -50,36 +50,44 @@ func GetNodeData() (Data, error) {
 	return data, err
 }

+func GetRedisNode(key string) (*Data, error) {
+	// 获取节点数据
+	value, err := database.RedisClient.HGet("nodes", key)
+	if err != nil {
+		log.Errorf(err.Error())
+		return nil, err
+	}
+
+	// 解析节点列表数据
+	var data Data
+	if err := json.Unmarshal([]byte(value), &data); err != nil {
+		log.Errorf(err.Error())
+		return nil, err
+	}
+	return &data, nil
+}
+
 // 更新所有节点状态
 func UpdateNodeStatus() {
 	// 从Redis获取节点keys
 	list, err := database.RedisClient.HKeys("nodes")
 	if err != nil {
-		log.Errorf(err.Error())
+		log.Errorf("get redis node keys error: %s", err.Error())
 		return
 	}

 	// 遍历节点keys
 	for _, key := range list {
-		// 获取节点数据
-		value, err := database.RedisClient.HGet("nodes", key)
+
+		data, err := GetRedisNode(key)
 		if err != nil {
-			log.Errorf(err.Error())
-			return
+			continue
 		}
-
-		// 解析节点列表数据
-		var data Data
-		if err := json.Unmarshal([]byte(value), &data); err != nil {
-			log.Errorf(err.Error())
-			return
-		}
-
 		// 如果记录的更新时间超过60秒，该节点被认为离线
 		if time.Now().Unix()-data.UpdateTsUnix > 60 {
 			// 在Redis中删除该节点
 			if err := database.RedisClient.HDel("nodes", data.Key); err != nil {
-				log.Errorf(err.Error())
+				log.Errorf("delete redis node key error:%s, key:%s", err.Error(), data.Key)
 			}
 			continue
 		}
@@ -94,20 +102,19 @@ func UpdateNodeStatus() {
 	model.ResetNodeStatusToOffline(list)
 }

-func handleNodeInfo(key string, data Data) {
+// 处理接到信息
+func handleNodeInfo(key string, data *Data) {
+	// 添加同步锁
+	v, err := database.RedisClient.Lock(key)
+	if err != nil {
+		return
+	}
+	defer database.RedisClient.UnLock(key, v)
+
 	// 更新节点信息到数据库
 	s, c := database.GetCol("nodes")
 	defer s.Close()

-	// 同个key可能因为并发，被注册多次
-	var nodes []model.Node
-	_ = c.Find(bson.M{"key": key}).All(&nodes)
-	if len(nodes) > 1 {
-		for _, node := range nodes {
-			_ = c.RemoveId(node.Id)
-		}
-	}
-
 	var node model.Node
 	if err := c.Find(bson.M{"key": key}).One(&node); err != nil {
 		// 数据库不存在该节点
@@ -160,27 +167,34 @@ func UpdateNodeData() {
 		debug.PrintStack()
 		return
 	}
-	// 构造节点数据
-	data := Data{
-		Key:          key,
-		Mac:          mac,
-		Ip:           ip,
-		Master:       model.IsMaster(),
-		UpdateTs:     time.Now(),
-		UpdateTsUnix: time.Now().Unix(),
+
+	//先获取所有Redis的nodekey
+	list, _ := database.RedisClient.HKeys("nodes")
+
+	if i := utils.Contains(list, key); i == false {
+		// 构造节点数据
+		data := Data{
+			Key:          key,
+			Mac:          mac,
+			Ip:           ip,
+			Master:       model.IsMaster(),
+			UpdateTs:     time.Now(),
+			UpdateTsUnix: time.Now().Unix(),
+		}
+
+		// 注册节点到Redis
+		dataBytes, err := json.Marshal(&data)
+		if err != nil {
+			log.Errorf(err.Error())
+			debug.PrintStack()
+			return
+		}
+		if err := database.RedisClient.HSet("nodes", key, utils.BytesToString(dataBytes)); err != nil {
+			log.Errorf(err.Error())
+			return
+		}
 	}

-	// 注册节点到Redis
-	dataBytes, err := json.Marshal(&data)
-	if err != nil {
-		log.Errorf(err.Error())
-		debug.PrintStack()
-		return
-	}
-	if err := database.RedisClient.HSet("nodes", key, utils.BytesToString(dataBytes)); err != nil {
-		log.Errorf(err.Error())
-		return
-	}
 }

 func MasterNodeCallback(message redis.Message) (err error) {
@@ -258,7 +272,7 @@ func InitNodeService() error {
 		return err
 	}

-	// 如果为主节点，每30秒刷新所有节点信息
+	// 如果为主节点，每10秒刷新所有节点信息
 	if model.IsMaster() {
 		spec := "*/10 * * * * *"
 		if _, err := c.AddFunc(spec, UpdateNodeStatus); err != nil {
--- a/backend/services/register/register.go
+++ b/backend/services/register/register.go
@@ -6,6 +6,7 @@ import (
 	"net"
 	"reflect"
 	"runtime/debug"
+	"sync"
 )

 type Register interface {
@@ -97,25 +98,31 @@ func getMac() (string, error) {
 var register Register

 // 获得注册器
-func GetRegister() Register {
-	if register != nil {
-		return register
-	}
+var once sync.Once

-	registerType := viper.GetString("server.register.type")
-	if registerType == "mac" {
-		register = &MacRegister{}
-	} else {
-		ip := viper.GetString("server.register.ip")
-		if ip == "" {
-			log.Error("server.register.ip is empty")
-			debug.PrintStack()
-			return nil
+func GetRegister() Register {
+	once.Do(func() {
+
+		if register != nil {
+			register = register
 		}
-		register = &IpRegister{
-			Ip: ip,
+
+		registerType := viper.GetString("server.register.type")
+		if registerType == "mac" {
+			register = &MacRegister{}
+		} else {
+			ip := viper.GetString("server.register.ip")
+			if ip == "" {
+				log.Error("server.register.ip is empty")
+				debug.PrintStack()
+				register = nil
+			}
+			register = &IpRegister{
+				Ip: ip,
+			}
 		}
-	}
-	log.Info("register type is :" + reflect.TypeOf(register).String())
+		log.Info("register type is :" + reflect.TypeOf(register).String())
+
+	})
 	return register
 }
--- a/backend/services/schedule.go
+++ b/backend/services/schedule.go
@@ -4,8 +4,10 @@ import (
 	"crawlab/constants"
 	"crawlab/lib/cron"
 	"crawlab/model"
+	"errors"
 	"github.com/apex/log"
-	"github.com/satori/go.uuid"
+	"github.com/globalsign/mgo/bson"
+	uuid "github.com/satori/go.uuid"
 	"runtime/debug"
 )

@@ -15,50 +17,89 @@ type Scheduler struct {
 	cron *cron.Cron
 }

-func AddTask(s model.Schedule) func() {
+func AddScheduleTask(s model.Schedule) func() {
 	return func() {
-		node, err := model.GetNodeByKey(s.NodeKey)
-		if err != nil || node.Id.Hex() == "" {
-			log.Errorf("get node by key error: %s", err.Error())
-			debug.PrintStack()
-			return
-		}
-
-		spider := model.GetSpiderByName(s.SpiderName)
-		if spider == nil || spider.Id.Hex() == "" {
-			log.Errorf("get spider by name error: %s", err.Error())
-			debug.PrintStack()
-			return
-		}
-
-		// 同步ID到定时任务
-		s.SyncNodeIdAndSpiderId(node, *spider)
-
 		// 生成任务ID
 		id := uuid.NewV4()

-		// 生成任务模型
-		t := model.Task{
-			Id:       id.String(),
-			SpiderId: spider.Id,
-			NodeId:   node.Id,
-			Status:   constants.StatusPending,
-			Param:    s.Param,
-		}
+		if s.RunType == constants.RunTypeAllNodes {
+			// 所有节点
+			nodes, err := model.GetNodeList(nil)
+			if err != nil {
+				return
+			}
+			for _, node := range nodes {
+				t := model.Task{
+					Id:       id.String(),
+					SpiderId: s.SpiderId,
+					NodeId:   node.Id,
+					Param:    s.Param,
+				}

-		// 将任务存入数据库
-		if err := model.AddTask(t); err != nil {
-			log.Errorf(err.Error())
-			debug.PrintStack()
+				if err := AddTask(t); err != nil {
+					return
+				}
+				if err := AssignTask(t); err != nil {
+					log.Errorf(err.Error())
+					debug.PrintStack()
+					return
+				}
+			}
+		} else if s.RunType == constants.RunTypeRandom {
+			// 随机
+			t := model.Task{
+				Id:       id.String(),
+				SpiderId: s.SpiderId,
+				Param:    s.Param,
+			}
+			if err := AddTask(t); err != nil {
+				return
+			}
+			if err := AssignTask(t); err != nil {
+				log.Errorf(err.Error())
+				debug.PrintStack()
+				return
+			}
+		} else if s.RunType == constants.RunTypeSelectedNodes {
+			// 指定节点
+			for _, nodeId := range s.NodeIds {
+				t := model.Task{
+					Id:       id.String(),
+					SpiderId: s.SpiderId,
+					NodeId:   nodeId,
+					Param:    s.Param,
+				}
+
+				if err := AddTask(t); err != nil {
+					return
+				}
+
+				if err := AssignTask(t); err != nil {
+					log.Errorf(err.Error())
+					debug.PrintStack()
+					return
+				}
+			}
+		} else {
 			return
 		}

-		// 加入任务队列
-		if err := AssignTask(t); err != nil {
-			log.Errorf(err.Error())
-			debug.PrintStack()
-			return
-		}
+		//node, err := model.GetNodeByKey(s.NodeKey)
+		//if err != nil || node.Id.Hex() == "" {
+		//	log.Errorf("get node by key error: %s", err.Error())
+		//	debug.PrintStack()
+		//	return
+		//}
+		//
+		//spider := model.GetSpiderByName(s.SpiderName)
+		//if spider == nil || spider.Id.Hex() == "" {
+		//	log.Errorf("get spider by name error: %s", err.Error())
+		//	debug.PrintStack()
+		//	return
+		//}
+		//
+		//// 同步ID到定时任务
+		//s.SyncNodeIdAndSpiderId(node, *spider)
 	}
 }

@@ -97,7 +138,7 @@ func (s *Scheduler) AddJob(job model.Schedule) error {
 	spec := job.Cron

 	// 添加任务
-	eid, err := s.cron.AddFunc(spec, AddTask(job))
+	eid, err := s.cron.AddFunc(spec, AddScheduleTask(job))
 	if err != nil {
 		log.Errorf("add func task error: %s", err.Error())
 		debug.PrintStack()
@@ -106,6 +147,7 @@ func (s *Scheduler) AddJob(job model.Schedule) error {

 	// 更新EntryID
 	job.EntryId = eid
+	job.Status = constants.ScheduleStatusRunning
 	if err := job.Save(); err != nil {
 		log.Errorf("job save error: %s", err.Error())
 		debug.PrintStack()
@@ -134,6 +176,36 @@ func ParserCron(spec string) error {
 	return nil
 }

+// 停止定时任务
+func (s *Scheduler) Stop(id bson.ObjectId) error {
+	schedule, err := model.GetSchedule(id)
+	if err != nil {
+		return err
+	}
+	if schedule.EntryId == 0 {
+		return errors.New("entry id not found")
+	}
+	s.cron.Remove(schedule.EntryId)
+	// 更新状态
+	schedule.Status = constants.ScheduleStatusStop
+	if err = schedule.Save(); err != nil {
+		return err
+	}
+	return nil
+}
+
+// 运行任务
+func (s *Scheduler) Run(id bson.ObjectId) error {
+	schedule, err := model.GetSchedule(id)
+	if err != nil {
+		return err
+	}
+	if err := s.AddJob(schedule); err != nil {
+		return err
+	}
+	return nil
+}
+
 func (s *Scheduler) Update() error {
 	// 删除所有定时任务
 	s.RemoveAll()
@@ -151,6 +223,10 @@ func (s *Scheduler) Update() error {
 		// 单个任务
 		job := sList[i]

+		if job.Status == constants.ScheduleStatusStop {
+			continue
+		}
+
 		// 添加到定时任务
 		if err := s.AddJob(job); err != nil {
 			log.Errorf("add job error: %s, job: %s, cron: %s", err.Error(), job.Name, job.Cron)
--- a/backend/services/spider.go
+++ b/backend/services/spider.go
@@ -12,6 +12,7 @@ import (
 	"github.com/apex/log"
 	"github.com/globalsign/mgo"
 	"github.com/globalsign/mgo/bson"
+	uuid "github.com/satori/go.uuid"
 	"github.com/spf13/viper"
 	"os"
 	"path/filepath"
@@ -30,6 +31,48 @@ type SpiderUploadMessage struct {
 	SpiderId string
 }

+// 从主节点上传爬虫到GridFS
+func UploadSpiderToGridFsFromMaster(spider model.Spider) error {
+	// 爬虫所在目录
+	spiderDir := spider.Src
+
+	// 打包为 zip 文件
+	files, err := utils.GetFilesFromDir(spiderDir)
+	if err != nil {
+		return err
+	}
+	randomId := uuid.NewV4()
+	tmpFilePath := filepath.Join(viper.GetString("other.tmppath"), spider.Name+"."+randomId.String()+".zip")
+	spiderZipFileName := spider.Name + ".zip"
+	if err := utils.Compress(files, tmpFilePath); err != nil {
+		return err
+	}
+
+	// 获取 GridFS 实例
+	s, gf := database.GetGridFs("files")
+	defer s.Close()
+
+	// 判断文件是否已经存在
+	var gfFile model.GridFs
+	if err := gf.Find(bson.M{"filename": spiderZipFileName}).One(&gfFile); err == nil {
+		// 已经存在文件，则删除
+		_ = gf.RemoveId(gfFile.Id)
+	}
+
+	// 上传到GridFs
+	fid, err := UploadToGridFs(spiderZipFileName, tmpFilePath)
+	if err != nil {
+		log.Errorf("upload to grid fs error: %s", err.Error())
+		return err
+	}
+
+	// 保存爬虫 FileId
+	spider.FileId = fid
+	_ = spider.Save()
+
+	return nil
+}
+
 // 上传zip文件到GridFS
 func UploadToGridFs(fileName string, filePath string) (fid bson.ObjectId, err error) {
 	fid = ""
@@ -116,12 +159,23 @@ func PublishAllSpiders() {

 // 发布爬虫
 func PublishSpider(spider model.Spider) {
-	// 查询gf file，不存在则删除
-	gfFile := model.GetGridFs(spider.FileId)
-	if gfFile == nil {
-		_ = model.RemoveSpider(spider.Id)
+	var gfFile *model.GridFs
+	if spider.FileId.Hex() != constants.ObjectIdNull {
+		// 查询gf file，不存在则标记为爬虫文件不存在
+		gfFile = model.GetGridFs(spider.FileId)
+		if gfFile == nil {
+			spider.FileId = constants.ObjectIdNull
+			_ = spider.Save()
+			return
+		}
+	}
+
+	// 如果FileId为空，表示还没有上传爬虫到GridFS，则跳过
+	if spider.FileId == bson.ObjectIdHex(constants.ObjectIdNull) {
 		return
 	}
+
+	// 获取爬虫同步实例
 	spiderSync := spider_handler.SpiderSync{
 		Spider: spider,
 	}
--- a/backend/services/spider_handler/spider.go
+++ b/backend/services/spider_handler/spider.go
@@ -10,6 +10,7 @@ import (
 	"github.com/spf13/viper"
 	"io"
 	"os"
+	"os/exec"
 	"path/filepath"
 	"runtime/debug"
 )
@@ -99,7 +100,6 @@ func (s *SpiderSync) Download() {
 	// 创建临时文件
 	tmpFilePath := filepath.Join(tmpPath, randomId.String()+".zip")
 	tmpFile := utils.OpenFile(tmpFilePath)
-	defer utils.Close(tmpFile)

 	// 将该文件写入临时文件
 	if _, err := io.Copy(tmpFile, f); err != nil {
@@ -119,6 +119,15 @@ func (s *SpiderSync) Download() {
 		return
 	}

+	//递归修改目标文件夹权限
+	// 解决scrapy.setting中开启LOG_ENABLED 和 LOG_FILE时不能创建log文件的问题
+	cmd := exec.Command("chmod", "-R", "777", dstPath)
+	if err := cmd.Run(); err != nil {
+		log.Errorf(err.Error())
+		debug.PrintStack()
+		return
+	}
+
 	// 关闭临时文件
 	if err := tmpFile.Close(); err != nil {
 		log.Errorf(err.Error())
--- a/backend/services/system.go
+++ b/backend/services/system.go
@@ -4,28 +4,60 @@ import (
 	"crawlab/constants"
 	"crawlab/database"
 	"crawlab/entity"
+	"crawlab/lib/cron"
 	"crawlab/model"
 	"crawlab/utils"
 	"encoding/json"
+	"errors"
+	"fmt"
+	"github.com/apex/log"
+	"github.com/imroc/req"
+	"os/exec"
+	"regexp"
+	"runtime/debug"
+	"sort"
+	"strings"
+	"sync"
 )

+type PythonDepJsonData struct {
+	Info PythonDepJsonDataInfo `json:"info"`
+}
+
+type PythonDepJsonDataInfo struct {
+	Name    string `json:"name"`
+	Summary string `json:"summary"`
+	Version string `json:"version"`
+}
+
+type PythonDepNameDict struct {
+	Name   string `json:"name"`
+	Weight int    `json:"weight"`
+}
+
+type PythonDepNameDictSlice []PythonDepNameDict
+
+func (s PythonDepNameDictSlice) Len() int           { return len(s) }
+func (s PythonDepNameDictSlice) Swap(i, j int)      { s[i], s[j] = s[j], s[i] }
+func (s PythonDepNameDictSlice) Less(i, j int) bool { return s[i].Weight > s[j].Weight }
+
 var SystemInfoChanMap = utils.NewChanMap()

-func GetRemoteSystemInfo(id string) (sysInfo entity.SystemInfo, err error) {
+func GetRemoteSystemInfo(nodeId string) (sysInfo entity.SystemInfo, err error) {
 	// 发送消息
 	msg := entity.NodeMessage{
 		Type:   constants.MsgTypeGetSystemInfo,
-		NodeId: id,
+		NodeId: nodeId,
 	}

 	// 序列化
 	msgBytes, _ := json.Marshal(&msg)
-	if _, err := database.RedisClient.Publish("nodes:"+id, utils.BytesToString(msgBytes)); err != nil {
+	if _, err := database.RedisClient.Publish("nodes:"+nodeId, utils.BytesToString(msgBytes)); err != nil {
 		return entity.SystemInfo{}, err
 	}

 	// 通道
-	ch := SystemInfoChanMap.ChanBlocked(id)
+	ch := SystemInfoChanMap.ChanBlocked(nodeId)

 	// 等待响应，阻塞
 	sysInfoStr := <-ch
@@ -38,11 +70,242 @@ func GetRemoteSystemInfo(id string) (sysInfo entity.SystemInfo, err error) {
 	return sysInfo, nil
 }

-func GetSystemInfo(id string) (sysInfo entity.SystemInfo, err error) {
-	if IsMasterNode(id) {
+func GetSystemInfo(nodeId string) (sysInfo entity.SystemInfo, err error) {
+	if IsMasterNode(nodeId) {
 		sysInfo, err = model.GetLocalSystemInfo()
 	} else {
-		sysInfo, err = GetRemoteSystemInfo(id)
+		sysInfo, err = GetRemoteSystemInfo(nodeId)
 	}
 	return
 }
+
+func GetLangList(nodeId string) []entity.Lang {
+	list := []entity.Lang{
+		{Name: "Python", ExecutableName: "python", ExecutablePath: "/usr/local/bin/python", DepExecutablePath: "/usr/local/bin/pip"},
+		{Name: "NodeJS", ExecutableName: "node", ExecutablePath: "/usr/local/bin/node"},
+		{Name: "Java", ExecutableName: "java", ExecutablePath: "/usr/local/bin/java"},
+	}
+	for i, lang := range list {
+		list[i].Installed = IsInstalledLang(nodeId, lang)
+	}
+	return list
+}
+
+func GetLangFromLangName(nodeId string, name string) entity.Lang {
+	langList := GetLangList(nodeId)
+	for _, lang := range langList {
+		if lang.ExecutableName == name {
+			return lang
+		}
+	}
+	return entity.Lang{}
+}
+
+func GetPythonDepList(nodeId string, searchDepName string) ([]entity.Dependency, error) {
+	var list []entity.Dependency
+
+	// 先从 Redis 获取
+	depList, err := GetPythonDepListFromRedis()
+	if err != nil {
+		return list, err
+	}
+
+	// 过滤相似的依赖
+	var depNameList PythonDepNameDictSlice
+	for _, depName := range depList {
+		if strings.HasPrefix(strings.ToLower(depName), strings.ToLower(searchDepName)) {
+			var weight int
+			if strings.ToLower(depName) == strings.ToLower(searchDepName) {
+				weight = 3
+			} else if strings.HasPrefix(strings.ToLower(depName), strings.ToLower(searchDepName)) {
+				weight = 2
+			} else {
+				weight = 1
+			}
+			depNameList = append(depNameList, PythonDepNameDict{
+				Name:   depName,
+				Weight: weight,
+			})
+		}
+	}
+
+	// 获取已安装依赖
+	installedDepList, err := GetPythonInstalledDepList(nodeId)
+	if err != nil {
+		return list, err
+	}
+
+	// 从依赖源获取数据
+	var goSync sync.WaitGroup
+	sort.Stable(depNameList)
+	for i, depNameDict := range depNameList {
+		if i > 10 {
+			break
+		}
+		goSync.Add(1)
+		go func(depName string, n *sync.WaitGroup) {
+			url := fmt.Sprintf("https://pypi.org/pypi/%s/json", depName)
+			res, err := req.Get(url)
+			if err != nil {
+				n.Done()
+				return
+			}
+			var data PythonDepJsonData
+			if err := res.ToJSON(&data); err != nil {
+				n.Done()
+				return
+			}
+			dep := entity.Dependency{
+				Name:        depName,
+				Version:     data.Info.Version,
+				Description: data.Info.Summary,
+			}
+			dep.Installed = IsInstalledDep(installedDepList, dep)
+			list = append(list, dep)
+			n.Done()
+		}(depNameDict.Name, &goSync)
+	}
+	goSync.Wait()
+
+	return list, nil
+}
+
+func GetPythonDepListFromRedis() ([]string, error) {
+	var list []string
+
+	// 从 Redis 获取字符串
+	rawData, err := database.RedisClient.HGet("system", "deps:python")
+	if err != nil {
+		return list, err
+	}
+
+	// 反序列化
+	if err := json.Unmarshal([]byte(rawData), &list); err != nil {
+		return list, err
+	}
+
+	// 如果为空，则从依赖源获取列表
+	if len(list) == 0 {
+		UpdatePythonDepList()
+	}
+
+	return list, nil
+}
+
+func IsInstalledLang(nodeId string, lang entity.Lang) bool {
+	sysInfo, err := GetSystemInfo(nodeId)
+	if err != nil {
+		return false
+	}
+	for _, exec := range sysInfo.Executables {
+		if exec.Path == lang.ExecutablePath {
+			return true
+		}
+	}
+	return false
+}
+
+func IsInstalledDep(installedDepList []entity.Dependency, dep entity.Dependency) bool {
+	for _, _dep := range installedDepList {
+		if strings.ToLower(_dep.Name) == strings.ToLower(dep.Name) {
+			return true
+		}
+	}
+	return false
+}
+
+func FetchPythonDepList() ([]string, error) {
+	// 依赖URL
+	url := "https://pypi.tuna.tsinghua.edu.cn/simple"
+
+	// 输出列表
+	var list []string
+
+	// 请求URL
+	res, err := req.Get(url)
+	if err != nil {
+		log.Error(err.Error())
+		debug.PrintStack()
+		return list, err
+	}
+
+	// 获取响应数据
+	text, err := res.ToString()
+	if err != nil {
+		log.Error(err.Error())
+		debug.PrintStack()
+		return list, err
+	}
+
+	// 从响应数据中提取依赖名
+	regex := regexp.MustCompile("<a href=\".*/\">(.*)</a>")
+	for _, line := range strings.Split(text, "\n") {
+		arr := regex.FindStringSubmatch(line)
+		if len(arr) < 2 {
+			continue
+		}
+		list = append(list, arr[1])
+	}
+
+	// 赋值给列表
+	return list, nil
+}
+
+func UpdatePythonDepList() {
+	// 从依赖源获取列表
+	list, _ := FetchPythonDepList()
+
+	// 序列化
+	listBytes, err := json.Marshal(list)
+	if err != nil {
+		log.Error(err.Error())
+		debug.PrintStack()
+		return
+	}
+
+	// 设置Redis
+	if err := database.RedisClient.HSet("system", "deps:python", string(listBytes)); err != nil {
+		log.Error(err.Error())
+		debug.PrintStack()
+		return
+	}
+}
+
+func GetPythonInstalledDepList(nodeId string) ([]entity.Dependency, error){
+	var list []entity.Dependency
+
+	lang := GetLangFromLangName(nodeId, constants.Python)
+	if !IsInstalledLang(nodeId, lang) {
+		return list, errors.New("python is not installed")
+	}
+	cmd := exec.Command("pip", "freeze")
+	outputBytes, err := cmd.Output()
+	if err != nil {
+		debug.PrintStack()
+		return list, err
+	}
+
+	for _, line := range strings.Split(string(outputBytes), "\n") {
+		arr := strings.Split(line, "==")
+		if len(arr) < 2 {
+			continue
+		}
+		dep := entity.Dependency{
+			Name:      strings.ToLower(arr[0]),
+			Version:   arr[1],
+			Installed: true,
+		}
+		list = append(list, dep)
+	}
+
+	return list, nil
+}
+
+func InitDepsFetcher() error {
+	c := cron.New(cron.WithSeconds())
+	c.Start()
+	if _, err := c.AddFunc("0 */5 * * * *", UpdatePythonDepList); err != nil {
+		return err
+	}
+	return nil
+}
--- a/backend/services/task.go
+++ b/backend/services/task.go
@@ -10,6 +10,8 @@ import (
 	"encoding/json"
 	"errors"
 	"github.com/apex/log"
+	"github.com/globalsign/mgo/bson"
+	uuid "github.com/satori/go.uuid"
 	"github.com/spf13/viper"
 	"os"
 	"os/exec"
@@ -224,7 +226,22 @@ func ExecuteShellCmd(cmdStr string, cwd string, t model.Task, s model.Spider) (e
 	}

 	// 环境变量配置
-	cmd = SetEnv(cmd, s.Envs, t.Id, s.Col)
+	envs := s.Envs
+	if s.Type == constants.Configurable {
+		// 数据库配置
+		envs = append(envs, model.Env{Name: "CRAWLAB_MONGO_HOST", Value: viper.GetString("mongo.host")})
+		envs = append(envs, model.Env{Name: "CRAWLAB_MONGO_PORT", Value: viper.GetString("mongo.port")})
+		envs = append(envs, model.Env{Name: "CRAWLAB_MONGO_DB", Value: viper.GetString("mongo.db")})
+		envs = append(envs, model.Env{Name: "CRAWLAB_MONGO_USERNAME", Value: viper.GetString("mongo.username")})
+		envs = append(envs, model.Env{Name: "CRAWLAB_MONGO_PASSWORD", Value: viper.GetString("mongo.password")})
+		envs = append(envs, model.Env{Name: "CRAWLAB_MONGO_AUTHSOURCE", Value: viper.GetString("mongo.authSource")})
+
+		// 设置配置
+		for envName, envValue := range s.Config.Settings {
+			envs = append(envs, model.Env{Name: "CRAWLAB_SETTING_" + envName, Value: envValue})
+		}
+	}
+	cmd = SetEnv(cmd, envs, t.Id, s.Col)

 	// 起一个goroutine来监控进程
 	ch := utils.TaskExecChanMap.ChanBlocked(t.Id)
@@ -302,9 +319,12 @@ func SaveTaskResultCount(id string) func() {

 // 执行任务
 func ExecuteTask(id int) {
-	if flag, _ := LockList.Load(id); flag.(bool) {
-		log.Debugf(GetWorkerPrefix(id) + "正在执行任务...")
-		return
+	if flag, ok := LockList.Load(id); ok {
+		if flag.(bool) {
+			log.Debugf(GetWorkerPrefix(id) + "正在执行任务...")
+			return
+		}
+
 	}

 	// 上锁
@@ -378,7 +398,14 @@ func ExecuteTask(id int) {
 	)

 	// 执行命令
-	cmd := spider.Cmd
+	var cmd string
+	if spider.Type == constants.Configurable {
+		// 可配置爬虫命令
+		cmd = "scrapy crawl config_spider"
+	} else {
+		// 自定义爬虫命令
+		cmd = spider.Cmd
+	}

 	// 加入参数
 	if t.Param != "" {
@@ -391,15 +418,23 @@ func ExecuteTask(id int) {
 	t.Status = constants.StatusRunning                   // 任务状态
 	t.WaitDuration = t.StartTs.Sub(t.CreateTs).Seconds() // 等待时长

+	// 判断爬虫文件是否存在
+	gfFile := model.GetGridFs(spider.FileId)
+	if gfFile == nil {
+		t.Error = "找不到爬虫文件，请重新上传"
+		t.Status = constants.StatusError
+		t.FinishTs = time.Now()                                 // 结束时间
+		t.RuntimeDuration = t.FinishTs.Sub(t.StartTs).Seconds() // 运行时长
+		t.TotalDuration = t.FinishTs.Sub(t.CreateTs).Seconds()  // 总时长
+		_ = t.Save()
+		return
+	}
+
 	// 开始执行任务
 	log.Infof(GetWorkerPrefix(id) + "开始执行任务(ID:" + t.Id + ")")

 	// 储存任务
-	if err := t.Save(); err != nil {
-		log.Errorf(err.Error())
-		HandleTaskError(t, err)
-		return
-	}
+	_ = t.Save()

 	// 起一个cron执行器来统计任务结果数
 	if spider.Col != "" {
@@ -461,6 +496,29 @@ func GetTaskLog(id string) (logStr string, err error) {
 	}

 	if IsMasterNode(task.NodeId.Hex()) {
+		if !utils.Exists(task.LogPath) {
+			fileDir, err := MakeLogDir(task)
+
+			if err != nil {
+				log.Errorf(err.Error())
+			}
+
+			fileP := GetLogFilePaths(fileDir)
+
+			// 获取日志文件路径
+			fLog, err := os.Create(fileP)
+			defer fLog.Close()
+			if err != nil {
+				log.Errorf("create task log file error: %s", fileP)
+				debug.PrintStack()
+			}
+			task.LogPath = fileP
+			if err := task.Save(); err != nil {
+				log.Errorf(err.Error())
+				debug.PrintStack()
+			}
+
+		}
 		// 若为主节点，获取本机日志
 		logBytes, err := model.GetLocalLog(task.LogPath)
 		if err != nil {
@@ -542,6 +600,32 @@ func CancelTask(id string) (err error) {
 	return nil
 }

+func AddTask(t model.Task) error {
+	// 生成任务ID
+	id := uuid.NewV4()
+	t.Id = id.String()
+
+	// 设置任务状态
+	t.Status = constants.StatusPending
+
+	// 如果没有传入node_id，则置为null
+	if t.NodeId.Hex() == "" {
+		t.NodeId = bson.ObjectIdHex(constants.ObjectIdNull)
+	}
+
+	// 将任务存入数据库
+	if err := model.AddTask(t); err != nil {
+		return err
+	}
+
+	// 加入任务队列
+	if err := AssignTask(t); err != nil {
+		return err
+	}
+
+	return nil
+}
+
 func HandleTaskError(t model.Task, err error) {
 	log.Error("handle task error:" + err.Error())
 	t.Status = constants.StatusError
--- a/backend/template/scrapy/config_spider/init.py
+++ b/backend/template/scrapy/config_spider/init.py
--- a/backend/template/scrapy/config_spider/items.py
+++ b/backend/template/scrapy/config_spider/items.py
@@ -0,0 +1,12 @@
+# -*- coding: utf-8 -*-
+
+# Define here the models for your scraped items
+#
+# See documentation in:
+# https://docs.scrapy.org/en/latest/topics/items.html
+
+import scrapy
+
+
+class Item(scrapy.Item):
+###ITEMS###
--- a/backend/template/scrapy/config_spider/middlewares.py
+++ b/backend/template/scrapy/config_spider/middlewares.py
@@ -0,0 +1,103 @@
+# -*- coding: utf-8 -*-
+
+# Define here the models for your spider middleware
+#
+# See documentation in:
+# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
+
+from scrapy import signals
+
+
+class ConfigSpiderSpiderMiddleware(object):
+    # Not all methods need to be defined. If a method is not defined,
+    # scrapy acts as if the spider middleware does not modify the
+    # passed objects.
+
+    @classmethod
+    def from_crawler(cls, crawler):
+        # This method is used by Scrapy to create your spiders.
+        s = cls()
+        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
+        return s
+
+    def process_spider_input(self, response, spider):
+        # Called for each response that goes through the spider
+        # middleware and into the spider.
+
+        # Should return None or raise an exception.
+        return None
+
+    def process_spider_output(self, response, result, spider):
+        # Called with the results returned from the Spider, after
+        # it has processed the response.
+
+        # Must return an iterable of Request, dict or Item objects.
+        for i in result:
+            yield i
+
+    def process_spider_exception(self, response, exception, spider):
+        # Called when a spider or process_spider_input() method
+        # (from other spider middleware) raises an exception.
+
+        # Should return either None or an iterable of Request, dict
+        # or Item objects.
+        pass
+
+    def process_start_requests(self, start_requests, spider):
+        # Called with the start requests of the spider, and works
+        # similarly to the process_spider_output() method, except
+        # that it doesn’t have a response associated.
+
+        # Must return only requests (not items).
+        for r in start_requests:
+            yield r
+
+    def spider_opened(self, spider):
+        spider.logger.info('Spider opened: %s' % spider.name)
+
+
+class ConfigSpiderDownloaderMiddleware(object):
+    # Not all methods need to be defined. If a method is not defined,
+    # scrapy acts as if the downloader middleware does not modify the
+    # passed objects.
+
+    @classmethod
+    def from_crawler(cls, crawler):
+        # This method is used by Scrapy to create your spiders.
+        s = cls()
+        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
+        return s
+
+    def process_request(self, request, spider):
+        # Called for each request that goes through the downloader
+        # middleware.
+
+        # Must either:
+        # - return None: continue processing this request
+        # - or return a Response object
+        # - or return a Request object
+        # - or raise IgnoreRequest: process_exception() methods of
+        #   installed downloader middleware will be called
+        return None
+
+    def process_response(self, request, response, spider):
+        # Called with the response returned from the downloader.
+
+        # Must either;
+        # - return a Response object
+        # - return a Request object
+        # - or raise IgnoreRequest
+        return response
+
+    def process_exception(self, request, exception, spider):
+        # Called when a download handler or a process_request()
+        # (from other downloader middleware) raises an exception.
+
+        # Must either:
+        # - return None: continue processing this exception
+        # - return a Response object: stops process_exception() chain
+        # - return a Request object: stops process_exception() chain
+        pass
+
+    def spider_opened(self, spider):
+        spider.logger.info('Spider opened: %s' % spider.name)
--- a/backend/template/scrapy/config_spider/pipelines.py
+++ b/backend/template/scrapy/config_spider/pipelines.py
@@ -0,0 +1,27 @@
+# -*- coding: utf-8 -*-
+
+# Define your item pipelines here
+#
+# Don't forget to add your pipeline to the ITEM_PIPELINES setting
+# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
+
+import os
+from pymongo import MongoClient
+
+mongo = MongoClient(
+    host=os.environ.get('CRAWLAB_MONGO_HOST') or 'localhost',
+    port=int(os.environ.get('CRAWLAB_MONGO_PORT') or 27017),
+    username=os.environ.get('CRAWLAB_MONGO_USERNAME'),
+    password=os.environ.get('CRAWLAB_MONGO_PASSWORD'),
+    authSource=os.environ.get('CRAWLAB_MONGO_AUTHSOURCE') or 'admin'
+)
+db = mongo[os.environ.get('CRAWLAB_MONGO_DB') or 'test']
+col = db[os.environ.get('CRAWLAB_COLLECTION') or 'test']
+task_id = os.environ.get('CRAWLAB_TASK_ID')
+
+class ConfigSpiderPipeline(object):
+    def process_item(self, item, spider):
+        item['task_id'] = task_id
+        if col is not None:
+            col.save(item)
+        return item
--- a/backend/template/scrapy/config_spider/settings.py
+++ b/backend/template/scrapy/config_spider/settings.py
@@ -0,0 +1,111 @@
+# -*- coding: utf-8 -*-
+import os
+import re
+import json
+
+# Scrapy settings for config_spider project
+#
+# For simplicity, this file contains only settings considered important or
+# commonly used. You can find more settings consulting the documentation:
+#
+#     https://docs.scrapy.org/en/latest/topics/settings.html
+#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
+#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html
+
+BOT_NAME = 'Crawlab Configurable Spider'
+
+SPIDER_MODULES = ['config_spider.spiders']
+NEWSPIDER_MODULE = 'config_spider.spiders'
+
+
+# Crawl responsibly by identifying yourself (and your website) on the user-agent
+USER_AGENT = 'Crawlab Spider'
+
+# Obey robots.txt rules
+ROBOTSTXT_OBEY = True
+
+# Configure maximum concurrent requests performed by Scrapy (default: 16)
+#CONCURRENT_REQUESTS = 32
+
+# Configure a delay for requests for the same website (default: 0)
+# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
+# See also autothrottle settings and docs
+#DOWNLOAD_DELAY = 3
+# The download delay setting will honor only one of:
+#CONCURRENT_REQUESTS_PER_DOMAIN = 16
+#CONCURRENT_REQUESTS_PER_IP = 16
+
+# Disable cookies (enabled by default)
+#COOKIES_ENABLED = False
+
+# Disable Telnet Console (enabled by default)
+#TELNETCONSOLE_ENABLED = False
+
+# Override the default request headers:
+#DEFAULT_REQUEST_HEADERS = {
+#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
+#   'Accept-Language': 'en',
+#}
+
+# Enable or disable spider middlewares
+# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
+#SPIDER_MIDDLEWARES = {
+#    'config_spider.middlewares.ConfigSpiderSpiderMiddleware': 543,
+#}
+
+# Enable or disable downloader middlewares
+# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
+#DOWNLOADER_MIDDLEWARES = {
+#    'config_spider.middlewares.ConfigSpiderDownloaderMiddleware': 543,
+#}
+
+# Enable or disable extensions
+# See https://docs.scrapy.org/en/latest/topics/extensions.html
+#EXTENSIONS = {
+#    'scrapy.extensions.telnet.TelnetConsole': None,
+#}
+
+# Configure item pipelines
+# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
+ITEM_PIPELINES = {
+   'config_spider.pipelines.ConfigSpiderPipeline': 300,
+}
+
+# Enable and configure the AutoThrottle extension (disabled by default)
+# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
+#AUTOTHROTTLE_ENABLED = True
+# The initial download delay
+#AUTOTHROTTLE_START_DELAY = 5
+# The maximum download delay to be set in case of high latencies
+#AUTOTHROTTLE_MAX_DELAY = 60
+# The average number of requests Scrapy should be sending in parallel to
+# each remote server
+#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
+# Enable showing throttling stats for every response received:
+#AUTOTHROTTLE_DEBUG = False
+
+# Enable and configure HTTP caching (disabled by default)
+# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
+#HTTPCACHE_ENABLED = True
+#HTTPCACHE_EXPIRATION_SECS = 0
+#HTTPCACHE_DIR = 'httpcache'
+#HTTPCACHE_IGNORE_HTTP_CODES = []
+#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
+
+for setting_env_name in [x for x in os.environ.keys() if x.startswith('CRAWLAB_SETTING_')]:
+    setting_name = setting_env_name.replace('CRAWLAB_SETTING_', '')
+    setting_value = os.environ.get(setting_env_name)
+    if setting_value.lower() == 'true':
+        setting_value = True
+    elif setting_value.lower() == 'false':
+        setting_value = False
+    elif re.search(r'^\d+$', setting_value) is not None:
+        setting_value = int(setting_value)
+    elif re.search(r'^\{.*\}$', setting_value.strip()) is not None:
+        setting_value = json.loads(setting_value)
+    elif re.search(r'^\[.*\]$', setting_value.strip()) is not None:
+        setting_value = json.loads(setting_value)
+    else:
+        pass
+    locals()[setting_name] = setting_value
+
--- a/backend/template/scrapy/config_spider/spiders/init.py
+++ b/backend/template/scrapy/config_spider/spiders/init.py
@@ -0,0 +1,4 @@
+# This package will contain the spiders of your Scrapy project
+#
+# Please refer to the documentation for information on how to create and manage
+# your spiders.
--- a/backend/template/scrapy/config_spider/spiders/spider.py
+++ b/backend/template/scrapy/config_spider/spiders/spider.py
@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+import scrapy
+import re
+from config_spider.items import Item
+from urllib.parse import urljoin
+
+def get_real_url(response, url):
+    if re.search(r'^https?|^\/\/', url):
+        return url
+    return urljoin(response.url, url)
+
+class ConfigSpider(scrapy.Spider):
+    name = 'config_spider'
+
+    def start_requests(self):
+        yield scrapy.Request(url='###START_URL###', callback=self.###START_STAGE###)
+
+###PARSERS###
--- a/backend/template/scrapy/scrapy.cfg
+++ b/backend/template/scrapy/scrapy.cfg
@@ -0,0 +1,11 @@
+# Automatically created by: scrapy startproject
+#
+# For more information about the [deploy] section see:
+# https://scrapyd.readthedocs.io/en/latest/deploy.html
+
+[settings]
+default = config_spider.settings
+
+[deploy]
+#url = http://localhost:6800/
+project = config_spider
--- a/backend/template/spiderfile/Spiderfile.163_news
+++ b/backend/template/spiderfile/Spiderfile.163_news
@@ -0,0 +1,20 @@
+version: "0.4.0"
+name: "toscrapy_books"
+start_url: "http://news.163.com/special/0001386F/rank_news.html"
+start_stage: "list"
+engine: "scrapy"
+stages:
+- name: list
+  is_list: true
+  list_css: "table tr:not(:first-child)"
+  fields:
+  - name: "title"
+    css: "td:nth-child(1) > a"
+  - name: "url"
+    css: "td:nth-child(1) > a"
+    attr: "href"
+  - name: "clicks"
+    css: "td.cBlue"
+settings:
+  ROBOTSTXT_OBEY: false
+  USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
--- a/backend/template/spiderfile/Spiderfile.baidu
+++ b/backend/template/spiderfile/Spiderfile.baidu
@@ -0,0 +1,22 @@
+version: 0.4.0
+name: toscrapy_books
+start_url: http://www.baidu.com/s?wd=crawlab
+start_stage: list
+engine: scrapy
+stages:
+- name: list
+  is_list: true
+  list_xpath: //*[contains(@class, "c-container")]
+  page_xpath: //*[@id="page"]//a[@class="n"][last()]
+  page_attr: href
+  fields:
+  - name: title
+    xpath: .//h3/a
+  - name: url
+    xpath: .//h3/a
+    attr: href
+  - name: abstract
+    xpath: .//*[@class="c-abstract"]
+settings:
+  ROBOTSTXT_OBEY: false
+  USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
--- a/backend/template/spiderfile/Spiderfile.toscrapy_books
+++ b/backend/template/spiderfile/Spiderfile.toscrapy_books
@@ -0,0 +1,28 @@
+version: "0.4.0"
+name: "toscrapy_books"
+start_url: "http://books.toscrape.com"
+start_stage: "list"
+engine: "scrapy"
+stages:
+- name: list
+  is_list: true
+  list_css: "section article.product_pod"
+  page_css: "ul.pager li.next a"
+  page_attr: "href"
+  fields:
+  - name: "title"
+    css: "h3 > a"
+  - name: "url"
+    css: "h3 > a"
+    attr: "href"
+    next_stage: "detail"
+  - name: "price"
+    css: ".product_price > .price_color"
+- name: detail
+  is_list: false
+  fields:
+  - name: "description"
+    css: "#product_description + p"
+settings:
+  ROBOTSTXT_OBEY: true
+  AUTOTHROTTLE_ENABLED: true
--- a/backend/utils/file.go
+++ b/backend/utils/file.go
@@ -3,11 +3,15 @@ package utils
 import (
 	"archive/zip"
 	"bufio"
+	"fmt"
 	"github.com/apex/log"
 	"io"
+	"io/ioutil"
 	"os"
+	"path"
 	"path/filepath"
 	"runtime/debug"
+	"strings"
 )

 // 删除文件
@@ -71,6 +75,16 @@ func IsDir(path string) bool {
 	return s.IsDir()
 }

+func ListDir(path string) []os.FileInfo {
+	list, err := ioutil.ReadDir(path)
+	if err != nil {
+		log.Errorf(err.Error())
+		debug.PrintStack()
+		return nil
+	}
+	return list
+}
+
 // 判断所给路径是否为文件
 func IsFile(path string) bool {
 	return !IsDir(path)
@@ -153,7 +167,6 @@ func DeCompress(srcFile *os.File, dstPath string) error {
 			debug.PrintStack()
 			continue
 		}
-		defer Close(newFile)

 		// 拷贝该文件到新文件中
 		if _, err := io.Copy(newFile, srcFile); err != nil {
@@ -185,8 +198,7 @@ func Compress(files []*os.File, dest string) error {
 	w := zip.NewWriter(d)
 	defer Close(w)
 	for _, file := range files {
-		err := _Compress(file, "", w)
-		if err != nil {
+		if err := _Compress(file, "", w); err != nil {
 			return err
 		}
 	}
@@ -239,3 +251,128 @@ func _Compress(file *os.File, prefix string, zw *zip.Writer) error {
 	}
 	return nil
 }
+
+func GetFilesFromDir(dirPath string) ([]*os.File, error) {
+	var res []*os.File
+	for _, fInfo := range ListDir(dirPath) {
+		f, err := os.Open(filepath.Join(dirPath, fInfo.Name()))
+		if err != nil {
+			return res, err
+		}
+		res = append(res, f)
+	}
+	return res, nil
+}
+
+func GetAllFilesFromDir(dirPath string) ([]*os.File, error) {
+	var res []*os.File
+	if err := filepath.Walk(dirPath, func(path string, info os.FileInfo, err error) error {
+		if !IsDir(path) {
+			f, err2 := os.Open(path)
+			if err2 != nil {
+				return err
+			}
+			res = append(res, f)
+		}
+		return nil
+	}); err != nil {
+		log.Error(err.Error())
+		debug.PrintStack()
+		return res, err
+	}
+	return res, nil
+}
+
+// File copies a single file from src to dst
+func CopyFile(src, dst string) error {
+	var err error
+	var srcFd *os.File
+	var dstFd *os.File
+	var srcInfo os.FileInfo
+
+	if srcFd, err = os.Open(src); err != nil {
+		return err
+	}
+	defer srcFd.Close()
+
+	if dstFd, err = os.Create(dst); err != nil {
+		return err
+	}
+	defer dstFd.Close()
+
+	if _, err = io.Copy(dstFd, srcFd); err != nil {
+		return err
+	}
+	if srcInfo, err = os.Stat(src); err != nil {
+		return err
+	}
+	return os.Chmod(dst, srcInfo.Mode())
+}
+
+// Dir copies a whole directory recursively
+func CopyDir(src string, dst string) error {
+	var err error
+	var fds []os.FileInfo
+	var srcInfo os.FileInfo
+
+	if srcInfo, err = os.Stat(src); err != nil {
+		return err
+	}
+
+	if err = os.MkdirAll(dst, srcInfo.Mode()); err != nil {
+		return err
+	}
+
+	if fds, err = ioutil.ReadDir(src); err != nil {
+		return err
+	}
+	for _, fd := range fds {
+		srcfp := path.Join(src, fd.Name())
+		dstfp := path.Join(dst, fd.Name())
+
+		if fd.IsDir() {
+			if err = CopyDir(srcfp, dstfp); err != nil {
+				fmt.Println(err)
+			}
+		} else {
+			if err = CopyFile(srcfp, dstfp); err != nil {
+				fmt.Println(err)
+			}
+		}
+	}
+	return nil
+}
+
+// 设置文件变量值
+// 可以理解为将文件中的变量占位符替换为想要设置的值
+func SetFileVariable(filePath string, key string, value string) error {
+	// 占位符标识
+	sep := "###"
+
+	// 读取文件到字节
+	contentBytes, err := ioutil.ReadFile(filePath)
+	if err != nil {
+		return err
+	}
+
+	// 将字节转化为文本
+	content := string(contentBytes)
+
+	// 替换文本
+	content = strings.Replace(content, fmt.Sprintf("%s%s%s", sep, key, sep), value, -1)
+
+	// 打开文件
+	f, err := os.OpenFile(filePath, os.O_WRONLY|os.O_TRUNC, 0777)
+	if err != nil {
+		return err
+	}
+
+	// 将替换后的内容写入文件
+	if _, err := f.Write([]byte(content)); err != nil {
+		return err
+	}
+
+	f.Close()
+
+	return nil
+}
--- a/backend/utils/helpers.go
+++ b/backend/utils/helpers.go
@@ -6,6 +6,7 @@ import (
 	"github.com/apex/log"
 	"github.com/gomodule/redigo/redis"
 	"io"
+	"reflect"
 	"runtime/debug"
 	"unsafe"
 )
@@ -40,3 +41,20 @@ func Close(c io.Closer) {
 		//log.WithError(err).Error("关闭资源文件失败。")
 	}
 }
+
+func Contains(array interface{}, val interface{}) (fla bool) {
+	fla = false
+	switch reflect.TypeOf(array).Kind() {
+	case reflect.Slice:
+		{
+			s := reflect.ValueOf(array)
+			for i := 0; i < s.Len(); i++ {
+				if reflect.DeepEqual(val, s.Index(i).Interface()) {
+					fla = true
+					return
+				}
+			}
+		}
+	}
+	return
+}
--- a/backend/vendor/github.com/globalsign/mgo/bson/bson_corpus_spec_test_generator.go
+++ b/backend/vendor/github.com/globalsign/mgo/bson/bson_corpus_spec_test_generator.go
@@ -1,294 +0,0 @@
-// +build ignore
-
-package main
-
-import (
-	"bytes"
-	"fmt"
-	"go/format"
-	"html/template"
-	"io/ioutil"
-	"log"
-	"path/filepath"
-	"strings"
-
-	"github.com/globalsign/mgo/internal/json"
-)
-
-func main() {
-	log.SetFlags(0)
-	log.SetPrefix(name + ": ")
-
-	var g Generator
-
-	fmt.Fprintf(&g, "// Code generated by \"%s.go\"; DO NOT EDIT\n\n", name)
-
-	src := g.generate()
-
-	err := ioutil.WriteFile(fmt.Sprintf("%s.go", strings.TrimSuffix(name, "_generator")), src, 0644)
-	if err != nil {
-		log.Fatalf("writing output: %s", err)
-	}
-}
-
-// Generator holds the state of the analysis. Primarily used to buffer
-// the output for format.Source.
-type Generator struct {
-	bytes.Buffer // Accumulated output.
-}
-
-// format returns the gofmt-ed contents of the Generator's buffer.
-func (g *Generator) format() []byte {
-	src, err := format.Source(g.Bytes())
-	if err != nil {
-		// Should never happen, but can arise when developing this code.
-		// The user can compile the output to see the error.
-		log.Printf("warning: internal error: invalid Go generated: %s", err)
-		log.Printf("warning: compile the package to analyze the error")
-		return g.Bytes()
-	}
-	return src
-}
-
-// EVERYTHING ABOVE IS CONSTANT BETWEEN THE GENERATORS
-
-const name = "bson_corpus_spec_test_generator"
-
-func (g *Generator) generate() []byte {
-
-	testFiles, err := filepath.Glob("./specdata/specifications/source/bson-corpus/tests/*.json")
-	if err != nil {
-		log.Fatalf("error reading bson-corpus files: %s", err)
-	}
-
-	tests, err := g.loadTests(testFiles)
-	if err != nil {
-		log.Fatalf("error loading tests: %s", err)
-	}
-
-	tmpl, err := g.getTemplate()
-	if err != nil {
-		log.Fatalf("error loading template: %s", err)
-	}
-
-	tmpl.Execute(&g.Buffer, tests)
-
-	return g.format()
-}
-
-func (g *Generator) loadTests(filenames []string) ([]*testDef, error) {
-	var tests []*testDef
-	for _, filename := range filenames {
-		test, err := g.loadTest(filename)
-		if err != nil {
-			return nil, err
-		}
-
-		tests = append(tests, test)
-	}
-
-	return tests, nil
-}
-
-func (g *Generator) loadTest(filename string) (*testDef, error) {
-	content, err := ioutil.ReadFile(filename)
-	if err != nil {
-		return nil, err
-	}
-
-	var testDef testDef
-	err = json.Unmarshal(content, &testDef)
-	if err != nil {
-		return nil, err
-	}
-
-	names := make(map[string]struct{})
-
-	for i := len(testDef.Valid) - 1; i >= 0; i-- {
-		if testDef.BsonType == "0x05" && testDef.Valid[i].Description == "subtype 0x02" {
-			testDef.Valid = append(testDef.Valid[:i], testDef.Valid[i+1:]...)
-			continue
-		}
-
-		name := cleanupFuncName(testDef.Description + "_" + testDef.Valid[i].Description)
-		nameIdx := name
-		j := 1
-		for {
-			if _, ok := names[nameIdx]; !ok {
-				break
-			}
-
-			nameIdx = fmt.Sprintf("%s_%d", name, j)
-		}
-
-		names[nameIdx] = struct{}{}
-
-		testDef.Valid[i].TestDef = &testDef
-		testDef.Valid[i].Name = nameIdx
-		testDef.Valid[i].StructTest = testDef.TestKey != "" &&
-			(testDef.BsonType != "0x05" || strings.Contains(testDef.Valid[i].Description, "0x00")) &&
-			!testDef.Deprecated
-	}
-
-	for i := len(testDef.DecodeErrors) - 1; i >= 0; i-- {
-		if strings.Contains(testDef.DecodeErrors[i].Description, "UTF-8") {
-			testDef.DecodeErrors = append(testDef.DecodeErrors[:i], testDef.DecodeErrors[i+1:]...)
-			continue
-		}
-
-		name := cleanupFuncName(testDef.Description + "_" + testDef.DecodeErrors[i].Description)
-		nameIdx := name
-		j := 1
-		for {
-			if _, ok := names[nameIdx]; !ok {
-				break
-			}
-
-			nameIdx = fmt.Sprintf("%s_%d", name, j)
-		}
-		names[nameIdx] = struct{}{}
-
-		testDef.DecodeErrors[i].Name = nameIdx
-	}
-
-	return &testDef, nil
-}
-
-func (g *Generator) getTemplate() (*template.Template, error) {
-	content := `package bson_test
-
-import (
-    "encoding/hex"
-	"time"
-
-	. "gopkg.in/check.v1"
-    "github.com/globalsign/mgo/bson"
-)
-
-func testValid(c *C, in []byte, expected []byte, result interface{}) {
-	err := bson.Unmarshal(in, result)
-	c.Assert(err, IsNil)
-
-	out, err := bson.Marshal(result)
-	c.Assert(err, IsNil)
-
-	c.Assert(string(expected), Equals, string(out), Commentf("roundtrip failed for %T, expected '%x' but got '%x'", result, expected, out))
-}
-
-func testDecodeSkip(c *C, in []byte) {
-	err := bson.Unmarshal(in, &struct{}{})
-	c.Assert(err, IsNil)
-}
-
-func testDecodeError(c *C, in []byte, result interface{}) {
-	err := bson.Unmarshal(in, result)
-	c.Assert(err, Not(IsNil))
-}
-
-{{range .}}
-{{range .Valid}}
-func (s *S) Test{{.Name}}(c *C) {
-    b, err := hex.DecodeString("{{.Bson}}")
-	c.Assert(err, IsNil)
-
-    {{if .CanonicalBson}}
-    cb, err := hex.DecodeString("{{.CanonicalBson}}")
-	c.Assert(err, IsNil)
-	{{else}}
-    cb := b
-    {{end}}
-
-    var resultD bson.D
-	testValid(c, b, cb, &resultD)
-	{{if .StructTest}}var resultS struct {
-		Element {{.TestDef.GoType}} ` + "`bson:\"{{.TestDef.TestKey}}\"`" + `
-	}
-	testValid(c, b, cb, &resultS){{end}}
-
-	testDecodeSkip(c, b)
-}
-{{end}}
-
-{{range .DecodeErrors}}
-func (s *S) Test{{.Name}}(c *C) {
-	b, err := hex.DecodeString("{{.Bson}}")
-	c.Assert(err, IsNil)
-
-	var resultD bson.D
-	testDecodeError(c, b, &resultD)
-}
-{{end}}
-{{end}}
-`
-	tmpl, err := template.New("").Parse(content)
-	if err != nil {
-		return nil, err
-	}
-	return tmpl, nil
-}
-
-func cleanupFuncName(name string) string {
-	return strings.Map(func(r rune) rune {
-		if (r >= 48 && r <= 57) || (r >= 65 && r <= 90) || (r >= 97 && r <= 122) {
-			return r
-		}
-		return '_'
-	}, name)
-}
-
-type testDef struct {
-	Description  string         `json:"description"`
-	BsonType     string         `json:"bson_type"`
-	TestKey      string         `json:"test_key"`
-	Valid        []*valid       `json:"valid"`
-	DecodeErrors []*decodeError `json:"decodeErrors"`
-	Deprecated   bool           `json:"deprecated"`
-}
-
-func (t *testDef) GoType() string {
-	switch t.BsonType {
-	case "0x01":
-		return "float64"
-	case "0x02":
-		return "string"
-	case "0x03":
-		return "bson.D"
-	case "0x04":
-		return "[]interface{}"
-	case "0x05":
-		return "[]byte"
-	case "0x07":
-		return "bson.ObjectId"
-	case "0x08":
-		return "bool"
-	case "0x09":
-		return "time.Time"
-	case "0x0E":
-		return "string"
-	case "0x10":
-		return "int32"
-	case "0x12":
-		return "int64"
-	case "0x13":
-		return "bson.Decimal"
-	default:
-		return "interface{}"
-	}
-}
-
-type valid struct {
-	Description   string `json:"description"`
-	Bson          string `json:"bson"`
-	CanonicalBson string `json:"canonical_bson"`
-
-	Name       string
-	StructTest bool
-	TestDef    *testDef
-}
-
-type decodeError struct {
-	Description string `json:"description"`
-	Bson        string `json:"bson"`
-
-	Name string
-}
--- a/backend/vendor/github.com/go-playground/locales/.gitignore
+++ b/backend/vendor/github.com/go-playground/locales/.gitignore
@@ -0,0 +1,24 @@
+# Compiled Object files, Static and Dynamic libs (Shared Objects)
+*.o
+*.a
+*.so
+
+# Folders
+_obj
+_test
+
+# Architecture specific extensions/prefixes
+*.[568vq]
+[568vq].out
+
+*.cgo1.go
+*.cgo2.c
+_cgo_defun.c
+_cgo_gotypes.go
+_cgo_export.*
+
+_testmain.go
+
+*.exe
+*.test
+*.prof
--- a/backend/vendor/github.com/go-playground/locales/LICENSE
+++ b/backend/vendor/github.com/go-playground/locales/LICENSE
@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Go Playground
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
--- a/backend/vendor/github.com/go-playground/locales/README.md
+++ b/backend/vendor/github.com/go-playground/locales/README.md
@@ -0,0 +1,172 @@
+## locales
+<img align="right" src="https://raw.githubusercontent.com/go-playground/locales/master/logo.png">![Project status](https://img.shields.io/badge/version-0.12.1-green.svg)
+[![Build Status](https://semaphoreci.com/api/v1/joeybloggs/locales/branches/master/badge.svg)](https://semaphoreci.com/joeybloggs/locales)
+[![Go Report Card](https://goreportcard.com/badge/github.com/go-playground/locales)](https://goreportcard.com/report/github.com/go-playground/locales)
+[![GoDoc](https://godoc.org/github.com/go-playground/locales?status.svg)](https://godoc.org/github.com/go-playground/locales)
+![License](https://img.shields.io/dub/l/vibe-d.svg)
+[![Gitter](https://badges.gitter.im/go-playground/locales.svg)](https://gitter.im/go-playground/locales?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+
+Locales is a set of locales generated from the [Unicode CLDR Project](http://cldr.unicode.org/) which can be used independently or within
+an i18n package; these were built for use with, but not exclusive to, [Universal Translator](https://github.com/go-playground/universal-translator).
+
+Features
+--------
+- [x] Rules generated from the latest [CLDR](http://cldr.unicode.org/index/downloads) data, v31.0.1
+- [x] Contains Cardinal, Ordinal and Range Plural Rules
+- [x] Contains Month, Weekday and Timezone translations built in
+- [x] Contains Date & Time formatting functions
+- [x] Contains Number, Currency, Accounting and Percent formatting functions
+- [x] Supports the "Gregorian" calendar only ( my time isn't unlimited, had to draw the line somewhere )
+
+Full Tests
+--------------------
+I could sure use your help adding tests for every locale, it is a huge undertaking and I just don't have the free time to do it all at the moment;
+any help would be **greatly appreciated!!!!** please see [issue](https://github.com/go-playground/locales/issues/1) for details.
+
+Installation
+-----------
+
+Use go get 
+
+```shell
+go get github.com/go-playground/locales
+```  
+
+NOTES
+--------
+You'll notice most return types are []byte, this is because most of the time the results will be concatenated with a larger body
+of text and can avoid some allocations if already appending to a byte array, otherwise just cast as string.
+
+Usage
+-------
+```go
+package main
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/go-playground/locales/currency"
+	"github.com/go-playground/locales/en_CA"
+)
+
+func main() {
+
+	loc, _ := time.LoadLocation("America/Toronto")
+	datetime := time.Date(2016, 02, 03, 9, 0, 1, 0, loc)
+
+	l := en_CA.New()
+
+	// Dates
+	fmt.Println(l.FmtDateFull(datetime))
+	fmt.Println(l.FmtDateLong(datetime))
+	fmt.Println(l.FmtDateMedium(datetime))
+	fmt.Println(l.FmtDateShort(datetime))
+
+	// Times
+	fmt.Println(l.FmtTimeFull(datetime))
+	fmt.Println(l.FmtTimeLong(datetime))
+	fmt.Println(l.FmtTimeMedium(datetime))
+	fmt.Println(l.FmtTimeShort(datetime))
+
+	// Months Wide
+	fmt.Println(l.MonthWide(time.January))
+	fmt.Println(l.MonthWide(time.February))
+	fmt.Println(l.MonthWide(time.March))
+	// ...
+
+	// Months Abbreviated
+	fmt.Println(l.MonthAbbreviated(time.January))
+	fmt.Println(l.MonthAbbreviated(time.February))
+	fmt.Println(l.MonthAbbreviated(time.March))
+	// ...
+
+	// Months Narrow
+	fmt.Println(l.MonthNarrow(time.January))
+	fmt.Println(l.MonthNarrow(time.February))
+	fmt.Println(l.MonthNarrow(time.March))
+	// ...
+
+	// Weekdays Wide
+	fmt.Println(l.WeekdayWide(time.Sunday))
+	fmt.Println(l.WeekdayWide(time.Monday))
+	fmt.Println(l.WeekdayWide(time.Tuesday))
+	// ...
+
+	// Weekdays Abbreviated
+	fmt.Println(l.WeekdayAbbreviated(time.Sunday))
+	fmt.Println(l.WeekdayAbbreviated(time.Monday))
+	fmt.Println(l.WeekdayAbbreviated(time.Tuesday))
+	// ...
+
+	// Weekdays Short
+	fmt.Println(l.WeekdayShort(time.Sunday))
+	fmt.Println(l.WeekdayShort(time.Monday))
+	fmt.Println(l.WeekdayShort(time.Tuesday))
+	// ...
+
+	// Weekdays Narrow
+	fmt.Println(l.WeekdayNarrow(time.Sunday))
+	fmt.Println(l.WeekdayNarrow(time.Monday))
+	fmt.Println(l.WeekdayNarrow(time.Tuesday))
+	// ...
+
+	var f64 float64
+
+	f64 = -10356.4523
+
+	// Number
+	fmt.Println(l.FmtNumber(f64, 2))
+
+	// Currency
+	fmt.Println(l.FmtCurrency(f64, 2, currency.CAD))
+	fmt.Println(l.FmtCurrency(f64, 2, currency.USD))
+
+	// Accounting
+	fmt.Println(l.FmtAccounting(f64, 2, currency.CAD))
+	fmt.Println(l.FmtAccounting(f64, 2, currency.USD))
+
+	f64 = 78.12
+
+	// Percent
+	fmt.Println(l.FmtPercent(f64, 0))
+
+	// Plural Rules for locale, so you know what rules you must cover
+	fmt.Println(l.PluralsCardinal())
+	fmt.Println(l.PluralsOrdinal())
+
+	// Cardinal Plural Rules
+	fmt.Println(l.CardinalPluralRule(1, 0))
+	fmt.Println(l.CardinalPluralRule(1.0, 0))
+	fmt.Println(l.CardinalPluralRule(1.0, 1))
+	fmt.Println(l.CardinalPluralRule(3, 0))
+
+	// Ordinal Plural Rules
+	fmt.Println(l.OrdinalPluralRule(21, 0)) // 21st
+	fmt.Println(l.OrdinalPluralRule(22, 0)) // 22nd
+	fmt.Println(l.OrdinalPluralRule(33, 0)) // 33rd
+	fmt.Println(l.OrdinalPluralRule(34, 0)) // 34th
+
+	// Range Plural Rules
+	fmt.Println(l.RangePluralRule(1, 0, 1, 0)) // 1-1
+	fmt.Println(l.RangePluralRule(1, 0, 2, 0)) // 1-2
+	fmt.Println(l.RangePluralRule(5, 0, 8, 0)) // 5-8
+}
+```
+
+NOTES:
+-------
+These rules were generated from the [Unicode CLDR Project](http://cldr.unicode.org/), if you encounter any issues
+I strongly encourage contributing to the CLDR project to get the locale information corrected and the next time 
+these locales are regenerated the fix will come with.
+
+I do however realize that time constraints are often important and so there are two options:
+
+1. Create your own locale, copy, paste and modify, and ensure it complies with the `Translator` interface.
+2. Add an exception in the locale generation code directly and once regenerated, fix will be in place.
+
+Please to not make fixes inside the locale files, they WILL get overwritten when the locales are regenerated.
+
+License
+------
+Distributed under MIT License, please see license file in code for more details.
--- a/backend/vendor/github.com/go-playground/locales/currency/currency.go
+++ b/backend/vendor/github.com/go-playground/locales/currency/currency.go
@@ -0,0 +1,308 @@
+package currency
+
+// Type is the currency type associated with the locales currency enum
+type Type int
+
+// locale currencies
+const (
+	ADP Type = iota
+	AED
+	AFA
+	AFN
+	ALK
+	ALL
+	AMD
+	ANG
+	AOA
+	AOK
+	AON
+	AOR
+	ARA
+	ARL
+	ARM
+	ARP
+	ARS
+	ATS
+	AUD
+	AWG
+	AZM
+	AZN
+	BAD
+	BAM
+	BAN
+	BBD
+	BDT
+	BEC
+	BEF
+	BEL
+	BGL
+	BGM
+	BGN
+	BGO
+	BHD
+	BIF
+	BMD
+	BND
+	BOB
+	BOL
+	BOP
+	BOV
+	BRB
+	BRC
+	BRE
+	BRL
+	BRN
+	BRR
+	BRZ
+	BSD
+	BTN
+	BUK
+	BWP
+	BYB
+	BYN
+	BYR
+	BZD
+	CAD
+	CDF
+	CHE
+	CHF
+	CHW
+	CLE
+	CLF
+	CLP
+	CNH
+	CNX
+	CNY
+	COP
+	COU
+	CRC
+	CSD
+	CSK
+	CUC
+	CUP
+	CVE
+	CYP
+	CZK
+	DDM
+	DEM
+	DJF
+	DKK
+	DOP
+	DZD
+	ECS
+	ECV
+	EEK
+	EGP
+	ERN
+	ESA
+	ESB
+	ESP
+	ETB
+	EUR
+	FIM
+	FJD
+	FKP
+	FRF
+	GBP
+	GEK
+	GEL
+	GHC
+	GHS
+	GIP
+	GMD
+	GNF
+	GNS
+	GQE
+	GRD
+	GTQ
+	GWE
+	GWP
+	GYD
+	HKD
+	HNL
+	HRD
+	HRK
+	HTG
+	HUF
+	IDR
+	IEP
+	ILP
+	ILR
+	ILS
+	INR
+	IQD
+	IRR
+	ISJ
+	ISK
+	ITL
+	JMD
+	JOD
+	JPY
+	KES
+	KGS
+	KHR
+	KMF
+	KPW
+	KRH
+	KRO
+	KRW
+	KWD
+	KYD
+	KZT
+	LAK
+	LBP
+	LKR
+	LRD
+	LSL
+	LTL
+	LTT
+	LUC
+	LUF
+	LUL
+	LVL
+	LVR
+	LYD
+	MAD
+	MAF
+	MCF
+	MDC
+	MDL
+	MGA
+	MGF
+	MKD
+	MKN
+	MLF
+	MMK
+	MNT
+	MOP
+	MRO
+	MTL
+	MTP
+	MUR
+	MVP
+	MVR
+	MWK
+	MXN
+	MXP
+	MXV
+	MYR
+	MZE
+	MZM
+	MZN
+	NAD
+	NGN
+	NIC
+	NIO
+	NLG
+	NOK
+	NPR
+	NZD
+	OMR
+	PAB
+	PEI
+	PEN
+	PES
+	PGK
+	PHP
+	PKR
+	PLN
+	PLZ
+	PTE
+	PYG
+	QAR
+	RHD
+	ROL
+	RON
+	RSD
+	RUB
+	RUR
+	RWF
+	SAR
+	SBD
+	SCR
+	SDD
+	SDG
+	SDP
+	SEK
+	SGD
+	SHP
+	SIT
+	SKK
+	SLL
+	SOS
+	SRD
+	SRG
+	SSP
+	STD
+	STN
+	SUR
+	SVC
+	SYP
+	SZL
+	THB
+	TJR
+	TJS
+	TMM
+	TMT
+	TND
+	TOP
+	TPE
+	TRL
+	TRY
+	TTD
+	TWD
+	TZS
+	UAH
+	UAK
+	UGS
+	UGX
+	USD
+	USN
+	USS
+	UYI
+	UYP
+	UYU
+	UZS
+	VEB
+	VEF
+	VND
+	VNN
+	VUV
+	WST
+	XAF
+	XAG
+	XAU
+	XBA
+	XBB
+	XBC
+	XBD
+	XCD
+	XDR
+	XEU
+	XFO
+	XFU
+	XOF
+	XPD
+	XPF
+	XPT
+	XRE
+	XSU
+	XTS
+	XUA
+	XXX
+	YDD
+	YER
+	YUD
+	YUM
+	YUN
+	YUR
+	ZAL
+	ZAR
+	ZMK
+	ZMW
+	ZRN
+	ZRZ
+	ZWD
+	ZWL
+	ZWR
+)
--- a/backend/vendor/github.com/go-playground/locales/logo.png
+++ b/backend/vendor/github.com/go-playground/locales/logo.png
--- a/backend/vendor/github.com/go-playground/locales/rules.go
+++ b/backend/vendor/github.com/go-playground/locales/rules.go
@@ -0,0 +1,293 @@
+package locales
+
+import (
+	"strconv"
+	"time"
+
+	"github.com/go-playground/locales/currency"
+)
+
+// // ErrBadNumberValue is returned when the number passed for
+// // plural rule determination cannot be parsed
+// type ErrBadNumberValue struct {
+// 	NumberValue string
+// 	InnerError  error
+// }
+
+// // Error returns ErrBadNumberValue error string
+// func (e *ErrBadNumberValue) Error() string {
+// 	return fmt.Sprintf("Invalid Number Value '%s' %s", e.NumberValue, e.InnerError)
+// }
+
+// var _ error = new(ErrBadNumberValue)
+
+// PluralRule denotes the type of plural rules
+type PluralRule int
+
+// PluralRule's
+const (
+	PluralRuleUnknown PluralRule = iota
+	PluralRuleZero               // zero
+	PluralRuleOne                // one - singular
+	PluralRuleTwo                // two - dual
+	PluralRuleFew                // few - paucal
+	PluralRuleMany               // many - also used for fractions if they have a separate class
+	PluralRuleOther              // other - required—general plural form—also used if the language only has a single form
+)
+
+const (
+	pluralsString = "UnknownZeroOneTwoFewManyOther"
+)
+
+// Translator encapsulates an instance of a locale
+// NOTE: some values are returned as a []byte just in case the caller
+// wishes to add more and can help avoid allocations; otherwise just cast as string
+type Translator interface {
+
+	// The following Functions are for overriding, debugging or developing
+	// with a Translator Locale
+
+	// Locale returns the string value of the translator
+	Locale() string
+
+	// returns an array of cardinal plural rules associated
+	// with this translator
+	PluralsCardinal() []PluralRule
+
+	// returns an array of ordinal plural rules associated
+	// with this translator
+	PluralsOrdinal() []PluralRule
+
+	// returns an array of range plural rules associated
+	// with this translator
+	PluralsRange() []PluralRule
+
+	// returns the cardinal PluralRule given 'num' and digits/precision of 'v' for locale
+	CardinalPluralRule(num float64, v uint64) PluralRule
+
+	// returns the ordinal PluralRule given 'num' and digits/precision of 'v' for locale
+	OrdinalPluralRule(num float64, v uint64) PluralRule
+
+	// returns the ordinal PluralRule given 'num1', 'num2' and digits/precision of 'v1' and 'v2' for locale
+	RangePluralRule(num1 float64, v1 uint64, num2 float64, v2 uint64) PluralRule
+
+	// returns the locales abbreviated month given the 'month' provided
+	MonthAbbreviated(month time.Month) string
+
+	// returns the locales abbreviated months
+	MonthsAbbreviated() []string
+
+	// returns the locales narrow month given the 'month' provided
+	MonthNarrow(month time.Month) string
+
+	// returns the locales narrow months
+	MonthsNarrow() []string
+
+	// returns the locales wide month given the 'month' provided
+	MonthWide(month time.Month) string
+
+	// returns the locales wide months
+	MonthsWide() []string
+
+	// returns the locales abbreviated weekday given the 'weekday' provided
+	WeekdayAbbreviated(weekday time.Weekday) string
+
+	// returns the locales abbreviated weekdays
+	WeekdaysAbbreviated() []string
+
+	// returns the locales narrow weekday given the 'weekday' provided
+	WeekdayNarrow(weekday time.Weekday) string
+
+	// WeekdaysNarrowreturns the locales narrow weekdays
+	WeekdaysNarrow() []string
+
+	// returns the locales short weekday given the 'weekday' provided
+	WeekdayShort(weekday time.Weekday) string
+
+	// returns the locales short weekdays
+	WeekdaysShort() []string
+
+	// returns the locales wide weekday given the 'weekday' provided
+	WeekdayWide(weekday time.Weekday) string
+
+	// returns the locales wide weekdays
+	WeekdaysWide() []string
+
+	// The following Functions are common Formatting functionsfor the Translator's Locale
+
+	// returns 'num' with digits/precision of 'v' for locale and handles both Whole and Real numbers based on 'v'
+	FmtNumber(num float64, v uint64) string
+
+	// returns 'num' with digits/precision of 'v' for locale and handles both Whole and Real numbers based on 'v'
+	// NOTE: 'num' passed into FmtPercent is assumed to be in percent already
+	FmtPercent(num float64, v uint64) string
+
+	// returns the currency representation of 'num' with digits/precision of 'v' for locale
+	FmtCurrency(num float64, v uint64, currency currency.Type) string
+
+	// returns the currency representation of 'num' with digits/precision of 'v' for locale
+	// in accounting notation.
+	FmtAccounting(num float64, v uint64, currency currency.Type) string
+
+	// returns the short date representation of 't' for locale
+	FmtDateShort(t time.Time) string
+
+	// returns the medium date representation of 't' for locale
+	FmtDateMedium(t time.Time) string
+
+	//  returns the long date representation of 't' for locale
+	FmtDateLong(t time.Time) string
+
+	// returns the full date representation of 't' for locale
+	FmtDateFull(t time.Time) string
+
+	// returns the short time representation of 't' for locale
+	FmtTimeShort(t time.Time) string
+
+	// returns the medium time representation of 't' for locale
+	FmtTimeMedium(t time.Time) string
+
+	// returns the long time representation of 't' for locale
+	FmtTimeLong(t time.Time) string
+
+	// returns the full time representation of 't' for locale
+	FmtTimeFull(t time.Time) string
+}
+
+// String returns the string value  of PluralRule
+func (p PluralRule) String() string {
+
+	switch p {
+	case PluralRuleZero:
+		return pluralsString[7:11]
+	case PluralRuleOne:
+		return pluralsString[11:14]
+	case PluralRuleTwo:
+		return pluralsString[14:17]
+	case PluralRuleFew:
+		return pluralsString[17:20]
+	case PluralRuleMany:
+		return pluralsString[20:24]
+	case PluralRuleOther:
+		return pluralsString[24:]
+	default:
+		return pluralsString[:7]
+	}
+}
+
+//
+// Precision Notes:
+//
+// must specify a precision >= 0, and here is why https://play.golang.org/p/LyL90U0Vyh
+//
+// 	v := float64(3.141)
+// 	i := float64(int64(v))
+//
+// 	fmt.Println(v - i)
+//
+// 	or
+//
+// 	s := strconv.FormatFloat(v-i, 'f', -1, 64)
+// 	fmt.Println(s)
+//
+// these will not print what you'd expect: 0.14100000000000001
+// and so this library requires a precision to be specified, or
+// inaccurate plural rules could be applied.
+//
+//
+//
+// n - absolute value of the source number (integer and decimals).
+// i - integer digits of n.
+// v - number of visible fraction digits in n, with trailing zeros.
+// w - number of visible fraction digits in n, without trailing zeros.
+// f - visible fractional digits in n, with trailing zeros.
+// t - visible fractional digits in n, without trailing zeros.
+//
+//
+// Func(num float64, v uint64) // v = digits/precision and prevents -1 as a special case as this can lead to very unexpected behaviour, see precision note's above.
+//
+// n := math.Abs(num)
+// i := int64(n)
+// v := v
+//
+//
+// w := strconv.FormatFloat(num-float64(i), 'f', int(v), 64)  // then parse backwards on string until no more zero's....
+// f := strconv.FormatFloat(n, 'f', int(v), 64) 			  // then turn everything after decimal into an int64
+// t := strconv.FormatFloat(n, 'f', int(v), 64) 			  // then parse backwards on string until no more zero's....
+//
+//
+//
+// General Inclusion Rules
+// - v will always be available inherently
+// - all require n
+// - w requires i
+//
+
+// W returns the number of visible fraction digits in N, without trailing zeros.
+func W(n float64, v uint64) (w int64) {
+
+	s := strconv.FormatFloat(n-float64(int64(n)), 'f', int(v), 64)
+
+	// with either be '0' or '0.xxxx', so if 1 then w will be zero
+	// otherwise need to parse
+	if len(s) != 1 {
+
+		s = s[2:]
+		end := len(s) + 1
+
+		for i := end; i >= 0; i-- {
+			if s[i] != '0' {
+				end = i + 1
+				break
+			}
+		}
+
+		w = int64(len(s[:end]))
+	}
+
+	return
+}
+
+// F returns the visible fractional digits in N, with trailing zeros.
+func F(n float64, v uint64) (f int64) {
+
+	s := strconv.FormatFloat(n-float64(int64(n)), 'f', int(v), 64)
+
+	// with either be '0' or '0.xxxx', so if 1 then f will be zero
+	// otherwise need to parse
+	if len(s) != 1 {
+
+		// ignoring error, because it can't fail as we generated
+		// the string internally from a real number
+		f, _ = strconv.ParseInt(s[2:], 10, 64)
+	}
+
+	return
+}
+
+// T returns the visible fractional digits in N, without trailing zeros.
+func T(n float64, v uint64) (t int64) {
+
+	s := strconv.FormatFloat(n-float64(int64(n)), 'f', int(v), 64)
+
+	// with either be '0' or '0.xxxx', so if 1 then t will be zero
+	// otherwise need to parse
+	if len(s) != 1 {
+
+		s = s[2:]
+		end := len(s) + 1
+
+		for i := end; i >= 0; i-- {
+			if s[i] != '0' {
+				end = i + 1
+				break
+			}
+		}
+
+		// ignoring error, because it can't fail as we generated
+		// the string internally from a real number
+		t, _ = strconv.ParseInt(s[:end], 10, 64)
+	}
+
+	return
+}
--- a/backend/vendor/github.com/go-playground/universal-translator/.gitignore
+++ b/backend/vendor/github.com/go-playground/universal-translator/.gitignore
@@ -0,0 +1,24 @@
+# Compiled Object files, Static and Dynamic libs (Shared Objects)
+*.o
+*.a
+*.so
+
+# Folders
+_obj
+_test
+
+# Architecture specific extensions/prefixes
+*.[568vq]
+[568vq].out
+
+*.cgo1.go
+*.cgo2.c
+_cgo_defun.c
+_cgo_gotypes.go
+_cgo_export.*
+
+_testmain.go
+
+*.exe
+*.test
+*.prof
--- a/backend/vendor/github.com/go-playground/universal-translator/LICENSE
+++ b/backend/vendor/github.com/go-playground/universal-translator/LICENSE
@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Go Playground
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
--- a/backend/vendor/github.com/go-playground/universal-translator/README.md
+++ b/backend/vendor/github.com/go-playground/universal-translator/README.md
@@ -0,0 +1,90 @@
+## universal-translator
+<img align="right" src="https://raw.githubusercontent.com/go-playground/universal-translator/master/logo.png">
+![Project status](https://img.shields.io/badge/version-0.16.0-green.svg)
+[![Build Status](https://semaphoreci.com/api/v1/joeybloggs/universal-translator/branches/master/badge.svg)](https://semaphoreci.com/joeybloggs/universal-translator)
+[![Coverage Status](https://coveralls.io/repos/github/go-playground/universal-translator/badge.svg)](https://coveralls.io/github/go-playground/universal-translator)
+[![Go Report Card](https://goreportcard.com/badge/github.com/go-playground/universal-translator)](https://goreportcard.com/report/github.com/go-playground/universal-translator)
+[![GoDoc](https://godoc.org/github.com/go-playground/universal-translator?status.svg)](https://godoc.org/github.com/go-playground/universal-translator)
+![License](https://img.shields.io/dub/l/vibe-d.svg)
+[![Gitter](https://badges.gitter.im/go-playground/universal-translator.svg)](https://gitter.im/go-playground/universal-translator?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+
+Universal Translator is an i18n Translator for Go/Golang using CLDR data + pluralization rules
+
+Why another i18n library?
+--------------------------
+Because none of the plural rules seem to be correct out there, including the previous implementation of this package,
+so I took it upon myself to create [locales](https://github.com/go-playground/locales) for everyone to use; this package 
+is a thin wrapper around [locales](https://github.com/go-playground/locales) in order to store and translate text for 
+use in your applications.
+
+Features
+--------
+- [x] Rules generated from the [CLDR](http://cldr.unicode.org/index/downloads) data, v30.0.3
+- [x] Contains Cardinal, Ordinal and Range Plural Rules
+- [x] Contains Month, Weekday and Timezone translations built in
+- [x] Contains Date & Time formatting functions
+- [x] Contains Number, Currency, Accounting and Percent formatting functions
+- [x] Supports the "Gregorian" calendar only ( my time isn't unlimited, had to draw the line somewhere )
+- [x] Support loading translations from files
+- [x] Exporting translations to file(s), mainly for getting them professionally translated
+- [ ] Code Generation for translation files -> Go code.. i.e. after it has been professionally translated
+- [ ] Tests for all languages, I need help with this, please see [here](https://github.com/go-playground/locales/issues/1)
+
+Installation
+-----------
+
+Use go get 
+
+```shell
+go get github.com/go-playground/universal-translator
+```
+
+Usage & Documentation
+-------
+
+Please see https://godoc.org/github.com/go-playground/universal-translator for usage docs
+
+##### Examples:
+
+- [Basic](https://github.com/go-playground/universal-translator/tree/master/examples/basic)
+- [Full - no files](https://github.com/go-playground/universal-translator/tree/master/examples/full-no-files)
+- [Full - with files](https://github.com/go-playground/universal-translator/tree/master/examples/full-with-files)
+
+File formatting
+--------------
+All types, Plain substitution, Cardinal, Ordinal and Range translations can all be contained withing the same file(s);
+they are only separated for easy viewing.
+
+##### Examples:
+
+- [Formats](https://github.com/go-playground/universal-translator/tree/master/examples/file-formats)
+
+##### Basic Makeup
+NOTE: not all fields are needed for all translation types, see [examples](https://github.com/go-playground/universal-translator/tree/master/examples/file-formats)
+```json
+{
+    "locale": "en",
+    "key": "days-left",
+    "trans": "You have {0} day left.",
+    "type": "Cardinal",
+    "rule": "One",
+    "override": false
+}
+```
+|Field|Description|
+|---|---|
+|locale|The locale for which the translation is for.|
+|key|The translation key that will be used to store and lookup each translation; normally it is a string or integer.|
+|trans|The actual translation text.|
+|type|The type of translation Cardinal, Ordinal, Range or "" for a plain substitution(not required to be defined if plain used)|
+|rule|The plural rule for which the translation is for eg. One, Two, Few, Many or Other.(not required to be defined if plain used)|
+|override|If you wish to override an existing translation that has already been registered, set this to 'true'. 99% of the time there is no need to define it.|
+
+Help With Tests
+---------------
+To anyone interesting in helping or contributing, I sure could use some help creating tests for each language.
+Please see issue [here](https://github.com/go-playground/locales/issues/1) for details.
+
+License
+------
+Distributed under MIT License, please see license file in code for more details.
--- a/backend/vendor/github.com/go-playground/universal-translator/errors.go
+++ b/backend/vendor/github.com/go-playground/universal-translator/errors.go
@@ -0,0 +1,148 @@
+package ut
+
+import (
+	"errors"
+	"fmt"
+
+	"github.com/go-playground/locales"
+)
+
+var (
+	// ErrUnknowTranslation indicates the translation could not be found
+	ErrUnknowTranslation = errors.New("Unknown Translation")
+)
+
+var _ error = new(ErrConflictingTranslation)
+var _ error = new(ErrRangeTranslation)
+var _ error = new(ErrOrdinalTranslation)
+var _ error = new(ErrCardinalTranslation)
+var _ error = new(ErrMissingPluralTranslation)
+var _ error = new(ErrExistingTranslator)
+
+// ErrExistingTranslator is the error representing a conflicting translator
+type ErrExistingTranslator struct {
+	locale string
+}
+
+// Error returns ErrExistingTranslator's internal error text
+func (e *ErrExistingTranslator) Error() string {
+	return fmt.Sprintf("error: conflicting translator for locale '%s'", e.locale)
+}
+
+// ErrConflictingTranslation is the error representing a conflicting translation
+type ErrConflictingTranslation struct {
+	locale string
+	key    interface{}
+	rule   locales.PluralRule
+	text   string
+}
+
+// Error returns ErrConflictingTranslation's internal error text
+func (e *ErrConflictingTranslation) Error() string {
+
+	if _, ok := e.key.(string); !ok {
+		return fmt.Sprintf("error: conflicting key '%#v' rule '%s' with text '%s' for locale '%s', value being ignored", e.key, e.rule, e.text, e.locale)
+	}
+
+	return fmt.Sprintf("error: conflicting key '%s' rule '%s' with text '%s' for locale '%s', value being ignored", e.key, e.rule, e.text, e.locale)
+}
+
+// ErrRangeTranslation is the error representing a range translation error
+type ErrRangeTranslation struct {
+	text string
+}
+
+// Error returns ErrRangeTranslation's internal error text
+func (e *ErrRangeTranslation) Error() string {
+	return e.text
+}
+
+// ErrOrdinalTranslation is the error representing an ordinal translation error
+type ErrOrdinalTranslation struct {
+	text string
+}
+
+// Error returns ErrOrdinalTranslation's internal error text
+func (e *ErrOrdinalTranslation) Error() string {
+	return e.text
+}
+
+// ErrCardinalTranslation is the error representing a cardinal translation error
+type ErrCardinalTranslation struct {
+	text string
+}
+
+// Error returns ErrCardinalTranslation's internal error text
+func (e *ErrCardinalTranslation) Error() string {
+	return e.text
+}
+
+// ErrMissingPluralTranslation is the error signifying a missing translation given
+// the locales plural rules.
+type ErrMissingPluralTranslation struct {
+	locale          string
+	key             interface{}
+	rule            locales.PluralRule
+	translationType string
+}
+
+// Error returns ErrMissingPluralTranslation's internal error text
+func (e *ErrMissingPluralTranslation) Error() string {
+
+	if _, ok := e.key.(string); !ok {
+		return fmt.Sprintf("error: missing '%s' plural rule '%s' for translation with key '%#v' and locale '%s'", e.translationType, e.rule, e.key, e.locale)
+	}
+
+	return fmt.Sprintf("error: missing '%s' plural rule '%s' for translation with key '%s' and locale '%s'", e.translationType, e.rule, e.key, e.locale)
+}
+
+// ErrMissingBracket is the error representing a missing bracket in a translation
+// eg. This is a {0 <-- missing ending '}'
+type ErrMissingBracket struct {
+	locale string
+	key    interface{}
+	text   string
+}
+
+// Error returns ErrMissingBracket error message
+func (e *ErrMissingBracket) Error() string {
+	return fmt.Sprintf("error: missing bracket '{}', in translation. locale: '%s' key: '%v' text: '%s'", e.locale, e.key, e.text)
+}
+
+// ErrBadParamSyntax is the error representing a bad parameter definition in a translation
+// eg. This is a {must-be-int}
+type ErrBadParamSyntax struct {
+	locale string
+	param  string
+	key    interface{}
+	text   string
+}
+
+// Error returns ErrBadParamSyntax error message
+func (e *ErrBadParamSyntax) Error() string {
+	return fmt.Sprintf("error: bad parameter syntax, missing parameter '%s' in translation. locale: '%s' key: '%v' text: '%s'", e.param, e.locale, e.key, e.text)
+}
+
+// import/export errors
+
+// ErrMissingLocale is the error representing an expected locale that could
+// not be found aka locale not registered with the UniversalTranslator Instance
+type ErrMissingLocale struct {
+	locale string
+}
+
+// Error returns ErrMissingLocale's internal error text
+func (e *ErrMissingLocale) Error() string {
+	return fmt.Sprintf("error: locale '%s' not registered.", e.locale)
+}
+
+// ErrBadPluralDefinition is the error representing an incorrect plural definition
+// usually found within translations defined within files during the import process.
+type ErrBadPluralDefinition struct {
+	tl translation
+}
+
+// Error returns ErrBadPluralDefinition's internal error text
+func (e *ErrBadPluralDefinition) Error() string {
+	return fmt.Sprintf("error: bad plural definition '%#v'", e.tl)
+}
--- a/backend/vendor/github.com/go-playground/universal-translator/import_export.go
+++ b/backend/vendor/github.com/go-playground/universal-translator/import_export.go
@@ -0,0 +1,274 @@
+package ut
+
+import (
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+
+	"io"
+
+	"github.com/go-playground/locales"
+)
+
+type translation struct {
+	Locale           string      `json:"locale"`
+	Key              interface{} `json:"key"` // either string or integer
+	Translation      string      `json:"trans"`
+	PluralType       string      `json:"type,omitempty"`
+	PluralRule       string      `json:"rule,omitempty"`
+	OverrideExisting bool        `json:"override,omitempty"`
+}
+
+const (
+	cardinalType = "Cardinal"
+	ordinalType  = "Ordinal"
+	rangeType    = "Range"
+)
+
+// ImportExportFormat is the format of the file import or export
+type ImportExportFormat uint8
+
+// supported Export Formats
+const (
+	FormatJSON ImportExportFormat = iota
+)
+
+// Export writes the translations out to a file on disk.
+//
+// NOTE: this currently only works with string or int translations keys.
+func (t *UniversalTranslator) Export(format ImportExportFormat, dirname string) error {
+
+	_, err := os.Stat(dirname)
+	fmt.Println(dirname, err, os.IsNotExist(err))
+	if err != nil {
+
+		if !os.IsNotExist(err) {
+			return err
+		}
+
+		if err = os.MkdirAll(dirname, 0744); err != nil {
+			return err
+		}
+	}
+
+	// build up translations
+	var trans []translation
+	var b []byte
+	var ext string
+
+	for _, locale := range t.translators {
+
+		for k, v := range locale.(*translator).translations {
+			trans = append(trans, translation{
+				Locale:      locale.Locale(),
+				Key:         k,
+				Translation: v.text,
+			})
+		}
+
+		for k, pluralTrans := range locale.(*translator).cardinalTanslations {
+
+			for i, plural := range pluralTrans {
+
+				// leave enough for all plural rules
+				// but not all are set for all languages.
+				if plural == nil {
+					continue
+				}
+
+				trans = append(trans, translation{
+					Locale:      locale.Locale(),
+					Key:         k.(string),
+					Translation: plural.text,
+					PluralType:  cardinalType,
+					PluralRule:  locales.PluralRule(i).String(),
+				})
+			}
+		}
+
+		for k, pluralTrans := range locale.(*translator).ordinalTanslations {
+
+			for i, plural := range pluralTrans {
+
+				// leave enough for all plural rules
+				// but not all are set for all languages.
+				if plural == nil {
+					continue
+				}
+
+				trans = append(trans, translation{
+					Locale:      locale.Locale(),
+					Key:         k.(string),
+					Translation: plural.text,
+					PluralType:  ordinalType,
+					PluralRule:  locales.PluralRule(i).String(),
+				})
+			}
+		}
+
+		for k, pluralTrans := range locale.(*translator).rangeTanslations {
+
+			for i, plural := range pluralTrans {
+
+				// leave enough for all plural rules
+				// but not all are set for all languages.
+				if plural == nil {
+					continue
+				}
+
+				trans = append(trans, translation{
+					Locale:      locale.Locale(),
+					Key:         k.(string),
+					Translation: plural.text,
+					PluralType:  rangeType,
+					PluralRule:  locales.PluralRule(i).String(),
+				})
+			}
+		}
+
+		switch format {
+		case FormatJSON:
+			b, err = json.MarshalIndent(trans, "", "    ")
+			ext = ".json"
+		}
+
+		if err != nil {
+			return err
+		}
+
+		err = ioutil.WriteFile(filepath.Join(dirname, fmt.Sprintf("%s%s", locale.Locale(), ext)), b, 0644)
+		if err != nil {
+			return err
+		}
+
+		trans = trans[0:0]
+	}
+
+	return nil
+}
+
+// Import reads the translations out of a file or directory on disk.
+//
+// NOTE: this currently only works with string or int translations keys.
+func (t *UniversalTranslator) Import(format ImportExportFormat, dirnameOrFilename string) error {
+
+	fi, err := os.Stat(dirnameOrFilename)
+	if err != nil {
+		return err
+	}
+
+	processFn := func(filename string) error {
+
+		f, err := os.Open(filename)
+		if err != nil {
+			return err
+		}
+		defer f.Close()
+
+		return t.ImportByReader(format, f)
+	}
+
+	if !fi.IsDir() {
+		return processFn(dirnameOrFilename)
+	}
+
+	// recursively go through directory
+	walker := func(path string, info os.FileInfo, err error) error {
+
+		if info.IsDir() {
+			return nil
+		}
+
+		switch format {
+		case FormatJSON:
+			// skip non JSON files
+			if filepath.Ext(info.Name()) != ".json" {
+				return nil
+			}
+		}
+
+		return processFn(path)
+	}
+
+	return filepath.Walk(dirnameOrFilename, walker)
+}
+
+// ImportByReader imports the the translations found within the contents read from the supplied reader.
+//
+// NOTE: generally used when assets have been embedded into the binary and are already in memory.
+func (t *UniversalTranslator) ImportByReader(format ImportExportFormat, reader io.Reader) error {
+
+	b, err := ioutil.ReadAll(reader)
+	if err != nil {
+		return err
+	}
+
+	var trans []translation
+
+	switch format {
+	case FormatJSON:
+		err = json.Unmarshal(b, &trans)
+	}
+
+	if err != nil {
+		return err
+	}
+
+	for _, tl := range trans {
+
+		locale, found := t.FindTranslator(tl.Locale)
+		if !found {
+			return &ErrMissingLocale{locale: tl.Locale}
+		}
+
+		pr := stringToPR(tl.PluralRule)
+
+		if pr == locales.PluralRuleUnknown {
+
+			err = locale.Add(tl.Key, tl.Translation, tl.OverrideExisting)
+			if err != nil {
+				return err
+			}
+
+			continue
+		}
+
+		switch tl.PluralType {
+		case cardinalType:
+			err = locale.AddCardinal(tl.Key, tl.Translation, pr, tl.OverrideExisting)
+		case ordinalType:
+			err = locale.AddOrdinal(tl.Key, tl.Translation, pr, tl.OverrideExisting)
+		case rangeType:
+			err = locale.AddRange(tl.Key, tl.Translation, pr, tl.OverrideExisting)
+		default:
+			return &ErrBadPluralDefinition{tl: tl}
+		}
+
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func stringToPR(s string) locales.PluralRule {
+
+	switch s {
+	case "One":
+		return locales.PluralRuleOne
+	case "Two":
+		return locales.PluralRuleTwo
+	case "Few":
+		return locales.PluralRuleFew
+	case "Many":
+		return locales.PluralRuleMany
+	case "Other":
+		return locales.PluralRuleOther
+	default:
+		return locales.PluralRuleUnknown
+	}
+
+}
--- a/backend/vendor/github.com/go-playground/universal-translator/logo.png
+++ b/backend/vendor/github.com/go-playground/universal-translator/logo.png
--- a/backend/vendor/github.com/go-playground/universal-translator/translator.go
+++ b/backend/vendor/github.com/go-playground/universal-translator/translator.go
@@ -0,0 +1,420 @@
+package ut
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+
+	"github.com/go-playground/locales"
+)
+
+const (
+	paramZero          = "{0}"
+	paramOne           = "{1}"
+	unknownTranslation = ""
+)
+
+// Translator is universal translators
+// translator instance which is a thin wrapper
+// around locales.Translator instance providing
+// some extra functionality
+type Translator interface {
+	locales.Translator
+
+	// adds a normal translation for a particular language/locale
+	// {#} is the only replacement type accepted and are ad infinitum
+	// eg. one: '{0} day left' other: '{0} days left'
+	Add(key interface{}, text string, override bool) error
+
+	// adds a cardinal plural translation for a particular language/locale
+	// {0} is the only replacement type accepted and only one variable is accepted as
+	// multiple cannot be used for a plural rule determination, unless it is a range;
+	// see AddRange below.
+	// eg. in locale 'en' one: '{0} day left' other: '{0} days left'
+	AddCardinal(key interface{}, text string, rule locales.PluralRule, override bool) error
+
+	// adds an ordinal plural translation for a particular language/locale
+	// {0} is the only replacement type accepted and only one variable is accepted as
+	// multiple cannot be used for a plural rule determination, unless it is a range;
+	// see AddRange below.
+	// eg. in locale 'en' one: '{0}st day of spring' other: '{0}nd day of spring'
+	// - 1st, 2nd, 3rd...
+	AddOrdinal(key interface{}, text string, rule locales.PluralRule, override bool) error
+
+	// adds a range plural translation for a particular language/locale
+	// {0} and {1} are the only replacement types accepted and only these are accepted.
+	// eg. in locale 'nl' one: '{0}-{1} day left' other: '{0}-{1} days left'
+	AddRange(key interface{}, text string, rule locales.PluralRule, override bool) error
+
+	// creates the translation for the locale given the 'key' and params passed in
+	T(key interface{}, params ...string) (string, error)
+
+	// creates the cardinal translation for the locale given the 'key', 'num' and 'digit' arguments
+	//  and param passed in
+	C(key interface{}, num float64, digits uint64, param string) (string, error)
+
+	// creates the ordinal translation for the locale given the 'key', 'num' and 'digit' arguments
+	// and param passed in
+	O(key interface{}, num float64, digits uint64, param string) (string, error)
+
+	//  creates the range translation for the locale given the 'key', 'num1', 'digit1', 'num2' and
+	//  'digit2' arguments and 'param1' and 'param2' passed in
+	R(key interface{}, num1 float64, digits1 uint64, num2 float64, digits2 uint64, param1, param2 string) (string, error)
+
+	// VerifyTranslations checks to ensures that no plural rules have been
+	// missed within the translations.
+	VerifyTranslations() error
+}
+
+var _ Translator = new(translator)
+var _ locales.Translator = new(translator)
+
+type translator struct {
+	locales.Translator
+	translations        map[interface{}]*transText
+	cardinalTanslations map[interface{}][]*transText // array index is mapped to locales.PluralRule index + the locales.PluralRuleUnknown
+	ordinalTanslations  map[interface{}][]*transText
+	rangeTanslations    map[interface{}][]*transText
+}
+
+type transText struct {
+	text    string
+	indexes []int
+}
+
+func newTranslator(trans locales.Translator) Translator {
+	return &translator{
+		Translator:          trans,
+		translations:        make(map[interface{}]*transText), // translation text broken up by byte index
+		cardinalTanslations: make(map[interface{}][]*transText),
+		ordinalTanslations:  make(map[interface{}][]*transText),
+		rangeTanslations:    make(map[interface{}][]*transText),
+	}
+}
+
+// Add adds a normal translation for a particular language/locale
+// {#} is the only replacement type accepted and are ad infinitum
+// eg. one: '{0} day left' other: '{0} days left'
+func (t *translator) Add(key interface{}, text string, override bool) error {
+
+	if _, ok := t.translations[key]; ok && !override {
+		return &ErrConflictingTranslation{locale: t.Locale(), key: key, text: text}
+	}
+
+	lb := strings.Count(text, "{")
+	rb := strings.Count(text, "}")
+
+	if lb != rb {
+		return &ErrMissingBracket{locale: t.Locale(), key: key, text: text}
+	}
+
+	trans := &transText{
+		text: text,
+	}
+
+	var idx int
+
+	for i := 0; i < lb; i++ {
+		s := "{" + strconv.Itoa(i) + "}"
+		idx = strings.Index(text, s)
+		if idx == -1 {
+			return &ErrBadParamSyntax{locale: t.Locale(), param: s, key: key, text: text}
+		}
+
+		trans.indexes = append(trans.indexes, idx)
+		trans.indexes = append(trans.indexes, idx+len(s))
+	}
+
+	t.translations[key] = trans
+
+	return nil
+}
+
+// AddCardinal adds a cardinal plural translation for a particular language/locale
+// {0} is the only replacement type accepted and only one variable is accepted as
+// multiple cannot be used for a plural rule determination, unless it is a range;
+// see AddRange below.
+// eg. in locale 'en' one: '{0} day left' other: '{0} days left'
+func (t *translator) AddCardinal(key interface{}, text string, rule locales.PluralRule, override bool) error {
+
+	var verified bool
+
+	// verify plural rule exists for locale
+	for _, pr := range t.PluralsCardinal() {
+		if pr == rule {
+			verified = true
+			break
+		}
+	}
+
+	if !verified {
+		return &ErrCardinalTranslation{text: fmt.Sprintf("error: cardinal plural rule '%s' does not exist for locale '%s' key: '%v' text: '%s'", rule, t.Locale(), key, text)}
+	}
+
+	tarr, ok := t.cardinalTanslations[key]
+	if ok {
+		// verify not adding a conflicting record
+		if len(tarr) > 0 && tarr[rule] != nil && !override {
+			return &ErrConflictingTranslation{locale: t.Locale(), key: key, rule: rule, text: text}
+		}
+
+	} else {
+		tarr = make([]*transText, 7, 7)
+		t.cardinalTanslations[key] = tarr
+	}
+
+	trans := &transText{
+		text:    text,
+		indexes: make([]int, 2, 2),
+	}
+
+	tarr[rule] = trans
+
+	idx := strings.Index(text, paramZero)
+	if idx == -1 {
+		tarr[rule] = nil
+		return &ErrCardinalTranslation{text: fmt.Sprintf("error: parameter '%s' not found, may want to use 'Add' instead of 'AddCardinal'. locale: '%s' key: '%v' text: '%s'", paramZero, t.Locale(), key, text)}
+	}
+
+	trans.indexes[0] = idx
+	trans.indexes[1] = idx + len(paramZero)
+
+	return nil
+}
+
+// AddOrdinal adds an ordinal plural translation for a particular language/locale
+// {0} is the only replacement type accepted and only one variable is accepted as
+// multiple cannot be used for a plural rule determination, unless it is a range;
+// see AddRange below.
+// eg. in locale 'en' one: '{0}st day of spring' other: '{0}nd day of spring' - 1st, 2nd, 3rd...
+func (t *translator) AddOrdinal(key interface{}, text string, rule locales.PluralRule, override bool) error {
+
+	var verified bool
+
+	// verify plural rule exists for locale
+	for _, pr := range t.PluralsOrdinal() {
+		if pr == rule {
+			verified = true
+			break
+		}
+	}
+
+	if !verified {
+		return &ErrOrdinalTranslation{text: fmt.Sprintf("error: ordinal plural rule '%s' does not exist for locale '%s' key: '%v' text: '%s'", rule, t.Locale(), key, text)}
+	}
+
+	tarr, ok := t.ordinalTanslations[key]
+	if ok {
+		// verify not adding a conflicting record
+		if len(tarr) > 0 && tarr[rule] != nil && !override {
+			return &ErrConflictingTranslation{locale: t.Locale(), key: key, rule: rule, text: text}
+		}
+
+	} else {
+		tarr = make([]*transText, 7, 7)
+		t.ordinalTanslations[key] = tarr
+	}
+
+	trans := &transText{
+		text:    text,
+		indexes: make([]int, 2, 2),
+	}
+
+	tarr[rule] = trans
+
+	idx := strings.Index(text, paramZero)
+	if idx == -1 {
+		tarr[rule] = nil
+		return &ErrOrdinalTranslation{text: fmt.Sprintf("error: parameter '%s' not found, may want to use 'Add' instead of 'AddOrdinal'. locale: '%s' key: '%v' text: '%s'", paramZero, t.Locale(), key, text)}
+	}
+
+	trans.indexes[0] = idx
+	trans.indexes[1] = idx + len(paramZero)
+
+	return nil
+}
+
+// AddRange adds a range plural translation for a particular language/locale
+// {0} and {1} are the only replacement types accepted and only these are accepted.
+// eg. in locale 'nl' one: '{0}-{1} day left' other: '{0}-{1} days left'
+func (t *translator) AddRange(key interface{}, text string, rule locales.PluralRule, override bool) error {
+
+	var verified bool
+
+	// verify plural rule exists for locale
+	for _, pr := range t.PluralsRange() {
+		if pr == rule {
+			verified = true
+			break
+		}
+	}
+
+	if !verified {
+		return &ErrRangeTranslation{text: fmt.Sprintf("error: range plural rule '%s' does not exist for locale '%s' key: '%v' text: '%s'", rule, t.Locale(), key, text)}
+	}
+
+	tarr, ok := t.rangeTanslations[key]
+	if ok {
+		// verify not adding a conflicting record
+		if len(tarr) > 0 && tarr[rule] != nil && !override {
+			return &ErrConflictingTranslation{locale: t.Locale(), key: key, rule: rule, text: text}
+		}
+
+	} else {
+		tarr = make([]*transText, 7, 7)
+		t.rangeTanslations[key] = tarr
+	}
+
+	trans := &transText{
+		text:    text,
+		indexes: make([]int, 4, 4),
+	}
+
+	tarr[rule] = trans
+
+	idx := strings.Index(text, paramZero)
+	if idx == -1 {
+		tarr[rule] = nil
+		return &ErrRangeTranslation{text: fmt.Sprintf("error: parameter '%s' not found, are you sure you're adding a Range Translation? locale: '%s' key: '%v' text: '%s'", paramZero, t.Locale(), key, text)}
+	}
+
+	trans.indexes[0] = idx
+	trans.indexes[1] = idx + len(paramZero)
+
+	idx = strings.Index(text, paramOne)
+	if idx == -1 {
+		tarr[rule] = nil
+		return &ErrRangeTranslation{text: fmt.Sprintf("error: parameter '%s' not found, a Range Translation requires two parameters. locale: '%s' key: '%v' text: '%s'", paramOne, t.Locale(), key, text)}
+	}
+
+	trans.indexes[2] = idx
+	trans.indexes[3] = idx + len(paramOne)
+
+	return nil
+}
+
+// T creates the translation for the locale given the 'key' and params passed in
+func (t *translator) T(key interface{}, params ...string) (string, error) {
+
+	trans, ok := t.translations[key]
+	if !ok {
+		return unknownTranslation, ErrUnknowTranslation
+	}
+
+	b := make([]byte, 0, 64)
+
+	var start, end, count int
+
+	for i := 0; i < len(trans.indexes); i++ {
+		end = trans.indexes[i]
+		b = append(b, trans.text[start:end]...)
+		b = append(b, params[count]...)
+		i++
+		start = trans.indexes[i]
+		count++
+	}
+
+	b = append(b, trans.text[start:]...)
+
+	return string(b), nil
+}
+
+// C creates the cardinal translation for the locale given the 'key', 'num' and 'digit' arguments and param passed in
+func (t *translator) C(key interface{}, num float64, digits uint64, param string) (string, error) {
+
+	tarr, ok := t.cardinalTanslations[key]
+	if !ok {
+		return unknownTranslation, ErrUnknowTranslation
+	}
+
+	rule := t.CardinalPluralRule(num, digits)
+
+	trans := tarr[rule]
+
+	b := make([]byte, 0, 64)
+	b = append(b, trans.text[:trans.indexes[0]]...)
+	b = append(b, param...)
+	b = append(b, trans.text[trans.indexes[1]:]...)
+
+	return string(b), nil
+}
+
+// O creates the ordinal translation for the locale given the 'key', 'num' and 'digit' arguments and param passed in
+func (t *translator) O(key interface{}, num float64, digits uint64, param string) (string, error) {
+
+	tarr, ok := t.ordinalTanslations[key]
+	if !ok {
+		return unknownTranslation, ErrUnknowTranslation
+	}
+
+	rule := t.OrdinalPluralRule(num, digits)
+
+	trans := tarr[rule]
+
+	b := make([]byte, 0, 64)
+	b = append(b, trans.text[:trans.indexes[0]]...)
+	b = append(b, param...)
+	b = append(b, trans.text[trans.indexes[1]:]...)
+
+	return string(b), nil
+}
+
+// R creates the range translation for the locale given the 'key', 'num1', 'digit1', 'num2' and 'digit2' arguments
+// and 'param1' and 'param2' passed in
+func (t *translator) R(key interface{}, num1 float64, digits1 uint64, num2 float64, digits2 uint64, param1, param2 string) (string, error) {
+
+	tarr, ok := t.rangeTanslations[key]
+	if !ok {
+		return unknownTranslation, ErrUnknowTranslation
+	}
+
+	rule := t.RangePluralRule(num1, digits1, num2, digits2)
+
+	trans := tarr[rule]
+
+	b := make([]byte, 0, 64)
+	b = append(b, trans.text[:trans.indexes[0]]...)
+	b = append(b, param1...)
+	b = append(b, trans.text[trans.indexes[1]:trans.indexes[2]]...)
+	b = append(b, param2...)
+	b = append(b, trans.text[trans.indexes[3]:]...)
+
+	return string(b), nil
+}
+
+// VerifyTranslations checks to ensures that no plural rules have been
+// missed within the translations.
+func (t *translator) VerifyTranslations() error {
+
+	for k, v := range t.cardinalTanslations {
+
+		for _, rule := range t.PluralsCardinal() {
+
+			if v[rule] == nil {
+				return &ErrMissingPluralTranslation{locale: t.Locale(), translationType: "plural", rule: rule, key: k}
+			}
+		}
+	}
+
+	for k, v := range t.ordinalTanslations {
+
+		for _, rule := range t.PluralsOrdinal() {
+
+			if v[rule] == nil {
+				return &ErrMissingPluralTranslation{locale: t.Locale(), translationType: "ordinal", rule: rule, key: k}
+			}
+		}
+	}
+
+	for k, v := range t.rangeTanslations {
+
+		for _, rule := range t.PluralsRange() {
+
+			if v[rule] == nil {
+				return &ErrMissingPluralTranslation{locale: t.Locale(), translationType: "range", rule: rule, key: k}
+			}
+		}
+	}
+
+	return nil
+}
--- a/backend/vendor/github.com/go-playground/universal-translator/universal_translator.go
+++ b/backend/vendor/github.com/go-playground/universal-translator/universal_translator.go
@@ -0,0 +1,113 @@
+package ut
+
+import (
+	"strings"
+
+	"github.com/go-playground/locales"
+)
+
+// UniversalTranslator holds all locale & translation data
+type UniversalTranslator struct {
+	translators map[string]Translator
+	fallback    Translator
+}
+
+// New returns a new UniversalTranslator instance set with
+// the fallback locale and locales it should support
+func New(fallback locales.Translator, supportedLocales ...locales.Translator) *UniversalTranslator {
+
+	t := &UniversalTranslator{
+		translators: make(map[string]Translator),
+	}
+
+	for _, v := range supportedLocales {
+
+		trans := newTranslator(v)
+		t.translators[strings.ToLower(trans.Locale())] = trans
+
+		if fallback.Locale() == v.Locale() {
+			t.fallback = trans
+		}
+	}
+
+	if t.fallback == nil && fallback != nil {
+		t.fallback = newTranslator(fallback)
+	}
+
+	return t
+}
+
+// FindTranslator trys to find a Translator based on an array of locales
+// and returns the first one it can find, otherwise returns the
+// fallback translator.
+func (t *UniversalTranslator) FindTranslator(locales ...string) (trans Translator, found bool) {
+
+	for _, locale := range locales {
+
+		if trans, found = t.translators[strings.ToLower(locale)]; found {
+			return
+		}
+	}
+
+	return t.fallback, false
+}
+
+// GetTranslator returns the specified translator for the given locale,
+// or fallback if not found
+func (t *UniversalTranslator) GetTranslator(locale string) (trans Translator, found bool) {
+
+	if trans, found = t.translators[strings.ToLower(locale)]; found {
+		return
+	}
+
+	return t.fallback, false
+}
+
+// GetFallback returns the fallback locale
+func (t *UniversalTranslator) GetFallback() Translator {
+	return t.fallback
+}
+
+// AddTranslator adds the supplied translator, if it already exists the override param
+// will be checked and if false an error will be returned, otherwise the translator will be
+// overridden; if the fallback matches the supplied translator it will be overridden as well
+// NOTE: this is normally only used when translator is embedded within a library
+func (t *UniversalTranslator) AddTranslator(translator locales.Translator, override bool) error {
+
+	lc := strings.ToLower(translator.Locale())
+	_, ok := t.translators[lc]
+	if ok && !override {
+		return &ErrExistingTranslator{locale: translator.Locale()}
+	}
+
+	trans := newTranslator(translator)
+
+	if t.fallback.Locale() == translator.Locale() {
+
+		// because it's optional to have a fallback, I don't impose that limitation
+		// don't know why you wouldn't but...
+		if !override {
+			return &ErrExistingTranslator{locale: translator.Locale()}
+		}
+
+		t.fallback = trans
+	}
+
+	t.translators[lc] = trans
+
+	return nil
+}
+
+// VerifyTranslations runs through all locales and identifies any issues
+// eg. missing plural rules for a locale
+func (t *UniversalTranslator) VerifyTranslations() (err error) {
+
+	for _, trans := range t.translators {
+		err = trans.VerifyTranslations()
+		if err != nil {
+			return
+		}
+	}
+
+	return
+}
--- a/backend/vendor/github.com/gopherjs/gopherjs/LICENSE
+++ b/backend/vendor/github.com/gopherjs/gopherjs/LICENSE
@@ -0,0 +1,24 @@
+Copyright (c) 2013 Richard Musiol. All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+   * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+   * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
--- a/backend/vendor/github.com/gopherjs/gopherjs/js/js.go
+++ b/backend/vendor/github.com/gopherjs/gopherjs/js/js.go
@@ -0,0 +1,168 @@
+// Package js provides functions for interacting with native JavaScript APIs. Calls to these functions are treated specially by GopherJS and translated directly to their corresponding JavaScript syntax.
+//
+// Use MakeWrapper to expose methods to JavaScript. When passing values directly, the following type conversions are performed:
+//
+//  | Go type               | JavaScript type       | Conversions back to interface{} |
+//  | --------------------- | --------------------- | ------------------------------- |
+//  | bool                  | Boolean               | bool                            |
+//  | integers and floats   | Number                | float64                         |
+//  | string                | String                | string                          |
+//  | []int8                | Int8Array             | []int8                          |
+//  | []int16               | Int16Array            | []int16                         |
+//  | []int32, []int        | Int32Array            | []int                           |
+//  | []uint8               | Uint8Array            | []uint8                         |
+//  | []uint16              | Uint16Array           | []uint16                        |
+//  | []uint32, []uint      | Uint32Array           | []uint                          |
+//  | []float32             | Float32Array          | []float32                       |
+//  | []float64             | Float64Array          | []float64                       |
+//  | all other slices      | Array                 | []interface{}                   |
+//  | arrays                | see slice type        | see slice type                  |
+//  | functions             | Function              | func(...interface{}) *js.Object |
+//  | time.Time             | Date                  | time.Time                       |
+//  | -                     | instanceof Node       | *js.Object                      |
+//  | maps, structs         | instanceof Object     | map[string]interface{}          |
+//
+// Additionally, for a struct containing a *js.Object field, only the content of the field will be passed to JavaScript and vice versa.
+package js
+
+// Object is a container for a native JavaScript object. Calls to its methods are treated specially by GopherJS and translated directly to their JavaScript syntax. A nil pointer to Object is equal to JavaScript's "null". Object can not be used as a map key.
+type Object struct{ object *Object }
+
+// Get returns the object's property with the given key.
+func (o *Object) Get(key string) *Object { return o.object.Get(key) }
+
+// Set assigns the value to the object's property with the given key.
+func (o *Object) Set(key string, value interface{}) { o.object.Set(key, value) }
+
+// Delete removes the object's property with the given key.
+func (o *Object) Delete(key string) { o.object.Delete(key) }
+
+// Length returns the object's "length" property, converted to int.
+func (o *Object) Length() int { return o.object.Length() }
+
+// Index returns the i'th element of an array.
+func (o *Object) Index(i int) *Object { return o.object.Index(i) }
+
+// SetIndex sets the i'th element of an array.
+func (o *Object) SetIndex(i int, value interface{}) { o.object.SetIndex(i, value) }
+
+// Call calls the object's method with the given name.
+func (o *Object) Call(name string, args ...interface{}) *Object { return o.object.Call(name, args...) }
+
+// Invoke calls the object itself. This will fail if it is not a function.
+func (o *Object) Invoke(args ...interface{}) *Object { return o.object.Invoke(args...) }
+
+// New creates a new instance of this type object. This will fail if it not a function (constructor).
+func (o *Object) New(args ...interface{}) *Object { return o.object.New(args...) }
+
+// Bool returns the object converted to bool according to JavaScript type conversions.
+func (o *Object) Bool() bool { return o.object.Bool() }
+
+// String returns the object converted to string according to JavaScript type conversions.
+func (o *Object) String() string { return o.object.String() }
+
+// Int returns the object converted to int according to JavaScript type conversions (parseInt).
+func (o *Object) Int() int { return o.object.Int() }
+
+// Int64 returns the object converted to int64 according to JavaScript type conversions (parseInt).
+func (o *Object) Int64() int64 { return o.object.Int64() }
+
+// Uint64 returns the object converted to uint64 according to JavaScript type conversions (parseInt).
+func (o *Object) Uint64() uint64 { return o.object.Uint64() }
+
+// Float returns the object converted to float64 according to JavaScript type conversions (parseFloat).
+func (o *Object) Float() float64 { return o.object.Float() }
+
+// Interface returns the object converted to interface{}. See table in package comment for details.
+func (o *Object) Interface() interface{} { return o.object.Interface() }
+
+// Unsafe returns the object as an uintptr, which can be converted via unsafe.Pointer. Not intended for public use.
+func (o *Object) Unsafe() uintptr { return o.object.Unsafe() }
+
+// Error encapsulates JavaScript errors. Those are turned into a Go panic and may be recovered, giving an *Error that holds the JavaScript error object.
+type Error struct {
+	*Object
+}
+
+// Error returns the message of the encapsulated JavaScript error object.
+func (err *Error) Error() string {
+	return "JavaScript error: " + err.Get("message").String()
+}
+
+// Stack returns the stack property of the encapsulated JavaScript error object.
+func (err *Error) Stack() string {
+	return err.Get("stack").String()
+}
+
+// Global gives JavaScript's global object ("window" for browsers and "GLOBAL" for Node.js).
+var Global *Object
+
+// Module gives the value of the "module" variable set by Node.js. Hint: Set a module export with 'js.Module.Get("exports").Set("exportName", ...)'.
+var Module *Object
+
+// Undefined gives the JavaScript value "undefined".
+var Undefined *Object
+
+// Debugger gets compiled to JavaScript's "debugger;" statement.
+func Debugger() {}
+
+// InternalObject returns the internal JavaScript object that represents i. Not intended for public use.
+func InternalObject(i interface{}) *Object {
+	return nil
+}
+
+// MakeFunc wraps a function and gives access to the values of JavaScript's "this" and "arguments" keywords.
+func MakeFunc(fn func(this *Object, arguments []*Object) interface{}) *Object {
+	return Global.Call("$makeFunc", InternalObject(fn))
+}
+
+// Keys returns the keys of the given JavaScript object.
+func Keys(o *Object) []string {
+	if o == nil || o == Undefined {
+		return nil
+	}
+	a := Global.Get("Object").Call("keys", o)
+	s := make([]string, a.Length())
+	for i := 0; i < a.Length(); i++ {
+		s[i] = a.Index(i).String()
+	}
+	return s
+}
+
+// MakeWrapper creates a JavaScript object which has wrappers for the exported methods of i. Use explicit getter and setter methods to expose struct fields to JavaScript.
+func MakeWrapper(i interface{}) *Object {
+	v := InternalObject(i)
+	o := Global.Get("Object").New()
+	o.Set("__internal_object__", v)
+	methods := v.Get("constructor").Get("methods")
+	for i := 0; i < methods.Length(); i++ {
+		m := methods.Index(i)
+		if m.Get("pkg").String() != "" { // not exported
+			continue
+		}
+		o.Set(m.Get("name").String(), func(args ...*Object) *Object {
+			return Global.Call("$externalizeFunction", v.Get(m.Get("prop").String()), m.Get("typ"), true).Call("apply", v, args)
+		})
+	}
+	return o
+}
+
+// NewArrayBuffer creates a JavaScript ArrayBuffer from a byte slice.
+func NewArrayBuffer(b []byte) *Object {
+	slice := InternalObject(b)
+	offset := slice.Get("$offset").Int()
+	length := slice.Get("$length").Int()
+	return slice.Get("$array").Get("buffer").Call("slice", offset, offset+length)
+}
+
+// M is a simple map type. It is intended as a shorthand for JavaScript objects (before conversion).
+type M map[string]interface{}
+
+// S is a simple slice type. It is intended as a shorthand for JavaScript arrays (before conversion).
+type S []interface{}
+
+func init() {
+	// avoid dead code elimination
+	e := Error{}
+	_ = e
+}
--- a/backend/vendor/github.com/imroc/req/LICENSE
+++ b/backend/vendor/github.com/imroc/req/LICENSE
@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "{}"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright {yyyy} {name of copyright owner}
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
--- a/backend/vendor/github.com/imroc/req/README.md
+++ b/backend/vendor/github.com/imroc/req/README.md
@@ -0,0 +1,302 @@
+# req
+[![GoDoc](https://godoc.org/github.com/imroc/req?status.svg)](https://godoc.org/github.com/imroc/req)
+
+A golang http request library for humans
+
+
+
+Features
+========
+
+- Light weight
+- Simple
+- Easy play with JSON and XML
+- Easy for debug and logging
+- Easy file uploads and downloads
+- Easy manage cookie
+- Easy set up proxy
+- Easy set timeout
+- Easy customize http client
+
+
+Document
+========
+[中文](doc/README_cn.md)
+
+
+Install
+=======
+``` sh
+go get github.com/imroc/req
+```
+
+Overview
+=======
+`req` implements a friendly API over Go's existing `net/http` library.  
+  
+`Req` and `Resp` are two most important struct, you can think of `Req` as a client that initiate HTTP requests, `Resp` as a information container for the request and response. They all provide simple and convenient APIs that allows you to do a lot of things.
+``` go
+func (r *Req) Post(url string, v ...interface{}) (*Resp, error)
+```  
+
+In most cases, only url is required, others are optional, like headers, params, files or body etc.
+
+There is a default `Req` object, all of its' public methods  are wrapped by the `req` package, so you can also think of `req` package as a `Req` object
+``` go
+// use Req object to initiate requests.
+r := req.New()
+r.Get(url)
+
+// use req package to initiate request.
+req.Get(url)
+```
+You can use `req.New()` to create lots of `*Req` as client with independent configuration
+
+Examples
+=======
+[Basic](#Basic)  
+[Set Header](#Set-Header)  
+[Set Param](#Set-Param)  
+[Set Body](#Set-Body)  
+[Debug](#Debug)  
+[Output Format](#Format)  
+[ToJSON & ToXML](#ToJSON-ToXML)  
+[Get *http.Response](#Response)  
+[Upload](#Upload)  
+[Download](#Download)  
+[Cookie](#Cookie)  
+[Set Timeout](#Set-Timeout)  
+[Set Proxy](#Set-Proxy)  
+[Customize Client](#Customize-Client)  
+
+## <a name="Basic">Basic</a>
+``` go
+header := req.Header{
+	"Accept":        "application/json",
+	"Authorization": "Basic YWRtaW46YWRtaW4=",
+}
+param := req.Param{
+	"name": "imroc",
+	"cmd":  "add",
+}
+// only url is required, others are optional.
+r, err = req.Post("http://foo.bar/api", header, param)
+if err != nil {
+	log.Fatal(err)
+}
+r.ToJSON(&foo)       // response => struct/map
+log.Printf("%+v", r) // print info (try it, you may surprise) 
+```
+
+## <a name="Set-Header">Set Header</a>
+Use `req.Header` (it is actually a `map[string]string`)
+``` go
+authHeader := req.Header{
+	"Accept":        "application/json",
+	"Authorization": "Basic YWRtaW46YWRtaW4=",
+}
+req.Get("https://www.baidu.com", authHeader, req.Header{"User-Agent": "V1.1"})
+```
+use `http.Header`
+``` go
+header := make(http.Header)
+header.Set("Accept", "application/json")
+req.Get("https://www.baidu.com", header)
+```
+
+## <a name="Set-Param">Set Param</a>
+Use `req.Param` (it is actually a `map[string]interface{}`)
+``` go
+param := req.Param{
+	"id":  "imroc",
+	"pwd": "roc",
+}
+req.Get("http://foo.bar/api", param) // http://foo.bar/api?id=imroc&pwd=roc
+req.Post(url, param)                  // body => id=imroc&pwd=roc
+```
+use `req.QueryParam` force to append params to the url (it is also actually a `map[string]interface{}`)
+``` go
+req.Post("http://foo.bar/api", req.Param{"name": "roc", "age": "22"}, req.QueryParam{"access_token": "fedledGF9Hg9ehTU"})
+/*
+POST /api?access_token=fedledGF9Hg9ehTU HTTP/1.1
+Host: foo.bar
+User-Agent: Go-http-client/1.1
+Content-Length: 15
+Content-Type: application/x-www-form-urlencoded;charset=UTF-8
+Accept-Encoding: gzip
+
+age=22&name=roc
+*/
+```
+
+## <a name="Set-Body">Set Body</a>
+Put `string`, `[]byte` and `io.Reader` as body directly.
+``` go
+req.Post(url, "id=roc&cmd=query")
+```
+Put object as xml or json body (add `Content-Type` header automatically)
+``` go
+req.Post(url, req.BodyJSON(&foo))
+req.Post(url, req.BodyXML(&bar))
+```
+
+## <a name="Debug">Debug</a>
+Set global variable `req.Debug` to true, it will print detail infomation for every request.
+``` go
+req.Debug = true
+req.Post("http://localhost/test" "hi")
+```
+![post](doc/post.png)
+
+## <a name="Format">Output Format</a>
+You can use different kind of output format to log the request and response infomation in your log file in defferent scenarios. For example, use `%+v` output format in the development phase, it allows you to observe the details. Use `%v` or `%-v` output format in production phase, just log the information necessarily.  
+
+### `%+v` or `%+s`
+Output in detail
+``` go
+r, _ := req.Post(url, header, param)
+log.Printf("%+v", r) // output the same format as Debug is enabled
+```
+
+### `%v` or `%s`
+Output in simple way (default format)
+``` go
+r, _ := req.Get(url, param)
+log.Printf("%v\n", r) // GET http://foo.bar/api?name=roc&cmd=add {"code":"0","msg":"success"}
+log.Prinln(r)         // smae as above
+```
+
+### `%-v` or `%-s`
+Output in simple way and keep all in one line (request body or response body may have multiple lines, this format will replace `"\r"` or `"\n"` with `" "`, it's useful when doing some search in your log file)
+
+### Flag
+You can call `SetFlags` to control the output content, decide which pieces can be output.
+``` go
+const (
+	LreqHead  = 1 << iota // output request head (request line and request header)
+	LreqBody              // output request body
+	LrespHead             // output response head (response line and response header)
+	LrespBody             // output response body
+	Lcost                 // output time costed by the request
+	LstdFlags = LreqHead | LreqBody | LrespHead | LrespBody
+)
+```
+``` go
+req.SetFlags(req.LreqHead | req.LreqBody | req.LrespHead)
+```
+
+### Monitoring time consuming
+``` go
+req.SetFlags(req.LstdFlags | req.Lcost) // output format add time costed by request
+r,_ := req.Get(url)
+log.Println(r) // http://foo.bar/api 3.260802ms {"code":0 "msg":"success"}
+if r.Cost() > 3 * time.Second { // check cost
+	log.Println("WARN: slow request:", r)
+}
+```
+
+## <a name="ToJSON-ToXML">ToJSON & ToXML</a>
+``` go
+r, _ := req.Get(url)
+r.ToJSON(&foo)
+r, _ = req.Post(url, req.BodyXML(&bar))
+r.ToXML(&baz)
+```
+
+## <a name="Response">Get *http.Response</a>
+```go
+// func (r *Req) Response() *http.Response
+r, _ := req.Get(url)
+resp := r.Response()
+fmt.Println(resp.StatusCode)
+```
+
+## <a name="Upload">Upload</a>
+Use `req.File` to match files
+``` go
+req.Post(url, req.File("imroc.png"), req.File("/Users/roc/Pictures/*.png"))
+```
+Use `req.FileUpload` to fully control
+``` go
+file, _ := os.Open("imroc.png")
+req.Post(url, req.FileUpload{
+	File:      file,
+	FieldName: "file",       // FieldName is form field name
+	FileName:  "avatar.png", //Filename is the name of the file that you wish to upload. We use this to guess the mimetype as well as pass it onto the server
+})
+```
+Use `req.UploadProgress` to listen upload progress
+```go
+progress := func(current, total int64) {
+	fmt.Println(float32(current)/float32(total)*100, "%")
+}
+req.Post(url, req.File("/Users/roc/Pictures/*.png"), req.UploadProgress(progress))
+fmt.Println("upload complete")
+```
+
+## <a name="Download">Download</a>
+``` go
+r, _ := req.Get(url)
+r.ToFile("imroc.png")
+```
+Use `req.DownloadProgress` to listen download progress
+```go
+progress := func(current, total int64) {
+	fmt.Println(float32(current)/float32(total)*100, "%")
+}
+r, _ := req.Get(url, req.DownloadProgress(progress))
+r.ToFile("hello.mp4")
+fmt.Println("download complete")
+```
+
+## <a name="Cookie">Cookie</a>
+By default, the underlying `*http.Client` will manage your cookie(send cookie header to server automatically if server has set a cookie for you), you can disable it by calling this function :
+``` go
+req.EnableCookie(false)
+```
+and you can set cookie in request just using `*http.Cookie`
+``` go
+cookie := new(http.Cookie)
+// ......
+req.Get(url, cookie)
+```
+
+## <a name="Set-Timeout">Set Timeout</a>
+``` go
+req.SetTimeout(50 * time.Second)
+```
+
+## <a name="Set-Proxy">Set Proxy</a>
+By default, req use proxy from system environment if `http_proxy` or `https_proxy` is specified, you can set a custom proxy or disable it by set `nil`
+``` go
+req.SetProxy(func(r *http.Request) (*url.URL, error) {
+	if strings.Contains(r.URL.Hostname(), "google") {
+		return url.Parse("http://my.vpn.com:23456")
+	}
+	return nil, nil
+})
+```
+Set a simple proxy (use fixed proxy url for every request)
+``` go
+req.SetProxyUrl("http://my.proxy.com:23456")
+```
+
+## <a name="Customize-Client">Customize Client</a>
+Use `SetClient` to change the default underlying `*http.Client`
+``` go
+req.SetClient(client)
+```
+Specify independent http client for some requests
+``` go
+client := &http.Client{Timeout: 30 * time.Second}
+req.Get(url, client)
+```
+Change some properties of default client you want
+``` go
+req.Client().Jar, _ = cookiejar.New(nil)
+trans, _ := req.Client().Transport.(*http.Transport)
+trans.MaxIdleConns = 20
+trans.TLSHandshakeTimeout = 20 * time.Second
+trans.DisableKeepAlives = true
+trans.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+```
--- a/backend/vendor/github.com/imroc/req/dump.go
+++ b/backend/vendor/github.com/imroc/req/dump.go
@@ -0,0 +1,216 @@
+package req
+
+import (
+	"bufio"
+	"bytes"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net"
+	"net/http"
+	"net/http/httputil"
+	"net/url"
+	"strings"
+	"time"
+)
+
+// Debug enable debug mode if set to true
+var Debug bool
+
+// dumpConn is a net.Conn which writes to Writer and reads from Reader
+type dumpConn struct {
+	io.Writer
+	io.Reader
+}
+
+func (c *dumpConn) Close() error                       { return nil }
+func (c *dumpConn) LocalAddr() net.Addr                { return nil }
+func (c *dumpConn) RemoteAddr() net.Addr               { return nil }
+func (c *dumpConn) SetDeadline(t time.Time) error      { return nil }
+func (c *dumpConn) SetReadDeadline(t time.Time) error  { return nil }
+func (c *dumpConn) SetWriteDeadline(t time.Time) error { return nil }
+
+// delegateReader is a reader that delegates to another reader,
+// once it arrives on a channel.
+type delegateReader struct {
+	c chan io.Reader
+	r io.Reader // nil until received from c
+}
+
+func (r *delegateReader) Read(p []byte) (int, error) {
+	if r.r == nil {
+		r.r = <-r.c
+	}
+	return r.r.Read(p)
+}
+
+type dummyBody struct {
+	N   int
+	off int
+}
+
+func (d *dummyBody) Read(p []byte) (n int, err error) {
+	if d.N <= 0 {
+		err = io.EOF
+		return
+	}
+	left := d.N - d.off
+	if left <= 0 {
+		err = io.EOF
+		return
+	}
+
+	if l := len(p); l > 0 {
+		if l >= left {
+			n = left
+			err = io.EOF
+		} else {
+			n = l
+		}
+		d.off += n
+		for i := 0; i < n; i++ {
+			p[i] = '*'
+		}
+	}
+
+	return
+}
+
+func (d *dummyBody) Close() error {
+	return nil
+}
+
+type dumpBuffer struct {
+	bytes.Buffer
+}
+
+func (b *dumpBuffer) Write(p []byte) {
+	if b.Len() > 0 {
+		b.Buffer.WriteString("\r\n\r\n")
+	}
+	b.Buffer.Write(p)
+}
+
+func (b *dumpBuffer) WriteString(s string) {
+	b.Write([]byte(s))
+}
+
+func (r *Resp) dumpRequest(dump *dumpBuffer) {
+	head := r.r.flag&LreqHead != 0
+	body := r.r.flag&LreqBody != 0
+
+	if head {
+		r.dumpReqHead(dump)
+	}
+	if body {
+		if r.multipartHelper != nil {
+			dump.Write(r.multipartHelper.Dump())
+		} else if len(r.reqBody) > 0 {
+			dump.Write(r.reqBody)
+		}
+	}
+}
+
+func (r *Resp) dumpReqHead(dump *dumpBuffer) {
+	reqSend := new(http.Request)
+	*reqSend = *r.req
+	if reqSend.URL.Scheme == "https" {
+		reqSend.URL = new(url.URL)
+		*reqSend.URL = *r.req.URL
+		reqSend.URL.Scheme = "http"
+	}
+
+	if reqSend.ContentLength > 0 {
+		reqSend.Body = &dummyBody{N: int(reqSend.ContentLength)}
+	} else {
+		reqSend.Body = &dummyBody{N: 1}
+	}
+
+	// Use the actual Transport code to record what we would send
+	// on the wire, but not using TCP.  Use a Transport with a
+	// custom dialer that returns a fake net.Conn that waits
+	// for the full input (and recording it), and then responds
+	// with a dummy response.
+	var buf bytes.Buffer // records the output
+	pr, pw := io.Pipe()
+	defer pw.Close()
+	dr := &delegateReader{c: make(chan io.Reader)}
+
+	t := &http.Transport{
+		Dial: func(net, addr string) (net.Conn, error) {
+			return &dumpConn{io.MultiWriter(&buf, pw), dr}, nil
+		},
+	}
+	defer t.CloseIdleConnections()
+
+	client := new(http.Client)
+	*client = *r.client
+	client.Transport = t
+
+	// Wait for the request before replying with a dummy response:
+	go func() {
+		req, err := http.ReadRequest(bufio.NewReader(pr))
+		if err == nil {
+			// Ensure all the body is read; otherwise
+			// we'll get a partial dump.
+			io.Copy(ioutil.Discard, req.Body)
+			req.Body.Close()
+		}
+
+		dr.c <- strings.NewReader("HTTP/1.1 204 No Content\r\nConnection: close\r\n\r\n")
+		pr.Close()
+	}()
+
+	_, err := client.Do(reqSend)
+	if err != nil {
+		dump.WriteString(err.Error())
+	} else {
+		reqDump := buf.Bytes()
+		if i := bytes.Index(reqDump, []byte("\r\n\r\n")); i >= 0 {
+			reqDump = reqDump[:i]
+		}
+		dump.Write(reqDump)
+	}
+}
+
+func (r *Resp) dumpResponse(dump *dumpBuffer) {
+	head := r.r.flag&LrespHead != 0
+	body := r.r.flag&LrespBody != 0
+	if head {
+		respDump, err := httputil.DumpResponse(r.resp, false)
+		if err != nil {
+			dump.WriteString(err.Error())
+		} else {
+			if i := bytes.Index(respDump, []byte("\r\n\r\n")); i >= 0 {
+				respDump = respDump[:i]
+			}
+			dump.Write(respDump)
+		}
+	}
+	if body && len(r.Bytes()) > 0 {
+		dump.Write(r.Bytes())
+	}
+}
+
+// Cost return the time cost of the request
+func (r *Resp) Cost() time.Duration {
+	return r.cost
+}
+
+// Dump dump the request
+func (r *Resp) Dump() string {
+	dump := new(dumpBuffer)
+	if r.r.flag&Lcost != 0 {
+		dump.WriteString(fmt.Sprint(r.cost))
+	}
+	r.dumpRequest(dump)
+	l := dump.Len()
+	if l > 0 {
+		dump.WriteString("=================================")
+		l = dump.Len()
+	}
+
+	r.dumpResponse(dump)
+
+	return dump.String()
+}
--- a/backend/vendor/github.com/imroc/req/req.go
+++ b/backend/vendor/github.com/imroc/req/req.go
@@ -0,0 +1,688 @@
+package req
+
+import (
+	"bytes"
+	"compress/gzip"
+	"context"
+	"encoding/json"
+	"encoding/xml"
+	"errors"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"mime/multipart"
+	"net/http"
+	"net/textproto"
+	"net/url"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// default *Req
+var std = New()
+
+// flags to decide which part can be outputed
+const (
+	LreqHead  = 1 << iota // output request head (request line and request header)
+	LreqBody              // output request body
+	LrespHead             // output response head (response line and response header)
+	LrespBody             // output response body
+	Lcost                 // output time costed by the request
+	LstdFlags = LreqHead | LreqBody | LrespHead | LrespBody
+)
+
+// Header represents http request header
+type Header map[string]string
+
+func (h Header) Clone() Header {
+	if h == nil {
+		return nil
+	}
+	hh := Header{}
+	for k, v := range h {
+		hh[k] = v
+	}
+	return hh
+}
+
+// Param represents  http request param
+type Param map[string]interface{}
+
+// QueryParam is used to force append http request param to the uri
+type QueryParam map[string]interface{}
+
+// Host is used for set request's Host
+type Host string
+
+// FileUpload represents a file to upload
+type FileUpload struct {
+	// filename in multipart form.
+	FileName string
+	// form field name
+	FieldName string
+	// file to uplaod, required
+	File io.ReadCloser
+}
+
+type DownloadProgress func(current, total int64)
+
+type UploadProgress func(current, total int64)
+
+// File upload files matching the name pattern such as
+// /usr/*/bin/go* (assuming the Separator is '/')
+func File(patterns ...string) interface{} {
+	matches := []string{}
+	for _, pattern := range patterns {
+		m, err := filepath.Glob(pattern)
+		if err != nil {
+			return err
+		}
+		matches = append(matches, m...)
+	}
+	if len(matches) == 0 {
+		return errors.New("req: no file have been matched")
+	}
+	uploads := []FileUpload{}
+	for _, match := range matches {
+		if s, e := os.Stat(match); e != nil || s.IsDir() {
+			continue
+		}
+		file, _ := os.Open(match)
+		uploads = append(uploads, FileUpload{
+			File:      file,
+			FileName:  filepath.Base(match),
+			FieldName: "media",
+		})
+	}
+
+	return uploads
+}
+
+type bodyJson struct {
+	v interface{}
+}
+
+type bodyXml struct {
+	v interface{}
+}
+
+// BodyJSON make the object be encoded in json format and set it to the request body
+func BodyJSON(v interface{}) *bodyJson {
+	return &bodyJson{v: v}
+}
+
+// BodyXML make the object be encoded in xml format and set it to the request body
+func BodyXML(v interface{}) *bodyXml {
+	return &bodyXml{v: v}
+}
+
+// Req is a convenient client for initiating requests
+type Req struct {
+	client      *http.Client
+	jsonEncOpts *jsonEncOpts
+	xmlEncOpts  *xmlEncOpts
+	flag        int
+}
+
+// New create a new *Req
+func New() *Req {
+	return &Req{flag: LstdFlags}
+}
+
+type param struct {
+	url.Values
+}
+
+func (p *param) getValues() url.Values {
+	if p.Values == nil {
+		p.Values = make(url.Values)
+	}
+	return p.Values
+}
+
+func (p *param) Copy(pp param) {
+	if pp.Values == nil {
+		return
+	}
+	vs := p.getValues()
+	for key, values := range pp.Values {
+		for _, value := range values {
+			vs.Add(key, value)
+		}
+	}
+}
+func (p *param) Adds(m map[string]interface{}) {
+	if len(m) == 0 {
+		return
+	}
+	vs := p.getValues()
+	for k, v := range m {
+		vs.Add(k, fmt.Sprint(v))
+	}
+}
+
+func (p *param) Empty() bool {
+	return p.Values == nil
+}
+
+// Do execute a http request with sepecify method and url,
+// and it can also have some optional params, depending on your needs.
+func (r *Req) Do(method, rawurl string, vs ...interface{}) (resp *Resp, err error) {
+	if rawurl == "" {
+		return nil, errors.New("req: url not specified")
+	}
+	req := &http.Request{
+		Method:     method,
+		Header:     make(http.Header),
+		Proto:      "HTTP/1.1",
+		ProtoMajor: 1,
+		ProtoMinor: 1,
+	}
+	resp = &Resp{req: req, r: r}
+
+	var queryParam param
+	var formParam param
+	var uploads []FileUpload
+	var uploadProgress UploadProgress
+	var progress func(int64, int64)
+	var delayedFunc []func()
+	var lastFunc []func()
+
+	for _, v := range vs {
+		switch vv := v.(type) {
+		case Header:
+			for key, value := range vv {
+				req.Header.Add(key, value)
+			}
+		case http.Header:
+			for key, values := range vv {
+				for _, value := range values {
+					req.Header.Add(key, value)
+				}
+			}
+		case *bodyJson:
+			fn, err := setBodyJson(req, resp, r.jsonEncOpts, vv.v)
+			if err != nil {
+				return nil, err
+			}
+			delayedFunc = append(delayedFunc, fn)
+		case *bodyXml:
+			fn, err := setBodyXml(req, resp, r.xmlEncOpts, vv.v)
+			if err != nil {
+				return nil, err
+			}
+			delayedFunc = append(delayedFunc, fn)
+		case url.Values:
+			p := param{vv}
+			if method == "GET" || method == "HEAD" {
+				queryParam.Copy(p)
+			} else {
+				formParam.Copy(p)
+			}
+		case Param:
+			if method == "GET" || method == "HEAD" {
+				queryParam.Adds(vv)
+			} else {
+				formParam.Adds(vv)
+			}
+		case QueryParam:
+			queryParam.Adds(vv)
+		case string:
+			setBodyBytes(req, resp, []byte(vv))
+		case []byte:
+			setBodyBytes(req, resp, vv)
+		case bytes.Buffer:
+			setBodyBytes(req, resp, vv.Bytes())
+		case *http.Client:
+			resp.client = vv
+		case FileUpload:
+			uploads = append(uploads, vv)
+		case []FileUpload:
+			uploads = append(uploads, vv...)
+		case *http.Cookie:
+			req.AddCookie(vv)
+		case Host:
+			req.Host = string(vv)
+		case io.Reader:
+			fn := setBodyReader(req, resp, vv)
+			lastFunc = append(lastFunc, fn)
+		case UploadProgress:
+			uploadProgress = vv
+		case DownloadProgress:
+			resp.downloadProgress = vv
+		case func(int64, int64):
+			progress = vv
+		case context.Context:
+			req = req.WithContext(vv)
+			resp.req = req
+		case error:
+			return nil, vv
+		}
+	}
+
+	if length := req.Header.Get("Content-Length"); length != "" {
+		if l, err := strconv.ParseInt(length, 10, 64); err == nil {
+			req.ContentLength = l
+		}
+	}
+
+	if len(uploads) > 0 && (req.Method == "POST" || req.Method == "PUT") { // multipart
+		var up UploadProgress
+		if uploadProgress != nil {
+			up = uploadProgress
+		} else if progress != nil {
+			up = UploadProgress(progress)
+		}
+		multipartHelper := &multipartHelper{
+			form:           formParam.Values,
+			uploads:        uploads,
+			uploadProgress: up,
+		}
+		multipartHelper.Upload(req)
+		resp.multipartHelper = multipartHelper
+	} else {
+		if progress != nil {
+			resp.downloadProgress = DownloadProgress(progress)
+		}
+		if !formParam.Empty() {
+			if req.Body != nil {
+				queryParam.Copy(formParam)
+			} else {
+				setBodyBytes(req, resp, []byte(formParam.Encode()))
+				setContentType(req, "application/x-www-form-urlencoded; charset=UTF-8")
+			}
+		}
+	}
+
+	if !queryParam.Empty() {
+		paramStr := queryParam.Encode()
+		if strings.IndexByte(rawurl, '?') == -1 {
+			rawurl = rawurl + "?" + paramStr
+		} else {
+			rawurl = rawurl + "&" + paramStr
+		}
+	}
+
+	u, err := url.Parse(rawurl)
+	if err != nil {
+		return nil, err
+	}
+	req.URL = u
+
+	if host := req.Header.Get("Host"); host != "" {
+		req.Host = host
+	}
+
+	for _, fn := range delayedFunc {
+		fn()
+	}
+
+	if resp.client == nil {
+		resp.client = r.Client()
+	}
+
+	var response *http.Response
+	if r.flag&Lcost != 0 {
+		before := time.Now()
+		response, err = resp.client.Do(req)
+		after := time.Now()
+		resp.cost = after.Sub(before)
+	} else {
+		response, err = resp.client.Do(req)
+	}
+	if err != nil {
+		return nil, err
+	}
+
+	for _, fn := range lastFunc {
+		fn()
+	}
+
+	resp.resp = response
+
+	if _, ok := resp.client.Transport.(*http.Transport); ok && response.Header.Get("Content-Encoding") == "gzip" && req.Header.Get("Accept-Encoding") != "" {
+		body, err := gzip.NewReader(response.Body)
+		if err != nil {
+			return nil, err
+		}
+		response.Body = body
+	}
+
+	// output detail if Debug is enabled
+	if Debug {
+		fmt.Println(resp.Dump())
+	}
+	return
+}
+
+func setBodyBytes(req *http.Request, resp *Resp, data []byte) {
+	resp.reqBody = data
+	req.Body = ioutil.NopCloser(bytes.NewReader(data))
+	req.ContentLength = int64(len(data))
+}
+
+func setBodyJson(req *http.Request, resp *Resp, opts *jsonEncOpts, v interface{}) (func(), error) {
+	var data []byte
+	switch vv := v.(type) {
+	case string:
+		data = []byte(vv)
+	case []byte:
+		data = vv
+	case *bytes.Buffer:
+		data = vv.Bytes()
+	default:
+		if opts != nil {
+			var buf bytes.Buffer
+			enc := json.NewEncoder(&buf)
+			enc.SetIndent(opts.indentPrefix, opts.indentValue)
+			enc.SetEscapeHTML(opts.escapeHTML)
+			err := enc.Encode(v)
+			if err != nil {
+				return nil, err
+			}
+			data = buf.Bytes()
+		} else {
+			var err error
+			data, err = json.Marshal(v)
+			if err != nil {
+				return nil, err
+			}
+		}
+	}
+	setBodyBytes(req, resp, data)
+	delayedFunc := func() {
+		setContentType(req, "application/json; charset=UTF-8")
+	}
+	return delayedFunc, nil
+}
+
+func setBodyXml(req *http.Request, resp *Resp, opts *xmlEncOpts, v interface{}) (func(), error) {
+	var data []byte
+	switch vv := v.(type) {
+	case string:
+		data = []byte(vv)
+	case []byte:
+		data = vv
+	case *bytes.Buffer:
+		data = vv.Bytes()
+	default:
+		if opts != nil {
+			var buf bytes.Buffer
+			enc := xml.NewEncoder(&buf)
+			enc.Indent(opts.prefix, opts.indent)
+			err := enc.Encode(v)
+			if err != nil {
+				return nil, err
+			}
+			data = buf.Bytes()
+		} else {
+			var err error
+			data, err = xml.Marshal(v)
+			if err != nil {
+				return nil, err
+			}
+		}
+	}
+	setBodyBytes(req, resp, data)
+	delayedFunc := func() {
+		setContentType(req, "application/xml; charset=UTF-8")
+	}
+	return delayedFunc, nil
+}
+
+func setContentType(req *http.Request, contentType string) {
+	if req.Header.Get("Content-Type") == "" {
+		req.Header.Set("Content-Type", contentType)
+	}
+}
+
+func setBodyReader(req *http.Request, resp *Resp, rd io.Reader) func() {
+	var rc io.ReadCloser
+	switch r := rd.(type) {
+	case *os.File:
+		stat, err := r.Stat()
+		if err == nil {
+			req.ContentLength = stat.Size()
+		}
+		rc = r
+
+	case io.ReadCloser:
+		rc = r
+	default:
+		rc = ioutil.NopCloser(rd)
+	}
+	bw := &bodyWrapper{
+		ReadCloser: rc,
+		limit:      102400,
+	}
+	req.Body = bw
+	lastFunc := func() {
+		resp.reqBody = bw.buf.Bytes()
+	}
+	return lastFunc
+}
+
+type bodyWrapper struct {
+	io.ReadCloser
+	buf   bytes.Buffer
+	limit int
+}
+
+func (b *bodyWrapper) Read(p []byte) (n int, err error) {
+	n, err = b.ReadCloser.Read(p)
+	if left := b.limit - b.buf.Len(); left > 0 && n > 0 {
+		if n <= left {
+			b.buf.Write(p[:n])
+		} else {
+			b.buf.Write(p[:left])
+		}
+	}
+	return
+}
+
+type multipartHelper struct {
+	form           url.Values
+	uploads        []FileUpload
+	dump           []byte
+	uploadProgress UploadProgress
+}
+
+func (m *multipartHelper) Upload(req *http.Request) {
+	pr, pw := io.Pipe()
+	bodyWriter := multipart.NewWriter(pw)
+	go func() {
+		for key, values := range m.form {
+			for _, value := range values {
+				bodyWriter.WriteField(key, value)
+			}
+		}
+		var upload func(io.Writer, io.Reader) error
+		if m.uploadProgress != nil {
+			var total int64
+			for _, up := range m.uploads {
+				if file, ok := up.File.(*os.File); ok {
+					stat, err := file.Stat()
+					if err != nil {
+						continue
+					}
+					total += stat.Size()
+				}
+			}
+			var current int64
+			buf := make([]byte, 1024)
+			var lastTime time.Time
+			upload = func(w io.Writer, r io.Reader) error {
+				for {
+					n, err := r.Read(buf)
+					if n > 0 {
+						_, _err := w.Write(buf[:n])
+						if _err != nil {
+							return _err
+						}
+						current += int64(n)
+						if now := time.Now(); now.Sub(lastTime) > 200*time.Millisecond {
+							lastTime = now
+							m.uploadProgress(current, total)
+						}
+					}
+					if err == io.EOF {
+						return nil
+					}
+					if err != nil {
+						return err
+					}
+				}
+			}
+		}
+
+		i := 0
+		for _, up := range m.uploads {
+			if up.FieldName == "" {
+				i++
+				up.FieldName = "file" + strconv.Itoa(i)
+			}
+			fileWriter, err := bodyWriter.CreateFormFile(up.FieldName, up.FileName)
+			if err != nil {
+				continue
+			}
+			//iocopy
+			if upload == nil {
+				io.Copy(fileWriter, up.File)
+			} else {
+				if _, ok := up.File.(*os.File); ok {
+					upload(fileWriter, up.File)
+				} else {
+					io.Copy(fileWriter, up.File)
+				}
+			}
+			up.File.Close()
+		}
+		bodyWriter.Close()
+		pw.Close()
+	}()
+	req.Header.Set("Content-Type", bodyWriter.FormDataContentType())
+	req.Body = ioutil.NopCloser(pr)
+}
+
+func (m *multipartHelper) Dump() []byte {
+	if m.dump != nil {
+		return m.dump
+	}
+	var buf bytes.Buffer
+	bodyWriter := multipart.NewWriter(&buf)
+	for key, values := range m.form {
+		for _, value := range values {
+			m.writeField(bodyWriter, key, value)
+		}
+	}
+	for _, up := range m.uploads {
+		m.writeFile(bodyWriter, up.FieldName, up.FileName)
+	}
+	bodyWriter.Close()
+	m.dump = buf.Bytes()
+	return m.dump
+}
+
+func (m *multipartHelper) writeField(w *multipart.Writer, fieldname, value string) error {
+	h := make(textproto.MIMEHeader)
+	h.Set("Content-Disposition",
+		fmt.Sprintf(`form-data; name="%s"`, fieldname))
+	p, err := w.CreatePart(h)
+	if err != nil {
+		return err
+	}
+	_, err = p.Write([]byte(value))
+	return err
+}
+
+func (m *multipartHelper) writeFile(w *multipart.Writer, fieldname, filename string) error {
+	h := make(textproto.MIMEHeader)
+	h.Set("Content-Disposition",
+		fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
+			fieldname, filename))
+	h.Set("Content-Type", "application/octet-stream")
+	p, err := w.CreatePart(h)
+	if err != nil {
+		return err
+	}
+	_, err = p.Write([]byte("******"))
+	return err
+}
+
+// Get execute a http GET request
+func (r *Req) Get(url string, v ...interface{}) (*Resp, error) {
+	return r.Do("GET", url, v...)
+}
+
+// Post execute a http POST request
+func (r *Req) Post(url string, v ...interface{}) (*Resp, error) {
+	return r.Do("POST", url, v...)
+}
+
+// Put execute a http PUT request
+func (r *Req) Put(url string, v ...interface{}) (*Resp, error) {
+	return r.Do("PUT", url, v...)
+}
+
+// Patch execute a http PATCH request
+func (r *Req) Patch(url string, v ...interface{}) (*Resp, error) {
+	return r.Do("PATCH", url, v...)
+}
+
+// Delete execute a http DELETE request
+func (r *Req) Delete(url string, v ...interface{}) (*Resp, error) {
+	return r.Do("DELETE", url, v...)
+}
+
+// Head execute a http HEAD request
+func (r *Req) Head(url string, v ...interface{}) (*Resp, error) {
+	return r.Do("HEAD", url, v...)
+}
+
+// Options execute a http OPTIONS request
+func (r *Req) Options(url string, v ...interface{}) (*Resp, error) {
+	return r.Do("OPTIONS", url, v...)
+}
+
+// Get execute a http GET request
+func Get(url string, v ...interface{}) (*Resp, error) {
+	return std.Get(url, v...)
+}
+
+// Post execute a http POST request
+func Post(url string, v ...interface{}) (*Resp, error) {
+	return std.Post(url, v...)
+}
+
+// Put execute a http PUT request
+func Put(url string, v ...interface{}) (*Resp, error) {
+	return std.Put(url, v...)
+}
+
+// Head execute a http HEAD request
+func Head(url string, v ...interface{}) (*Resp, error) {
+	return std.Head(url, v...)
+}
+
+// Options execute a http OPTIONS request
+func Options(url string, v ...interface{}) (*Resp, error) {
+	return std.Options(url, v...)
+}
+
+// Delete execute a http DELETE request
+func Delete(url string, v ...interface{}) (*Resp, error) {
+	return std.Delete(url, v...)
+}
+
+// Patch execute a http PATCH request
+func Patch(url string, v ...interface{}) (*Resp, error) {
+	return std.Patch(url, v...)
+}
+
+// Do execute request.
+func Do(method, url string, v ...interface{}) (*Resp, error) {
+	return std.Do(method, url, v...)
+}
--- a/backend/vendor/github.com/imroc/req/resp.go
+++ b/backend/vendor/github.com/imroc/req/resp.go
@@ -0,0 +1,215 @@
+package req
+
+import (
+	"encoding/json"
+	"encoding/xml"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"os"
+	"regexp"
+	"time"
+)
+
+// Resp represents a request with it's response
+type Resp struct {
+	r      *Req
+	req    *http.Request
+	resp   *http.Response
+	client *http.Client
+	cost   time.Duration
+	*multipartHelper
+	reqBody          []byte
+	respBody         []byte
+	downloadProgress DownloadProgress
+	err              error // delayed error
+}
+
+// Request returns *http.Request
+func (r *Resp) Request() *http.Request {
+	return r.req
+}
+
+// Response returns *http.Response
+func (r *Resp) Response() *http.Response {
+	return r.resp
+}
+
+// Bytes returns response body as []byte
+func (r *Resp) Bytes() []byte {
+	data, _ := r.ToBytes()
+	return data
+}
+
+// ToBytes returns response body as []byte,
+// return error if error happend when reading
+// the response body
+func (r *Resp) ToBytes() ([]byte, error) {
+	if r.err != nil {
+		return nil, r.err
+	}
+	if r.respBody != nil {
+		return r.respBody, nil
+	}
+	defer r.resp.Body.Close()
+	respBody, err := ioutil.ReadAll(r.resp.Body)
+	if err != nil {
+		r.err = err
+		return nil, err
+	}
+	r.respBody = respBody
+	return r.respBody, nil
+}
+
+// String returns response body as string
+func (r *Resp) String() string {
+	data, _ := r.ToBytes()
+	return string(data)
+}
+
+// ToString returns response body as string,
+// return error if error happend when reading
+// the response body
+func (r *Resp) ToString() (string, error) {
+	data, err := r.ToBytes()
+	return string(data), err
+}
+
+// ToJSON convert json response body to struct or map
+func (r *Resp) ToJSON(v interface{}) error {
+	data, err := r.ToBytes()
+	if err != nil {
+		return err
+	}
+	return json.Unmarshal(data, v)
+}
+
+// ToXML convert xml response body to struct or map
+func (r *Resp) ToXML(v interface{}) error {
+	data, err := r.ToBytes()
+	if err != nil {
+		return err
+	}
+	return xml.Unmarshal(data, v)
+}
+
+// ToFile download the response body to file with optional download callback
+func (r *Resp) ToFile(name string) error {
+	//TODO set name to the suffix of url path if name == ""
+	file, err := os.Create(name)
+	if err != nil {
+		return err
+	}
+	defer file.Close()
+
+	if r.respBody != nil {
+		_, err = file.Write(r.respBody)
+		return err
+	}
+
+	if r.downloadProgress != nil && r.resp.ContentLength > 0 {
+		return r.download(file)
+	}
+
+	defer r.resp.Body.Close()
+	_, err = io.Copy(file, r.resp.Body)
+	return err
+}
+
+func (r *Resp) download(file *os.File) error {
+	p := make([]byte, 1024)
+	b := r.resp.Body
+	defer b.Close()
+	total := r.resp.ContentLength
+	var current int64
+	var lastTime time.Time
+	for {
+		l, err := b.Read(p)
+		if l > 0 {
+			_, _err := file.Write(p[:l])
+			if _err != nil {
+				return _err
+			}
+			current += int64(l)
+			if now := time.Now(); now.Sub(lastTime) > 200*time.Millisecond {
+				lastTime = now
+				r.downloadProgress(current, total)
+			}
+		}
+		if err != nil {
+			if err == io.EOF {
+				return nil
+			}
+			return err
+		}
+	}
+}
+
+var regNewline = regexp.MustCompile(`\n|\r`)
+
+func (r *Resp) autoFormat(s fmt.State) {
+	req := r.req
+	if r.r.flag&Lcost != 0 {
+		fmt.Fprint(s, req.Method, " ", req.URL.String(), " ", r.cost)
+	} else {
+		fmt.Fprint(s, req.Method, " ", req.URL.String())
+	}
+
+	// test if it is should be outputed pretty
+	var pretty bool
+	var parts []string
+	addPart := func(part string) {
+		if part == "" {
+			return
+		}
+		parts = append(parts, part)
+		if !pretty && regNewline.MatchString(part) {
+			pretty = true
+		}
+	}
+	if r.r.flag&LreqBody != 0 { // request body
+		addPart(string(r.reqBody))
+	}
+	if r.r.flag&LrespBody != 0 { // response body
+		addPart(r.String())
+	}
+
+	for _, part := range parts {
+		if pretty {
+			fmt.Fprint(s, "\n")
+		}
+		fmt.Fprint(s, " ", part)
+	}
+}
+
+func (r *Resp) miniFormat(s fmt.State) {
+	req := r.req
+	if r.r.flag&Lcost != 0 {
+		fmt.Fprint(s, req.Method, " ", req.URL.String(), " ", r.cost)
+	} else {
+		fmt.Fprint(s, req.Method, " ", req.URL.String())
+	}
+	if r.r.flag&LreqBody != 0 && len(r.reqBody) > 0 { // request body
+		str := regNewline.ReplaceAllString(string(r.reqBody), " ")
+		fmt.Fprint(s, " ", str)
+	}
+	if r.r.flag&LrespBody != 0 && r.String() != "" { // response body
+		str := regNewline.ReplaceAllString(r.String(), " ")
+		fmt.Fprint(s, " ", str)
+	}
+}
+
+// Format fort the response
+func (r *Resp) Format(s fmt.State, verb rune) {
+	if r == nil || r.req == nil {
+		return
+	}
+	if s.Flag('+') { // include header and format pretty.
+		fmt.Fprint(s, r.Dump())
+	} else if s.Flag('-') { // keep all informations in one line.
+		r.miniFormat(s)
+	} else { // auto
+		r.autoFormat(s)
+	}
+}
--- a/backend/vendor/github.com/imroc/req/setting.go
+++ b/backend/vendor/github.com/imroc/req/setting.go
@@ -0,0 +1,236 @@
+package req
+
+import (
+	"crypto/tls"
+	"errors"
+	"net"
+	"net/http"
+	"net/http/cookiejar"
+	"net/url"
+	"time"
+)
+
+// create a default client
+func newClient() *http.Client {
+	jar, _ := cookiejar.New(nil)
+	transport := &http.Transport{
+		Proxy: http.ProxyFromEnvironment,
+		DialContext: (&net.Dialer{
+			Timeout:   30 * time.Second,
+			KeepAlive: 30 * time.Second,
+			DualStack: true,
+		}).DialContext,
+		MaxIdleConns:          100,
+		IdleConnTimeout:       90 * time.Second,
+		TLSHandshakeTimeout:   10 * time.Second,
+		ExpectContinueTimeout: 1 * time.Second,
+	}
+	return &http.Client{
+		Jar:       jar,
+		Transport: transport,
+		Timeout:   2 * time.Minute,
+	}
+}
+
+// Client return the default underlying http client
+func (r *Req) Client() *http.Client {
+	if r.client == nil {
+		r.client = newClient()
+	}
+	return r.client
+}
+
+// Client return the default underlying http client
+func Client() *http.Client {
+	return std.Client()
+}
+
+// SetClient sets the underlying http.Client.
+func (r *Req) SetClient(client *http.Client) {
+	r.client = client // use default if client == nil
+}
+
+// SetClient sets the default http.Client for requests.
+func SetClient(client *http.Client) {
+	std.SetClient(client)
+}
+
+// SetFlags control display format of *Resp
+func (r *Req) SetFlags(flags int) {
+	r.flag = flags
+}
+
+// SetFlags control display format of *Resp
+func SetFlags(flags int) {
+	std.SetFlags(flags)
+}
+
+// Flags return output format for the *Resp
+func (r *Req) Flags() int {
+	return r.flag
+}
+
+// Flags return output format for the *Resp
+func Flags() int {
+	return std.Flags()
+}
+
+func (r *Req) getTransport() *http.Transport {
+	trans, _ := r.Client().Transport.(*http.Transport)
+	return trans
+}
+
+// EnableInsecureTLS allows insecure https
+func (r *Req) EnableInsecureTLS(enable bool) {
+	trans := r.getTransport()
+	if trans == nil {
+		return
+	}
+	if trans.TLSClientConfig == nil {
+		trans.TLSClientConfig = &tls.Config{}
+	}
+	trans.TLSClientConfig.InsecureSkipVerify = enable
+}
+
+func EnableInsecureTLS(enable bool) {
+	std.EnableInsecureTLS(enable)
+}
+
+// EnableCookieenable or disable cookie manager
+func (r *Req) EnableCookie(enable bool) {
+	if enable {
+		jar, _ := cookiejar.New(nil)
+		r.Client().Jar = jar
+	} else {
+		r.Client().Jar = nil
+	}
+}
+
+// EnableCookieenable or disable cookie manager
+func EnableCookie(enable bool) {
+	std.EnableCookie(enable)
+}
+
+// SetTimeout sets the timeout for every request
+func (r *Req) SetTimeout(d time.Duration) {
+	r.Client().Timeout = d
+}
+
+// SetTimeout sets the timeout for every request
+func SetTimeout(d time.Duration) {
+	std.SetTimeout(d)
+}
+
+// SetProxyUrl set the simple proxy with fixed proxy url
+func (r *Req) SetProxyUrl(rawurl string) error {
+	trans := r.getTransport()
+	if trans == nil {
+		return errors.New("req: no transport")
+	}
+	u, err := url.Parse(rawurl)
+	if err != nil {
+		return err
+	}
+	trans.Proxy = http.ProxyURL(u)
+	return nil
+}
+
+// SetProxyUrl set the simple proxy with fixed proxy url
+func SetProxyUrl(rawurl string) error {
+	return std.SetProxyUrl(rawurl)
+}
+
+// SetProxy sets the proxy for every request
+func (r *Req) SetProxy(proxy func(*http.Request) (*url.URL, error)) error {
+	trans := r.getTransport()
+	if trans == nil {
+		return errors.New("req: no transport")
+	}
+	trans.Proxy = proxy
+	return nil
+}
+
+// SetProxy sets the proxy for every request
+func SetProxy(proxy func(*http.Request) (*url.URL, error)) error {
+	return std.SetProxy(proxy)
+}
+
+type jsonEncOpts struct {
+	indentPrefix string
+	indentValue  string
+	escapeHTML   bool
+}
+
+func (r *Req) getJSONEncOpts() *jsonEncOpts {
+	if r.jsonEncOpts == nil {
+		r.jsonEncOpts = &jsonEncOpts{escapeHTML: true}
+	}
+	return r.jsonEncOpts
+}
+
+// SetJSONEscapeHTML specifies whether problematic HTML characters
+// should be escaped inside JSON quoted strings.
+// The default behavior is to escape &, <, and > to \u0026, \u003c, and \u003e
+// to avoid certain safety problems that can arise when embedding JSON in HTML.
+//
+// In non-HTML settings where the escaping interferes with the readability
+// of the output, SetEscapeHTML(false) disables this behavior.
+func (r *Req) SetJSONEscapeHTML(escape bool) {
+	opts := r.getJSONEncOpts()
+	opts.escapeHTML = escape
+}
+
+// SetJSONEscapeHTML specifies whether problematic HTML characters
+// should be escaped inside JSON quoted strings.
+// The default behavior is to escape &, <, and > to \u0026, \u003c, and \u003e
+// to avoid certain safety problems that can arise when embedding JSON in HTML.
+//
+// In non-HTML settings where the escaping interferes with the readability
+// of the output, SetEscapeHTML(false) disables this behavior.
+func SetJSONEscapeHTML(escape bool) {
+	std.SetJSONEscapeHTML(escape)
+}
+
+// SetJSONIndent instructs the encoder to format each subsequent encoded
+// value as if indented by the package-level function Indent(dst, src, prefix, indent).
+// Calling SetIndent("", "") disables indentation.
+func (r *Req) SetJSONIndent(prefix, indent string) {
+	opts := r.getJSONEncOpts()
+	opts.indentPrefix = prefix
+	opts.indentValue = indent
+}
+
+// SetJSONIndent instructs the encoder to format each subsequent encoded
+// value as if indented by the package-level function Indent(dst, src, prefix, indent).
+// Calling SetIndent("", "") disables indentation.
+func SetJSONIndent(prefix, indent string) {
+	std.SetJSONIndent(prefix, indent)
+}
+
+type xmlEncOpts struct {
+	prefix string
+	indent string
+}
+
+func (r *Req) getXMLEncOpts() *xmlEncOpts {
+	if r.xmlEncOpts == nil {
+		r.xmlEncOpts = &xmlEncOpts{}
+	}
+	return r.xmlEncOpts
+}
+
+// SetXMLIndent sets the encoder to generate XML in which each element
+// begins on a new indented line that starts with prefix and is followed by
+// one or more copies of indent according to the nesting depth.
+func (r *Req) SetXMLIndent(prefix, indent string) {
+	opts := r.getXMLEncOpts()
+	opts.prefix = prefix
+	opts.indent = indent
+}
+
+// SetXMLIndent sets the encoder to generate XML in which each element
+// begins on a new indented line that starts with prefix and is followed by
+// one or more copies of indent according to the nesting depth.
+func SetXMLIndent(prefix, indent string) {
+	std.SetXMLIndent(prefix, indent)
+}
--- a/backend/vendor/github.com/json-iterator/go/Gopkg.lock
+++ b/backend/vendor/github.com/json-iterator/go/Gopkg.lock
@@ -1,21 +0,0 @@
-# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
-
-
-[[projects]]
-  name = "github.com/modern-go/concurrent"
-  packages = ["."]
-  revision = "e0a39a4cb4216ea8db28e22a69f4ec25610d513a"
-  version = "1.0.0"
-
-[[projects]]
-  name = "github.com/modern-go/reflect2"
-  packages = ["."]
-  revision = "4b7aa43c6742a2c18fdef89dd197aaae7dac7ccd"
-  version = "1.0.1"
-
-[solve-meta]
-  analyzer-name = "dep"
-  analyzer-version = 1
-  inputs-digest = "ea54a775e5a354cb015502d2e7aa4b74230fc77e894f34a838b268c25ec8eeb8"
-  solver-name = "gps-cdcl"
-  solver-version = 1
--- a/backend/vendor/github.com/jtolds/gls/LICENSE
+++ b/backend/vendor/github.com/jtolds/gls/LICENSE
@@ -0,0 +1,18 @@
+Copyright (c) 2013, Space Monkey, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
--- a/backend/vendor/github.com/jtolds/gls/README.md
+++ b/backend/vendor/github.com/jtolds/gls/README.md
@@ -0,0 +1,89 @@
+gls
+===
+
+Goroutine local storage
+
+### IMPORTANT NOTE ###
+
+It is my duty to point you to https://blog.golang.org/context, which is how 
+Google solves all of the problems you'd perhaps consider using this package
+for at scale. 
+
+One downside to Google's approach is that *all* of your functions must have
+a new first argument, but after clearing that hurdle everything else is much
+better.
+
+If you aren't interested in this warning, read on.
+
+### Huhwaht? Why? ###
+
+Every so often, a thread shows up on the
+[golang-nuts](https://groups.google.com/d/forum/golang-nuts) asking for some
+form of goroutine-local-storage, or some kind of goroutine id, or some kind of
+context. There are a few valid use cases for goroutine-local-storage, one of
+the most prominent being log line context. One poster was interested in being
+able to log an HTTP request context id in every log line in the same goroutine
+as the incoming HTTP request, without having to change every library and
+function call he was interested in logging.
+
+This would be pretty useful. Provided that you could get some kind of
+goroutine-local-storage, you could call
+[log.SetOutput](http://golang.org/pkg/log/#SetOutput) with your own logging
+writer that checks goroutine-local-storage for some context information and
+adds that context to your log lines.
+
+But alas, Andrew Gerrand's typically diplomatic answer to the question of
+goroutine-local variables was:
+
+> We wouldn't even be having this discussion if thread local storage wasn't
+> useful. But every feature comes at a cost, and in my opinion the cost of
+> threadlocals far outweighs their benefits. They're just not a good fit for
+> Go.
+
+So, yeah, that makes sense. That's a pretty good reason for why the language
+won't support a specific and (relatively) unuseful feature that requires some
+runtime changes, just for the sake of a little bit of log improvement.
+
+But does Go require runtime changes?
+
+### How it works ###
+
+Go has pretty fantastic introspective and reflective features, but one thing Go
+doesn't give you is any kind of access to the stack pointer, or frame pointer,
+or goroutine id, or anything contextual about your current stack. It gives you
+access to your list of callers, but only along with program counters, which are
+fixed at compile time.
+
+But it does give you the stack.
+
+So, we define 16 special functions and embed base-16 tags into the stack using
+the call order of those 16 functions. Then, we can read our tags back out of
+the stack looking at the callers list.
+
+We then use these tags as an index into a traditional map for implementing
+this library.
+
+### What are people saying? ###
+
+"Wow, that's horrifying."
+
+"This is the most terrible thing I have seen in a very long time."
+
+"Where is it getting a context from? Is this serializing all the requests? 
+What the heck is the client being bound to? What are these tags? Why does he 
+need callers? Oh god no. No no no."
+
+### Docs ###
+
+Please see the docs at http://godoc.org/github.com/jtolds/gls
+
+### Related ###
+
+If you're okay relying on the string format of the current runtime stacktrace 
+including a unique goroutine id (not guaranteed by the spec or anything, but 
+very unlikely to change within a Go release), you might be able to squeeze 
+out a bit more performance by using this similar library, inspired by some 
+code Brad Fitzpatrick wrote for debugging his HTTP/2 library: 
+https://github.com/tylerb/gls (in contrast, jtolds/gls doesn't require 
+any knowledge of the string format of the runtime stacktrace, which 
+probably adds unnecessary overhead).
--- a/backend/vendor/github.com/jtolds/gls/context.go
+++ b/backend/vendor/github.com/jtolds/gls/context.go
@@ -0,0 +1,153 @@
+// Package gls implements goroutine-local storage.
+package gls
+
+import (
+	"sync"
+)
+
+var (
+	mgrRegistry    = make(map[*ContextManager]bool)
+	mgrRegistryMtx sync.RWMutex
+)
+
+// Values is simply a map of key types to value types. Used by SetValues to
+// set multiple values at once.
+type Values map[interface{}]interface{}
+
+// ContextManager is the main entrypoint for interacting with
+// Goroutine-local-storage. You can have multiple independent ContextManagers
+// at any given time. ContextManagers are usually declared globally for a given
+// class of context variables. You should use NewContextManager for
+// construction.
+type ContextManager struct {
+	mtx    sync.Mutex
+	values map[uint]Values
+}
+
+// NewContextManager returns a brand new ContextManager. It also registers the
+// new ContextManager in the ContextManager registry which is used by the Go
+// method. ContextManagers are typically defined globally at package scope.
+func NewContextManager() *ContextManager {
+	mgr := &ContextManager{values: make(map[uint]Values)}
+	mgrRegistryMtx.Lock()
+	defer mgrRegistryMtx.Unlock()
+	mgrRegistry[mgr] = true
+	return mgr
+}
+
+// Unregister removes a ContextManager from the global registry, used by the
+// Go method. Only intended for use when you're completely done with a
+// ContextManager. Use of Unregister at all is rare.
+func (m *ContextManager) Unregister() {
+	mgrRegistryMtx.Lock()
+	defer mgrRegistryMtx.Unlock()
+	delete(mgrRegistry, m)
+}
+
+// SetValues takes a collection of values and a function to call for those
+// values to be set in. Anything further down the stack will have the set
+// values available through GetValue. SetValues will add new values or replace
+// existing values of the same key and will not mutate or change values for
+// previous stack frames.
+// SetValues is slow (makes a copy of all current and new values for the new
+// gls-context) in order to reduce the amount of lookups GetValue requires.
+func (m *ContextManager) SetValues(new_values Values, context_call func()) {
+	if len(new_values) == 0 {
+		context_call()
+		return
+	}
+
+	mutated_keys := make([]interface{}, 0, len(new_values))
+	mutated_vals := make(Values, len(new_values))
+
+	EnsureGoroutineId(func(gid uint) {
+		m.mtx.Lock()
+		state, found := m.values[gid]
+		if !found {
+			state = make(Values, len(new_values))
+			m.values[gid] = state
+		}
+		m.mtx.Unlock()
+
+		for key, new_val := range new_values {
+			mutated_keys = append(mutated_keys, key)
+			if old_val, ok := state[key]; ok {
+				mutated_vals[key] = old_val
+			}
+			state[key] = new_val
+		}
+
+		defer func() {
+			if !found {
+				m.mtx.Lock()
+				delete(m.values, gid)
+				m.mtx.Unlock()
+				return
+			}
+
+			for _, key := range mutated_keys {
+				if val, ok := mutated_vals[key]; ok {
+					state[key] = val
+				} else {
+					delete(state, key)
+				}
+			}
+		}()
+
+		context_call()
+	})
+}
+
+// GetValue will return a previously set value, provided that the value was set
+// by SetValues somewhere higher up the stack. If the value is not found, ok
+// will be false.
+func (m *ContextManager) GetValue(key interface{}) (
+	value interface{}, ok bool) {
+	gid, ok := GetGoroutineId()
+	if !ok {
+		return nil, false
+	}
+
+	m.mtx.Lock()
+	state, found := m.values[gid]
+	m.mtx.Unlock()
+
+	if !found {
+		return nil, false
+	}
+	value, ok = state[key]
+	return value, ok
+}
+
+func (m *ContextManager) getValues() Values {
+	gid, ok := GetGoroutineId()
+	if !ok {
+		return nil
+	}
+	m.mtx.Lock()
+	state, _ := m.values[gid]
+	m.mtx.Unlock()
+	return state
+}
+
+// Go preserves ContextManager values and Goroutine-local-storage across new
+// goroutine invocations. The Go method makes a copy of all existing values on
+// all registered context managers and makes sure they are still set after
+// kicking off the provided function in a new goroutine. If you don't use this
+// Go method instead of the standard 'go' keyword, you will lose values in
+// ContextManagers, as goroutines have brand new stacks.
+func Go(cb func()) {
+	mgrRegistryMtx.RLock()
+	defer mgrRegistryMtx.RUnlock()
+
+	for mgr := range mgrRegistry {
+		values := mgr.getValues()
+		if len(values) > 0 {
+			cb = func(mgr *ContextManager, cb func()) func() {
+				return func() { mgr.SetValues(values, cb) }
+			}(mgr, cb)
+		}
+	}
+
+	go cb()
+}
--- a/backend/vendor/github.com/jtolds/gls/gen_sym.go
+++ b/backend/vendor/github.com/jtolds/gls/gen_sym.go
@@ -0,0 +1,21 @@
+package gls
+
+import (
+	"sync"
+)
+
+var (
+	keyMtx     sync.Mutex
+	keyCounter uint64
+)
+
+// ContextKey is a throwaway value you can use as a key to a ContextManager
+type ContextKey struct{ id uint64 }
+
+// GenSym will return a brand new, never-before-used ContextKey
+func GenSym() ContextKey {
+	keyMtx.Lock()
+	defer keyMtx.Unlock()
+	keyCounter += 1
+	return ContextKey{id: keyCounter}
+}
--- a/backend/vendor/github.com/jtolds/gls/gid.go
+++ b/backend/vendor/github.com/jtolds/gls/gid.go
@@ -0,0 +1,25 @@
+package gls
+
+var (
+	stackTagPool = &idPool{}
+)
+
+// Will return this goroutine's identifier if set. If you always need a
+// goroutine identifier, you should use EnsureGoroutineId which will make one
+// if there isn't one already.
+func GetGoroutineId() (gid uint, ok bool) {
+	return readStackTag()
+}
+
+// Will call cb with the current goroutine identifier. If one hasn't already
+// been generated, one will be created and set first. The goroutine identifier
+// might be invalid after cb returns.
+func EnsureGoroutineId(cb func(gid uint)) {
+	if gid, ok := readStackTag(); ok {
+		cb(gid)
+		return
+	}
+	gid := stackTagPool.Acquire()
+	defer stackTagPool.Release(gid)
+	addStackTag(gid, func() { cb(gid) })
+}
--- a/backend/vendor/github.com/jtolds/gls/id_pool.go
+++ b/backend/vendor/github.com/jtolds/gls/id_pool.go
@@ -0,0 +1,34 @@
+package gls
+
+// though this could probably be better at keeping ids smaller, the goal of
+// this class is to keep a registry of the smallest unique integer ids
+// per-process possible
+
+import (
+	"sync"
+)
+
+type idPool struct {
+	mtx      sync.Mutex
+	released []uint
+	max_id   uint
+}
+
+func (p *idPool) Acquire() (id uint) {
+	p.mtx.Lock()
+	defer p.mtx.Unlock()
+	if len(p.released) > 0 {
+		id = p.released[len(p.released)-1]
+		p.released = p.released[:len(p.released)-1]
+		return id
+	}
+	id = p.max_id
+	p.max_id++
+	return id
+}
+
+func (p *idPool) Release(id uint) {
+	p.mtx.Lock()
+	defer p.mtx.Unlock()
+	p.released = append(p.released, id)
+}
--- a/Show More
+++ b/Show More