+
## 架构
-Crawlab的架构包括了一个主节点(Master Node)和多个工作节点(Worker Node),以及负责通信和数据储存的Redis和MongoDB数据库。
+Crawlab的架构包括了一个主节点(Master Node)和多个工作节点(Worker Node),以及 [SeaweeFS](https://github.com/chrislusf/seaweedfs) (分布式文件系统) 和 MongoDB 数据库。
-
+
-前端应用向主节点请求数据,主节点通过MongoDB和Redis来执行任务派发调度以及部署,工作节点收到任务之后,开始执行爬虫任务,并将任务结果储存到MongoDB。架构相对于`v0.3.0`之前的Celery版本有所精简,去除了不必要的节点监控模块Flower,节点监控主要由Redis完成。
+前端应用与主节点 (Master Node) 进行交互,主节点与其他模块(例如 MongoDB、SeaweedFS、工作节点)进行通信。主节点和工作节点 (Worker Nodes) 通过 [gRPC](https://grpc.io) (一种 RPC 框架) 进行通信。任务通过主节点上的任务调度器 (Task Scheduler) 进行调度分发,并被工作节点上的任务处理模块 (Task Handler) 接收,然后分配到任务执行器 (Task Runners) 中。任务执行器实际上是执行爬虫程序的进程,它可以通过 gRPC (内置于 SDK) 发送数据到其他数据源中,例如 MongoDB。
### 主节点
@@ -185,7 +180,7 @@ Crawlab的架构包括了一个主节点(Master Node)和多个工作节点
4. 前端以及API服务
5. 执行任务(可以将主节点当成工作节点)
-主节点负责与前端应用进行通信,并通过Redis将爬虫任务派发给工作节点。同时,主节点会同步(部署)爬虫给工作节点,通过Redis和MongoDB的GridFS。
+主节点负责与前端应用进行通信,并将爬虫任务派发给工作节点。同时,主节点会同步(部署)爬虫到分布式文件系统 SeaweedFS,用于工作节点的文件同步。
### 工作节点
@@ -193,15 +188,17 @@ Crawlab的架构包括了一个主节点(Master Node)和多个工作节点
### MongoDB
-MongoDB是Crawlab的运行数据库,储存有节点、爬虫、任务、定时任务等数据,另外GridFS文件储存方式是主节点储存爬虫文件并同步到工作节点的中间媒介。
+MongoDB是Crawlab的运行数据库,储存有节点、爬虫、任务、定时任务等数据。任务队列也储存在 MongoDB 里。
-### Redis
+### SeaweedFS
-Redis是非常受欢迎的Key-Value数据库,在Crawlab中主要实现节点间数据通信的功能。例如,节点会将自己信息通过`HSET`储存在Redis的`nodes`哈希列表中,主节点根据哈希列表来判断在线节点。
+SeaweedFS 是开源分布式文件系统,由 [Chris Lu](https://github.com/chrislusf) 开发和维护。它能在分布式系统中有效稳定的储存和共享文件。在 Crawlab 中,SeaweedFS 主要用作文件同步和日志存储。
### 前端
-前端是一个基于[Vue-Element-Admin](https://github.com/PanJiaChen/vue-element-admin)的单页应用。其中重用了很多Element-UI的控件来支持相应的展示。
+Frontend app is built upon [Element-Plus](https://github.com/element-plus/element-plus), a popular [Vue 3](https://github.com/vuejs/vue-next)-based UI framework. It interacts with API hosted on the Master Node, and indirectly controls Worker Nodes.
+
+前端应用是基于 [Element-Plus](https://github.com/element-plus/element-plus) 构建的,它是基于 [Vue 3](https://github.com/vuejs/vue-next) 的 UI 框架。前端应用与主节点上的 API 进行交互,并间接控制工作节点。
## 与其他框架的集成
@@ -213,11 +210,11 @@ Redis是非常受欢迎的Key-Value数据库,在Crawlab中主要实现节点
```python
ITEM_PIPELINES = {
- 'crawlab.pipelines.CrawlabMongoPipeline': 888,
+ 'crawlab.scrapy.pipelines.CrawlabPipeline': 888,
}
```
-然后,启动 Scrapy 爬虫,运行完成之后,您就应该能看到抓取结果出现在 **任务详情-结果** 里。
+然后,启动 Scrapy 爬虫,运行完成之后,您就应该能看到抓取结果出现在 **任务详情 -> 数据** 里。
### 通用 Python 爬虫
@@ -234,13 +231,11 @@ result = {'name': 'crawlab'}
save_item(result)
```
-然后,启动爬虫,运行完成之后,您就应该能看到抓取结果出现在 **任务详情-结果** 里。
+然后,启动爬虫,运行完成之后,您就应该能看到抓取结果出现在 **任务详情 -> 数据** 里。
### 其他框架和语言
-爬虫任务本质上是由一个shell命令来实现的。任务ID将以环境变量`CRAWLAB_TASK_ID`的形式存在于爬虫任务运行的进程中,并以此来关联抓取数据。另外,`CRAWLAB_COLLECTION`是Crawlab传过来的所存放collection的名称。
-
-在爬虫程序中,需要将`CRAWLAB_TASK_ID`的值以`task_id`作为可以存入数据库中`CRAWLAB_COLLECTION`的collection中。这样Crawlab就知道如何将爬虫任务与抓取数据关联起来了。当前,Crawlab只支持MongoDB。
+爬虫任务实际上是通过 shell 命令执行的。任务 ID (Task ID) 作为环境变量 `CRAWLAB_TASK_ID` 被传入爬虫任务进程中,从而抓取的数据可以跟任务管理。
## 与其他框架比较
@@ -257,38 +252,6 @@ Crawlab使用起来很方便,也很通用,可以适用于几乎任何主流
| [Gerapy](https://github.com/Gerapy/Gerapy) | Python Django + Vue|Gerapy 是崔庆才大神开发的爬虫管理平台,安装部署非常简单,同样基于 scrapyd,有精美的 UI 界面,支持节点管理、代码编辑、可配置规则等功能|同样不支持 scrapy 以外的爬虫,而且据使用者反馈,1.0 版本有很多 bug,期待 2.0 版本会有一定程度的改进|   |
| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | Python Flask|基于 scrapyd,开源版 Scrapyhub,非常简洁的 UI 界面,支持定时任务|可能有些过于简洁了,不支持分页,不支持节点管理,不支持 scrapy 以外的爬虫|   |
-## Q&A
-
-#### 1. 为何我访问 http://localhost:8080 提示访问不了?
-
-假如您是Docker部署的,请检查一下您是否用了Docker Machine,这样的话您需要输入地址 http://192.168.99.100:8080 才行。
-
-另外,请确保您用了`-p 8080:8080`来映射端口,并检查宿主机是否开放了8080端口。
-
-#### 2. 我可以看到登录页面了,但为何我点击登陆的时候按钮一直转圈圈?
-
-绝大多数情况下,您可能是没有正确配置`CRAWLAB_API_ADDRESS`这个环境变量。这个变量是告诉前端应该通过哪个地址来请求API数据的,因此需要将它设置为宿主机的IP地址+端口,例如 `192.168.0.1:8000`。接着,重启容器,在浏览器中输入宿主机IP+端口,就可以顺利登陆了。
-
-请注意,8080是前端端口,8000是后端端口,您在浏览器中只需要输入前端的地址就可以了,要注意区分。
-
-#### 3. 在爬虫页面有一些不认识的爬虫列表,这些是什么呢?
-
-这些是demo爬虫,如果需要添加您自己的爬虫,请将您的爬虫文件打包成zip文件,再在爬虫页面中点击**添加爬虫**上传就可以了。
-
-注意,Crawlab将取文件名作为爬虫名称,这个您可以后期更改。另外,请不要将zip文件名设置为中文,可能会导致上传不成功。
-
-## 相关文章
-
-- [爬虫管理平台Crawlab v0.3.0发布(Golang版本)](https://juejin.im/post/5d418deff265da03c926d75c)
-- [爬虫平台Crawlab核心原理--分布式架构](https://juejin.im/post/5d4ba9d1e51d4561cf15df79)
-- [爬虫平台Crawlab核心原理--自动提取字段算法](https://juejin.im/post/5cf4a7fa5188254c5879facd)
-- [爬虫管理平台Crawlab部署指南(Docker and more)](https://juejin.im/post/5d01027a518825142939320f)
-- [[爬虫手记] 我是如何在3分钟内开发完一个爬虫的](https://juejin.im/post/5ceb4342f265da1bc8540660)
-- [手把手教你如何用Crawlab构建技术文章聚合平台(二)](https://juejin.im/post/5c92365d6fb9a070c5510e71)
-- [手把手教你如何用Crawlab构建技术文章聚合平台(一)](https://juejin.im/user/5a1ba6def265da430b7af463/posts)
-
-**注意: v0.3.0版本已将基于Celery的Python版本切换为了Golang版本,如何部署请参照文档**
-
## 贡献者
-
-
+
+
@@ -35,29 +35,28 @@ Golang-based distributed web crawler management platform, supporting various lan
## Installation
Three methods:
-1. [Docker](http://docs.crawlab.cn/en/Installation/Docker.html) (Recommended)
-2. [Direct Deploy](http://docs.crawlab.cn/en/Installation/Direct.html) (Check Internal Kernel)
-3. [Kubernetes](http://docs.crawlab.cn/en/Installation/Kubernetes.html) (Multi-Node Deployment)
+1. [Docker](http://docs.crawlab.cn/Installation/Docker.html) (Recommended)
+2. [Direct Deploy](http://docs.crawlab.cn/Installation/Direct.html) (Check Internal Kernel)
+3. [Kubernetes](http://docs.crawlab.cn/Installation/Kubernetes.html) (Multi-Node Deployment)
### Pre-requisite (Docker)
- Docker 18.03+
-- Redis 5.x+
- MongoDB 3.6+
- Docker Compose 1.24+ (optional but recommended)
### Pre-requisite (Direct Deploy)
-- Go 1.12+
+- Go 1.15+
- Node 8.12+
-- Redis 5.x+
- MongoDB 3.6+
+- [SeaweedFS](https://github.com/chrislusf/seaweedfs) 2.59+
## Quick Start
Please open the command line prompt and execute the command below. Make sure you have installed `docker-compose` in advance.
```bash
-git clone https://github.com/crawlab-team/crawlab
-cd crawlab
+git clone https://github.com/crawlab-team/examples
+cd examples/docker/basic
docker-compose up -d
```
@@ -73,34 +72,51 @@ Please use `docker-compose` to one-click to start up. By doing so, you don't eve
```yaml
version: '3.3'
services:
- master:
- image: tikazyq/crawlab:latest
- container_name: master
+ master:
+ image: crawlabteam/crawlab:latest
+ container_name: crawlab_example_master
environment:
CRAWLAB_SERVER_MASTER: "Y"
CRAWLAB_MONGO_HOST: "mongo"
- CRAWLAB_REDIS_ADDRESS: "redis"
- ports:
+ volumes:
+ - "./.crawlab/master:/root/.crawlab"
+ ports:
- "8080:8080"
depends_on:
- mongo
- - redis
+
+ worker01:
+ image: crawlabteam/crawlab:latest
+ container_name: crawlab_example_worker01
+ environment:
+ CRAWLAB_SERVER_MASTER: "N"
+ CRAWLAB_GRPC_ADDRESS: "master"
+ volumes:
+ - "./.crawlab/worker01:/root/.crawlab"
+ depends_on:
+ - master
+
+ worker02:
+ image: crawlabteam/crawlab:latest
+ container_name: crawlab_example_worker02
+ environment:
+ CRAWLAB_SERVER_MASTER: "N"
+ CRAWLAB_GRPC_ADDRESS: "master"
+ volumes:
+ - "./.crawlab/worker02:/root/.crawlab"
+ depends_on:
+ - master
+
mongo:
image: mongo:latest
+ container_name: crawlab_example_mongo
restart: always
- ports:
- - "27017:27017"
- redis:
- image: redis:latest
- restart: always
- ports:
- - "6379:6379"
```
-Then execute the command below, and Crawlab Master Node + MongoDB + Redis will start up. Open the browser and enter `http://localhost:8080` to see the UI interface.
+Then execute the command below, and Crawlab Master and Worker Nodes + MongoDB will start up. Open the browser and enter `http://localhost:8080` to see the UI interface.
```bash
-docker-compose up
+docker-compose up -d
```
For Docker Deployment details, please refer to [relevant documentation](https://tikazyq.github.io/crawlab-docs/Installation/Docker.html).
@@ -110,115 +126,92 @@ For Docker Deployment details, please refer to [relevant documentation](https://
#### Login
-
+
#### Home Page
-
+
#### Node List
-
-
-#### Node Network
-
-
+
#### Spider List
-
+
#### Spider Overview
-
+
-#### Spider Analytics
+#### Spider Files
-
-
-#### Spider File Edit
-
-
+
#### Task Log
-
+
#### Task Results
-
+
#### Cron Job
-
-
-#### Language Installation
-
-
-
-#### Dependency Installation
-
-
-
-#### Notifications
-
-
+
## Architecture
-The architecture of Crawlab is consisted of the Master Node and multiple Worker Nodes, and Redis and MongoDB databases which are mainly for nodes communication and data storage.
+The architecture of Crawlab is consisted of a master node, worker nodes, [SeaweeFS](https://github.com/chrislusf/seaweedfs) (a distributed file system) and MongoDB database.
-
+
-The frontend app makes requests to the Master Node, which assigns tasks and deploys spiders through MongoDB and Redis. When a Worker Node receives a task, it begins to execute the crawling task, and stores the results to MongoDB. The architecture is much more concise compared with versions before `v0.3.0`. It has removed unnecessary Flower module which offers node monitoring services. They are now done by Redis.
+The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. Master node and worker nodes communicate with each other via [gRPC](https://grpc.io) (a RPC framework). Tasks are scheduled by the task scheduler module in the master node, and received by the task handler module in worker nodes, which executes these tasks in task runners. Task runners are actually processes running spider or crawler programs, and can also send data through gRPC (integrated in SDK) to other data sources, e.g. MongoDB.
### Master Node
The Master Node is the core of the Crawlab architecture. It is the center control system of Crawlab.
-The Master Node offers below services:
-1. Crawling Task Coordination;
+The Master Node provides below services:
+1. Task Scheduling;
2. Worker Node Management and Communication;
3. Spider Deployment;
4. Frontend and API Services;
-5. Task Execution (one can regard the Master Node as a Worker Node)
+5. Task Execution (you can regard the Master Node as a Worker Node)
-The Master Node communicates with the frontend app, and send crawling tasks to Worker Nodes. In the mean time, the Master Node synchronizes (deploys) spiders to Worker Nodes, via Redis and MongoDB GridFS.
+The Master Node communicates with the frontend app, and send crawling tasks to Worker Nodes. In the mean time, the Master Node uploads (deploys) spiders to the distributed file system SeaweedFS, for synchronization by worker nodes.
### Worker Node
-The main functionality of the Worker Nodes is to execute crawling tasks and store results and logs, and communicate with the Master Node through Redis `PubSub`. By increasing the number of Worker Nodes, Crawlab can scale horizontally, and different crawling tasks can be assigned to different nodes to execute.
+The main functionality of the Worker Nodes is to execute crawling tasks and store results and logs, and communicate with the Master Node through gRPC. By increasing the number of Worker Nodes, Crawlab can scale horizontally, and different crawling tasks can be assigned to different nodes to execute.
### MongoDB
-MongoDB is the operational database of Crawlab. It stores data of nodes, spiders, tasks, schedules, etc. The MongoDB GridFS file system is the medium for the Master Node to store spider files and synchronize to the Worker Nodes.
+MongoDB is the operational database of Crawlab. It stores data of nodes, spiders, tasks, schedules, etc. Task queue is also stored in MongoDB.
-### Redis
+### SeaweedFS
-Redis is a very popular Key-Value database. It offers node communication services in Crawlab. For example, nodes will execute `HSET` to set their info into a hash list named `nodes` in Redis, and the Master Node will identify online nodes according to the hash list.
+SeaweedFS is an open source distributed file system authored by [Chris Lu](https://github.com/chrislusf). It can robustly store and share files across a distributed system. In Crawlab, SeaweedFS mainly plays the role as file synchronization system and the place where task log files are stored.
### Frontend
-Frontend is a SPA based on
-[Vue-Element-Admin](https://github.com/PanJiaChen/vue-element-admin). It has re-used many Element-UI components to support corresponding display.
+Frontend app is built upon [Element-Plus](https://github.com/element-plus/element-plus), a popular [Vue 3](https://github.com/vuejs/vue-next)-based UI framework. It interacts with API hosted on the Master Node, and indirectly controls Worker Nodes.
## Integration with Other Frameworks
[Crawlab SDK](https://github.com/crawlab-team/crawlab-sdk) provides some `helper` methods to make it easier for you to integrate your spiders into Crawlab, e.g. saving results.
-⚠️Note: make sure you have already installed `crawlab-sdk` using pip.
-
### Scrapy
In `settings.py` in your Scrapy project, find the variable named `ITEM_PIPELINES` (a `dict` variable). Add content below.
```python
ITEM_PIPELINES = {
- 'crawlab.pipelines.CrawlabMongoPipeline': 888,
+ 'crawlab.scrapy.pipelines.CrawlabPipeline': 888,
}
```
-Then, start the Scrapy spider. After it's done, you should be able to see scraped results in **Task Detail -> Result**
+Then, start the Scrapy spider. After it's done, you should be able to see scraped results in **Task Detail -> Data**
### General Python Spider
@@ -235,11 +228,11 @@ result = {'name': 'crawlab'}
save_item(result)
```
-Then, start the spider. After it's done, you should be able to see scraped results in **Task Detail -> Result**
+Then, start the spider. After it's done, you should be able to see scraped results in **Task Detail -> Data**
### Other Frameworks / Languages
-A crawling task is actually executed through a shell command. The Task ID will be passed to the crawling task process in the form of environment variable named `CRAWLAB_TASK_ID`. By doing so, the data can be related to a task. Also, another environment variable `CRAWLAB_COLLECTION` is passed by Crawlab as the name of the collection to store results data.
+A crawling task is actually executed through a shell command. The Task ID will be passed to the crawling task process in the form of environment variable named `CRAWLAB_TASK_ID`. By doing so, the data can be related to a task.
## Comparison with Other Frameworks
diff --git a/backend/apps/api.go b/backend/apps/api.go
new file mode 100644
index 00000000..43a34bc8
--- /dev/null
+++ b/backend/apps/api.go
@@ -0,0 +1,79 @@
+package apps
+
+import (
+ "context"
+ "github.com/apex/log"
+ "github.com/crawlab-team/crawlab-core/controllers"
+ "github.com/crawlab-team/crawlab-core/interfaces"
+ "github.com/crawlab-team/crawlab-core/middlewares"
+ "github.com/crawlab-team/crawlab-core/routes"
+ "github.com/gin-gonic/gin"
+ "github.com/spf13/viper"
+ "net"
+ "net/http"
+ "time"
+)
+
+type Api struct {
+ // dependencies
+ interfaces.WithConfigPath
+
+ // internals
+ app *gin.Engine
+ srv *http.Server
+}
+
+func (app *Api) Init() {
+ // initialize controllers
+ _ = initModule("controllers", controllers.InitControllers)
+
+ // initialize middlewares
+ _ = app.initModuleWithApp("middlewares", middlewares.InitMiddlewares)
+
+ // initialize routes
+ _ = app.initModuleWithApp("routes", routes.InitRoutes)
+}
+
+func (app *Api) Start() {
+ host := viper.GetString("server.host")
+ port := viper.GetString("server.port")
+ address := net.JoinHostPort(host, port)
+ app.srv = &http.Server{
+ Handler: app.app,
+ Addr: address,
+ }
+ if err := app.srv.ListenAndServe(); err != nil {
+ if err != http.ErrServerClosed {
+ log.Error("run server error:" + err.Error())
+ } else {
+ log.Info("server graceful down")
+ }
+ }
+}
+
+func (app *Api) Wait() {
+ DefaultWait()
+}
+
+func (app *Api) Stop() {
+ ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
+ defer cancel()
+
+ if err := app.srv.Shutdown(ctx); err != nil {
+ log.Error("run server error:" + err.Error())
+ }
+}
+
+func (app *Api) initModuleWithApp(name string, fn func(app *gin.Engine) error) (err error) {
+ return initModule(name, func() error {
+ return fn(app.app)
+ })
+}
+
+func NewApi() *Api {
+ api := &Api{
+ app: gin.New(),
+ }
+ api.Init()
+ return api
+}
diff --git a/backend/apps/handler.go b/backend/apps/handler.go
new file mode 100644
index 00000000..9bf010bf
--- /dev/null
+++ b/backend/apps/handler.go
@@ -0,0 +1,26 @@
+package apps
+
+import (
+ "github.com/apex/log"
+)
+
+type Handler struct {
+}
+
+func (app *Handler) Init() {
+}
+
+func (app *Handler) Start() {
+}
+
+func (app *Handler) Wait() {
+ DefaultWait()
+}
+
+func (app *Handler) Stop() {
+ log.Info("handler has stopped")
+}
+
+func NewHandler() *Handler {
+ return &Handler{}
+}
diff --git a/backend/apps/interfaces.go b/backend/apps/interfaces.go
new file mode 100644
index 00000000..dbdd24fd
--- /dev/null
+++ b/backend/apps/interfaces.go
@@ -0,0 +1,25 @@
+package apps
+
+import "github.com/crawlab-team/crawlab-core/interfaces"
+
+type App interface {
+ Init()
+ Start()
+ Wait()
+ Stop()
+}
+
+type NodeApp interface {
+ App
+ interfaces.WithConfigPath
+ SetGrpcAddress(address interfaces.Address)
+}
+
+type MasterApp interface {
+ NodeApp
+ SetRunOnMaster(ok bool)
+}
+
+type WorkerApp interface {
+ NodeApp
+}
diff --git a/backend/apps/master.go b/backend/apps/master.go
new file mode 100644
index 00000000..e7a50e05
--- /dev/null
+++ b/backend/apps/master.go
@@ -0,0 +1,87 @@
+package apps
+
+import (
+ "github.com/crawlab-team/crawlab-core/config"
+ "github.com/crawlab-team/crawlab-core/controllers"
+ "github.com/crawlab-team/crawlab-core/interfaces"
+ "github.com/crawlab-team/crawlab-core/node/service"
+ "go.uber.org/dig"
+)
+
+type Master struct {
+ // settings
+ runOnMaster bool
+ grpcAddress interfaces.Address
+
+ // dependencies
+ interfaces.WithConfigPath
+ masterSvc interfaces.NodeMasterService
+
+ // modules
+ api *Api
+
+ // internals
+ quit chan int
+}
+
+func (app *Master) SetGrpcAddress(address interfaces.Address) {
+ app.grpcAddress = address
+}
+
+func (app *Master) SetRunOnMaster(ok bool) {
+ app.runOnMaster = ok
+}
+
+func (app *Master) Init() {
+ // initialize controllers
+ if err := controllers.InitControllers(); err != nil {
+ panic(err)
+ }
+}
+
+func (app *Master) Start() {
+ go app.api.Start()
+ go app.masterSvc.Start()
+}
+
+func (app *Master) Wait() {
+ <-app.quit
+}
+
+func (app *Master) Stop() {
+ app.api.Stop()
+ app.quit <- 1
+}
+
+func NewMaster(opts ...MasterOption) (app MasterApp) {
+ // master
+ m := &Master{
+ WithConfigPath: config.NewConfigPathService(),
+ api: NewApi(),
+ quit: make(chan int, 1),
+ }
+
+ // apply options
+ for _, opt := range opts {
+ opt(m)
+ }
+
+ // service options
+ var svcOpts []service.Option
+ if m.grpcAddress != nil {
+ svcOpts = append(svcOpts, service.WithAddress(m.grpcAddress))
+ }
+
+ // dependency injection
+ c := dig.New()
+ if err := c.Provide(service.ProvideMasterService(m.GetConfigPath(), svcOpts...)); err != nil {
+ panic(err)
+ }
+ if err := c.Invoke(func(masterSvc interfaces.NodeMasterService) {
+ m.masterSvc = masterSvc
+ }); err != nil {
+ panic(err)
+ }
+
+ return m
+}
diff --git a/backend/apps/options.go b/backend/apps/options.go
new file mode 100644
index 00000000..70280058
--- /dev/null
+++ b/backend/apps/options.go
@@ -0,0 +1,37 @@
+package apps
+
+import "github.com/crawlab-team/crawlab-core/interfaces"
+
+type MasterOption func(app MasterApp)
+
+func WithMasterConfigPath(path string) MasterOption {
+ return func(app MasterApp) {
+ app.SetConfigPath(path)
+ }
+}
+
+func WithMasterGrpcAddress(address interfaces.Address) MasterOption {
+ return func(app MasterApp) {
+ app.SetGrpcAddress(address)
+ }
+}
+
+func WithRunOnMaster(ok bool) MasterOption {
+ return func(app MasterApp) {
+ app.SetRunOnMaster(ok)
+ }
+}
+
+type WorkerOption func(app WorkerApp)
+
+func WithWorkerConfigPath(path string) WorkerOption {
+ return func(app WorkerApp) {
+ app.SetConfigPath(path)
+ }
+}
+
+func WithWorkerGrpcAddress(address interfaces.Address) WorkerOption {
+ return func(app WorkerApp) {
+ app.SetGrpcAddress(address)
+ }
+}
diff --git a/backend/apps/scheduler.go b/backend/apps/scheduler.go
new file mode 100644
index 00000000..8bb8662a
--- /dev/null
+++ b/backend/apps/scheduler.go
@@ -0,0 +1,27 @@
+package apps
+
+import (
+ "github.com/apex/log"
+)
+
+type Scheduler struct {
+}
+
+func (app *Scheduler) Init() {
+}
+
+func (app *Scheduler) Start() {
+ log.Info("scheduler has started")
+}
+
+func (app *Scheduler) Wait() {
+ DefaultWait()
+}
+
+func (app *Scheduler) Stop() {
+ log.Info("scheduler has stopped")
+}
+
+func NewScheduler() *Scheduler {
+ return &Scheduler{}
+}
diff --git a/backend/apps/utils.go b/backend/apps/utils.go
new file mode 100644
index 00000000..c223f8c5
--- /dev/null
+++ b/backend/apps/utils.go
@@ -0,0 +1,40 @@
+package apps
+
+import (
+ "fmt"
+ "github.com/apex/log"
+ "github.com/crawlab-team/crawlab-core/utils"
+ "github.com/crawlab-team/go-trace"
+)
+
+func Start(app App) {
+ start(app)
+}
+
+func start(app App) {
+ app.Init()
+ go app.Start()
+ app.Wait()
+ app.Stop()
+}
+
+func DefaultWait() {
+ utils.DefaultWait()
+}
+
+func initModule(name string, fn func() error) (err error) {
+ if err := fn(); err != nil {
+ log.Error(fmt.Sprintf("init %s error: %s", name, err.Error()))
+ _ = trace.TraceError(err)
+ panic(err)
+ }
+ log.Info(fmt.Sprintf("initialized %s successfully", name))
+ return nil
+}
+
+func initApp(name string, app App) {
+ _ = initModule(name, func() error {
+ app.Init()
+ return nil
+ })
+}
diff --git a/backend/apps/worker.go b/backend/apps/worker.go
new file mode 100644
index 00000000..7d1200d7
--- /dev/null
+++ b/backend/apps/worker.go
@@ -0,0 +1,71 @@
+package apps
+
+import (
+ "github.com/crawlab-team/crawlab-core/config"
+ "github.com/crawlab-team/crawlab-core/interfaces"
+ "github.com/crawlab-team/crawlab-core/node/service"
+ "go.uber.org/dig"
+)
+
+type Worker struct {
+ // settings
+ grpcAddress interfaces.Address
+
+ // dependencies
+ interfaces.WithConfigPath
+ workerSvc interfaces.NodeWorkerService
+
+ // internals
+ quit chan int
+}
+
+func (app *Worker) SetGrpcAddress(address interfaces.Address) {
+ app.grpcAddress = address
+}
+
+func (app *Worker) Init() {
+}
+
+func (app *Worker) Start() {
+ go app.workerSvc.Start()
+}
+
+func (app *Worker) Wait() {
+ <-app.quit
+}
+
+func (app *Worker) Stop() {
+ app.quit <- 1
+}
+
+func NewWorker(opts ...WorkerOption) (app *Worker) {
+ // worker
+ w := &Worker{
+ WithConfigPath: config.NewConfigPathService(),
+ quit: make(chan int, 1),
+ }
+
+ // apply options
+ for _, opt := range opts {
+ opt(w)
+ }
+
+ // service options
+ var svcOpts []service.Option
+ if w.grpcAddress != nil {
+ svcOpts = append(svcOpts, service.WithAddress(w.grpcAddress))
+ }
+
+ // dependency injection
+ c := dig.New()
+ if err := c.Provide(service.ProvideWorkerService(w.GetConfigPath(), svcOpts...)); err != nil {
+ panic(err)
+ }
+ if err := c.Invoke(func(workerSvc interfaces.NodeWorkerService) {
+ w.workerSvc = workerSvc
+ }); err != nil {
+ panic(err)
+ }
+
+ return w
+}
diff --git a/backend/cmd/api.go b/backend/cmd/api.go
new file mode 100644
index 00000000..931bdbe9
--- /dev/null
+++ b/backend/cmd/api.go
@@ -0,0 +1,21 @@
+package cmd
+
+import (
+ "crawlab/apps"
+ "github.com/spf13/cobra"
+)
+
+func init() {
+ rootCmd.AddCommand(apiCmd)
+}
+
+var apiCmd = &cobra.Command{
+ Use: "api",
+ Aliases: []string{"A"},
+ Short: "Start API server",
+ Long: `Start API server of Crawlab which serves data to frontend`,
+ Run: func(cmd *cobra.Command, args []string) {
+ api := apps.NewApi()
+ apps.Start(api)
+ },
+}
diff --git a/backend/cmd/handler.go b/backend/cmd/handler.go
new file mode 100644
index 00000000..67abf39f
--- /dev/null
+++ b/backend/cmd/handler.go
@@ -0,0 +1,22 @@
+package cmd
+
+import (
+ "crawlab/apps"
+ "github.com/spf13/cobra"
+)
+
+func init() {
+ rootCmd.AddCommand(handlerCmd)
+}
+
+var handlerCmd = &cobra.Command{
+ Use: "handler",
+ Aliases: []string{"H"},
+ Short: "Start handler",
+ Long: `Start a handler instance of Crawlab
+which runs tasks with given parameters`,
+ Run: func(cmd *cobra.Command, args []string) {
+ handler := apps.NewHandler()
+ apps.Start(handler)
+ },
+}
diff --git a/backend/cmd/master.go b/backend/cmd/master.go
new file mode 100644
index 00000000..d85e3e51
--- /dev/null
+++ b/backend/cmd/master.go
@@ -0,0 +1,64 @@
+package cmd
+
+import (
+ "crawlab/apps"
+ "fmt"
+ "github.com/crawlab-team/crawlab-core/entity"
+ "github.com/spf13/cobra"
+ "github.com/spf13/viper"
+)
+
+var (
+ runOnMaster bool
+ masterConfigPath string
+ masterGrpcAddress string
+ masterGrpcAuthKey string
+)
+
+func init() {
+ rootCmd.AddCommand(masterCmd)
+
+ masterCmd.PersistentFlags().StringVarP(&masterConfigPath, "config-path", "c", "", "Config path of master node")
+ _ = viper.BindPFlag("configPath", masterCmd.PersistentFlags().Lookup("configPath"))
+
+ masterCmd.PersistentFlags().StringVarP(&masterGrpcAddress, "grpc-address", "g", "", "gRPC address of master node")
+ _ = viper.BindPFlag("grpcAddress", masterCmd.PersistentFlags().Lookup("grpcAddress"))
+
+ masterCmd.PersistentFlags().StringVarP(&masterGrpcAuthKey, "grpc-auth-key", "a", "", "gRPC auth key of master node")
+ _ = viper.BindPFlag("grpcAuthKey", masterCmd.PersistentFlags().Lookup("grpcAuthKey"))
+}
+
+var masterCmd = &cobra.Command{
+ Use: "master",
+ Aliases: []string{"M"},
+ Short: "Start master",
+ Long: `Start a master instance of Crawlab
+which runs api and assign tasks to worker nodes`,
+ Run: func(cmd *cobra.Command, args []string) {
+ // options
+ var opts []apps.MasterOption
+ if masterConfigPath != "" {
+ opts = append(opts, apps.WithMasterConfigPath(masterConfigPath))
+ viper.Set("config.path", masterConfigPath)
+ }
+ opts = append(opts, apps.WithRunOnMaster(runOnMaster))
+ if masterGrpcAddress != "" {
+ address, err := entity.NewAddressFromString(masterGrpcAddress)
+ if err != nil {
+ fmt.Println(fmt.Sprintf("invalid grpc-address: %s", masterGrpcAddress))
+ }
+ opts = append(opts, apps.WithMasterGrpcAddress(address))
+ viper.Set("grpc.address", masterGrpcAddress)
+ viper.Set("grpc.server.address", masterGrpcAddress)
+ }
+ if masterGrpcAuthKey != "" {
+ viper.Set("grpc.authKey", masterGrpcAuthKey)
+ }
+
+ // app
+ master := apps.NewMaster(opts...)
+
+ // start
+ apps.Start(master)
+ },
+}
diff --git a/backend/cmd/root.go b/backend/cmd/root.go
new file mode 100644
index 00000000..11285130
--- /dev/null
+++ b/backend/cmd/root.go
@@ -0,0 +1,56 @@
+package cmd
+
+import (
+ "fmt"
+ "strings"
+
+ "github.com/spf13/cobra"
+ "github.com/spf13/viper"
+)
+
+var (
+ // Used for flags.
+ cfgFile string
+
+ rootCmd = &cobra.Command{
+ Use: "crawlab",
+ Short: "CLI tool for Crawlab",
+ Long: `The CLI tool is for controlling against Crawlab.
+Crawlab is a distributed web crawler and task admin platform
+aimed at making web crawling and task management easier.
+`,
+ }
+)
+
+// Execute executes the root command.
+func Execute() error {
+ return rootCmd.Execute()
+}
+
+func init() {
+ cobra.OnInitialize(initConfig)
+}
+
+func initConfig() {
+ if cfgFile != "" {
+ viper.SetConfigFile(cfgFile)
+ } else {
+ viper.AddConfigPath("./conf")
+ viper.SetConfigName("config")
+ }
+
+ // file format as yaml
+ viper.SetConfigType("yaml")
+
+ // auto load env
+ viper.AutomaticEnv()
+
+ // env prefix as CRAWLAB
+ viper.SetEnvPrefix("CRAWLAB")
+ replacer := strings.NewReplacer(".", "_")
+ viper.SetEnvKeyReplacer(replacer)
+
+ if err := viper.ReadInConfig(); err == nil {
+ fmt.Println("Using config file:", viper.ConfigFileUsed())
+ }
+}
diff --git a/backend/cmd/scheduler.go b/backend/cmd/scheduler.go
new file mode 100644
index 00000000..ae44a06a
--- /dev/null
+++ b/backend/cmd/scheduler.go
@@ -0,0 +1,22 @@
+package cmd
+
+import (
+ "crawlab/apps"
+ "github.com/spf13/cobra"
+)
+
+func init() {
+ rootCmd.AddCommand(schedulerCmd)
+}
+
+var schedulerCmd = &cobra.Command{
+ Use: "scheduler",
+ Aliases: []string{"S"},
+ Short: "Start scheduler",
+ Long: `Start a scheduler instance of Crawlab
+which assigns tasks to worker nodes to execute`,
+ Run: func(cmd *cobra.Command, args []string) {
+ scheduler := apps.NewScheduler()
+ apps.Start(scheduler)
+ },
+}
diff --git a/backend/cmd/worker.go b/backend/cmd/worker.go
new file mode 100644
index 00000000..490cdd47
--- /dev/null
+++ b/backend/cmd/worker.go
@@ -0,0 +1,63 @@
+package cmd
+
+import (
+ "crawlab/apps"
+ "fmt"
+ "github.com/crawlab-team/crawlab-core/entity"
+ "github.com/spf13/cobra"
+ "github.com/spf13/viper"
+)
+
+var (
+ workerConfigPath string
+ workerGrpcAddress string
+ workerGrpcAuthKey string
+)
+
+func init() {
+ rootCmd.AddCommand(workerCmd)
+
+ workerCmd.PersistentFlags().StringVarP(&workerConfigPath, "config-path", "c", "", "Config path of worker node")
+ _ = viper.BindPFlag("configPath", workerCmd.PersistentFlags().Lookup("configPath"))
+
+ workerCmd.PersistentFlags().StringVarP(&workerGrpcAddress, "grpc-address", "g", "", "gRPC address of worker node")
+ _ = viper.BindPFlag("grpcAddress", workerCmd.PersistentFlags().Lookup("grpcAddress"))
+
+ workerCmd.PersistentFlags().StringVarP(&workerGrpcAuthKey, "grpc-auth-key", "a", "", "gRPC auth key of worker node")
+ _ = viper.BindPFlag("grpcAuthKey", workerCmd.PersistentFlags().Lookup("grpcAuthKey"))
+}
+
+var workerCmd = &cobra.Command{
+ Use: "worker",
+ Aliases: []string{"W"},
+ Short: "Start worker",
+ Long: `Start a worker instance of Crawlab
+serving in the worker node and executes tasks
+assigned by the master node`,
+ Run: func(cmd *cobra.Command, args []string) {
+ // options
+ var opts []apps.WorkerOption
+ if workerConfigPath != "" {
+ opts = append(opts, apps.WithWorkerConfigPath(workerConfigPath))
+ viper.Set("config.path", workerConfigPath)
+ }
+ if workerGrpcAddress != "" {
+ address, err := entity.NewAddressFromString(workerGrpcAddress)
+ if err != nil {
+ fmt.Println(fmt.Sprintf("invalid grpc-address: %s", workerGrpcAddress))
+ return
+ }
+ opts = append(opts, apps.WithWorkerGrpcAddress(address))
+ viper.Set("grpc.address", workerGrpcAddress)
+ }
+ if workerGrpcAuthKey != "" {
+ viper.Set("grpc.authKey", workerGrpcAuthKey)
+ }
+
+ // app
+ master := apps.NewWorker(opts...)
+
+ // start
+ apps.Start(master)
+ },
+}
diff --git a/backend/conf/config.yml b/backend/conf/config.yml
index bc06935b..b4e90764 100644
--- a/backend/conf/config.yml
+++ b/backend/conf/config.yml
@@ -1,5 +1,3 @@
-api:
- address: "localhost:8000"
mongo:
host: localhost
port: 27017
@@ -7,61 +5,59 @@ mongo:
username: ""
password: ""
authSource: "admin"
-redis:
- address: localhost
- password: ""
- database: 1
- port: 6379
-log:
- level: info
- path: "/var/logs/crawlab"
- isDeletePeriodically: "N"
- deleteFrequency: "@hourly"
server:
host: 0.0.0.0
port: 8000
- master: "Y"
- secret: "crawlab"
- register:
- # type 填 mac/ip/customName, 如果是ip,则需要手动指定IP, 如果是 customName, 需填写你的 customNodeName
- type: "mac"
- customNodeName: "" # 自定义节点名称, default node1,只有在type = customName 时生效
- ip: ""
- lang: # 安装语言环境, Y 为安装,N 为不安装
- python: "Y"
- node: "N"
- java: "N"
- dotnet: "N"
- php: "N"
- scripts: "/app/backend/scripts"
+# master: true
+# secret: "crawlab"
+# register:
+# # type 填 mac/ip/customName, 如果是ip,则需要手动指定IP, 如果是 customName, 需填写你的 customNodeName
+# type: "mac"
+# customNodeName: "" # 自定义节点名称, default node1,只有在type = customName 时生效
+# ip: ""
+# lang: # 安装语言环境, Y 为安装,N 为不安装
+# python: "Y"
+# node: "N"
+# java: "N"
+# dotnet: "N"
+# php: "N"
+# scripts: "/app/backend/scripts"
spider:
- path: "/app/spiders"
+ fs: "/spiders"
+ workspace: "/workspace"
+ repo: "/repo"
task:
workers: 16
-other:
- tmppath: "/tmp"
-version: 0.5.1
-setting:
- crawlabLogToES: "N" # Send crawlab runtime log to ES, open this option "Y", remember to set esClient
- crawlabLogIndex: "crawlab-log"
- allowRegister: "N"
- enableTutorial: "N"
- runOnMaster: "Y"
- demoSpiders: "N"
- checkScrapy: "Y"
- autoInstall: "Y"
- esClient: "" # Your ES client, for example, http://192.168.1.1:9200 or http://your-domain.com, if not use es, set empty
- spiderLogIndex: "spider-log" # Index pattern for kibana, need to config on kibana
-notification:
- mail:
- server: ''
- port: ''
- senderEmail: ''
- senderIdentity: ''
- smtp:
- user: ''
- password: ''
-repo:
- apiUrl: "https://center.crawlab.cn/api"
-# apiUrl: "http://localhost:8002"
- ossUrl: "https://repo.crawlab.cn"
\ No newline at end of file
+ cancelWaitSeconds: 30
+# TODO: implement
+#setting:
+# crawlabLogToES: "N" # Send crawlab runtime log to ES, open this option "Y", remember to set esClient
+# crawlabLogIndex: "crawlab-log"
+# allowRegister: "N"
+# enableTutorial: "N"
+# runOnMaster: "Y"
+# demoSpiders: "N"
+# checkScrapy: "Y"
+# autoInstall: "Y"
+# esClient: "" # Your ES client, for example, http://192.168.1.1:9200 or http://your-domain.com, if not use es, set empty
+# spiderLogIndex: "spider-log" # Index pattern for kibana, need to config on kibana
+# TODO: implement
+#notification:
+# mail:
+# server: ''
+# port: ''
+# senderEmail: ''
+# senderIdentity: ''
+# smtp:
+# user: ''
+# password: ''
+grpc:
+ address: localhost:9666
+ server:
+ address: 0.0.0.0:9666
+ authKey: Crawlab2021!
+fs:
+ filer:
+ proxy: http://localhost:8888
+ url: http://localhost:8000/filer
+ authKey: Crawlab2021!
diff --git a/backend/config/config.go b/backend/config/config.go
deleted file mode 100644
index 79be808e..00000000
--- a/backend/config/config.go
+++ /dev/null
@@ -1,57 +0,0 @@
-package config
-
-import (
- "github.com/fsnotify/fsnotify"
- "github.com/spf13/viper"
- "log"
- "strings"
-)
-
-type Config struct {
- Name string
-}
-
-// 监控配置文件变化并热加载程序
-func (c *Config) WatchConfig() {
- viper.WatchConfig()
- viper.OnConfigChange(func(e fsnotify.Event) {
- log.Printf("Config file changed: %s", e.Name)
- })
-}
-
-func (c *Config) Init() error {
- if c.Name != "" {
- viper.SetConfigFile(c.Name) // 如果指定了配置文件,则解析指定的配置文件
- } else {
- viper.AddConfigPath("./conf") // 如果没有指定配置文件,则解析默认的配置文件
- viper.SetConfigName("config")
- }
- viper.SetConfigType("yaml") // 设置配置文件格式为YAML
- viper.AutomaticEnv() // 读取匹配的环境变量
- viper.SetEnvPrefix("CRAWLAB") // 读取环境变量的前缀为CRAWLAB
- replacer := strings.NewReplacer(".", "_")
- viper.SetEnvKeyReplacer(replacer)
- if err := viper.ReadInConfig(); err != nil { // viper解析配置文件
- return err
- }
-
- return nil
-}
-
-func InitConfig(cfg string) error {
- c := Config{
- Name: cfg,
- }
-
- // 初始化配置文件
- if err := c.Init(); err != nil {
- return err
- }
-
- // 监控配置文件变化并热加载程序
- c.WatchConfig()
-
- return nil
-}
-
-
diff --git a/backend/config/config_test.go b/backend/config/config_test.go
deleted file mode 100644
index 0068e6ad..00000000
--- a/backend/config/config_test.go
+++ /dev/null
@@ -1,16 +0,0 @@
-package config
-
-import (
- . "github.com/smartystreets/goconvey/convey"
- "testing"
-)
-
-func TestInitConfig(t *testing.T) {
- Convey("Test InitConfig func", t, func() {
- x := InitConfig("../conf/config.yml")
-
- Convey("The value should be nil", func() {
- So(x, ShouldEqual, nil)
- })
- })
-}
diff --git a/backend/constants/action.go b/backend/constants/action.go
deleted file mode 100644
index 389a11bc..00000000
--- a/backend/constants/action.go
+++ /dev/null
@@ -1,8 +0,0 @@
-package constants
-
-const (
- ActionTypeVisit = "visit"
- ActionTypeInstallDep = "install_dep"
- ActionTypeInstallLang = "install_lang"
- ActionTypeViewDisclaimer = "view_disclaimer"
-)
diff --git a/backend/constants/anchor.go b/backend/constants/anchor.go
deleted file mode 100644
index f462135f..00000000
--- a/backend/constants/anchor.go
+++ /dev/null
@@ -1,8 +0,0 @@
-package constants
-
-const (
- AnchorStartStage = "START_STAGE"
- AnchorStartUrl = "START_URL"
- AnchorItems = "ITEMS"
- AnchorParsers = "PARSERS"
-)
diff --git a/backend/constants/auth.go b/backend/constants/auth.go
deleted file mode 100644
index 136391a0..00000000
--- a/backend/constants/auth.go
+++ /dev/null
@@ -1,7 +0,0 @@
-package constants
-
-const (
- OwnerTypeAll = "all"
- OwnerTypeMe = "me"
- OwnerTypePublic = "public"
-)
diff --git a/backend/constants/challenge.go b/backend/constants/challenge.go
deleted file mode 100644
index 5c056e91..00000000
--- a/backend/constants/challenge.go
+++ /dev/null
@@ -1,20 +0,0 @@
-package constants
-
-const (
- ChallengeLogin7d = "login_7d"
- ChallengeLogin30d = "login_30d"
- ChallengeLogin90d = "login_90d"
- ChallengeLogin180d = "login_180d"
- ChallengeCreateCustomizedSpider = "create_customized_spider"
- ChallengeCreateConfigurableSpider = "create_configurable_spider"
- ChallengeCreateSchedule = "create_schedule"
- ChallengeCreateNodes = "create_nodes"
- ChallengeCreateUser = "create_user"
- ChallengeRunRandom = "run_random"
- ChallengeScrape1k = "scrape_1k"
- ChallengeScrape10k = "scrape_10k"
- ChallengeScrape100k = "scrape_100k"
- ChallengeInstallDep = "install_dep"
- ChallengeInstallLang = "install_lang"
- ChallengeViewDisclaimer = "view_disclaimer"
-)
diff --git a/backend/constants/channels.go b/backend/constants/channels.go
deleted file mode 100644
index c38a5ac9..00000000
--- a/backend/constants/channels.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package constants
-
-const (
- ChannelAllNode = "nodes:public"
-
- ChannelWorkerNode = "nodes:"
-
- ChannelMasterNode = "nodes:master"
-)
diff --git a/backend/constants/common.go b/backend/constants/common.go
deleted file mode 100644
index 9ac6cdbc..00000000
--- a/backend/constants/common.go
+++ /dev/null
@@ -1,6 +0,0 @@
-package constants
-
-const (
- ASCENDING = "ascending"
- DESCENDING = "descending"
-)
diff --git a/backend/constants/config_spider.go b/backend/constants/config_spider.go
deleted file mode 100644
index c29624dc..00000000
--- a/backend/constants/config_spider.go
+++ /dev/null
@@ -1,6 +0,0 @@
-package constants
-
-const (
- EngineScrapy = "scrapy"
- EngineColly = "colly"
-)
diff --git a/backend/constants/context.go b/backend/constants/context.go
deleted file mode 100644
index 0759b54b..00000000
--- a/backend/constants/context.go
+++ /dev/null
@@ -1,5 +0,0 @@
-package constants
-
-const (
- ContextUser = "currentUser"
-)
diff --git a/backend/constants/errors.go b/backend/constants/errors.go
deleted file mode 100644
index a273cb75..00000000
--- a/backend/constants/errors.go
+++ /dev/null
@@ -1,13 +0,0 @@
-package constants
-
-import (
- "crawlab/errors"
- "net/http"
-)
-
-var (
- ErrorMongoError = errors.NewSystemOPError(1001, "system error:[mongo]%s", http.StatusInternalServerError)
- //users
- ErrorUserNotFound = errors.NewBusinessError(10001, "user not found.", http.StatusUnauthorized)
- ErrorUsernameOrPasswordInvalid = errors.NewBusinessError(11001, "username or password invalid", http.StatusUnauthorized)
-)
diff --git a/backend/constants/log.go b/backend/constants/log.go
deleted file mode 100644
index 5f0b4a66..00000000
--- a/backend/constants/log.go
+++ /dev/null
@@ -1,5 +0,0 @@
-package constants
-
-const (
- ErrorRegexPattern = "(?:[ :,.]|^)((?:error|exception|traceback)s?)(?:[ :,.]|$)"
-)
diff --git a/backend/constants/message.go b/backend/constants/message.go
deleted file mode 100644
index 72e5fab2..00000000
--- a/backend/constants/message.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package constants
-
-const (
- MsgTypeGetLog = "get-log"
- MsgTypeGetSystemInfo = "get-sys-info"
- MsgTypeCancelTask = "cancel-task"
- MsgTypeRemoveLog = "remove-log"
- MsgTypeRemoveSpider = "remove-spider"
-)
diff --git a/backend/constants/model.go b/backend/constants/model.go
deleted file mode 100644
index da66b15f..00000000
--- a/backend/constants/model.go
+++ /dev/null
@@ -1,6 +0,0 @@
-package constants
-
-const (
- ObjectIdNull = "000000000000000000000000"
- Infinite = 999999999
-)
diff --git a/backend/constants/node.go b/backend/constants/node.go
deleted file mode 100644
index 29b0b7c1..00000000
--- a/backend/constants/node.go
+++ /dev/null
@@ -1,6 +0,0 @@
-package constants
-
-const (
- StatusOnline = "online"
- StatusOffline = "offline"
-)
diff --git a/backend/constants/notification.go b/backend/constants/notification.go
deleted file mode 100644
index cf3da062..00000000
--- a/backend/constants/notification.go
+++ /dev/null
@@ -1,13 +0,0 @@
-package constants
-
-const (
- NotificationTriggerOnTaskEnd = "notification_trigger_on_task_end"
- NotificationTriggerOnTaskError = "notification_trigger_on_task_error"
- NotificationTriggerNever = "notification_trigger_never"
-)
-
-const (
- NotificationTypeMail = "notification_type_mail"
- NotificationTypeDingTalk = "notification_type_ding_talk"
- NotificationTypeWechat = "notification_type_wechat"
-)
diff --git a/backend/constants/register.go b/backend/constants/register.go
deleted file mode 100644
index 4ed1e396..00000000
--- a/backend/constants/register.go
+++ /dev/null
@@ -1,8 +0,0 @@
-package constants
-
-const (
- RegisterTypeMac = "mac"
- RegisterTypeIp = "ip"
- RegisterTypeHostname = "hostname"
- RegisterTypeCustomName = "customName"
-)
diff --git a/backend/constants/rpc.go b/backend/constants/rpc.go
deleted file mode 100644
index 0fd7ad9f..00000000
--- a/backend/constants/rpc.go
+++ /dev/null
@@ -1,12 +0,0 @@
-package constants
-
-const (
- RpcInstallLang = "install_lang"
- RpcInstallDep = "install_dep"
- RpcUninstallDep = "uninstall_dep"
- RpcGetInstalledDepList = "get_installed_dep_list"
- RpcGetLang = "get_lang"
- RpcCancelTask = "cancel_task"
- RpcGetSystemInfoService = "get_system_info"
- RpcRemoveSpider = "remove_spider"
-)
diff --git a/backend/constants/schedule.go b/backend/constants/schedule.go
deleted file mode 100644
index 520626a9..00000000
--- a/backend/constants/schedule.go
+++ /dev/null
@@ -1,10 +0,0 @@
-package constants
-
-const (
- ScheduleStatusStop = "stopped"
- ScheduleStatusRunning = "running"
- ScheduleStatusError = "error"
-
- ScheduleStatusErrorNotFoundNode = "Not Found Node"
- ScheduleStatusErrorNotFoundSpider = "Not Found Spider"
-)
diff --git a/backend/constants/scrapy.go b/backend/constants/scrapy.go
deleted file mode 100644
index bc82508f..00000000
--- a/backend/constants/scrapy.go
+++ /dev/null
@@ -1,5 +0,0 @@
-package constants
-
-const ScrapyProtectedStageNames = ""
-
-const ScrapyProtectedFieldNames = "_id,task_id,ts"
diff --git a/backend/constants/spider.go b/backend/constants/spider.go
deleted file mode 100644
index 5119aa67..00000000
--- a/backend/constants/spider.go
+++ /dev/null
@@ -1,7 +0,0 @@
-package constants
-
-const (
- Customized = "customized"
- Configurable = "configurable"
- Plugin = "plugin"
-)
diff --git a/backend/constants/system.go b/backend/constants/system.go
deleted file mode 100644
index 14c45698..00000000
--- a/backend/constants/system.go
+++ /dev/null
@@ -1,25 +0,0 @@
-package constants
-
-const (
- Windows = "windows"
- Linux = "linux"
- Darwin = "darwin"
-)
-
-const (
- Python = "python"
- Nodejs = "node"
- Java = "java"
-)
-
-const (
- InstallStatusNotInstalled = "not-installed"
- InstallStatusInstalling = "installing"
- InstallStatusInstallingOther = "installing-other"
- InstallStatusInstalled = "installed"
-)
-
-const (
- LangTypeLang = "lang"
- LangTypeWebDriver = "webdriver"
-)
diff --git a/backend/constants/task.go b/backend/constants/task.go
deleted file mode 100644
index 08539432..00000000
--- a/backend/constants/task.go
+++ /dev/null
@@ -1,32 +0,0 @@
-package constants
-
-const (
- // 调度中
- StatusPending string = "pending"
- // 运行中
- StatusRunning string = "running"
- // 已完成
- StatusFinished string = "finished"
- // 错误
- StatusError string = "error"
- // 取消
- StatusCancelled string = "cancelled"
- // 节点重启导致的异常终止
- StatusAbnormal string = "abnormal"
-)
-
-const (
- TaskFinish string = "finish"
- TaskCancel string = "cancel"
-)
-
-const (
- RunTypeAllNodes string = "all-nodes"
- RunTypeRandom string = "random"
- RunTypeSelectedNodes string = "selected-nodes"
-)
-
-const (
- TaskTypeSpider string = "spider"
- TaskTypeSystem string = "system"
-)
diff --git a/backend/constants/user.go b/backend/constants/user.go
deleted file mode 100644
index bc225c94..00000000
--- a/backend/constants/user.go
+++ /dev/null
@@ -1,6 +0,0 @@
-package constants
-
-const (
- RoleAdmin = "admin"
- RoleNormal = "normal"
-)
diff --git a/backend/constants/variable.go b/backend/constants/variable.go
deleted file mode 100644
index 713fbe2d..00000000
--- a/backend/constants/variable.go
+++ /dev/null
@@ -1,9 +0,0 @@
-package constants
-
-const (
- String = "string"
- Number = "number"
- Boolean = "boolean"
- Array = "array"
- Object = "object"
-)
diff --git a/backend/data/challenge_data.json b/backend/data/challenge_data.json
deleted file mode 100644
index 5a51dc33..00000000
--- a/backend/data/challenge_data.json
+++ /dev/null
@@ -1,142 +0,0 @@
-[
- {
- "name": "login_7d",
- "title_cn": "连续登录 7 天",
- "title_en": "Logged-in for 7 days",
- "description_cn": "连续 7 天登录 Crawlab,即可完成挑战!",
- "description_en": "Logged-in for consecutive 7 days to complete the challenge",
- "difficulty": 1
- },
- {
- "name": "login_30d",
- "title_cn": "连续登录 30 天",
- "title_en": "Logged-in for 30 days",
- "description_cn": "连续 30 天登录 Crawlab,即可完成挑战!",
- "description_en": "Logged-in for consecutive 30 days to complete the challenge",
- "difficulty": 2
- },
- {
- "name": "login_90d",
- "title_cn": "连续登录 90 天",
- "title_en": "Logged-in for 90 days",
- "description_cn": "连续 90 天登录 Crawlab,即可完成挑战!",
- "description_en": "Logged-in for consecutive 90 days to complete the challenge",
- "difficulty": 3
- },
- {
- "name": "login_180d",
- "title_cn": "连续登录 180 天",
- "title_en": "Logged-in for 180 days",
- "description_cn": "连续 180 天登录 Crawlab,即可完成挑战!",
- "description_en": "Logged-in for consecutive 180 days to complete the challenge",
- "difficulty": 4
- },
- {
- "name": "create_customized_spider",
- "title_cn": "创建 1 个自定义爬虫",
- "title_en": "Create a customized spider",
- "description_cn": "在爬虫列表中,点击 '添加爬虫',选择 '自定义爬虫',输入相应的参数,点击添加,即可完成挑战!",
- "description_en": "In Spider List page, click 'Add Spider', select 'Customized Spider', enter params, click 'Add' to finish the challenge.",
- "difficulty": 1,
- "path": "/spiders"
- },
- {
- "name": "create_configurable_spider",
- "title_cn": "创建 1 个可配置爬虫",
- "title_en": "Create a configurable spider",
- "description_cn": "在爬虫列表中,点击 '添加爬虫',选择 '可配置爬虫',输入相应的参数,点击添加,即可完成挑战!",
- "description_en": "In Spider List page, click 'Add Spider', select 'Configurable Spider', enter params, click 'Add' to finish the challenge.",
- "difficulty": 1,
- "path": "/spiders"
- },
- {
- "name": "run_random",
- "title_cn": "用随机模式成功运行爬虫",
- "title_en": "Run a spider in random mode successfully",
- "description_cn": "在您创建好的爬虫中,导航到其对应的详情页(爬虫列表中点击爬虫),选择随机模式运行一个爬虫,并能运行成功。",
- "description_en": "In your created spiders, navigate to corresponding detail page (click spider in Spider List page), run a spider in random mode successfully.",
- "difficulty": 1,
- "path": "/spiders"
- },
- {
- "name": "scrape_1k",
- "title_cn": "抓取 1 千条数据",
- "title_en": "Scrape 1k records",
- "description_cn": "运行您创建好的爬虫,抓取 1 千条及以上的结果数据,即可完成挑战!",
- "description_en": "Run your created spiders, scrape 1k and more results to finish the challenge.",
- "difficulty": 2,
- "path": "/spiders"
- },
- {
- "name": "scrape_10k",
- "title_cn": "抓取 1 万条数据",
- "title_en": "Scrape 10k records",
- "description_cn": "运行您创建好的爬虫,抓取 1 万条及以上的结果数据,即可完成挑战!",
- "description_en": "Run your created spiders, scrape 10k and more results to finish the challenge.",
- "difficulty": 3,
- "path": "/spiders"
- },
- {
- "name": "scrape_100k",
- "title_cn": "抓取 10 万条数据",
- "title_en": "Scrape 100k records",
- "description_cn": "运行您创建好的爬虫,抓取 10 万条及以上的结果数据,即可完成挑战!",
- "description_en": "Run your created spiders, scrape 100k and more results to finish the challenge.",
- "difficulty": 4,
- "path": "/spiders"
- },
- {
- "name": "create_schedule",
- "title_cn": "创建 1 个定时任务",
- "title_en": "Create a schedule",
- "description_cn": "在定时任务列表中,创建一个定时任务,正确设置好 Cron 表达式,即可完成挑战!",
- "description_en": "In Schedule List page, create a schedule and configure cron expression to finish the task.",
- "difficulty": 1,
- "path": "/schedules"
- },
- {
- "name": "create_nodes",
- "title_cn": "创建 1 个节点集群",
- "title_en": "Create a node cluster",
- "description_cn": "按照文档的部署指南,部署含有 3 个节点的集群,即可完成挑战!",
- "description_en": "Deploy a 3-node cluster according to the deployment guidance in documentation to finish the task.",
- "difficulty": 3,
- "path": "/nodes"
- },
- {
- "name": "install_dep",
- "title_cn": "安装 1 个依赖",
- "title_en": "Install a dependency successfully",
- "description_cn": "在 '节点列表->安装' 或 '节点详情->安装' 中,搜索并安装所需的 1 个依赖,即可完成挑战!",
- "description_en": "In 'Node List -> Installation' or 'Node Detail -> Installation', search and install a dependency.",
- "difficulty": 3,
- "path": "/nodes"
- },
- {
- "name": "install_lang",
- "title_cn": "安装 1 个语言环境",
- "title_en": "Install a language successfully",
- "description_cn": "在 '节点列表->安装' 或 '节点详情->安装' 中,点击安装所需的 1 个语言环境,即可完成挑战!",
- "description_en": "In 'Node List -> Installation' or 'Node Detail -> Installation', install a language.",
- "difficulty": 3,
- "path": "/nodes"
- },
- {
- "name": "view_disclaimer",
- "title_cn": "阅读免责声明",
- "title_en": "View disclaimer",
- "description_cn": "在左侧菜单栏,点击 '免责声明' 查看其内容,即可完成挑战!",
- "description_en": "In the left side menu, click 'Disclaimer' and view its content to finish the challenge.",
- "difficulty": 1,
- "path": "/disclaimer"
- },
- {
- "name": "create_user",
- "title_cn": "创建 1 个用户",
- "title_en": "Create a user",
- "description_cn": "在用户管理页面中创建一个新用户,即可完成挑战!",
- "description_en": "In User Admin page, create a new user to finish the challenge.",
- "difficulty": 1,
- "path": "/users"
- }
-]
\ No newline at end of file
diff --git a/backend/database/es_base.go b/backend/database/es_base.go
deleted file mode 100644
index b255958a..00000000
--- a/backend/database/es_base.go
+++ /dev/null
@@ -1,44 +0,0 @@
-package database
-
-import (
- "context"
- "github.com/apex/log"
- "github.com/olivere/elastic/v7"
- "github.com/satori/go.uuid"
- "github.com/spf13/viper"
- "sync"
- "time"
-)
-
-var doOnce sync.Once
-var ctx context.Context
-var ESClient *elastic.Client
-
-func InitEsClient() {
- esClientStr := viper.GetString("setting.esClient")
- ctx = context.Background()
- ESClient, _ = elastic.NewClient(elastic.SetURL(esClientStr), elastic.SetSniff(false))
-}
-
-// WriteMsg will write the msg and level into es
-func WriteMsgToES(when time.Time, msg chan string, index string) {
- doOnce.Do(InitEsClient)
- vals := make(map[string]interface{})
- vals["@timestamp"] = when.Format(time.RFC3339)
- for {
- select {
- case vals["@msg"] = <-msg:
- uid := uuid.NewV4().String()
- _, err := ESClient.Index().Index(index).Id(uid).BodyJson(vals).Refresh("wait_for").Do(ctx)
- if err != nil {
- log.Error(err.Error())
- log.Error("send msg log to es error")
- return
- }
- case <-time.After(6 * time.Second):
- return
- }
- }
-
- return
-}
diff --git a/backend/database/mongo.go b/backend/database/mongo.go
deleted file mode 100644
index 01204a70..00000000
--- a/backend/database/mongo.go
+++ /dev/null
@@ -1,112 +0,0 @@
-package database
-
-import (
- "crawlab/constants"
- "github.com/apex/log"
- "github.com/cenkalti/backoff/v4"
- "github.com/globalsign/mgo"
- "github.com/globalsign/mgo/bson"
- "github.com/spf13/viper"
- "net"
- "reflect"
- "time"
-)
-
-var Session *mgo.Session
-
-func GetSession() *mgo.Session {
- return Session.Copy()
-}
-
-func GetDb() (*mgo.Session, *mgo.Database) {
- s := GetSession()
- return s, s.DB(viper.GetString("mongo.db"))
-}
-
-func GetCol(collectionName string) (*mgo.Session, *mgo.Collection) {
- s := GetSession()
- db := s.DB(viper.GetString("mongo.db"))
- col := db.C(collectionName)
- return s, col
-}
-
-func GetGridFs(prefix string) (*mgo.Session, *mgo.GridFS) {
- s, db := GetDb()
- gf := db.GridFS(prefix)
- return s, gf
-}
-
-func FillNullObjectId(doc interface{}) {
- t := reflect.TypeOf(doc)
- if t.Kind() == reflect.Ptr {
- t = t.Elem()
- }
- if t.Kind() != reflect.Struct {
- return
- }
- v := reflect.ValueOf(doc)
- for i := 0; i < t.NumField(); i++ {
- ft := t.Field(i)
- fv := v.Elem().Field(i)
- val := fv.Interface()
- switch val.(type) {
- case bson.ObjectId:
- if !val.(bson.ObjectId).Valid() {
- v.FieldByName(ft.Name).Set(reflect.ValueOf(bson.ObjectIdHex(constants.ObjectIdNull)))
- }
- }
- }
-}
-
-func InitMongo() error {
- var mongoHost = viper.GetString("mongo.host")
- var mongoPort = viper.GetString("mongo.port")
- var mongoDb = viper.GetString("mongo.db")
- var mongoUsername = viper.GetString("mongo.username")
- var mongoPassword = viper.GetString("mongo.password")
- var mongoAuth = viper.GetString("mongo.authSource")
-
- if Session == nil {
- var dialInfo mgo.DialInfo
- addr := net.JoinHostPort(mongoHost, mongoPort)
- timeout := time.Second * 10
- dialInfo = mgo.DialInfo{
- Addrs: []string{addr},
- Timeout: timeout,
- Database: mongoDb,
- PoolLimit: 100,
- PoolTimeout: timeout,
- ReadTimeout: timeout,
- WriteTimeout: timeout,
- AppName: "crawlab",
- FailFast: true,
- MinPoolSize: 10,
- MaxIdleTimeMS: 1000 * 30,
- }
- if mongoUsername != "" {
- dialInfo.Username = mongoUsername
- dialInfo.Password = mongoPassword
- dialInfo.Source = mongoAuth
- }
- bp := backoff.NewExponentialBackOff()
- var err error
-
- err = backoff.Retry(func() error {
- Session, err = mgo.DialWithInfo(&dialInfo)
- if err != nil {
- log.WithError(err).Warnf("waiting for connect mongo database, after %f seconds try again.", bp.NextBackOff().Seconds())
- }
- return err
- }, bp)
- }
- //Add Unique index for 'key'
- keyIndex := mgo.Index{
- Key: []string{"key"},
- Unique: true,
- }
- s, c := GetCol("nodes")
- defer s.Close()
- c.EnsureIndex(keyIndex)
-
- return nil
-}
diff --git a/backend/database/mongo_test.go b/backend/database/mongo_test.go
deleted file mode 100644
index ed6044ee..00000000
--- a/backend/database/mongo_test.go
+++ /dev/null
@@ -1,70 +0,0 @@
-package database
-
-import (
- "crawlab/config"
- "github.com/apex/log"
- "github.com/globalsign/mgo"
- . "github.com/smartystreets/goconvey/convey"
- "github.com/spf13/viper"
- "reflect"
- "testing"
-)
-
-func init() {
- if err := config.InitConfig("../conf/config.yml"); err != nil {
- log.Fatal("Init config failed")
- }
- log.Infof("初始化配置成功")
- err := InitMongo()
- if err != nil {
- log.Fatal("Init mongodb failed")
- }
-
-}
-
-func TestGetDb(t *testing.T) {
- Convey("Test GetDb", t, func() {
- if err := config.InitConfig("../conf/config.yml"); err != nil {
- t.Fatal("Init config failed")
- }
- t.Log("初始化配置成功")
- err := InitMongo()
- if err != nil {
- t.Fatal("Init mongodb failed")
- }
- s, db := GetDb()
- Convey("The value should be Session.Copy", func() {
- So(s, ShouldResemble, Session.Copy())
- })
- Convey("The value should be reference of database", func() {
- So(db, ShouldResemble, s.DB(viper.GetString("mongo.db")))
- })
- })
-}
-
-func TestGetCol(t *testing.T) {
- var c = "nodes"
- var colActual *mgo.Collection
- Convey("Test GetCol", t, func() {
- s, col := GetCol(c)
- Convey("s should resemble Session.Copy", func() {
- So(s, ShouldResemble, Session.Copy())
- So(reflect.TypeOf(col), ShouldResemble, reflect.TypeOf(colActual))
- })
- })
-}
-
-func TestGetGridFs(t *testing.T) {
- var prefix = "files"
- var gfActual *mgo.GridFS
-
- Convey("Test GetGridFs", t, func() {
- s, gf := GetGridFs(prefix)
- Convey("s should be session.copy", func() {
- So(s, ShouldResemble, Session.Copy())
- })
- Convey("gf should be *mgo.GridFS", func() {
- So(reflect.TypeOf(gf), ShouldResemble, reflect.TypeOf(gfActual))
- })
- })
-}
diff --git a/backend/database/pubsub.go b/backend/database/pubsub.go
deleted file mode 100644
index f9eae535..00000000
--- a/backend/database/pubsub.go
+++ /dev/null
@@ -1,96 +0,0 @@
-package database
-
-import (
- "context"
- "crawlab/utils"
- "fmt"
- "github.com/apex/log"
- "github.com/gomodule/redigo/redis"
- errors2 "github.com/pkg/errors"
- "time"
-)
-
-type ConsumeFunc func(message redis.Message) error
-
-func (r *Redis) Close() {
- err := r.pool.Close()
- if err != nil {
- log.Errorf("redis close error.")
- }
-}
-func (r *Redis) subscribe(ctx context.Context, consume ConsumeFunc, channel ...string) error {
- psc := redis.PubSubConn{Conn: r.pool.Get()}
- if err := psc.Subscribe(redis.Args{}.AddFlat(channel)...); err != nil {
- return err
- }
- done := make(chan error, 1)
- tick := time.NewTicker(time.Second * 3)
- defer tick.Stop()
- go func() {
- defer utils.Close(psc)
- for {
- switch msg := psc.Receive().(type) {
- case error:
- done <- fmt.Errorf("redis pubsub receive err: %v", msg)
- return
- case redis.Message:
- if err := consume(msg); err != nil {
- fmt.Printf("redis pubsub consume message err: %v", err)
- continue
- }
- case redis.Subscription:
- if msg.Count == 0 {
- // all channels are unsubscribed
- return
- }
- }
-
- }
- }()
- // start a new goroutine to receive message
- for {
- select {
- case <-ctx.Done():
- if err := psc.Unsubscribe(); err != nil {
- fmt.Printf("redis pubsub unsubscribe err: %v \n", err)
- }
- done <- nil
- case <-tick.C:
- if err := psc.Ping(""); err != nil {
- fmt.Printf("ping message error: %s \n", err)
- //done <- err
- }
- case err := <-done:
- close(done)
- return err
- }
- }
-
-}
-func (r *Redis) Subscribe(ctx context.Context, consume ConsumeFunc, channel ...string) error {
- index := 0
- go func() {
- for {
- err := r.subscribe(ctx, consume, channel...)
- fmt.Println(err)
-
- if err == nil {
- index = 0
- break
- }
- time.Sleep(5 * time.Second)
- index += 1
- fmt.Printf("try reconnect %d times \n", index)
- }
- }()
- return nil
-}
-func (r *Redis) Publish(channel, message string) (n int, err error) {
- conn := r.pool.Get()
- defer utils.Close(conn)
- n, err = redis.Int(conn.Do("PUBLISH", channel, message))
- if err != nil {
- return 0, errors2.Wrapf(err, "redis publish %s %s", channel, message)
- }
- return
-}
diff --git a/backend/database/redis.go b/backend/database/redis.go
deleted file mode 100644
index c14b9a16..00000000
--- a/backend/database/redis.go
+++ /dev/null
@@ -1,290 +0,0 @@
-package database
-
-import (
- "context"
- "crawlab/entity"
- "crawlab/utils"
- "errors"
- "github.com/apex/log"
- "github.com/cenkalti/backoff/v4"
- "github.com/gomodule/redigo/redis"
- "github.com/spf13/viper"
- "net/url"
- "runtime/debug"
- "strings"
- "time"
-)
-
-var RedisClient *Redis
-
-type Redis struct {
- pool *redis.Pool
-}
-
-type Mutex struct {
- Name string
- expiry time.Duration
- tries int
- delay time.Duration
- value string
-}
-
-func NewRedisClient() *Redis {
- return &Redis{pool: NewRedisPool()}
-}
-
-func (r *Redis) RPush(collection string, value interface{}) error {
- c := r.pool.Get()
- defer utils.Close(c)
-
- if _, err := c.Do("RPUSH", collection, value); err != nil {
- log.Error(err.Error())
- debug.PrintStack()
- return err
- }
- return nil
-}
-
-func (r *Redis) LPush(collection string, value interface{}) error {
- c := r.pool.Get()
- defer utils.Close(c)
-
- if _, err := c.Do("RPUSH", collection, value); err != nil {
- log.Error(err.Error())
- debug.PrintStack()
- return err
- }
- return nil
-}
-
-func (r *Redis) LPop(collection string) (string, error) {
- c := r.pool.Get()
- defer utils.Close(c)
-
- value, err2 := redis.String(c.Do("LPOP", collection))
- if err2 != nil {
- return value, err2
- }
- return value, nil
-}
-
-func (r *Redis) HSet(collection string, key string, value string) error {
- c := r.pool.Get()
- defer utils.Close(c)
-
- if _, err := c.Do("HSET", collection, key, value); err != nil {
- log.Error(err.Error())
- debug.PrintStack()
- return err
- }
- return nil
-}
-func (r *Redis) Ping() error {
- c := r.pool.Get()
- defer utils.Close(c)
- _, err2 := redis.String(c.Do("PING"))
- return err2
-}
-func (r *Redis) HGet(collection string, key string) (string, error) {
- c := r.pool.Get()
- defer utils.Close(c)
- value, err2 := redis.String(c.Do("HGET", collection, key))
- if err2 != nil && err2 != redis.ErrNil {
- log.Error(err2.Error())
- debug.PrintStack()
- return value, err2
- }
- return value, nil
-}
-
-func (r *Redis) HDel(collection string, key string) error {
- c := r.pool.Get()
- defer utils.Close(c)
-
- if _, err := c.Do("HDEL", collection, key); err != nil {
- log.Error(err.Error())
- debug.PrintStack()
- return err
- }
- return nil
-}
-func (r *Redis) HScan(collection string) (results []string, err error) {
- c := r.pool.Get()
- defer utils.Close(c)
- var (
- cursor int64
- items []string
- )
-
- for {
- values, err := redis.Values(c.Do("HSCAN", collection, cursor))
- if err != nil {
- return results, err
- }
-
- values, err = redis.Scan(values, &cursor, &items)
- if err != nil {
- return results, err
- }
- for i := 0; i < len(items); i += 2 {
- cur := items[i+1]
- results = append(results, cur)
- }
- if cursor == 0 {
- break
- }
- }
- return results, err
-
-}
-func (r *Redis) HKeys(collection string) ([]string, error) {
- c := r.pool.Get()
- defer utils.Close(c)
-
- value, err2 := redis.Strings(c.Do("HKEYS", collection))
- if err2 != nil {
- log.Error(err2.Error())
- debug.PrintStack()
- return []string{}, err2
- }
- return value, nil
-}
-
-func (r *Redis) BRPop(collection string, timeout int) (string, error) {
- if timeout <= 0 {
- timeout = 60
- }
- c := r.pool.Get()
- defer utils.Close(c)
-
- values, err := redis.Strings(c.Do("BRPOP", collection, timeout))
- if err != nil {
- return "", err
- }
- return values[1], nil
-}
-
-func NewRedisPool() *redis.Pool {
- var address = viper.GetString("redis.address")
- var port = viper.GetString("redis.port")
- var database = viper.GetString("redis.database")
- var password = url.QueryEscape(viper.GetString("redis.password"))
-
- var redisUrl string
- if password == "" {
- redisUrl = "redis://" + address + ":" + port + "/" + database
- } else {
- redisUrl = "redis://x:" + password + "@" + address + ":" + port + "/" + database
- }
- return &redis.Pool{
- Dial: func() (conn redis.Conn, e error) {
- return redis.DialURL(redisUrl,
- redis.DialConnectTimeout(time.Second*10),
- redis.DialReadTimeout(time.Second*600),
- redis.DialWriteTimeout(time.Second*10),
- )
- },
- TestOnBorrow: func(c redis.Conn, t time.Time) error {
- if time.Since(t) < time.Minute {
- return nil
- }
- _, err := c.Do("PING")
- return err
- },
- MaxIdle: 10,
- MaxActive: 0,
- IdleTimeout: 300 * time.Second,
- Wait: false,
- MaxConnLifetime: 0,
- }
-}
-
-func InitRedis() error {
- RedisClient = NewRedisClient()
- b := backoff.NewExponentialBackOff()
- b.MaxInterval = 20 * time.Second
- err := backoff.Retry(func() error {
- err := RedisClient.Ping()
-
- if err != nil {
- log.WithError(err).Warnf("waiting for redis pool active connection. will after %f seconds try again.", b.NextBackOff().Seconds())
- }
- return err
- }, b)
- return err
-}
-
-func Pub(channel string, msg entity.NodeMessage) error {
- if _, err := RedisClient.Publish(channel, utils.GetJson(msg)); err != nil {
- log.Errorf("publish redis error: %s", err.Error())
- debug.PrintStack()
- return err
- }
- return nil
-}
-
-func Sub(channel string, consume ConsumeFunc) error {
- ctx := context.Background()
- if err := RedisClient.Subscribe(ctx, consume, channel); err != nil {
- log.Errorf("subscribe redis error: %s", err.Error())
- debug.PrintStack()
- return err
- }
- return nil
-}
-
-// 构建同步锁key
-func (r *Redis) getLockKey(lockKey string) string {
- lockKey = strings.ReplaceAll(lockKey, ":", "-")
- return "nodes:lock:" + lockKey
-}
-
-// 获得锁
-func (r *Redis) Lock(lockKey string) (int64, error) {
- c := r.pool.Get()
- defer utils.Close(c)
- lockKey = r.getLockKey(lockKey)
-
- ts := time.Now().Unix()
- ok, err := c.Do("SET", lockKey, ts, "NX", "PX", 30000)
- if err != nil {
- log.Errorf("get lock fail with error: %s", err.Error())
- debug.PrintStack()
- return 0, err
- }
- if ok == nil {
- log.Errorf("the lockKey is locked: key=%s", lockKey)
- return 0, errors.New("the lockKey is locked")
- }
- return ts, nil
-}
-
-func (r *Redis) UnLock(lockKey string, value int64) {
- c := r.pool.Get()
- defer utils.Close(c)
- lockKey = r.getLockKey(lockKey)
-
- getValue, err := redis.Int64(c.Do("GET", lockKey))
- if err != nil {
- log.Errorf("get lockKey error: %s", err.Error())
- debug.PrintStack()
- return
- }
-
- if getValue != value {
- log.Errorf("the lockKey value diff: %d, %d", value, getValue)
- return
- }
-
- v, err := redis.Int64(c.Do("DEL", lockKey))
- if err != nil {
- log.Errorf("unlock failed, error: %s", err.Error())
- debug.PrintStack()
- return
- }
-
- if v == 0 {
- log.Errorf("unlock failed: key=%s", lockKey)
- return
- }
-}
diff --git a/backend/docs/docs.go b/backend/docs/docs.go
deleted file mode 100644
index feaa3214..00000000
--- a/backend/docs/docs.go
+++ /dev/null
@@ -1,4808 +0,0 @@
-// GENERATED BY THE COMMAND ABOVE; DO NOT EDIT
-// This file was generated by swaggo/swag at
-// 2020-05-05 11:09:10.499886 +0800 CST m=+0.084916029
-
-package docs
-
-import (
- "bytes"
- "encoding/json"
- "strings"
-
- "github.com/alecthomas/template"
- "github.com/swaggo/swag"
-)
-
-var doc = `{
- "schemes": {{ marshal .Schemes }},
- "swagger": "2.0",
- "info": {
- "description": "{{.Description}}",
- "title": "{{.Title}}",
- "contact": {},
- "license": {},
- "version": "{{.Version}}"
- },
- "host": "{{.Host}}",
- "basePath": "{{.BasePath}}",
- "paths": {
- "/config_spiders": {
- "put": {
- "description": "Put config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Put config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders/{id}/config": {
- "get": {
- "description": "Get config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Get config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post config spider config",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Post config spider config",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders/{id}/spiderfile": {
- "post": {
- "description": "Post config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Post config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders/{id}/upload": {
- "post": {
- "description": "Upload config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Upload config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders_templates": {
- "get": {
- "description": "Get config spider template list",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Get config spider template list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/docs": {
- "get": {
- "description": "Get docs",
- "produces": [
- "application/json"
- ],
- "tags": [
- "docs"
- ],
- "summary": "Get docs",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/file": {
- "get": {
- "description": "Get file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "file"
- ],
- "summary": "Get file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes": {
- "get": {
- "description": "Get nodes",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get nodes",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}": {
- "get": {
- "description": "Get node",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post node",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Post node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "post node",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete node",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Delete node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps": {
- "get": {
- "description": "Get dep list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get dep list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "query",
- "required": true
- },
- {
- "type": "string",
- "description": "dep name",
- "name": "dep_name",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps/install": {
- "post": {
- "description": "Install dep",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Install dep",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps/installed": {
- "get": {
- "description": "Get installed dep list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get installed dep list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps/uninstall": {
- "post": {
- "description": "Uninstall dep",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Uninstall dep",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/langs": {
- "get": {
- "description": "Get language list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get language list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/langs/install": {
- "post": {
- "description": "Install language",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Install language",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/system": {
- "get": {
- "description": "Get system info",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get system info",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/tasks": {
- "get": {
- "description": "Get tasks on node",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get tasks on node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/projects": {
- "get": {
- "description": "Get projects",
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Get projects",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "projects",
- "name": "tag",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put project",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Put project",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "post project",
- "name": "p",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Project"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/projects/tags": {
- "get": {
- "description": "Get projects tags",
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Get project tags",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/projects/{id}": {
- "post": {
- "description": "Post project",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Post project",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "project id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "project item",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Project"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete project",
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Delete project",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "project id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/releases/latest": {
- "get": {
- "description": "Get latest release",
- "produces": [
- "application/json"
- ],
- "tags": [
- "version"
- ],
- "summary": "Get latest release",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules": {
- "get": {
- "description": "Get spider list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "page num",
- "name": "page_num",
- "in": "query"
- },
- {
- "type": "string",
- "description": "page size",
- "name": "page_size",
- "in": "query"
- },
- {
- "type": "string",
- "description": "keyword",
- "name": "keyword",
- "in": "query"
- },
- {
- "type": "string",
- "description": "project_id",
- "name": "project_id",
- "in": "query"
- },
- {
- "type": "string",
- "description": "type",
- "name": "type",
- "in": "query"
- },
- {
- "type": "string",
- "description": "sort_key",
- "name": "sort_key",
- "in": "query"
- },
- {
- "type": "string",
- "description": "sort_direction",
- "name": "sort_direction",
- "in": "query"
- },
- {
- "type": "string",
- "description": "owner_type",
- "name": "owner_type",
- "in": "query"
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Put schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "schedule item",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Schedule"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules/{id}": {
- "get": {
- "description": "Get schedule by id",
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Get schedule by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Post schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "schedule item",
- "name": "newItem",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Schedule"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete schedule",
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Delete schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules/{id}/disable": {
- "post": {
- "description": "disable schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "disable schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules/{id}/enable": {
- "post": {
- "description": "enable schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "enable schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/setting": {
- "get": {
- "description": "Get setting",
- "produces": [
- "application/json"
- ],
- "tags": [
- "setting"
- ],
- "summary": "Get setting",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders": {
- "put": {
- "description": "Put spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Put spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "delete spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "delete spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders-cancel": {
- "post": {
- "description": "cancel spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "cancel spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders-run": {
- "post": {
- "description": "run spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "run spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}": {
- "get": {
- "description": "Get spider by id",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "spider item",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete spider by id",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Delete spider by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/copy": {
- "post": {
- "description": "Copy spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Copy spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/dir": {
- "get": {
- "description": "Get spider dir",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider dir",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "path",
- "name": "path",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/file": {
- "get": {
- "description": "Get spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "path",
- "name": "path",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Post spider dir",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider dir",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Put spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Put spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Delete spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/file/rename": {
- "post": {
- "description": "Rename spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Rename spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/file/tree": {
- "get": {
- "description": "Get spider dir",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider dir",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/git/reset": {
- "post": {
- "description": "Post spider reset git",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider reset git",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/git/sync": {
- "post": {
- "description": "Post spider sync git",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider sync git",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/publish": {
- "post": {
- "description": "Publish spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Publish spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/schedules": {
- "get": {
- "description": "Get schedules",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get schedules",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/items": {
- "get": {
- "description": "Get scrapy spider items",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider items",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post scrapy spider items",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post scrapy spider items",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "req data",
- "name": "reqData",
- "in": "body",
- "required": true,
- "schema": {
- "type": "entity.ScrapyItem",
- "items": {
- "$ref": "#/definitions/entity.ScrapyItem"
- }
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/pipelines": {
- "get": {
- "description": "Get scrapy spider pipelines",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider pipelines",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/settings": {
- "get": {
- "description": "Get scrapy spider settings",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider settings",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Get scrapy spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "req data",
- "name": "reqData",
- "in": "body",
- "required": true,
- "schema": {
- "type": "entity.ScrapySettingParam",
- "items": {
- "$ref": "#/definitions/entity.ScrapySettingParam"
- }
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/spider/filepath": {
- "get": {
- "description": "Get scrapy spider file path",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider file path",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/spiders": {
- "get": {
- "description": "Get scrapy spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put scrapy spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Put scrapy spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/stats": {
- "get": {
- "description": "Get spider stats",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider stats",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/tasks": {
- "get": {
- "description": "Get task list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get task list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/upload": {
- "post": {
- "description": "Upload spider by id",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Upload spider by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "file",
- "description": "spider file to upload",
- "name": "file",
- "in": "formData",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/stats/home": {
- "get": {
- "description": "Get home stats",
- "produces": [
- "application/json"
- ],
- "tags": [
- "version"
- ],
- "summary": "Get home stats",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/system/deps/": {
- "get": {
- "description": "Get all dep list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get all dep list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "dep name",
- "name": "dep_nane",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/system/deps/{lang}/{dep_name}/json": {
- "get": {
- "description": "Get dep json",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get dep json",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "dep name",
- "name": "dep_name",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/task/{id}": {
- "delete": {
- "description": "Delete task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Delete task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks": {
- "get": {
- "description": "Get task list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "req data",
- "name": "data",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.TaskListRequestData"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Put task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete tasks",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Delete tasks",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}": {
- "get": {
- "description": "Get task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/cancel": {
- "post": {
- "description": "Cancel task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Cancel task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/error-log": {
- "delete": {
- "description": "Get task error log",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task error log",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/log": {
- "delete": {
- "description": "Get task log",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task log",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/restart": {
- "post": {
- "description": "Restart task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Restart task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/results": {
- "get": {
- "description": "Get task list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "req data",
- "name": "data",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.TaskResultsRequestData"
- }
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/results/download": {
- "get": {
- "description": "Get task results",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task results",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks_by_status": {
- "delete": {
- "description": "Delete task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Delete task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task status",
- "name": "status",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tokens": {
- "get": {
- "description": "token",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Get token",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "token",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Put token",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tokens/{id}": {
- "delete": {
- "description": "Delete token",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Delete token",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "token id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/users": {
- "get": {
- "description": "Get user list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Get user list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "data body",
- "name": "data",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.UserListRequestData"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Put user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "reqData body",
- "name": "reqData",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.UserRequestData"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/users/{id}": {
- "get": {
- "description": "user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Get user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "user id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Post user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "user body",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.User"
- }
- },
- {
- "type": "string",
- "description": "user id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Delete user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "user id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/variable": {
- "put": {
- "description": "Put variable",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Put variable",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "reqData body",
- "name": "variable",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Variable"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/variable/{id}": {
- "post": {
- "description": "Post variable",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Post variable",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "reqData body",
- "name": "variable",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Variable"
- }
- },
- {
- "type": "string",
- "description": "variable id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete variable",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Delete variable",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "variable id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/variables": {
- "get": {
- "description": "Get variable list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Get variable list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/version": {
- "get": {
- "description": "Get version",
- "produces": [
- "application/json"
- ],
- "tags": [
- "setting"
- ],
- "summary": "Get version",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- }
- },
- "definitions": {
- "entity.ConfigSpiderData": {
- "type": "object",
- "properties": {
- "cmd": {
- "description": "自定义爬虫",
- "type": "string"
- },
- "col": {
- "type": "string"
- },
- "display_name": {
- "type": "string"
- },
- "engine": {
- "description": "可配置爬虫",
- "type": "string"
- },
- "name": {
- "description": "通用",
- "type": "string"
- },
- "remark": {
- "type": "string"
- },
- "settings": {
- "type": "object"
- },
- "stages": {
- "type": "array",
- "items": {
- "$ref": "#/definitions/entity.Stage"
- }
- },
- "start_stage": {
- "type": "string"
- },
- "start_url": {
- "type": "string"
- },
- "type": {
- "type": "string"
- }
- }
- },
- "entity.Field": {
- "type": "object",
- "properties": {
- "attr": {
- "type": "string"
- },
- "css": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "next_stage": {
- "type": "string"
- },
- "remark": {
- "type": "string"
- },
- "xpath": {
- "type": "string"
- }
- }
- },
- "entity.ScrapyItem": {
- "type": "object",
- "properties": {
- "fields": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "name": {
- "type": "string"
- }
- }
- },
- "entity.ScrapySettingParam": {
- "type": "object",
- "properties": {
- "key": {
- "type": "string"
- },
- "type": {
- "type": "string"
- },
- "value": {
- "type": "object"
- }
- }
- },
- "entity.Stage": {
- "type": "object",
- "properties": {
- "fields": {
- "type": "array",
- "items": {
- "$ref": "#/definitions/entity.Field"
- }
- },
- "is_list": {
- "type": "boolean"
- },
- "list_css": {
- "type": "string"
- },
- "list_xpath": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "page_attr": {
- "type": "string"
- },
- "page_css": {
- "type": "string"
- },
- "page_xpath": {
- "type": "string"
- }
- }
- },
- "model.Env": {
- "type": "object",
- "properties": {
- "name": {
- "type": "string"
- },
- "value": {
- "type": "string"
- }
- }
- },
- "model.Node": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "description": {
- "type": "string"
- },
- "hostname": {
- "type": "string"
- },
- "ip": {
- "type": "string"
- },
- "is_master": {
- "description": "前端展示",
- "type": "boolean"
- },
- "key": {
- "description": "用于唯一标识节点,可能是mac地址,可能是ip地址",
- "type": "string"
- },
- "mac": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "port": {
- "type": "string"
- },
- "status": {
- "type": "string"
- },
- "update_ts": {
- "type": "string"
- },
- "update_ts_unix": {
- "type": "integer"
- }
- }
- },
- "model.Project": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "description": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "spiders": {
- "description": "前端展示",
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Spider"
- }
- },
- "tags": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "username": {
- "type": "string"
- }
- }
- },
- "model.Schedule": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "cron": {
- "type": "string"
- },
- "description": {
- "type": "string"
- },
- "enabled": {
- "type": "boolean"
- },
- "entry_id": {
- "type": "integer"
- },
- "message": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "node_ids": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "nodes": {
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Node"
- }
- },
- "param": {
- "type": "string"
- },
- "run_type": {
- "type": "string"
- },
- "scrapy_log_level": {
- "type": "string"
- },
- "scrapy_spider": {
- "type": "string"
- },
- "spider_id": {
- "type": "string"
- },
- "spider_name": {
- "description": "前端展示",
- "type": "string"
- },
- "status": {
- "type": "string"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "user_name": {
- "type": "string"
- }
- }
- },
- "model.Spider": {
- "type": "object",
- "properties": {
- "_id": {
- "description": "爬虫ID",
- "type": "string"
- },
- "cmd": {
- "description": "自定义爬虫",
- "type": "string"
- },
- "col": {
- "description": "结果储存位置",
- "type": "string"
- },
- "config": {
- "description": "可配置爬虫配置",
- "type": "object",
- "$ref": "#/definitions/entity.ConfigSpiderData"
- },
- "create_ts": {
- "type": "string"
- },
- "dedup_field": {
- "description": "去重字段",
- "type": "string"
- },
- "dedup_method": {
- "description": "去重方式",
- "type": "string"
- },
- "display_name": {
- "description": "爬虫显示名称",
- "type": "string"
- },
- "envs": {
- "description": "环境变量",
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Env"
- }
- },
- "file_id": {
- "description": "GridFS文件ID",
- "type": "string"
- },
- "git_auto_sync": {
- "description": "Git 是否自动同步",
- "type": "boolean"
- },
- "git_branch": {
- "description": "Git 分支",
- "type": "string"
- },
- "git_has_credential": {
- "description": "Git 是否加密",
- "type": "boolean"
- },
- "git_password": {
- "description": "Git 密码",
- "type": "string"
- },
- "git_sync_error": {
- "description": "Git 同步错误",
- "type": "string"
- },
- "git_sync_frequency": {
- "description": "Git 同步频率",
- "type": "string"
- },
- "git_url": {
- "description": "Git URL",
- "type": "string"
- },
- "git_username": {
- "description": "Git 用户名",
- "type": "string"
- },
- "is_dedup": {
- "description": "去重",
- "type": "boolean"
- },
- "is_git": {
- "description": "Git 设置",
- "type": "boolean"
- },
- "is_long_task": {
- "description": "长任务",
- "type": "boolean"
- },
- "is_public": {
- "description": "是否公开",
- "type": "boolean"
- },
- "is_scrapy": {
- "description": "Scrapy 爬虫(属于自定义爬虫)",
- "type": "boolean"
- },
- "is_web_hook": {
- "description": "Web Hook",
- "type": "boolean"
- },
- "last_run_ts": {
- "description": "前端展示",
- "type": "string"
- },
- "last_status": {
- "description": "最后执行状态",
- "type": "string"
- },
- "latest_tasks": {
- "description": "最近任务列表",
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Task"
- }
- },
- "name": {
- "description": "爬虫名称(唯一)",
- "type": "string"
- },
- "project_id": {
- "description": "项目ID",
- "type": "string"
- },
- "remark": {
- "description": "备注",
- "type": "string"
- },
- "site": {
- "description": "爬虫网站",
- "type": "string"
- },
- "spider_names": {
- "description": "爬虫名称列表",
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "src": {
- "description": "源码位置",
- "type": "string"
- },
- "template": {
- "description": "可配置爬虫",
- "type": "string"
- },
- "type": {
- "description": "爬虫类别",
- "type": "string"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "description": "时间",
- "type": "string"
- },
- "username": {
- "description": "用户名称",
- "type": "string"
- },
- "web_hook_url": {
- "description": "Web Hook URL",
- "type": "string"
- }
- }
- },
- "model.Task": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "cmd": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "error": {
- "type": "string"
- },
- "error_log_count": {
- "type": "integer"
- },
- "finish_ts": {
- "type": "string"
- },
- "log_path": {
- "type": "string"
- },
- "node_id": {
- "type": "string"
- },
- "node_name": {
- "type": "string"
- },
- "param": {
- "type": "string"
- },
- "pid": {
- "type": "integer"
- },
- "result_count": {
- "type": "integer"
- },
- "run_type": {
- "type": "string"
- },
- "runtime_duration": {
- "type": "number"
- },
- "schedule_id": {
- "type": "string"
- },
- "spider_id": {
- "type": "string"
- },
- "spider_name": {
- "description": "前端数据",
- "type": "string"
- },
- "start_ts": {
- "type": "string"
- },
- "status": {
- "type": "string"
- },
- "total_duration": {
- "type": "number"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "username": {
- "type": "string"
- },
- "wait_duration": {
- "type": "number"
- }
- }
- },
- "model.User": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "email": {
- "type": "string"
- },
- "password": {
- "type": "string"
- },
- "role": {
- "type": "string"
- },
- "setting": {
- "type": "object",
- "$ref": "#/definitions/model.UserSetting"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "username": {
- "type": "string"
- }
- }
- },
- "model.UserSetting": {
- "type": "object",
- "properties": {
- "ding_talk_robot_webhook": {
- "type": "string"
- },
- "enabled_notifications": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "error_regex_pattern": {
- "type": "string"
- },
- "log_expire_duration": {
- "type": "integer"
- },
- "max_error_log": {
- "type": "integer"
- },
- "notification_trigger": {
- "type": "string"
- },
- "wechat_robot_webhook": {
- "type": "string"
- }
- }
- },
- "model.Variable": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "key": {
- "type": "string"
- },
- "remark": {
- "type": "string"
- },
- "value": {
- "type": "string"
- }
- }
- },
- "routes.SpiderFileReqBody": {
- "type": "object",
- "properties": {
- "content": {
- "type": "string"
- },
- "new_path": {
- "type": "string"
- },
- "path": {
- "type": "string"
- }
- }
- },
- "routes.TaskListRequestData": {
- "type": "object",
- "properties": {
- "nodeId": {
- "type": "string"
- },
- "pageNum": {
- "type": "integer"
- },
- "pageSize": {
- "type": "integer"
- },
- "scheduleId": {
- "type": "string"
- },
- "spiderId": {
- "type": "string"
- },
- "status": {
- "type": "string"
- }
- }
- },
- "routes.TaskResultsRequestData": {
- "type": "object",
- "properties": {
- "pageNum": {
- "type": "integer"
- },
- "pageSize": {
- "type": "integer"
- }
- }
- },
- "routes.UserListRequestData": {
- "type": "object",
- "properties": {
- "pageNum": {
- "type": "integer"
- },
- "pageSize": {
- "type": "integer"
- }
- }
- },
- "routes.UserRequestData": {
- "type": "object",
- "properties": {
- "email": {
- "type": "string"
- },
- "password": {
- "type": "string"
- },
- "role": {
- "type": "string"
- },
- "username": {
- "type": "string"
- }
- }
- }
- }
-}`
-
-type swaggerInfo struct {
- Version string
- Host string
- BasePath string
- Schemes []string
- Title string
- Description string
-}
-
-// SwaggerInfo holds exported Swagger Info so clients can modify it
-var SwaggerInfo = swaggerInfo{
- Version: "",
- Host: "",
- BasePath: "",
- Schemes: []string{},
- Title: "",
- Description: "",
-}
-
-type s struct{}
-
-func (s *s) ReadDoc() string {
- sInfo := SwaggerInfo
- sInfo.Description = strings.Replace(sInfo.Description, "\n", "\\n", -1)
-
- t, err := template.New("swagger_info").Funcs(template.FuncMap{
- "marshal": func(v interface{}) string {
- a, _ := json.Marshal(v)
- return string(a)
- },
- }).Parse(doc)
- if err != nil {
- return doc
- }
-
- var tpl bytes.Buffer
- if err := t.Execute(&tpl, sInfo); err != nil {
- return doc
- }
-
- return tpl.String()
-}
-
-func init() {
- swag.Register(swag.Name, &s{})
-}
diff --git a/backend/docs/swagger.json b/backend/docs/swagger.json
deleted file mode 100644
index 47986662..00000000
--- a/backend/docs/swagger.json
+++ /dev/null
@@ -1,4740 +0,0 @@
-{
- "swagger": "2.0",
- "info": {
- "contact": {},
- "license": {}
- },
- "paths": {
- "/config_spiders": {
- "put": {
- "description": "Put config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Put config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders/{id}/config": {
- "get": {
- "description": "Get config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Get config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post config spider config",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Post config spider config",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders/{id}/spiderfile": {
- "post": {
- "description": "Post config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Post config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders/{id}/upload": {
- "post": {
- "description": "Upload config spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Upload config spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/config_spiders_templates": {
- "get": {
- "description": "Get config spider template list",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "config spider"
- ],
- "summary": "Get config spider template list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/docs": {
- "get": {
- "description": "Get docs",
- "produces": [
- "application/json"
- ],
- "tags": [
- "docs"
- ],
- "summary": "Get docs",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/file": {
- "get": {
- "description": "Get file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "file"
- ],
- "summary": "Get file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes": {
- "get": {
- "description": "Get nodes",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get nodes",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}": {
- "get": {
- "description": "Get node",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post node",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Post node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "post node",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete node",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Delete node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps": {
- "get": {
- "description": "Get dep list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get dep list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "query",
- "required": true
- },
- {
- "type": "string",
- "description": "dep name",
- "name": "dep_name",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps/install": {
- "post": {
- "description": "Install dep",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Install dep",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps/installed": {
- "get": {
- "description": "Get installed dep list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get installed dep list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/deps/uninstall": {
- "post": {
- "description": "Uninstall dep",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Uninstall dep",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/langs": {
- "get": {
- "description": "Get language list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get language list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/langs/install": {
- "post": {
- "description": "Install language",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Install language",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/system": {
- "get": {
- "description": "Get system info",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get system info",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/nodes/{id}/tasks": {
- "get": {
- "description": "Get tasks on node",
- "produces": [
- "application/json"
- ],
- "tags": [
- "node"
- ],
- "summary": "Get tasks on node",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "node id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/projects": {
- "get": {
- "description": "Get projects",
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Get projects",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "projects",
- "name": "tag",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put project",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Put project",
- "parameters": [
- {
- "type": "string",
- "description": "With the bearer started",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "post project",
- "name": "p",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Project"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/projects/tags": {
- "get": {
- "description": "Get projects tags",
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Get project tags",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/projects/{id}": {
- "post": {
- "description": "Post project",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Post project",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "project id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "project item",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Project"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete project",
- "produces": [
- "application/json"
- ],
- "tags": [
- "project"
- ],
- "summary": "Delete project",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "project id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/releases/latest": {
- "get": {
- "description": "Get latest release",
- "produces": [
- "application/json"
- ],
- "tags": [
- "version"
- ],
- "summary": "Get latest release",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules": {
- "get": {
- "description": "Get spider list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "page num",
- "name": "page_num",
- "in": "query"
- },
- {
- "type": "string",
- "description": "page size",
- "name": "page_size",
- "in": "query"
- },
- {
- "type": "string",
- "description": "keyword",
- "name": "keyword",
- "in": "query"
- },
- {
- "type": "string",
- "description": "project_id",
- "name": "project_id",
- "in": "query"
- },
- {
- "type": "string",
- "description": "type",
- "name": "type",
- "in": "query"
- },
- {
- "type": "string",
- "description": "sort_key",
- "name": "sort_key",
- "in": "query"
- },
- {
- "type": "string",
- "description": "sort_direction",
- "name": "sort_direction",
- "in": "query"
- },
- {
- "type": "string",
- "description": "owner_type",
- "name": "owner_type",
- "in": "query"
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Put schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "schedule item",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Schedule"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules/{id}": {
- "get": {
- "description": "Get schedule by id",
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Get schedule by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Post schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "schedule item",
- "name": "newItem",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Schedule"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete schedule",
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "Delete schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules/{id}/disable": {
- "post": {
- "description": "disable schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "disable schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/schedules/{id}/enable": {
- "post": {
- "description": "enable schedule",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "schedule"
- ],
- "summary": "enable schedule",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/setting": {
- "get": {
- "description": "Get setting",
- "produces": [
- "application/json"
- ],
- "tags": [
- "setting"
- ],
- "summary": "Get setting",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders": {
- "put": {
- "description": "Put spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Put spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "spider item",
- "name": "spider",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "delete spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "delete spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders-cancel": {
- "post": {
- "description": "cancel spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "cancel spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders-run": {
- "post": {
- "description": "run spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "run spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}": {
- "get": {
- "description": "Get spider by id",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "spider item",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Spider"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete spider by id",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Delete spider by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/copy": {
- "post": {
- "description": "Copy spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Copy spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/dir": {
- "get": {
- "description": "Get spider dir",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider dir",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "path",
- "name": "path",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/file": {
- "get": {
- "description": "Get spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "path",
- "name": "path",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Post spider dir",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider dir",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Put spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Put spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Delete spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/file/rename": {
- "post": {
- "description": "Rename spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Rename spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "path",
- "name": "reqBody",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.SpiderFileReqBody"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/file/tree": {
- "get": {
- "description": "Get spider dir",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider dir",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/git/reset": {
- "post": {
- "description": "Post spider reset git",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider reset git",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/git/sync": {
- "post": {
- "description": "Post spider sync git",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post spider sync git",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/publish": {
- "post": {
- "description": "Publish spider",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Publish spider",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "schedule id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/schedules": {
- "get": {
- "description": "Get schedules",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get schedules",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/items": {
- "get": {
- "description": "Get scrapy spider items",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider items",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post scrapy spider items",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Post scrapy spider items",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "req data",
- "name": "reqData",
- "in": "body",
- "required": true,
- "schema": {
- "type": "entity.ScrapyItem",
- "items": {
- "$ref": "#/definitions/entity.ScrapyItem"
- }
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/pipelines": {
- "get": {
- "description": "Get scrapy spider pipelines",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider pipelines",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/settings": {
- "get": {
- "description": "Get scrapy spider settings",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider settings",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Get scrapy spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- },
- {
- "description": "req data",
- "name": "reqData",
- "in": "body",
- "required": true,
- "schema": {
- "type": "entity.ScrapySettingParam",
- "items": {
- "$ref": "#/definitions/entity.ScrapySettingParam"
- }
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/spider/filepath": {
- "get": {
- "description": "Get scrapy spider file path",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider file path",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/scrapy/spiders": {
- "get": {
- "description": "Get scrapy spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get scrapy spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put scrapy spider file",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Put scrapy spider file",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/stats": {
- "get": {
- "description": "Get spider stats",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get spider stats",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/tasks": {
- "get": {
- "description": "Get task list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Get task list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/spiders/{id}/upload": {
- "post": {
- "description": "Upload spider by id",
- "consumes": [
- "application/json"
- ],
- "produces": [
- "application/json"
- ],
- "tags": [
- "spider"
- ],
- "summary": "Upload spider by id",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "file",
- "description": "spider file to upload",
- "name": "file",
- "in": "formData",
- "required": true
- },
- {
- "type": "string",
- "description": "spider id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "500": {
- "description": "Internal Server Error",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/stats/home": {
- "get": {
- "description": "Get home stats",
- "produces": [
- "application/json"
- ],
- "tags": [
- "version"
- ],
- "summary": "Get home stats",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/system/deps/": {
- "get": {
- "description": "Get all dep list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get all dep list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "dep name",
- "name": "dep_nane",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/system/deps/{lang}/{dep_name}/json": {
- "get": {
- "description": "Get dep json",
- "produces": [
- "application/json"
- ],
- "tags": [
- "system"
- ],
- "summary": "Get dep json",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "language",
- "name": "lang",
- "in": "path",
- "required": true
- },
- {
- "type": "string",
- "description": "dep name",
- "name": "dep_name",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/task/{id}": {
- "delete": {
- "description": "Delete task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Delete task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks": {
- "get": {
- "description": "Get task list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "req data",
- "name": "data",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.TaskListRequestData"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Put task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete tasks",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Delete tasks",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}": {
- "get": {
- "description": "Get task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/cancel": {
- "post": {
- "description": "Cancel task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Cancel task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/error-log": {
- "delete": {
- "description": "Get task error log",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task error log",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/log": {
- "delete": {
- "description": "Get task log",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task log",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/restart": {
- "post": {
- "description": "Restart task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Restart task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/results": {
- "get": {
- "description": "Get task list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "req data",
- "name": "data",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.TaskResultsRequestData"
- }
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks/{id}/results/download": {
- "get": {
- "description": "Get task results",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Get task results",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tasks_by_status": {
- "delete": {
- "description": "Delete task",
- "produces": [
- "application/json"
- ],
- "tags": [
- "task"
- ],
- "summary": "Delete task",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "task status",
- "name": "status",
- "in": "query",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tokens": {
- "get": {
- "description": "token",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Get token",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "token",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Put token",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/tokens/{id}": {
- "delete": {
- "description": "Delete token",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Delete token",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "token id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/users": {
- "get": {
- "description": "Get user list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "token"
- ],
- "summary": "Get user list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "data body",
- "name": "data",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.UserListRequestData"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "put": {
- "description": "Put user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Put user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "reqData body",
- "name": "reqData",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/routes.UserRequestData"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/users/{id}": {
- "get": {
- "description": "user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Get user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "user id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "post": {
- "description": "Post user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Post user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "user body",
- "name": "item",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.User"
- }
- },
- {
- "type": "string",
- "description": "user id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete user",
- "produces": [
- "application/json"
- ],
- "tags": [
- "user"
- ],
- "summary": "Delete user",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "user id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/variable": {
- "put": {
- "description": "Put variable",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Put variable",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "reqData body",
- "name": "variable",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Variable"
- }
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/variable/{id}": {
- "post": {
- "description": "Post variable",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Post variable",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "description": "reqData body",
- "name": "variable",
- "in": "body",
- "required": true,
- "schema": {
- "type": "object",
- "$ref": "#/definitions/model.Variable"
- }
- },
- {
- "type": "string",
- "description": "variable id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- },
- "delete": {
- "description": "Delete variable",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Delete variable",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- },
- {
- "type": "string",
- "description": "variable id",
- "name": "id",
- "in": "path",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/variables": {
- "get": {
- "description": "Get variable list",
- "produces": [
- "application/json"
- ],
- "tags": [
- "variable"
- ],
- "summary": "Get variable list",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- },
- "/version": {
- "get": {
- "description": "Get version",
- "produces": [
- "application/json"
- ],
- "tags": [
- "setting"
- ],
- "summary": "Get version",
- "parameters": [
- {
- "type": "string",
- "description": "Authorization token",
- "name": "Authorization",
- "in": "header",
- "required": true
- }
- ],
- "responses": {
- "200": {
- "description": "OK",
- "schema": {
- "type": "json"
- }
- },
- "400": {
- "description": "Bad Request",
- "schema": {
- "type": "json"
- }
- }
- }
- }
- }
- },
- "definitions": {
- "entity.ConfigSpiderData": {
- "type": "object",
- "properties": {
- "cmd": {
- "description": "自定义爬虫",
- "type": "string"
- },
- "col": {
- "type": "string"
- },
- "display_name": {
- "type": "string"
- },
- "engine": {
- "description": "可配置爬虫",
- "type": "string"
- },
- "name": {
- "description": "通用",
- "type": "string"
- },
- "remark": {
- "type": "string"
- },
- "settings": {
- "type": "object"
- },
- "stages": {
- "type": "array",
- "items": {
- "$ref": "#/definitions/entity.Stage"
- }
- },
- "start_stage": {
- "type": "string"
- },
- "start_url": {
- "type": "string"
- },
- "type": {
- "type": "string"
- }
- }
- },
- "entity.Field": {
- "type": "object",
- "properties": {
- "attr": {
- "type": "string"
- },
- "css": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "next_stage": {
- "type": "string"
- },
- "remark": {
- "type": "string"
- },
- "xpath": {
- "type": "string"
- }
- }
- },
- "entity.ScrapyItem": {
- "type": "object",
- "properties": {
- "fields": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "name": {
- "type": "string"
- }
- }
- },
- "entity.ScrapySettingParam": {
- "type": "object",
- "properties": {
- "key": {
- "type": "string"
- },
- "type": {
- "type": "string"
- },
- "value": {
- "type": "object"
- }
- }
- },
- "entity.Stage": {
- "type": "object",
- "properties": {
- "fields": {
- "type": "array",
- "items": {
- "$ref": "#/definitions/entity.Field"
- }
- },
- "is_list": {
- "type": "boolean"
- },
- "list_css": {
- "type": "string"
- },
- "list_xpath": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "page_attr": {
- "type": "string"
- },
- "page_css": {
- "type": "string"
- },
- "page_xpath": {
- "type": "string"
- }
- }
- },
- "model.Env": {
- "type": "object",
- "properties": {
- "name": {
- "type": "string"
- },
- "value": {
- "type": "string"
- }
- }
- },
- "model.Node": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "description": {
- "type": "string"
- },
- "hostname": {
- "type": "string"
- },
- "ip": {
- "type": "string"
- },
- "is_master": {
- "description": "前端展示",
- "type": "boolean"
- },
- "key": {
- "description": "用于唯一标识节点,可能是mac地址,可能是ip地址",
- "type": "string"
- },
- "mac": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "port": {
- "type": "string"
- },
- "status": {
- "type": "string"
- },
- "update_ts": {
- "type": "string"
- },
- "update_ts_unix": {
- "type": "integer"
- }
- }
- },
- "model.Project": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "description": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "spiders": {
- "description": "前端展示",
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Spider"
- }
- },
- "tags": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "username": {
- "type": "string"
- }
- }
- },
- "model.Schedule": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "cron": {
- "type": "string"
- },
- "description": {
- "type": "string"
- },
- "enabled": {
- "type": "boolean"
- },
- "entry_id": {
- "type": "integer"
- },
- "message": {
- "type": "string"
- },
- "name": {
- "type": "string"
- },
- "node_ids": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "nodes": {
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Node"
- }
- },
- "param": {
- "type": "string"
- },
- "run_type": {
- "type": "string"
- },
- "scrapy_log_level": {
- "type": "string"
- },
- "scrapy_spider": {
- "type": "string"
- },
- "spider_id": {
- "type": "string"
- },
- "spider_name": {
- "description": "前端展示",
- "type": "string"
- },
- "status": {
- "type": "string"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "user_name": {
- "type": "string"
- }
- }
- },
- "model.Spider": {
- "type": "object",
- "properties": {
- "_id": {
- "description": "爬虫ID",
- "type": "string"
- },
- "cmd": {
- "description": "自定义爬虫",
- "type": "string"
- },
- "col": {
- "description": "结果储存位置",
- "type": "string"
- },
- "config": {
- "description": "可配置爬虫配置",
- "type": "object",
- "$ref": "#/definitions/entity.ConfigSpiderData"
- },
- "create_ts": {
- "type": "string"
- },
- "dedup_field": {
- "description": "去重字段",
- "type": "string"
- },
- "dedup_method": {
- "description": "去重方式",
- "type": "string"
- },
- "display_name": {
- "description": "爬虫显示名称",
- "type": "string"
- },
- "envs": {
- "description": "环境变量",
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Env"
- }
- },
- "file_id": {
- "description": "GridFS文件ID",
- "type": "string"
- },
- "git_auto_sync": {
- "description": "Git 是否自动同步",
- "type": "boolean"
- },
- "git_branch": {
- "description": "Git 分支",
- "type": "string"
- },
- "git_has_credential": {
- "description": "Git 是否加密",
- "type": "boolean"
- },
- "git_password": {
- "description": "Git 密码",
- "type": "string"
- },
- "git_sync_error": {
- "description": "Git 同步错误",
- "type": "string"
- },
- "git_sync_frequency": {
- "description": "Git 同步频率",
- "type": "string"
- },
- "git_url": {
- "description": "Git URL",
- "type": "string"
- },
- "git_username": {
- "description": "Git 用户名",
- "type": "string"
- },
- "is_dedup": {
- "description": "去重",
- "type": "boolean"
- },
- "is_git": {
- "description": "Git 设置",
- "type": "boolean"
- },
- "is_long_task": {
- "description": "长任务",
- "type": "boolean"
- },
- "is_public": {
- "description": "是否公开",
- "type": "boolean"
- },
- "is_scrapy": {
- "description": "Scrapy 爬虫(属于自定义爬虫)",
- "type": "boolean"
- },
- "is_web_hook": {
- "description": "Web Hook",
- "type": "boolean"
- },
- "last_run_ts": {
- "description": "前端展示",
- "type": "string"
- },
- "last_status": {
- "description": "最后执行状态",
- "type": "string"
- },
- "latest_tasks": {
- "description": "最近任务列表",
- "type": "array",
- "items": {
- "$ref": "#/definitions/model.Task"
- }
- },
- "name": {
- "description": "爬虫名称(唯一)",
- "type": "string"
- },
- "project_id": {
- "description": "项目ID",
- "type": "string"
- },
- "remark": {
- "description": "备注",
- "type": "string"
- },
- "site": {
- "description": "爬虫网站",
- "type": "string"
- },
- "spider_names": {
- "description": "爬虫名称列表",
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "src": {
- "description": "源码位置",
- "type": "string"
- },
- "template": {
- "description": "可配置爬虫",
- "type": "string"
- },
- "type": {
- "description": "爬虫类别",
- "type": "string"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "description": "时间",
- "type": "string"
- },
- "username": {
- "description": "用户名称",
- "type": "string"
- },
- "web_hook_url": {
- "description": "Web Hook URL",
- "type": "string"
- }
- }
- },
- "model.Task": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "cmd": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "error": {
- "type": "string"
- },
- "error_log_count": {
- "type": "integer"
- },
- "finish_ts": {
- "type": "string"
- },
- "log_path": {
- "type": "string"
- },
- "node_id": {
- "type": "string"
- },
- "node_name": {
- "type": "string"
- },
- "param": {
- "type": "string"
- },
- "pid": {
- "type": "integer"
- },
- "result_count": {
- "type": "integer"
- },
- "run_type": {
- "type": "string"
- },
- "runtime_duration": {
- "type": "number"
- },
- "schedule_id": {
- "type": "string"
- },
- "spider_id": {
- "type": "string"
- },
- "spider_name": {
- "description": "前端数据",
- "type": "string"
- },
- "start_ts": {
- "type": "string"
- },
- "status": {
- "type": "string"
- },
- "total_duration": {
- "type": "number"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "username": {
- "type": "string"
- },
- "wait_duration": {
- "type": "number"
- }
- }
- },
- "model.User": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "create_ts": {
- "type": "string"
- },
- "email": {
- "type": "string"
- },
- "password": {
- "type": "string"
- },
- "role": {
- "type": "string"
- },
- "setting": {
- "type": "object",
- "$ref": "#/definitions/model.UserSetting"
- },
- "update_ts": {
- "type": "string"
- },
- "user_id": {
- "type": "string"
- },
- "username": {
- "type": "string"
- }
- }
- },
- "model.UserSetting": {
- "type": "object",
- "properties": {
- "ding_talk_robot_webhook": {
- "type": "string"
- },
- "enabled_notifications": {
- "type": "array",
- "items": {
- "type": "string"
- }
- },
- "error_regex_pattern": {
- "type": "string"
- },
- "log_expire_duration": {
- "type": "integer"
- },
- "max_error_log": {
- "type": "integer"
- },
- "notification_trigger": {
- "type": "string"
- },
- "wechat_robot_webhook": {
- "type": "string"
- }
- }
- },
- "model.Variable": {
- "type": "object",
- "properties": {
- "_id": {
- "type": "string"
- },
- "key": {
- "type": "string"
- },
- "remark": {
- "type": "string"
- },
- "value": {
- "type": "string"
- }
- }
- },
- "routes.SpiderFileReqBody": {
- "type": "object",
- "properties": {
- "content": {
- "type": "string"
- },
- "new_path": {
- "type": "string"
- },
- "path": {
- "type": "string"
- }
- }
- },
- "routes.TaskListRequestData": {
- "type": "object",
- "properties": {
- "nodeId": {
- "type": "string"
- },
- "pageNum": {
- "type": "integer"
- },
- "pageSize": {
- "type": "integer"
- },
- "scheduleId": {
- "type": "string"
- },
- "spiderId": {
- "type": "string"
- },
- "status": {
- "type": "string"
- }
- }
- },
- "routes.TaskResultsRequestData": {
- "type": "object",
- "properties": {
- "pageNum": {
- "type": "integer"
- },
- "pageSize": {
- "type": "integer"
- }
- }
- },
- "routes.UserListRequestData": {
- "type": "object",
- "properties": {
- "pageNum": {
- "type": "integer"
- },
- "pageSize": {
- "type": "integer"
- }
- }
- },
- "routes.UserRequestData": {
- "type": "object",
- "properties": {
- "email": {
- "type": "string"
- },
- "password": {
- "type": "string"
- },
- "role": {
- "type": "string"
- },
- "username": {
- "type": "string"
- }
- }
- }
- }
-}
\ No newline at end of file
diff --git a/backend/docs/swagger.yaml b/backend/docs/swagger.yaml
deleted file mode 100644
index 26598b52..00000000
--- a/backend/docs/swagger.yaml
+++ /dev/null
@@ -1,3177 +0,0 @@
-definitions:
- entity.ConfigSpiderData:
- properties:
- cmd:
- description: 自定义爬虫
- type: string
- col:
- type: string
- display_name:
- type: string
- engine:
- description: 可配置爬虫
- type: string
- name:
- description: 通用
- type: string
- remark:
- type: string
- settings:
- type: object
- stages:
- items:
- $ref: '#/definitions/entity.Stage'
- type: array
- start_stage:
- type: string
- start_url:
- type: string
- type:
- type: string
- type: object
- entity.Field:
- properties:
- attr:
- type: string
- css:
- type: string
- name:
- type: string
- next_stage:
- type: string
- remark:
- type: string
- xpath:
- type: string
- type: object
- entity.ScrapyItem:
- properties:
- fields:
- items:
- type: string
- type: array
- name:
- type: string
- type: object
- entity.ScrapySettingParam:
- properties:
- key:
- type: string
- type:
- type: string
- value:
- type: object
- type: object
- entity.Stage:
- properties:
- fields:
- items:
- $ref: '#/definitions/entity.Field'
- type: array
- is_list:
- type: boolean
- list_css:
- type: string
- list_xpath:
- type: string
- name:
- type: string
- page_attr:
- type: string
- page_css:
- type: string
- page_xpath:
- type: string
- type: object
- model.Env:
- properties:
- name:
- type: string
- value:
- type: string
- type: object
- model.Node:
- properties:
- _id:
- type: string
- create_ts:
- type: string
- description:
- type: string
- hostname:
- type: string
- ip:
- type: string
- is_master:
- description: 前端展示
- type: boolean
- key:
- description: 用于唯一标识节点,可能是mac地址,可能是ip地址
- type: string
- mac:
- type: string
- name:
- type: string
- port:
- type: string
- status:
- type: string
- update_ts:
- type: string
- update_ts_unix:
- type: integer
- type: object
- model.Project:
- properties:
- _id:
- type: string
- create_ts:
- type: string
- description:
- type: string
- name:
- type: string
- spiders:
- description: 前端展示
- items:
- $ref: '#/definitions/model.Spider'
- type: array
- tags:
- items:
- type: string
- type: array
- update_ts:
- type: string
- user_id:
- type: string
- username:
- type: string
- type: object
- model.Schedule:
- properties:
- _id:
- type: string
- create_ts:
- type: string
- cron:
- type: string
- description:
- type: string
- enabled:
- type: boolean
- entry_id:
- type: integer
- message:
- type: string
- name:
- type: string
- node_ids:
- items:
- type: string
- type: array
- nodes:
- items:
- $ref: '#/definitions/model.Node'
- type: array
- param:
- type: string
- run_type:
- type: string
- scrapy_log_level:
- type: string
- scrapy_spider:
- type: string
- spider_id:
- type: string
- spider_name:
- description: 前端展示
- type: string
- status:
- type: string
- update_ts:
- type: string
- user_id:
- type: string
- user_name:
- type: string
- type: object
- model.Spider:
- properties:
- _id:
- description: 爬虫ID
- type: string
- cmd:
- description: 自定义爬虫
- type: string
- col:
- description: 结果储存位置
- type: string
- config:
- $ref: '#/definitions/entity.ConfigSpiderData'
- description: 可配置爬虫配置
- type: object
- create_ts:
- type: string
- dedup_field:
- description: 去重字段
- type: string
- dedup_method:
- description: 去重方式
- type: string
- display_name:
- description: 爬虫显示名称
- type: string
- envs:
- description: 环境变量
- items:
- $ref: '#/definitions/model.Env'
- type: array
- file_id:
- description: GridFS文件ID
- type: string
- git_auto_sync:
- description: Git 是否自动同步
- type: boolean
- git_branch:
- description: Git 分支
- type: string
- git_has_credential:
- description: Git 是否加密
- type: boolean
- git_password:
- description: Git 密码
- type: string
- git_sync_error:
- description: Git 同步错误
- type: string
- git_sync_frequency:
- description: Git 同步频率
- type: string
- git_url:
- description: Git URL
- type: string
- git_username:
- description: Git 用户名
- type: string
- is_dedup:
- description: 去重
- type: boolean
- is_git:
- description: Git 设置
- type: boolean
- is_long_task:
- description: 长任务
- type: boolean
- is_public:
- description: 是否公开
- type: boolean
- is_scrapy:
- description: Scrapy 爬虫(属于自定义爬虫)
- type: boolean
- is_web_hook:
- description: Web Hook
- type: boolean
- last_run_ts:
- description: 前端展示
- type: string
- last_status:
- description: 最后执行状态
- type: string
- latest_tasks:
- description: 最近任务列表
- items:
- $ref: '#/definitions/model.Task'
- type: array
- name:
- description: 爬虫名称(唯一)
- type: string
- project_id:
- description: 项目ID
- type: string
- remark:
- description: 备注
- type: string
- site:
- description: 爬虫网站
- type: string
- spider_names:
- description: 爬虫名称列表
- items:
- type: string
- type: array
- src:
- description: 源码位置
- type: string
- template:
- description: 可配置爬虫
- type: string
- type:
- description: 爬虫类别
- type: string
- update_ts:
- type: string
- user_id:
- description: 时间
- type: string
- username:
- description: 用户名称
- type: string
- web_hook_url:
- description: Web Hook URL
- type: string
- type: object
- model.Task:
- properties:
- _id:
- type: string
- cmd:
- type: string
- create_ts:
- type: string
- error:
- type: string
- error_log_count:
- type: integer
- finish_ts:
- type: string
- log_path:
- type: string
- node_id:
- type: string
- node_name:
- type: string
- param:
- type: string
- pid:
- type: integer
- result_count:
- type: integer
- run_type:
- type: string
- runtime_duration:
- type: number
- schedule_id:
- type: string
- spider_id:
- type: string
- spider_name:
- description: 前端数据
- type: string
- start_ts:
- type: string
- status:
- type: string
- total_duration:
- type: number
- update_ts:
- type: string
- user_id:
- type: string
- username:
- type: string
- wait_duration:
- type: number
- type: object
- model.User:
- properties:
- _id:
- type: string
- create_ts:
- type: string
- email:
- type: string
- password:
- type: string
- role:
- type: string
- setting:
- $ref: '#/definitions/model.UserSetting'
- type: object
- update_ts:
- type: string
- user_id:
- type: string
- username:
- type: string
- type: object
- model.UserSetting:
- properties:
- ding_talk_robot_webhook:
- type: string
- enabled_notifications:
- items:
- type: string
- type: array
- error_regex_pattern:
- type: string
- log_expire_duration:
- type: integer
- max_error_log:
- type: integer
- notification_trigger:
- type: string
- wechat_robot_webhook:
- type: string
- type: object
- model.Variable:
- properties:
- _id:
- type: string
- key:
- type: string
- remark:
- type: string
- value:
- type: string
- type: object
- routes.SpiderFileReqBody:
- properties:
- content:
- type: string
- new_path:
- type: string
- path:
- type: string
- type: object
- routes.TaskListRequestData:
- properties:
- nodeId:
- type: string
- pageNum:
- type: integer
- pageSize:
- type: integer
- scheduleId:
- type: string
- spiderId:
- type: string
- status:
- type: string
- type: object
- routes.TaskResultsRequestData:
- properties:
- pageNum:
- type: integer
- pageSize:
- type: integer
- type: object
- routes.UserListRequestData:
- properties:
- pageNum:
- type: integer
- pageSize:
- type: integer
- type: object
- routes.UserRequestData:
- properties:
- email:
- type: string
- password:
- type: string
- role:
- type: string
- username:
- type: string
- type: object
-info:
- contact: {}
- license: {}
-paths:
- /config_spiders:
- put:
- consumes:
- - application/json
- description: Put config spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider item
- in: body
- name: spider
- required: true
- schema:
- $ref: '#/definitions/model.Spider'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Put config spider
- tags:
- - config spider
- /config_spiders/{id}/config:
- get:
- consumes:
- - application/json
- description: Get config spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Get config spider
- tags:
- - config spider
- post:
- consumes:
- - application/json
- description: Post config spider config
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider item
- in: body
- name: spider
- required: true
- schema:
- $ref: '#/definitions/model.Spider'
- type: object
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Post config spider config
- tags:
- - config spider
- /config_spiders/{id}/spiderfile:
- post:
- consumes:
- - application/json
- description: Post config spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Post config spider
- tags:
- - config spider
- /config_spiders/{id}/upload:
- post:
- consumes:
- - application/json
- description: Upload config spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider item
- in: body
- name: spider
- required: true
- schema:
- $ref: '#/definitions/model.Spider'
- type: object
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Upload config spider
- tags:
- - config spider
- /config_spiders_templates:
- get:
- consumes:
- - application/json
- description: Get config spider template list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Get config spider template list
- tags:
- - config spider
- /docs:
- get:
- description: Get docs
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get docs
- tags:
- - docs
- /file:
- get:
- description: Get file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get file
- tags:
- - file
- /nodes:
- get:
- description: Get nodes
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get nodes
- tags:
- - node
- /nodes/{id}:
- delete:
- description: Delete node
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete node
- tags:
- - node
- get:
- description: Get node
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- - description: id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get node
- tags:
- - node
- post:
- consumes:
- - application/json
- description: Post node
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- - description: post node
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Post node
- tags:
- - node
- /nodes/{id}/deps:
- get:
- description: Get dep list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- - description: language
- in: query
- name: lang
- required: true
- type: string
- - description: dep name
- in: query
- name: dep_name
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get dep list
- tags:
- - system
- /nodes/{id}/deps/install:
- post:
- description: Install dep
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Install dep
- tags:
- - system
- /nodes/{id}/deps/installed:
- get:
- description: Get installed dep list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- - description: language
- in: query
- name: lang
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get installed dep list
- tags:
- - system
- /nodes/{id}/deps/uninstall:
- post:
- description: Uninstall dep
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Uninstall dep
- tags:
- - system
- /nodes/{id}/langs:
- get:
- description: Get language list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get language list
- tags:
- - system
- /nodes/{id}/langs/install:
- post:
- description: Install language
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Install language
- tags:
- - system
- /nodes/{id}/system:
- get:
- description: Get system info
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get system info
- tags:
- - node
- /nodes/{id}/tasks:
- get:
- description: Get tasks on node
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- - description: node id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get tasks on node
- tags:
- - node
- /projects:
- get:
- description: Get projects
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- - description: projects
- in: query
- name: tag
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get projects
- tags:
- - project
- put:
- consumes:
- - application/json
- description: Put project
- parameters:
- - description: With the bearer started
- in: header
- name: Authorization
- required: true
- type: string
- - description: post project
- in: body
- name: p
- required: true
- schema:
- $ref: '#/definitions/model.Project'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Put project
- tags:
- - project
- /projects/{id}:
- delete:
- description: Delete project
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: project id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete project
- tags:
- - project
- post:
- consumes:
- - application/json
- description: Post project
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: project id
- in: path
- name: id
- required: true
- type: string
- - description: project item
- in: body
- name: item
- required: true
- schema:
- $ref: '#/definitions/model.Project'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Post project
- tags:
- - project
- /projects/tags:
- get:
- description: Get projects tags
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get project tags
- tags:
- - project
- /releases/latest:
- get:
- description: Get latest release
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get latest release
- tags:
- - version
- /schedules:
- get:
- description: Get spider list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: page num
- in: query
- name: page_num
- type: string
- - description: page size
- in: query
- name: page_size
- type: string
- - description: keyword
- in: query
- name: keyword
- type: string
- - description: project_id
- in: query
- name: project_id
- type: string
- - description: type
- in: query
- name: type
- type: string
- - description: sort_key
- in: query
- name: sort_key
- type: string
- - description: sort_direction
- in: query
- name: sort_direction
- type: string
- - description: owner_type
- in: query
- name: owner_type
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get spider list
- tags:
- - spider
- put:
- consumes:
- - application/json
- description: Put schedule
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule item
- in: body
- name: item
- required: true
- schema:
- $ref: '#/definitions/model.Schedule'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Put schedule
- tags:
- - schedule
- /schedules/{id}:
- delete:
- description: Delete schedule
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete schedule
- tags:
- - schedule
- get:
- description: Get schedule by id
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get schedule by id
- tags:
- - schedule
- post:
- consumes:
- - application/json
- description: Post schedule
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- - description: schedule item
- in: body
- name: newItem
- required: true
- schema:
- $ref: '#/definitions/model.Schedule'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Post schedule
- tags:
- - schedule
- /schedules/{id}/disable:
- post:
- consumes:
- - application/json
- description: disable schedule
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: disable schedule
- tags:
- - schedule
- /schedules/{id}/enable:
- post:
- consumes:
- - application/json
- description: enable schedule
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: enable schedule
- tags:
- - schedule
- /setting:
- get:
- description: Get setting
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get setting
- tags:
- - setting
- /spiders:
- post:
- consumes:
- - application/json
- description: delete spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: delete spider
- tags:
- - spider
- put:
- consumes:
- - application/json
- description: Put spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider item
- in: body
- name: spider
- required: true
- schema:
- $ref: '#/definitions/model.Spider'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Put spider
- tags:
- - spider
- /spiders-cancel:
- post:
- consumes:
- - application/json
- description: cancel spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: cancel spider
- tags:
- - spider
- /spiders-run:
- post:
- consumes:
- - application/json
- description: run spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: run spider
- tags:
- - spider
- /spiders/{id}:
- delete:
- description: Delete spider by id
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete spider by id
- tags:
- - spider
- get:
- description: Get spider by id
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get spider by id
- tags:
- - spider
- post:
- consumes:
- - application/json
- description: Post spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- - description: spider item
- in: body
- name: item
- required: true
- schema:
- $ref: '#/definitions/model.Spider'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Post spider
- tags:
- - spider
- /spiders/{id}/copy:
- post:
- consumes:
- - application/json
- description: Copy spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Copy spider
- tags:
- - spider
- /spiders/{id}/dir:
- get:
- description: Get spider dir
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: path
- in: query
- name: path
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get spider dir
- tags:
- - spider
- /spiders/{id}/file:
- delete:
- description: Delete spider file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: path
- in: body
- name: reqBody
- required: true
- schema:
- $ref: '#/definitions/routes.SpiderFileReqBody'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete spider file
- tags:
- - spider
- get:
- description: Get spider file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: path
- in: query
- name: path
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get spider file
- tags:
- - spider
- post:
- description: Put spider file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: path
- in: body
- name: reqBody
- required: true
- schema:
- $ref: '#/definitions/routes.SpiderFileReqBody'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Put spider file
- tags:
- - spider
- put:
- description: Post spider dir
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: path
- in: body
- name: reqBody
- required: true
- schema:
- $ref: '#/definitions/routes.SpiderFileReqBody'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Post spider dir
- tags:
- - spider
- /spiders/{id}/file/rename:
- post:
- description: Rename spider file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: path
- in: body
- name: reqBody
- required: true
- schema:
- $ref: '#/definitions/routes.SpiderFileReqBody'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Rename spider file
- tags:
- - spider
- /spiders/{id}/file/tree:
- get:
- description: Get spider dir
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get spider dir
- tags:
- - spider
- /spiders/{id}/git/reset:
- post:
- description: Post spider reset git
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Post spider reset git
- tags:
- - spider
- /spiders/{id}/git/sync:
- post:
- description: Post spider sync git
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Post spider sync git
- tags:
- - spider
- /spiders/{id}/publish:
- post:
- consumes:
- - application/json
- description: Publish spider
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: schedule id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Publish spider
- tags:
- - spider
- /spiders/{id}/schedules:
- get:
- description: Get schedules
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get schedules
- tags:
- - spider
- /spiders/{id}/scrapy/items:
- get:
- description: Get scrapy spider items
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get scrapy spider items
- tags:
- - spider
- post:
- description: Post scrapy spider items
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: req data
- in: body
- name: reqData
- required: true
- schema:
- items:
- $ref: '#/definitions/entity.ScrapyItem'
- type: entity.ScrapyItem
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Post scrapy spider items
- tags:
- - spider
- /spiders/{id}/scrapy/pipelines:
- get:
- description: Get scrapy spider pipelines
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get scrapy spider pipelines
- tags:
- - spider
- /spiders/{id}/scrapy/settings:
- get:
- description: Get scrapy spider settings
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get scrapy spider settings
- tags:
- - spider
- post:
- description: Get scrapy spider file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- - description: req data
- in: body
- name: reqData
- required: true
- schema:
- items:
- $ref: '#/definitions/entity.ScrapySettingParam'
- type: entity.ScrapySettingParam
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get scrapy spider file
- tags:
- - spider
- /spiders/{id}/scrapy/spider/filepath:
- get:
- description: Get scrapy spider file path
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get scrapy spider file path
- tags:
- - spider
- /spiders/{id}/scrapy/spiders:
- get:
- description: Get scrapy spider file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get scrapy spider file
- tags:
- - spider
- put:
- description: Put scrapy spider file
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Put scrapy spider file
- tags:
- - spider
- /spiders/{id}/stats:
- get:
- description: Get spider stats
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get spider stats
- tags:
- - spider
- /spiders/{id}/tasks:
- get:
- description: Get task list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get task list
- tags:
- - spider
- /spiders/{id}/upload:
- post:
- consumes:
- - application/json
- description: Upload spider by id
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: spider file to upload
- in: formData
- name: file
- required: true
- type: file
- - description: spider id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "500":
- description: Internal Server Error
- schema:
- type: json
- summary: Upload spider by id
- tags:
- - spider
- /stats/home:
- get:
- description: Get home stats
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get home stats
- tags:
- - version
- /system/deps/:
- get:
- description: Get all dep list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: language
- in: path
- name: lang
- required: true
- type: string
- - description: dep name
- in: query
- name: dep_nane
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get all dep list
- tags:
- - system
- /system/deps/{lang}/{dep_name}/json:
- get:
- description: Get dep json
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: language
- in: path
- name: lang
- required: true
- type: string
- - description: dep name
- in: path
- name: dep_name
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get dep json
- tags:
- - system
- /task/{id}:
- delete:
- description: Delete task
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete task
- tags:
- - task
- /tasks:
- delete:
- description: Delete tasks
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete tasks
- tags:
- - task
- get:
- description: Get task list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: req data
- in: body
- name: data
- required: true
- schema:
- $ref: '#/definitions/routes.TaskListRequestData'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get task list
- tags:
- - task
- put:
- description: Put task
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Put task
- tags:
- - task
- /tasks/{id}:
- get:
- description: Get task
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get task
- tags:
- - task
- /tasks/{id}/cancel:
- post:
- description: Cancel task
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Cancel task
- tags:
- - task
- /tasks/{id}/error-log:
- delete:
- description: Get task error log
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get task error log
- tags:
- - task
- /tasks/{id}/log:
- delete:
- description: Get task log
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get task log
- tags:
- - task
- /tasks/{id}/restart:
- post:
- description: Restart task
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Restart task
- tags:
- - task
- /tasks/{id}/results:
- get:
- description: Get task list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: req data
- in: body
- name: data
- required: true
- schema:
- $ref: '#/definitions/routes.TaskResultsRequestData'
- type: object
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get task list
- tags:
- - task
- /tasks/{id}/results/download:
- get:
- description: Get task results
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get task results
- tags:
- - task
- /tasks_by_status:
- delete:
- description: Delete task
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: task status
- in: query
- name: status
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete task
- tags:
- - task
- /tokens:
- get:
- description: token
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get token
- tags:
- - token
- put:
- description: token
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Put token
- tags:
- - token
- /tokens/{id}:
- delete:
- description: Delete token
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: token id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete token
- tags:
- - token
- /users:
- get:
- description: Get user list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: data body
- in: body
- name: data
- required: true
- schema:
- $ref: '#/definitions/routes.UserListRequestData'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get user list
- tags:
- - token
- put:
- description: Put user
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: reqData body
- in: body
- name: reqData
- required: true
- schema:
- $ref: '#/definitions/routes.UserRequestData'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Put user
- tags:
- - user
- /users/{id}:
- delete:
- description: Delete user
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: user id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete user
- tags:
- - user
- get:
- description: user
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: user id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get user
- tags:
- - user
- post:
- description: Post user
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: user body
- in: body
- name: item
- required: true
- schema:
- $ref: '#/definitions/model.User'
- type: object
- - description: user id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Post user
- tags:
- - user
- /variable:
- put:
- description: Put variable
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: reqData body
- in: body
- name: variable
- required: true
- schema:
- $ref: '#/definitions/model.Variable'
- type: object
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Put variable
- tags:
- - variable
- /variable/{id}:
- delete:
- description: Delete variable
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: variable id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Delete variable
- tags:
- - variable
- post:
- description: Post variable
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- - description: reqData body
- in: body
- name: variable
- required: true
- schema:
- $ref: '#/definitions/model.Variable'
- type: object
- - description: variable id
- in: path
- name: id
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Post variable
- tags:
- - variable
- /variables:
- get:
- description: Get variable list
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get variable list
- tags:
- - variable
- /version:
- get:
- description: Get version
- parameters:
- - description: Authorization token
- in: header
- name: Authorization
- required: true
- type: string
- produces:
- - application/json
- responses:
- "200":
- description: OK
- schema:
- type: json
- "400":
- description: Bad Request
- schema:
- type: json
- summary: Get version
- tags:
- - setting
-swagger: "2.0"
diff --git a/backend/dump.rdb b/backend/dump.rdb
deleted file mode 100644
index 6f8d62d2..00000000
Binary files a/backend/dump.rdb and /dev/null differ
diff --git a/backend/entity/common.go b/backend/entity/common.go
deleted file mode 100644
index c46ae4f9..00000000
--- a/backend/entity/common.go
+++ /dev/null
@@ -1,17 +0,0 @@
-package entity
-
-import "strconv"
-
-type Page struct {
- Skip int
- Limit int
- PageNum int
- PageSize int
-}
-
-func (p *Page) GetPage(pageNum string, pageSize string) {
- p.PageNum, _ = strconv.Atoi(pageNum)
- p.PageSize, _ = strconv.Atoi(pageSize)
- p.Skip = p.PageSize * (p.PageNum - 1)
- p.Limit = p.PageSize
-}
diff --git a/backend/entity/config_spider.go b/backend/entity/config_spider.go
deleted file mode 100644
index 054ee2fe..00000000
--- a/backend/entity/config_spider.go
+++ /dev/null
@@ -1,40 +0,0 @@
-package entity
-
-type ConfigSpiderData struct {
- // 通用
- Name string `yaml:"name" json:"name"`
- DisplayName string `yaml:"display_name" json:"display_name"`
- Col string `yaml:"col" json:"col"`
- Remark string `yaml:"remark" json:"remark"`
- Type string `yaml:"type" bson:"type"`
-
- // 可配置爬虫
- Engine string `yaml:"engine" json:"engine"`
- StartUrl string `yaml:"start_url" json:"start_url"`
- StartStage string `yaml:"start_stage" json:"start_stage"`
- Stages []Stage `yaml:"stages" json:"stages"`
- Settings map[string]string `yaml:"settings" json:"settings"`
-
- // 自定义爬虫
- Cmd string `yaml:"cmd" json:"cmd"`
-}
-
-type Stage struct {
- Name string `yaml:"name" json:"name"`
- IsList bool `yaml:"is_list" json:"is_list"`
- ListCss string `yaml:"list_css" json:"list_css"`
- ListXpath string `yaml:"list_xpath" json:"list_xpath"`
- PageCss string `yaml:"page_css" json:"page_css"`
- PageXpath string `yaml:"page_xpath" json:"page_xpath"`
- PageAttr string `yaml:"page_attr" json:"page_attr"`
- Fields []Field `yaml:"fields" json:"fields"`
-}
-
-type Field struct {
- Name string `yaml:"name" json:"name"`
- Css string `yaml:"css" json:"css"`
- Xpath string `yaml:"xpath" json:"xpath"`
- Attr string `yaml:"attr" json:"attr"`
- NextStage string `yaml:"next_stage" json:"next_stage"`
- Remark string `yaml:"remark" json:"remark"`
-}
diff --git a/backend/entity/doc.go b/backend/entity/doc.go
deleted file mode 100644
index b356d38a..00000000
--- a/backend/entity/doc.go
+++ /dev/null
@@ -1,8 +0,0 @@
-package entity
-
-type DocItem struct {
- Title string `json:"title"`
- Url string `json:"url"`
- Path string `json:"path"`
- Children []DocItem `json:"children"`
-}
diff --git a/backend/entity/node.go b/backend/entity/node.go
deleted file mode 100644
index ebe2047f..00000000
--- a/backend/entity/node.go
+++ /dev/null
@@ -1,28 +0,0 @@
-package entity
-
-type NodeMessage struct {
- // 通信类别
- Type string `json:"type"`
-
- // 任务相关
- TaskId string `json:"task_id"` // 任务ID
-
- // 节点相关
- NodeId string `json:"node_id"` // 节点ID
-
- // 日志相关
- LogPath string `json:"log_path"` // 日志路径
- Log string `json:"log"` // 日志
-
- // 系统信息
- SysInfo SystemInfo `json:"sys_info"`
-
- // 爬虫相关
- SpiderId string `json:"spider_id"` //爬虫ID
-
- // 语言相关
- Lang Lang `json:"lang"`
-
- // 错误相关
- Error string `json:"error"`
-}
diff --git a/backend/entity/rpc.go b/backend/entity/rpc.go
deleted file mode 100644
index 48f14b26..00000000
--- a/backend/entity/rpc.go
+++ /dev/null
@@ -1,11 +0,0 @@
-package entity
-
-type RpcMessage struct {
- Id string `json:"id"` // 消息ID
- Method string `json:"method"` // 消息方法
- NodeId string `json:"node_id"` // 节点ID
- Params map[string]string `json:"params"` // 参数
- Timeout int `json:"timeout"` // 超时
- Result string `json:"result"` // 结果
- Error string `json:"error"` // 错误
-}
diff --git a/backend/entity/spider.go b/backend/entity/spider.go
deleted file mode 100644
index 616d3bbf..00000000
--- a/backend/entity/spider.go
+++ /dev/null
@@ -1,17 +0,0 @@
-package entity
-
-type SpiderType struct {
- Type string `json:"type" bson:"_id"`
- Count int `json:"count" bson:"count"`
-}
-
-type ScrapySettingParam struct {
- Key string `json:"key"`
- Value interface{} `json:"value"`
- Type string `json:"type"`
-}
-
-type ScrapyItem struct {
- Name string `json:"name"`
- Fields []string `json:"fields"`
-}
diff --git a/backend/entity/system.go b/backend/entity/system.go
deleted file mode 100644
index f1a24f4b..00000000
--- a/backend/entity/system.go
+++ /dev/null
@@ -1,39 +0,0 @@
-package entity
-
-type SystemInfo struct {
- ARCH string `json:"arch"`
- OS string `json:"os"`
- Hostname string `json:"host_name"`
- NumCpu int `json:"num_cpu"`
- Executables []Executable `json:"executables"`
-}
-
-type Executable struct {
- Path string `json:"path"`
- FileName string `json:"file_name"`
- DisplayName string `json:"display_name"`
-}
-
-type Lang struct {
- Name string `json:"name"`
- ExecutableName string `json:"executable_name"`
- ExecutablePaths []string `json:"executable_paths"`
- DepExecutablePath string `json:"dep_executable_path"`
- LockPath string `json:"lock_path"`
- InstallScript string `json:"install_script"`
- InstallStatus string `json:"install_status"`
- DepFileName string `json:"dep_file_name"`
- InstallDepArgs string `json:"install_dep_cmd"`
- Type string `json:"type"`
-}
-
-type Dependency struct {
- Name string `json:"name"`
- Version string `json:"version"`
- Description string `json:"description"`
- Installed bool `json:"installed"`
-}
-
-type PackageJson struct {
- Dependencies map[string]string `json:"dependencies"`
-}
diff --git a/backend/entity/version.go b/backend/entity/version.go
deleted file mode 100644
index 97a0278d..00000000
--- a/backend/entity/version.go
+++ /dev/null
@@ -1,23 +0,0 @@
-package entity
-
-type Release struct {
- Name string `json:"name"`
- Draft bool `json:"draft"`
- PreRelease bool `json:"pre_release"`
- PublishedAt string `json:"published_at"`
- Body string `json:"body"`
-}
-
-type ReleaseSlices []Release
-
-func (r ReleaseSlices) Len() int {
- return len(r)
-}
-
-func (r ReleaseSlices) Less(i, j int) bool {
- return r[i].PublishedAt < r[j].PublishedAt
-}
-
-func (r ReleaseSlices) Swap(i, j int) {
- r[i], r[j] = r[j], r[i]
-}
diff --git a/backend/errors/errors.go b/backend/errors/errors.go
deleted file mode 100644
index d896e4d4..00000000
--- a/backend/errors/errors.go
+++ /dev/null
@@ -1,54 +0,0 @@
-package errors
-
-import (
- "fmt"
- "net/http"
-)
-
-type Scope int
-
-const (
- ScopeSystem Scope = 1
- ScopeBusiness Scope = 2
-)
-
-type OPError struct {
- HttpCode int
- Message string
- Code int
- Scope Scope
-}
-
-func (O OPError) Error() string {
- var scope string
- switch O.Scope {
- case ScopeSystem:
- scope = "system"
- case ScopeBusiness:
- scope = "business"
- }
- return fmt.Sprintf("%s error: [%d]%s.", scope, O.Code, O.Message)
-}
-
-func NewSystemOPError(code int, message string, httpCodes ...int) *OPError {
- httpCode := http.StatusOK
- if len(httpCodes) > 0 {
- httpCode = httpCodes[0]
- }
- return NewOpError(code, message, ScopeSystem, httpCode)
-}
-func NewOpError(code int, message string, scope Scope, httpCode int) *OPError {
- return &OPError{
- Message: message,
- Code: code,
- Scope: scope,
- HttpCode: httpCode,
- }
-}
-func NewBusinessError(code int, message string, httpCodes ...int) *OPError {
- httpCode := http.StatusOK
- if len(httpCodes) > 0 {
- httpCode = httpCodes[0]
- }
- return NewOpError(code, message, ScopeBusiness, httpCode)
-}
diff --git a/backend/go.mod b/backend/go.mod
index 3f1fda3a..b70e650e 100644
--- a/backend/go.mod
+++ b/backend/go.mod
@@ -1,43 +1,14 @@
module crawlab
-go 1.12
+go 1.15
require (
- github.com/Masterminds/semver v1.4.2 // indirect
- github.com/Masterminds/sprig v2.16.0+incompatible // indirect
- github.com/Unknwon/goconfig v0.0.0-20191126170842-860a72fb44fd
- github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
- github.com/aokoli/goutils v1.0.1 // indirect
- github.com/apex/log v1.1.4
- github.com/cenkalti/backoff/v4 v4.0.2
- github.com/dgrijalva/jwt-go v3.2.0+incompatible
- github.com/fsnotify/fsnotify v1.4.9
+ github.com/apex/log v1.9.0
+ github.com/crawlab-team/crawlab-core v0.6.0-beta.20210802.1344
+ github.com/crawlab-team/go-trace v0.1.0
github.com/gin-gonic/gin v1.6.3
- github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
- github.com/go-playground/validator/v10 v10.3.0
- github.com/gomodule/redigo v2.0.0+incompatible
- github.com/hashicorp/go-sockaddr v1.0.0
- github.com/huandu/xstrings v1.2.0 // indirect
- github.com/imdario/mergo v0.3.6 // indirect
- github.com/imroc/req v0.3.0
- github.com/jaytaylor/html2text v0.0.0-20180606194806-57d518f124b0 // indirect
- github.com/matcornic/hermes v1.2.0
- github.com/mattn/go-runewidth v0.0.3 // indirect
- github.com/olekukonko/tablewriter v0.0.1 // indirect
- github.com/olivere/elastic/v7 v7.0.15
- github.com/pkg/errors v0.9.1
- github.com/satori/go.uuid v1.2.0
- github.com/smartystreets/goconvey v1.6.4
- github.com/spf13/viper v1.7.0
- github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf // indirect
- github.com/swaggo/gin-swagger v1.2.0
- github.com/swaggo/swag v1.6.6
- go.uber.org/atomic v1.6.0
- golang.org/x/sys v0.0.0-20200420163511-1957bb5e6d1f // indirect
- gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
- gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
- gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df
- gopkg.in/russross/blackfriday.v2 v2.0.0 // indirect
- gopkg.in/src-d/go-git.v4 v4.13.1
- gopkg.in/yaml.v2 v2.3.0
+ github.com/spf13/cobra v1.1.3
+ github.com/spf13/viper v1.7.1
+ go.mongodb.org/mongo-driver v1.6.0 // indirect
+ go.uber.org/dig v1.10.0
)
diff --git a/backend/go.sum b/backend/go.sum
index c53a2aaf..cfa705ff 100644
--- a/backend/go.sum
+++ b/backend/go.sum
@@ -4,6 +4,7 @@ cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSR
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
+cloud.google.com/go v0.46.3 h1:AVXDdKsrtX33oR9fbCMu/+c1o8Ofjq6Ku/MInaLVg5Y=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
@@ -14,35 +15,35 @@ dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
-github.com/KyleBanks/depth v1.2.1 h1:5h8fQADFrWtarTdtDudMmGsC7GPbOAu6RVB3ffsVFHc=
github.com/KyleBanks/depth v1.2.1/go.mod h1:jzSb9d0L43HxTQfT+oSA1EEp2q+ne2uh6XgeJcm8brE=
github.com/Masterminds/semver v1.4.2 h1:WBLTQ37jOCzSLtXNdoo8bNM8876KhNqOKvrlGITgsTc=
github.com/Masterminds/semver v1.4.2/go.mod h1:MB6lktGJrhw8PrUyiEoblNEGEQ+RzHPF078ddwwvV3Y=
github.com/Masterminds/sprig v2.16.0+incompatible h1:QZbMUPxRQ50EKAq3LFMnxddMu88/EUUG3qmxwtDmPsY=
github.com/Masterminds/sprig v2.16.0+incompatible/go.mod h1:y6hNFY5UBTIWBxnzTeuNhlNS5hqE0NB0E6fgfo2Br3o=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
-github.com/PuerkitoBio/purell v1.1.0 h1:rmGxhojJlM0tuKtfdvliR84CFHljx9ag64t2xmVkjK4=
github.com/PuerkitoBio/purell v1.1.0/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
-github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI=
github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
-github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578/go.mod h1:uGdkoq3SwY9Y+13GIhn11/XLaGBb4BfwItxLd5jeuXE=
+github.com/StackExchange/wmi v1.2.0 h1:noJEYkMQVlFCEAc+2ma5YyRhlfjcWfZqk5sBRYozdyM=
+github.com/StackExchange/wmi v1.2.0/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
github.com/Unknwon/goconfig v0.0.0-20191126170842-860a72fb44fd h1:+CYOsXi89xOqBkj7CuEJjA2It+j+R3ngUZEydr6mtkw=
github.com/Unknwon/goconfig v0.0.0-20191126170842-860a72fb44fd/go.mod h1:wngxua9XCNjvHjDiTiV26DaKDT+0c63QR6H5hjVUUxw=
+github.com/ajg/form v1.5.1/go.mod h1:uL1WgH+h2mgNtvBq0339dVnzXdBETtL2LeUXaIv25UY=
github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7 h1:uSoVVbwJiQipAclBbw+8quDsfcvFjOpI5iCf4p/cqCs=
github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7/go.mod h1:6zEj6s6u/ghQa61ZWa/C2Aw3RkjiTBOix7dkqa1VLIs=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc h1:cAKDfWh5VpdgMhJosfJnn5/FoN2SRZ4p7fJNX58YPaU=
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751 h1:JYp7IbQjafoB+tBA3gMyHYHrpOtNuDiK/uB5uXxq5wM=
github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZqLG4oE62mJzwPIB8+Tee4RNCL9ulrY=
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239 h1:kFOfPq6dUM1hTo4JG6LR5AXSUEsOjtdm0kw0FtQtMJA=
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
github.com/aokoli/goutils v1.0.1 h1:7fpzNGoJ3VA8qcrm++XEE1QUe0mIwNeLa02Nwq7RDkg=
github.com/aokoli/goutils v1.0.1/go.mod h1:SijmP0QR8LtwsmDs8Yii5Z/S4trXFGFC2oO5g9DP+DQ=
-github.com/apex/log v1.1.4 h1:3Zk+boorIQAAGBrHn0JUtAau4ihMamT4WdnfdnXM1zQ=
-github.com/apex/log v1.1.4/go.mod h1:AlpoD9aScyQfJDVHmLMEcx4oU6LqzkWp4Mg9GdAcEvQ=
-github.com/apex/logs v0.0.4/go.mod h1:XzxuLZ5myVHDy9SAmYpamKKRNApGj54PfYLcFrXqDwo=
+github.com/apex/log v1.9.0 h1:FHtw/xuaM8AgmvDDTI9fiwoAL25Sq2cxojnZICUU8l0=
+github.com/apex/log v1.9.0/go.mod h1:m82fZlWIuiWzWP04XCTXmnX0xRkYYbCdYn8jbJeLBEA=
+github.com/apex/logs v1.0.0/go.mod h1:XzxuLZ5myVHDy9SAmYpamKKRNApGj54PfYLcFrXqDwo=
github.com/aphistic/golf v0.0.0-20180712155816-02c07f170c5a/go.mod h1:3NqKYiepwy8kCu4PNA+aP7WUV72eXWJeP9/r3/K9aLE=
github.com/aphistic/sweet v0.2.0/go.mod h1:fWDlIh/isSE9n6EPsRmC0det+whmX6dJid3stzu0Xys=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
@@ -52,23 +53,49 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPd
github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
github.com/aws/aws-sdk-go v1.20.6/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/aws/aws-sdk-go v1.30.7/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
+github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I=
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJmJgSg28kpZDP6UIiPt0e0Oz0kqKNGyRaWEPv84=
-github.com/cenkalti/backoff/v4 v4.0.2 h1:JIufpQLbh4DkbQoii76ItQIUFzevQSqOLZca4eamEDs=
-github.com/cenkalti/backoff/v4 v4.0.2/go.mod h1:eEew/i+1Q6OrCDZh3WiXYv3+nJwBASZ8Bog/87DQnVg=
-github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
+github.com/cenkalti/backoff/v4 v4.1.0 h1:c8LkOFQTzuO0WBM/ae5HdGQuZPfPxp7lqBRwQRm4fSc=
+github.com/cenkalti/backoff/v4 v4.1.0/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw=
+github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
+github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
+github.com/crawlab-team/crawlab-core v0.0.1 h1:tZ9Rlji5L5uoZ6nRZFqnSGmEl5TdmdxAyxueJDKfUl8=
+github.com/crawlab-team/crawlab-core v0.0.1/go.mod h1:6dJHMvrmIJbfYHhYNeGZkGOLEBvur+yGiFzLCRXx92k=
+github.com/crawlab-team/crawlab-core v0.6.0-beta.20210802.1344 h1:iRFkt5TAQM4T6NWeJfcM/yT1USqskv5SmqKYiDfHb9I=
+github.com/crawlab-team/crawlab-core v0.6.0-beta.20210802.1344/go.mod h1:3K3ZIbZOmks7nJT/nfR338SrXFPHzD0AT1F7TSVuI/w=
+github.com/crawlab-team/crawlab-db v0.0.2/go.mod h1:o7o4rbcyAWlFGHg9VS7V7tM/GqRq+N2mnAXO71cZA78=
+github.com/crawlab-team/crawlab-db v0.1.1 h1:156h2fbbFKXAHs1mxprqRFC8zs2nrdyaG9JKG7patVw=
+github.com/crawlab-team/crawlab-db v0.1.1/go.mod h1:t0VidSjXKzQgACqNSQV5wusXncFtL6lGEiQTbLfNR04=
+github.com/crawlab-team/crawlab-fs v0.0.0/go.mod h1:k2VXprQspLAmbgO5sSpqMjg/xP4iKDkW4RyTWY8eTZM=
+github.com/crawlab-team/crawlab-fs v0.1.0 h1:iKSJJY4Wvea8Qss+zC/tLiZ371VeV75Z3cuqlsxydzY=
+github.com/crawlab-team/crawlab-fs v0.1.0/go.mod h1:dOE0TeWPDz9krwzt1H72rjj0Fn/aHe53yn7GoOZHD0s=
+github.com/crawlab-team/crawlab-grpc v0.1.0 h1:cCAvQMyXCuUumek2EmYBFRU4UwnC84454vwc+vg4PeE=
+github.com/crawlab-team/crawlab-grpc v0.1.0/go.mod h1:W9Yee6xfesxoaqS5K1sF1I1zlH+i6xqwy4lyoBTOdkc=
+github.com/crawlab-team/crawlab-log v0.1.0 h1:0t+lZEojs3Vqb/bMkk2qs3I+1+XdwKG3pMTfeK5PZWM=
+github.com/crawlab-team/crawlab-log v0.1.0/go.mod h1:N8nTTKEbr9ZQSlmw0+HNB4ZAMQF4yVMaJLx8YhXvhNo=
+github.com/crawlab-team/crawlab-vcs v0.1.0 h1:LjtKOOFzx1o7vvgGppC7jt/8lznyvFwwXBYggbSW9+4=
+github.com/crawlab-team/crawlab-vcs v0.1.0/go.mod h1:G6Hnt/3255QCGHO5Q0xJe1AbJE7m5t65E0v7flRJBJM=
+github.com/crawlab-team/go-trace v0.1.0 h1:uCqfdqNfb+NwqdkQrBkcYfQ9iqGJ76MbPw1wK8n7xGg=
+github.com/crawlab-team/go-trace v0.1.0/go.mod h1:LcWyn68HoT+d29CHM8L41pFHxsAcBMF1xjqJmWdyFh8=
+github.com/crawlab-team/goseaweedfs v0.1.6/go.mod h1:u+rwfqb0rnYllTLjCctE/z1Yp+TC8L+CbbWH8E2NstA=
+github.com/crawlab-team/goseaweedfs v0.2.0/go.mod h1:u+rwfqb0rnYllTLjCctE/z1Yp+TC8L+CbbWH8E2NstA=
+github.com/crawlab-team/goseaweedfs v0.6.0-beta.20210725.1917 h1:Kb8AErE3357UO0jPf8Q2wqG/qcmL0hKDwPMaOZ/JjcY=
+github.com/crawlab-team/goseaweedfs v0.6.0-beta.20210725.1917/go.mod h1:u+rwfqb0rnYllTLjCctE/z1Yp+TC8L+CbbWH8E2NstA=
github.com/creack/pty v1.1.7/go.mod h1:lj5s0c3V2DBrqTV7llrYr5NG6My20zk30Fl46Y7DoTY=
+github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -77,25 +104,28 @@ github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZm
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
+github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
+github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
+github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po=
+github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
+github.com/fasthttp/websocket v1.4.2/go.mod h1:smsv/h4PBEBaU0XDTY5UwJTpZv69fQ0FfcLJr21mA6Y=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/fatih/structs v1.0.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568 h1:BHsljHzVlRcyQhjrss6TZTdY2VfCqZPbv5k3iBFa2ZQ=
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568/go.mod h1:xEzjJPgXI435gkrCt3MPfRiAkVrwSbHsst4LCFVfpJc=
github.com/fortytw2/leaktest v1.3.0 h1:u8491cBMTQ8ft8aeV+adlcytMZylmA5nnwwkRZjI8vw=
github.com/fortytw2/leaktest v1.3.0/go.mod h1:jDsjWgpAGjm2CA7WthBh/CdZYEPF31XHquHwclZch5g=
-github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
+github.com/gavv/httpexpect/v2 v2.2.0/go.mod h1:lnd0TqJLrP+wkJk3SFwtrpSlOAZQ7HaaIFuOYbgqgUM=
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
-github.com/gin-contrib/gzip v0.0.1 h1:ezvKOL6jH+jlzdHNE4h9h8q8uMpDQjyl0NN0Jd7jozc=
github.com/gin-contrib/gzip v0.0.1/go.mod h1:fGBJBCdt6qCZuCAOwWuFhBB4OOq9EFqlo5dEaFhhu5w=
github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s=
-github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3 h1:t8FVkw33L+wilf2QiWkw0UV77qRpcH/JHPKGpKa2E8g=
github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-gonic/gin v1.3.0/go.mod h1:7cKuhb5qV2ggCFctp2fJQ+ErvciLZrIeoOSOm6mUr7Y=
-github.com/gin-gonic/gin v1.4.0 h1:3tMoCCfM7ppqsR0ptz/wi1impNpT7/9wQtMZ8lr1mCQ=
github.com/gin-gonic/gin v1.4.0/go.mod h1:OW2EZn3DO8Ln9oIKOvM++LBO+5UPHJJDH72/q/3rZdM=
github.com/gin-gonic/gin v1.6.3 h1:ahKqKTFpO5KTPHxWZjEdPScmYaGtLo8Y4DMHoEsnp14=
github.com/gin-gonic/gin v1.6.3/go.mod h1:75u5sXoLsGZoRN5Sgbi1eraJ4GU3++wFwWzhwvtwp4M=
@@ -103,29 +133,31 @@ github.com/gliderlabs/ssh v0.2.2 h1:6zsha5zo/TWhRhwqCD3+EarCAgZ2yN28ipRnGPnwkI0=
github.com/gliderlabs/ssh v0.2.2/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8 h1:DujepqpGd1hyOd7aW59XpK7Qymp8iy83xq74fLr21is=
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
+github.com/go-git/gcfg v1.5.0 h1:Q5ViNfGF8zFgyJWPqYwA7qGFoMTEiBmdlkcfRmpIMa4=
+github.com/go-git/gcfg v1.5.0/go.mod h1:5m20vg6GwYabIxaOonVkTdrILxQMpEShl1xiMF4ua+E=
+github.com/go-git/go-billy/v5 v5.0.0 h1:7NQHvd9FVid8VL4qVUMm8XifBK+2xCoZ2lSk0agRrHM=
+github.com/go-git/go-billy/v5 v5.0.0/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0=
+github.com/go-git/go-git-fixtures/v4 v4.0.2-0.20200613231340-f56387b50c12 h1:PbKy9zOy4aAKrJ5pibIRpVO2BXnK1Tlcg+caKI7Ox5M=
+github.com/go-git/go-git-fixtures/v4 v4.0.2-0.20200613231340-f56387b50c12/go.mod h1:m+ICp2rF3jDhFgEZ/8yziagdT1C+ZpZcrJjappBCDSw=
+github.com/go-git/go-git/v5 v5.2.0 h1:YPBLG/3UK1we1ohRkncLjaXWLW+HKp5QNM/jTli2JgI=
+github.com/go-git/go-git/v5 v5.2.0/go.mod h1:kh02eMX+wdqqxgNMEyq8YgwlIOsDOa9homkUq1PoTMs=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
-github.com/go-openapi/jsonpointer v0.17.0 h1:nH6xp8XdXHx8dqveo0ZuJBluCO2qGrPbDNZ0dwoRHP0=
+github.com/go-ole/go-ole v1.2.5 h1:t4MGB5xEDZvXI+0rMjjsfBsD7yAgp/s9ZDkL1JndXwY=
+github.com/go-ole/go-ole v1.2.5/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-openapi/jsonpointer v0.17.0/go.mod h1:cOnomiV+CVVwFLk0A/MExoFMjwdsUdVpsRhURCKh+3M=
github.com/go-openapi/jsonpointer v0.19.2/go.mod h1:3akKfEdA7DF1sugOqz1dVQHBcuDBPKZGEoHC/NkiQRg=
-github.com/go-openapi/jsonpointer v0.19.3 h1:gihV7YNZK1iK6Tgwwsxo2rJbD1GTbdm72325Bq8FI3w=
github.com/go-openapi/jsonpointer v0.19.3/go.mod h1:Pl9vOtqEWErmShwVjC8pYs9cog34VGT37dQOVbmoatg=
github.com/go-openapi/jsonreference v0.17.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
-github.com/go-openapi/jsonreference v0.19.0 h1:BqWKpV1dFd+AuiKlgtddwVIFQsuMpxfBDBHGfM2yNpk=
github.com/go-openapi/jsonreference v0.19.0/go.mod h1:g4xxGn04lDIRh0GJb5QlpE3HfopLOL6uZrK/VgnsK9I=
github.com/go-openapi/jsonreference v0.19.2/go.mod h1:jMjeRr2HHw6nAVajTXJ4eiUwohSTlpa0o73RUL1owJc=
-github.com/go-openapi/jsonreference v0.19.3 h1:5cxNfTy0UVC3X8JL5ymxzyoUZmo8iZb+jeTWn7tUa8o=
github.com/go-openapi/jsonreference v0.19.3/go.mod h1:rjx6GuL8TTa9VaixXglHmQmIL98+wF9xc8zWvFonSJ8=
-github.com/go-openapi/spec v0.19.0 h1:A4SZ6IWh3lnjH0rG0Z5lkxazMGBECtrZcbyYQi+64k4=
github.com/go-openapi/spec v0.19.0/go.mod h1:XkF/MOi14NmjsfZ8VtAKf8pIlbZzyoTvZsdfssdxcBI=
-github.com/go-openapi/spec v0.19.4 h1:ixzUSnHTd6hCemgtAJgluaTSGYpLNpJY4mA2DIkdOAo=
github.com/go-openapi/spec v0.19.4/go.mod h1:FpwSN1ksY1eteniUU7X0N/BgJ7a4WvBFVA8Lj9mJglo=
-github.com/go-openapi/swag v0.17.0 h1:iqrgMg7Q7SvtbWLlltPrkMs0UBJI6oTSs79JFRUi880=
github.com/go-openapi/swag v0.17.0/go.mod h1:AByQ+nYG6gQg71GINrmuDXCPWdL640yX49/kXLo40Tg=
github.com/go-openapi/swag v0.19.2/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
-github.com/go-openapi/swag v0.19.5 h1:lTz6Ys4CmqqCQmZPBlbQENR1/GucA2bzYTE12Pw4tFY=
github.com/go-openapi/swag v0.19.5/go.mod h1:POnQmlKehdgb5mhVOsnJFsivZCEZ/vjK9gh66Z9tfKk=
github.com/go-playground/assert/v2 v2.0.1 h1:MsBgLAaY856+nPRTKrp3/OZK38U/wa0CcBYNjji3q3A=
github.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
@@ -133,13 +165,39 @@ github.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8c
github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
github.com/go-playground/universal-translator v0.17.0 h1:icxd5fm+REJzpZx7ZfpaD876Lmtgy7VtROAbHHXk8no=
github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
-github.com/go-playground/validator/v10 v10.2.0 h1:KgJ0snyC2R9VXYN2rneOtQcw5aHQB1Vv0sFl1UcHBOY=
github.com/go-playground/validator/v10 v10.2.0/go.mod h1:uOYAAleCW8F/7oMFd6aG0GOhaH6EGOAJShg8Id5JGkI=
github.com/go-playground/validator/v10 v10.3.0 h1:nZU+7q+yJoFmwvNgv/LnPUkwPal62+b2xXj0AU1Es7o=
github.com/go-playground/validator/v10 v10.3.0/go.mod h1:uOYAAleCW8F/7oMFd6aG0GOhaH6EGOAJShg8Id5JGkI=
+github.com/go-sql-driver/mysql v1.4.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
+github.com/go-stack/stack v1.8.0 h1:5SgMzNM5HxrEjV0ww2lTmX6E2Izsfxas4+YHWRs3Lsk=
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
+github.com/gobuffalo/attrs v0.0.0-20190224210810-a9411de4debd/go.mod h1:4duuawTqi2wkkpB4ePgWMaai6/Kc6WEz83bhFwpHzj0=
+github.com/gobuffalo/depgen v0.0.0-20190329151759-d478694a28d3/go.mod h1:3STtPUQYuzV0gBVOY3vy6CfMm/ljR4pABfrTeHNLHUY=
+github.com/gobuffalo/depgen v0.1.0/go.mod h1:+ifsuy7fhi15RWncXQQKjWS9JPkdah5sZvtHc2RXGlg=
+github.com/gobuffalo/envy v1.6.15/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
+github.com/gobuffalo/envy v1.7.0/go.mod h1:n7DRkBerg/aorDM8kbduw5dN3oXGswK5liaSCx4T5NI=
+github.com/gobuffalo/flect v0.1.0/go.mod h1:d2ehjJqGOH/Kjqcoz+F7jHTBbmDb38yXA598Hb50EGs=
+github.com/gobuffalo/flect v0.1.1/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
+github.com/gobuffalo/flect v0.1.3/go.mod h1:8JCgGVbRjJhVgD6399mQr4fx5rRfGKVzFjbj6RE/9UI=
+github.com/gobuffalo/genny v0.0.0-20190329151137-27723ad26ef9/go.mod h1:rWs4Z12d1Zbf19rlsn0nurr75KqhYp52EAGGxTbBhNk=
+github.com/gobuffalo/genny v0.0.0-20190403191548-3ca520ef0d9e/go.mod h1:80lIj3kVJWwOrXWWMRzzdhW3DsrdjILVil/SFKBzF28=
+github.com/gobuffalo/genny v0.1.0/go.mod h1:XidbUqzak3lHdS//TPu2OgiFB+51Ur5f7CSnXZ/JDvo=
+github.com/gobuffalo/genny v0.1.1/go.mod h1:5TExbEyY48pfunL4QSXxlDOmdsD44RRq4mVZ0Ex28Xk=
+github.com/gobuffalo/gitgen v0.0.0-20190315122116-cc086187d211/go.mod h1:vEHJk/E9DmhejeLeNt7UVvlSGv3ziL+djtTr3yyzcOw=
+github.com/gobuffalo/gogen v0.0.0-20190315121717-8f38393713f5/go.mod h1:V9QVDIxsgKNZs6L2IYiGR8datgMhB577vzTDqypH360=
+github.com/gobuffalo/gogen v0.1.0/go.mod h1:8NTelM5qd8RZ15VjQTFkAW6qOMx5wBbW4dSCS3BY8gg=
+github.com/gobuffalo/gogen v0.1.1/go.mod h1:y8iBtmHmGc4qa3urIyo1shvOD8JftTtfcKi+71xfDNE=
+github.com/gobuffalo/logger v0.0.0-20190315122211-86e12af44bc2/go.mod h1:QdxcLw541hSGtBnhUc4gaNIXRjiDppFGaDqzbrBd3v8=
+github.com/gobuffalo/mapi v1.0.1/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
+github.com/gobuffalo/mapi v1.0.2/go.mod h1:4VAGh89y6rVOvm5A8fKFxYG+wIW6LO1FMTG9hnKStFc=
+github.com/gobuffalo/packd v0.0.0-20190315124812-a385830c7fc0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
+github.com/gobuffalo/packd v0.1.0/go.mod h1:M2Juc+hhDXf/PnmBANFCqx4DM3wRbgDvnVWeG2RIxq4=
+github.com/gobuffalo/packr/v2 v2.0.9/go.mod h1:emmyGweYTm6Kdper+iywB6YK5YzuKchGtJQZ0Odn4pQ=
+github.com/gobuffalo/packr/v2 v2.2.0/go.mod h1:CaAwI0GPIAv+5wKLtv8Afwl+Cm78K/I/VCm/3ptBN+0=
+github.com/gobuffalo/syncx v0.0.0-20190224160051-33c29581e754/go.mod h1:HhnNqWY95UYwwW3uSASeV7vtgYkT2t16hJgV3AEPUpw=
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE=
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
@@ -148,33 +206,49 @@ github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfb
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/protobuf v1.3.3 h1:gyjaxf+svBWX08ZjK86iN9geUJF0H6gp2IRKX6Nf6/I=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
+github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
+github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
+github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
+github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
+github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
+github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
+github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
+github.com/golang/protobuf v1.4.3 h1:JjCZWpVbqXDqFVmTfYWEVTMIYrL/NPdPSCHPJ0T/raM=
+github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
+github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=
+github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/gomodule/redigo v2.0.0+incompatible h1:K/R+8tc58AaqLkqG2Ol3Qk+DR/TlNuhuh457pBFPtt0=
github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
-github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
-github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
+github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-cmp v0.5.2 h1:X2ev0eStA3AbceY54o37/0PQ/UWqKEiiO2dKL5OPaFM=
+github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
-github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y=
+github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
+github.com/gorilla/websocket v1.0.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
github.com/gorilla/websocket v1.4.2/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
+github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4 h1:z53tR0945TRRQO/fLEVPI6SMv7ZflF0TEaTAoU7tOzg=
+github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/hashicorp/consul/api v1.1.0/go.mod h1:VmuI/Lkw1nC05EYQWNKwWGbkg+FbDBtguAZLlVdkD9Q=
@@ -202,10 +276,14 @@ github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/J
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/huandu/xstrings v1.2.0 h1:yPeWdRnmynF7p+lLYz0H2tthW9lqhMJrQV/U7yy4wX0=
github.com/huandu/xstrings v1.2.0/go.mod h1:DvyZB1rfVYsBIigL8HwpZgxHwXozlTgGqn63UyNX5k4=
-github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/imdario/mergo v0.3.9 h1:UauaLniWCFHWd+Jp9oCEkTBj8VO/9DKg3PV3VCNMDIg=
+github.com/imdario/mergo v0.3.9/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/imkira/go-interpol v1.0.0/go.mod h1:z0h2/2T3XF8kyEPpRgJ3kmNv+C43p+I/CoI+jC3w2iA=
github.com/imroc/req v0.3.0 h1:3EioagmlSG+z+KySToa+Ylo3pTFZs+jh3Brl7ngU12U=
github.com/imroc/req v0.3.0/go.mod h1:F+NZ+2EFSo6EFXdeIbpfE9hcC233id70kf0byW97Caw=
+github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
+github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
github.com/jaytaylor/html2text v0.0.0-20180606194806-57d518f124b0 h1:xqgexXAGQgY3HAjNPSaCqn5Aahbo5TKsmhp8VRfr1iQ=
github.com/jaytaylor/html2text v0.0.0-20180606194806-57d518f124b0/go.mod h1:CVKlgaMiht+LXvHG173ujK6JUhZXKb2u/BQtjPDIvyk=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
@@ -213,10 +291,15 @@ github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i
github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik=
+github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
+github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
+github.com/jmoiron/sqlx v1.2.0/go.mod h1:1FEQNm3xlJgrMD+FBdI9+xvCksHtbpVBBw5dYhBSsks=
+github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901 h1:rp+c0RAYOWj8l6qbCUTSiRLG/iKnW3K3/QfPPuSsBt4=
+github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901/go.mod h1:Z86h9688Y0wesXCyonoVr47MasHilkuLMqGhRZ4Hpak=
+github.com/joho/godotenv v1.3.0/go.mod h1:7hK45KPybAkOC6peb+G5yklZfMxEjkZhHbwpqxOKXbg=
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=
github.com/json-iterator/go v1.1.5/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
-github.com/json-iterator/go v1.1.6 h1:MrUvLMLTMxbqFJ9kzlvat/rYZqZnW3u4wkLzWTaFwKs=
github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
github.com/json-iterator/go v1.1.9 h1:9yzud/Ht36ygwatGx56VwCZtlI/2AD15T1X2sjSuGns=
github.com/json-iterator/go v1.1.9/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
@@ -224,21 +307,38 @@ github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
+github.com/k0kubun/colorstring v0.0.0-20150214042306-9440f1994b88/go.mod h1:3w7q1U84EfirKl04SVQ/s7nPm1ZPhiXd34z40TNz36k=
+github.com/karrick/godirwalk v1.8.0/go.mod h1:H5KPZjojv4lE+QYImBI8xVtrBRgYrIVsaRPx4tDPEn4=
+github.com/karrick/godirwalk v1.10.3/go.mod h1:RoGL9dQei4vP9ilrpETWE8CLOZ1kiN0LhBygSwrAsHA=
github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd h1:Coekwdh0v2wtGp9Gmz1Ze3eVRAWJMLokvN3QjdzCHLY=
github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/klauspost/compress v1.8.2/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
+github.com/klauspost/compress v1.9.5 h1:U+CaK85mrNNb4k8BNOfgJtJ/gr6kswUCFj6miSzVC6M=
+github.com/klauspost/compress v1.9.5/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
+github.com/klauspost/cpuid v1.2.1/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/konsorten/go-windows-terminal-sequences v1.0.2 h1:DB17ag19krx9CFsz4o3enTrPXyIXCl+2iCXH/aMAp9s=
+github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
-github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
-github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
+github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=
github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
+github.com/lib/pq v1.0.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
+github.com/lib/pq v1.1.1/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
+github.com/linxGnu/goseaweedfs v0.1.5/go.mod h1:Zwe/7H7FJaPQyMTNKXgv6fhVDw6qi34MMJQp1K0VLNc=
+github.com/linxGnu/gumble v1.0.0 h1:OAJud8Hy4rmV9I5p/KTRiVpwwklMTd9Ankza3Mz7a4M=
+github.com/linxGnu/gumble v1.0.0/go.mod h1:iyhNJpBHvJ0q2Hr41iiZRJyj6LLF47i2a9C9zLiucVY=
+github.com/logrusorgru/aurora v0.0.0-20181002194514-a7b3b318ed4e h1:9MlwzLdW7QSDrhDjFlsEYmxpFyIoXmYRon3dt0io31k=
+github.com/logrusorgru/aurora v0.0.0-20181002194514-a7b3b318ed4e/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4=
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
github.com/mailru/easyjson v0.0.0-20180823135443-60711f1a8329/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
@@ -246,6 +346,8 @@ github.com/mailru/easyjson v0.0.0-20190614124828-94de47d64c63/go.mod h1:C1wdFJiN
github.com/mailru/easyjson v0.0.0-20190626092158-b2ccc519800e/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
github.com/mailru/easyjson v0.7.1 h1:mdxE1MF9o53iCb2Ghj1VfWvh7ZOwHpnVG/xwXrV90U8=
github.com/mailru/easyjson v0.7.1/go.mod h1:KAzv3t3aY1NaHWoQz1+4F1ccyAH66Jk7yos7ldAVICs=
+github.com/markbates/oncer v0.0.0-20181203154359-bf2de49a0be2/go.mod h1:Ld9puTsIW75CHf65OeIOkyKbteujpZVXDpWK6YGZbxE=
+github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kNSCBdG0=
github.com/matcornic/hermes v1.2.0 h1:AuqZpYcTOtTB7cahdevLfnhIpfzmpqw5Czv8vpdnFDU=
github.com/matcornic/hermes v1.2.0/go.mod h1:lujJomb016Xjv8wBnWlNvUdtmvowjjfkqri5J/+1hYc=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
@@ -255,13 +357,12 @@ github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNx
github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
-github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE=
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
github.com/mattn/go-runewidth v0.0.3 h1:a+kO+98RDGEfo6asOGMmpodZq4FNtnGP54yps8BzLR4=
github.com/mattn/go-runewidth v0.0.3/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
-github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
+github.com/mattn/go-sqlite3 v1.9.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
@@ -281,21 +382,26 @@ github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJ
github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
+github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs=
+github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/olekukonko/tablewriter v0.0.1 h1:b3iUnf1v+ppJiOfNX4yxxqfWKMQPZR5yoh8urCTFX88=
github.com/olekukonko/tablewriter v0.0.1/go.mod h1:vsDQFd/mU46D+Z4whnwzcISnGGzXWMclvtLoiIKAKIo=
github.com/olivere/elastic/v7 v7.0.15 h1:v7kX5S+oMFfYKS4ZyzD37GH6lfZSpBo9atynRwBUywE=
github.com/olivere/elastic/v7 v7.0.15/go.mod h1:+FgncZ8ho1QF3NlBo77XbuoTKYHhvEOfFZKIAfHnnDE=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
+github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
+github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/pascaldekloe/goe v0.0.0-20180627143212-57f6aae5913c/go.mod h1:lzWF7FIEvWOWxwDKqyGYQf6ZUaNfKdP144TG7ZOy1lc=
github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtbSNq5BcowyYOWdKo=
-github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/pelletier/go-toml v1.7.0 h1:7utD74fnzVc/cpcyy8sjrlFr5vYpypUixARcHIMIGuI=
+github.com/pelletier/go-toml v1.7.0/go.mod h1:vwGMzjaWMwyfHwgIBhI2YUM4fB6nL6lVAvS1LBMMhTE=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
-github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -306,24 +412,39 @@ github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXP
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
+github.com/prometheus/common v0.4.0 h1:7etb9YClo3a6HjLzfl6rIQaU+FDfi0VSX39io3aQ+DM=
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
+github.com/robfig/cron/v3 v3.0.0 h1:kQ6Cb7aHOHTSzNVNEhmp8EcWKLb4CbiMW9h9VyIhO4E=
+github.com/robfig/cron/v3 v3.0.0/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
github.com/rogpeppe/fastuuid v1.1.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
+github.com/rogpeppe/go-internal v1.1.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
+github.com/rogpeppe/go-internal v1.2.2/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
github.com/satori/go.uuid v1.2.0 h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
+github.com/savsgio/gotils v0.0.0-20200117113501-90175b0fbe3f/go.mod h1:lHhJedqxCoHN+zMtwGNTXWmF0u9Jt363FYRhV6g0CdY=
+github.com/scryner/lfreequeue v0.0.0-20121212074822-473f33702129/go.mod h1:0OrdloYlIayHGsgKYlwEnmdrPWmuYtbdS6Dm71PprFM=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
-github.com/sergi/go-diff v1.0.0 h1:Kpca3qRNrduNnOQeazBd0ysaKrUJiIuISHxogkT9RPQ=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
+github.com/sergi/go-diff v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
+github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
+github.com/shirou/gopsutil v3.20.11+incompatible h1:LJr4ZQK4mPpIV5gOa4jCOKOGb4ty4DZO54I4FGqIpto=
+github.com/shirou/gopsutil v3.20.11+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shurcooL/sanitized_anchor_name v1.0.0 h1:PdmoCO6wvbs+7yrJyMORt4/BmY5IYyJwS/kOiWx8mHo=
github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
+github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
+github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
github.com/smartystreets/assertions v1.0.0 h1:UVQPSSmc3qtTi+zPPkCXvZX9VvW/xT/NsRvKfwY81a8=
github.com/smartystreets/assertions v1.0.0/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
@@ -337,12 +458,17 @@ github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
+github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
+github.com/spf13/cobra v1.1.3 h1:xghbfqPkxzxP3C/f3n5DdpAbdKLj4ZE4BWQI362l53M=
+github.com/spf13/cobra v1.1.3/go.mod h1:pGADOWyqRD/YMrPZigI/zbliZ2wVD/23d+is3pSWzOo=
github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
-github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
-github.com/spf13/viper v1.7.0 h1:xVKxvI7ouOI5I+U9s2eeiUfMaWBVoXA3AWskkrqK0VM=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.7.0/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5qpdg=
+github.com/spf13/viper v1.7.1 h1:pM5oEahlgWv/WnHXpgbKz7iLIxRf65tye2Ci+XFK5sk=
+github.com/spf13/viper v1.7.1/go.mod h1:8WkrPz2fc9jxqZNCJI/76HCieCp4Q8HaLFoCha5qpdg=
github.com/src-d/gcfg v1.4.0 h1:xXbNR5AlLSA315x2UO+fTSSAXCDf+Ar38/6oyGbDKQ4=
github.com/src-d/gcfg v1.4.0/go.mod h1:p/UMsR43ujA89BJY9duynAwIpvqEujIH/jFlfL7jWoI=
github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf h1:pvbZ0lM0XWPBqUKqFU8cmavspvIl9nulOYwdy6IFRRo=
@@ -351,62 +477,91 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.2.0/go.mod h1:qt09Ya8vawLte6SNmTgCsAVtYtaKzEcn8ATUoHMkEqE=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
-github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
-github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
+github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
+github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
github.com/swaggo/files v0.0.0-20190704085106-630677cd5c14/go.mod h1:gxQT6pBGRuIGunNf/+tSOB5OHvguWi8Tbt82WOkf35E=
-github.com/swaggo/gin-swagger v1.2.0 h1:YskZXEiv51fjOMTsXrOetAjrMDfFaXD79PEoQBOe2W0=
github.com/swaggo/gin-swagger v1.2.0/go.mod h1:qlH2+W7zXGZkczuL+r2nEBR2JTT+/lX05Nn6vPhc7OI=
-github.com/swaggo/swag v1.5.1 h1:2Agm8I4K5qb00620mHq0VJ05/KT4FtmALPIcQR9lEZM=
github.com/swaggo/swag v1.5.1/go.mod h1:1Bl9F/ZBpVWh22nY0zmYyASPO1lI/zIwRDrpZU+tv8Y=
-github.com/swaggo/swag v1.6.6 h1:3YX5hmuUyCMT/OqqnjW92gULAfHg3hVjpcPm53N64RY=
github.com/swaggo/swag v1.6.6/go.mod h1:xDhTyuFIujYiN3DKWC/H/83xcfHp+UE/IzWWampG7Zc=
+github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4=
+github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
github.com/tj/assert v0.0.0-20171129193455-018094318fb0/go.mod h1:mZ9/Rh9oLWpLLDRpvE+3b7gP/C2YyLFYxNmcLnPTMe0=
+github.com/tj/assert v0.0.3 h1:Df/BlaZ20mq6kuai7f5z2TvPFiwC3xaWJSDQNiIS3Rk=
+github.com/tj/assert v0.0.3/go.mod h1:Ne6X72Q+TB1AteidzQncjw9PabbMp4PBMZ1k+vd1Pvk=
+github.com/tj/go-buffer v1.1.0/go.mod h1:iyiJpfFcR2B9sXu7KvjbT9fpM4mOelRSDTbntVj52Uc=
github.com/tj/go-elastic v0.0.0-20171221160941-36157cbbebc2/go.mod h1:WjeM0Oo1eNAjXGDx2yma7uG2XoyRZTq1uv3M/o7imD0=
github.com/tj/go-kinesis v0.0.0-20171128231115-08b17f58cb1b/go.mod h1:/yhzCV0xPfx6jb1bBgRFjl5lytqVqZXEaeqWP8lTEao=
github.com/tj/go-spin v1.1.0/go.mod h1:Mg1mzmePZm4dva8Qz60H2lHwmJ2loum4VIrLgVnKwh4=
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
-github.com/ugorji/go v1.1.4 h1:j4s+tAvLfL3bZyefP2SEWmhBzmuIlH/eqNuPdFPgngw=
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
-github.com/ugorji/go v1.1.5-pre h1:jyJKFOSEbdOc2HODrf2qcCkYOdq7zzXqA9bhW5oV4fM=
github.com/ugorji/go v1.1.5-pre/go.mod h1:FwP/aQVg39TXzItUBMwnWp9T9gPQnXw4Poh4/oBQZ/0=
github.com/ugorji/go v1.1.7 h1:/68gy2h+1mWMrwZFeD1kQialdSzAb432dtpeJ42ovdo=
github.com/ugorji/go v1.1.7/go.mod h1:kZn38zHttfInRq0xu/PH0az30d+z6vm202qpg1oXVMw=
github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
-github.com/ugorji/go/codec v1.1.5-pre h1:5YV9PsFAN+ndcCtTM7s60no7nY7eTG3LPtxhSwuxzCs=
github.com/ugorji/go/codec v1.1.5-pre/go.mod h1:tULtS6Gy1AE1yCENaw4Vb//HLH5njI2tfCQDUqRd8fI=
github.com/ugorji/go/codec v1.1.7 h1:2SvQaVZ1ouYrrKKwoSk2pzd4A9evlKJb9oTL+OaLUSs=
github.com/ugorji/go/codec v1.1.7/go.mod h1:Ax+UKWsSmolVDwsd+7N3ZtXu+yMGCf907BLYF3GoBXY=
github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA=
github.com/urfave/cli/v2 v2.1.1/go.mod h1:SE9GqnLQmjVa0iPEY0f1w3ygNIYcIJ0OKPMoW2caLfQ=
+github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
+github.com/valyala/fasthttp v1.9.0/go.mod h1:FstJa9V+Pj9vQ7OJie2qMHdwemEDaDiSdBnvPM1Su9w=
+github.com/valyala/fastrand v1.0.0/go.mod h1:HWqCzkrkg6QXT8V2EXWvXCoow7vLwOFN002oeRzjapQ=
+github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV9WzVtRmSR+PDvWpU/qWl4Wa5LApYYX4ZtKbio=
github.com/xanzy/ssh-agent v0.2.1 h1:TCbipTQL2JiiCprBWx9frJ2eJlCYT00NmctrHxVAr70=
github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4=
+github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
+github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
+github.com/xdg-go/scram v1.0.2 h1:akYIkZ28e6A96dkWNJQu3nmCzH3YfwMPQExUYDaRv7w=
+github.com/xdg-go/scram v1.0.2/go.mod h1:1WAq6h33pAW+iRreB34OORO2Nf7qel3VV3fjBj+hCSs=
+github.com/xdg-go/stringprep v1.0.2 h1:6iq84/ryjjeRmMJwxutI51F2GIPlP5BfTvXHeYjyhBc=
+github.com/xdg-go/stringprep v1.0.2/go.mod h1:8F9zXuvzgwmyT5DUm4GUfZGDdT3W+LCvS6+da4O5kxM=
+github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c/go.mod h1:lB8K/P019DLNhemzwFU4jHLhdvlE6uDZjXFejJXr49I=
+github.com/xdg/stringprep v0.0.0-20180714160509-73f8eece6fdc/go.mod h1:Jhud4/sHMO4oL310DaZAKk9ZaJ08SJfe+sJh0HrGL1Y=
+github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
+github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ=
+github.com/xeipuuv/gojsonschema v1.1.0/go.mod h1:5yf86TLmAcydyeJq5YvxkGPE2fm/u4myDekKRoLuqhs=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
+github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0/go.mod h1:/LWChgwKmvncFJFHJ7Gvn9wZArjbV5/FppcK2fKk/tI=
+github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d h1:splanxYIlg+5LfHAM6xpdFEAYOk8iySO56hMFq6uLyA=
+github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d/go.mod h1:rHwXgn7JulP+udvsHwJoVG1YGAP6VLg4y9I5dyZdqmA=
+github.com/yudai/gojsondiff v1.0.0/go.mod h1:AY32+k2cwILAkW1fbgxQ5mUmMiZFgLIV+FBNExI05xg=
+github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82/go.mod h1:lgjkn3NuSvDfVJdfcVVdX+jpBxNmX4rDAzaS45IcYoM=
+github.com/yudai/pp v2.0.1+incompatible/go.mod h1:PuxR/8QJ7cyCkFp/aUDS+JY727OFEZkTdatxwunjIkc=
+github.com/ztrue/tracerr v0.3.0 h1:lDi6EgEYhPYPnKcjsYzmWw4EkFEoA/gfe+I9Y5f+h6Y=
+github.com/ztrue/tracerr v0.3.0/go.mod h1:qEalzze4VN9O8tnhBXScfCrmoJo10o8TN5ciKjm6Mww=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
+go.mongodb.org/mongo-driver v1.4.5/go.mod h1:WcMNYLx/IlOxLe6JRJiv2uXuCz6zBLndR4SoGjYphSc=
+go.mongodb.org/mongo-driver v1.6.0 h1:ccc26ylcoRWJQRbjU7GvqfxNzwKcoIcEL3BPuFR/pJ0=
+go.mongodb.org/mongo-driver v1.6.0/go.mod h1:Q4oFMbo1+MSNqICAdYMlC/zSTrwCogR4R8NzkI+yfU8=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
-go.uber.org/atomic v1.4.0 h1:cxzIVoETapQEqDhQu3QfnvXAV4AlzcvUCxkVUFw3+EU=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
+go.uber.org/dig v1.10.0 h1:yLmDDj9/zuDjv3gz8GQGviXMs9TfysIUMUilCpgzUJY=
+go.uber.org/dig v1.10.0/go.mod h1:X34SnWGr8Fyla9zQNO2GSO2D+TIuqB14OS8JhYocIyw=
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190219172222-a4c6cb3142f2/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734 h1:p/H982KKEjUnLJkM3tt/LemDnOc1GiZL5FCVlORJ5zo=
+golang.org/x/crypto v0.0.0-20190422162423-af44ce270edf/go.mod h1:WFFai1msRO1wXaEeE5yQxYXgSfI8pQAWXbQop6sCtWE=
golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190530122614-20be4c3c3ed5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4 h1:HuIa8hRrWRSrqYzx1qI49NNxhdi2PrY7gxVSq1JjLDc=
golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97 h1:/UOmuWzQfxxo9UtlXMwuQU8CMgg1eZXqTRwkSQJWKOI=
+golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -443,21 +598,25 @@ golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190611141213-3f473d35a33a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190613194153-d28f0bde5980/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20190724013045-ca1201d0de80 h1:Ao/3l156eZf2AW5wK8a7/smtodRU+gha3+BeqJ69lRk=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20200202094626-16171245cfb2 h1:CCH4IOTTfewWjGOlSp+zGcjutRKlBEZQ6wTn8ozI/nI=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20210226172049-e18ecbb05110 h1:qWPm9rbaAMKs8Bq/9LRpbMqxWRVUAQwMI9fVrssnTfw=
+golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
+golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45 h1:SVwTIAaPC2U/AvvLNZ2a7OVsmBpC8L5BlwK1whH3hm0=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190412183630-56d357773e84/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e h1:vcxGaoTs7kV8m5Np9uUNQin4BrLOthgV7252N8V+FwY=
+golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -470,26 +629,35 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h
golang.org/x/sys v0.0.0-20190221075227-b4e8571b14e0/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI=
+golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190419153524-e8e3143a4f4a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190531175056-4c3a928424d2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190610200419-93c9922d18ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e h1:D5TXcfTk7xF7hvieo4QErS3qqCB4teTffacDWr7CI+0=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200116001909-b77594299b42 h1:vEOn+mP2zCOVzKckCZy6YsCtDblrpj/w7B9nxGNELpg=
golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200420163511-1957bb5e6d1f h1:gWF768j/LaZugp8dyS4UwsslYCYz9XgFxvlgsn0n9H8=
+golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200420163511-1957bb5e6d1f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c h1:F1jZWGFhYfh0Ci55sIpILtKKK8p3i2/krTr0H1rg74I=
+golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.5 h1:i6eZZ+zk0SOf0xgBpEpPD18qWcJda6q1sxt3S0kzyUQ=
+golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -500,20 +668,25 @@ golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190329151228-23e29df326fe/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190416151739-9c9e1878f421/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190420181800-aa740d480789/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
+golang.org/x/tools v0.0.0-20190531172133-b3315ee88b7d/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190606050223-4d9ae51c2468/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190611222205-d73e1c7e250b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
-golang.org/x/tools v0.0.0-20190729092621-ff9f1409240a h1:mEQZbbaBjWyLNy0tmZmgEuQAR8XOQ3hL8GYi3J/NG64=
golang.org/x/tools v0.0.0-20190729092621-ff9f1409240a/go.mod h1:jcCCGcm9btYwXyDqrUWc6MKQKKGJCWEQ3AfLSRIbEuI=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191030062658-86caa796c7ab/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191112195655-aa38f8e97acc h1:NCy3Ohtk6Iny5V/reW2Ktypo4zIpWBdRJ1uFMjBxdg8=
golang.org/x/tools v0.0.0-20191112195655-aa38f8e97acc/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
@@ -527,33 +700,49 @@ google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsb
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
+google.golang.org/appengine v1.6.1 h1:QzqyMA1tlu6CgqCDUtU9V+ZKhLFT2dkJuANu5QaxI3I=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
-google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb h1:i1Ppqkc3WQXikh8bXiwHqAN5Rv3/qDCcRk0/Otx73BY=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
+google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=
+google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
+google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
+google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
+google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
+google.golang.org/grpc v1.34.0 h1:raiipEjMOIC/TO2AvyTxP25XFdLxNIBwzDh3FM3XztI=
+google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA51WJ8=
+google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
+google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
+google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
+google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
+google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
+google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
+google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
+google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
+google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
+google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc h1:2gGKlE2+asNV9m7xrywl36YYNnBG5ZQ0r/BOOxqPpmk=
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc/go.mod h1:m7x9LTH6d71AHyAX77c9yqWCCa3UKHcVEj9y7hAtKDk=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
+gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
-gopkg.in/go-playground/assert.v1 v1.2.1 h1:xoYuJVE7KT85PYWrN730RguIQO0ePzVRfFMXadIrXTM=
gopkg.in/go-playground/assert.v1 v1.2.1/go.mod h1:9RXL0bg/zibRAgZUYszZSwO/z8Y/a8bDuhia5mkpMnE=
-gopkg.in/go-playground/validator.v8 v8.18.2 h1:lFB4DoMU6B626w8ny76MV7VX6W2VHct2GVOI3xgiMrQ=
gopkg.in/go-playground/validator.v8 v8.18.2/go.mod h1:RX2a/7Ha8BgOhfk7j780h4/u/RRjR0eouCJSH80/M2Y=
gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df h1:n7WqCuqOuCbNr617RXOY0AWRXxgwEyPp2z+p0+hgMuE=
gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df/go.mod h1:LRQQ+SO6ZHR7tOkpBDuZnXENFzX8qRjMDMyPD6BRkCw=
@@ -573,15 +762,19 @@ gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10=
gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c h1:grhR+C34yXImVGp7EzNk+DTIk+323eIUWOmEevy6bDo=
+gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
+honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
+moul.io/http2curl v1.0.1-0.20190925090545-5cd742060b0e/go.mod h1:nejbQVfXh96n9dSF6cH3Jsk/QI1Z2oEL7sSI2ifXFNA=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
diff --git a/backend/lib/cron/.gitignore b/backend/lib/cron/.gitignore
deleted file mode 100644
index 00268614..00000000
--- a/backend/lib/cron/.gitignore
+++ /dev/null
@@ -1,22 +0,0 @@
-# Compiled Object files, Static and Dynamic libs (Shared Objects)
-*.o
-*.a
-*.so
-
-# Folders
-_obj
-_test
-
-# Architecture specific extensions/prefixes
-*.[568vq]
-[568vq].out
-
-*.cgo1.go
-*.cgo2.c
-_cgo_defun.c
-_cgo_gotypes.go
-_cgo_export.*
-
-_testmain.go
-
-*.exe
diff --git a/backend/lib/cron/.travis.yml b/backend/lib/cron/.travis.yml
deleted file mode 100644
index 4f2ee4d9..00000000
--- a/backend/lib/cron/.travis.yml
+++ /dev/null
@@ -1 +0,0 @@
-language: go
diff --git a/backend/lib/cron/LICENSE b/backend/lib/cron/LICENSE
deleted file mode 100644
index 3a0f627f..00000000
--- a/backend/lib/cron/LICENSE
+++ /dev/null
@@ -1,21 +0,0 @@
-Copyright (C) 2012 Rob Figueiredo
-All Rights Reserved.
-
-MIT LICENSE
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of
-this software and associated documentation files (the "Software"), to deal in
-the Software without restriction, including without limitation the rights to
-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
-the Software, and to permit persons to whom the Software is furnished to do so,
-subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
-FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
-COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
-IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
diff --git a/backend/lib/cron/README.md b/backend/lib/cron/README.md
deleted file mode 100644
index 979f71e6..00000000
--- a/backend/lib/cron/README.md
+++ /dev/null
@@ -1,125 +0,0 @@
-[](http://godoc.org/github.com/robfig/cron)
-[](https://travis-ci.org/robfig/cron)
-
-# cron
-
-Cron V3 has been released!
-
-To download the specific tagged release, run:
-
- go get github.com/robfig/cron/v3@v3.0.0
-
-Import it in your program as:
-
- import "github.com/robfig/cron/v3"
-
-It requires Go 1.11 or later due to usage of Go Modules.
-
-Refer to the documentation here:
-http://godoc.org/github.com/robfig/cron
-
-The rest of this document describes the the advances in v3 and a list of
-breaking changes for users that wish to upgrade from an earlier version.
-
-## Upgrading to v3 (June 2019)
-
-cron v3 is a major upgrade to the library that addresses all outstanding bugs,
-feature requests, and rough edges. It is based on a merge of master which
-contains various fixes to issues found over the years and the v2 branch which
-contains some backwards-incompatible features like the ability to remove cron
-jobs. In addition, v3 adds support for Go Modules, cleans up rough edges like
-the timezone support, and fixes a number of bugs.
-
-New features:
-
-- Support for Go modules. Callers must now import this library as
- `github.com/robfig/cron/v3`, instead of `gopkg.in/...`
-
-- Fixed bugs:
- - 0f01e6b parser: fix combining of Dow and Dom (#70)
- - dbf3220 adjust times when rolling the clock forward to handle non-existent midnight (#157)
- - eeecf15 spec_test.go: ensure an error is returned on 0 increment (#144)
- - 70971dc cron.Entries(): update request for snapshot to include a reply channel (#97)
- - 1cba5e6 cron: fix: removing a job causes the next scheduled job to run too late (#206)
-
-- Standard cron spec parsing by default (first field is "minute"), with an easy
- way to opt into the seconds field (quartz-compatible). Although, note that the
- year field (optional in Quartz) is not supported.
-
-- Extensible, key/value logging via an interface that complies with
- the https://github.com/go-logr/logr project.
-
-- The new Chain & JobWrapper types allow you to install "interceptors" to add
- cross-cutting behavior like the following:
- - Recover any panics from jobs
- - Delay a job's execution if the previous run hasn't completed yet
- - Skip a job's execution if the previous run hasn't completed yet
- - Log each job's invocations
- - Notification when jobs are completed
-
-It is backwards incompatible with both v1 and v2. These updates are required:
-
-- The v1 branch accepted an optional seconds field at the beginning of the cron
- spec. This is non-standard and has led to a lot of confusion. The new default
- parser conforms to the standard as described by [the Cron wikipedia page].
-
- UPDATING: To retain the old behavior, construct your Cron with a custom
- parser:
-
- // Seconds field, required
- cron.New(cron.WithSeconds())
-
- // Seconds field, optional
- cron.New(
- cron.WithParser(
- cron.SecondOptional | cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor))
-
-- The Cron type now accepts functional options on construction rather than the
- previous ad-hoc behavior modification mechanisms (setting a field, calling a setter).
-
- UPDATING: Code that sets Cron.ErrorLogger or calls Cron.SetLocation must be
- updated to provide those values on construction.
-
-- CRON_TZ is now the recommended way to specify the timezone of a single
- schedule, which is sanctioned by the specification. The legacy "TZ=" prefix
- will continue to be supported since it is unambiguous and easy to do so.
-
- UPDATING: No update is required.
-
-- By default, cron will no longer recover panics in jobs that it runs.
- Recovering can be surprising (see issue #192) and seems to be at odds with
- typical behavior of libraries. Relatedly, the `cron.WithPanicLogger` option
- has been removed to accommodate the more general JobWrapper type.
-
- UPDATING: To opt into panic recovery and configure the panic logger:
-
- cron.New(cron.WithChain(
- cron.Recover(logger), // or use cron.DefaultLogger
- ))
-
-- In adding support for https://github.com/go-logr/logr, `cron.WithVerboseLogger` was
- removed, since it is duplicative with the leveled logging.
-
- UPDATING: Callers should use `WithLogger` and specify a logger that does not
- discard `Info` logs. For convenience, one is provided that wraps `*log.Logger`:
-
- cron.New(
- cron.WithLogger(cron.VerbosePrintfLogger(logger)))
-
-
-### Background - Cron spec format
-
-There are two cron spec formats in common usage:
-
-- The "standard" cron format, described on [the Cron wikipedia page] and used by
- the cron Linux system utility.
-
-- The cron format used by [the Quartz Scheduler], commonly used for scheduled
- jobs in Java software
-
-[the Cron wikipedia page]: https://en.wikipedia.org/wiki/Cron
-[the Quartz Scheduler]: http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/crontrigger.html
-
-The original version of this package included an optional "seconds" field, which
-made it incompatible with both of these formats. Now, the "standard" format is
-the default format accepted, and the Quartz format is opt-in.
diff --git a/backend/lib/cron/chain.go b/backend/lib/cron/chain.go
deleted file mode 100644
index 118e5bbe..00000000
--- a/backend/lib/cron/chain.go
+++ /dev/null
@@ -1,92 +0,0 @@
-package cron
-
-import (
- "fmt"
- "runtime"
- "sync"
- "time"
-)
-
-// JobWrapper decorates the given Job with some behavior.
-type JobWrapper func(Job) Job
-
-// Chain is a sequence of JobWrappers that decorates submitted jobs with
-// cross-cutting behaviors like logging or synchronization.
-type Chain struct {
- wrappers []JobWrapper
-}
-
-// NewChain returns a Chain consisting of the given JobWrappers.
-func NewChain(c ...JobWrapper) Chain {
- return Chain{c}
-}
-
-// Then decorates the given job with all JobWrappers in the chain.
-//
-// This:
-// NewChain(m1, m2, m3).Then(job)
-// is equivalent to:
-// m1(m2(m3(job)))
-func (c Chain) Then(j Job) Job {
- for i := range c.wrappers {
- j = c.wrappers[len(c.wrappers)-i-1](j)
- }
- return j
-}
-
-// Recover panics in wrapped jobs and log them with the provided logger.
-func Recover(logger Logger) JobWrapper {
- return func(j Job) Job {
- return FuncJob(func() {
- defer func() {
- if r := recover(); r != nil {
- const size = 64 << 10
- buf := make([]byte, size)
- buf = buf[:runtime.Stack(buf, false)]
- err, ok := r.(error)
- if !ok {
- err = fmt.Errorf("%v", r)
- }
- logger.Error(err, "panic", "stack", "...\n"+string(buf))
- }
- }()
- j.Run()
- })
- }
-}
-
-// DelayIfStillRunning serializes jobs, delaying subsequent runs until the
-// previous one is complete. Jobs running after a delay of more than a minute
-// have the delay logged at Info.
-func DelayIfStillRunning(logger Logger) JobWrapper {
- return func(j Job) Job {
- var mu sync.Mutex
- return FuncJob(func() {
- start := time.Now()
- mu.Lock()
- defer mu.Unlock()
- if dur := time.Since(start); dur > time.Minute {
- logger.Info("delay", "duration", dur)
- }
- j.Run()
- })
- }
-}
-
-// SkipIfStillRunning skips an invocation of the Job if a previous invocation is
-// still running. It logs skips to the given logger at Info level.
-func SkipIfStillRunning(logger Logger) JobWrapper {
- var ch = make(chan struct{}, 1)
- ch <- struct{}{}
- return func(j Job) Job {
- return FuncJob(func() {
- select {
- case v := <-ch:
- j.Run()
- ch <- v
- default:
- logger.Info("skip")
- }
- })
- }
-}
diff --git a/backend/lib/cron/chain_test.go b/backend/lib/cron/chain_test.go
deleted file mode 100644
index 2561bd7f..00000000
--- a/backend/lib/cron/chain_test.go
+++ /dev/null
@@ -1,221 +0,0 @@
-package cron
-
-import (
- "io/ioutil"
- "log"
- "reflect"
- "sync"
- "testing"
- "time"
-)
-
-func appendingJob(slice *[]int, value int) Job {
- var m sync.Mutex
- return FuncJob(func() {
- m.Lock()
- *slice = append(*slice, value)
- m.Unlock()
- })
-}
-
-func appendingWrapper(slice *[]int, value int) JobWrapper {
- return func(j Job) Job {
- return FuncJob(func() {
- appendingJob(slice, value).Run()
- j.Run()
- })
- }
-}
-
-func TestChain(t *testing.T) {
- var nums []int
- var (
- append1 = appendingWrapper(&nums, 1)
- append2 = appendingWrapper(&nums, 2)
- append3 = appendingWrapper(&nums, 3)
- append4 = appendingJob(&nums, 4)
- )
- NewChain(append1, append2, append3).Then(append4).Run()
- if !reflect.DeepEqual(nums, []int{1, 2, 3, 4}) {
- t.Error("unexpected order of calls:", nums)
- }
-}
-
-func TestChainRecover(t *testing.T) {
- panickingJob := FuncJob(func() {
- panic("panickingJob panics")
- })
-
- t.Run("panic exits job by default", func(t *testing.T) {
- defer func() {
- if err := recover(); err == nil {
- t.Errorf("panic expected, but none received")
- }
- }()
- NewChain().Then(panickingJob).
- Run()
- })
-
- t.Run("Recovering JobWrapper recovers", func(t *testing.T) {
- NewChain(Recover(PrintfLogger(log.New(ioutil.Discard, "", 0)))).
- Then(panickingJob).
- Run()
- })
-
- t.Run("composed with the *IfStillRunning wrappers", func(t *testing.T) {
- NewChain(Recover(PrintfLogger(log.New(ioutil.Discard, "", 0)))).
- Then(panickingJob).
- Run()
- })
-}
-
-type countJob struct {
- m sync.Mutex
- started int
- done int
- delay time.Duration
-}
-
-func (j *countJob) Run() {
- j.m.Lock()
- j.started++
- j.m.Unlock()
- time.Sleep(j.delay)
- j.m.Lock()
- j.done++
- j.m.Unlock()
-}
-
-func (j *countJob) Started() int {
- defer j.m.Unlock()
- j.m.Lock()
- return j.started
-}
-
-func (j *countJob) Done() int {
- defer j.m.Unlock()
- j.m.Lock()
- return j.done
-}
-
-func TestChainDelayIfStillRunning(t *testing.T) {
-
- t.Run("runs immediately", func(t *testing.T) {
- var j countJob
- wrappedJob := NewChain(DelayIfStillRunning(DiscardLogger)).Then(&j)
- go wrappedJob.Run()
- time.Sleep(2 * time.Millisecond) // Give the job 2ms to complete.
- if c := j.Done(); c != 1 {
- t.Errorf("expected job run once, immediately, got %d", c)
- }
- })
-
- t.Run("second run immediate if first done", func(t *testing.T) {
- var j countJob
- wrappedJob := NewChain(DelayIfStillRunning(DiscardLogger)).Then(&j)
- go func() {
- go wrappedJob.Run()
- time.Sleep(time.Millisecond)
- go wrappedJob.Run()
- }()
- time.Sleep(3 * time.Millisecond) // Give both jobs 3ms to complete.
- if c := j.Done(); c != 2 {
- t.Errorf("expected job run twice, immediately, got %d", c)
- }
- })
-
- t.Run("second run delayed if first not done", func(t *testing.T) {
- var j countJob
- j.delay = 10 * time.Millisecond
- wrappedJob := NewChain(DelayIfStillRunning(DiscardLogger)).Then(&j)
- go func() {
- go wrappedJob.Run()
- time.Sleep(time.Millisecond)
- go wrappedJob.Run()
- }()
-
- // After 5ms, the first job is still in progress, and the second job was
- // run but should be waiting for it to finish.
- time.Sleep(5 * time.Millisecond)
- started, done := j.Started(), j.Done()
- if started != 1 || done != 0 {
- t.Error("expected first job started, but not finished, got", started, done)
- }
-
- // Verify that the second job completes.
- time.Sleep(25 * time.Millisecond)
- started, done = j.Started(), j.Done()
- if started != 2 || done != 2 {
- t.Error("expected both jobs done, got", started, done)
- }
- })
-
-}
-
-func TestChainSkipIfStillRunning(t *testing.T) {
-
- t.Run("runs immediately", func(t *testing.T) {
- var j countJob
- wrappedJob := NewChain(SkipIfStillRunning(DiscardLogger)).Then(&j)
- go wrappedJob.Run()
- time.Sleep(2 * time.Millisecond) // Give the job 2ms to complete.
- if c := j.Done(); c != 1 {
- t.Errorf("expected job run once, immediately, got %d", c)
- }
- })
-
- t.Run("second run immediate if first done", func(t *testing.T) {
- var j countJob
- wrappedJob := NewChain(SkipIfStillRunning(DiscardLogger)).Then(&j)
- go func() {
- go wrappedJob.Run()
- time.Sleep(time.Millisecond)
- go wrappedJob.Run()
- }()
- time.Sleep(3 * time.Millisecond) // Give both jobs 3ms to complete.
- if c := j.Done(); c != 2 {
- t.Errorf("expected job run twice, immediately, got %d", c)
- }
- })
-
- t.Run("second run skipped if first not done", func(t *testing.T) {
- var j countJob
- j.delay = 10 * time.Millisecond
- wrappedJob := NewChain(SkipIfStillRunning(DiscardLogger)).Then(&j)
- go func() {
- go wrappedJob.Run()
- time.Sleep(time.Millisecond)
- go wrappedJob.Run()
- }()
-
- // After 5ms, the first job is still in progress, and the second job was
- // aleady skipped.
- time.Sleep(5 * time.Millisecond)
- started, done := j.Started(), j.Done()
- if started != 1 || done != 0 {
- t.Error("expected first job started, but not finished, got", started, done)
- }
-
- // Verify that the first job completes and second does not run.
- time.Sleep(25 * time.Millisecond)
- started, done = j.Started(), j.Done()
- if started != 1 || done != 1 {
- t.Error("expected second job skipped, got", started, done)
- }
- })
-
- t.Run("skip 10 jobs on rapid fire", func(t *testing.T) {
- var j countJob
- j.delay = 10 * time.Millisecond
- wrappedJob := NewChain(SkipIfStillRunning(DiscardLogger)).Then(&j)
- for i := 0; i < 11; i++ {
- go wrappedJob.Run()
- }
- time.Sleep(200 * time.Millisecond)
- done := j.Done()
- if done != 1 {
- t.Error("expected 1 jobs executed, 10 jobs dropped, got", done)
- }
- })
-
-}
diff --git a/backend/lib/cron/constantdelay.go b/backend/lib/cron/constantdelay.go
deleted file mode 100644
index cd6e7b1b..00000000
--- a/backend/lib/cron/constantdelay.go
+++ /dev/null
@@ -1,27 +0,0 @@
-package cron
-
-import "time"
-
-// ConstantDelaySchedule represents a simple recurring duty cycle, e.g. "Every 5 minutes".
-// It does not support jobs more frequent than once a second.
-type ConstantDelaySchedule struct {
- Delay time.Duration
-}
-
-// Every returns a crontab Schedule that activates once every duration.
-// Delays of less than a second are not supported (will round up to 1 second).
-// Any fields less than a Second are truncated.
-func Every(duration time.Duration) ConstantDelaySchedule {
- if duration < time.Second {
- duration = time.Second
- }
- return ConstantDelaySchedule{
- Delay: duration - time.Duration(duration.Nanoseconds())%time.Second,
- }
-}
-
-// Next returns the next time this should be run.
-// This rounds so that the next activation time will be on the second.
-func (schedule ConstantDelaySchedule) Next(t time.Time) time.Time {
- return t.Add(schedule.Delay - time.Duration(t.Nanosecond())*time.Nanosecond)
-}
diff --git a/backend/lib/cron/constantdelay_test.go b/backend/lib/cron/constantdelay_test.go
deleted file mode 100644
index f43a58ad..00000000
--- a/backend/lib/cron/constantdelay_test.go
+++ /dev/null
@@ -1,54 +0,0 @@
-package cron
-
-import (
- "testing"
- "time"
-)
-
-func TestConstantDelayNext(t *testing.T) {
- tests := []struct {
- time string
- delay time.Duration
- expected string
- }{
- // Simple cases
- {"Mon Jul 9 14:45 2012", 15*time.Minute + 50*time.Nanosecond, "Mon Jul 9 15:00 2012"},
- {"Mon Jul 9 14:59 2012", 15 * time.Minute, "Mon Jul 9 15:14 2012"},
- {"Mon Jul 9 14:59:59 2012", 15 * time.Minute, "Mon Jul 9 15:14:59 2012"},
-
- // Wrap around hours
- {"Mon Jul 9 15:45 2012", 35 * time.Minute, "Mon Jul 9 16:20 2012"},
-
- // Wrap around days
- {"Mon Jul 9 23:46 2012", 14 * time.Minute, "Tue Jul 10 00:00 2012"},
- {"Mon Jul 9 23:45 2012", 35 * time.Minute, "Tue Jul 10 00:20 2012"},
- {"Mon Jul 9 23:35:51 2012", 44*time.Minute + 24*time.Second, "Tue Jul 10 00:20:15 2012"},
- {"Mon Jul 9 23:35:51 2012", 25*time.Hour + 44*time.Minute + 24*time.Second, "Thu Jul 11 01:20:15 2012"},
-
- // Wrap around months
- {"Mon Jul 9 23:35 2012", 91*24*time.Hour + 25*time.Minute, "Thu Oct 9 00:00 2012"},
-
- // Wrap around minute, hour, day, month, and year
- {"Mon Dec 31 23:59:45 2012", 15 * time.Second, "Tue Jan 1 00:00:00 2013"},
-
- // Round to nearest second on the delay
- {"Mon Jul 9 14:45 2012", 15*time.Minute + 50*time.Nanosecond, "Mon Jul 9 15:00 2012"},
-
- // Round up to 1 second if the duration is less.
- {"Mon Jul 9 14:45:00 2012", 15 * time.Millisecond, "Mon Jul 9 14:45:01 2012"},
-
- // Round to nearest second when calculating the next time.
- {"Mon Jul 9 14:45:00.005 2012", 15 * time.Minute, "Mon Jul 9 15:00 2012"},
-
- // Round to nearest second for both.
- {"Mon Jul 9 14:45:00.005 2012", 15*time.Minute + 50*time.Nanosecond, "Mon Jul 9 15:00 2012"},
- }
-
- for _, c := range tests {
- actual := Every(c.delay).Next(getTime(c.time))
- expected := getTime(c.expected)
- if actual != expected {
- t.Errorf("%s, \"%s\": (expected) %v != %v (actual)", c.time, c.delay, expected, actual)
- }
- }
-}
diff --git a/backend/lib/cron/cron.go b/backend/lib/cron/cron.go
deleted file mode 100644
index f6e451db..00000000
--- a/backend/lib/cron/cron.go
+++ /dev/null
@@ -1,350 +0,0 @@
-package cron
-
-import (
- "context"
- "sort"
- "sync"
- "time"
-)
-
-// Cron keeps track of any number of entries, invoking the associated func as
-// specified by the schedule. It may be started, stopped, and the entries may
-// be inspected while running.
-type Cron struct {
- entries []*Entry
- chain Chain
- stop chan struct{}
- add chan *Entry
- remove chan EntryID
- snapshot chan chan []Entry
- running bool
- logger Logger
- runningMu sync.Mutex
- location *time.Location
- parser Parser
- nextID EntryID
- jobWaiter sync.WaitGroup
-}
-
-// Job is an interface for submitted cron jobs.
-type Job interface {
- Run()
-}
-
-// Schedule describes a job's duty cycle.
-type Schedule interface {
- // Next returns the next activation time, later than the given time.
- // Next is invoked initially, and then each time the job is run.
- Next(time.Time) time.Time
-}
-
-// EntryID identifies an entry within a Cron instance
-type EntryID int
-
-// Entry consists of a schedule and the func to execute on that schedule.
-type Entry struct {
- // ID is the cron-assigned ID of this entry, which may be used to look up a
- // snapshot or remove it.
- ID EntryID
-
- // Schedule on which this job should be run.
- Schedule Schedule
-
- // Next time the job will run, or the zero time if Cron has not been
- // started or this entry's schedule is unsatisfiable
- Next time.Time
-
- // Prev is the last time this job was run, or the zero time if never.
- Prev time.Time
-
- // WrappedJob is the thing to run when the Schedule is activated.
- WrappedJob Job
-
- // Job is the thing that was submitted to cron.
- // It is kept around so that user code that needs to get at the job later,
- // e.g. via Entries() can do so.
- Job Job
-}
-
-// Valid returns true if this is not the zero entry.
-func (e Entry) Valid() bool { return e.ID != 0 }
-
-// byTime is a wrapper for sorting the entry array by time
-// (with zero time at the end).
-type byTime []*Entry
-
-func (s byTime) Len() int { return len(s) }
-func (s byTime) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
-func (s byTime) Less(i, j int) bool {
- // Two zero times should return false.
- // Otherwise, zero is "greater" than any other time.
- // (To sort it at the end of the list.)
- if s[i].Next.IsZero() {
- return false
- }
- if s[j].Next.IsZero() {
- return true
- }
- return s[i].Next.Before(s[j].Next)
-}
-
-// New returns a new Cron job runner, modified by the given options.
-//
-// Available Settings
-//
-// Time Zone
-// Description: The time zone in which schedules are interpreted
-// Default: time.Local
-//
-// Parser
-// Description: Parser converts cron spec strings into cron.Schedules.
-// Default: Accepts this spec: https://en.wikipedia.org/wiki/Cron
-//
-// Chain
-// Description: Wrap submitted jobs to customize behavior.
-// Default: A chain that recovers panics and logs them to stderr.
-//
-// See "cron.With*" to modify the default behavior.
-func New(opts ...Option) *Cron {
- c := &Cron{
- entries: nil,
- chain: NewChain(),
- add: make(chan *Entry),
- stop: make(chan struct{}),
- snapshot: make(chan chan []Entry),
- remove: make(chan EntryID),
- running: false,
- runningMu: sync.Mutex{},
- logger: DefaultLogger,
- location: time.Local,
- parser: standardParser,
- }
- for _, opt := range opts {
- opt(c)
- }
- return c
-}
-
-// FuncJob is a wrapper that turns a func() into a cron.Job
-type FuncJob func()
-
-func (f FuncJob) Run() { f() }
-
-// AddFunc adds a func to the Cron to be run on the given schedule.
-// The spec is parsed using the time zone of this Cron instance as the default.
-// An opaque ID is returned that can be used to later remove it.
-func (c *Cron) AddFunc(spec string, cmd func()) (EntryID, error) {
- return c.AddJob(spec, FuncJob(cmd))
-}
-
-// AddJob adds a Job to the Cron to be run on the given schedule.
-// The spec is parsed using the time zone of this Cron instance as the default.
-// An opaque ID is returned that can be used to later remove it.
-func (c *Cron) AddJob(spec string, cmd Job) (EntryID, error) {
- schedule, err := c.parser.Parse(spec)
- if err != nil {
- return 0, err
- }
- return c.Schedule(schedule, cmd), nil
-}
-
-// Schedule adds a Job to the Cron to be run on the given schedule.
-// The job is wrapped with the configured Chain.
-func (c *Cron) Schedule(schedule Schedule, cmd Job) EntryID {
- c.runningMu.Lock()
- defer c.runningMu.Unlock()
- c.nextID++
- entry := &Entry{
- ID: c.nextID,
- Schedule: schedule,
- WrappedJob: c.chain.Then(cmd),
- Job: cmd,
- }
- if !c.running {
- c.entries = append(c.entries, entry)
- } else {
- c.add <- entry
- }
- return entry.ID
-}
-
-// Entries returns a snapshot of the cron entries.
-func (c *Cron) Entries() []Entry {
- c.runningMu.Lock()
- defer c.runningMu.Unlock()
- if c.running {
- replyChan := make(chan []Entry, 1)
- c.snapshot <- replyChan
- return <-replyChan
- }
- return c.entrySnapshot()
-}
-
-// Location gets the time zone location
-func (c *Cron) Location() *time.Location {
- return c.location
-}
-
-// Entry returns a snapshot of the given entry, or nil if it couldn't be found.
-func (c *Cron) Entry(id EntryID) Entry {
- for _, entry := range c.Entries() {
- if id == entry.ID {
- return entry
- }
- }
- return Entry{}
-}
-
-// Remove an entry from being run in the future.
-func (c *Cron) Remove(id EntryID) {
- c.runningMu.Lock()
- defer c.runningMu.Unlock()
- if c.running {
- c.remove <- id
- } else {
- c.removeEntry(id)
- }
-}
-
-// Start the cron scheduler in its own goroutine, or no-op if already started.
-func (c *Cron) Start() {
- c.runningMu.Lock()
- defer c.runningMu.Unlock()
- if c.running {
- return
- }
- c.running = true
- go c.run()
-}
-
-// Run the cron scheduler, or no-op if already running.
-func (c *Cron) Run() {
- c.runningMu.Lock()
- if c.running {
- c.runningMu.Unlock()
- return
- }
- c.running = true
- c.runningMu.Unlock()
- c.run()
-}
-
-// run the scheduler.. this is private just due to the need to synchronize
-// access to the 'running' state variable.
-func (c *Cron) run() {
- c.logger.Info("start")
-
- // Figure out the next activation times for each entry.
- now := c.now()
- for _, entry := range c.entries {
- entry.Next = entry.Schedule.Next(now)
- c.logger.Info("schedule", "now", now, "entry", entry.ID, "next", entry.Next)
- }
-
- for {
- // Determine the next entry to run.
- sort.Sort(byTime(c.entries))
-
- var timer *time.Timer
- if len(c.entries) == 0 || c.entries[0].Next.IsZero() {
- // If there are no entries yet, just sleep - it still handles new entries
- // and stop requests.
- timer = time.NewTimer(100000 * time.Hour)
- } else {
- timer = time.NewTimer(c.entries[0].Next.Sub(now))
- }
-
- for {
- select {
- case now = <-timer.C:
- now = now.In(c.location)
- c.logger.Info("wake", "now", now)
-
- // Run every entry whose next time was less than now
- for _, e := range c.entries {
- if e.Next.After(now) || e.Next.IsZero() {
- break
- }
- c.startJob(e.WrappedJob)
- e.Prev = e.Next
- e.Next = e.Schedule.Next(now)
- c.logger.Info("run", "now", now, "entry", e.ID, "next", e.Next)
- }
-
- case newEntry := <-c.add:
- timer.Stop()
- now = c.now()
- newEntry.Next = newEntry.Schedule.Next(now)
- c.entries = append(c.entries, newEntry)
- c.logger.Info("added", "now", now, "entry", newEntry.ID, "next", newEntry.Next)
-
- case replyChan := <-c.snapshot:
- replyChan <- c.entrySnapshot()
- continue
-
- case <-c.stop:
- timer.Stop()
- c.logger.Info("stop")
- return
-
- case id := <-c.remove:
- timer.Stop()
- now = c.now()
- c.removeEntry(id)
- c.logger.Info("removed", "entry", id)
- }
-
- break
- }
- }
-}
-
-// startJob runs the given job in a new goroutine.
-func (c *Cron) startJob(j Job) {
- c.jobWaiter.Add(1)
- go func() {
- defer c.jobWaiter.Done()
- j.Run()
- }()
-}
-
-// now returns current time in c location
-func (c *Cron) now() time.Time {
- return time.Now().In(c.location)
-}
-
-// Stop stops the cron scheduler if it is running; otherwise it does nothing.
-// A context is returned so the caller can wait for running jobs to complete.
-func (c *Cron) Stop() context.Context {
- c.runningMu.Lock()
- defer c.runningMu.Unlock()
- if c.running {
- c.stop <- struct{}{}
- c.running = false
- }
- ctx, cancel := context.WithCancel(context.Background())
- go func() {
- c.jobWaiter.Wait()
- cancel()
- }()
- return ctx
-}
-
-// entrySnapshot returns a copy of the current cron entry list.
-func (c *Cron) entrySnapshot() []Entry {
- var entries = make([]Entry, len(c.entries))
- for i, e := range c.entries {
- entries[i] = *e
- }
- return entries
-}
-
-func (c *Cron) removeEntry(id EntryID) {
- var entries []*Entry
- for _, e := range c.entries {
- if e.ID != id {
- entries = append(entries, e)
- }
- }
- c.entries = entries
-}
diff --git a/backend/lib/cron/cron_test.go b/backend/lib/cron/cron_test.go
deleted file mode 100644
index 35266df1..00000000
--- a/backend/lib/cron/cron_test.go
+++ /dev/null
@@ -1,699 +0,0 @@
-package cron
-
-import (
- "bytes"
- "fmt"
- "log"
- "strings"
- "sync"
- "sync/atomic"
- "testing"
- "time"
-)
-
-// Many tests schedule a job for every second, and then wait at most a second
-// for it to run. This amount is just slightly larger than 1 second to
-// compensate for a few milliseconds of runtime.
-const OneSecond = 1*time.Second + 50*time.Millisecond
-
-type syncWriter struct {
- wr bytes.Buffer
- m sync.Mutex
-}
-
-func (sw *syncWriter) Write(data []byte) (n int, err error) {
- sw.m.Lock()
- n, err = sw.wr.Write(data)
- sw.m.Unlock()
- return
-}
-
-func (sw *syncWriter) String() string {
- sw.m.Lock()
- defer sw.m.Unlock()
- return sw.wr.String()
-}
-
-func newBufLogger(sw *syncWriter) Logger {
- return PrintfLogger(log.New(sw, "", log.LstdFlags))
-}
-
-func TestFuncPanicRecovery(t *testing.T) {
- var buf syncWriter
- cron := New(WithParser(secondParser),
- WithChain(Recover(newBufLogger(&buf))))
- cron.Start()
- defer cron.Stop()
- _, _ = cron.AddFunc("* * * * * ?", func() {
- panic("YOLO")
- })
- <-time.After(OneSecond)
- if !strings.Contains(buf.String(), "YOLO") {
- t.Error("expected a panic to be logged, got none")
- }
-
-}
-
-type DummyJob struct{}
-
-func (d DummyJob) Run() {
- panic("YOLO")
-}
-
-func TestJobPanicRecovery(t *testing.T) {
- var job DummyJob
-
- var buf syncWriter
- cron := New(WithParser(secondParser),
- WithChain(Recover(newBufLogger(&buf))))
- cron.Start()
- defer cron.Stop()
- _, _ = cron.AddJob("* * * * * ?", job)
-
- select {
- case <-time.After(OneSecond):
- if !strings.Contains(buf.String(), "YOLO") {
- t.Error("expected a panic to be logged, got none")
- }
- return
- }
-}
-
-// Start and stop cron with no entries.
-func TestNoEntries(t *testing.T) {
- cron := newWithSeconds()
- cron.Start()
-
- select {
- case <-time.After(OneSecond):
- t.Fatal("expected cron will be stopped immediately")
- case <-stop(cron):
- }
-}
-
-// Start, stop, then add an entry. Verify entry doesn't run.
-func TestStopCausesJobsToNotRun(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := newWithSeconds()
- cron.Start()
- cron.Stop()
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
-
- select {
- case <-time.After(OneSecond):
- // No job ran!
- case <-wait(wg):
- t.Fatal("expected stopped cron does not run any job")
- }
-}
-
-// Add a job, start cron, expect it runs.
-func TestAddBeforeRunning(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := newWithSeconds()
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
- cron.Start()
- defer cron.Stop()
-
- // Give cron 2 seconds to run our job (which is always activated).
- select {
- case <-time.After(OneSecond):
- t.Fatal("expected job runs")
- case <-wait(wg):
- }
-}
-
-// Start cron, add a job, expect it runs.
-func TestAddWhileRunning(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := newWithSeconds()
- cron.Start()
- defer cron.Stop()
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
-
- select {
- case <-time.After(OneSecond):
- t.Fatal("expected job runs")
- case <-wait(wg):
- }
-}
-
-// Test for #34. Adding a job after calling start results in multiple job invocations
-func TestAddWhileRunningWithDelay(t *testing.T) {
- cron := newWithSeconds()
- cron.Start()
- defer cron.Stop()
- time.Sleep(5 * time.Second)
- var calls int64
- _, _ = cron.AddFunc("* * * * * *", func() { atomic.AddInt64(&calls, 1) })
-
- <-time.After(OneSecond)
- if atomic.LoadInt64(&calls) != 1 {
- t.Errorf("called %d times, expected 1\n", calls)
- }
-}
-
-// Add a job, remove a job, start cron, expect nothing runs.
-func TestRemoveBeforeRunning(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := newWithSeconds()
- id, _ := cron.AddFunc("* * * * * ?", func() { wg.Done() })
- cron.Remove(id)
- cron.Start()
- defer cron.Stop()
-
- select {
- case <-time.After(OneSecond):
- // Success, shouldn't run
- case <-wait(wg):
- t.FailNow()
- }
-}
-
-// Start cron, add a job, remove it, expect it doesn't run.
-func TestRemoveWhileRunning(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := newWithSeconds()
- cron.Start()
- defer cron.Stop()
- id, _ := cron.AddFunc("* * * * * ?", func() { wg.Done() })
- cron.Remove(id)
-
- select {
- case <-time.After(OneSecond):
- case <-wait(wg):
- t.FailNow()
- }
-}
-
-// Test timing with Entries.
-func TestSnapshotEntries(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := New()
- _, _ = cron.AddFunc("@every 2s", func() { wg.Done() })
- cron.Start()
- defer cron.Stop()
-
- // Cron should fire in 2 seconds. After 1 second, call Entries.
- select {
- case <-time.After(OneSecond):
- cron.Entries()
- }
-
- // Even though Entries was called, the cron should fire at the 2 second mark.
- select {
- case <-time.After(OneSecond):
- t.Error("expected job runs at 2 second mark")
- case <-wait(wg):
- }
-}
-
-// Test that the entries are correctly sorted.
-// Add a bunch of long-in-the-future entries, and an immediate entry, and ensure
-// that the immediate entry runs immediately.
-// Also: Test that multiple jobs run in the same instant.
-func TestMultipleEntries(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(2)
-
- cron := newWithSeconds()
- _, _ = cron.AddFunc("0 0 0 1 1 ?", func() {})
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
- id1, _ := cron.AddFunc("* * * * * ?", func() { t.Fatal() })
- id2, _ := cron.AddFunc("* * * * * ?", func() { t.Fatal() })
- _, _ = cron.AddFunc("0 0 0 31 12 ?", func() {})
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
-
- cron.Remove(id1)
- cron.Start()
- cron.Remove(id2)
- defer cron.Stop()
-
- select {
- case <-time.After(OneSecond):
- t.Error("expected job run in proper order")
- case <-wait(wg):
- }
-}
-
-// Test running the same job twice.
-func TestRunningJobTwice(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(2)
-
- cron := newWithSeconds()
- _, _ = cron.AddFunc("0 0 0 1 1 ?", func() {})
- _, _ = cron.AddFunc("0 0 0 31 12 ?", func() {})
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
-
- cron.Start()
- defer cron.Stop()
-
- select {
- case <-time.After(2 * OneSecond):
- t.Error("expected job fires 2 times")
- case <-wait(wg):
- }
-}
-
-func TestRunningMultipleSchedules(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(2)
-
- cron := newWithSeconds()
- _, _ = cron.AddFunc("0 0 0 1 1 ?", func() {})
- _, _ = cron.AddFunc("0 0 0 31 12 ?", func() {})
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
- cron.Schedule(Every(time.Minute), FuncJob(func() {}))
- cron.Schedule(Every(time.Second), FuncJob(func() { wg.Done() }))
- cron.Schedule(Every(time.Hour), FuncJob(func() {}))
-
- cron.Start()
- defer cron.Stop()
-
- select {
- case <-time.After(2 * OneSecond):
- t.Error("expected job fires 2 times")
- case <-wait(wg):
- }
-}
-
-// Test that the cron is run in the local time zone (as opposed to UTC).
-func TestLocalTimezone(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(2)
-
- now := time.Now()
- // FIX: Issue #205
- // This calculation doesn't work in seconds 58 or 59.
- // Take the easy way out and sleep.
- if now.Second() >= 58 {
- time.Sleep(2 * time.Second)
- now = time.Now()
- }
- spec := fmt.Sprintf("%d,%d %d %d %d %d ?",
- now.Second()+1, now.Second()+2, now.Minute(), now.Hour(), now.Day(), now.Month())
-
- cron := newWithSeconds()
- _, _ = cron.AddFunc(spec, func() { wg.Done() })
- cron.Start()
- defer cron.Stop()
-
- select {
- case <-time.After(OneSecond * 2):
- t.Error("expected job fires 2 times")
- case <-wait(wg):
- }
-}
-
-// Test that the cron is run in the given time zone (as opposed to local).
-func TestNonLocalTimezone(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(2)
-
- loc, err := time.LoadLocation("Atlantic/Cape_Verde")
- if err != nil {
- fmt.Printf("Failed to load time zone Atlantic/Cape_Verde: %+v", err)
- t.Fail()
- }
-
- now := time.Now().In(loc)
- // FIX: Issue #205
- // This calculation doesn't work in seconds 58 or 59.
- // Take the easy way out and sleep.
- if now.Second() >= 58 {
- time.Sleep(2 * time.Second)
- now = time.Now().In(loc)
- }
- spec := fmt.Sprintf("%d,%d %d %d %d %d ?",
- now.Second()+1, now.Second()+2, now.Minute(), now.Hour(), now.Day(), now.Month())
-
- cron := New(WithLocation(loc), WithParser(secondParser))
- _, _ = cron.AddFunc(spec, func() { wg.Done() })
- cron.Start()
- defer cron.Stop()
-
- select {
- case <-time.After(OneSecond * 2):
- t.Error("expected job fires 2 times")
- case <-wait(wg):
- }
-}
-
-// Test that calling stop before start silently returns without
-// blocking the stop channel.
-func TestStopWithoutStart(t *testing.T) {
- cron := New()
- cron.Stop()
-}
-
-type testJob struct {
- wg *sync.WaitGroup
- name string
-}
-
-func (t testJob) Run() {
- t.wg.Done()
-}
-
-// Test that adding an invalid job spec returns an error
-func TestInvalidJobSpec(t *testing.T) {
- cron := New()
- _, err := cron.AddJob("this will not parse", nil)
- if err == nil {
- t.Errorf("expected an error with invalid spec, got nil")
- }
-}
-
-// Test blocking run method behaves as Start()
-func TestBlockingRun(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := newWithSeconds()
- _, _ = cron.AddFunc("* * * * * ?", func() { wg.Done() })
-
- var unblockChan = make(chan struct{})
-
- go func() {
- cron.Run()
- close(unblockChan)
- }()
- defer cron.Stop()
-
- select {
- case <-time.After(OneSecond):
- t.Error("expected job fires")
- case <-unblockChan:
- t.Error("expected that Run() blocks")
- case <-wait(wg):
- }
-}
-
-// Test that double-running is a no-op
-func TestStartNoop(t *testing.T) {
- var tickChan = make(chan struct{}, 2)
-
- cron := newWithSeconds()
- _, _ = cron.AddFunc("* * * * * ?", func() {
- tickChan <- struct{}{}
- })
-
- cron.Start()
- defer cron.Stop()
-
- // Wait for the first firing to ensure the runner is going
- <-tickChan
-
- cron.Start()
-
- <-tickChan
-
- // Fail if this job fires again in a short period, indicating a double-run
- select {
- case <-time.After(time.Millisecond):
- case <-tickChan:
- t.Error("expected job fires exactly twice")
- }
-}
-
-// Simple test using Runnables.
-func TestJob(t *testing.T) {
- wg := &sync.WaitGroup{}
- wg.Add(1)
-
- cron := newWithSeconds()
- _, _ = cron.AddJob("0 0 0 30 Feb ?", testJob{wg, "job0"})
- _, _ = cron.AddJob("0 0 0 1 1 ?", testJob{wg, "job1"})
- job2, _ := cron.AddJob("* * * * * ?", testJob{wg, "job2"})
- _, _ = cron.AddJob("1 0 0 1 1 ?", testJob{wg, "job3"})
- cron.Schedule(Every(5*time.Second+5*time.Nanosecond), testJob{wg, "job4"})
- job5 := cron.Schedule(Every(5*time.Minute), testJob{wg, "job5"})
-
- // Test getting an Entry pre-Start.
- if actualName := cron.Entry(job2).Job.(testJob).name; actualName != "job2" {
- t.Error("wrong job retrieved:", actualName)
- }
- if actualName := cron.Entry(job5).Job.(testJob).name; actualName != "job5" {
- t.Error("wrong job retrieved:", actualName)
- }
-
- cron.Start()
- defer cron.Stop()
-
- select {
- case <-time.After(OneSecond):
- t.FailNow()
- case <-wait(wg):
- }
-
- // Ensure the entries are in the right order.
- expecteds := []string{"job2", "job4", "job5", "job1", "job3", "job0"}
-
- var actuals = make([]string, 0, len(cron.Entries()))
- for _, entry := range cron.Entries() {
- actuals = append(actuals, entry.Job.(testJob).name)
- }
-
- for i, expected := range expecteds {
- if actuals[i] != expected {
- t.Fatalf("Jobs not in the right order. (expected) %s != %s (actual)", expecteds, actuals)
- }
- }
-
- // Test getting Entries.
- if actualName := cron.Entry(job2).Job.(testJob).name; actualName != "job2" {
- t.Error("wrong job retrieved:", actualName)
- }
- if actualName := cron.Entry(job5).Job.(testJob).name; actualName != "job5" {
- t.Error("wrong job retrieved:", actualName)
- }
-}
-
-// Issue #206
-// Ensure that the next run of a job after removing an entry is accurate.
-func TestScheduleAfterRemoval(t *testing.T) {
- var wg1 sync.WaitGroup
- var wg2 sync.WaitGroup
- wg1.Add(1)
- wg2.Add(1)
-
- // The first time this job is run, set a timer and remove the other job
- // 750ms later. Correct behavior would be to still run the job again in
- // 250ms, but the bug would cause it to run instead 1s later.
-
- var calls int
- var mu sync.Mutex
-
- cron := newWithSeconds()
- hourJob := cron.Schedule(Every(time.Hour), FuncJob(func() {}))
- cron.Schedule(Every(time.Second), FuncJob(func() {
- mu.Lock()
- defer mu.Unlock()
- switch calls {
- case 0:
- wg1.Done()
- calls++
- case 1:
- time.Sleep(750 * time.Millisecond)
- cron.Remove(hourJob)
- calls++
- case 2:
- calls++
- wg2.Done()
- case 3:
- panic("unexpected 3rd call")
- }
- }))
-
- cron.Start()
- defer cron.Stop()
-
- // the first run might be any length of time 0 - 1s, since the schedule
- // rounds to the second. wait for the first run to true up.
- wg1.Wait()
-
- select {
- case <-time.After(2 * OneSecond):
- t.Error("expected job fires 2 times")
- case <-wait(&wg2):
- }
-}
-
-type ZeroSchedule struct{}
-
-func (*ZeroSchedule) Next(time.Time) time.Time {
- return time.Time{}
-}
-
-// Tests that job without time does not run
-func TestJobWithZeroTimeDoesNotRun(t *testing.T) {
- cron := newWithSeconds()
- var calls int64
- _, _ = cron.AddFunc("* * * * * *", func() { atomic.AddInt64(&calls, 1) })
- cron.Schedule(new(ZeroSchedule), FuncJob(func() { t.Error("expected zero task will not run") }))
- cron.Start()
- defer cron.Stop()
- <-time.After(OneSecond)
- if atomic.LoadInt64(&calls) != 1 {
- t.Errorf("called %d times, expected 1\n", calls)
- }
-}
-
-func TestStopAndWait(t *testing.T) {
- t.Run("nothing running, returns immediately", func(t *testing.T) {
- cron := newWithSeconds()
- cron.Start()
- ctx := cron.Stop()
- select {
- case <-ctx.Done():
- case <-time.After(time.Millisecond):
- t.Error("context was not done immediately")
- }
- })
-
- t.Run("repeated calls to Stop", func(t *testing.T) {
- cron := newWithSeconds()
- cron.Start()
- _ = cron.Stop()
- time.Sleep(time.Millisecond)
- ctx := cron.Stop()
- select {
- case <-ctx.Done():
- case <-time.After(time.Millisecond):
- t.Error("context was not done immediately")
- }
- })
-
- t.Run("a couple fast jobs added, still returns immediately", func(t *testing.T) {
- cron := newWithSeconds()
- _, _ = cron.AddFunc("* * * * * *", func() {})
- cron.Start()
- _, _ = cron.AddFunc("* * * * * *", func() {})
- _, _ = cron.AddFunc("* * * * * *", func() {})
- _, _ = cron.AddFunc("* * * * * *", func() {})
- time.Sleep(time.Second)
- ctx := cron.Stop()
- select {
- case <-ctx.Done():
- case <-time.After(time.Millisecond):
- t.Error("context was not done immediately")
- }
- })
-
- t.Run("a couple fast jobs and a slow job added, waits for slow job", func(t *testing.T) {
- cron := newWithSeconds()
- _, _ = cron.AddFunc("* * * * * *", func() {})
- cron.Start()
- _, _ = cron.AddFunc("* * * * * *", func() { time.Sleep(2 * time.Second) })
- _, _ = cron.AddFunc("* * * * * *", func() {})
- time.Sleep(time.Second)
-
- ctx := cron.Stop()
-
- // Verify that it is not done for at least 750ms
- select {
- case <-ctx.Done():
- t.Error("context was done too quickly immediately")
- case <-time.After(750 * time.Millisecond):
- // expected, because the job sleeping for 1 second is still running
- }
-
- // Verify that it IS done in the next 500ms (giving 250ms buffer)
- select {
- case <-ctx.Done():
- // expected
- case <-time.After(1500 * time.Millisecond):
- t.Error("context not done after job should have completed")
- }
- })
-
- t.Run("repeated calls to stop, waiting for completion and after", func(t *testing.T) {
- cron := newWithSeconds()
- _, _ = cron.AddFunc("* * * * * *", func() {})
- _, _ = cron.AddFunc("* * * * * *", func() { time.Sleep(2 * time.Second) })
- cron.Start()
- _, _ = cron.AddFunc("* * * * * *", func() {})
- time.Sleep(time.Second)
- ctx := cron.Stop()
- ctx2 := cron.Stop()
-
- // Verify that it is not done for at least 1500ms
- select {
- case <-ctx.Done():
- t.Error("context was done too quickly immediately")
- case <-ctx2.Done():
- t.Error("context2 was done too quickly immediately")
- case <-time.After(1500 * time.Millisecond):
- // expected, because the job sleeping for 2 seconds is still running
- }
-
- // Verify that it IS done in the next 1s (giving 500ms buffer)
- select {
- case <-ctx.Done():
- // expected
- case <-time.After(time.Second):
- t.Error("context not done after job should have completed")
- }
-
- // Verify that ctx2 is also done.
- select {
- case <-ctx2.Done():
- // expected
- case <-time.After(time.Millisecond):
- t.Error("context2 not done even though context1 is")
- }
-
- // Verify that a new context retrieved from stop is immediately done.
- ctx3 := cron.Stop()
- select {
- case <-ctx3.Done():
- // expected
- case <-time.After(time.Millisecond):
- t.Error("context not done even when cron Stop is completed")
- }
-
- })
-}
-
-func TestMultiThreadedStartAndStop(t *testing.T) {
- cron := New()
- go cron.Run()
- time.Sleep(2 * time.Millisecond)
- cron.Stop()
-}
-
-func wait(wg *sync.WaitGroup) chan bool {
- ch := make(chan bool)
- go func() {
- wg.Wait()
- ch <- true
- }()
- return ch
-}
-
-func stop(cron *Cron) chan bool {
- ch := make(chan bool)
- go func() {
- cron.Stop()
- ch <- true
- }()
- return ch
-}
-
-// newWithSeconds returns a Cron with the seconds field enabled.
-func newWithSeconds() *Cron {
- return New(WithParser(secondParser), WithChain())
-}
diff --git a/backend/lib/cron/doc.go b/backend/lib/cron/doc.go
deleted file mode 100644
index fbee72c1..00000000
--- a/backend/lib/cron/doc.go
+++ /dev/null
@@ -1,212 +0,0 @@
-/*
-Package cron implements a cron spec parser and job runner.
-
-Usage
-
-Callers may register Funcs to be invoked on a given schedule. Cron will run
-them in their own goroutines.
-
- c := cron.New()
- c.AddFunc("30 * * * *", func() { fmt.Println("Every hour on the half hour") })
- c.AddFunc("30 3-6,20-23 * * *", func() { fmt.Println(".. in the range 3-6am, 8-11pm") })
- c.AddFunc("CRON_TZ=Asia/Tokyo 30 04 * * * *", func() { fmt.Println("Runs at 04:30 Tokyo time every day") })
- c.AddFunc("@hourly", func() { fmt.Println("Every hour, starting an hour from now") })
- c.AddFunc("@every 1h30m", func() { fmt.Println("Every hour thirty, starting an hour thirty from now") })
- c.Start()
- ..
- // Funcs are invoked in their own goroutine, asynchronously.
- ...
- // Funcs may also be added to a running Cron
- c.AddFunc("@daily", func() { fmt.Println("Every day") })
- ..
- // Inspect the cron job entries' next and previous run times.
- inspect(c.Entries())
- ..
- c.Stop() // Stop the scheduler (does not stop any jobs already running).
-
-CRON Expression Format
-
-A cron expression represents a set of times, using 5 space-separated fields.
-
- Field name | Mandatory? | Allowed values | Allowed special characters
- ---------- | ---------- | -------------- | --------------------------
- Minutes | Yes | 0-59 | * / , -
- Hours | Yes | 0-23 | * / , -
- Day of month | Yes | 1-31 | * / , - ?
- Month | Yes | 1-12 or JAN-DEC | * / , -
- Day of week | Yes | 0-6 or SUN-SAT | * / , - ?
-
-Month and Day-of-week field values are case insensitive. "SUN", "Sun", and
-"sun" are equally accepted.
-
-The specific interpretation of the format is based on the Cron Wikipedia page:
-https://en.wikipedia.org/wiki/Cron
-
-Alternative Formats
-
-Alternative Cron expression formats support other fields like seconds. You can
-implement that by creating a custom Parser as follows.
-
- cron.New(
- cron.WithParser(
- cron.SecondOptional | cron.Minute | cron.Hour | cron.Dom | cron.Month | cron.Dow | cron.Descriptor))
-
-The most popular alternative Cron expression format is Quartz:
-http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/crontrigger.html
-
-Special Characters
-
-Asterisk ( * )
-
-The asterisk indicates that the cron expression will match for all values of the
-field; e.g., using an asterisk in the 5th field (month) would indicate every
-month.
-
-Slash ( / )
-
-Slashes are used to describe increments of ranges. For example 3-59/15 in the
-1st field (minutes) would indicate the 3rd minute of the hour and every 15
-minutes thereafter. The form "*\/..." is equivalent to the form "first-last/...",
-that is, an increment over the largest possible range of the field. The form
-"N/..." is accepted as meaning "N-MAX/...", that is, starting at N, use the
-increment until the end of that specific range. It does not wrap around.
-
-Comma ( , )
-
-Commas are used to separate items of a list. For example, using "MON,WED,FRI" in
-the 5th field (day of week) would mean Mondays, Wednesdays and Fridays.
-
-Hyphen ( - )
-
-Hyphens are used to define ranges. For example, 9-17 would indicate every
-hour between 9am and 5pm inclusive.
-
-Question mark ( ? )
-
-Question mark may be used instead of '*' for leaving either day-of-month or
-day-of-week blank.
-
-Predefined schedules
-
-You may use one of several pre-defined schedules in place of a cron expression.
-
- Entry | Description | Equivalent To
- ----- | ----------- | -------------
- @yearly (or @annually) | Run once a year, midnight, Jan. 1st | 0 0 1 1 *
- @monthly | Run once a month, midnight, first of month | 0 0 1 * *
- @weekly | Run once a week, midnight between Sat/Sun | 0 0 * * 0
- @daily (or @midnight) | Run once a day, midnight | 0 0 * * *
- @hourly | Run once an hour, beginning of hour | 0 * * * *
-
-Intervals
-
-You may also schedule a job to execute at fixed intervals, starting at the time it's added
-or cron is run. This is supported by formatting the cron spec like this:
-
- @every
-
-
-
-
-
-
-
-
-
-
-
-
- {{ $t('This language is not installed yet.') }}
- {{ $t('Task Info') }}
-
-
- {{ $t('Spider Info') }}
-
-
- {{ $t('Node Info') }}
-
-
-
-
-
-
-
- {{ $t('Daily Tasks') }}
-
- {{ $t('Daily Avg Duration (sec)') }}
-
- {{ title }}
- {{ title }}
- Files to Upload
+
+
+
+
+
+
+
+
+
diff --git a/frontend/src/components/form/Form.vue b/frontend/src/components/form/Form.vue
new file mode 100644
index 00000000..51ae8670
--- /dev/null
+++ b/frontend/src/components/form/Form.vue
@@ -0,0 +1,90 @@
+
+
+