From 220dfdb12ed4f57521ce02196c616bd30994d462 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Tue, 30 Jul 2019 23:07:30 +0800 Subject: [PATCH 01/16] updated README.md --- README.md | 55 ++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 441a3ead..7d29f3d6 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,62 @@ # Crawlab ![](http://114.67.75.98:8081/buildStatus/icon?job=crawlab%2Fdevelop) -![](https://img.shields.io/badge/version-v0.2.3-blue.svg) +![](https://img.shields.io/badge/version-v0.3.0-blue.svg) [中文](https://github.com/tikazyq/crawlab/blob/master/README-zh.md) | English -Celery-based web crawler admin platform for managing distributed web spiders regardless of languages and frameworks. +基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。 +Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. [Demo](http://114.67.75.98:8080) | [Documentation](https://tikazyq.github.io/crawlab-docs) -## Pre-requisite -- Go 1.12+ -- Node.js 8.12+ -- MongoDB 3.6+ -- Redis - ## Installation -Threee methods: +Two methods: 1. [Docker](https://tikazyq.github.io/crawlab/Installation/Docker.md) (Recommended) -2. [Direct Deploy](https://tikazyq.github.io/crawlab/Installation/Direct.md) -3. [Preview](https://tikazyq.github.io/crawlab/Installation/Direct.md) (Quick start) +2. [Direct Deploy](https://tikazyq.github.io/crawlab/Installation/Direct.md) (Check Internal Kernel) + +### Pre-requisite (Docker) +- Docker 18.03+ +- Redis +- MongoDB 3.6+ + +### Pre-requisite (Direct Deploy) +- Go 1.12+ +- Node 8.12+ +- Redis +- MongoDB 3.6+ + +## Run + +### Docker + +Run Master Node for example. `192.168.99.1` is the host machine IP address in Docker Machine network. `192.168.99.100` is the Master Node's IP address. + +```bash +docker run -d --rm --name crawlab \ + -e CRAWLAB_REDIS_ADDRESS=192.168.99.1:6379 \ + -e CRAWLAB_MONGO_HOST=192.168.99.1 \ + -e CRAWLAB_SERVER_MASTER=Y \ + -e CRAWLAB_API_ADDRESS=192.168.99.100:8000 \ + -e CRAWLAB_SPIDER_PATH=/app/spiders \ + -p 8080:8080 \ + -p 8000:8000 \ + -v /var/logs/crawlab:/var/logs/crawlab \ + tikazyq/crawlab:0.3.0 +``` + +Surely you can use `docker-compose` to one-click to start up. By doing so, you don't even have to configure MongoDB and Redis databases. + +```bash +docker-compose up +``` + +For Docker Deployment details, please refer to [relevant documentation](https://tikazyq.github.io/crawlab/Installation/Docker.md). + ## Screenshot From 9e6171143fe5b4d69a61e825b9069a285d1588f1 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 12:22:40 +0800 Subject: [PATCH 02/16] updated README --- README-zh.md | 55 ++++++++++++++++++++----- README.md | 112 ++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 127 insertions(+), 40 deletions(-) diff --git a/README-zh.md b/README-zh.md index 4dc87800..226271a6 100644 --- a/README-zh.md +++ b/README-zh.md @@ -8,6 +8,8 @@ 中文 | [English](https://github.com/tikazyq/crawlab) +[安装](#安装) | [运行](#运行) | [截图](#截图) | [架构](#架构) | [集成](#与其他框架的集成) | [比较](#与其他框架比较) | [相关文章](#相关文章) | [社区&赞助](#社区--赞助) + 基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。 [查看演示 Demo](http://114.67.75.98:8080) | [文档](https://tikazyq.github.io/crawlab-docs) @@ -48,7 +50,38 @@ docker run -d --rm --name crawlab \ tikazyq/crawlab:0.3.0 ``` -当然也可以用`docker-compose`来一键启动，甚至不用配置MongoDB和Redis数据库。 +当然也可以用`docker-compose`来一键启动，甚至不用配置MongoDB和Redis数据库。在当前目录中创建`docker-compose.yml`文件，输入以下内容。 + +```bash +version: '3.3' +services: + master: + image: tikazyq/crawlab:latest + container_name: crawlab-master + environment: + CRAWLAB_API_ADDRESS: "192.168.99.100:8000" + CRAWLAB_SERVER_MASTER: "Y" + CRAWLAB_MONGO_HOST: "mongo" + CRAWLAB_REDIS_ADDRESS: "redis:6379" + ports: + - "8080:8080" # frontend + - "8000:8000" # backend + depends_on: + - mongo + - redis + mongo: + image: mongo:latest + restart: always + ports: + - "27017:27017" + redis: + image: redis:latest + restart: always + ports: + - "6379:6379" +``` + +然后执行以下命令，Crawlab主节点＋MongoDB＋Redis就启动了。打开`http://localhost:8080`就能看到界面。 ```bash docker-compose up @@ -64,43 +97,43 @@ Docker部署的详情，请见[相关文档](https://tikazyq.github.io/crawlab/I #### 登录 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/login.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/login.png?v0.3.0) #### 首页 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/home.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/home.png?v0.3.0) #### 节点列表 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-list.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-list.png?v0.3.0) #### 节点拓扑图 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-network.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-network.png?v0.3.0) #### 爬虫列表 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-list.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-list.png?v0.3.0) #### 爬虫概览 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-overview.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-overview.png?v0.3.0) #### 爬虫分析 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-analytics.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-analytics.png?v0.3.0) #### 爬虫文件 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-file.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-file.png?v0.3.0) #### 任务详情 - 抓取结果 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0) #### 定时任务 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/schedule.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/schedule.png?v0.3.0) ## 架构 diff --git a/README.md b/README.md index 7d29f3d6..3032d0fe 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,39 @@ docker run -d --rm --name crawlab \ tikazyq/crawlab:0.3.0 ``` -Surely you can use `docker-compose` to one-click to start up. By doing so, you don't even have to configure MongoDB and Redis databases. +Surely you can use `docker-compose` to one-click to start up. By doing so, you don't even have to configure MongoDB and Redis databases. Create a file named `docker-compose.yml` and input the code below. + + +```bash +version: '3.3' +services: + master: + image: tikazyq/crawlab:latest + container_name: crawlab-master + environment: + CRAWLAB_API_ADDRESS: "192.168.99.100:8000" + CRAWLAB_SERVER_MASTER: "Y" + CRAWLAB_MONGO_HOST: "mongo" + CRAWLAB_REDIS_ADDRESS: "redis:6379" + ports: + - "8080:8080" # frontend + - "8000:8000" # backend + depends_on: + - mongo + - redis + mongo: + image: mongo:latest + restart: always + ports: + - "27017:27017" + redis: + image: redis:latest + restart: always + ports: + - "6379:6379" +``` + +Then execute the command below, and Crawlab Master Node + MongoDB + Redis will start up. Open the browser and enter `http://localhost:8080` to see the UI interface. ```bash docker-compose up @@ -60,65 +92,87 @@ For Docker Deployment details, please refer to [relevant documentation](https:// ## Screenshot +#### Login + +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/login.png?v0.3.0) + #### Home Page -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/gitbook/home.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/home.png?v0.3.0) + +#### Node List + +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-list.png?v0.3.0) + +#### Node Network + +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-network.png?v0.3.0) #### Spider List -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/gitbook/spider-list.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-list.png?v0.3.0) -#### Spider Detail - Overview +#### Spider Overview -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/gitbook/spider-detail-overview.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-overview.png?v0.3.0) -#### Spider Detail - Analytics +#### Spider Analytics -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/gitbook/spider-detail-analytics.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-analytics.png?v0.3.0) -#### Task Detail - Results +#### Spider Files -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/gitbook/task-detail-results.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-file.png?v0.3.0) -#### Cron Schedule +#### Task Results -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/gitbook/schedule-generate-cron.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0) + +#### Cron Job + +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/schedule.png?v0.3.0) ## Architecture -Crawlab's architecture is very similar to Celery's, but a few more modules including Frontend, Spiders and Flower are added to feature the crawling management functionality. +The architecture of Crawlab is consisted of the Master Node and multiple Worker Nodes, and Redis and MongoDB databases which are mainly for nodes communication and data storage. -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/gitbook/architecture.png) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/architecture.png) -### Nodes +The frontend app makes requests to the Master Node, which assigns tasks and deploys spiders through MongoDB and Redis. When a Worker Node receives a task, it begins to execute the crawling task, and stores the results to MongoDB. The architecture is much more concise compared with versions before `v0.3.0`. It has removed unnecessary Flower module which offers node monitoring services. They are now done by Redis. -Nodes are actually the workers defined in Celery. A node is running and connected to a task queue, redis for example, to receive and run tasks. As spiders need to be deployed to the nodes, users should specify their ip addresses and ports before the deployment. +### Master Node -### Spiders +The Master Node is the core of the Crawlab architecture. It is the center control system of Crawlab. -The spider source codes and configured crawling rules are stored on `App`, which need to be deployed to each `worker` node. +The Master Node offers below services: +1. Crawling Task Coordination; +2. Worker Node Management and Communication; +3. Spider Deployment; +4. Frontend and API Services; +5. Task Execution (one can regard the Master Node as a Worker Node) -### Tasks +The Master Node communicates with the frontend app, and send crawling tasks to Worker Nodes. In the mean time, the Master Node synchronizes (deploys) spiders to Worker Nodes, via Redis and MongoDB GridFS. -Tasks are triggered and run by the workers. Users can view the task status, logs and results in the task detail page. +### Worker Node -### App +The main functionality of the Worker Nodes is to execute crawling tasks and store results and logs, and communicate with the Master Node through Redis `PubSub`. By increasing the number of Worker Nodes, Crawlab can scale horizontally, and different crawling tasks can be assigned to different nodes to execute. -This is a Flask app that provides necessary API for common operations such as CRUD, spider deployment and task running. Each node has to run the flask app to get spiders deployed on this machine. Simply run `python manage.py app` or `python ./bin/run_app.py` to start the app. +### MongoDB -### Broker +MongoDB is the operational database of Crawlab. It stores data of nodes, spiders, tasks, schedules, etc. The MongoDB GridFS file system is the medium for the Master Node to store spider files and synchronize to the Worker Nodes. -Broker is the same as defined in Celery. It is the queue for running async tasks. +### Redis + +Redis is a very popular Key-Value database. It offers node communication services in Crawlab. For example, nodes will execute `HSET` to set their info into a hash list named `nodes` in Redis, and the Master Node will identify online nodes according to the hash list. ### Frontend -Frontend is basically a Vue SPA that inherits from [Vue-Element-Admin](https://github.com/PanJiaChen/vue-element-admin) of [PanJiaChen](https://github.com/PanJiaChen). Thanks for his awesome template. +Frontend is a SPA based on +[Vue-Element-Admin](https://github.com/PanJiaChen/vue-element-admin). It has re-used many Element-UI components to support correspoinding display. ## Integration with Other Frameworks -A task is triggered via `Popen` in python `subprocess` module. A Task ID is will be defined as a variable `CRAWLAB_TASK_ID` in the shell environment to link the data to the task. - -In your spider program, you should store the `CRAWLAB_TASK_ID` value in the database with key `task_id`. Then Crawlab would know how to link those results to a particular task. For now, Crawlab only supports MongoDB. +A crawling task is actually executed through a shell command. The Task ID will be passed to the crawling task process in the form of environment variable named `CRAWLAB_TASK_ID`. By doing so, the data can be related to a task. Also, another environment variable `CRAWLAB_COLLECTION` is passed by Crawlab as the name of the collection to store results data. ### Scrapy @@ -158,9 +212,9 @@ Crawlab is easy to use, general enough to adapt spiders in any language and any |Framework | Type | Distributed | Frontend | Scrapyd-Dependent | |:---:|:---:|:---:|:---:|:---:| | [Crawlab](https://github.com/tikazyq/crawlab) | Admin Platform | Y | Y | N -| [Gerapy](https://github.com/Gerapy/Gerapy) | Admin Platform | Y | Y | Y -| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | Admin Platform | Y | Y | Y | [ScrapydWeb](https://github.com/my8100/scrapydweb) | Admin Platform | Y | Y | Y +| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | Admin Platform | Y | Y | Y +| [Gerapy](https://github.com/Gerapy/Gerapy) | Admin Platform | Y | Y | Y | [Scrapyd](https://github.com/scrapy/scrapyd) | Web Service | Y | N | N/A ## Community & Sponsorship From c7c4e483eb44ddf12cfaf2a6cc185f80fa8a1e03 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 12:23:47 +0800 Subject: [PATCH 03/16] updated README --- README-zh.md | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README-zh.md b/README-zh.md index 226271a6..0b4d9849 100644 --- a/README-zh.md +++ b/README-zh.md @@ -129,7 +129,7 @@ Docker部署的详情，请见[相关文档](https://tikazyq.github.io/crawlab/I #### 任务详情 - 抓取结果 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0_1) #### 定时任务 diff --git a/README.md b/README.md index 3032d0fe..abf7d104 100644 --- a/README.md +++ b/README.md @@ -126,7 +126,7 @@ For Docker Deployment details, please refer to [relevant documentation](https:// #### Task Results -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0) +![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0_1) #### Cron Job From ab9f89d1fd21d5d486687313a5763273dc8ebc77 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 12:29:33 +0800 Subject: [PATCH 04/16] updated README --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index abf7d104..8da20e5a 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,8 @@ [中文](https://github.com/tikazyq/crawlab/blob/master/README-zh.md) | English -基于Golang的分布式爬虫管理平台，支持Python、NodeJS、Go、Java、PHP等多种编程语言以及多种爬虫框架。 +[Installation](#installation) | [Run](#run) | [Screenshot](#screenshot) | [Architecture](#architecture) | [Integration](#integration-with-other-frameworks) | [Compare](#comparison-with-other-frameworks) | [Community & Sponsorship](#community--sponsorship) + Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. [Demo](http://114.67.75.98:8080) | [Documentation](https://tikazyq.github.io/crawlab-docs) From b6f9ed1dca778bb6e5341aa4fb17361384bf581a Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 12:31:25 +0800 Subject: [PATCH 05/16] code cleanup --- manage.py | 82 ------------------------------------------------------- setup.cfg | 2 -- setup.py | 39 -------------------------- 3 files changed, 123 deletions(-) delete mode 100644 manage.py delete mode 100644 setup.cfg delete mode 100644 setup.py diff --git a/manage.py b/manage.py deleted file mode 100644 index e9806a79..00000000 --- a/manage.py +++ /dev/null @@ -1,82 +0,0 @@ -import argparse -import os -import subprocess -from multiprocessing import Process -import sys - -BASE_DIR = os.path.dirname(__file__) - -APP_DESC = """ -Crawlab CLI tool. - -usage: python manage.py [action] - -action: - serve: start all necessary services to run crawlab. This is for quick start, please checkout Deployment guide for production environment. - app: start app + flower services, normally run on master node. - worker: start app + worker services, normally run on worker nodes. - flower: start flower service only. - frontend: start frontend/client service only. -""" -ACTION_LIST = [ - 'serve', - 'app', - 'worker', - 'flower', - 'frontend', -] -if len(sys.argv) == 1: - print(APP_DESC) - sys.argv.append('--help') -parser = argparse.ArgumentParser() -parser.add_argument('action', type=str) -args = parser.parse_args() - - -def run_app(): - p = subprocess.Popen([sys.executable, os.path.join(BASE_DIR, 'crawlab', 'app.py')]) - p.communicate() - - -def run_flower(): - p = subprocess.Popen([sys.executable, os.path.join(BASE_DIR, 'crawlab', 'flower.py')]) - p.communicate() - - -def run_worker(): - p = subprocess.Popen([sys.executable, os.path.join(BASE_DIR, 'crawlab', 'worker.py')]) - p.communicate() - - -def run_frontend(): - p = subprocess.Popen(['npm', 'run', 'serve'], - cwd=os.path.abspath(os.path.join(BASE_DIR, 'frontend'))) - p.communicate() - - -def main(): - p_app = Process(target=run_app) - p_flower = Process(target=run_flower) - p_worker = Process(target=run_worker) - p_frontend = Process(target=run_frontend) - if args.action == 'serve': - p_app.start() - p_flower.start() - p_worker.start() - p_frontend.start() - elif args.action == 'app': - p_app.start() - p_flower.start() - elif args.action == 'worker': - p_app.start() - p_worker.start() - elif args.action == 'flower': - p_flower.start() - elif args.action == 'frontend': - p_frontend.start() - else: - print(f'Invalid action: {args.action}') - - -if __name__ == '__main__': - main() diff --git a/setup.cfg b/setup.cfg deleted file mode 100644 index 224a7795..00000000 --- a/setup.cfg +++ /dev/null @@ -1,2 +0,0 @@ -[metadata] -description-file = README.md \ No newline at end of file diff --git a/setup.py b/setup.py deleted file mode 100644 index ce239009..00000000 --- a/setup.py +++ /dev/null @@ -1,39 +0,0 @@ -#-*- encoding: UTF-8 -*- -from setuptools import setup, find_packages - -VERSION = '0.2.3' - -with open('README.md') as fp: - readme = fp.read() - -setup(name='crawlab-server', - version=VERSION, - description="Celery-based web crawler admin platform for managing distributed web spiders regardless of languages and frameworks.", - long_description=readme, - classifiers=['Python', 'Javascript', 'Scrapy'], # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers - keywords='python crawlab celery crawler spider platform scrapy', - author='tikazyq', - author_email='tikazyq@163.com', - url='https://github.com/tikazyq/crawlab', - license='BSD', - packages=find_packages(), - include_package_data=True, - zip_safe=True, - install_requires=[ - 'celery', - 'flower', - 'requests', - 'pymongo', - 'flask', - 'flask_cors', - 'flask_restful', - 'lxml', - 'gevent', - 'scrapy', - ], - entry_points={ - 'console_scripts':[ - 'crawlab = crawlab.manage:main' - ] - }, -) \ No newline at end of file From 7497d41b7ac5d880b03bf787e4b3f357093a58fb Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 13:10:58 +0800 Subject: [PATCH 06/16] updated favicon.ico --- frontend/favicon.ico | Bin 67646 -> 1150 bytes frontend/index.html | 1 + frontend/public/favicon.ico | Bin 0 -> 16958 bytes frontend/public/index.html | 16 ++++++++++++++++ frontend/src/assets/favicon.ico | Bin 0 -> 1150 bytes frontend/static/favicon.ico | Bin 0 -> 1150 bytes 6 files changed, 17 insertions(+) create mode 100644 frontend/public/favicon.ico create mode 100644 frontend/public/index.html create mode 100644 frontend/src/assets/favicon.ico create mode 100644 frontend/static/favicon.ico diff --git a/frontend/favicon.ico b/frontend/favicon.ico index 34b63ac63a87ee3ea8e7a0f3f5b5406c437e2112..12b5c475b32588363c7b7d59e4a569bb5f881303 100644 GIT binary patch literal 1150 zcmbW0J4;;R%BVgnPvIWPrJiCaFj9;J6Hp(d<1gmh+5$WIQ|KIX-ei1WBkt-9OE@_PC8b| zlUV@{=77uJZu@ftXkESrF=G4g-hr>JK(Rm-fw|5w4j5}-jx*1(+!NW0WU1YVUmNw9 zlhr6D@~vioi&J3hThgaJx{hKFmz-CgWG(NXi;ekTi;v;82LImrxbv!YXbZ>5*TDA< zV5%kQ6C2lW`kmKKHO#ifal>_hYOnUAH?}*Q-oL%9$K`Xt{tU1&0sK97oAz;}KIFsw z`{(?;0@gnR`73ACYd_C+xb`<83+`KResl&oFQ?Mycf1jp{Q%5&@f`EN2AMakg~tZx zSG*ee;QnwdB;t~czA0ayVx3C&^ab~q{sGQa+gu>O%d+Wir=Y7Uv%{xm3s20tOI)dd L#7pQ`tQfJkBIvd6 literal 67646 zcmeHQ33wdEm2O$a2gEi66S8a^!){2(X2Y@}*zze`)?rJU(MUFiMYeol`2frK#KsuO zmSjtF$~K2NVhlNfB)}R7SkfFaCWOr{`-RN#4YIW(H- zo|aUj@2RJ{r>m>p|LWDNSFfrSr2u~|Eeif$u3R;tK>3)WC|7}SiYLW+e&x3_lNCW) zTK<C+ zR1SCpF}#VnD^NAY(^oSQ_)Y=-Q*rK69RC8x%Ygl*pi6-NM{#|U$L|<_q}x%@6|58T zLYZ(QKeyk8bhWAVkPjO?2iki|(6)h5$e}pLQj0&7qVaE>Z7c&zo=1K1Eq%6 zPiCH&1BdlEaj^n`p2H2XH-4&`R zaKVOzju8668Ca6LijVS~I{X!|<*Jo?14|3sA^XR{_hyjUxF?dKKcgH!1N|%L9(TY| z2K#cdCuAS%3oX;eA1Jr{WXAz_z@{ANU04A7>+jL-p9Zi007`^MKF@uCGW`m47X8B( z*pZ(BUEm4TVO(ifyuDHmh9U*zoy%WY2;57-+wX(^1j;AQ>7GrUKw0=Ps1sD|_1iAy z_{598q3GG6aAEY`)74uI>}zU4Z-CPA-ip`%8|8iwRO1cVF6jz!jA|RbfM!s07yWm^ znsU&OKvu*ylQR7rb@)2yMo*~vLfA1%M@ZU;iNwK!@f7@aw4c5Jew`fw*z4tl`rs7s zU)>q3yb!iRLCeb(&=m7g`Yy$q5yZX>4Kn_O3$Axhb>N*GY|4&dlK8^J2zXK1R2Q`8xQ`s+) zM)Jqyw-=zk$3cGtjdXkmTfbj}7ta9uTF^vKpqe~V=o24FANcs-=d*tvb^G^WgMFYz zbtfUr4-XdK!Tyfp#z!&VGy{FZcwcC#LVY<>K41Nh-SqYB0?{5GEKlRk_1 zsQd0NA4tc_#kJ07Z7SMqF3U*U;2qF@#Jv1f+lysmFz1AJQ#b6z0ArphsMnLj0eg3^ zUEOh@L|s`qLr?>bdRn<;hPq~Hk&ppzUrzCl$P)X!^eHMa- zg^#}j?RxuN3)PM5=c=n~W~wVHAP2x)Pb(^}Q*Yb5Ky^N!a~%+o0Umz~dKj@u^ST0c zV-S-vEJeWCEByT1Q5VKu>FZ~?@9gh)9a^Mr**0Ho{aU4ZM^~A;sd28_V4K+w|CNhp zsH)B}$$R-xOBuE)Q@hj=^iXI!h&)?GHNko8#fO=uCw^Vie4acA5pxyKg@Go)ttH&OFu~s>PSmsmRw_08n z;OC&-|FY%rN88_a$AT!%TW*`Lc0OlQ+n%UWH{Lu~Wt-Rc0ZcdTC^ote(CZfZ5#NE1 z+5x&`f8Qd@+XX(v*CS?X3ChiJuD&kT`i{0wtT%0#8}08`*=MSat7og5HqKMm-7qUE z7kU|3v#dz;6DHS<@llT>Hu~D5y>*4qTb9)UzFylHjAz%I%%6;9i+23MuPjm5ub8dj zkG8#{{5o|7j+BGdj-rJ4H&o3~ceO84Ic6}qj?@LOf$A}rK8bdL6+gb9?E;kXu*tm1 z91H0F-LU%_s$u5=bGGZPyNlKJD`yYX`#KwdW5lMN#j2~vbo?WAWPEBn@SldGa=*o5 z44|7XMY$fcGX8A)?Bh4AofFmlYwC)`7=X6mfPP<(KfbT~hgoVn?$2W#e~)KB0s6Zy zbuTM$`K=RY0=c{b<>LHzKKYaK*>1mGy}fhMK>ZFqpzrIGePwzM zakDRBook6Z=oo92IYspSSD{>%jql^w{m_%N@6q-LiFYD7*t~gOzVVO9${WzR%YpxR z#uw!K1MC^=!Sz*<@>%Lg-On*UeL0D69n?9<4ef+O-$ox`yf0`l{QD{J-w6CC4J5Z5c^O$`A1nIMi;l9e;B z=c)TS=1YY=$No3pG%KZya6W+Z1)Mjuhz=lM!!lrl9>~x5(9-y0%*Q$2RBfL=Uity= zyr*2)`N{MEf45=H95D_sVLl*}HuDQ$-vpZ2b*2h?UYNVTftB&6?%(4oO^g3ljPE<1 zsaCh&xiI-}(Lb!h9MSF$_<|r;7SnhCg|%t*DAU-^fVr^*R>q(H-p!ll4IKCB#`kR3 zP1}oA#(K1zT&%9bd~mY9g1@Ezn0BDWbO7b?Cx|5|+0(t=R9nEx_}_VNx#ri^;T?}H z`*n`(ou}>40XB8*^~uKx#J_RX9JTcum8wB;A=&DcvzUwd8#mUfn)R_V{`B`!XUAh6 zO+%+M&S(Gea?ze$-FEndF+Z3>UYOqxpm691pHtUL?i_ZvDI-@b3Wr8!=CmY#b=^!kplp_m%70oYlT_uE2va!4&RU zVdMwT`7Fpfai74j^E;Z|&-wj$?3v~`zUAS?1Lt&cFXKIL+XEXQncfK728+`xlDXom|(teZVcZ%+uzCtbzAHS%yylbJm~k>NRdGfO)8k!MhU?{^j)u?5O)0 zvzbiyNARRCKUw?dIFRcD=o^UqKGBhJV-_9@QXVibXWxwZp+cj20PJ@$5zpWry@}+{ z)N{1Q>BAGToGJQm>h8Us(xhc$K8y{gjhLzrApU#3mh}a&e(VFD1&ucfe@|~E*RyT} zCFQ3n*U+LYxf7J* zm*}I7GXNl1@HunOhc^>E+dRGi|2FvL*WHLX{uH{O@w|IHWr*WQEvxJU*tf5))#8N* z^#90*&*3;>|M3bd`-0wsjl12b_RoA_Uwj(a`}2uEqRc#hxbn@O;{v3Gp%d zJmb99Ek_(z{cLRxH$fl3HU5m>81z~W;J?RRD%Oank}LWNb~KkHw^`O0VvXWT~NPo|G2l?}kY zf%3JpwPf&pi+OzdE$s7+#^YbjKA;U`?w+r}hx>UX)cIkXjo5IV-+@Om)BO=yZhKtY zGa{W0z*rL6p%&8tKZE{XXf*!J-xGAu4`klnkLYXJ_C>r#0(&ol7w7xw%Ngq04c@n_ zEy4Oh-I#!Vz^z;7ix@(S;r}M~oV(N<{Oy+kf75J$n2#p8pPsm|?PnXW!?{ASUziSa zAz$Bi8;+%d8P&Kl~@O{l_pqA7=*shkMH~7hrz?&;29Fcsl^?|8{4o7PFnC z{YUIR#`k7ZCdLFL!hh}cv!H*pcp)*qL|-&iJ(ErOlHB_T=%O`7&j0D<>JZ=M8L}@$ zS>8=jo~$s?=Slw^+Zt_pw(*Frk3SNB&I#!6&G)%JK5{0i_bCuslr^+h*xuK?-* zGvq)0Gk$_SWUk;l_ssZoVC@Osq4#mr{f9=^-+RU({PV)@3*%3$N&Wo)xV8oTa#npZ z@TdNuFR)+U@v?xlur;ub$RawxiM<8SN1HSQex)O@lyd>&&<7j?{@m--uy#QFwe=m^ z{*4V)GsXH$uH|EFPXs5n+2${0AE!lIgTAA6HzwXMBqDn}Zdfzd3gf_Y;Q!3Gwtd7D z>?04sUPX)tCrsA9j9#A*P~SUUfAx~7c_03kGSvfnh)4zjL{mjzc-b<;&~!2 z2;+!|Y>B=A`i1krTd;r7kzQKKj8PBEzrd|+P{%Hfb%^4e2sV(bqxCkt~KR;qKpfr-`L3dr^)+O z*he$kU%nyIEKNOn{f=>nt!Y9T8Ha8NANJj&&CN3gM_?Z5i-8Bv(I3kg4X)M8RT+ps z_l)IyfUq;d&!pWi_K?GKx#ooWXY*$46@WFxxq5a6`TQR3Z>O)XX6%uk{Jew6eY%dm zz^g%TWKizG^9nvc&db%8VgJ6c%@;X}Hh-{0rG5}L7=D|pG7x{Uj~M2e6Foap2F?f3 z=fE*TR(5CRZHG43Q+N-dlT*R2cZip|tfoaBIl>+~GI80A-%z?7- z-Je*8l6xMoPiGtl{dxz#TB#AfA9~W@b(tIsLbveQZI9vkk!Qyp4;x*Mfjs?R@OkSI zf30D$-kA!KJ0#`=A2tGi=E*irovbHQ^evy8;Mt~^BPIj%w;sj2wx{?k^d5i5mvMd$ob4~jh8)I;cf^x|D9_bNr0eOoJ8{x9?3dVZ#F~3&;Jv(HKjK0wss5~Dy zJPtC{Cp^Nj*6SIz`!m3P0mx|hvd&*sRp_RgVw73h0onIeY(w7!?q6)~E6ExgGcxG- zZWVmOjtM9?{Q>6024v4?eS5RY_un9v2XlIkEA9?nubl9&Gy}e@pK%oL!1kRLK0&M# zvYQqk1X;5RZrPTDZRS!r@0;-D9DuI>!pU!~QGh>ac>!&ko_SdDgDfc+rH z#ct57&c4bCoI4|p(Wjk6y*~gM-L|ZjKi`>h4#Y9>wTPRXjLhndP(vJ_&jrM4y(`FfyV%+$ZNpAP?mKb7%(>Iz!O&z}`x8cgZlz<@wfNnS#CP zXbaFDJPR^{-&Vc$_HACj?J7^717qNk-M42d58OYZqvuAvN7%7o1j)cVsLTDpeL3PX zrn&mi&QVXVfqQXg%6BXM-sP`TI(wFn!8=SS1K$S?H+z8NcKU42BKCO~j@O}2zo;v; z3^oq>b_>MhSqbk%WpQKun6q~o-zR=8_}2@v`nZ5H%5^SpV;pk=WA`PPm%9vm%Zo5a@7~-Dj)hRUI#smxwuC7XRieQ6EJow z;CwvVzvAk({=9q=2=ac~9f1D>1Ya@Ud+*o;KL0a_`ZZVh4J9AiaV*C-Kuxe!{ubl4 z^MHQ={j>DV3%?EGo#ZDs9N=8R&euxOe~R%T{UkpI4F$WQ-b4TNX^;>4Xcfk4SG)XG zSl_CR*}DAJi?_=S=dF{A@D+pKt{$voM!XZ^AH3lITcDw0n(DsaAYb|+I6q0fQ;IhI z*PX%2v9#ehR(HcEFF)pU0P8)3j>TAUGWuBVHMa%Cwn@EAUn*lH*heH9H}E|D?EDRX z%RQ>-kD)#}1#)8C!MKDX$k?ZV^M%J=s;A!3bb1hTufufC-F&2sd!iQ@h%q!{NkTQ} zp%0$w4m#jFcdP?JFW8>|etp306%c)GJiY{ZLHxP>5qx)IOtux@?VziX{!1J$Mm`rD zdSUq(AKEzkaoNs|!!Q>)yZw02F>wM_oVXiyVr?Pt=e`pYaDF1rGu}8#z#p-P_VK`e z9DHwMy@Bd6zB3$`GY_#&4Y5hJ%HiPBTog~{VIl`y7%z&L5)oGtZogsK(C`)e5kzsO z@6|(nj(%MBFqYOb(5k~swGzeBT8Uz5tx8&TMIl6ny7Y8DK$jL7hO5sTL;~ zlvtnU1N7C!QI(h$xxtqM^J|3iKI-A_RE*W=R^fEL{kfJe%7F|G%7zi&qR zT#7U<>j8-u*!VQ%P2T`mJbijP(>jJ0$EBN@)-ixd9Rsum8LObkURU8_@$*;(o3(VL zg5lev6(TeFiAY>HyJ&&6tI-0B%JE#dfRUd^3RoQZeYk+pq*kEl?O~2I$F7qjy2~QN X;@D+E7Ad=S_FbjlW&KY_+xPzm3MBT? diff --git a/frontend/index.html b/frontend/index.html index 701c2e13..2c943e7e 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -5,6 +5,7 @@ + Crawlab diff --git a/frontend/public/favicon.ico b/frontend/public/favicon.ico new file mode 100644 index 0000000000000000000000000000000000000000..3fb253bdbf7a46c65a6880838d3b0fd1ecb20701 GIT binary patch literal 16958 zcmd^GU1%Le6yCI8d?;=4L42yA_|RvOy!N6X2!beBL;N9%D2k#eNUTs3G$<57DJp_m zS`5{OriRAGQcJEsMJrZ~SmO^h38JyB(MY8=8a;mB?wxk`-rb$qv%8mEGvQ?Jot<;$ zd}q$goSC`nd2RTsR6P9e@HSlOc^7-0w*iDAuMfm>KUIRPng8Y0$~sUtXaKYqbPO~L z`W^Iljrf^me8xJgOP+Pj)U}AVptA?6fL;Mjfg~r*)MQa+5h|7<1G=pO^@5Ir@?uWn zr);*dsuU0kq0nV5XbWf#l%#jju2!gA2n*V#pWgyH0V*1wt30+%9SXtZDg^Cy zfW|?tdX`;}x~zBAKTkch!+DwWb=jD*%BN1s-PmFXx?YeKXIy>C0gbf~c4GtToXZB_r_b+dLF~o`)LlImz`Lz<&YovYtgvF z^3KeO?C#RqfY{b1*#OU;P-CZW7Ul0@ym6oBXG8|C)7qdv4nOSz`b~}h=*w}MmRI)E zXIlKk*x&;w+pNaE?_T%$ec^Pl29EcO4EJDezdoQnw_<&KN@V(dcbmOCptZr~dLIDx zNfrJbSBspUb<_E)VYgUkei-4EF*UZF$4F`al}2MJLYiRLGT?_@p4{EOWfn@bZRCau}`V=w9S1*js0?M zNbI~z%5HO8eD=*~c9|MI^6~Kg=&TCAE$-pEolm!B@DuxjO3(ife%+qew&&qIz4ZH^ zV}DAer)_W2I`f}y{j=@UTVwq@g1%=Aklk0C8UJYXbK@a)#(>%J^Zdq^4m^@wu0|I< z53%d^KfIH$rK0A*!_UOWy(Si}KB=`cvFrE09OG$;`$PRX?RiYLSS!c=w{`|Fp&A1p zzmSGLc&6W*2Ke_sXm5jcvX24({G*0ncmBn`wVeTQA2^1+a(4Yhc*gLH#mCOhYwyLv z=U<&F{fLQizAaJ5H}A}51K0Z${XXMz_Pt5KuRH&y5AaOimUxc&!B#gL@XU<&8!l=1 zb~l+(|BIY|(+0q=zyJ8~Sr<)%I-E0EKki(RKEksM+nKLvO@Fo;cmI*^{>yeAVssAY zoIALWAh}{T{2WudwI%lctL6i`vq1V5&$@U&-HhbVsSj!Vo; zJdTAOEnow_cc=c%x51l~L-uW;{A*+JzE7P>d;cYD546MhU>uY*_L^NzT{^-#mi-y+ zac*3e@4tm@3mcgCA9eR=Iv?V_72khM-OGgaEc-KbSq+K5s92XT3*NPcX z-nn3^CW|ubn_JS~p)Y-cXRBN@20(j3uJ6B-=L%4%@AzWtRbA=tzZN}jBR|Z0-Wujr z&+{M9#Q2KGeuY1oUy9!;GjI2+`4#*4y-kD}82%0N$4&0DLx9f?0VYRlOlzk7%xkaD ztDc`$n!GRg-j#ttSob~1J|xv{JCQ6TJGWsVM- dW{!UF1)yI{Klu#kw{!hy`V|#6vOMVb{{SrgFHryh literal 0 HcmV?d00001 diff --git a/frontend/public/index.html b/frontend/public/index.html new file mode 100644 index 00000000..c703d044 --- /dev/null +++ b/frontend/public/index.html @@ -0,0 +1,16 @@ + + + + + + + + + Crawlab + + + +
+ + + diff --git a/frontend/src/assets/favicon.ico b/frontend/src/assets/favicon.ico new file mode 100644 index 0000000000000000000000000000000000000000..12b5c475b32588363c7b7d59e4a569bb5f881303 GIT binary patch literal 1150 zcmbW0J4;;R%BVgnPvIWPrJiCaFj9;J6Hp(d<1gmh+5$WIQ|KIX-ei1WBkt-9OE@_PC8b| zlUV@{=77uJZu@ftXkESrF=G4g-hr>JK(Rm-fw|5w4j5}-jx*1(+!NW0WU1YVUmNw9 zlhr6D@~vioi&J3hThgaJx{hKFmz-CgWG(NXi;ekTi;v;82LImrxbv!YXbZ>5*TDA< zV5%kQ6C2lW`kmKKHO#ifal>_hYOnUAH?}*Q-oL%9$K`Xt{tU1&0sK97oAz;}KIFsw z`{(?;0@gnR`73ACYd_C+xb`<83+`KResl&oFQ?Mycf1jp{Q%5&@f`EN2AMakg~tZx zSG*ee;QnwdB;t~czA0ayVx3C&^ab~q{sGQa+gu>O%d+Wir=Y7Uv%{xm3s20tOI)dd L#7pQ`tQfJkBIvd6 literal 0 HcmV?d00001 diff --git a/frontend/static/favicon.ico b/frontend/static/favicon.ico new file mode 100644 index 0000000000000000000000000000000000000000..12b5c475b32588363c7b7d59e4a569bb5f881303 GIT binary patch literal 1150 zcmbW0J4;;R%BVgnPvIWPrJiCaFj9;J6Hp(d<1gmh+5$WIQ|KIX-ei1WBkt-9OE@_PC8b| zlUV@{=77uJZu@ftXkESrF=G4g-hr>JK(Rm-fw|5w4j5}-jx*1(+!NW0WU1YVUmNw9 zlhr6D@~vioi&J3hThgaJx{hKFmz-CgWG(NXi;ekTi;v;82LImrxbv!YXbZ>5*TDA< zV5%kQ6C2lW`kmKKHO#ifal>_hYOnUAH?}*Q-oL%9$K`Xt{tU1&0sK97oAz;}KIFsw z`{(?;0@gnR`73ACYd_C+xb`<83+`KResl&oFQ?Mycf1jp{Q%5&@f`EN2AMakg~tZx zSG*ee;QnwdB;t~czA0ayVx3C&^ab~q{sGQa+gu>O%d+Wir=Y7Uv%{xm3s20tOI)dd L#7pQ`tQfJkBIvd6 literal 0 HcmV?d00001 From d701e235580ece3bc0f55fbc218f52bab6ca81ab Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 13:29:56 +0800 Subject: [PATCH 07/16] updated Jenkinsfile --- Dockerfile.local | 1 - Jenkinsfile | 10 ++-------- 2 files changed, 2 insertions(+), 9 deletions(-) diff --git a/Dockerfile.local b/Dockerfile.local index c08719aa..fc10b586 100644 --- a/Dockerfile.local +++ b/Dockerfile.local @@ -1,4 +1,3 @@ - FROM golang:1.12 AS backend-build WORKDIR /go/src/app diff --git a/Jenkinsfile b/Jenkinsfile index b22488cc..af9d6a13 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -24,7 +24,7 @@ pipeline { steps { echo "Building..." sh """ - docker build -t tikazyq/crawlab:latest . + docker build -t tikazyq/crawlab:latest -f Dockerfile.local . """ } } @@ -37,13 +37,7 @@ pipeline { steps { echo 'Deploying....' sh """ - docker rm -f crawlab | true - docker run -d --rm --name crawlab \ - -p 8080:8080 \ - -p 8000:8000 \ - -v /home/yeqing/.env.production:/opt/crawlab/frontend/.env.production \ - -v /home/yeqing/config.py:/opt/crawlab/crawlab/config/config.py \ - tikazyq/crawlab master + docker-compose up """ } } From f6028542b7f6931bfcba5252e402a03674523fdd Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 13:34:22 +0800 Subject: [PATCH 08/16] updated Jenkinsfile --- Dockerfile.local | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile.local b/Dockerfile.local index fc10b586..ed4e7e96 100644 --- a/Dockerfile.local +++ b/Dockerfile.local @@ -8,7 +8,7 @@ ENV GOPROXY https://mirrors.aliyun.com/goproxy/ RUN go install -v ./... -FROM node:8.16.0-alpine AS frontend-build +FROM node:8.16.0 AS frontend-build ADD ./frontend /app WORKDIR /app From d855fb5114149c69ae935ad49dcc4892d51b1710 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 17:01:17 +0800 Subject: [PATCH 09/16] updated Jenkinsfile --- Jenkinsfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Jenkinsfile b/Jenkinsfile index af9d6a13..af8f4768 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -37,6 +37,7 @@ pipeline { steps { echo 'Deploying....' sh """ + docker-compose stop | true docker-compose up """ } From 3087e925a7d7950569d35829bd7eaf015fa5c29a Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 17:13:48 +0800 Subject: [PATCH 10/16] updated Jenkinsfile --- Jenkinsfile | 2 +- docker-compose.yml | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/Jenkinsfile b/Jenkinsfile index af8f4768..4e3f86cb 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -38,7 +38,7 @@ pipeline { echo 'Deploying....' sh """ docker-compose stop | true - docker-compose up + docker-compose up -d """ } } diff --git a/docker-compose.yml b/docker-compose.yml index 8b7019f9..536b7d6b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -8,6 +8,8 @@ services: CRAWLAB_SERVER_MASTER: "Y" CRAWLAB_MONGO_HOST: "mongo" CRAWLAB_REDIS_ADDRESS: "redis:6379" + networks: + - host ports: - "8080:8080" # frontend - "8000:8000" # backend From 682a64d7517bfbbfbb47b75dc3b3b387937e50a4 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 19:38:03 +0800 Subject: [PATCH 11/16] updated Jenkinsfile --- Jenkinsfile | 3 ++- README-zh.md | 18 ++++++++++++++---- docker-compose.yml | 12 +++++------- 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/Jenkinsfile b/Jenkinsfile index 4e3f86cb..64d123f8 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -11,7 +11,7 @@ pipeline { echo "Running Setup..." script { if (env.GIT_BRANCH == 'develop') { - env.MODE = 'test' + env.MODE = 'develop' } else if (env.GIT_BRANCH == 'master') { env.MODE = 'production' } else { @@ -37,6 +37,7 @@ pipeline { steps { echo 'Deploying....' sh """ + cd ./jenkins docker-compose stop | true docker-compose up -d """ diff --git a/README-zh.md b/README-zh.md index 0b4d9849..49877554 100644 --- a/README-zh.md +++ b/README-zh.md @@ -50,25 +50,35 @@ docker run -d --rm --name crawlab \ tikazyq/crawlab:0.3.0 ``` -当然也可以用`docker-compose`来一键启动，甚至不用配置MongoDB和Redis数据库。在当前目录中创建`docker-compose.yml`文件，输入以下内容。 +当然也可以用`docker-compose`来一键启动，甚至不用配置MongoDB和Redis数据库，**当然我们推荐这样做**。在当前目录中创建`docker-compose.yml`文件，输入以下内容。 ```bash version: '3.3' services: master: image: tikazyq/crawlab:latest - container_name: crawlab-master + container_name: master environment: - CRAWLAB_API_ADDRESS: "192.168.99.100:8000" + CRAWLAB_API_ADDRESS: "localhost:8000" CRAWLAB_SERVER_MASTER: "Y" CRAWLAB_MONGO_HOST: "mongo" - CRAWLAB_REDIS_ADDRESS: "redis:6379" + CRAWLAB_REDIS_ADDRESS: "redis" ports: - "8080:8080" # frontend - "8000:8000" # backend depends_on: - mongo - redis + worker: + image: tikazyq/crawlab:latest + container_name: worker + environment: + CRAWLAB_SERVER_MASTER: "N" + CRAWLAB_MONGO_HOST: "mongo" + CRAWLAB_REDIS_ADDRESS: "redis" + depends_on: + - mongo + - redis mongo: image: mongo:latest restart: always diff --git a/docker-compose.yml b/docker-compose.yml index 536b7d6b..d44d5024 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,14 +2,12 @@ version: '3.3' services: master: image: tikazyq/crawlab:latest - container_name: crawlab-master + container_name: master environment: - CRAWLAB_API_ADDRESS: "192.168.99.100:8000" + CRAWLAB_API_ADDRESS: "localhost:8000" CRAWLAB_SERVER_MASTER: "Y" CRAWLAB_MONGO_HOST: "mongo" - CRAWLAB_REDIS_ADDRESS: "redis:6379" - networks: - - host + CRAWLAB_REDIS_ADDRESS: "redis" ports: - "8080:8080" # frontend - "8000:8000" # backend @@ -18,11 +16,11 @@ services: - redis worker: image: tikazyq/crawlab:latest - container_name: crawlab-worker + container_name: worker environment: CRAWLAB_SERVER_MASTER: "N" CRAWLAB_MONGO_HOST: "mongo" - CRAWLAB_REDIS_ADDRESS: "redis:6379" + CRAWLAB_REDIS_ADDRESS: "redis" depends_on: - mongo - redis From 218d8c5db3afb36e2fe3577bb08c10caf4a67149 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 19:45:43 +0800 Subject: [PATCH 12/16] updated config.yml --- README-zh.md | 10 ---------- README.md | 6 +++--- backend/conf/config.yml | 2 +- 3 files changed, 4 insertions(+), 14 deletions(-) diff --git a/README-zh.md b/README-zh.md index 49877554..bcb0c75e 100644 --- a/README-zh.md +++ b/README-zh.md @@ -69,16 +69,6 @@ services: depends_on: - mongo - redis - worker: - image: tikazyq/crawlab:latest - container_name: worker - environment: - CRAWLAB_SERVER_MASTER: "N" - CRAWLAB_MONGO_HOST: "mongo" - CRAWLAB_REDIS_ADDRESS: "redis" - depends_on: - - mongo - - redis mongo: image: mongo:latest restart: always diff --git a/README.md b/README.md index 8da20e5a..50c289b1 100644 --- a/README.md +++ b/README.md @@ -58,12 +58,12 @@ version: '3.3' services: master: image: tikazyq/crawlab:latest - container_name: crawlab-master + container_name: master environment: - CRAWLAB_API_ADDRESS: "192.168.99.100:8000" + CRAWLAB_API_ADDRESS: "localhost:8000" CRAWLAB_SERVER_MASTER: "Y" CRAWLAB_MONGO_HOST: "mongo" - CRAWLAB_REDIS_ADDRESS: "redis:6379" + CRAWLAB_REDIS_ADDRESS: "redis" ports: - "8080:8080" # frontend - "8000:8000" # backend diff --git a/backend/conf/config.yml b/backend/conf/config.yml index 820ff077..30b7e786 100644 --- a/backend/conf/config.yml +++ b/backend/conf/config.yml @@ -7,7 +7,7 @@ mongo: username: "" password: "" redis: - address: 127.0.0.1 + address: localhost password: "" database: 1 port: 6379 From b963512b2c212cfd54a3cf5aee3e76939688f2f7 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 19:46:55 +0800 Subject: [PATCH 13/16] added jenkins --- jenkins/docker-compose.yml | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 jenkins/docker-compose.yml diff --git a/jenkins/docker-compose.yml b/jenkins/docker-compose.yml new file mode 100644 index 00000000..57c85e1e --- /dev/null +++ b/jenkins/docker-compose.yml @@ -0,0 +1,36 @@ +version: '3.3' +services: + master: + image: tikazyq/crawlab:latest + container_name: master + environment: + CRAWLAB_API_ADDRESS: "114.67.75.98:8000" + CRAWLAB_SERVER_MASTER: "Y" + CRAWLAB_MONGO_HOST: "mongo" + CRAWLAB_REDIS_ADDRESS: "redis" + ports: + - "8080:8080" # frontend + - "8000:8000" # backend + depends_on: + - mongo + - redis + worker: + image: tikazyq/crawlab:latest + container_name: worker + environment: + CRAWLAB_SERVER_MASTER: "N" + CRAWLAB_MONGO_HOST: "mongo" + CRAWLAB_REDIS_ADDRESS: "redis" + depends_on: + - mongo + - redis + mongo: + image: mongo:latest + restart: always + ports: + - "27017:27017" + redis: + image: redis:latest + restart: always + ports: + - "6379:6379" \ No newline at end of file From 6e1a70124e5e2a8de5c176a2603c06d905e71a76 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 20:06:37 +0800 Subject: [PATCH 14/16] added jenkins badge --- README-zh.md | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README-zh.md b/README-zh.md index bcb0c75e..5203cd66 100644 --- a/README-zh.md +++ b/README-zh.md @@ -1,6 +1,6 @@ # Crawlab -![](http://114.67.75.98:8081/buildStatus/icon?job=crawlab%2Fdevelop) +![](http://114.67.75.98:8082/buildStatus/icon?job=crawlab%2Fdevelop) ![](https://img.shields.io/badge/版本-v0.3.0-blue.svg) diff --git a/README.md b/README.md index 50c289b1..fa3c5625 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Crawlab -![](http://114.67.75.98:8081/buildStatus/icon?job=crawlab%2Fdevelop) +![](http://114.67.75.98:8082/buildStatus/icon?job=crawlab%2Fdevelop) ![](https://img.shields.io/badge/version-v0.3.0-blue.svg) From b0850899037d3e4e08160c5b9722c912ea542863 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 20:14:03 +0800 Subject: [PATCH 15/16] updated README --- README-zh.md | 2 +- README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README-zh.md b/README-zh.md index 5203cd66..e7651d3b 100644 --- a/README-zh.md +++ b/README-zh.md @@ -39,7 +39,7 @@ ```bash docker run -d --rm --name crawlab \ - -e CRAWLAB_REDIS_ADDRESS=192.168.99.1:6379 \ + -e CRAWLAB_REDIS_ADDRESS=192.168.99.1 \ -e CRAWLAB_MONGO_HOST=192.168.99.1 \ -e CRAWLAB_SERVER_MASTER=Y \ -e CRAWLAB_API_ADDRESS=192.168.99.100:8000 \ diff --git a/README.md b/README.md index fa3c5625..31773883 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ Run Master Node for example. `192.168.99.1` is the host machine IP address in Do ```bash docker run -d --rm --name crawlab \ - -e CRAWLAB_REDIS_ADDRESS=192.168.99.1:6379 \ + -e CRAWLAB_REDIS_ADDRESS=192.168.99.1 \ -e CRAWLAB_MONGO_HOST=192.168.99.1 \ -e CRAWLAB_SERVER_MASTER=Y \ -e CRAWLAB_API_ADDRESS=192.168.99.100:8000 \ From b3e6dea24de4bde0297ee47453053dff114bfd18 Mon Sep 17 00:00:00 2001 From: Marvin Zhang Date: Wed, 31 Jul 2019 20:26:43 +0800 Subject: [PATCH 16/16] updated README --- README-zh.md | 20 ++++++++++---------- README.md | 22 +++++++++++----------- 2 files changed, 21 insertions(+), 21 deletions(-) diff --git a/README-zh.md b/README-zh.md index e7651d3b..8a1cfd26 100644 --- a/README-zh.md +++ b/README-zh.md @@ -97,43 +97,43 @@ Docker部署的详情，请见[相关文档](https://tikazyq.github.io/crawlab/I #### 登录 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/login.png?v0.3.0) + #### 首页 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/home.png?v0.3.0) + #### 节点列表 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-list.png?v0.3.0) + #### 节点拓扑图 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-network.png?v0.3.0) + #### 爬虫列表 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-list.png?v0.3.0) + #### 爬虫概览 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-overview.png?v0.3.0) + #### 爬虫分析 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-analytics.png?v0.3.0) + #### 爬虫文件 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-file.png?v0.3.0) + #### 任务详情 - 抓取结果 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0_1) + #### 定时任务 -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/schedule.png?v0.3.0) + ## 架构 diff --git a/README.md b/README.md index 31773883..cc474050 100644 --- a/README.md +++ b/README.md @@ -95,49 +95,49 @@ For Docker Deployment details, please refer to [relevant documentation](https:// #### Login -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/login.png?v0.3.0) + #### Home Page -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/home.png?v0.3.0) + #### Node List -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-list.png?v0.3.0) + #### Node Network -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/node-network.png?v0.3.0) + #### Spider List -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-list.png?v0.3.0) + #### Spider Overview -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-overview.png?v0.3.0) + #### Spider Analytics -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-analytics.png?v0.3.0) + #### Spider Files -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/spider-file.png?v0.3.0) + #### Task Results -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/task-results.png?v0.3.0_1) + #### Cron Job -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/schedule.png?v0.3.0) + ## Architecture The architecture of Crawlab is consisted of the Master Node and multiple Worker Nodes, and Redis and MongoDB databases which are mainly for nodes communication and data storage. -![](https://crawlab.oss-cn-hangzhou.aliyuncs.com/v0.3.0/architecture.png) + The frontend app makes requests to the Master Node, which assigns tasks and deploys spiders through MongoDB and Redis. When a Worker Node receives a task, it begins to execute the crawling task, and stores the results to MongoDB. The architecture is much more concise compared with versions before `v0.3.0`. It has removed unnecessary Flower module which offers node monitoring services. They are now done by Redis.