git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-23 17:31:11 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #626 from crawlab-team/release

Browse Source 
Release
This commit is contained in:
Marvin Zhang
2020-03-11 17:50:32 +08:00
committed by GitHub
parent 21df57c82e 05d2d57214
commit e58c6c44a3
57 changed files with 2378 additions and 1057 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.github/workflows/dockerpush.yml vendored
View File

@@ -64,3 +64,6 @@ jobs:
if [ "$VERSION" == "develop" ]; then
curl ${{ secrets.JENKINS_DEVELOP_URL }}
fi
if [ "$VERSION" == "master" ]; then
curl ${{ secrets.JENKINS_DEMO_URL }}
fi

18
CHANGELOG-zh.md
View File

@@ -1,3 +1,21 @@
# 0.4.8 (2020-03-11)
### 功能 / 优化
- **支持更多编程语言安装**. 现在用户可以安装或预装更多的编程语言，包括 Java、.Net Core、PHP.
- **安装 UI 优化**. 用户能够更好的查看和管理节点列表页的安装.
- **更多 Git 支持**. 允许用户查看 Git Commits 记录，并 Checkout 到相应 Commit.
- **支持用 Hostname 作为节点注册类型**. 用户可以将 hostname 作为节点的唯一识别号.
- **RPC 支持**. 加入 RPC 支持来更好的管理节点通信.
- **是否在主节点运行开关**. 用户可以决定是否在主节点运行，如果为否，则所有任务将在工作节点上运行.
- **默认禁用教程**.
- **加入相关文档侧边栏**.
- **加载页面优化**.
### Bug 修复
- **重复节点**. [#391](https://github.com/crawlab-team/crawlab/issues/391)
- **重复上传爬虫**. [#603](https://github.com/crawlab-team/crawlab/issues/603)
- **节点第三方模块安装失败导致 节点安装第三方部分无法使用**. [#609](https://github.com/crawlab-team/crawlab/issues/609)
- **离线节点也会创建任务**. [#622](https://github.com/crawlab-team/crawlab/issues/622)
# 0.4.7 (2020-02-24)
### 功能 / 优化
- **更好的支持 Scrapy**. 爬虫识别，`settings.py` 配置，日志级别选择，爬虫选择. [#435](https://github.com/crawlab-team/crawlab/issues/435)

18
CHANGELOG.md
View File

@@ -1,3 +1,21 @@
# 0.4.8 (2020-03-11)
### Features / Enhancement
- **Support Installations of More Programming Languages**. Now users can install or pre-install more programming languages including Java, .Net Core and PHP.
- **Installation UI Optimization**. Users can better view and manage installations on Node List page.
- **More Git Support**. Allow users to view Git Commits record, and allow checkout to corresponding commit.
- **Support Hostname Node Registration Type**. Users can set hostname as the node key as the unique identifier.
- **RPC Support**. Added RPC support to better manage node communication.
- **Run On Master Switch**. Users can determine whether to run tasks on master. If not, all tasks will be run only on worker nodes.
- **Disabled Tutorial by Default**.
- **Added Related Documentation Sidebar**.
- **Loading Page Optimization**.
### Bug Fixes
- **Duplicated Nodes**. [#391](https://github.com/crawlab-team/crawlab/issues/391)
- **Duplicated Spider Upload**. [#603](https://github.com/crawlab-team/crawlab/issues/603)
- **Failure in dependencies installation results in unusable dependency installation functionalities.**. [#609](https://github.com/crawlab-team/crawlab/issues/609)
- **Create Tasks for Offline Nodes**. [#622](https://github.com/crawlab-team/crawlab/issues/622)
# 0.4.7 (2020-02-24)
### Features / Enhancement
- **Better Support for Scrapy**. Spiders identification, `settings.py` configuration, log level selection, spider selection. [#435](https://github.com/crawlab-team/crawlab/issues/435)

3
Dockerfile
View File

@@ -29,7 +29,8 @@ ENV DEBIAN_FRONTEND noninteractive
ENV CRAWLAB_IS_DOCKER Y
# install packages
RUN apt-get update \
RUN chmod 777 /tmp \
&& apt-get update \
&& apt-get install -y curl git net-tools iputils-ping ntp ntpdate python3 python3-pip nginx wget \
&& ln -s /usr/bin/pip3 /usr/local/bin/pip \
&& ln -s /usr/bin/python3 /usr/local/bin/python

3
Dockerfile.local
View File

@@ -25,6 +25,9 @@ FROM ubuntu:latest
# set as non-interactive
ENV DEBIAN_FRONTEND noninteractive
# set CRAWLAB_IS_DOCKER
ENV CRAWLAB_IS_DOCKER Y
# install packages
RUN chmod 777 /tmp \
&& apt-get update \

4
README-zh.md
View File

@@ -154,6 +154,10 @@ Docker部署的详情，请见[相关文档](https://tikazyq.github.io/crawlab-d
![](http://static-docs.crawlab.cn/schedule-v0.4.4.png)
#### 语言安装
![](http://static-docs.crawlab.cn/node-install-langs.png)
#### 依赖安装
![](http://static-docs.crawlab.cn/node-install-dependencies.png)

4
README.md
View File

@@ -152,6 +152,10 @@ For Docker Deployment details, please refer to [relevant documentation](https://
![](http://static-docs.crawlab.cn/schedule-v0.4.4.png)
#### Language Installation
![](http://static-docs.crawlab.cn/node-install-langs.png)
#### Dependency Installation
![](http://static-docs.crawlab.cn/node-install-dependencies.png)

9
backend/conf/config.yml
View File

@@ -26,19 +26,24 @@ server:
# mac地址/ip地址/hostname, 如果是ip，则需要手动指定IP
type: "mac"
ip: ""
lang: # 安装语言环境, Y 为安装，N 为不安装，只对 Docker 有效
lang: # 安装语言环境, Y 为安装，N 为不安装
python: "Y"
node: "N"
java: "N"
dotnet: "N"
php: "N"
spider:
path: "/app/spiders"
task:
workers: 4
other:
tmppath: "/tmp"
version: 0.4.7
version: 0.4.8
setting:
allowRegister: "N"
enableTutorial: "N"
runOnMaster: "Y"
demoSpiders: "N"
notification:
mail:
server: ''

2
backend/constants/rpc.go
View File

@@ -4,6 +4,6 @@ const (
RpcInstallLang = "install_lang"
RpcInstallDep = "install_dep"
RpcUninstallDep = "uninstall_dep"
RpcGetDepList = "get_dep_list"
RpcGetInstalledDepList = "get_installed_dep_list"
RpcGetLang = "get_lang"
)

7
backend/constants/system.go
View File

@@ -11,3 +11,10 @@ const (
Nodejs = "node"
Java = "java"
)
const (
InstallStatusNotInstalled = "not-installed"
InstallStatusInstalling = "installing"
InstallStatusInstallingOther = "installing-other"
InstallStatusInstalled = "installed"
)

3
backend/entity/node.go
View File

@@ -20,6 +20,9 @@ type NodeMessage struct {
// 爬虫相关
SpiderId string `json:"spider_id"` //爬虫ID
// 语言相关
Lang Lang `json:"lang"`
// 错误相关
Error string `json:"error"`
}

11
backend/entity/rpc.go Normal file
View File

@@ -0,0 +1,11 @@
package entity
type RpcMessage struct {
Id string `json:"id"`
Method string `json:"method"`
NodeId string `json:"node_id"`
Params map[string]string `json:"params"`
Timeout int `json:"timeout"`
Result string `json:"result"`
Error string `json:"error"`
}

4
backend/entity/system.go
View File

@@ -19,7 +19,9 @@ type Lang struct {
ExecutableName string `json:"executable_name"`
ExecutablePaths []string `json:"executable_paths"`
DepExecutablePath string `json:"dep_executable_path"`
Installed bool `json:"installed"`
LockPath string `json:"lock_path"`
InstallScript string `json:"install_script"`
InstallStatus string `json:"install_status"`
}
type Dependency struct {

15
backend/main.go
View File

@@ -9,6 +9,7 @@ import (
"crawlab/model"
"crawlab/routes"
"crawlab/services"
"crawlab/services/rpc"
"github.com/apex/log"
"github.com/gin-gonic/gin"
"github.com/gin-gonic/gin/binding"
@@ -116,7 +117,7 @@ func main() {
log.Info("initialized spider service successfully")
// 初始化RPC服务
if err := services.InitRpcService(); err != nil {
if err := rpc.InitRpcService(); err != nil {
log.Error("init rpc service error:" + err.Error())
debug.PrintStack()
panic(err)
@@ -242,24 +243,26 @@ func main() {
{
authGroup.GET("/variables", routes.GetVariableList) // 列表
authGroup.PUT("/variable", routes.PutVariable) // 新增
authGroup.POST("/variable/:id", routes.PostVariable) //修改
authGroup.DELETE("/variable/:id", routes.DeleteVariable) //删除
authGroup.POST("/variable/:id", routes.PostVariable) // 修改
authGroup.DELETE("/variable/:id", routes.DeleteVariable) // 删除
}
// 项目
{
authGroup.GET("/projects", routes.GetProjectList) // 列表
authGroup.GET("/projects/tags", routes.GetProjectTags) // 项目标签
authGroup.PUT("/projects", routes.PutProject) //修改
authGroup.PUT("/projects", routes.PutProject) // 修改
authGroup.POST("/projects/:id", routes.PostProject) // 新增
authGroup.DELETE("/projects/:id", routes.DeleteProject) //删除
authGroup.DELETE("/projects/:id", routes.DeleteProject) // 删除
}
// 统计数据
authGroup.GET("/stats/home", routes.GetHomeStats) // 首页统计数据
// 文件
authGroup.GET("/file", routes.GetFile) // 获取文件
// Git
authGroup.GET("/git/branches", routes.GetGitBranches) // 获取 Git 分支
authGroup.GET("/git/branches", routes.GetGitRemoteBranches) // 获取 Git 分支
authGroup.GET("/git/public-key", routes.GetGitSshPublicKey) // 获取 SSH 公钥
authGroup.GET("/git/commits", routes.GetGitCommits) // 获取 Git Commits
authGroup.POST("/git/checkout", routes.PostGitCheckout) // 获取 Git Commits
}
}

57
backend/routes/git.go
View File

@@ -1,16 +1,19 @@
package routes
import (
"crawlab/model"
"crawlab/services"
"github.com/gin-gonic/gin"
"github.com/globalsign/mgo/bson"
"net/http"
)
func GetGitBranches(c *gin.Context) {
func GetGitRemoteBranches(c *gin.Context) {
url := c.Query("url")
branches, err := services.GetGitBranches(url)
branches, err := services.GetGitRemoteBranchesPlain(url)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
c.JSON(http.StatusOK, Response{
Status: "ok",
@@ -27,3 +30,53 @@ func GetGitSshPublicKey(c *gin.Context) {
Data: content,
})
}
func GetGitCommits(c *gin.Context) {
spiderId := c.Query("spider_id")
if spiderId == "" || !bson.IsObjectIdHex(spiderId) {
HandleErrorF(http.StatusInternalServerError, c, "invalid request")
return
}
spider, err := model.GetSpider(bson.ObjectIdHex(spiderId))
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
commits, err := services.GetGitCommits(spider)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
c.JSON(http.StatusOK, Response{
Status: "ok",
Message: "success",
Data: commits,
})
}
func PostGitCheckout(c *gin.Context) {
type ReqBody struct {
SpiderId string `json:"spider_id"`
Hash string `json:"hash"`
}
var reqBody ReqBody
if err := c.ShouldBindJSON(&reqBody); err != nil {
}
if reqBody.SpiderId == "" || !bson.IsObjectIdHex(reqBody.SpiderId) {
HandleErrorF(http.StatusInternalServerError, c, "invalid request")
return
}
spider, err := model.GetSpider(bson.ObjectIdHex(reqBody.SpiderId))
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
if err := services.GitCheckout(spider, reqBody.Hash); err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
c.JSON(http.StatusOK, Response{
Status: "ok",
Message: "success",
})
}

12
backend/routes/setting.go
View File

@@ -7,8 +7,10 @@ import (
)
type SettingBody struct {
AllowRegister string `json:"allow_register"`
EnableTutorial string `json:"enable_tutorial"`
AllowRegister string `json:"allow_register"`
EnableTutorial string `json:"enable_tutorial"`
RunOnMaster string `json:"run_on_master"`
EnableDemoSpiders string `json:"enable_demo_spiders"`
}
func GetVersion(c *gin.Context) {
@@ -23,8 +25,10 @@ func GetVersion(c *gin.Context) {
func GetSetting(c *gin.Context) {
body := SettingBody{
AllowRegister: viper.GetString("setting.allowRegister"),
EnableTutorial: viper.GetString("setting.enableTutorial"),
AllowRegister: viper.GetString("setting.allowRegister"),
EnableTutorial: viper.GetString("setting.enableTutorial"),
RunOnMaster: viper.GetString("setting.runOnMaster"),
EnableDemoSpiders: viper.GetString("setting.enableDemoSpiders"),
}
c.JSON(http.StatusOK, Response{

149
backend/routes/system.go
View File

@@ -4,6 +4,7 @@ import (
"crawlab/constants"
"crawlab/entity"
"crawlab/services"
"crawlab/services/rpc"
"fmt"
"github.com/gin-gonic/gin"
"net/http"
@@ -55,41 +56,20 @@ func GetInstalledDepList(c *gin.Context) {
nodeId := c.Param("id")
lang := c.Query("lang")
var depList []entity.Dependency
if lang == constants.Python {
if services.IsMasterNode(nodeId) {
list, err := services.GetPythonLocalInstalledDepList(nodeId)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
depList = list
} else {
list, err := services.GetPythonRemoteInstalledDepList(nodeId)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
depList = list
}
} else if lang == constants.Nodejs {
if services.IsMasterNode(nodeId) {
list, err := services.GetNodejsLocalInstalledDepList(nodeId)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
depList = list
} else {
list, err := services.GetNodejsRemoteInstalledDepList(nodeId)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
depList = list
if services.IsMasterNode(nodeId) {
list, err := rpc.GetInstalledDepsLocal(lang)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
depList = list
} else {
HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("%s is not implemented", lang))
return
list, err := rpc.GetInstalledDepsRemote(nodeId, lang)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
depList = list
}
c.JSON(http.StatusOK, Response{
@@ -155,41 +135,18 @@ func InstallDep(c *gin.Context) {
return
}
if reqBody.Lang == constants.Python {
if services.IsMasterNode(nodeId) {
_, err := services.InstallPythonLocalDep(reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
_, err := services.InstallPythonRemoteDep(nodeId, reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
}
} else if reqBody.Lang == constants.Nodejs {
if services.IsMasterNode(nodeId) {
_, err := services.InstallNodejsLocalDep(reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
_, err := services.InstallNodejsRemoteDep(nodeId, reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
if services.IsMasterNode(nodeId) {
if err := rpc.InstallDepLocal(reqBody.Lang, reqBody.DepName); err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("%s is not implemented", reqBody.Lang))
return
if err := rpc.InstallDepRemote(nodeId, reqBody.Lang, reqBody.DepName); err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
}
// TODO: check if install is successful
c.JSON(http.StatusOK, Response{
Status: "ok",
Message: "success",
@@ -209,41 +166,18 @@ func UninstallDep(c *gin.Context) {
HandleError(http.StatusBadRequest, c, err)
}
if reqBody.Lang == constants.Python {
if services.IsMasterNode(nodeId) {
_, err := services.UninstallPythonLocalDep(reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
_, err := services.UninstallPythonRemoteDep(nodeId, reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
}
} else if reqBody.Lang == constants.Nodejs {
if services.IsMasterNode(nodeId) {
_, err := services.UninstallNodejsLocalDep(reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
_, err := services.UninstallNodejsRemoteDep(nodeId, reqBody.DepName)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
if services.IsMasterNode(nodeId) {
if err := rpc.UninstallDepLocal(reqBody.Lang, reqBody.DepName); err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("%s is not implemented", reqBody.Lang))
return
if err := rpc.UninstallDepRemote(nodeId, reqBody.Lang, reqBody.DepName); err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
}
// TODO: check if uninstall is successful
c.JSON(http.StatusOK, Response{
Status: "ok",
Message: "success",
@@ -288,23 +222,18 @@ func InstallLang(c *gin.Context) {
return
}
if reqBody.Lang == constants.Nodejs {
if services.IsMasterNode(nodeId) {
_, err := services.InstallNodejsLocalLang()
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
_, err := services.InstallNodejsRemoteLang(nodeId)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
if services.IsMasterNode(nodeId) {
_, err := rpc.InstallLangLocal(reqBody.Lang)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
} else {
HandleErrorF(http.StatusBadRequest, c, fmt.Sprintf("%s is not implemented", reqBody.Lang))
return
_, err := rpc.InstallLangRemote(nodeId, reqBody.Lang)
if err != nil {
HandleError(http.StatusInternalServerError, c, err)
return
}
}
// TODO: check if install is successful

17
backend/scripts/install-dotnet.sh Executable file
View File

@@ -0,0 +1,17 @@
# lock global
touch /tmp/install.lock
# lock
touch /tmp/install-dotnet.lock
wget -q https://packages.microsoft.com/config/ubuntu/16.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
dpkg -i packages-microsoft-prod.deb
apt-get install -y apt-transport-https
apt-get update
apt-get install -y dotnet-sdk-2.1 dotnet-runtime-2.1 aspnetcore-runtime-2.1
# unlock global
rm /tmp/install.lock
# unlock
rm /tmp/install-dotnet.lock

12
backend/scripts/install-java.sh
View File

@@ -1,11 +1,19 @@
#!/bin/env bash
#!/bin/bash
# lock global
touch /tmp/install.lock
# lock
touch /tmp/install-java.lock
# install java
apt-get update && apt-get install -y default-jdk --fix-missing
apt-get clean && \
apt-get update --fix-missing && \
apt-get install -y --fix-missing default-jdk
ln -s /usr/bin/java /usr/local/bin/java
# unlock
rm /tmp/install-java.lock
# unlock global
rm /tmp/install.lock

8
backend/scripts/install-nodejs.sh
View File

@@ -1,4 +1,7 @@
#!/bin/env bash
#!/bin/bash
# lock global
touch /tmp/install.lock
# lock
touch /tmp/install-nodejs.lock
@@ -39,3 +42,6 @@ npm install puppeteer-chromium-resolver crawlab-sdk -g --unsafe-perm=true --regi
# unlock
rm /tmp/install-nodejs.lock
# unlock global
rm /tmp/install.lock

13
backend/scripts/install-php.sh Executable file
View File

@@ -0,0 +1,13 @@
# lock global
touch /tmp/install.lock
# lock
touch /tmp/install-php.lock
apt-get install -y php
# unlock global
rm /tmp/install.lock
# unlock
rm /tmp/install-php.lock

25
backend/scripts/install.sh Normal file
View File

@@ -0,0 +1,25 @@
#!/bin/bash
# install node.js
if [ "${CRAWLAB_SERVER_LANG_NODE}" = "Y" ];
then
echo "installing node.js"
/bin/sh /app/backend/scripts/install-nodejs.sh
echo "installed node.js"
fi
# install java
if [ "${CRAWLAB_SERVER_LANG_JAVA}" = "Y" ];
then
echo "installing java"
/bin/sh /app/backend/scripts/install-java.sh
echo "installed java"
fi
# install dotnet
if [ "${CRAWLAB_SERVER_LANG_DOTNET}" = "Y" ];
then
echo "installing dotnet"
/bin/sh /app/backend/scripts/install-dotnet.sh
echo "installed dotnet"
fi

378
backend/services/git.go
View File

@@ -12,6 +12,7 @@ import (
"gopkg.in/src-d/go-git.v4"
"gopkg.in/src-d/go-git.v4/config"
"gopkg.in/src-d/go-git.v4/plumbing"
"gopkg.in/src-d/go-git.v4/plumbing/object"
"gopkg.in/src-d/go-git.v4/plumbing/transport/ssh"
"io/ioutil"
"net/url"
@@ -21,6 +22,7 @@ import (
"regexp"
"runtime/debug"
"strings"
"time"
)
var GitCron *GitCronScheduler
@@ -29,6 +31,102 @@ type GitCronScheduler struct {
cron *cron.Cron
}
type GitBranch struct {
Hash string `json:"hash"`
Name string `json:"name"`
Label string `json:"label"`
}
type GitTag struct {
Hash string `json:"hash"`
Name string `json:"name"`
Label string `json:"label"`
}
type GitCommit struct {
Hash string `json:"hash"`
TreeHash string `json:"tree_hash"`
Author string `json:"author"`
Email string `json:"email"`
Message string `json:"message"`
IsHead bool `json:"is_head"`
Ts time.Time `json:"ts"`
Branches []GitBranch `json:"branches"`
RemoteBranches []GitBranch `json:"remote_branches"`
Tags []GitTag `json:"tags"`
}
func (g *GitCronScheduler) Start() error {
c := cron.New(cron.WithSeconds())
// 启动cron服务
g.cron.Start()
// 更新任务列表
if err := g.Update(); err != nil {
log.Errorf("update scheduler error: %s", err.Error())
debug.PrintStack()
return err
}
// 每30秒更新一次任务列表
spec := "*/30 * * * * *"
if _, err := c.AddFunc(spec, UpdateGitCron); err != nil {
log.Errorf("add func update schedulers error: %s", err.Error())
debug.PrintStack()
return err
}
return nil
}
func (g *GitCronScheduler) RemoveAll() {
entries := g.cron.Entries()
for i := 0; i < len(entries); i++ {
g.cron.Remove(entries[i].ID)
}
}
func (g *GitCronScheduler) Update() error {
// 删除所有定时任务
g.RemoveAll()
// 获取开启 Git 自动同步的爬虫
spiders, err := model.GetSpiderAllList(bson.M{"git_auto_sync": true})
if err != nil {
log.Errorf("get spider list error: %s", err.Error())
debug.PrintStack()
return err
}
// 遍历任务列表
for _, s := range spiders {
// 添加到定时任务
if err := g.AddJob(s); err != nil {
log.Errorf("add job error: %s, job: %s, cron: %s", err.Error(), s.Name, s.GitSyncFrequency)
debug.PrintStack()
return err
}
}
return nil
}
func (g *GitCronScheduler) AddJob(s model.Spider) error {
spec := s.GitSyncFrequency
// 添加定时任务
_, err := g.cron.AddFunc(spec, AddGitCronJob(s))
if err != nil {
log.Errorf("add func task error: %s", err.Error())
debug.PrintStack()
return err
}
return nil
}
// 保存爬虫Git同步错误
func SaveSpiderGitSyncError(s model.Spider, errMsg string) {
s, _ = model.GetSpider(s.Id)
s.GitSyncError = errMsg
@@ -39,7 +137,8 @@ func SaveSpiderGitSyncError(s model.Spider, errMsg string) {
}
}
func GetGitBranches(url string) (branches []string, err error) {
// 获得Git分支
func GetGitRemoteBranchesPlain(url string) (branches []string, err error) {
var stdout bytes.Buffer
var stderr bytes.Buffer
@@ -63,6 +162,7 @@ func GetGitBranches(url string) (branches []string, err error) {
return branches, nil
}
// 重置爬虫Git
func ResetSpiderGit(s model.Spider) (err error) {
// 删除文件夹
if err := os.RemoveAll(s.Src); err != nil {
@@ -86,6 +186,7 @@ func ResetSpiderGit(s model.Spider) (err error) {
return nil
}
// 同步爬虫Git
func SyncSpiderGit(s model.Spider) (err error) {
// 如果 .git 不存在，初始化一个仓库
if !utils.Exists(path.Join(s.Src, ".git")) {
@@ -165,6 +266,7 @@ func SyncSpiderGit(s model.Spider) (err error) {
RemoteName: "origin",
Force: true,
Auth: auth,
Tags: git.AllTags,
})
// 获得 WorkTree
@@ -178,8 +280,10 @@ func SyncSpiderGit(s model.Spider) (err error) {
// 拉取 repo
if err := wt.Pull(&git.PullOptions{
RemoteName: "origin",
Auth: auth,
RemoteName: "origin",
Auth: auth,
ReferenceName: plumbing.HEAD,
SingleBranch: false,
}); err != nil {
if err.Error() == "already up-to-date" {
// 检查是否为 Scrapy
@@ -221,76 +325,7 @@ func SyncSpiderGit(s model.Spider) (err error) {
return nil
}
func (g *GitCronScheduler) Start() error {
c := cron.New(cron.WithSeconds())
// 启动cron服务
g.cron.Start()
// 更新任务列表
if err := g.Update(); err != nil {
log.Errorf("update scheduler error: %s", err.Error())
debug.PrintStack()
return err
}
// 每30秒更新一次任务列表
spec := "*/30 * * * * *"
if _, err := c.AddFunc(spec, UpdateGitCron); err != nil {
log.Errorf("add func update schedulers error: %s", err.Error())
debug.PrintStack()
return err
}
return nil
}
func (g *GitCronScheduler) RemoveAll() {
entries := g.cron.Entries()
for i := 0; i < len(entries); i++ {
g.cron.Remove(entries[i].ID)
}
}
func (g *GitCronScheduler) Update() error {
// 删除所有定时任务
g.RemoveAll()
// 获取开启 Git 自动同步的爬虫
spiders, err := model.GetSpiderAllList(bson.M{"git_auto_sync": true})
if err != nil {
log.Errorf("get spider list error: %s", err.Error())
debug.PrintStack()
return err
}
// 遍历任务列表
for _, s := range spiders {
// 添加到定时任务
if err := g.AddJob(s); err != nil {
log.Errorf("add job error: %s, job: %s, cron: %s", err.Error(), s.Name, s.GitSyncFrequency)
debug.PrintStack()
return err
}
}
return nil
}
func (g *GitCronScheduler) AddJob(s model.Spider) error {
spec := s.GitSyncFrequency
// 添加定时任务
_, err := g.cron.AddFunc(spec, AddGitCronJob(s))
if err != nil {
log.Errorf("add func task error: %s", err.Error())
debug.PrintStack()
return err
}
return nil
}
// 添加Git定时任务
func AddGitCronJob(s model.Spider) func() {
return func() {
if err := SyncSpiderGit(s); err != nil {
@@ -301,6 +336,7 @@ func AddGitCronJob(s model.Spider) func() {
}
}
// 更新Git定时任务
func UpdateGitCron() {
if err := GitCron.Update(); err != nil {
log.Errorf(err.Error())
@@ -308,6 +344,7 @@ func UpdateGitCron() {
}
}
// 获取SSH公钥
func GetGitSshPublicKey() string {
if !utils.Exists(path.Join(os.Getenv("HOME"), ".ssh")) ||
!utils.Exists(path.Join(os.Getenv("HOME"), ".ssh", "id_rsa")) ||
@@ -322,3 +359,198 @@ func GetGitSshPublicKey() string {
}
return string(content)
}
// 获取Git分支
func GetGitBranches(s model.Spider) (branches []GitBranch, err error) {
// 打开 repo
repo, err := git.PlainOpen(s.Src)
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return branches, err
}
iter, err := repo.Branches()
if iter == nil {
return branches, nil
}
if err := iter.ForEach(func(reference *plumbing.Reference) error {
branches = append(branches, GitBranch{
Hash: reference.Hash().String(),
Name: reference.Name().String(),
Label: reference.Name().Short(),
})
return nil
}); err != nil {
return branches, err
}
return branches, nil
}
// 获取Git Tags
func GetGitTags(s model.Spider) (tags []GitTag, err error) {
// 打开 repo
repo, err := git.PlainOpen(s.Src)
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return tags, err
}
iter, err := repo.Tags()
if iter == nil {
return tags, nil
}
if err := iter.ForEach(func(reference *plumbing.Reference) error {
tags = append(tags, GitTag{
Hash: reference.Hash().String(),
Name: reference.Name().String(),
Label: reference.Name().Short(),
})
return nil
}); err != nil {
return tags, err
}
return tags, nil
}
// 获取Git Head Hash
func GetGitHeadHash(repo *git.Repository) string {
head, _ := repo.Head()
return head.Hash().String()
}
// 获取Git远端分支
func GetGitRemoteBranches(s model.Spider) (branches []GitBranch, err error) {
// 打开 repo
repo, err := git.PlainOpen(s.Src)
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return branches, err
}
iter, err := repo.References()
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return branches, err
}
if err := iter.ForEach(func(reference *plumbing.Reference) error {
if reference.Name().IsRemote() {
log.Infof(reference.Hash().String())
log.Infof(reference.Name().String())
branches = append(branches, GitBranch{
Hash: reference.Hash().String(),
Name: reference.Name().String(),
Label: reference.Name().Short(),
})
}
return nil
}); err != nil {
log.Error(err.Error())
debug.PrintStack()
return branches, err
}
return branches, err
}
// 获取Git Commits
func GetGitCommits(s model.Spider) (commits []GitCommit, err error) {
// 打开 repo
repo, err := git.PlainOpen(s.Src)
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return commits, err
}
// 获取分支列表
branches, err := GetGitBranches(s)
branchesDict := map[string][]GitBranch{}
for _, b := range branches {
branchesDict[b.Hash] = append(branchesDict[b.Hash], b)
}
// 获取分支列表
remoteBranches, err := GetGitRemoteBranches(s)
remoteBranchesDict := map[string][]GitBranch{}
for _, b := range remoteBranches {
remoteBranchesDict[b.Hash] = append(remoteBranchesDict[b.Hash], b)
}
// 获取标签列表
tags, err := GetGitTags(s)
tagsDict := map[string][]GitTag{}
for _, t := range tags {
tagsDict[t.Hash] = append(tagsDict[t.Hash], t)
}
// 获取日志遍历器
iter, err := repo.Log(&git.LogOptions{
All: true,
})
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return commits, err
}
// 遍历日志
if err := iter.ForEach(func(commit *object.Commit) error {
gc := GitCommit{
Hash: commit.Hash.String(),
TreeHash: commit.TreeHash.String(),
Message: commit.Message,
Author: commit.Author.Name,
Email: commit.Author.Email,
Ts: commit.Author.When,
IsHead: commit.Hash.String() == GetGitHeadHash(repo),
Branches: branchesDict[commit.Hash.String()],
RemoteBranches: remoteBranchesDict[commit.Hash.String()],
Tags: tagsDict[commit.Hash.String()],
}
commits = append(commits, gc)
return nil
}); err != nil {
log.Error(err.Error())
debug.PrintStack()
return commits, err
}
return commits, nil
}
func GitCheckout(s model.Spider, hash string) (err error) {
// 打开 repo
repo, err := git.PlainOpen(s.Src)
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return err
}
// 获取worktree
wt, err := repo.Worktree()
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return err
}
// Checkout
if err := wt.Checkout(&git.CheckoutOptions{
Hash: plumbing.NewHash(hash),
Create: false,
Force: true,
Keep: false,
}); err != nil {
log.Error(err.Error())
debug.PrintStack()
return err
}
return nil
}

2
backend/services/msg_handler/handler.go
View File

@@ -11,7 +11,7 @@ type Handler interface {
}
func GetMsgHandler(msg entity.NodeMessage) Handler {
log.Infof("received msg , type is : %s", msg.Type)
log.Debugf("received msg , type is : %s", msg.Type)
if msg.Type == constants.MsgTypeGetLog || msg.Type == constants.MsgTypeRemoveLog {
// 日志相关
return &Log{

16
backend/services/node.go
View File

@@ -15,6 +15,7 @@ import (
"github.com/globalsign/mgo"
"github.com/globalsign/mgo/bson"
"github.com/gomodule/redigo/redis"
"github.com/spf13/viper"
"runtime/debug"
"time"
)
@@ -104,6 +105,19 @@ func UpdateNodeStatus() {
model.ResetNodeStatusToOffline(list)
}
func getNodeName(data *Data) string {
registerType := viper.GetString("server.register.type")
if registerType == constants.RegisterTypeMac {
return data.Ip
} else if registerType == constants.RegisterTypeIp {
return data.Ip
} else if registerType == constants.RegisterTypeHostname {
return data.Hostname
} else {
return data.Ip
}
}
// 处理节点信息
func handleNodeInfo(key string, data *Data) {
// 添加同步锁
@@ -122,7 +136,7 @@ func handleNodeInfo(key string, data *Data) {
// 数据库不存在该节点
node = model.Node{
Key: key,
Name: data.Ip,
Name: getNodeName(data),
Ip: data.Ip,
Port: "8000",
Mac: data.Mac,

8
backend/services/register/register.go
View File

@@ -10,6 +10,7 @@ import (
"os/exec"
"reflect"
"runtime/debug"
"strings"
"sync"
)
@@ -145,7 +146,7 @@ func getHostname() (string, error) {
return "", err
}
return stdout.String(), nil
return strings.Replace(stdout.String(), "\n", "", -1), nil
}
// ===================== 获得注册简单工厂 =====================
@@ -156,11 +157,6 @@ var once sync.Once
func GetRegister() Register {
once.Do(func() {
if register != nil {
register = register
}
registerType := viper.GetString("server.register.type")
if registerType == constants.RegisterTypeMac {
register = &MacRegister{}

234
backend/services/rpc.go
View File

@@ -1,234 +0,0 @@
package services
import (
"crawlab/constants"
"crawlab/database"
"crawlab/entity"
"crawlab/model"
"crawlab/utils"
"encoding/json"
"fmt"
"github.com/apex/log"
"github.com/gomodule/redigo/redis"
uuid "github.com/satori/go.uuid"
"runtime/debug"
)
type RpcMessage struct {
Id string `json:"id"`
Method string `json:"method"`
Params map[string]string `json:"params"`
Result string `json:"result"`
}
func RpcServerInstallLang(msg RpcMessage) RpcMessage {
lang := GetRpcParam("lang", msg.Params)
if lang == constants.Nodejs {
output, _ := InstallNodejsLocalLang()
msg.Result = output
}
return msg
}
func RpcClientInstallLang(nodeId string, lang string) (output string, err error) {
params := map[string]string{}
params["lang"] = lang
data, err := RpcClientFunc(nodeId, constants.RpcInstallLang, params, 600)()
if err != nil {
return
}
output = data
return
}
func RpcServerInstallDep(msg RpcMessage) RpcMessage {
lang := GetRpcParam("lang", msg.Params)
depName := GetRpcParam("dep_name", msg.Params)
if lang == constants.Python {
output, _ := InstallPythonLocalDep(depName)
msg.Result = output
}
return msg
}
func RpcClientInstallDep(nodeId string, lang string, depName string) (output string, err error) {
params := map[string]string{}
params["lang"] = lang
params["dep_name"] = depName
data, err := RpcClientFunc(nodeId, constants.RpcInstallDep, params, 10)()
if err != nil {
return
}
output = data
return
}
func RpcServerUninstallDep(msg RpcMessage) RpcMessage {
lang := GetRpcParam("lang", msg.Params)
depName := GetRpcParam("dep_name", msg.Params)
if lang == constants.Python {
output, _ := UninstallPythonLocalDep(depName)
msg.Result = output
}
return msg
}
func RpcClientUninstallDep(nodeId string, lang string, depName string) (output string, err error) {
params := map[string]string{}
params["lang"] = lang
params["dep_name"] = depName
data, err := RpcClientFunc(nodeId, constants.RpcUninstallDep, params, 60)()
if err != nil {
return
}
output = data
return
}
func RpcServerGetInstalledDepList(nodeId string, msg RpcMessage) RpcMessage {
lang := GetRpcParam("lang", msg.Params)
if lang == constants.Python {
depList, _ := GetPythonLocalInstalledDepList(nodeId)
resultStr, _ := json.Marshal(depList)
msg.Result = string(resultStr)
} else if lang == constants.Nodejs {
depList, _ := GetNodejsLocalInstalledDepList(nodeId)
resultStr, _ := json.Marshal(depList)
msg.Result = string(resultStr)
}
return msg
}
func RpcClientGetInstalledDepList(nodeId string, lang string) (list []entity.Dependency, err error) {
params := map[string]string{}
params["lang"] = lang
data, err := RpcClientFunc(nodeId, constants.RpcGetInstalledDepList, params, 10)()
if err != nil {
return
}
// 反序列化结果
if err := json.Unmarshal([]byte(data), &list); err != nil {
return list, err
}
return
}
func RpcClientFunc(nodeId string, method string, params map[string]string, timeout int) func() (string, error) {
return func() (result string, err error) {
// 请求ID
id := uuid.NewV4().String()
// 构造RPC消息
msg := RpcMessage{
Id: id,
Method: method,
Params: params,
Result: "",
}
// 发送RPC消息
msgStr := ObjectToString(msg)
if err := database.RedisClient.LPush(fmt.Sprintf("rpc:%s", nodeId), msgStr); err != nil {
return result, err
}
// 获取RPC回复消息
dataStr, err := database.RedisClient.BRPop(fmt.Sprintf("rpc:%s", nodeId), timeout)
if err != nil {
return result, err
}
// 反序列化消息
if err := json.Unmarshal([]byte(dataStr), &msg); err != nil {
return result, err
}
return msg.Result, err
}
}
func GetRpcParam(key string, params map[string]string) string {
return params[key]
}
func ObjectToString(params interface{}) string {
bytes, _ := json.Marshal(params)
return utils.BytesToString(bytes)
}
var IsRpcStopped = false
func StopRpcService() {
IsRpcStopped = true
}
func InitRpcService() error {
go func() {
for {
// 获取当前节点
node, err := model.GetCurrentNode()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
continue
}
// 获取获取消息队列信息
dataStr, err := database.RedisClient.BRPop(fmt.Sprintf("rpc:%s", node.Id.Hex()), 0)
if err != nil {
if err != redis.ErrNil {
log.Errorf(err.Error())
debug.PrintStack()
}
continue
}
// 反序列化消息
var msg RpcMessage
if err := json.Unmarshal([]byte(dataStr), &msg); err != nil {
log.Errorf(err.Error())
debug.PrintStack()
continue
}
// 根据Method调用本地方法
var replyMsg RpcMessage
if msg.Method == constants.RpcInstallDep {
replyMsg = RpcServerInstallDep(msg)
} else if msg.Method == constants.RpcUninstallDep {
replyMsg = RpcServerUninstallDep(msg)
} else if msg.Method == constants.RpcInstallLang {
replyMsg = RpcServerInstallLang(msg)
} else if msg.Method == constants.RpcGetInstalledDepList {
replyMsg = RpcServerGetInstalledDepList(node.Id.Hex(), msg)
} else {
continue
}
// 发送返回消息
if err := database.RedisClient.LPush(fmt.Sprintf("rpc:%s", node.Id.Hex()), ObjectToString(replyMsg)); err != nil {
log.Errorf(err.Error())
debug.PrintStack()
continue
}
// 如果停止RPC服务，则返回
if IsRpcStopped {
return
}
}
}()
return nil
}

132
backend/services/rpc/base.go Normal file
View File

@@ -0,0 +1,132 @@
package rpc
import (
"crawlab/constants"
"crawlab/database"
"crawlab/entity"
"crawlab/model"
"crawlab/utils"
"encoding/json"
"errors"
"fmt"
"github.com/apex/log"
"github.com/gomodule/redigo/redis"
uuid "github.com/satori/go.uuid"
"runtime/debug"
)
// RPC服务基础类
type Service interface {
ServerHandle() (entity.RpcMessage, error)
ClientHandle() (interface{}, error)
}
// 客户端处理消息函数
func ClientFunc(msg entity.RpcMessage) func() (entity.RpcMessage, error) {
return func() (replyMsg entity.RpcMessage, err error) {
// 请求ID
msg.Id = uuid.NewV4().String()
// 发送RPC消息
msgStr := utils.ObjectToString(msg)
if err := database.RedisClient.LPush(fmt.Sprintf("rpc:%s", msg.NodeId), msgStr); err != nil {
log.Errorf("RpcClientFunc error: " + err.Error())
debug.PrintStack()
return replyMsg, err
}
// 获取RPC回复消息
dataStr, err := database.RedisClient.BRPop(fmt.Sprintf("rpc:%s:%s", msg.NodeId, msg.Id), msg.Timeout)
if err != nil {
log.Errorf("RpcClientFunc error: " + err.Error())
debug.PrintStack()
return replyMsg, err
}
// 反序列化消息
if err := json.Unmarshal([]byte(dataStr), &replyMsg); err != nil {
log.Errorf("RpcClientFunc error: " + err.Error())
debug.PrintStack()
return replyMsg, err
}
// 如果返回消息有错误，返回错误
if replyMsg.Error != "" {
return replyMsg, errors.New(replyMsg.Error)
}
return
}
}
// 获取RPC服务
func GetService(msg entity.RpcMessage) Service {
switch msg.Method {
case constants.RpcInstallLang:
return &InstallLangService{msg: msg}
case constants.RpcInstallDep:
return &InstallDepService{msg: msg}
case constants.RpcUninstallDep:
return &UninstallDepService{msg: msg}
case constants.RpcGetLang:
return &GetLangService{msg: msg}
case constants.RpcGetInstalledDepList:
return &GetInstalledDepsService{msg: msg}
}
return nil
}
// 处理RPC消息
func handleMsg(msgStr string, node model.Node) {
// 反序列化消息
var msg entity.RpcMessage
if err := json.Unmarshal([]byte(msgStr), &msg); err != nil {
log.Errorf(err.Error())
debug.PrintStack()
}
// 获取service
service := GetService(msg)
// 根据Method调用本地方法
replyMsg, err := service.ServerHandle()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
}
// 发送返回消息
if err := database.RedisClient.LPush(fmt.Sprintf("rpc:%s:%s", node.Id.Hex(), replyMsg.Id), utils.ObjectToString(replyMsg)); err != nil {
log.Errorf(err.Error())
debug.PrintStack()
}
}
// 初始化服务端RPC服务
func InitRpcService() error {
go func() {
for {
// 获取当前节点
node, err := model.GetCurrentNode()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
continue
}
// 获取获取消息队列信息
msgStr, err := database.RedisClient.BRPop(fmt.Sprintf("rpc:%s", node.Id.Hex()), 0)
if err != nil {
if err != redis.ErrNil {
log.Errorf(err.Error())
debug.PrintStack()
}
continue
}
// 处理消息
go handleMsg(msgStr, node)
}
}()
return nil
}

123
backend/services/rpc/get_installed_deps.go Normal file
View File

@@ -0,0 +1,123 @@
package rpc
import (
"crawlab/constants"
"crawlab/entity"
"crawlab/utils"
"encoding/json"
"os/exec"
"regexp"
"runtime/debug"
"strings"
)
type GetInstalledDepsService struct {
msg entity.RpcMessage
}
func (s *GetInstalledDepsService) ServerHandle() (entity.RpcMessage, error) {
lang := utils.GetRpcParam("lang", s.msg.Params)
deps, err := GetInstalledDepsLocal(lang)
if err != nil {
s.msg.Error = err.Error()
return s.msg, err
}
resultStr, _ := json.Marshal(deps)
s.msg.Result = string(resultStr)
return s.msg, nil
}
func (s *GetInstalledDepsService) ClientHandle() (o interface{}, err error) {
// 发起 RPC 请求，获取服务端数据
s.msg, err = ClientFunc(s.msg)()
if err != nil {
return o, err
}
// 反序列化
var output []entity.Dependency
if err := json.Unmarshal([]byte(s.msg.Result), &output); err != nil {
return o, err
}
o = output
return
}
// 获取本地已安装依赖列表
func GetInstalledDepsLocal(lang string) (deps []entity.Dependency, err error) {
if lang == constants.Python {
deps, err = GetPythonInstalledDepListLocal()
} else if lang == constants.Nodejs {
deps, err = GetNodejsInstalledDepListLocal()
}
return deps, err
}
// 获取Python本地已安装依赖列表
func GetPythonInstalledDepListLocal() ([]entity.Dependency, error) {
var list []entity.Dependency
cmd := exec.Command("pip", "freeze")
outputBytes, err := cmd.Output()
if err != nil {
debug.PrintStack()
return list, err
}
for _, line := range strings.Split(string(outputBytes), "\n") {
arr := strings.Split(line, "==")
if len(arr) < 2 {
continue
}
dep := entity.Dependency{
Name: strings.ToLower(arr[0]),
Version: arr[1],
Installed: true,
}
list = append(list, dep)
}
return list, nil
}
// 获取Node.js本地已安装依赖列表
func GetNodejsInstalledDepListLocal() ([]entity.Dependency, error) {
var list []entity.Dependency
cmd := exec.Command("npm", "ls", "-g", "--depth", "0")
outputBytes, _ := cmd.Output()
regex := regexp.MustCompile("\\s(.*)@(.*)")
for _, line := range strings.Split(string(outputBytes), "\n") {
arr := regex.FindStringSubmatch(line)
if len(arr) < 3 {
continue
}
dep := entity.Dependency{
Name: strings.ToLower(arr[1]),
Version: arr[2],
Installed: true,
}
list = append(list, dep)
}
return list, nil
}
func GetInstalledDepsRemote(nodeId string, lang string) (deps []entity.Dependency, err error) {
params := make(map[string]string)
params["lang"] = lang
s := GetService(entity.RpcMessage{
NodeId: nodeId,
Method: constants.RpcGetInstalledDepList,
Params: params,
Timeout: 60,
})
o, err := s.ClientHandle()
if err != nil {
return
}
deps = o.([]entity.Dependency)
return
}

82
backend/services/rpc/get_lang.go Normal file
View File

@@ -0,0 +1,82 @@
package rpc
import (
"crawlab/constants"
"crawlab/entity"
"crawlab/utils"
"encoding/json"
)
type GetLangService struct {
msg entity.RpcMessage
}
func (s *GetLangService) ServerHandle() (entity.RpcMessage, error) {
langName := utils.GetRpcParam("lang", s.msg.Params)
lang := utils.GetLangFromLangNamePlain(langName)
l := GetLangLocal(lang)
lang.InstallStatus = l.InstallStatus
// 序列化
resultStr, _ := json.Marshal(lang)
s.msg.Result = string(resultStr)
return s.msg, nil
}
func (s *GetLangService) ClientHandle() (o interface{}, err error) {
// 发起 RPC 请求，获取服务端数据
s.msg, err = ClientFunc(s.msg)()
if err != nil {
return o, err
}
var output entity.Lang
if err := json.Unmarshal([]byte(s.msg.Result), &output); err != nil {
return o, err
}
o = output
return
}
func GetLangLocal(lang entity.Lang) entity.Lang {
// 检查是否存在执行路径
for _, p := range lang.ExecutablePaths {
if utils.Exists(p) {
lang.InstallStatus = constants.InstallStatusInstalled
return lang
}
}
// 检查是否正在安装
if utils.Exists(lang.LockPath) {
lang.InstallStatus = constants.InstallStatusInstalling
return lang
}
// 检查其他语言是否在安装
if utils.Exists("/tmp/install.lock") {
lang.InstallStatus = constants.InstallStatusInstallingOther
return lang
}
lang.InstallStatus = constants.InstallStatusNotInstalled
return lang
}
func GetLangRemote(nodeId string, lang entity.Lang) (l entity.Lang, err error) {
params := make(map[string]string)
params["lang"] = lang.ExecutableName
s := GetService(entity.RpcMessage{
NodeId: nodeId,
Method: constants.RpcGetLang,
Params: params,
Timeout: 60,
})
o, err := s.ClientHandle()
if err != nil {
return
}
l = o.(entity.Lang)
return
}

100
backend/services/rpc/install_dep.go Normal file
View File

@@ -0,0 +1,100 @@
package rpc
import (
"crawlab/constants"
"crawlab/entity"
"crawlab/utils"
"errors"
"fmt"
"github.com/apex/log"
"os/exec"
"runtime/debug"
)
type InstallDepService struct {
msg entity.RpcMessage
}
func (s *InstallDepService) ServerHandle() (entity.RpcMessage, error) {
lang := utils.GetRpcParam("lang", s.msg.Params)
depName := utils.GetRpcParam("dep_name", s.msg.Params)
if err := InstallDepLocal(lang, depName); err != nil {
s.msg.Error = err.Error()
return s.msg, err
}
s.msg.Result = "success"
return s.msg, nil
}
func (s *InstallDepService) ClientHandle() (o interface{}, err error) {
// 发起 RPC 请求，获取服务端数据
_, err = ClientFunc(s.msg)()
if err != nil {
return
}
return
}
func InstallDepLocal(lang string, depName string) error {
if lang == constants.Python {
_, err := InstallPythonDepLocal(depName)
if err != nil {
return err
}
} else if lang == constants.Nodejs {
_, err := InstallNodejsDepLocal(depName)
if err != nil {
return err
}
} else {
return errors.New(fmt.Sprintf("%s is not implemented", lang))
}
return nil
}
// 安装Python本地依赖
func InstallPythonDepLocal(depName string) (string, error) {
// 依赖镜像URL
url := "https://pypi.tuna.tsinghua.edu.cn/simple"
cmd := exec.Command("pip", "install", depName, "-i", url)
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
func InstallNodejsDepLocal(depName string) (string, error) {
// 依赖镜像URL
url := "https://registry.npm.taobao.org"
cmd := exec.Command("npm", "install", depName, "-g", "--registry", url)
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
func InstallDepRemote(nodeId string, lang string, depName string) (err error) {
params := make(map[string]string)
params["lang"] = lang
params["dep_name"] = depName
s := GetService(entity.RpcMessage{
NodeId: nodeId,
Method: constants.RpcInstallDep,
Params: params,
Timeout: 300,
})
_, err = s.ClientHandle()
if err != nil {
return
}
return
}

73
backend/services/rpc/install_lang.go Normal file
View File

@@ -0,0 +1,73 @@
package rpc
import (
"crawlab/constants"
"crawlab/entity"
"crawlab/utils"
"errors"
"fmt"
"github.com/apex/log"
"os/exec"
"path"
"runtime/debug"
)
type InstallLangService struct {
msg entity.RpcMessage
}
func (s *InstallLangService) ServerHandle() (entity.RpcMessage, error) {
lang := utils.GetRpcParam("lang", s.msg.Params)
output, err := InstallLangLocal(lang)
s.msg.Result = output
if err != nil {
s.msg.Error = err.Error()
return s.msg, err
}
return s.msg, nil
}
func (s *InstallLangService) ClientHandle() (o interface{}, err error) {
// 发起 RPC 请求，获取服务端数据
go func() {
_, err := ClientFunc(s.msg)()
if err != nil {
return
}
}()
return
}
// 本地安装语言
func InstallLangLocal(lang string) (o string, err error) {
l := utils.GetLangFromLangNamePlain(lang)
if l.Name == "" || l.InstallScript == "" {
return "", errors.New(fmt.Sprintf("%s is not implemented", lang))
}
cmd := exec.Command("/bin/sh", path.Join("scripts", l.InstallScript))
output, err := cmd.Output()
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return string(output), err
}
return
}
// 远端安装语言
func InstallLangRemote(nodeId string, lang string) (o string, err error) {
params := make(map[string]string)
params["lang"] = lang
s := GetService(entity.RpcMessage{
NodeId: nodeId,
Method: constants.RpcInstallLang,
Params: params,
Timeout: 60,
})
_, err = s.ClientHandle()
if err != nil {
return
}
return
}

96
backend/services/rpc/uninstall_dep.go Normal file
View File

@@ -0,0 +1,96 @@
package rpc
import (
"crawlab/constants"
"crawlab/entity"
"crawlab/utils"
"errors"
"fmt"
"github.com/apex/log"
"os/exec"
"runtime/debug"
)
type UninstallDepService struct {
msg entity.RpcMessage
}
func (s *UninstallDepService) ServerHandle() (entity.RpcMessage, error) {
lang := utils.GetRpcParam("lang", s.msg.Params)
depName := utils.GetRpcParam("dep_name", s.msg.Params)
if err := UninstallDepLocal(lang, depName); err != nil {
s.msg.Error = err.Error()
return s.msg, err
}
s.msg.Result = "success"
return s.msg, nil
}
func (s *UninstallDepService) ClientHandle() (o interface{}, err error) {
// 发起 RPC 请求，获取服务端数据
_, err = ClientFunc(s.msg)()
if err != nil {
return
}
return
}
func UninstallDepLocal(lang string, depName string) error {
if lang == constants.Python {
output, err := UninstallPythonDepLocal(depName)
if err != nil {
log.Debugf(output)
return err
}
} else if lang == constants.Nodejs {
output, err := UninstallNodejsDepLocal(depName)
if err != nil {
log.Debugf(output)
return err
}
} else {
return errors.New(fmt.Sprintf("%s is not implemented", lang))
}
return nil
}
func UninstallPythonDepLocal(depName string) (string, error) {
cmd := exec.Command("pip", "uninstall", "-y", depName)
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(string(outputBytes))
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
func UninstallNodejsDepLocal(depName string) (string, error) {
cmd := exec.Command("npm", "uninstall", depName, "-g")
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
func UninstallDepRemote(nodeId string, lang string, depName string) (err error) {
params := make(map[string]string)
params["lang"] = lang
params["dep_name"] = depName
s := GetService(entity.RpcMessage{
NodeId: nodeId,
Method: constants.RpcUninstallDep,
Params: params,
Timeout: 300,
})
_, err = s.ClientHandle()
if err != nil {
return
}
return
}

213
backend/services/spider.go
View File

@@ -412,6 +412,111 @@ func CopySpider(spider model.Spider, newName string) error {
return nil
}
func InitDemoSpiders () {
// 添加Demo爬虫
templateSpidersDir := "./template/spiders"
for _, info := range utils.ListDir(templateSpidersDir) {
if !info.IsDir() {
continue
}
spiderName := info.Name()
// 如果爬虫在数据库中不存在，则添加
spider := model.GetSpiderByName(spiderName)
if spider.Name != "" {
// 存在同名爬虫，跳过
continue
}
// 拷贝爬虫
templateSpiderPath := path.Join(templateSpidersDir, spiderName)
spiderPath := path.Join(viper.GetString("spider.path"), spiderName)
if utils.Exists(spiderPath) {
utils.RemoveFiles(spiderPath)
}
if err := utils.CopyDir(templateSpiderPath, spiderPath); err != nil {
log.Errorf("copy error: " + err.Error())
debug.PrintStack()
continue
}
// 构造配置数据
configData := entity.ConfigSpiderData{}
// 读取YAML文件
yamlFile, err := ioutil.ReadFile(path.Join(spiderPath, "Spiderfile"))
if err != nil {
log.Errorf("read yaml error: " + err.Error())
//debug.PrintStack()
continue
}
// 反序列化
if err := yaml.Unmarshal(yamlFile, &configData); err != nil {
log.Errorf("unmarshal error: " + err.Error())
debug.PrintStack()
continue
}
if configData.Type == constants.Customized {
// 添加该爬虫到数据库
spider = model.Spider{
Id: bson.NewObjectId(),
Name: spiderName,
DisplayName: configData.DisplayName,
Type: constants.Customized,
Col: configData.Col,
Src: spiderPath,
Remark: configData.Remark,
ProjectId: bson.ObjectIdHex(constants.ObjectIdNull),
FileId: bson.ObjectIdHex(constants.ObjectIdNull),
Cmd: configData.Cmd,
}
if err := spider.Add(); err != nil {
log.Errorf("add spider error: " + err.Error())
debug.PrintStack()
continue
}
// 上传爬虫到GridFS
if err := UploadSpiderToGridFsFromMaster(spider); err != nil {
log.Errorf("upload spider error: " + err.Error())
debug.PrintStack()
continue
}
} else if configData.Type == constants.Configurable || configData.Type == "config" {
// 添加该爬虫到数据库
spider = model.Spider{
Id: bson.NewObjectId(),
Name: configData.Name,
DisplayName: configData.DisplayName,
Type: constants.Configurable,
Col: configData.Col,
Src: spiderPath,
Remark: configData.Remark,
ProjectId: bson.ObjectIdHex(constants.ObjectIdNull),
FileId: bson.ObjectIdHex(constants.ObjectIdNull),
Config: configData,
}
if err := spider.Add(); err != nil {
log.Errorf("add spider error: " + err.Error())
debug.PrintStack()
continue
}
// 根据序列化后的数据处理爬虫文件
if err := ProcessSpiderFilesFromConfigData(spider, configData); err != nil {
log.Errorf("add spider error: " + err.Error())
debug.PrintStack()
continue
}
}
}
// 发布所有爬虫
PublishAllSpiders()
}
// 启动爬虫服务
func InitSpiderService() error {
// 构造定时任务执行器
@@ -423,110 +528,12 @@ func InitSpiderService() error {
// 启动定时任务
cPub.Start()
if model.IsMaster() && viper.GetString("setting.demoSpiders") == "Y" {
// 初始化Demo爬虫
InitDemoSpiders()
}
if model.IsMaster() {
// 添加Demo爬虫
templateSpidersDir := "./template/spiders"
for _, info := range utils.ListDir(templateSpidersDir) {
if !info.IsDir() {
continue
}
spiderName := info.Name()
// 如果爬虫在数据库中不存在，则添加
spider := model.GetSpiderByName(spiderName)
if spider.Name != "" {
// 存在同名爬虫，跳过
continue
}
// 拷贝爬虫
templateSpiderPath := path.Join(templateSpidersDir, spiderName)
spiderPath := path.Join(viper.GetString("spider.path"), spiderName)
if utils.Exists(spiderPath) {
utils.RemoveFiles(spiderPath)
}
if err := utils.CopyDir(templateSpiderPath, spiderPath); err != nil {
log.Errorf("copy error: " + err.Error())
debug.PrintStack()
continue
}
// 构造配置数据
configData := entity.ConfigSpiderData{}
// 读取YAML文件
yamlFile, err := ioutil.ReadFile(path.Join(spiderPath, "Spiderfile"))
if err != nil {
log.Errorf("read yaml error: " + err.Error())
//debug.PrintStack()
continue
}
// 反序列化
if err := yaml.Unmarshal(yamlFile, &configData); err != nil {
log.Errorf("unmarshal error: " + err.Error())
debug.PrintStack()
continue
}
if configData.Type == constants.Customized {
// 添加该爬虫到数据库
spider = model.Spider{
Id: bson.NewObjectId(),
Name: spiderName,
DisplayName: configData.DisplayName,
Type: constants.Customized,
Col: configData.Col,
Src: spiderPath,
Remark: configData.Remark,
ProjectId: bson.ObjectIdHex(constants.ObjectIdNull),
FileId: bson.ObjectIdHex(constants.ObjectIdNull),
Cmd: configData.Cmd,
}
if err := spider.Add(); err != nil {
log.Errorf("add spider error: " + err.Error())
debug.PrintStack()
continue
}
// 上传爬虫到GridFS
if err := UploadSpiderToGridFsFromMaster(spider); err != nil {
log.Errorf("upload spider error: " + err.Error())
debug.PrintStack()
continue
}
} else if configData.Type == constants.Configurable || configData.Type == "config" {
// 添加该爬虫到数据库
spider = model.Spider{
Id: bson.NewObjectId(),
Name: configData.Name,
DisplayName: configData.DisplayName,
Type: constants.Configurable,
Col: configData.Col,
Src: spiderPath,
Remark: configData.Remark,
ProjectId: bson.ObjectIdHex(constants.ObjectIdNull),
FileId: bson.ObjectIdHex(constants.ObjectIdNull),
Config: configData,
}
if err := spider.Add(); err != nil {
log.Errorf("add spider error: " + err.Error())
debug.PrintStack()
continue
}
// 根据序列化后的数据处理爬虫文件
if err := ProcessSpiderFilesFromConfigData(spider, configData); err != nil {
log.Errorf("add spider error: " + err.Error())
debug.PrintStack()
continue
}
}
}
// 发布所有爬虫
PublishAllSpiders()
// 构造 Git 定时任务
GitCron = &GitCronScheduler{
cron: cron.New(cron.WithSeconds()),

259
backend/services/system.go
View File

@@ -6,6 +6,7 @@ import (
"crawlab/entity"
"crawlab/lib/cron"
"crawlab/model"
"crawlab/services/rpc"
"crawlab/utils"
"encoding/json"
"errors"
@@ -13,7 +14,6 @@ import (
"github.com/apex/log"
"github.com/imroc/req"
"os/exec"
"path"
"regexp"
"runtime/debug"
"sort"
@@ -64,42 +64,25 @@ func GetSystemInfo(nodeId string) (sysInfo entity.SystemInfo, err error) {
// 获取语言列表
func GetLangList(nodeId string) []entity.Lang {
list := []entity.Lang{
{Name: "Python", ExecutableName: "python", ExecutablePaths: []string{"/usr/bin/python", "/usr/local/bin/python"}, DepExecutablePath: "/usr/local/bin/pip"},
{Name: "Node.js", ExecutableName: "node", ExecutablePaths: []string{"/usr/bin/node", "/usr/local/bin/node"}, DepExecutablePath: "/usr/local/bin/npm"},
//{Name: "Java", ExecutableName: "java", ExecutablePaths: []string{"/usr/bin/java", "/usr/local/bin/java"}},
}
list := utils.GetLangList()
for i, lang := range list {
list[i].Installed = IsInstalledLang(nodeId, lang)
status, _ := GetLangInstallStatus(nodeId, lang)
list[i].InstallStatus = status
}
return list
}
// 根据语言名获取语言实例
func GetLangFromLangName(nodeId string, name string) entity.Lang {
langList := GetLangList(nodeId)
for _, lang := range langList {
if lang.ExecutableName == name {
return lang
func GetLangInstallStatus(nodeId string, lang entity.Lang) (string, error) {
if IsMasterNode(nodeId) {
lang := rpc.GetLangLocal(lang)
return lang.InstallStatus, nil
} else {
lang, err := rpc.GetLangRemote(nodeId, lang)
if err != nil {
return "", err
}
return lang.InstallStatus, nil
}
return entity.Lang{}
}
// 是否已安装该依赖
func IsInstalledLang(nodeId string, lang entity.Lang) bool {
sysInfo, err := GetSystemInfo(nodeId)
if err != nil {
return false
}
for _, exec := range sysInfo.Executables {
for _, path := range lang.ExecutablePaths {
if exec.Path == path {
return true
}
}
}
return false
}
// 是否已安装该依赖
@@ -112,6 +95,8 @@ func IsInstalledDep(installedDepList []entity.Dependency, dep entity.Dependency)
return false
}
// ========Python========
// 初始化函数
func InitDepsFetcher() error {
c := cron.New(cron.WithSeconds())
@@ -126,10 +111,6 @@ func InitDepsFetcher() error {
return nil
}
// =========
// Python
// =========
type PythonDepJsonData struct {
Info PythonDepJsonDataInfo `json:"info"`
}
@@ -183,12 +164,12 @@ func GetPythonDepList(nodeId string, searchDepName string) ([]entity.Dependency,
// 获取已安装依赖列表
var installedDepList []entity.Dependency
if IsMasterNode(nodeId) {
installedDepList, err = GetPythonLocalInstalledDepList(nodeId)
installedDepList, err = rpc.GetInstalledDepsLocal(constants.Python)
if err != nil {
return list, err
}
} else {
installedDepList, err = GetPythonRemoteInstalledDepList(nodeId)
installedDepList, err = rpc.GetInstalledDepsRemote(nodeId, constants.Python)
if err != nil {
return list, err
}
@@ -351,205 +332,9 @@ func UpdatePythonDepList() {
}
}
// 获取Python本地已安装的依赖列表
func GetPythonLocalInstalledDepList(nodeId string) ([]entity.Dependency, error) {
var list []entity.Dependency
// ========./Python========
lang := GetLangFromLangName(nodeId, constants.Python)
if !IsInstalledLang(nodeId, lang) {
return list, errors.New("python is not installed")
}
cmd := exec.Command("pip", "freeze")
outputBytes, err := cmd.Output()
if err != nil {
debug.PrintStack()
return list, err
}
for _, line := range strings.Split(string(outputBytes), "\n") {
arr := strings.Split(line, "==")
if len(arr) < 2 {
continue
}
dep := entity.Dependency{
Name: strings.ToLower(arr[0]),
Version: arr[1],
Installed: true,
}
list = append(list, dep)
}
return list, nil
}
// 获取Python远端依赖列表
func GetPythonRemoteInstalledDepList(nodeId string) ([]entity.Dependency, error) {
depList, err := RpcClientGetInstalledDepList(nodeId, constants.Python)
if err != nil {
return depList, err
}
return depList, nil
}
// 安装Python本地依赖
func InstallPythonLocalDep(depName string) (string, error) {
// 依赖镜像URL
url := "https://pypi.tuna.tsinghua.edu.cn/simple"
cmd := exec.Command("pip", "install", depName, "-i", url)
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
// 获取Python远端依赖列表
func InstallPythonRemoteDep(nodeId string, depName string) (string, error) {
output, err := RpcClientInstallDep(nodeId, constants.Python, depName)
if err != nil {
return output, err
}
return output, nil
}
// 安装Python本地依赖
func UninstallPythonLocalDep(depName string) (string, error) {
cmd := exec.Command("pip", "uninstall", "-y", depName)
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(string(outputBytes))
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
// 获取Python远端依赖列表
func UninstallPythonRemoteDep(nodeId string, depName string) (string, error) {
output, err := RpcClientUninstallDep(nodeId, constants.Python, depName)
if err != nil {
return output, err
}
return output, nil
}
// ==============
// Node.js
// ==============
func InstallNodejsLocalLang() (string, error) {
cmd := exec.Command("/bin/sh", path.Join("scripts", "install-nodejs.sh"))
output, err := cmd.Output()
if err != nil {
log.Error(err.Error())
debug.PrintStack()
return string(output), err
}
// TODO: check if Node.js is installed successfully
return string(output), nil
}
// 获取Node.js远端依赖列表
func InstallNodejsRemoteLang(nodeId string) (string, error) {
output, err := RpcClientInstallLang(nodeId, constants.Nodejs)
if err != nil {
return output, err
}
return output, nil
}
// 获取Nodejs本地已安装的依赖列表
func GetNodejsLocalInstalledDepList(nodeId string) ([]entity.Dependency, error) {
var list []entity.Dependency
lang := GetLangFromLangName(nodeId, constants.Nodejs)
if !IsInstalledLang(nodeId, lang) {
return list, errors.New("nodejs is not installed")
}
cmd := exec.Command("npm", "ls", "-g", "--depth", "0")
outputBytes, _ := cmd.Output()
//if err != nil {
// log.Error("error: " + string(outputBytes))
// debug.PrintStack()
// return list, err
//}
regex := regexp.MustCompile("\\s(.*)@(.*)")
for _, line := range strings.Split(string(outputBytes), "\n") {
arr := regex.FindStringSubmatch(line)
if len(arr) < 3 {
continue
}
dep := entity.Dependency{
Name: strings.ToLower(arr[1]),
Version: arr[2],
Installed: true,
}
list = append(list, dep)
}
return list, nil
}
// 获取Nodejs远端依赖列表
func GetNodejsRemoteInstalledDepList(nodeId string) ([]entity.Dependency, error) {
depList, err := RpcClientGetInstalledDepList(nodeId, constants.Nodejs)
if err != nil {
return depList, err
}
return depList, nil
}
// 安装Nodejs本地依赖
func InstallNodejsLocalDep(depName string) (string, error) {
// 依赖镜像URL
url := "https://registry.npm.taobao.org"
cmd := exec.Command("npm", "install", depName, "-g", "--registry", url)
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
// 获取Nodejs远端依赖列表
func InstallNodejsRemoteDep(nodeId string, depName string) (string, error) {
output, err := RpcClientInstallDep(nodeId, constants.Nodejs, depName)
if err != nil {
return output, err
}
return output, nil
}
// 安装Nodejs本地依赖
func UninstallNodejsLocalDep(depName string) (string, error) {
cmd := exec.Command("npm", "uninstall", depName, "-g")
outputBytes, err := cmd.Output()
if err != nil {
log.Errorf(err.Error())
debug.PrintStack()
return fmt.Sprintf("error: %s", err.Error()), err
}
return string(outputBytes), nil
}
// 获取Nodejs远端依赖列表
func UninstallNodejsRemoteDep(nodeId string, depName string) (string, error) {
output, err := RpcClientUninstallDep(nodeId, constants.Nodejs, depName)
if err != nil {
return output, err
}
return output, nil
}
// ========Node.js========
// 获取Nodejs本地依赖列表
func GetNodejsDepList(nodeId string, searchDepName string) (depList []entity.Dependency, err error) {
@@ -560,12 +345,12 @@ func GetNodejsDepList(nodeId string, searchDepName string) (depList []entity.Dep
// 获取已安装依赖列表
var installedDepList []entity.Dependency
if IsMasterNode(nodeId) {
installedDepList, err = GetNodejsLocalInstalledDepList(nodeId)
installedDepList, err = rpc.GetInstalledDepsLocal(constants.Nodejs)
if err != nil {
return depList, err
}
} else {
installedDepList, err = GetNodejsRemoteInstalledDepList(nodeId)
installedDepList, err = rpc.GetInstalledDepsRemote(nodeId, constants.Nodejs)
if err != nil {
return depList, err
}
@@ -585,3 +370,5 @@ func GetNodejsDepList(nodeId string, searchDepName string) (depList []entity.Dep
return depList, nil
}
// ========./Node.js========

16
backend/services/task.go
View File

@@ -852,19 +852,19 @@ func SendNotifications(u model.User, t model.Task, s model.Spider) {
}
}
func UnlockLongTask(s model.Spider, n model.Node) {
if s.IsLongTask {
colName := "long-tasks"
key := fmt.Sprintf("%s:%s", s.Id.Hex(), n.Id.Hex())
_ = database.RedisClient.HDel(colName, key)
}
}
func InitTaskExecutor() error {
// 构造任务执行器
c := cron.New(cron.WithSeconds())
Exec = &Executor{
Cron: c,
}
// 如果不允许主节点运行任务，则跳过
if model.IsMaster() && viper.GetString("setting.runOnMaster") == "N" {
return nil
}
// 运行定时任务
if err := Exec.Start(); err != nil {
return err
}

14
backend/utils/rpc.go Normal file
View File

@@ -0,0 +1,14 @@
package utils
import "encoding/json"
// Object 转化为 String
func ObjectToString(params interface{}) string {
bytes, _ := json.Marshal(params)
return BytesToString(bytes)
}
// 获取 RPC 参数
func GetRpcParam(key string, params map[string]string) string {
return params[key]
}

62
backend/utils/system.go Normal file
View File

@@ -0,0 +1,62 @@
package utils
import "crawlab/entity"
func GetLangList() []entity.Lang {
list := []entity.Lang{
{
Name: "Python",
ExecutableName: "python",
ExecutablePaths: []string{"/usr/bin/python", "/usr/local/bin/python"},
DepExecutablePath: "/usr/local/bin/pip",
LockPath: "/tmp/install-python.lock",
},
{
Name: "Node.js",
ExecutableName: "node",
ExecutablePaths: []string{"/usr/bin/node", "/usr/local/bin/node"},
DepExecutablePath: "/usr/local/bin/npm",
LockPath: "/tmp/install-nodejs.lock",
InstallScript: "install-nodejs.sh",
},
{
Name: "Java",
ExecutableName: "java",
ExecutablePaths: []string{"/usr/bin/java", "/usr/local/bin/java"},
LockPath: "/tmp/install-java.lock",
InstallScript: "install-java.sh",
},
{
Name: ".Net Core",
ExecutableName: "dotnet",
ExecutablePaths: []string{"/usr/bin/dotnet", "/usr/local/bin/dotnet"},
LockPath: "/tmp/install-dotnet.lock",
InstallScript: "install-dotnet.sh",
},
{
Name: "PHP",
ExecutableName: "php",
ExecutablePaths: []string{"/usr/bin/php", "/usr/local/bin/php"},
LockPath: "/tmp/install-php.lock",
InstallScript: "install-php.sh",
},
}
return list
}
// 获取语言列表
func GetLangListPlain() []entity.Lang {
list := GetLangList()
return list
}
// 根据语言名获取语言实例，不包含状态
func GetLangFromLangNamePlain(name string) entity.Lang {
langList := GetLangListPlain()
for _, lang := range langList {
if lang.ExecutableName == name {
return lang
}
}
return entity.Lang{}
}

13
devops/develop/crawlab-master.yaml
View File

@@ -13,13 +13,12 @@ spec:
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
kind: StatefulSet
metadata:
name: crawlab-master
namespace: crawlab-develop
spec:
strategy:
type: Recreate
serviceName: crawlab-master
selector:
matchLabels:
app: crawlab-master
@@ -42,9 +41,13 @@ spec:
- name: CRAWLAB_SETTING_ALLOWREGISTER
value: "Y"
- name: CRAWLAB_SERVER_LANG_NODE
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_JAVA
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_DOTNET
value: "N"
- name: CRAWLAB_SERVER_REGISTER_TYPE
value: "hostname"
ports:
- containerPort: 8080
name: crawlab

14
devops/develop/crawlab-worker.yaml
View File

@@ -1,12 +1,11 @@
apiVersion: apps/v1
kind: Deployment
kind: StatefulSet
metadata:
name: crawlab-worker
namespace: crawlab-develop
spec:
serviceName: crawlab-worker
replicas: 2
strategy:
type: Recreate
selector:
matchLabels:
app: crawlab-worker
@@ -27,7 +26,10 @@ spec:
- name: CRAWLAB_REDIS_ADDRESS
value: "redis"
- name: CRAWLAB_SERVER_LANG_NODE
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_JAVA
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_DOTNET
value: "N"
- name: CRAWLAB_SERVER_REGISTER_TYPE
value: "hostname"

13
devops/release/crawlab-master.yaml
View File

@@ -13,13 +13,12 @@ spec:
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
kind: StatefulSet
metadata:
name: crawlab-master
namespace: crawlab-release
spec:
strategy:
type: Recreate
serviceName: crawlab-master
selector:
matchLabels:
app: crawlab-master
@@ -42,9 +41,13 @@ spec:
- name: CRAWLAB_SETTING_ALLOWREGISTER
value: "Y"
- name: CRAWLAB_SERVER_LANG_NODE
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_JAVA
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_DOTNET
value: "N"
- name: CRAWLAB_SERVER_REGISTER_TYPE
value: "hostname"
ports:
- containerPort: 8080
name: crawlab

13
devops/release/crawlab-worker.yaml
View File

@@ -1,12 +1,11 @@
apiVersion: apps/v1
kind: Deployment
kind: StatefulSet
metadata:
name: crawlab-worker
namespace: crawlab-release
spec:
serviceName: crawlab-worker
replicas: 2
strategy:
type: Recreate
selector:
matchLabels:
app: crawlab-worker
@@ -27,6 +26,10 @@ spec:
- name: CRAWLAB_REDIS_ADDRESS
value: "redis"
- name: CRAWLAB_SERVER_LANG_NODE
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_JAVA
value: "Y"
value: "N"
- name: CRAWLAB_SERVER_LANG_DOTNET
value: "N"
- name: CRAWLAB_SERVER_REGISTER_TYPE
value: "hostname"

6
docker-compose.yml
View File

@@ -21,11 +21,15 @@ services:
# CRAWLAB_LOG_DELETEFREQUENCY: "@hourly" # frequency of deleting log files 删除日志文件的频率. 默认为每小时
# CRAWLAB_SERVER_REGISTER_TYPE: "mac" # node register type 节点注册方式. 默认为 mac 地址，也可设置为 ip（防止 mac 地址冲突）
# CRAWLAB_SERVER_REGISTER_IP: "127.0.0.1" # node register ip 节点注册IP. 节点唯一识别号，只有当 CRAWLAB_SERVER_REGISTER_TYPE 为 "ip" 时才生效
# CRAWLAB_TASK_WORKERS: 4 # number of task executors 任务执行器个数（并行执行任务数）
# CRAWLAB_TASK_WORKERS: 8 # number of task executors 任务执行器个数（并行执行任务数）
# CRAWLAB_SERVER_LANG_NODE: "Y" # whether to pre-install Node.js 预安装 Node.js 语言环境
# CRAWLAB_SERVER_LANG_JAVA: "Y" # whether to pre-install Java 预安装 Java 语言环境
# CRAWLAB_SERVER_LANG_DOTNET: "Y" # whether to pre-install .Net core 预安装 .Net Core 语言环境
# CRAWLAB_SERVER_LANG_PHP: "Y" # whether to pre-install PHP 预安装 PHP 语言环境
# CRAWLAB_SETTING_ALLOWREGISTER: "N" # whether to allow user registration 是否允许用户注册
# CRAWLAB_SETTING_ENABLETUTORIAL: "N" # whether to enable tutorial 是否启用教程
# CRAWLAB_SETTING_RUNONMASTER: "N" # whether to run on master node 是否在主节点上运行任务
# CRAWLAB_SETTING_DEMOSPIDERS: "Y" # whether to init demo spiders 是否使用Demo爬虫
# CRAWLAB_NOTIFICATION_MAIL_SERVER: smtp.exmaple.com # STMP server address STMP 服务器地址
# CRAWLAB_NOTIFICATION_MAIL_PORT: 465 # STMP server port STMP 服务器端口
# CRAWLAB_NOTIFICATION_MAIL_SENDEREMAIL: admin@exmaple.com # sender email 发送者邮箱

16
docker_init.sh
View File

@@ -22,18 +22,12 @@ fi
# start nginx
service nginx start
# install languages: Node.js
if [ "${CRAWLAB_SERVER_LANG_NODE}" = "Y" ];
# install languages
if [ "${CRAWLAB_SERVER_LANG_NODE}" = "Y" ] || [ "${CRAWLAB_SERVER_LANG_JAVA}" = "Y" ];
then
echo "installing node.js"
/bin/sh /app/backend/scripts/install-nodejs.sh >> /var/log/install-nodejs.sh.log 2>&1 &
fi
# install languages: Java
if [ "${CRAWLAB_SERVER_LANG_JAVA}" = "Y" ];
then
echo "installing java"
/bin/sh /app/backend/scripts/install-java.sh >> /var/log/install-java.sh.log 2>&1 &
echo "installing languages"
echo "you can view log at /var/log/install.sh.log"
/bin/sh /app/backend/scripts/install.sh >> /var/log/install.sh.log 2>&1 &
fi
# generate ssh

2
frontend/package.json
View File

@@ -1,6 +1,6 @@
{
"name": "crawlab",
"version": "0.4.7",
"version": "0.4.8",
"private": true,
"scripts": {
"serve": "vue-cli-service serve --ip=0.0.0.0 --mode=development",

4
frontend/public/font-awesome.min.css vendored Normal file
View File

File diff suppressed because one or more lines are too long

25
frontend/public/index.html
View File

@@ -6,12 +6,14 @@
<meta name="renderer" content="webkit">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
<link rel="icon" href="/favicon.ico" type="image/x-icon">
<link rel="stylesheet" href="/font-awesome.min.css" type="text/css">
<!-- Place this tag in your head or just before your close body tag. -->
<script async defer src="https://buttons.github.io/buttons.js"></script>
<style>
#loading-placeholder {
position: fixed;
background: white;
z-index: -1;
top: 0;
left: 0;
@@ -70,13 +72,31 @@
animation-delay: calc(1s / 7 * 6 / 2);
}
#loading-placeholder .sub-title-wrapper {
display: flex;
align-items: center;
justify-content: center;
margin-bottom: 10px;
height: 28px;
}
#loading-placeholder .sub-title-wrapper .sub-title {
font-size: 18px;
font-weight: 300;
font-family: "Verdana", serif;
font-style: italic;
color: #67C23A;
/*color: #E6A23C;*/
/*color: #F56C6C;*/
}
#loading-placeholder .loading-text {
text-align: center;
font-weight: bolder;
font-family: "Verdana", serif;
font-style: italic;
color: #889aa4;
font-size: 18px;
font-size: 14px;
animation: blink-loading 2s ease-in infinite;
}
@@ -129,6 +149,9 @@
<span>B</span>
</h3>
</div>
<div class="sub-title-wrapper">
<span class="sub-title"><i class="fa fa-check-square-o"></i> Easy crawling</span>
</div>
<div class="loading-text">
Loading...
</div>

10
frontend/src/components/Common/CrawlConfirmDialog.vue
View File

@@ -29,7 +29,7 @@
v-for="op in nodeList"
:key="op._id"
:value="op._id"
:disabled="op.status !== 'online'"
:disabled="isNodeDisabled(op)"
:label="op.name"
/>
</el-select>
@@ -139,6 +139,9 @@ export default {
...mapState('spider', [
'spiderForm'
]),
...mapState('setting', [
'setting'
]),
isConfirmDisabled () {
if (this.isLoading) return true
if (!this.isAllowDisclaimer) return true
@@ -291,6 +294,11 @@ export default {
onParametersConfirm (value) {
this.form.param = value
this.isParametersVisible = false
},
isNodeDisabled (node) {
if (node.status !== 'online') return true
if (node.is_master && this.setting.run_on_master === 'N') return true
return false
}
}
}

160
frontend/src/components/Node/NodeInstallation.vue
View File

@@ -38,65 +38,98 @@
<el-tabs v-model="activeTab" @tab-click="onTabChange">
<el-tab-pane v-for="lang in langList" :key="lang.name" :label="lang.name" :name="lang.executable_name"/>
</el-tabs>
<template v-if="activeLang.installed">
<el-table
height="calc(100vh - 280px)"
:data="computedDepList"
:empty-text="depName ? $t('No Data') : $t('Please search dependencies')"
v-loading="loading"
border
>
<el-table-column
:label="$t('Name')"
prop="name"
width="180"
/>
<el-table-column
:label="!isShowInstalled ? $t('Latest Version') : $t('Version')"
prop="version"
width="100"
/>
<el-table-column
v-if="!isShowInstalled"
:label="$t('Description')"
prop="description"
/>
<el-table-column
:label="$t('Action')"
<template v-if="activeLang.install_status === 'installed'">
<template v-if="!['python', 'node'].includes(activeLang.executable_name)">
<div class="install-wrapper">
<el-button
icon="el-icon-check"
disabled
type="success"
>
{{$t('Installed')}}
</el-button>
</div>
</template>
<template v-else>
<el-table
height="calc(100vh - 280px)"
:data="computedDepList"
:empty-text="depName ? $t('No Data') : $t('Please search dependencies')"
v-loading="loading"
border
>
<template slot-scope="scope">
<el-button
v-if="!scope.row.installed"
v-loading="getDepLoading(scope.row)"
:disabled="getDepLoading(scope.row)"
size="mini"
type="primary"
@click="onClickInstallDep(scope.row)"
>
{{$t('Install')}}
</el-button>
<el-button
v-else
v-loading="getDepLoading(scope.row)"
:disabled="getDepLoading(scope.row)"
size="mini"
type="danger"
@click="onClickUninstallDep(scope.row)"
>
{{$t('Uninstall')}}
</el-button>
</template>
</el-table-column>
</el-table>
<el-table-column
:label="$t('Name')"
prop="name"
width="180"
/>
<el-table-column
:label="!isShowInstalled ? $t('Latest Version') : $t('Version')"
prop="version"
width="100"
/>
<el-table-column
v-if="!isShowInstalled"
:label="$t('Description')"
prop="description"
/>
<el-table-column
:label="$t('Action')"
>
<template slot-scope="scope">
<el-button
v-if="!scope.row.installed"
:icon="getDepLoading(scope.row) ? 'el-icon-loading' : ''"
:disabled="getDepLoading(scope.row)"
size="mini"
type="primary"
@click="onClickInstallDep(scope.row)"
>
{{$t('Install')}}
</el-button>
<el-button
v-else
:icon="getDepLoading(scope.row) ? 'el-icon-loading' : ''"
:disabled="getDepLoading(scope.row)"
size="mini"
type="danger"
@click="onClickUninstallDep(scope.row)"
>
{{$t('Uninstall')}}
</el-button>
</template>
</el-table-column>
</el-table>
</template>
</template>
<template v-else>
<template v-else-if="activeLang.install_status === 'installing'">
<div class="install-wrapper">
<h3>{{activeLang.name + $t(' is not installed, do you want to install it?')}}</h3>
<el-button
v-loading="isLoadingInstallLang"
:disabled="isLoadingInstallLang"
icon="el-icon-loading"
disabled
type="warning"
>
{{$t('Installing')}}
</el-button>
</div>
</template>
<template v-else-if="activeLang.install_status === 'installing-other'">
<div class="install-wrapper">
<el-button
loading="el-icon-close"
disabled
type="warning"
>
{{$t('Other language installing')}}
</el-button>
</div>
</template>
<template v-else-if="activeLang.install_status === 'not-installed'">
<div class="install-wrapper">
<h4>{{$t('This language is not installed yet.')}}</h4>
<el-button
icon="el-icon-check"
type="primary"
style="width: 240px;font-weight: bolder;font-size: 18px"
@click="onClickInstallLang"
>
{{$t('Install')}}
@@ -175,6 +208,9 @@ export default {
}
},
async getInstalledDepList () {
if (this.activeLang.install_status !== 'installed') return
if (!['Python', 'Node.js'].includes(this.activeLang.name)) return
this.loading = true
this.installedDepList = []
const res = await this.$request.get(`/nodes/${this.nodeForm._id}/deps/installed`, {
@@ -295,19 +331,17 @@ export default {
const res = await this.$request.get(`/nodes/${id}/langs`)
this.langList = res.data.data
this.activeTab = this.langList[0].executable_name || ''
await this.getInstalledDepList()
setTimeout(() => {
this.getInstalledDepList()
}, 100)
}
}
</script>
<style scoped>
.node-installation >>> .el-button .el-loading-spinner {
margin-top: -13px;
height: 28px;
}
.node-installation >>> .el-button .el-loading-spinner .circular {
width: 28px;
height: 28px;
.node-installation >>> .install-wrapper .el-button {
min-width: 240px;
font-weight: bolder;
font-size: 18px
}
</style>

436
frontend/src/components/Node/NodeInstallationMatrix.vue
View File

@@ -1,9 +1,214 @@
<template>
<div class="node-installation-matrix">
<el-tabs v-model="activeTabName">
<el-tab-pane :label="$t('Languages')" name="lang">
<div class="lang-table">
<el-table
class="table"
:data="nodeList"
:header-cell-style="{background:'rgb(48, 65, 86)',color:'white',height:'50px'}"
border
@row-click="onLangTableRowClick"
>
<el-table-column
:label="$t('Node')"
width="240px"
prop="name"
fixed
/>
<el-table-column
:label="$t('nodeList.type')"
width="120px"
fixed
>
<template slot-scope="scope">
<el-tag type="primary" v-if="scope.row.is_master">{{$t('Master')}}</el-tag>
<el-tag type="warning" v-else>{{$t('Worker')}}</el-tag>
</template>
</el-table-column>
<el-table-column
:label="$t('Status')"
width="120px"
fixed
>
<template slot-scope="scope">
<el-tag type="info" v-if="scope.row.status === 'offline'">{{$t('Offline')}}</el-tag>
<el-tag type="success" v-else-if="scope.row.status === 'online'">{{$t('Online')}}</el-tag>
<el-tag type="danger" v-else>{{$t('Unavailable')}}</el-tag>
</template>
</el-table-column>
<el-table-column
v-for="l in langs"
:key="l.name"
:label="l.label"
width="220px"
>
<template slot="header" slot-scope="scope">
<div class="header-with-action">
<span>{{scope.column.label}}</span>
<el-button type="primary" size="mini" @click="onInstallLangAll(scope.column.label, $event)">
{{$t('Install')}}
</el-button>
</div>
</template>
<template slot-scope="scope">
<template v-if="getLangInstallStatus(scope.row._id, l.name) === 'installed'">
<el-tag type="success">
<i class="el-icon-check"></i>
{{$t('Installed')}}
</el-tag>
</template>
<template v-else-if="getLangInstallStatus(scope.row._id, l.name) === 'installing'">
<el-tag type="warning">
<i class="el-icon-loading"></i>
{{$t('Installing')}}
</el-tag>
</template>
<template
v-else-if="['installing-other', 'not-installed'].includes(getLangInstallStatus(scope.row._id, l.name))"
>
<div class="cell-with-action">
<el-tag type="danger">
<i class="el-icon-error"></i>
{{$t('Not Installed')}}
</el-tag>
<el-button type="primary" size="mini" @click="onInstallLang(scope.row._id, scope.column.label, $event)">
{{$t('Install')}}
</el-button>
</div>
</template>
<template v-else-if="getLangInstallStatus(scope.row._id, l.name) === 'na'">
<el-tag type="info">
<i class="el-icon-question"></i>
{{$t('N/A')}}
</el-tag>
</template>
</template>
</el-table-column>
</el-table>
</div>
</el-tab-pane>
<el-tab-pane :label="$t('Dependencies')" name="dep">
<el-form class="search-form" inline>
<el-form-item>
<el-input
v-model="depName"
style="width: 240px"
:placeholder="$t('Search Dependencies')"
/>
</el-form-item>
<el-form-item>
<el-button size="small"
icon="el-icon-search"
type="success"
@click="onSearch"
>
{{$t('Search')}}
</el-button>
</el-form-item>
<el-form-item>
<el-checkbox v-model="isShowInstalled" :label="$t('Show installed')" @change="onIsShowInstalledChange"/>
</el-form-item>
</el-form>
<el-tabs v-model="activeLang">
<el-tab-pane
v-for="l in langsWithDeps"
:key="l.name"
:name="l.name"
:label="l.label"
/>
</el-tabs>
<el-table
v-loading="isDepsLoading"
class="table"
height="calc(100vh - 320px)"
:data="computedDepsSet"
:header-cell-style="{background:'rgb(48, 65, 86)',color:'white',height:'50px'}"
border
>
<el-table-column
:label="$t('Dependency')"
prop="name"
width="180px"
fixed
/>
<el-table-column
v-if="false"
:label="$t('Install on All Nodes')"
width="120px"
align="center"
fixed
>
<template>
<el-button
size="mini"
type="primary"
>
{{$t('Install')}}
</el-button>
</template>
</el-table-column>
<el-table-column
v-for="n in activeNodes"
:key="n._id"
:label="n.name"
width="220px"
align="center"
>
<template slot="header" slot-scope="scope">
{{scope.column.label}}
</template>
<template slot-scope="scope">
<div
v-if="getDepStatus(n, scope.row) === 'installed'"
class="cell-with-action"
>
<el-tag type="success">
{{$t('Installed')}}
</el-tag>
<el-button
size="mini"
type="danger"
@click="uninstallDep(n, scope.row)"
>
{{$t('Uninstall')}}
</el-button>
</div>
<div
v-else-if="getDepStatus(n, scope.row) === 'installing'"
class="cell-with-action"
>
<el-tag type="warning">
<i class="el-icon-loading"></i>
{{$t('Installing')}}
</el-tag>
</div>
<div
v-else-if="getDepStatus(n, scope.row) === 'uninstalled'"
class="cell-with-action"
>
<el-tag type="danger">
{{$t('Not Installed')}}
</el-tag>
<el-button
size="mini"
type="primary"
@click="installDep(n, scope.row)"
>
{{$t('Install')}}
</el-button>
</div>
</template>
</el-table-column>
</el-table>
</el-tab-pane>
</el-tabs>
</div>
</template>
<script>
import { mapState } from 'vuex'
export default {
name: 'NodeInstallationMatrix',
props: {
@@ -11,10 +216,241 @@ export default {
type: String,
default: ''
}
},
data () {
return {
langs: [
{ label: 'Python', name: 'python', hasDeps: true },
{ label: 'Node.js', name: 'node', hasDeps: true },
{ label: 'Java', name: 'java', hasDeps: false },
{ label: '.Net Core', name: 'dotnet', hasDeps: false },
{ label: 'PHP', name: 'php', hasDeps: false }
],
langsDataDict: {},
handle: undefined,
activeTabName: 'lang',
depsDataDict: {},
depsSet: new Set(),
activeLang: 'python',
isDepsLoading: false,
depName: '',
isShowInstalled: true,
depList: []
}
},
computed: {
...mapState('node', [
'nodeList'
]),
activeNodes () {
return this.nodeList.filter(d => d.status === 'online')
},
computedDepsSet () {
return Array.from(this.depsSet).map(d => {
return {
name: d
}
})
},
langsWithDeps () {
return this.langs.filter(l => l.hasDeps)
}
},
watch: {
activeLang () {
this.getDepsData()
}
},
methods: {
async getLangsData () {
await Promise.all(this.nodeList.map(async n => {
if (n.status !== 'online') return
const res = await this.$request.get(`/nodes/${n._id}/langs`)
res.data.data.forEach(l => {
const key = n._id + '|' + l.executable_name
this.$set(this.langsDataDict, key, l)
})
}))
},
async getDepsData () {
this.isDepsLoading = true
this.depsDataDict = {}
this.depsSet = new Set()
const depsSet = new Set()
await Promise.all(this.nodeList.map(async n => {
if (n.status !== 'online') return
const res = await this.$request.get(`/nodes/${n._id}/deps/installed`, { lang: this.activeLang })
res.data.data.forEach(d => {
depsSet.add(d.name)
const key = n._id + '|' + d.name
this.$set(this.depsDataDict, key, 'installed')
})
}))
this.depsSet = depsSet
this.isDepsLoading = false
},
getLang (nodeId, langName) {
const key = nodeId + '|' + langName
return this.langsDataDict[key]
},
getLangInstallStatus (nodeId, langName) {
const lang = this.getLang(nodeId, langName)
if (!lang || !lang.install_status) return 'na'
return lang.install_status
},
getLangFromLabel (label) {
for (let i = 0; i < this.langs.length; i++) {
const lang = this.langs[i]
if (lang.label === label) {
return lang
}
}
},
async onInstallLang (nodeId, langLabel, ev) {
if (ev) {
ev.stopPropagation()
}
const lang = this.getLangFromLabel(langLabel)
this.$request.post(`/nodes/${nodeId}/langs/install`, {
lang: lang.name
})
const key = nodeId + '|' + lang.name
this.$set(this.langsDataDict[key], 'install_status', 'installing')
setTimeout(() => {
this.getLangsData()
}, 1000)
this.$st.sendEv('节点列表', '安装', '安装语言')
},
async onInstallLangAll (langLabel, ev) {
ev.stopPropagation()
this.nodeList
.filter(n => {
if (n.status !== 'online') return false
const lang = this.getLangFromLabel(langLabel)
const key = n._id + '|' + lang.name
if (!this.langsDataDict[key]) return false
if (['installing', 'installed'].includes(this.langsDataDict[key].install_status)) return false
return true
})
.forEach(n => {
this.onInstallLang(n._id, langLabel, ev)
})
setTimeout(() => {
this.getLangsData()
}, 1000)
this.$st.sendEv('节点列表', '安装', '安装语言-所有节点')
},
onLangTableRowClick (row) {
this.$router.push(`/nodes/${row._id}`)
this.$st.sendEv('节点列表', '安装', '查看节点详情')
},
getDepStatus (node, dep) {
const key = node._id + '|' + dep.name
if (!this.depsDataDict[key]) {
return 'uninstalled'
} else {
return this.depsDataDict[key]
}
},
async installDep (node, dep) {
const key = node._id + '|' + dep.name
this.$set(this.depsDataDict, key, 'installing')
const data = await this.$request.post(`/nodes/${node._id}/deps/install`, {
lang: this.activeLang,
dep_name: dep.name
})
if (!data || data.error) {
this.$notify.error({
title: this.$t('Installing dependency failed'),
message: this.$t('The dependency installation is unsuccessful: ') + dep.name
})
this.$set(this.depsDataDict, key, 'uninstalled')
} else {
this.$notify.success({
title: this.$t('Installing dependency successful'),
message: this.$t('You have successfully installed a dependency: ') + dep.name
})
this.$set(this.depsDataDict, key, 'installed')
}
this.$st.sendEv('节点列表', '安装', '安装依赖')
},
async uninstallDep (node, dep) {
const key = node._id + '|' + dep.name
this.$set(this.depsDataDict, key, 'installing')
const data = await this.$request.post(`/nodes/${node._id}/deps/uninstall`, {
lang: this.activeLang,
dep_name: dep.name
})
if (!data || data.error) {
this.$notify.error({
title: this.$t('Uninstalling dependency failed'),
message: this.$t('The dependency uninstallation is unsuccessful: ') + dep.name
})
this.$set(this.depsDataDict, key, 'installed')
} else {
this.$notify.success({
title: this.$t('Uninstalling dependency successful'),
message: this.$t('You have successfully uninstalled a dependency: ') + dep.name
})
this.$set(this.depsDataDict, key, 'uninstalled')
}
this.$st.sendEv('节点列表', '安装', '卸载依赖')
},
onSearch () {
this.isShowInstalled = false
this.getDepList()
this.$st.sendEv('节点列表', '安装', '搜索依赖')
},
async getDepList () {
const masterNode = this.nodeList.filter(n => n.is_master)[0]
this.depsSet = []
this.isDepsLoading = true
const res = await this.$request.get(`/nodes/${masterNode._id}/deps`, {
lang: this.activeLang,
dep_name: this.depName
})
this.isDepsLoading = false
this.depsSet = new Set(res.data.data.map(d => d.name))
},
onIsShowInstalledChange (val) {
if (val) {
this.getDepsData()
} else {
this.depsSet = []
}
this.$st.sendEv('节点列表', '安装', '点击查看已安装')
}
},
async created () {
setTimeout(() => {
this.getLangsData()
this.getDepsData()
}, 1000)
this.handle = setInterval(() => {
this.getLangsData()
}, 10000)
},
destroyed () {
clearInterval(this.handle)
}
}
</script>
<style scoped>
.table {
margin-top: 20px;
border-radius: 5px;
}
.el-table tr {
cursor: pointer;
}
.el-table .header-with-action,
.el-table .cell-with-action {
display: flex;
justify-content: space-between;
align-items: center;
}
</style>

436
frontend/src/components/Settings/GitSettings.vue
View File

@@ -1,159 +1,237 @@
<template>
<div class="git-settings">
<h3 class="title">{{$t('Git Settings')}}</h3>
<el-form
class="git-settings-form"
label-width="150px"
:model="spiderForm"
ref="git-settings-form"
>
<el-form-item
:label="$t('Git URL')"
prop="git_url"
required
<el-tabs
v-model="activeTabName"
class="git-settings"
>
<el-tab-pane :label="$t('Settings')" name="settings">
<el-form
class="git-settings-form"
label-width="150px"
:model="spiderForm"
ref="git-settings-form"
>
<el-input
v-model="spiderForm.git_url"
:placeholder="$t('Git URL')"
@blur="onGitUrlChange"
<el-form-item
:label="$t('Git URL')"
prop="git_url"
required
>
</el-input>
</el-form-item>
<el-form-item
:label="$t('Has Credential')"
prop="git_has_credential"
>
<el-switch
v-model="spiderForm.git_has_credential"
size="small"
active-color="#67C23A"
/>
</el-form-item>
<el-form-item
v-if="spiderForm.git_has_credential"
:label="$t('Git Username')"
prop="git_username"
>
<el-input
v-model="spiderForm.git_username"
:placeholder="$t('Git Username')"
<el-input
v-model="spiderForm.git_url"
:placeholder="$t('Git URL')"
@blur="onGitUrlChange"
>
</el-input>
</el-form-item>
<el-form-item
:label="$t('Has Credential')"
prop="git_has_credential"
>
</el-input>
</el-form-item>
<el-form-item
v-if="spiderForm.git_has_credential"
:label="$t('Git Password')"
prop="git_password"
>
<el-input
v-model="spiderForm.git_password"
:placeholder="$t('Git Password')"
type="password"
>
</el-input>
</el-form-item>
<el-form-item
:label="$t('Git Branch')"
prop="git_branch"
required
>
<el-select
v-model="spiderForm.git_branch"
:placeholder="$t('Git Branch')"
:disabled="!spiderForm.git_url || isGitBranchesLoading"
>
<el-option
v-for="op in gitBranches"
:key="op"
:value="op"
:label="op"
<el-switch
v-model="spiderForm.git_has_credential"
size="small"
active-color="#67C23A"
/>
</el-select>
</el-form-item>
<el-form-item
:label="$t('Auto Sync')"
prop="git_auto_sync"
>
<el-switch
v-model="spiderForm.git_auto_sync"
size="small"
active-color="#67C23A"
/>
</el-form-item>
<el-form-item
v-if="spiderForm.git_auto_sync"
:label="$t('Sync Frequency')"
prop="git_sync_frequency"
required
>
<el-select
v-model="spiderForm.git_sync_frequency"
:placeholder="$t('Sync Frequency')"
</el-form-item>
<el-form-item
v-if="spiderForm.git_has_credential"
:label="$t('Git Username')"
prop="git_username"
>
<el-option
v-for="op in syncFrequencies"
:key="op.value"
:value="op.value"
:label="op.label"
<el-input
v-model="spiderForm.git_username"
:placeholder="$t('Git Username')"
>
</el-input>
</el-form-item>
<el-form-item
v-if="spiderForm.git_has_credential"
:label="$t('Git Password')"
prop="git_password"
>
<el-input
v-model="spiderForm.git_password"
:placeholder="$t('Git Password')"
type="password"
>
</el-input>
</el-form-item>
<el-form-item
:label="$t('Git Branch')"
prop="git_branch"
required
>
<el-select
v-model="spiderForm.git_branch"
:placeholder="$t('Git Branch')"
:disabled="!spiderForm.git_url || isGitBranchesLoading"
>
<el-option
v-for="op in gitBranches"
:key="op"
:value="op"
:label="op"
/>
</el-select>
</el-form-item>
<el-form-item
:label="$t('Auto Sync')"
prop="git_auto_sync"
>
<el-switch
v-model="spiderForm.git_auto_sync"
size="small"
active-color="#67C23A"
/>
</el-select>
</el-form-item>
<el-form-item
v-if="spiderForm.git_sync_error"
:label="$t('Error Message')"
prop="git_git_sync_error"
>
<el-alert
type="error"
:closable="false"
</el-form-item>
<el-form-item
v-if="spiderForm.git_auto_sync"
:label="$t('Sync Frequency')"
prop="git_sync_frequency"
required
>
{{spiderForm.git_sync_error}}
</el-alert>
</el-form-item>
<el-form-item
v-if="sshPublicKey"
:label="$t('SSH Public Key')"
>
<el-alert
type="info"
:closable="false"
<el-select
v-model="spiderForm.git_sync_frequency"
:placeholder="$t('Sync Frequency')"
>
<el-option
v-for="op in syncFrequencies"
:key="op.value"
:value="op.value"
:label="op.label"
/>
</el-select>
</el-form-item>
<el-form-item
v-if="spiderForm.git_sync_error"
:label="$t('Error Message')"
prop="git_git_sync_error"
>
{{sshPublicKey}}
</el-alert>
<span class="copy" @click="copySshPublicKey">
<el-alert
type="error"
:closable="false"
>
{{spiderForm.git_sync_error}}
</el-alert>
</el-form-item>
<el-form-item
v-if="sshPublicKey"
:label="$t('SSH Public Key')"
>
<el-alert
type="info"
:closable="false"
>
{{sshPublicKey}}
</el-alert>
<span class="copy" @click="copySshPublicKey">
<i class="el-icon-copy-document"></i>
{{$t('Copy')}}
</span>
<input id="ssh-public-key" v-model="sshPublicKey" v-show="true">
</el-form-item>
</el-form>
<div class="action-wrapper">
<el-button
size="small"
type="warning"
:disabled="isGitResetLoading"
:icon="isGitResetLoading ? 'el-icon-loading' : 'el-icon-refresh-left'"
@click="onReset"
<input id="ssh-public-key" v-model="sshPublicKey" v-show="true">
</el-form-item>
</el-form>
<div class="action-wrapper">
<el-button
size="small"
type="warning"
:disabled="isGitResetLoading"
:icon="isGitResetLoading ? 'el-icon-loading' : 'el-icon-refresh-left'"
@click="onReset"
>
{{$t('Reset')}}
</el-button>
<el-button
size="small"
type="danger"
:icon="isGitSyncLoading ? 'el-icon-loading' : 'el-icon-refresh'"
:disabled="!spiderForm.git_url || isGitSyncLoading"
@click="onSync"
>
{{$t('Sync')}}
</el-button>
<el-button size="small" type="success" @click="onSave" icon="el-icon-check">
{{$t('Save')}}
</el-button>
</div>
</el-tab-pane>
<el-tab-pane label="Log" name="log">
<el-timeline
class="log"
>
{{$t('Reset')}}
</el-button>
<el-button
size="small"
type="danger"
:icon="isGitSyncLoading ? 'el-icon-loading' : 'el-icon-refresh'"
:disabled="!spiderForm.git_url || isGitSyncLoading"
@click="onSync"
>
{{$t('Sync')}}
</el-button>
<el-button size="small" type="success" @click="onSave" icon="el-icon-check">
{{$t('Save')}}
</el-button>
</div>
</div>
<el-timeline-item
v-for="c in commits"
:key="c.hash"
:timestamp="c.ts"
:type="getCommitType(c)"
>
<div class="commit">
<div class="row">
<div class="message">
{{c.message}}
</div>
<div class="author">
{{c.author}} ({{c.email}})
</div>
</div>
<div class="row" style="margin-top: 10px">
<div class="tags">
<el-tag
v-if="c.is_head"
type="primary"
size="mini"
>
<i class="fa fa-tag"></i>
HEAD
</el-tag>
<el-tag
v-for="b in c.branches"
:key="b.name"
:type="b.label === 'master' ? 'danger' : 'warning'"
size="mini"
>
<i class="fa fa-tag"></i>
{{b.label}}
</el-tag>
<el-tag
v-for="b in c.remote_branches"
:key="b.name"
type="info"
size="mini"
>
<i class="fa fa-tag"></i>
{{b.label}}
</el-tag>
<el-tag
v-for="t in c.tags"
:key="t.name"
type="success"
size="mini"
>
<i class="fa fa-tag"></i>
{{t.label}}
</el-tag>
</div>
<div class="actions">
<el-button
v-if="!c.is_head"
type="danger"
:icon="isGitCheckoutLoading ? 'el-icon-loading' : 'el-icon-position'"
size="mini"
@click="checkout(c)"
>
Checkout
</el-button>
</div>
</div>
</div>
</el-timeline-item>
</el-timeline>
</el-tab-pane>
</el-tabs>
</template>
<script>
import dayjs from 'dayjs'
import {
mapState
} from 'vuex'
@@ -166,6 +244,7 @@ export default {
isGitBranchesLoading: false,
isGitSyncLoading: false,
isGitResetLoading: false,
isGitCheckoutLoading: false,
syncFrequencies: [
{ label: '1m', value: '0 * * * * *' },
{ label: '5m', value: '0 0/5 * * * *' },
@@ -176,7 +255,9 @@ export default {
{ label: '12h', value: '0 0 0/12 * * *' },
{ label: '1d', value: '0 0 0 0 * *' }
],
sshPublicKey: ''
sshPublicKey: '',
activeTabName: 'settings',
commits: []
}
},
computed: {
@@ -217,6 +298,7 @@ export default {
}
} finally {
this.isGitSyncLoading = false
await this.updateGit()
await this.$store.dispatch('spider/getSpiderData', this.$route.params.id)
}
this.$st.sendEv('爬虫详情', 'Git 设置', '同步')
@@ -240,6 +322,7 @@ export default {
}
} finally {
this.isGitResetLoading = false
await this.updateGit()
}
})
this.$st.sendEv('爬虫详情', 'Git 设置', '点击重置')
@@ -255,13 +338,53 @@ export default {
document.execCommand('copy')
this.$message.success(this.$t('SSH Public Key is copied to the clipboard'))
this.$st.sendEv('爬虫详情', 'Git 设置', '拷贝 SSH 公钥')
},
async getCommits () {
const res = await this.$request.get('/git/commits', { spider_id: this.spiderForm._id })
this.commits = res.data.data.map(d => {
d.ts = dayjs(d.ts).format('YYYY-MM-DD HH:mm:ss')
return d
})
},
async checkout (c) {
this.isGitCheckoutLoading = true
try {
const res = await this.$request.post('/git/checkout', { spider_id: this.spiderForm._id, hash: c.hash })
if (!res.data.error) {
this.$message.success(this.$t('Checkout success'))
}
} finally {
this.isGitCheckoutLoading = false
await this.getCommits()
}
this.$st.sendEv('爬虫详情', 'Git', 'Checkout')
},
async updateGit () {
this.getCommits()
},
getCommitType (c) {
if (c.is_head) return 'primary'
if (c.branches && c.branches.length) {
if (c.branches.map(d => d.label).includes('master')) {
return 'danger'
} else {
return 'warning'
}
}
if (c.tags && c.tags.length) {
return 'success'
}
if (c.remote_branches && c.remote_branches.length) {
return 'info'
}
}
},
async created () {
if (this.spiderForm.git_url) {
this.onGitUrlChange()
}
await this.getSshPublicKey()
this.getSshPublicKey()
this.getCommits()
}
}
</script>
@@ -318,4 +441,33 @@ export default {
text-align: right;
margin-top: 10px;
}
.git-settings .log {
height: calc(100vh - 280px);
overflow: auto;
}
.git-settings .log .commit {
border-top: 1px solid rgb(244, 244, 245);
padding: 10px 0;
}
.git-settings .log .commit .row {
display: flex;
justify-content: space-between;
}
.git-settings .log .el-timeline-item {
/*cursor: pointer;*/
}
.git-settings .log .commit .row .tags .el-tag {
margin-right: 5px;
}
.git-settings .log .commit .row .actions {
right: 0;
bottom: 5px;
position: absolute;
}
</style>

9
frontend/src/i18n/zh.js
View File

@@ -346,6 +346,15 @@ export default {
'Executables': '执行文件',
'Latest Version': '最新版本',
'Version': '版本',
'Installed': '已安装',
'Not Installed': '未安装',
'Installing': '正在安装',
'Install All': '安装全部',
'Other language installing': '其他语言正在安装',
'This language is not installed yet.': '语言还未安装',
'Languages': '语言',
'Dependencies': '依赖',
'Install on All Nodes': '安装在所有节点',
// 弹出框
'Notification': '提示',

19
frontend/src/views/login/index.vue
View File

@@ -71,8 +71,12 @@
</div>
<div class="lang">
<span @click="setLang('zh')" :class="lang==='zh'?'active':''">中文</span>
|
<span @click="setLang('en')" :class="lang==='en'?'active':''">English</span>
</div>
<div class="documentation">
<a href="http://docs.crawlab.cn" target="_blank">{{$t('Documentation')}}</a>
</div>
<div v-if="isShowMobileWarning" class="mobile-warning">
<el-alert type="error" :closable="false">
{{$t('You are running on a mobile device, which is not optimized yet. Please try with a laptop or desktop.')}}
@@ -449,16 +453,17 @@ const initCanvas = () => {
.lang {
margin-top: 20px;
text-align: center;
color: #666;
span {
cursor: pointer;
margin: 10px;
color: #666;
font-size: 14px;
}
span.active {
font-weight: 600;
text-decoration: underline;
}
span:hover {
@@ -466,6 +471,18 @@ const initCanvas = () => {
}
}
.documentation {
margin-top: 20px;
text-align: center;
font-size: 14px;
color: #409eff;
font-weight: bolder;
&:hover {
text-decoration: underline;
}
}
.mobile-warning {
margin-top: 20px;
}

6
frontend/src/views/node/NodeList.vue
View File

@@ -139,12 +139,12 @@
</div>
<!--./table list-->
</el-tab-pane>
<el-tab-pane :label="$t('Network')">
<node-network :active-tab="activeTab"/>
</el-tab-pane>
<el-tab-pane :label="$t('Installation')">
<node-installation-matrix :active-tab="activeTab"/>
</el-tab-pane>
<el-tab-pane :label="$t('Network')">
<node-network :active-tab="activeTab"/>
</el-tab-pane>
</el-tabs>
</div>
</template>

2
frontend/src/views/spider/SpiderDetail.vue
View File

@@ -22,7 +22,7 @@
<el-tab-pane :label="$t('Overview')" name="overview">
<spider-overview/>
</el-tab-pane>
<el-tab-pane v-if="isGit" :label="$t('Git Settings')" name="git-settings">
<el-tab-pane v-if="isGit" :label="$t('Git')" name="git-settings">
<git-settings/>
</el-tab-pane>
<el-tab-pane v-if="isScrapy" :label="$t('Scrapy Settings')" name="scrapy-settings">

25
k8s/crawlab-master.yaml
View File

@@ -2,7 +2,7 @@ apiVersion: v1
kind: Service
metadata:
name: crawlab
namespace: crawlab
namespace: crawlab-master
spec:
ports:
- port: 8080
@@ -13,13 +13,12 @@ spec:
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
kind: StatefulSet
metadata:
name: crawlab-master
namespace: crawlab
namespace: crawlab-master
spec:
strategy:
type: Recreate
serviceName: crawlab-master
selector:
matchLabels:
app: crawlab-master
@@ -39,10 +38,16 @@ spec:
value: "mongo"
- name: CRAWLAB_REDIS_ADDRESS
value: "redis"
# - name: CRAWLAB_SERVER_LANG_NODE
# value: "Y"
# - name: CRAWLAB_SERVER_LANG_JAVA
# value: "Y"
- name: CRAWLAB_SETTING_ALLOWREGISTER
value: "Y"
- name: CRAWLAB_SERVER_LANG_NODE
value: "N"
- name: CRAWLAB_SERVER_LANG_JAVA
value: "N"
- name: CRAWLAB_SERVER_LANG_DOTNET
value: "N"
- name: CRAWLAB_SERVER_REGISTER_TYPE
value: "hostname"
ports:
- containerPort: 8080
name: crawlab
name: crawlab

21
k8s/crawlab-worker.yaml
View File

@@ -1,12 +1,11 @@
apiVersion: apps/v1
kind: Deployment
kind: StatefulSet
metadata:
name: crawlab-worker
namespace: crawlab
namespace: crawlab-master
spec:
replicas: 4
strategy:
type: Recreate
serviceName: crawlab-worker
replicas: 2
selector:
matchLabels:
app: crawlab-worker
@@ -26,7 +25,11 @@ spec:
value: "mongo"
- name: CRAWLAB_REDIS_ADDRESS
value: "redis"
# - name: CRAWLAB_SERVER_LANG_NODE
# value: "Y"
# - name: CRAWLAB_SERVER_LANG_JAVA
# value: "Y"
- name: CRAWLAB_SERVER_LANG_NODE
value: "Y"
- name: CRAWLAB_SERVER_LANG_JAVA
value: "Y"
- name: CRAWLAB_SERVER_LANG_DOTNET
value: "Y"
- name: CRAWLAB_SERVER_REGISTER_TYPE
value: "hostname"