git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-23 17:31:11 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #490 from crawlab-team/release

Browse Source 
Release
This commit is contained in:
Marvin Zhang
2020-01-30 11:21:39 +08:00
committed by GitHub
parent e37153ddf2 f4ae4bead0
commit d845989923
19 changed files with 457 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
README-zh.md
View File

@@ -242,13 +242,12 @@ save_item(result)
Crawlab使用起来很方便，也很通用，可以适用于几乎任何主流语言和框架。它还有一个精美的前端界面，让用户可以方便的管理和运行爬虫。
|框架 | 类型 | 分布式 | 前端 | 依赖于Scrapyd |
|:---:|:---:|:---:|:---:|:---:|
| [Crawlab](https://github.com/crawlab-team/crawlab) | 管理平台 | Y | Y | N
| [ScrapydWeb](https://github.com/my8100/scrapydweb) | 管理平台 | Y | Y | Y
| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | 管理平台 | Y | Y | Y
| [Gerapy](https://github.com/Gerapy/Gerapy) | 管理平台 | Y | Y | Y
| [Scrapyd](https://github.com/scrapy/scrapyd) | 网络服务 | Y | N | N/A
|框架 | 技术 | 优点 | 缺点 | Github 统计数据 |
|:---|:---|:---|-----| :---- |
| [Crawlab](https://github.com/crawlab-team/crawlab) | Golang + Vue|不局限于 scrapy，可以运行任何语言和框架的爬虫，精美的 UI 界面，天然支持分布式爬虫，支持节点管理、爬虫管理、任务管理、定时任务、结果导出、数据统计、消息通知、可配置爬虫、在线编辑代码等功能|暂时不支持爬虫版本管理| ![](https://img.shields.io/github/stars/crawlab-team/crawlab) ![](https://img.shields.io/github/forks/crawlab-team/crawlab) |
| [ScrapydWeb](https://github.com/my8100/scrapydweb) | Python Flask + Vue|精美的 UI 界面，内置了 scrapy 日志解析器，有较多任务运行统计图表，支持节点管理、定时任务、邮件提醒、移动界面，算是 scrapy-based 中功能完善的爬虫管理平台|不支持 scrapy 以外的爬虫，Python Flask 为后端，性能上有一定局限性| ![](https://img.shields.io/github/stars/my8100/scrapydweb) ![](https://img.shields.io/github/forks/my8100/scrapydweb) |
| [Gerapy](https://github.com/Gerapy/Gerapy) | Python Django + Vue|Gerapy 是崔庆才大神开发的爬虫管理平台，安装部署非常简单，同样基于 scrapyd，有精美的 UI 界面，支持节点管理、代码编辑、可配置规则等功能|同样不支持 scrapy 以外的爬虫，而且据使用者反馈，1.0 版本有很多 bug，期待 2.0 版本会有一定程度的改进| ![](https://img.shields.io/github/stars/Gerapy/Gerapy) ![](https://img.shields.io/github/forks/Gerapy/Gerapy) |
| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | Python Flask|基于 scrapyd，开源版 Scrapyhub，非常简洁的 UI 界面，支持定时任务|可能有些过于简洁了，不支持分页，不支持节点管理，不支持 scrapy 以外的爬虫| ![](https://img.shields.io/github/stars/DormyMo/SpiderKeeper) ![](https://img.shields.io/github/forks/DormyMo/SpiderKeeper) |
## Q&A

13
README.md
View File

@@ -241,13 +241,12 @@ The reason is that most of the existing platforms are depending on Scrapyd, whic
Crawlab is easy to use, general enough to adapt spiders in any language and any framework. It has also a beautiful frontend interface for users to manage spiders much more easily.
|Framework | Type | Distributed | Frontend | Scrapyd-Dependent |
|:---:|:---:|:---:|:---:|:---:|
| [Crawlab](https://github.com/crawlab-team/crawlab) | Admin Platform | Y | Y | N
| [ScrapydWeb](https://github.com/my8100/scrapydweb) | Admin Platform | Y | Y | Y
| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | Admin Platform | Y | Y | Y
| [Gerapy](https://github.com/Gerapy/Gerapy) | Admin Platform | Y | Y | Y
| [Scrapyd](https://github.com/scrapy/scrapyd) | Web Service | Y | N | N/A
|Framework | Technology | Pros | Cons | Github Stats |
|:---|:---|:---|-----| :---- |
| [Crawlab](https://github.com/crawlab-team/crawlab) | Golang + Vue|Not limited to Scrapy, available for all programming languages and frameworks. Beautiful UI interface. Naturally support distributed spiders. Support spider mangement, task management, cron job, result export, analytics, notifications, configurable spiders, online code editor, etc.|Not yet support spider versioning| ![](https://img.shields.io/github/stars/crawlab-team/crawlab) ![](https://img.shields.io/github/forks/crawlab-team/crawlab) |
| [ScrapydWeb](https://github.com/my8100/scrapydweb) | Python Flask + Vue|Beautiful UI interface, built-in Scrapy log parser, stats and graphs for task execution, support node management, cron job, mail notification, mobile. Full-feature spider management platform.|Not support spiders other than Scrapy. Limited performance because of Python Flask backend.| ![](https://img.shields.io/github/stars/my8100/scrapydweb) ![](https://img.shields.io/github/forks/my8100/scrapydweb) |
| [Gerapy](https://github.com/Gerapy/Gerapy) | Python Django + Vue|Gerapy is built by web crawler guru [Germey Cui](https://github.com/Germey). Simple installation and deployment. Beautiful UI interface. Support node management, code edit, configurable crawl rules, etc.|Again not support spiders other than Scrapy. A lot of bugs based on user feedback in v1.0. Look forward to improvement in v2.0| ![](https://img.shields.io/github/stars/Gerapy/Gerapy) ![](https://img.shields.io/github/forks/Gerapy/Gerapy) |
| [SpiderKeeper](https://github.com/DormyMo/SpiderKeeper) | Python Flask|Open-source Scrapyhub. Concise and simple UI interface. Support cron job.|Perhaps too simplified, not support pagination, not support node management, not support spiders other than Scrapy.| ![](https://img.shields.io/github/stars/DormyMo/SpiderKeeper) ![](https://img.shields.io/github/forks/DormyMo/SpiderKeeper) |
## Contributors
<a href="https://github.com/tikazyq">

2
backend/model/task.go
View File

@@ -95,7 +95,7 @@ func (t *Task) GetResults(pageNum int, pageSize int) (results []interface{}, tot
query := bson.M{
"task_id": t.Id,
}
if err = c.Find(query).Skip((pageNum - 1) * pageSize).Limit(pageSize).Sort("-create_ts").All(&results); err != nil {
if err = c.Find(query).Skip((pageNum - 1) * pageSize).Limit(pageSize).All(&results); err != nil {
return
}

5
docker-compose.yml
View File

@@ -60,3 +60,8 @@ services:
# - "/opt/crawlab/redis/data:/data" # make data persistent 持久化
# ports:
# - "6379:6379" # expose port to host machine 暴露接口到宿主机
# splash: # use Splash to run spiders on dynamic pages
# image: scrapinghub/splash
# container_name: splash
# ports:
# - "8050:8050"

1
frontend/package.json
View File

@@ -40,6 +40,7 @@
"vue-github-button": "^1.1.2",
"vue-i18n": "^8.9.0",
"vue-router": "^3.0.1",
"vue-tour": "^1.2.0",
"vue-virtual-scroll-list": "^1.3.9",
"vuex": "^3.0.1"
},

19
frontend/src/App.vue
View File

@@ -139,4 +139,23 @@ export default {
border-color: #f56c6c;
color: #fff;
}
.v-tour__target--highlighted {
box-shadow: none !important;
/*box-shadow: 0 0 0 4px #f56c6c !important;*/
border: 3px solid #f56c6c !important;
}
.v-step__button {
background: #67c23a !important;
border: none !important;
color: white !important;
}
.v-step__button:hover {
background: #67c23a !important;
border: none !important;
color: white !important;
opacity: 0.9 !important;
}
</style>

8
frontend/src/components/Config/ConfigList.vue
View File

@@ -815,18 +815,14 @@ ${f.css || f.xpath} ${f.attr ? ('(' + f.attr + ')') : ''} ${f.next_stage ? (' --
}
},
isList (stage) {
return !!stage.list_css || !!stage.list_xpath
return !!stage.is_list
},
onCheckIsList (value, stage) {
stage.is_list = value
if (value) {
this.$st.sendEv('爬虫详情', '配置', '勾选列表页')
if (!stage.list_css && !stage.list_xpath) {
stage.list_xpath = '//body'
}
} else {
this.$st.sendEv('爬虫详情', '配置', '取消勾选列表页')
stage.list_css = ''
stage.list_xpath = ''
}
},
onClickStageList ($event, stage, type) {

50
frontend/src/components/File/FileList.vue
View File

@@ -34,6 +34,7 @@
class="file-tree-wrapper"
>
<el-tree
class="tree"
:data="computedFileTree"
ref="tree"
node-key="path"
@@ -87,33 +88,33 @@
</el-popover>
</span>
</el-tree>
<el-popover trigger="click" placement="right"
popper-class="create-item-popover" :visible-arrow="false">
<ul class="action-item-list">
<li class="action-item" @click="fileDialogVisible = true">
<font-awesome-icon icon="file-alt" color="rgba(3,47,98,.5)"/>
<span class="action-item-text">{{$t('Create File')}}</span>
</li>
<li class="action-item" @click="dirDialogVisible = true">
<font-awesome-icon :icon="['fa', 'folder']" color="rgba(3,47,98,.5)"/>
<span class="action-item-text">{{$t('Create Directory')}}</span>
</li>
</ul>
<div
class="add-btn-wrapper"
slot="reference"
>
<div
class="add-btn-wrapper"
>
<el-popover trigger="click" placement="right"
popper-class="create-item-popover" :visible-arrow="false">
<ul class="action-item-list">
<li class="action-item" @click="fileDialogVisible = true">
<font-awesome-icon icon="file-alt" color="rgba(3,47,98,.5)"/>
<span class="action-item-text">{{$t('Create File')}}</span>
</li>
<li class="action-item" @click="dirDialogVisible = true">
<font-awesome-icon :icon="['fa', 'folder']" color="rgba(3,47,98,.5)"/>
<span class="action-item-text">{{$t('Create Directory')}}</span>
</li>
</ul>
<el-button
class="add-btn"
size="small"
type="primary"
icon="el-icon-plus"
slot="reference"
@click="onEmptyClick"
>
{{$t('Add')}}
</el-button>
</div>
</el-popover>
</el-popover>
</div>
</div>
<div class="main-content">
@@ -202,7 +203,18 @@ export default {
dirDialogVisible: false,
fileDialogVisible: false,
nodeExpandedDict: {},
isShowDeleteNav: false
isShowDeleteNav: false,
tourSteps: [
{
target: '.add-btn',
content: this.$t('You can add a file or directory')
}
],
tourCallbacks: {
onStop: () => {
this.$utils.tour.finishTour('spider-detail-file-list')
}
}
}
},
computed: {

29
frontend/src/i18n/zh.js
View File

@@ -406,6 +406,35 @@ docker run -d --restart always --name crawlab_worker \\
更多信息，请参考 [官方文档](https://docs.crawlab.cn)。
`,
// 教程
'Skip': '跳过',
'Previous': '上一步',
'Next': '下一步',
'Finish': '结束',
'Click to add a new spider': '点击并添加爬虫',
'You can view your created spiders here.<br>Click a table row to view <strong>spider details</strong>.': '您可以查看创建的爬虫<br>点击行来查看<strong>爬虫详情</strong>',
'View a list of <strong>Configurable Spiders</strong>': '查看<strong>可配置爬虫</strong>列表',
'View a list of <strong>Customized Spiders</strong>': '查看<strong>自定义爬虫</strong>列表',
'<strong>Customized Spider</strong> is a highly customized spider, which is able to run on any programming language and any web crawler framework.': '<strong>自定义爬虫</strong>是高度自定义化的爬虫，能够运行任何编程语言和爬虫框架',
'<strong>Configurable Spider</strong> is a spider defined by config data, aimed at streamlining spider development and improving dev efficiency.': '<strong>可配置爬虫</strong>被配置数据所定义，旨在将爬虫开发流程化以及提高爬虫开发效率',
'Unique identifier for the spider': '爬虫的唯一识别符',
'How the spider is displayed on Crawlab': '爬虫在 Crawlab 上的展示名称',
'A shell command to be executed when the spider is triggered to run (only available for <strong>Customized Spider</strong>': '当爬虫被触发时执行的一行 Shell 命令（仅<strong>自定义爬虫</strong>有效）',
'Where the results are stored in the database': '抓取结果在数据库中储存的位置',
'Upload a zip file containing all spider files to create the spider (only available for <strong>Customized Spider</strong>)': '上传一个包含所有爬虫文件的 zip 文件，然后创建爬虫（仅<strong>自定义爬虫</strong>有效）',
'The spider template to create from (only available for <strong>Configurable Spider</strong>)': '创建爬虫时引用的模版（仅<strong>可配置爬虫</strong>有效）',
'Click to confirm to add the spider': '点击并确认添加爬虫',
'You can switch to each section of the spider detail.': '您可以切换到爬虫详情的每一个部分',
'You can switch to different spider using this selector.': '您可以通过这个选择器切换不同的爬虫',
'You can view latest tasks for this spider and click each row to view task detail.': '您可以查看最近的爬虫任务以及点击行来查看任务详情',
'You can edit the detail info for this spider.': '您可以编辑爬虫详情信息',
'Here you can action on the spider, including running a task, uploading a zip file and save the spider info.': '这里您可以对爬虫进行操作，包括运行爬虫任务、上传 zip 文件以及保存爬虫信息',
'File navigation panel.<br><br>You can right click on <br>each item to create or delete<br> a file/directory.': '文件导航栏<br><br>您可以右键点击一个元素<br>来添加或删除文件/文件夹',
'Click to add a file or directory<br> on the root directory.': '点击并添加一个文件<br>或文件夹',
'You can edit, save, rename<br> and delete the selected file <br>in this box.': '在这个栏位中，您可以<br>编辑、保存、重命名、<br>删除所选择的文件',
'Here you can add environment variables that will be passed to the spider program when running a task.': '这里您可以添加环境变量，这些环境变量会被传入运行的爬虫程序中',
'You can add, edit and delete schedules (cron jobs) for the spider.': '您可以添加、修改、删除爬虫的定时任务',
// 其他
'Star crawlab-team/crawlab on GitHub': '在 GitHub 上为 Crawlab 加星吧'
}

7
frontend/src/main.js
View File

@@ -31,6 +31,10 @@ import request from './api/request'
import i18n from './i18n'
import utils from './utils'
// vue-tour
import VueTour from 'vue-tour'
import 'vue-tour/dist/vue-tour.css'
// code mirror
Vue.use(codemirror)
@@ -45,6 +49,9 @@ Vue.component('font-awesome-icon', FontAwesomeIcon)
Vue.component('font-awesome-layers', FontAwesomeLayers)
Vue.component('font-awesome-layers-text', FontAwesomeLayersText)
// vue-tour
Vue.use(VueTour)
Vue.config.productionTip = false
// 百度统计

6
frontend/src/store/modules/spider.js
View File

@@ -105,15 +105,9 @@ const actions = {
},
editSpider ({ state, dispatch }) {
return request.post(`/spiders/${state.spiderForm._id}`, state.spiderForm)
.then(() => {
dispatch('getSpiderList')
})
},
deleteSpider ({ state, dispatch }, id) {
return request.delete(`/spiders/${id}`)
.then(() => {
dispatch('getSpiderList')
})
},
getSpiderData ({ state, commit }, id) {
return request.get(`/spiders/${id}`)

4
frontend/src/utils/index.js
View File

@@ -1,7 +1,9 @@
import stats from './stats'
import encrypt from './encrypt'
import tour from './tour'
export default {
stats,
encrypt
encrypt,
tour
}

41
frontend/src/utils/tour.js Normal file
View File

@@ -0,0 +1,41 @@
import i18n from '../i18n'
export default {
isFinishedTour: (tourName) => {
if (!localStorage.getItem('tour')) {
localStorage.setItem('tour', JSON.stringify({}))
return false
}
let data
try {
data = JSON.parse(localStorage.getItem('tour'))
} catch (e) {
localStorage.setItem('tour', JSON.stringify({}))
return false
}
return !!data[tourName]
},
finishTour: (tourName) => {
let data
try {
data = JSON.parse(localStorage.getItem('tour'))
} catch (e) {
localStorage.setItem('tour', JSON.stringify({}))
data = {}
}
data[tourName] = 1
localStorage.setItem('tour', JSON.stringify(data))
},
getOptions: (isShowHighlight) => {
return {
labels: {
buttonSkip: i18n.t('Skip'),
buttonPrevious: i18n.t('Previous'),
buttonNext: i18n.t('Next'),
buttonStop: i18n.t('Finish')
},
highlight: isShowHighlight
}
}
}

13
frontend/src/views/layout/components/Sidebar/index.vue
View File

@@ -12,7 +12,13 @@
:active-text-color="variables.menuActiveText"
mode="vertical"
>
<sidebar-item v-for="route in routes" :key="route.path" :item="route" :base-path="route.path"/>
<sidebar-item
v-for="route in routes"
:class="route.path.replace('/', '')"
:key="route.path"
:item="route"
:base-path="route.path"
/>
</el-menu>
</el-scrollbar>
</template>
@@ -53,10 +59,11 @@ export default {
}
},
data () {
return {
}
return {}
},
async created () {
},
mounted () {
}
}
</script>

2
frontend/src/views/schedule/ScheduleList.vue
View File

@@ -108,7 +108,7 @@
<!--<vcrontab @hide="showCron=false" @fill="onCrontabFill" :expression="expression"></vcrontab>-->
<!--</el-dialog>-->
<el-card style="border-radius: 0">
<el-card style="border-radius: 0" class="schedule-list">
<!--filter-->
<div class="filter">
<div class="right">

100
frontend/src/views/spider/SpiderDetail.vue
View File

@@ -1,9 +1,18 @@
<template>
<div class="app-container">
<!--tour-->
<v-tour
name="spider-detail"
:steps="tourSteps"
:callbacks="tourCallbacks"
:options="$utils.tour.getOptions(true)"
/>
<!--./tour-->
<!--selector-->
<div class="selector">
<label class="label">{{$t('Spider')}}: </label>
<el-select v-model="spiderForm._id" @change="onSpiderChange">
<el-select id="spider-select" v-model="spiderForm._id" @change="onSpiderChange">
<el-option v-for="op in spiderList" :key="op._id" :value="op._id" :label="op.name"></el-option>
</el-select>
</div>
@@ -62,7 +71,89 @@ export default {
},
data () {
return {
activeTabName: 'overview'
activeTabName: 'overview',
tourSteps: [
// top bar
{
target: '.el-tabs__nav.is-top',
content: this.$t('You can switch to each section of the spider detail.')
},
{
target: '#spider-select',
content: this.$t('You can switch to different spider using this selector.')
},
// overview
{
target: '.task-table-view',
content: this.$t('You can view latest tasks for this spider and click each row to view task detail.'),
params: {
placement: 'right'
}
},
{
target: '.spider-form',
content: this.$t('You can edit the detail info for this spider.'),
params: {
placement: 'left'
}
},
{
target: '.button-container',
content: this.$t('Here you can action on the spider, including running a task, uploading a zip file and save the spider info.'),
params: {
placement: 'top'
}
},
// file
{
target: '.tree',
content: this.$t('File navigation panel.<br><br>You can right click on <br>each item to create or delete<br> a file/directory.')
},
{
target: '.add-btn',
content: this.$t('Click to add a file or directory<br> on the root directory.')
},
{
target: '.main-content',
content: this.$t('You can edit, save, rename<br> and delete the selected file <br>in this box.'),
params: {
placement: 'left'
}
},
// environment
{
target: '.environment-list',
content: this.$t('Here you can add environment variables that will be passed to the spider program when running a task.')
},
// schedules
{
target: '.schedule-list',
content: this.$t('You can add, edit and delete schedules (cron jobs) for the spider.')
}
],
tourCallbacks: {
onStop: () => {
this.$utils.tour.finishTour('spider-detail')
},
onPreviousStep: (currentStep) => {
if (currentStep === 5) {
this.activeTabName = 'overview'
} else if (currentStep === 8) {
this.activeTabName = 'files'
} else if (currentStep === 9) {
this.activeTabName = 'environment'
}
},
onNextStep: (currentStep) => {
if (currentStep === 4) {
this.activeTabName = 'files'
} else if (currentStep === 7) {
this.activeTabName = 'environment'
} else if (currentStep === 8) {
this.activeTabName = 'schedules'
}
}
}
}
},
computed: {
@@ -121,6 +212,11 @@ export default {
// if (this.spiderForm.type === 'configurable') {
// this.activeTabName = 'config'
// }
},
mounted () {
if (!this.$utils.tour.isFinishedTour('spider-detail')) {
this.$tours['spider-detail'].start()
}
}
}
</script>

188
frontend/src/views/spider/SpiderList.vue
View File

@@ -1,5 +1,20 @@
<template>
<div class="app-container">
<!--tour-->
<v-tour
name="spider-list"
:steps="tourSteps"
:callbacks="tourCallbacks"
:options="$utils.tour.getOptions(true)"
/>
<v-tour
name="spider-list-add"
:steps="tourAddSteps"
:callbacks="tourAddCallbacks"
:options="$utils.tour.getOptions(true)"
/>
<!--./tour-->
<!--import popup-->
<el-dialog
:title="$t('Import Spider')"
@@ -38,16 +53,16 @@
<el-tab-pane name="customized" :label="$t('Customized')">
<el-form :model="spiderForm" ref="addCustomizedForm" inline-message label-width="120px">
<el-form-item :label="$t('Spider Name')" prop="name" required>
<el-input v-model="spiderForm.name" :placeholder="$t('Spider Name')"/>
<el-input id="spider-name" v-model="spiderForm.name" :placeholder="$t('Spider Name')"/>
</el-form-item>
<el-form-item :label="$t('Display Name')" prop="display_name" required>
<el-input v-model="spiderForm.display_name" :placeholder="$t('Display Name')"/>
<el-input id="display-name" v-model="spiderForm.display_name" :placeholder="$t('Display Name')"/>
</el-form-item>
<el-form-item :label="$t('Execute Command')" prop="cmd" required>
<el-input v-model="spiderForm.cmd" :placeholder="$t('Execute Command')"/>
<el-input id="cmd" v-model="spiderForm.cmd" :placeholder="$t('Execute Command')"/>
</el-form-item>
<el-form-item :label="$t('Results')" prop="col" required>
<el-input v-model="spiderForm.col" :placeholder="$t('Results')"/>
<el-input id="col" v-model="spiderForm.col" :placeholder="$t('Results')"/>
</el-form-item>
<el-form-item :label="$t('Upload Zip File')" label-width="120px" name="site">
<el-upload
@@ -58,7 +73,7 @@
:file-list="fileList"
:before-upload="beforeUpload"
>
<el-button size="small" type="primary" icon="el-icon-upload">
<el-button id="upload" size="small" type="primary" icon="el-icon-upload">
{{$t('Upload')}}
</el-button>
</el-upload>
@@ -90,7 +105,8 @@
<el-input v-model="spiderForm.display_name" :placeholder="$t('Display Name')"/>
</el-form-item>
<el-form-item :label="$t('Template')" prop="template" required>
<el-select v-model="spiderForm.template" :value="spiderForm.template" :placeholder="$t('Template')">
<el-select id="template" v-model="spiderForm.template" :value="spiderForm.template"
:placeholder="$t('Template')">
<el-option
v-for="template in templateList"
:key="template"
@@ -104,7 +120,7 @@
</el-form-item>
</el-form>
<div class="actions">
<el-button size="small" type="primary" @click="onAddConfigurable">{{$t('Add')}}</el-button>
<el-button id="add" size="small" type="primary" @click="onAddConfigurable">{{$t('Add')}}</el-button>
</div>
</el-tab-pane>
</el-tabs>
@@ -164,10 +180,10 @@
<!--./filter-->
<!--tabs-->
<el-tabs v-model="filter.type" @tab-click="onClickTab">
<el-tab-pane :label="$t('All')" name="all"></el-tab-pane>
<el-tab-pane :label="$t('Customized')" name="customized"></el-tab-pane>
<el-tab-pane :label="$t('Configurable')" name="configurable"></el-tab-pane>
<el-tabs v-model="filter.type" @tab-click="onClickTab" class="tabs">
<el-tab-pane :label="$t('All')" name="all" class="all"></el-tab-pane>
<el-tab-pane :label="$t('Customized')" name="customized" class="customized"></el-tab-pane>
<el-tab-pane :label="$t('Configurable')" name="configurable" class="configuable"></el-tab-pane>
</el-tabs>
<!--./tabs-->
@@ -339,7 +355,121 @@ export default {
name: [{ required: true, message: 'Required Field', trigger: 'change' }]
},
fileList: [],
spiderType: 'customized'
spiderType: 'customized',
tourSteps: [
{
target: '#tab-customized',
content: this.$t('View a list of <strong>Customized Spiders</strong>'),
params: {
highlight: false
}
},
{
target: '#tab-configurable',
content: this.$t('View a list of <strong>Configurable Spiders</strong>'),
params: {
highlight: false
}
},
{
target: '.table',
content: this.$t('You can view your created spiders here.<br>Click a table row to view <strong>spider details</strong>.'),
params: {
placement: 'top'
}
},
{
target: '.btn.add',
content: this.$t('Click to add a new spider')
}
],
tourCallbacks: {
onStop: () => {
this.$utils.tour.finishTour('spider-list')
}
},
tourAddSteps: [
{
target: '#tab-customized',
content: this.$t('<strong>Customized Spider</strong> is a highly customized spider, which is able to run on any programming language and any web crawler framework.'),
params: {
placement: 'bottom',
highlight: false
}
},
{
target: '#tab-configurable',
content: this.$t('<strong>Configurable Spider</strong> is a spider defined by config data, aimed at streamlining spider development and improving dev efficiency.'),
params: {
placement: 'bottom',
highlight: false
}
},
{
target: '#spider-name',
content: this.$t('Unique identifier for the spider'),
params: {
placement: 'right'
}
},
{
target: '#display-name',
content: this.$t('How the spider is displayed on Crawlab'),
params: {
placement: 'right'
}
},
{
target: '#cmd',
content: this.$t('A shell command to be executed when the spider is triggered to run (only available for <strong>Customized Spider</strong>'),
params: {
placement: 'right'
}
},
{
target: '#col',
content: this.$t('Where the results are stored in the database'),
params: {
placement: 'right'
}
},
{
target: '#upload',
content: this.$t('Upload a zip file containing all spider files to create the spider (only available for <strong>Customized Spider</strong>)'),
params: {
placement: 'right'
}
},
{
target: '#template',
content: this.$t('The spider template to create from (only available for <strong>Configurable Spider</strong>)'),
params: {
placement: 'right'
}
},
{
target: '#add',
content: this.$t('Click to confirm to add the spider'),
params: {
placement: 'right'
}
}
],
tourAddCallbacks: {
onStop: () => {
this.$utils.tour.finishTour('spider-list-add')
},
onPreviousStep: (currentStep) => {
if (currentStep === 7) {
this.spiderType = 'customized'
}
},
onNextStep: (currentStep) => {
if (currentStep === 6) {
this.spiderType = 'configurable'
}
}
}
}
},
computed: {
@@ -383,6 +513,12 @@ export default {
template: this.templateList[0]
})
this.addDialogVisible = true
setTimeout(() => {
if (!this.$utils.tour.isFinishedTour('spider-list-add')) {
this.$tours['spider-list-add'].start()
}
}, 300)
},
onAddConfigurable () {
this.$refs['addConfigurableForm'].validate(async res => {
@@ -395,8 +531,8 @@ export default {
this.$message.error(this.$t('Something wrong happened'))
return
}
await this.$store.dispatch('spider/getSpiderList')
this.$router.push(`/spiders/${res2.data.data._id}`)
await this.$store.dispatch('spider/getSpiderList')
this.$st.sendEv('爬虫列表', '添加爬虫', '可配置爬虫')
})
},
@@ -410,8 +546,8 @@ export default {
this.$message.error(this.$t('Something wrong happened'))
return
}
await this.$store.dispatch('spider/getSpiderList')
this.$router.push(`/spiders/${res2.data.data._id}`)
await this.$store.dispatch('spider/getSpiderList')
this.$st.sendEv('爬虫列表', '添加爬虫', '自定义爬虫')
})
},
@@ -463,14 +599,13 @@ export default {
confirmButtonText: this.$t('Confirm'),
cancelButtonText: this.$t('Cancel'),
type: 'warning'
}).then(() => {
this.$store.dispatch('spider/deleteSpider', row._id)
.then(() => {
this.$message({
type: 'success',
message: 'Deleted successfully'
})
})
}).then(async () => {
await this.$store.dispatch('spider/deleteSpider', row._id)
this.$message({
type: 'success',
message: 'Deleted successfully'
})
await this.getList()
this.$st.sendEv('爬虫列表', '删除爬虫')
})
},
@@ -586,7 +721,7 @@ export default {
this.filter.type = tab.name
this.getList()
},
getList () {
async getList () {
let params = {
page_num: this.pagination.pageNum,
page_size: this.pagination.pageSize,
@@ -595,7 +730,7 @@ export default {
keyword: this.filter.keyword,
type: this.filter.type
}
this.$store.dispatch('spider/getSpiderList', params)
await this.$store.dispatch('spider/getSpiderList', params)
}
},
async created () {
@@ -609,11 +744,14 @@ export default {
await this.$store.dispatch('spider/getTemplateList')
},
mounted () {
console.log(this.spiderForm)
const vm = this
this.$nextTick(() => {
vm.$store.commit('spider/SET_SPIDER_FORM', this.spiderForm)
})
if (!this.$utils.tour.isFinishedTour('spider-list')) {
this.$tours['spider-list'].start()
}
}
}
</script>

30
frontend/yarn.lock
View File

@@ -4142,6 +4142,11 @@ hash-sum@^1.0.2:
version "1.0.2"
resolved "http://registry.npm.taobao.org/hash-sum/download/hash-sum-1.0.2.tgz#33b40777754c6432573c120cc3808bbd10d47f04"
hash-sum@^2.0.0:
version "2.0.0"
resolved "https://registry.npm.taobao.org/hash-sum/download/hash-sum-2.0.0.tgz#81d01bb5de8ea4a214ad5d6ead1b523460b0b45a"
integrity sha1-gdAbtd6OpKIUrV1urRtSNGCwtFo=
hash.js@^1.0.0, hash.js@^1.0.3:
version "1.1.7"
resolved "http://registry.npm.taobao.org/hash.js/download/hash.js-1.1.7.tgz#0babca538e8d4ee4a0f8988d68866537a003cf42"
@@ -5322,6 +5327,11 @@ jsprim@^1.2.2:
json-schema "0.2.3"
verror "1.10.0"
jump.js@^1.0.2:
version "1.0.2"
resolved "https://registry.npm.taobao.org/jump.js/download/jump.js-1.0.2.tgz#e0641b47f40a38f2139c25fda0500bf28e43015a"
integrity sha1-4GQbR/QKOPITnCX9oFAL8o5DAVo=
killable@^1.0.0:
version "1.0.1"
resolved "http://registry.npm.taobao.org/killable/download/killable-1.0.1.tgz#4c8ce441187a061c7474fb87ca08e2a638194892"
@@ -6551,6 +6561,11 @@ pn@^1.1.0:
version "1.1.0"
resolved "http://registry.npm.taobao.org/pn/download/pn-1.1.0.tgz#e2f4cef0e219f463c179ab37463e4e1ecdccbafb"
popper.js@^1.16.0:
version "1.16.1"
resolved "https://registry.npm.taobao.org/popper.js/download/popper.js-1.16.1.tgz#2a223cb3dc7b6213d740e40372be40de43e65b1b"
integrity sha1-KiI8s9x7YhPXQOQDcr5A3kPmWxs=
portfinder@^1.0.20, portfinder@^1.0.9:
version "1.0.20"
resolved "http://registry.npm.taobao.org/portfinder/download/portfinder-1.0.20.tgz#bea68632e54b2e13ab7b0c4775e9b41bf270e44a"
@@ -8674,6 +8689,16 @@ vue-template-es2015-compiler@^1.6.0, vue-template-es2015-compiler@^1.8.2:
version "1.8.2"
resolved "http://registry.npm.taobao.org/vue-template-es2015-compiler/download/vue-template-es2015-compiler-1.8.2.tgz#dd73e80ba58bb65dd7a8aa2aeef6089cf6116f2a"
vue-tour@^1.2.0:
version "1.2.0"
resolved "https://registry.npm.taobao.org/vue-tour/download/vue-tour-1.2.0.tgz#340bbfec8f3778a9ea0c87176ce3611c69e0a4d4"
integrity sha1-NAu/7I83eKnqDIcXbONhHGngpNQ=
dependencies:
hash-sum "^2.0.0"
jump.js "^1.0.2"
popper.js "^1.16.0"
vue "^2.6.10"
vue-virtual-scroll-list@^1.3.9:
version "1.3.9"
resolved "https://registry.npm.taobao.org/vue-virtual-scroll-list/download/vue-virtual-scroll-list-1.3.9.tgz#ba3ce6425374fb323ea83ab33daa2727117808ed"
@@ -8688,6 +8713,11 @@ vue@^2.5.17, vue@^2.5.22:
version "2.6.6"
resolved "https://registry.yarnpkg.com/vue/-/vue-2.6.6.tgz#dde41e483c11c46a7bf523909f4f2f816ab60d25"
vue@^2.6.10:
version "2.6.11"
resolved "https://registry.npm.taobao.org/vue/download/vue-2.6.11.tgz#76594d877d4b12234406e84e35275c6d514125c5"
integrity sha1-dllNh31LEiNEBuhONSdcbVFBJcU=
vuex@^3.0.1:
version "3.1.0"
resolved "https://registry.yarnpkg.com/vuex/-/vuex-3.1.0.tgz#634b81515cf0cfe976bd1ffe9601755e51f843b9"

5
jenkins/master/docker-compose.yaml
View File

@@ -6,13 +6,14 @@ services:
CRAWLAB_API_ADDRESS: "https://crawlab.cn/api"
CRAWLAB_BASE_URL: "/demo"
CRAWLAB_SERVER_MASTER: "Y"
CRAWLAB_SERVER_REGISTER_TYPE: "ip"
CRAWLAB_SERVER_REGISTER_IP: "172.19.0.1"
CRAWLAB_MONGO_HOST: "mongo"
CRAWLAB_REDIS_ADDRESS: "redis"
CRAWLAB_LOG_PATH: "/var/logs/crawlab"
CRAWLAB_SETTING_ALLOWREGISTER: "Y"
ports:
- "8080:8080" # frontend
- "8000:8000" # backend
depends_on:
- mongo
- redis
@@ -20,6 +21,8 @@ services:
image: "tikazyq/crawlab:master"
environment:
CRAWLAB_SERVER_MASTER: "N"
CRAWLAB_SERVER_REGISTER_TYPE: "ip"
CRAWLAB_SERVER_REGISTER_IP: "172.19.0.2"
CRAWLAB_MONGO_HOST: "mongo"
CRAWLAB_REDIS_ADDRESS: "redis"
depends_on: