git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-22 17:31:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
792 Commits 6 Branches 35 Tags
169a7d0df90ce109705c558d0feabbebdae73fcd
Commit Graph

137 Commits

Author SHA1 Message Date
陈景阳
169a7d0df9 fix 可能直接退出的问题 2019-09-26 16:49:01 +08:00
陈景阳
31be4c1839 优化爬虫获取逻辑 2019-09-26 16:43:32 +08:00
陈景阳
a11544d809 优化爬虫获取逻辑 2019-09-26 16:26:32 +08:00
陈景阳
947b561653 优化爬虫获取逻辑 2019-09-26 11:38:13 +08:00
陈景阳
05c28230b7 爬虫逻辑修改为从GridFS获取 2019-09-26 11:28:20 +08:00
陈景阳
96e926b486 fix 无法终止任务的问题 2019-09-26 10:29:58 +08:00
陈景阳
e235076a11 fix 无法终止任务的问题 2019-09-26 10:23:08 +08:00
陈景阳
7218a4f4de 添加日志打印 2019-09-26 09:37:10 +08:00
陈景阳
fda7e56e1a 添加日志打印 2019-09-25 16:19:58 +08:00
陈景阳
82999225de fix 无法获取task的错误 2019-09-25 15:11:27 +08:00
陈景阳
951f6a9f07 fix 2019-09-25 14:38:46 +08:00
陈景阳
3a272b404c fix 空指针引用 2019-09-10 14:47:23 +08:00
陈景阳
42e7647cd4 fix 消息无法订阅问题 
fix 可能出现重复爬虫的问题
 2019-09-10 14:26:50 +08:00
陈景阳
000b4ff730 删除日志 2019-09-10 09:51:27 +08:00
陈景阳
cdeef4b4a5 重构msg的包 2019-09-10 09:32:48 +08:00
陈景阳
8cf48d871e fix 删除spider没有删除task 2019-09-05 20:07:08 +08:00
Marvin Zhang
4893439ffb added param to task 2019-09-04 20:13:12 +08:00
hantmac
2fb1754e56 bug fix:fix data race caused by LockList 2019-09-04 13:51:02 +08:00
Marvin Zhang
4e17158a46 Merge pull request #208 from yaziming/v0.4.0-imporve-error-response 
feat(backend): add some feat for error message system
 2019-09-03 15:25:04 +08:00
yaziming
71c83251e5 feat(backend): add some feat for error message system 
errors support custom http code and support return invalid validated
error
 2019-09-03 15:23:08 +08:00
yaziming
e10d8fd996 refactor(backend): Use more efficient bytes to string methods and remove unnecessary type conversions 
detail:
    1. add utils.BytesToString function instead of string() convert bytes to string.
    2. use bytes.NewReader instead of strings.NewReader(string(sb)).
    3. use w.Body.Bytes() instead of []byte(w.Body.String()).
 2019-09-03 15:17:32 +08:00
hantmac
cb3d7263b2 bug fix:fix go test ./.. bug 2019-09-03 12:47:18 +08:00
陈景阳
cd78e6c745 还原代码 2019-09-03 09:06:04 +08:00
陈景阳
eb92d34f9b Merge branch 'develop' of https://github.com/crawlab-team/crawlab into develop 2019-09-03 08:57:21 +08:00
yaziming
361134f8dd Remove redundant fmt.Println 2019-09-02 22:57:29 +08:00
陈景阳
352fac4096 fix 删除spider没有删除task的问题 2019-09-02 20:49:00 +08:00
yaziming
10c3488caf break loop when subscribe count is 0 2019-09-02 18:32:47 +08:00
陈景阳
b5084c964b 还原代码 2019-09-02 18:14:34 +08:00
陈景阳
8b237ddc2a fix 无法正常删除有问题的爬虫 2019-09-02 18:04:47 +08:00
陈景阳
93dd3b714a fix 如果从dir读取爬虫为空，则移除所有的爬虫 2019-09-02 17:37:48 +08:00
陈景阳
f9fdf5b8d8 fix 打印日志只有2KB 2019-09-02 16:45:56 +08:00
陈景阳
033ca493aa 添加排序 2019-09-02 10:40:16 +08:00
yaziming
a9346a0934 backend: 
1. Mongo dial add 5 seconds connection timeout.
 2. Redis uses connection pool mode.
 3. Redis pool new connection have 10 seconds write timeout and read timeout and connection timeout.
 2019-09-01 17:18:08 +08:00
Marvin Zhang
d0c5f25bfe Merge pull request #195 from hantmac/master 
bug fix:fix read log file oom
 2019-09-01 11:09:02 +08:00
yaziming
c054b9bd33 Backend: 
upgrade gin framework depend validator version : v8 -> v9
 2019-09-01 10:13:53 +08:00
Jeremy
92c37a9e5c Merge branch 'develop' into master 2019-09-01 00:13:19 +08:00
hantmac
1de80f7779 add unit test for GetLocalLog 2019-09-01 00:07:18 +08:00
hantmac
b766be577a bug fix:fix read log file oom 2019-09-01 00:06:26 +08:00
yaziming
81f6cf021f Backend: 
improve
     - AuthMiddleware 注入当前用户的信息
     - 增加Context服务支持快捷获取当前登录者信息
     - 重构Login/GetMe接口逻辑避免重复的数据库查询
     - 规范化error信息声明(向下兼容,旧代码可逐渐迁移规范化)
     - 修正部分不符合规范的代码
 2019-08-31 21:26:56 +08:00
陈景阳
b9ed176950 添加日志打印 2019-08-31 17:56:42 +08:00
陈景阳
e423ff564c fix 爬虫目录无法打开的问题 2019-08-31 17:04:49 +08:00
陈景阳
f8e50fc829 Merge branch 'develop' of https://github.com/crawlab-team/crawlab into v0.4.0 2019-08-31 16:54:26 +08:00
陈景阳
0a0f943264 添加备注 
添加上次运行状态
 2019-08-31 16:09:41 +08:00
陈景阳
8115f06496 优化代码 2019-08-31 15:59:43 +08:00
Marvin Zhang
3b468f92f4 Merge pull request #192 from yaziming/forked/develop 
Some Change.
 2019-08-31 15:15:35 +08:00
陈景阳
5e6aec2393 fix 无法获取日志的问题 2019-08-31 15:03:57 +08:00
yaziming
b3cdb231ac Backend: 
1.[improve]
       使用gin提供的RouteGroup功能简化中间件逻辑
    2.[break change]
       移除Authorization Middleware中对登录注册下载特殊处理逻辑
    3.[unsafe problem]
       下载任务csv增加登录验证
Frontend:
    1. 更改csv下载方式
    2. task list页面table section 移除固定width设置,采用自适应,防止大屏空白断裂问题
 2019-08-31 15:00:38 +08:00
陈景阳
b61445acb0 优化代码 2019-08-31 13:57:14 +08:00
陈景阳
4944883f95 fix 节点注册异常情况 2019-08-31 13:49:34 +08:00
陈景阳
57dddc84ed FIX #178 
FIX 当日志文件小于2048的时候无法正常拉取日志的问题
 2019-08-31 12:42:53 +08:00