git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-24 17:41:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
875 Commits 6 Branches 35 Tags
484a7ec17c4da9bf550e82bb958deff58e00a22a
Commit Graph

32 Commits

Author SHA1 Message Date
yaziming
2f6b517356 refactor(all): refactor code 
remove redundant code and some code refactor
 2019-10-11 16:01:57 +08:00
陈景阳
7c75c24f65 fix md5值不一致的问题 2019-10-07 12:49:37 +08:00
陈景阳
0d10d6d045 fix 创建目录错误的问题 2019-10-07 12:21:32 +08:00
陈景阳
d80642aa0c fix 删除爬虫的问题 2019-09-30 12:09:37 +08:00
陈景阳
412e0fd983 fix bug 2019-09-26 21:13:25 +08:00
陈景阳
41e9edf940 完成爬虫列表 2019-09-26 20:53:05 +08:00
陈景阳
5416b7ac60 fix 上传的问题 2019-09-26 19:44:12 +08:00
陈景阳
79ee09aacb 完成爬虫获取 2019-09-26 19:12:02 +08:00
陈景阳
6c37731cca 优化爬虫获取逻辑 2019-09-26 16:43:32 +08:00
陈景阳
6df6155daf 优化爬虫获取逻辑 2019-09-26 16:26:32 +08:00
陈景阳
475bf4c69e 优化爬虫获取逻辑 2019-09-26 11:38:13 +08:00
陈景阳
9237e62a48 爬虫逻辑修改为从GridFS获取 2019-09-26 11:28:20 +08:00
陈景阳
f20039fefd fix 消息无法订阅问题 
fix 可能出现重复爬虫的问题
 2019-09-10 14:26:50 +08:00
yaziming
888a3cab79 refactor(backend): Use more efficient bytes to string methods and remove unnecessary type conversions 
detail:
    1. add utils.BytesToString function instead of string() convert bytes to string.
    2. use bytes.NewReader instead of strings.NewReader(string(sb)).
    3. use w.Body.Bytes() instead of []byte(w.Body.String()).
 2019-09-03 15:17:32 +08:00
陈景阳
4255b2f6b6 还原代码 2019-09-03 09:06:04 +08:00
陈景阳
9e9b117669 Merge branch 'develop' of https://github.com/crawlab-team/crawlab into develop 2019-09-03 08:57:21 +08:00
陈景阳
41c3b98333 还原代码 2019-09-02 18:14:34 +08:00
陈景阳
45aaaa2c46 fix 无法正常删除有问题的爬虫 2019-09-02 18:04:47 +08:00
陈景阳
69fc10fcc3 fix 如果从dir读取爬虫为空，则移除所有的爬虫 2019-09-02 17:37:48 +08:00
yaziming
443d697c6f backend: 
1. Mongo dial add 5 seconds connection timeout.
 2. Redis uses connection pool mode.
 3. Redis pool new connection have 10 seconds write timeout and read timeout and connection timeout.
 2019-09-01 17:18:08 +08:00
yaziming
9f43e08ff9 Backend: 
improve
     - AuthMiddleware 注入当前用户的信息
     - 增加Context服务支持快捷获取当前登录者信息
     - 重构Login/GetMe接口逻辑避免重复的数据库查询
     - 规范化error信息声明(向下兼容,旧代码可逐渐迁移规范化)
     - 修正部分不符合规范的代码
 2019-08-31 21:26:56 +08:00
陈景阳
e027aeb71f 添加日志打印 2019-08-31 17:56:42 +08:00
陈景阳
1a383dac17 fix 爬虫目录无法打开的问题 2019-08-31 17:04:49 +08:00
陈景阳
494601ab62 fix 节点注册异常情况 2019-08-31 13:49:34 +08:00
陈景阳
c634e2b5cf fix 前端控制台报错的问题 \ 
fix 无法打印中文的问题
 2019-08-31 12:04:12 +08:00
陈景阳
f098546a65 Merge branch 'develop' of https://github.com/crawlab-team/crawlab into v0.4.0 2019-08-28 16:00:27 +08:00
陈景阳
1736d00161 添加打印日志 2019-08-28 16:00:09 +08:00
陈景阳
0186d08112 删除爬虫顺带删除文件 2019-08-27 09:39:16 +08:00
hantmac
f7c0e8310a bug fix:fix permission bug when create spiders dir caused by umask 2019-08-26 17:44:01 +08:00
hantmac
21a68693c3 Code optimization:change the crawler zip file to slice read 2019-08-20 17:09:43 +08:00
hantmac
005abc68c8 Code optimization:change the crawler zip file to slice read 2019-08-20 16:59:53 +08:00
Marvin Zhang
53cebde962 added golang 2019-07-22 12:51:52 +08:00