git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-23 17:31:11 +01:00
Code Issues Packages Projects Releases Wiki Activity
1,802 Commits 6 Branches 35 Tags
a14e1663200fdd5668d6fa4373e593eaeee85a31
Commit Graph

57 Commits

Author SHA1 Message Date
marvzhang
b0a930ca32 结果集自动生成 2020-04-19 17:49:21 +08:00
marvzhang
ef43e2a4ff 加入自动安装依赖 2020-04-19 16:56:07 +08:00
marvzhang
5473f2436c fixed issue 2020-03-30 13:09:38 +08:00
marvzhang
bff9356a0f 支持去重 2020-03-29 11:17:38 +08:00
marvzhang
83c657dc49 fixed https://github.com/crawlab-team/crawlab/issues/643 https://github.com/crawlab-team/crawlab/issues/640 2020-03-24 08:19:45 +08:00
marvzhang
fa62e1a2dc 加入权限管理 2020-03-20 17:43:11 +08:00
marvzhang
daef59be07 updated contributors 2020-03-19 11:13:25 +08:00
marvzhang
3123752038 配置加载demo爬虫 2020-03-11 08:16:52 +08:00
marvzhang
7315deefee 加入复制爬虫 2020-02-24 09:12:03 +08:00
marvzhang
6000667b7d updated dockerpush.yml 2020-02-23 09:19:18 +08:00
marvzhang
2e6b6e4e70 加入批量删除爬虫任务 2020-02-22 17:35:51 +08:00
marvzhang
2ce289c8f3 加入Git同步 2020-02-18 12:15:40 +08:00
marvzhang
534a29ebc3 加入添加scrapy爬虫 2020-02-17 14:06:16 +08:00
marvzhang
23f0b0c9e2 fixed unable to sync spiders to nodes error 2020-02-03 16:08:43 +08:00
marvzhang
2c410feed3 changed dir 2020-02-03 11:58:05 +08:00
marvzhang
0d7b97ead5 added demo for general spiders 2020-02-03 10:30:04 +08:00
marvzhang
6b70868326 added demo spiders 2020-02-03 09:21:41 +08:00
marvzhang
98e5ead285 添加demo爬虫 2020-02-02 22:56:11 +08:00
marvzhang
85a4df9bfb fixed https://github.com/crawlab-team/crawlab/issues/485 2020-02-01 19:18:30 +08:00
陈景阳
4825653ae0 fix 无法及时同步爬虫的问题 2020-01-28 15:43:57 +08:00
marvzhang
2c3f440c8e 加入爬虫列表排序 2020-01-06 13:18:29 +08:00
marvzhang
6e0718e646 添加文件管理功能（后端） 2019-12-25 20:49:43 +08:00
marvzhang
08c7aec49c 加入可配置爬虫阶段设置 2019-11-30 10:58:54 +08:00
marvzhang
38d103da39 加入可配置爬虫 2019-11-24 17:57:12 +08:00
marvzhang
442396a5ff fixed https://github.com/crawlab-team/crawlab/issues/315 2019-11-24 12:20:44 +08:00
yaziming
2f6b517356 refactor(all): refactor code 
remove redundant code and some code refactor
 2019-10-11 16:01:57 +08:00
陈景阳
7c75c24f65 fix md5值不一致的问题 2019-10-07 12:49:37 +08:00
陈景阳
0d10d6d045 fix 创建目录错误的问题 2019-10-07 12:21:32 +08:00
陈景阳
d80642aa0c fix 删除爬虫的问题 2019-09-30 12:09:37 +08:00
陈景阳
412e0fd983 fix bug 2019-09-26 21:13:25 +08:00
陈景阳
41e9edf940 完成爬虫列表 2019-09-26 20:53:05 +08:00
陈景阳
5416b7ac60 fix 上传的问题 2019-09-26 19:44:12 +08:00
陈景阳
79ee09aacb 完成爬虫获取 2019-09-26 19:12:02 +08:00
陈景阳
6c37731cca 优化爬虫获取逻辑 2019-09-26 16:43:32 +08:00
陈景阳
6df6155daf 优化爬虫获取逻辑 2019-09-26 16:26:32 +08:00
陈景阳
475bf4c69e 优化爬虫获取逻辑 2019-09-26 11:38:13 +08:00
陈景阳
9237e62a48 爬虫逻辑修改为从GridFS获取 2019-09-26 11:28:20 +08:00
陈景阳
f20039fefd fix 消息无法订阅问题 
fix 可能出现重复爬虫的问题
 2019-09-10 14:26:50 +08:00
yaziming
888a3cab79 refactor(backend): Use more efficient bytes to string methods and remove unnecessary type conversions 
detail:
    1. add utils.BytesToString function instead of string() convert bytes to string.
    2. use bytes.NewReader instead of strings.NewReader(string(sb)).
    3. use w.Body.Bytes() instead of []byte(w.Body.String()).
 2019-09-03 15:17:32 +08:00
陈景阳
4255b2f6b6 还原代码 2019-09-03 09:06:04 +08:00
陈景阳
9e9b117669 Merge branch 'develop' of https://github.com/crawlab-team/crawlab into develop 2019-09-03 08:57:21 +08:00
陈景阳
41c3b98333 还原代码 2019-09-02 18:14:34 +08:00
陈景阳
45aaaa2c46 fix 无法正常删除有问题的爬虫 2019-09-02 18:04:47 +08:00
陈景阳
69fc10fcc3 fix 如果从dir读取爬虫为空，则移除所有的爬虫 2019-09-02 17:37:48 +08:00
yaziming
443d697c6f backend: 
1. Mongo dial add 5 seconds connection timeout.
 2. Redis uses connection pool mode.
 3. Redis pool new connection have 10 seconds write timeout and read timeout and connection timeout.
 2019-09-01 17:18:08 +08:00
yaziming
9f43e08ff9 Backend: 
improve
     - AuthMiddleware 注入当前用户的信息
     - 增加Context服务支持快捷获取当前登录者信息
     - 重构Login/GetMe接口逻辑避免重复的数据库查询
     - 规范化error信息声明(向下兼容,旧代码可逐渐迁移规范化)
     - 修正部分不符合规范的代码
 2019-08-31 21:26:56 +08:00
陈景阳
e027aeb71f 添加日志打印 2019-08-31 17:56:42 +08:00
陈景阳
1a383dac17 fix 爬虫目录无法打开的问题 2019-08-31 17:04:49 +08:00
陈景阳
494601ab62 fix 节点注册异常情况 2019-08-31 13:49:34 +08:00
陈景阳
c634e2b5cf fix 前端控制台报错的问题 \ 
fix 无法打印中文的问题
 2019-08-31 12:04:12 +08:00