Merge pull request #500 from crawlab-team/develop

Develop

CHANGELOG-zh.md

@@ -1,15 +1,17 @@
 # 0.4.5 (unkown)
 ### 功能 / 优化
 - **交互式教程**. 引导用户了解 Crawlab 的主要功能.
+- **加入全局环境变量**. 可以设置全局环境变量，然后传入到所有爬虫程序中.
 
 ### Bug 修复
+- **无法找到爬虫文件错误**. [#485](https://github.com/crawlab-team/crawlab/issues/485)
 - **点击删除按钮导致跳转**. [#480](https://github.com/crawlab-team/crawlab/issues/480)
 - **无法在空爬虫里创建文件**. [#479](https://github.com/crawlab-team/crawlab/issues/479)
 - **下载结果错误**. [#465](https://github.com/crawlab-team/crawlab/issues/465)
 - **crawlab-sdk CLI 错误**. [#458](https://github.com/crawlab-team/crawlab/issues/458)
 - **页面刷新问题**. [#441](https://github.com/crawlab-team/crawlab/issues/441)
 - **修复“删除爬虫后获取所有爬虫”错误**.
-- **修复 i18n warning 警告**.
+- **修复 i18n 警告**.
 
 # 0.4.4 (2020-01-17)
 

CHANGELOG.md

@@ -1,8 +1,10 @@
 # 0.4.5 (unkown)
 ### Features / Enhancement
 - **Interactive Tutorial**. Guide users through the main functionalities of Crawlab.
+- **Global Environment Variables**. Allow users to set global environment variables, which will be passed into all spider programs.
 
 ### Bug Fixes
+- **Unable to find spider file error**. [#485](https://github.com/crawlab-team/crawlab/issues/485)
 - **Click delete button results in redirect**. [#480](https://github.com/crawlab-team/crawlab/issues/480)
 - **Unable to create files in an empty spider**. [#479](https://github.com/crawlab-team/crawlab/issues/479)
 - **Download results error**. [#465](https://github.com/crawlab-team/crawlab/issues/465)

backend/database/redis.go

@@ -83,7 +83,7 @@ func (r *Redis) HGet(collection string, key string) (string, error) {
 	defer utils.Close(c)
 
 	value, err2 := redis.String(c.Do("HGET", collection, key))
-	if err2 != nil {
+	if err2 != nil && err2 != redis.ErrNil {
 		log.Error(err2.Error())
 		debug.PrintStack()
 		return value, err2

backend/services/spider.go

@@ -69,6 +69,17 @@ func UploadSpiderToGridFsFromMaster(spider model.Spider) error {
 	spider.FileId = fid
 	_ = spider.Save()
 
+	// 获取爬虫同步实例
+	spiderSync := spider_handler.SpiderSync{
+		Spider: spider,
+	}
+
+	// 获取gfFile
+	gfFile2 := model.GetGridFs(spider.FileId)
+
+	// 生成MD5
+	spiderSync.CreateMd5File(gfFile2.Md5)
+
 	return nil
 }
 
@@ -101,6 +112,7 @@ func UploadToGridFs(fileName string, filePath string) (fid bson.ObjectId, err er
 	}
 	// 关闭文件，提交写入
 	if err = f.Close(); err != nil {
+		debug.PrintStack()
 		return "", err
 	}
 	// 文件ID