updated docs

This commit is contained in:
Marvin Zhang
2019-06-16 22:04:16 +08:00
parent 01a1b39ad1
commit 5bca22489e
91 changed files with 15220 additions and 5995 deletions

View File

View File

@@ -1,2 +1,4 @@
# Examples
## 样例
1. [与Scrapy集成](/Examples/ScrapyIntegration.md)

View File

@@ -0,0 +1,26 @@
### 与Scrapy集成
以下是Crawlab跟`Scrapy`集成的例子利用了Crawlab传过来的`task_id``collection_name`
```python
import os
from pymongo import MongoClient
MONGO_HOST = '192.168.99.100'
MONGO_PORT = 27017
MONGO_DB = 'crawlab_test'
# scrapy example in the pipeline
class JuejinPipeline(object):
mongo = MongoClient(host=MONGO_HOST, port=MONGO_PORT)
db = mongo[MONGO_DB]
col_name = os.environ.get('CRAWLAB_COLLECTION')
if not col_name:
col_name = 'test'
col = db[col_name]
def process_item(self, item, spider):
item['task_id'] = os.environ.get('CRAWLAB_TASK_ID')
self.col.save(item)
return item
```