git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-22 17:31:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
29 Commits 6 Branches 35 Tags
f3ff0a464af7d0899b71dad00c70281063a6d4c2
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Marvin Zhang f3ff0a464a updated execute spider logic
2019-03-01 12:07:28 +08:00
.idea/httpRequests
update SpiderDetail
2019-02-17 11:14:22 +08:00
bin
updated Dockerfile
2019-02-28 18:57:44 +08:00
constants
return existing nodes
2019-02-21 14:18:26 +08:00
db
updated Dockerfile
2019-02-28 18:57:44 +08:00
model
adapt changes for SpiderList and NodeList
2019-02-16 22:56:58 +08:00
routes
updated execute spider logic
2019-03-01 12:07:28 +08:00
spiders
fixed sorting issue
2019-02-25 19:32:43 +08:00
tasks
updated Dockerfile
2019-02-28 18:57:44 +08:00
test
update SpiderDetail
2019-02-17 11:14:22 +08:00
utils
updated Dockerfile
2019-02-28 18:57:44 +08:00
.gitignore
added celery functionality
2019-02-11 13:25:33 +08:00
app.py
updated execute spider logic
2019-03-01 12:07:28 +08:00
config.py
updated execute spider logic
2019-03-01 12:07:28 +08:00
Dockerfile
updated execute spider logic
2019-03-01 12:07:28 +08:00
LICENSE
Initial commit
2019-02-10 14:02:00 +08:00
README.md
Initial commit
2019-02-10 14:02:00 +08:00
requirements.txt
updated requirements.txt
2019-02-26 12:42:12 +08:00

README.md

crawlab

Centralized admin platform for managing web crawlers including scrapy, pyspider, webmagic and much more

Reference in New Issue View Git Blame Copy Permalink
Description
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
crawlabcrawlercrawling-tasksdockergoplatformscrapyscrapyd-uispiderspiders-managementweb-crawlerwebcrawlerwebspider
Readme BSD-3-Clause 30 MiB
Languages
Go 99.3%
Shell 0.6%
Dockerfile 0.1%