git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-21 17:21:09 +01:00
Code Issues Packages Projects Releases Wiki Activity

feat: Implement Crawlab MCP Server with tools and prompts

Browse Source 
- Added main server file (index.ts) to initialize the Crawlab MCP Server.
- Created prompts for spider analysis, task debugging, spider setup, and system monitoring.
- Developed tools for managing spiders, tasks, nodes, schedules, and system health.
- Implemented a mock client for testing server functionality.
- Added versioning support with version.ts.
- Created a test script (test-server.mjs) to validate tool configurations and server responses.
- Included build validation script (validate-build.mjs) to ensure proper setup and functionality.
- Configured TypeScript settings with tsconfig.json for better development experience.
This commit is contained in:
Marvin Zhang
2025-06-19 15:38:45 +08:00
parent 38c3a7534a
commit c29e21deec
21 changed files with 7612 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
mcp/.eslintrc.cjs Normal file
View File

@@ -0,0 +1,24 @@
module.exports = {
env: {
node: true,
es2022: true
},
extends: [
'eslint:recommended',
'@typescript-eslint/recommended'
],
parser: '@typescript-eslint/parser',
parserOptions: {
ecmaVersion: 'latest',
sourceType: 'module'
},
plugins: [
'@typescript-eslint'
],
rules: {
'@typescript-eslint/no-explicit-any': 'warn',
'@typescript-eslint/no-unused-vars': 'error',
'prefer-const': 'error',
'no-var': 'error'
}
};

10
mcp/.gitignore vendored Normal file
View File

@@ -0,0 +1,10 @@
node_modules/
dist/
*.log
.env
.env.local
.env.*.local
coverage/
*.tsbuildinfo
.DS_Store
Thumbs.db

12
mcp/.prettierrc Normal file
View File

@@ -0,0 +1,12 @@
{
"semi": true,
"trailingComma": "es5",
"singleQuote": true,
"printWidth": 80,
"tabWidth": 2,
"useTabs": false,
"bracketSpacing": true,
"jsxBracketSameLine": false,
"arrowParens": "avoid",
"endOfLine": "lf"
}

241
mcp/README.md Normal file
View File

@@ -0,0 +1,241 @@
# Crawlab MCP Server
A Model Context Protocol (MCP) server for interacting with [Crawlab](https://github.com/crawlab-team/crawlab), a distributed web crawler management platform. This server provides tools to manage spiders, tasks, schedules, and monitor your Crawlab cluster through an AI assistant.
## Features
### Spider Management
- List, create, update, and delete spiders
- Run spiders with custom parameters
- Browse and edit spider files
- View spider execution history
### Task Management
- Monitor running and completed tasks
- Cancel, restart, and delete tasks
- View task logs and results
- Filter tasks by spider, status, or time range
### Schedule Management
- Create and manage cron-based schedules
- Enable/disable schedules
- View scheduled task history
### Node Monitoring
- List cluster nodes and their status
- Monitor node health and availability
### System Monitoring
- Health checks and system status
- Comprehensive cluster overview
## Installation
```bash
npm install
npm run build
```
## Usage
### Basic Usage
```bash
# Start the MCP server
mcp-server-crawlab <crawlab_url> [api_token]
# Examples:
mcp-server-crawlab http://localhost:8080
mcp-server-crawlab https://crawlab.example.com your-api-token
```
### Environment Variables
You can also set the API token via environment variable:
```bash
export CRAWLAB_API_TOKEN=your-api-token
mcp-server-crawlab http://localhost:8080
```
### With MCP Inspector
For development and testing, you can use the MCP Inspector:
```bash
npm run inspect
```
### Integration with AI Assistants
This MCP server is designed to work with AI assistants that support the Model Context Protocol. Configure your AI assistant to connect to this server to enable Crawlab management capabilities.
## Available Tools
### Spider Tools
- `crawlab_list_spiders` - List all spiders with optional pagination
- `crawlab_get_spider` - Get detailed information about a specific spider
- `crawlab_create_spider` - Create a new spider
- `crawlab_update_spider` - Update spider configuration
- `crawlab_delete_spider` - Delete a spider
- `crawlab_run_spider` - Execute a spider
- `crawlab_list_spider_files` - Browse spider files and directories
- `crawlab_get_spider_file_content` - Read spider file content
- `crawlab_save_spider_file` - Save content to spider files
### Task Tools
- `crawlab_list_tasks` - List tasks with filtering options
- `crawlab_get_task` - Get detailed task information
- `crawlab_cancel_task` - Cancel a running task
- `crawlab_restart_task` - Restart a completed or failed task
- `crawlab_delete_task` - Delete a task
- `crawlab_get_task_logs` - Retrieve task execution logs
- `crawlab_get_task_results` - Get data collected by a task
### Schedule Tools
- `crawlab_list_schedules` - List all schedules
- `crawlab_get_schedule` - Get schedule details
- `crawlab_create_schedule` - Create a new cron schedule
- `crawlab_update_schedule` - Update schedule configuration
- `crawlab_delete_schedule` - Delete a schedule
- `crawlab_enable_schedule` - Enable a schedule
- `crawlab_disable_schedule` - Disable a schedule
### Node Tools
- `crawlab_list_nodes` - List cluster nodes
- `crawlab_get_node` - Get node details and status
### System Tools
- `crawlab_health_check` - Check system health
- `crawlab_system_status` - Get comprehensive system overview
## Available Prompts
The server includes several helpful prompts for common workflows:
### `spider-analysis`
Analyze spider performance and provide optimization insights.
**Parameters:**
- `spider_id` (required) - ID of the spider to analyze
- `time_range` (optional) - Time range for analysis (e.g., '7d', '30d', '90d')
### `task-debugging`
Debug failed tasks and identify root causes.
**Parameters:**
- `task_id` (required) - ID of the failed task
### `spider-setup`
Guide for creating and configuring new spiders.
**Parameters:**
- `spider_name` (required) - Name for the new spider
- `target_website` (optional) - Target website to scrape
- `spider_type` (optional) - Type of spider (scrapy, selenium, custom)
### `system-monitoring`
Monitor system health and performance.
**Parameters:**
- `focus_area` (optional) - Area to focus on (nodes, tasks, storage, overall)
## Example Interactions
### Create and Run a Spider
```
AI: I'll help you create a new spider for scraping news articles.
[Uses crawlab_create_spider with appropriate parameters]
[Uses crawlab_run_spider to test the spider]
[Uses crawlab_get_task_logs to check execution]
```
### Debug a Failed Task
```
User: "My task abc123 failed, can you help me debug it?"
[Uses task-debugging prompt]
[AI retrieves task details, logs, and provides analysis]
```
### Monitor System Health
```
User: "How is my Crawlab cluster performing?"
[Uses system-monitoring prompt]
[AI provides comprehensive health overview and recommendations]
```
## Configuration
### Crawlab Setup
Ensure your Crawlab instance is accessible and optionally configure API authentication:
1. Make sure Crawlab is running and accessible at the specified URL
2. If using authentication, obtain an API token from your Crawlab instance
3. Configure the token via command line argument or environment variable
### MCP Client Configuration
Add this server to your MCP client configuration:
```json
{
"servers": {
"crawlab": {
"command": "mcp-server-crawlab",
"args": ["http://localhost:8080", "your-api-token"]
}
}
}
```
## Development
### Building
```bash
npm run build
```
### Watching for Changes
```bash
npm run watch
```
### Testing
```bash
npm test
```
### Linting
```bash
npm run lint
npm run lint:fix
```
## Requirements
- Node.js 18+
- A running Crawlab instance
- Valid network access to the Crawlab API
## License
MIT License
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request
## Support
For issues and questions:
- Check the [Crawlab documentation](https://docs.crawlab.cn)
- Review the [MCP specification](https://modelcontextprotocol.io)
- Open an issue in this repository

15
mcp/jest.config.cjs Normal file
View File

@@ -0,0 +1,15 @@
module.exports = {
preset: 'ts-jest',
testEnvironment: 'node',
extensionsToTreatAsEsm: ['.ts'],
globals: {
'ts-jest': {
useESM: true
}
},
moduleNameMapping: {
'^@tools/(.*)$': '<rootDir>/src/tools/$1',
'^@types/(.*)$': '<rootDir>/src/types/$1',
'^@utils/(.*)$': '<rootDir>/src/utils/$1'
}
};

11
mcp/mcp-config.example.json Normal file
View File

@@ -0,0 +1,11 @@
{
"mcpServers": {
"crawlab": {
"command": "node",
"args": ["/home/marvin/projects/crawlab-team/crawlab-pro/mcp/dist/index.js", "http://localhost:8080"],
"env": {
"CRAWLAB_API_TOKEN": "your-api-token-here"
}
}
}
}

46
mcp/package.json Normal file
View File

@@ -0,0 +1,46 @@
{
"name": "@crawlab/mcp",
"version": "0.1.0",
"description": "MCP server for interacting with Crawlab web crawler management platform",
"license": "MIT",
"author": "Crawlab Team",
"type": "module",
"bin": {
"mcp-server-crawlab": "dist/index.js"
},
"files": [
"dist"
],
"scripts": {
"prebuild": "node -p \"'export const packageVersion = ' + JSON.stringify(require('./package.json').version) + ';\\r'\" > src/version.ts",
"build": "tsc && shx chmod +x dist/*.js",
"prepare": "npm run build",
"watch": "tsc --watch",
"inspect": "npx @modelcontextprotocol/inspector node dist/index.js",
"start": "node dist/index.js",
"lint": "eslint src/**/*.ts",
"lint:fix": "eslint src/**/*.ts --fix",
"clean": "shx rm -rf dist",
"test": "jest"
},
"dependencies": {
"@modelcontextprotocol/sdk": "^1.12.2",
"axios": "^1.6.7",
"tailwindcss": "^4.1.10",
"zod": "^3.25.63",
"zod-to-json-schema": "^3.24.5"
},
"devDependencies": {
"@modelcontextprotocol/inspector": "^0.14.0",
"@types/jest": "^29.5.14",
"@types/node": "^22",
"@typescript-eslint/eslint-plugin": "^6.21.0",
"@typescript-eslint/parser": "^6.21.0",
"eslint": "^8.57.0",
"jest": "^29.7.0",
"prettier": "^3.5.3",
"shx": "^0.4.0",
"ts-jest": "^29.4.0",
"typescript": "^5.8.3"
}
}

5523
mcp/pnpm-lock.yaml generated Normal file
View File

File diff suppressed because it is too large Load Diff

280
mcp/src/client.ts Normal file
View File

@@ -0,0 +1,280 @@
import axios, { AxiosInstance, AxiosResponse } from 'axios';
export interface CrawlabConfig {
url: string;
apiToken?: string;
timeout?: number;
}
export interface ApiResponse<T = any> {
success: boolean;
data?: T;
error?: string;
total?: number;
}
export interface PaginationParams {
page?: number;
size?: number;
}
export interface Spider {
_id: string;
name: string;
description?: string;
cmd: string;
param?: string;
project_id?: string;
type?: string;
tags?: string[];
created_ts?: Date;
updated_ts?: Date;
}
export interface Task {
_id: string;
spider_id: string;
spider_name?: string;
cmd: string;
param?: string;
priority?: number;
status: string;
log_path?: string;
result_count?: number;
error?: string;
start_ts?: Date;
end_ts?: Date;
created_ts?: Date;
updated_ts?: Date;
}
export interface Node {
_id: string;
name: string;
description?: string;
ip: string;
mac: string;
hostname: string;
status: string;
is_master: boolean;
created_ts?: Date;
updated_ts?: Date;
}
export interface Schedule {
_id: string;
name: string;
description?: string;
spider_id: string;
spider_name?: string;
cron: string;
cmd?: string;
param?: string;
enabled: boolean;
created_ts?: Date;
updated_ts?: Date;
}
export class CrawlabClient {
private client: AxiosInstance;
private baseURL: string;
constructor(baseURL: string, apiToken?: string, timeout: number = 30000) {
this.baseURL = baseURL.replace(/\/$/, ''); // Remove trailing slash
this.client = axios.create({
baseURL: `${this.baseURL}/api`,
timeout,
headers: {
'Content-Type': 'application/json',
...(apiToken && { Authorization: `Bearer ${apiToken}` }),
},
});
// Add response interceptor for error handling
this.client.interceptors.response.use(
(response: AxiosResponse) => response,
error => {
const message = error.response?.data?.error || error.message;
throw new Error(`Crawlab API Error: ${message}`);
}
);
}
// Spiders
async getSpiders(params?: PaginationParams): Promise<ApiResponse<Spider[]>> {
const response = await this.client.get('/spiders', { params });
return response.data;
}
async getSpider(id: string): Promise<ApiResponse<Spider>> {
const response = await this.client.get(`/spiders/${id}`);
return response.data;
}
async createSpider(spider: Partial<Spider>): Promise<ApiResponse<Spider>> {
const response = await this.client.post('/spiders', spider);
return response.data;
}
async updateSpider(
id: string,
spider: Partial<Spider>
): Promise<ApiResponse<Spider>> {
const response = await this.client.put(`/spiders/${id}`, spider);
return response.data;
}
async deleteSpider(id: string): Promise<ApiResponse<void>> {
const response = await this.client.delete(`/spiders/${id}`);
return response.data;
}
async runSpider(
id: string,
params?: {
cmd?: string;
param?: string;
priority?: number;
}
): Promise<ApiResponse<string[]>> {
const response = await this.client.post(`/spiders/${id}/run`, params);
return response.data;
}
async getSpiderFiles(id: string, path?: string): Promise<ApiResponse<any[]>> {
const params = path ? { path } : {};
const response = await this.client.get(`/spiders/${id}/files`, { params });
return response.data;
}
async getSpiderFileContent(
id: string,
path: string
): Promise<ApiResponse<string>> {
const response = await this.client.get(`/spiders/${id}/files/content`, {
params: { path },
});
return response.data;
}
async saveSpiderFile(
id: string,
path: string,
content: string
): Promise<ApiResponse<void>> {
const response = await this.client.post(`/spiders/${id}/files/save`, {
path,
content,
});
return response.data;
}
// Tasks
async getTasks(
params?: PaginationParams & { spider_id?: string; status?: string }
): Promise<ApiResponse<Task[]>> {
const response = await this.client.get('/tasks', { params });
return response.data;
}
async getTask(id: string): Promise<ApiResponse<Task>> {
const response = await this.client.get(`/tasks/${id}`);
return response.data;
}
async cancelTask(id: string): Promise<ApiResponse<void>> {
const response = await this.client.post(`/tasks/${id}/cancel`);
return response.data;
}
async restartTask(id: string): Promise<ApiResponse<string[]>> {
const response = await this.client.post(`/tasks/${id}/restart`);
return response.data;
}
async deleteTask(id: string): Promise<ApiResponse<void>> {
const response = await this.client.delete(`/tasks/${id}`);
return response.data;
}
async getTaskLogs(
id: string,
params?: { page?: number; size?: number }
): Promise<ApiResponse<string[]>> {
const response = await this.client.get(`/tasks/${id}/logs`, { params });
return response.data;
}
async getTaskResults(
id: string,
params?: PaginationParams
): Promise<ApiResponse<any[]>> {
const response = await this.client.get(`/tasks/${id}/results`, { params });
return response.data;
}
// Nodes
async getNodes(params?: PaginationParams): Promise<ApiResponse<Node[]>> {
const response = await this.client.get('/nodes', { params });
return response.data;
}
async getNode(id: string): Promise<ApiResponse<Node>> {
const response = await this.client.get(`/nodes/${id}`);
return response.data;
}
// Schedules
async getSchedules(
params?: PaginationParams
): Promise<ApiResponse<Schedule[]>> {
const response = await this.client.get('/schedules', { params });
return response.data;
}
async getSchedule(id: string): Promise<ApiResponse<Schedule>> {
const response = await this.client.get(`/schedules/${id}`);
return response.data;
}
async createSchedule(
schedule: Partial<Schedule>
): Promise<ApiResponse<Schedule>> {
const response = await this.client.post('/schedules', schedule);
return response.data;
}
async updateSchedule(
id: string,
schedule: Partial<Schedule>
): Promise<ApiResponse<Schedule>> {
const response = await this.client.put(`/schedules/${id}`, schedule);
return response.data;
}
async deleteSchedule(id: string): Promise<ApiResponse<void>> {
const response = await this.client.delete(`/schedules/${id}`);
return response.data;
}
async enableSchedule(id: string): Promise<ApiResponse<void>> {
const response = await this.client.post(`/schedules/${id}/enable`);
return response.data;
}
async disableSchedule(id: string): Promise<ApiResponse<void>> {
const response = await this.client.post(`/schedules/${id}/disable`);
return response.data;
}
// Health check
async healthCheck(): Promise<boolean> {
try {
const response = await this.client.get('/health');
return response.status === 200;
} catch {
return false;
}
}
}

46
mcp/src/index.ts Normal file
View File

@@ -0,0 +1,46 @@
#!/usr/bin/env node
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { configurePrompts } from './prompts';
import { configureAllTools } from './tools';
import { CrawlabClient } from './client';
import { packageVersion } from './version';
const args = process.argv.slice(2);
if (args.length < 1) {
console.error('Usage: mcp-server-crawlab <crawlab_url> [api_token]');
console.error('Example: mcp-server-crawlab http://localhost:8080');
console.error(
'Example: mcp-server-crawlab http://localhost:8080 your-api-token'
);
process.exit(1);
}
const crawlabUrl = args[0];
const apiToken = args[1] || process.env.CRAWLAB_API_TOKEN;
async function main() {
const server = new McpServer({
name: 'Crawlab MCP Server',
version: packageVersion,
});
// Initialize Crawlab client
const client = new CrawlabClient(crawlabUrl, apiToken);
// Configure prompts and tools
configurePrompts(server);
configureAllTools(server, client);
const transport = new StdioServerTransport();
console.error(`Crawlab MCP Server version: ${packageVersion}`);
console.error(`Connecting to Crawlab at: ${crawlabUrl}`);
await server.connect(transport);
}
main().catch(error => {
console.error('Fatal error in main():', error);
process.exit(1);
});

142
mcp/src/prompts.ts Normal file
View File

@@ -0,0 +1,142 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
export function configurePrompts(server: McpServer) {
server.prompt(
"spider-analysis",
"Analyze spider performance and provide insights",
{
spider_id: z.string().describe("ID of the spider to analyze"),
time_range: z.string().optional().describe("Time range for analysis (e.g., '7d', '30d', '90d')"),
},
async (args) => {
const spiderId = args.spider_id as string;
const timeRange = (args.time_range as string) || "7d";
return {
description: `Spider Performance Analysis for Spider ${spiderId}`,
messages: [
{
role: "user",
content: {
type: "text",
text: `Please analyze the performance of spider "${spiderId}" over the last ${timeRange}.
I would like to understand:
1. Task success/failure rates
2. Average execution time
3. Data collection patterns
4. Any errors or issues
5. Recommendations for optimization
Please use the available Crawlab tools to gather task data, logs, and results for this analysis.`
}
}
]
};
}
);
server.prompt(
"task-debugging",
"Help debug a failed task",
{
task_id: z.string().describe("ID of the failed task to debug"),
},
async (args) => {
const taskId = args.task_id as string;
return {
description: `Task Debugging Assistant for Task ${taskId}`,
messages: [
{
role: "user",
content: {
type: "text",
text: `I have a failed task with ID "${taskId}" that needs debugging.
Please help me:
1. Get the task details and status
2. Retrieve and analyze the task logs
3. Identify the root cause of the failure
4. Suggest specific fixes or improvements
5. Provide guidance on preventing similar issues
Use the Crawlab tools to gather all relevant information about this task.`
}
}
]
};
}
);
server.prompt(
"spider-setup",
"Guide for setting up a new spider",
{
spider_name: z.string().describe("Name for the new spider"),
target_website: z.string().optional().describe("Target website or data source"),
spider_type: z.string().optional().describe("Type of spider (scrapy, selenium, custom, etc.)"),
},
async (args) => {
const spiderName = args.spider_name as string;
const targetWebsite = args.target_website as string;
const spiderType = (args.spider_type as string) || "scrapy";
return {
description: `Spider Setup Guide for "${spiderName}"`,
messages: [
{
role: "user",
content: {
type: "text",
text: `I want to create a new ${spiderType} spider named "${spiderName}"${targetWebsite ? ` to scrape ${targetWebsite}` : ""}.
Please help me:
1. Create the spider configuration
2. Set up the basic file structure
3. Configure appropriate settings
4. Create a simple test to verify the setup
5. Set up a schedule if needed
Guide me through the process step by step using the available Crawlab tools.`
}
}
]
};
}
);
server.prompt(
"system-monitoring",
"Monitor Crawlab system health and performance",
{
focus_area: z.string().optional().describe("Specific area to focus on (nodes, tasks, storage, overall)"),
},
async (args) => {
const focusArea = (args.focus_area as string) || "overall";
return {
description: `Crawlab System Monitoring - ${focusArea}`,
messages: [
{
role: "user",
content: {
type: "text",
text: `Please perform a system health check focusing on ${focusArea}.
I need you to:
1. Check node status and availability
2. Review recent task performance
3. Identify any system bottlenecks
4. Check for error patterns
5. Provide recommendations for optimization
Use the Crawlab monitoring tools to gather comprehensive system information.`
}
}
]
};
}
);
}

16
mcp/src/tools.ts Normal file
View File

@@ -0,0 +1,16 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { CrawlabClient } from "./client.js";
import { configureSpiderTools } from "./tools/spiders.js";
import { configureTaskTools } from "./tools/tasks.js";
import { configureNodeTools } from "./tools/nodes.js";
import { configureScheduleTools } from "./tools/schedules.js";
import { configureSystemTools } from "./tools/system.js";
export function configureAllTools(server: McpServer, client: CrawlabClient) {
configureSpiderTools(server, client);
configureTaskTools(server, client);
configureNodeTools(server, client);
configureScheduleTools(server, client);
configureSystemTools(server, client);
}

75
mcp/src/tools/nodes.ts Normal file
View File

@@ -0,0 +1,75 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { CrawlabClient } from "../client";
import { z } from "zod";
const NODE_TOOLS = {
list_nodes: "crawlab_list_nodes",
get_node: "crawlab_get_node",
};
export function configureNodeTools(server: McpServer, client: CrawlabClient) {
server.tool(
NODE_TOOLS.list_nodes,
"List all nodes in Crawlab cluster",
{
page: z.number().optional().describe("Page number for pagination (default: 1)"),
size: z.number().optional().describe("Number of nodes per page (default: 10)"),
},
async ({ page, size }) => {
try {
const response = await client.getNodes({ page, size });
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error listing nodes: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
NODE_TOOLS.get_node,
"Get details of a specific node",
{
node_id: z.string().describe("The ID of the node to retrieve"),
},
async ({ node_id }) => {
try {
const response = await client.getNode(node_id);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting node: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
}
export { NODE_TOOLS };

266
mcp/src/tools/schedules.ts Normal file
View File

@@ -0,0 +1,266 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { CrawlabClient } from "../client.js";
import { z } from "zod";
const SCHEDULE_TOOLS = {
list_schedules: "crawlab_list_schedules",
get_schedule: "crawlab_get_schedule",
create_schedule: "crawlab_create_schedule",
update_schedule: "crawlab_update_schedule",
delete_schedule: "crawlab_delete_schedule",
enable_schedule: "crawlab_enable_schedule",
disable_schedule: "crawlab_disable_schedule",
};
export function configureScheduleTools(server: McpServer, client: CrawlabClient) {
server.tool(
SCHEDULE_TOOLS.list_schedules,
"List all schedules in Crawlab",
{
page: z.number().optional().describe("Page number for pagination (default: 1)"),
size: z.number().optional().describe("Number of schedules per page (default: 10)"),
},
async ({ page, size }) => {
try {
const response = await client.getSchedules({ page, size });
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error listing schedules: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SCHEDULE_TOOLS.get_schedule,
"Get details of a specific schedule",
{
schedule_id: z.string().describe("The ID of the schedule to retrieve"),
},
async ({ schedule_id }) => {
try {
const response = await client.getSchedule(schedule_id);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting schedule: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SCHEDULE_TOOLS.create_schedule,
"Create a new schedule",
{
name: z.string().describe("Name of the schedule"),
description: z.string().optional().describe("Description of the schedule"),
spider_id: z.string().describe("ID of the spider to schedule"),
cron: z.string().describe("Cron expression for the schedule (e.g., '0 0 * * *' for daily at midnight)"),
cmd: z.string().optional().describe("Command to override for scheduled runs"),
param: z.string().optional().describe("Parameters to override for scheduled runs"),
enabled: z.boolean().optional().describe("Whether the schedule is enabled (default: true)"),
},
async ({ name, description, spider_id, cron, cmd, param, enabled = true }) => {
try {
const scheduleData = {
name,
description,
spider_id,
cron,
cmd,
param,
enabled,
};
const response = await client.createSchedule(scheduleData);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error creating schedule: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SCHEDULE_TOOLS.update_schedule,
"Update an existing schedule",
{
schedule_id: z.string().describe("The ID of the schedule to update"),
name: z.string().optional().describe("New name for the schedule"),
description: z.string().optional().describe("New description for the schedule"),
spider_id: z.string().optional().describe("New spider ID for the schedule"),
cron: z.string().optional().describe("New cron expression for the schedule"),
cmd: z.string().optional().describe("New command for the schedule"),
param: z.string().optional().describe("New parameters for the schedule"),
enabled: z.boolean().optional().describe("Whether the schedule is enabled"),
},
async ({ schedule_id, name, description, spider_id, cron, cmd, param, enabled }) => {
try {
const updateData = {
...(name && { name }),
...(description && { description }),
...(spider_id && { spider_id }),
...(cron && { cron }),
...(cmd && { cmd }),
...(param && { param }),
...(enabled !== undefined && { enabled }),
};
const response = await client.updateSchedule(schedule_id, updateData);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error updating schedule: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SCHEDULE_TOOLS.delete_schedule,
"Delete a schedule",
{
schedule_id: z.string().describe("The ID of the schedule to delete"),
},
async ({ schedule_id }) => {
try {
const response = await client.deleteSchedule(schedule_id);
return {
content: [
{
type: "text",
text: `Schedule ${schedule_id} deleted successfully.`,
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error deleting schedule: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SCHEDULE_TOOLS.enable_schedule,
"Enable a schedule",
{
schedule_id: z.string().describe("The ID of the schedule to enable"),
},
async ({ schedule_id }) => {
try {
const response = await client.enableSchedule(schedule_id);
return {
content: [
{
type: "text",
text: `Schedule ${schedule_id} enabled successfully.`,
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error enabling schedule: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SCHEDULE_TOOLS.disable_schedule,
"Disable a schedule",
{
schedule_id: z.string().describe("The ID of the schedule to disable"),
},
async ({ schedule_id }) => {
try {
const response = await client.disableSchedule(schedule_id);
return {
content: [
{
type: "text",
text: `Schedule ${schedule_id} disabled successfully.`,
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error disabling schedule: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
}
export { SCHEDULE_TOOLS };

342
mcp/src/tools/spiders.ts Normal file
View File

@@ -0,0 +1,342 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { CrawlabClient } from "../client.js";
import { z } from "zod";
const SPIDER_TOOLS = {
list_spiders: "crawlab_list_spiders",
get_spider: "crawlab_get_spider",
create_spider: "crawlab_create_spider",
update_spider: "crawlab_update_spider",
delete_spider: "crawlab_delete_spider",
run_spider: "crawlab_run_spider",
list_spider_files: "crawlab_list_spider_files",
get_spider_file_content: "crawlab_get_spider_file_content",
save_spider_file: "crawlab_save_spider_file",
};
export function configureSpiderTools(server: McpServer, client: CrawlabClient) {
server.tool(
SPIDER_TOOLS.list_spiders,
"List all spiders in Crawlab",
{
page: z.number().optional().describe("Page number for pagination (default: 1)"),
size: z.number().optional().describe("Number of spiders per page (default: 10)"),
},
async ({ page, size }) => {
try {
const response = await client.getSpiders({ page, size });
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error listing spiders: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.get_spider,
"Get details of a specific spider",
{
spider_id: z.string().describe("The ID of the spider to retrieve"),
},
async ({ spider_id }) => {
try {
const response = await client.getSpider(spider_id);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting spider: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.create_spider,
"Create a new spider",
{
name: z.string().describe("Name of the spider"),
description: z.string().optional().describe("Description of the spider"),
cmd: z.string().describe("Command to execute the spider"),
param: z.string().optional().describe("Parameters for the spider command"),
project_id: z.string().optional().describe("Project ID to associate with the spider"),
type: z.string().optional().describe("Type of spider (e.g., 'scrapy', 'selenium', 'custom')"),
tags: z.array(z.string()).optional().describe("Tags for categorizing the spider"),
},
async ({ name, description, cmd, param, project_id, type, tags }) => {
try {
const spiderData = {
name,
description,
cmd,
param,
project_id,
type,
tags,
};
const response = await client.createSpider(spiderData);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error creating spider: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.update_spider,
"Update an existing spider",
{
spider_id: z.string().describe("The ID of the spider to update"),
name: z.string().optional().describe("New name for the spider"),
description: z.string().optional().describe("New description for the spider"),
cmd: z.string().optional().describe("New command to execute the spider"),
param: z.string().optional().describe("New parameters for the spider command"),
project_id: z.string().optional().describe("New project ID to associate with the spider"),
type: z.string().optional().describe("New type of spider"),
tags: z.array(z.string()).optional().describe("New tags for the spider"),
},
async ({ spider_id, name, description, cmd, param, project_id, type, tags }) => {
try {
const updateData = {
...(name && { name }),
...(description && { description }),
...(cmd && { cmd }),
...(param && { param }),
...(project_id && { project_id }),
...(type && { type }),
...(tags && { tags }),
};
const response = await client.updateSpider(spider_id, updateData);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error updating spider: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.delete_spider,
"Delete a spider",
{
spider_id: z.string().describe("The ID of the spider to delete"),
},
async ({ spider_id }) => {
try {
const response = await client.deleteSpider(spider_id);
return {
content: [
{
type: "text",
text: `Spider ${spider_id} deleted successfully.`,
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error deleting spider: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.run_spider,
"Run a spider",
{
spider_id: z.string().describe("The ID of the spider to run"),
cmd: z.string().optional().describe("Override command for this run"),
param: z.string().optional().describe("Override parameters for this run"),
priority: z.number().optional().describe("Priority of the task (1-10, higher = more priority)"),
},
async ({ spider_id, cmd, param, priority }) => {
try {
const runData = {
...(cmd && { cmd }),
...(param && { param }),
...(priority && { priority }),
};
const response = await client.runSpider(spider_id, runData);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error running spider: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.list_spider_files,
"List files in a spider directory",
{
spider_id: z.string().describe("The ID of the spider"),
path: z.string().optional().describe("Path within the spider directory (default: root)"),
},
async ({ spider_id, path }) => {
try {
const response = await client.getSpiderFiles(spider_id, path);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error listing spider files: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.get_spider_file_content,
"Get the content of a spider file",
{
spider_id: z.string().describe("The ID of the spider"),
file_path: z.string().describe("Path to the file within the spider directory"),
},
async ({ spider_id, file_path }) => {
try {
const response = await client.getSpiderFileContent(spider_id, file_path);
return {
content: [
{
type: "text",
text: response.data || "File is empty",
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting file content: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SPIDER_TOOLS.save_spider_file,
"Save content to a spider file",
{
spider_id: z.string().describe("The ID of the spider"),
file_path: z.string().describe("Path to the file within the spider directory"),
content: z.string().describe("Content to save to the file"),
},
async ({ spider_id, file_path, content }) => {
try {
const response = await client.saveSpiderFile(spider_id, file_path, content);
return {
content: [
{
type: "text",
text: `File ${file_path} saved successfully for spider ${spider_id}.`,
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error saving file: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
}
export { SPIDER_TOOLS };

101
mcp/src/tools/system.ts Normal file
View File

@@ -0,0 +1,101 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { CrawlabClient } from "../client.js";
const SYSTEM_TOOLS = {
health_check: "crawlab_health_check",
system_status: "crawlab_system_status",
};
export function configureSystemTools(server: McpServer, client: CrawlabClient) {
server.tool(
SYSTEM_TOOLS.health_check,
"Check if Crawlab system is healthy and reachable",
{},
async () => {
try {
const isHealthy = await client.healthCheck();
return {
content: [
{
type: "text",
text: JSON.stringify({
healthy: isHealthy,
status: isHealthy ? "System is healthy and reachable" : "System is not reachable",
timestamp: new Date().toISOString(),
}, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error checking system health: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
SYSTEM_TOOLS.system_status,
"Get comprehensive system status including nodes, recent tasks, and schedules",
{},
async () => {
try {
// Get system overview data
const [nodesResponse, tasksResponse, schedulesResponse] = await Promise.all([
client.getNodes({ page: 1, size: 50 }),
client.getTasks({ page: 1, size: 20 }),
client.getSchedules({ page: 1, size: 50 }),
]);
const summary = {
timestamp: new Date().toISOString(),
nodes: {
total: nodesResponse.total || 0,
online: nodesResponse.data?.filter(node => node.status === 'online').length || 0,
offline: nodesResponse.data?.filter(node => node.status === 'offline').length || 0,
},
tasks: {
total_recent: tasksResponse.total || 0,
running: tasksResponse.data?.filter(task => task.status === 'running').length || 0,
pending: tasksResponse.data?.filter(task => task.status === 'pending').length || 0,
completed: tasksResponse.data?.filter(task => task.status === 'success').length || 0,
failed: tasksResponse.data?.filter(task => task.status === 'error').length || 0,
},
schedules: {
total: schedulesResponse.total || 0,
enabled: schedulesResponse.data?.filter(schedule => schedule.enabled).length || 0,
disabled: schedulesResponse.data?.filter(schedule => !schedule.enabled).length || 0,
},
health: await client.healthCheck(),
};
return {
content: [
{
type: "text",
text: JSON.stringify(summary, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting system status: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
}
export { SYSTEM_TOOLS };

243
mcp/src/tools/tasks.ts Normal file
View File

@@ -0,0 +1,243 @@
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { CrawlabClient } from "../client.js";
import { z } from "zod";
const TASK_TOOLS = {
list_tasks: "crawlab_list_tasks",
get_task: "crawlab_get_task",
cancel_task: "crawlab_cancel_task",
restart_task: "crawlab_restart_task",
delete_task: "crawlab_delete_task",
get_task_logs: "crawlab_get_task_logs",
get_task_results: "crawlab_get_task_results",
};
export function configureTaskTools(server: McpServer, client: CrawlabClient) {
server.tool(
TASK_TOOLS.list_tasks,
"List tasks in Crawlab",
{
page: z.number().optional().describe("Page number for pagination (default: 1)"),
size: z.number().optional().describe("Number of tasks per page (default: 10)"),
spider_id: z.string().optional().describe("Filter by spider ID"),
status: z.string().optional().describe("Filter by task status (pending, running, success, error, cancelled)"),
},
async ({ page, size, spider_id, status }) => {
try {
const response = await client.getTasks({ page, size, spider_id, status });
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error listing tasks: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
TASK_TOOLS.get_task,
"Get details of a specific task",
{
task_id: z.string().describe("The ID of the task to retrieve"),
},
async ({ task_id }) => {
try {
const response = await client.getTask(task_id);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting task: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
TASK_TOOLS.cancel_task,
"Cancel a running task",
{
task_id: z.string().describe("The ID of the task to cancel"),
},
async ({ task_id }) => {
try {
const response = await client.cancelTask(task_id);
return {
content: [
{
type: "text",
text: `Task ${task_id} cancelled successfully.`,
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error cancelling task: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
TASK_TOOLS.restart_task,
"Restart a task",
{
task_id: z.string().describe("The ID of the task to restart"),
},
async ({ task_id }) => {
try {
const response = await client.restartTask(task_id);
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error restarting task: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
TASK_TOOLS.delete_task,
"Delete a task",
{
task_id: z.string().describe("The ID of the task to delete"),
},
async ({ task_id }) => {
try {
const response = await client.deleteTask(task_id);
return {
content: [
{
type: "text",
text: `Task ${task_id} deleted successfully.`,
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error deleting task: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
TASK_TOOLS.get_task_logs,
"Get logs from a task",
{
task_id: z.string().describe("The ID of the task"),
page: z.number().optional().describe("Page number for pagination (default: 1)"),
size: z.number().optional().describe("Number of log lines per page (default: 100)"),
},
async ({ task_id, page, size }) => {
try {
const response = await client.getTaskLogs(task_id, { page, size });
return {
content: [
{
type: "text",
text: Array.isArray(response.data)
? response.data.join('\n')
: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting task logs: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
server.tool(
TASK_TOOLS.get_task_results,
"Get results data from a task",
{
task_id: z.string().describe("The ID of the task"),
page: z.number().optional().describe("Page number for pagination (default: 1)"),
size: z.number().optional().describe("Number of results per page (default: 10)"),
},
async ({ task_id, page, size }) => {
try {
const response = await client.getTaskResults(task_id, { page, size });
return {
content: [
{
type: "text",
text: JSON.stringify(response, null, 2),
},
],
};
} catch (error) {
return {
content: [
{
type: "text",
text: `Error getting task results: ${error instanceof Error ? error.message : String(error)}`,
},
],
isError: true,
};
}
}
);
}
export { TASK_TOOLS };

1
mcp/src/version.ts Normal file
View File

@@ -0,0 +1 @@
export const packageVersion = "0.1.0";

135
mcp/test-server.mjs Normal file
View File

@@ -0,0 +1,135 @@
#!/usr/bin/env node
/**
* Test script for Crawlab MCP Server
* This script validates that all tools are properly configured and can handle basic requests
*/
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { configureAllTools } from "./dist/tools.js";
import { CrawlabClient } from "./dist/client.js";
// Mock Crawlab client for testing
class MockCrawlabClient extends CrawlabClient {
constructor() {
super("http://localhost:8080", "test-token");
}
async getSpiders() {
return {
success: true,
data: [
{
_id: "test-spider-1",
name: "Test Spider",
description: "A test spider",
cmd: "scrapy crawl test",
type: "scrapy",
created_ts: new Date()
}
],
total: 1
};
}
async getTasks() {
return {
success: true,
data: [
{
_id: "test-task-1",
spider_id: "test-spider-1",
spider_name: "Test Spider",
cmd: "scrapy crawl test",
status: "success",
created_ts: new Date()
}
],
total: 1
};
}
async getNodes() {
return {
success: true,
data: [
{
_id: "test-node-1",
name: "Master Node",
ip: "127.0.0.1",
mac: "00:00:00:00:00:00",
hostname: "localhost",
status: "online",
is_master: true
}
],
total: 1
};
}
async healthCheck() {
return true;
}
}
async function testMcpServer() {
console.log("🧪 Testing Crawlab MCP Server...\n");
const server = new McpServer({
name: "Crawlab MCP Server Test",
version: "0.1.0-test",
});
const mockClient = new MockCrawlabClient();
configureAllTools(server, mockClient);
// Get list of registered tools
const tools = server.listTools();
console.log(`✅ Server initialized with ${tools.length} tools:`);
tools.forEach((tool, index) => {
console.log(` ${index + 1}. ${tool.name}: ${tool.description}`);
});
console.log("\n🔧 Testing sample tool executions...\n");
// Test spider listing tool
try {
const spiderResult = await server.callTool("crawlab_list_spiders", {});
console.log("✅ crawlab_list_spiders - SUCCESS");
console.log(" Sample output:", JSON.stringify(spiderResult, null, 2).substring(0, 200) + "...\n");
} catch (error) {
console.log("❌ crawlab_list_spiders - FAILED");
console.log(" Error:", error.message, "\n");
}
// Test task listing tool
try {
const taskResult = await server.callTool("crawlab_list_tasks", {});
console.log("✅ crawlab_list_tasks - SUCCESS");
console.log(" Sample output:", JSON.stringify(taskResult, null, 2).substring(0, 200) + "...\n");
} catch (error) {
console.log("❌ crawlab_list_tasks - FAILED");
console.log(" Error:", error.message, "\n");
}
// Test node listing tool
try {
const nodeResult = await server.callTool("crawlab_list_nodes", {});
console.log("✅ crawlab_list_nodes - SUCCESS");
console.log(" Sample output:", JSON.stringify(nodeResult, null, 2).substring(0, 200) + "...\n");
} catch (error) {
console.log("❌ crawlab_list_nodes - FAILED");
console.log(" Error:", error.message, "\n");
}
console.log("🎉 Test completed! The MCP server appears to be working correctly.");
console.log("\n📋 Next steps:");
console.log(" 1. Configure your MCP client (e.g., Claude Desktop) to use this server");
console.log(" 2. Point it to a real Crawlab instance");
console.log(" 3. Set up proper API authentication");
console.log(" 4. Start managing your Crawlab spiders through AI!");
}
// Run the test
testMcpServer().catch(console.error);

34
mcp/tsconfig.json Normal file
View File

@@ -0,0 +1,34 @@
{
"compilerOptions": {
"outDir": "dist",
"rootDir": "src",
"baseUrl": "src",
"target": "ES2022",
"module": "ESNext",
"moduleResolution": "bundler",
"strict": true,
"esModuleInterop": true,
"allowSyntheticDefaultImports": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"resolveJsonModule": true,
"paths": {
"@tools/*": [
"tools/*"
],
"@types/*": [
"types/*"
],
"@utils/*": [
"utils/*"
]
}
},
"include": [
"./src/**/*"
],
"exclude": [
"node_modules",
"dist"
]
}

49
mcp/validate-build.mjs Normal file
View File

@@ -0,0 +1,49 @@
#!/usr/bin/env node
/**
* Simple build validation test for Crawlab MCP Server
*/
import { CrawlabClient } from "./dist/client.js";
console.log("🧪 Testing Crawlab MCP Server build...\n");
// Test that we can instantiate the client
try {
const client = new CrawlabClient("http://localhost:8080", "test-token");
console.log("✅ CrawlabClient class - OK");
} catch (error) {
console.log("❌ CrawlabClient class - FAILED");
console.log(" Error:", error.message);
process.exit(1);
}
// Test that the main entry point is valid
try {
const entryPoint = await import("./dist/index.js");
console.log("✅ Main entry point - OK");
} catch (error) {
console.log("❌ Main entry point - FAILED");
console.log(" Error:", error.message);
process.exit(1);
}
// Test tools module
try {
const toolsModule = await import("./dist/tools.js");
if (typeof toolsModule.configureAllTools === 'function') {
console.log("✅ Tools configuration - OK");
} else {
console.log("❌ Tools configuration - FAILED (configureAllTools not a function)");
}
} catch (error) {
console.log("❌ Tools configuration - FAILED");
console.log(" Error:", error.message);
}
console.log("\n🎉 Build validation completed successfully!");
console.log("\n📋 Ready to use:");
console.log(" npm start <crawlab_url> [api_token]");
console.log(" Example: npm start http://localhost:8080 your-token");
console.log("\n Or use the binary directly:");
console.log(" ./dist/index.js http://localhost:8080 your-token");