Marvin Zhang
babecc46c0
refactor: update DependencySetupDialog to use language key for tag label; modify useDependencyList to dispatch node retrieval on mount
2025-08-08 11:12:05 +08:00
Marvin Zhang
45913ad7e4
refactor: implement health service for master and worker nodes; add health check script and integrate health checks into service lifecycle
2025-08-08 00:05:00 +08:00
Marvin Zhang
78f9e0ca8d
refactor: update task worker pool to support dynamic max workers and improve queue management; enhance configuration defaults for node runners and task queue size
2025-08-07 18:16:23 +08:00
Marvin Zhang
6340a9b880
refactor: Move context initialization for graceful shutdown to appropriate locations
2025-08-07 17:27:11 +08:00
Marvin Zhang
6912b92501
refactor: enhance context handling across task runner and service components; ensure proper cancellation chains and prevent goroutine leaks
2025-08-07 15:40:48 +08:00
Marvin Zhang
e1251d808b
refactor: update method receivers to value type for cleanup and connection methods; enhance context usage for task client operations
2025-08-07 11:53:42 +08:00
Marvin Zhang
f5f83a010f
refactor: update metric list structure for improved layout and responsiveness; enhance styling with grid display
2025-08-07 11:12:50 +08:00
Marvin Zhang
d042bc8cd7
refactor: improve connection readiness check and enhance goroutine management in gRPC client; ensure proper context handling in stream listeners
2025-08-07 11:12:46 +08:00
Marvin Zhang
060396af3d
refactor: enhance data structure annotations and improve layout responsiveness in Home component
2025-08-07 09:58:11 +08:00
Marvin Zhang
44dd68918f
refactor: improve goroutine management and context handling in task and stream operations; ensure graceful shutdown and prevent leaks
2025-08-07 00:16:46 +08:00
Marvin Zhang
784ffc8b52
feat: implement task management service operations, stream manager, and worker pool
...
- Added service_operations.go for task management including run, cancel, and execution logic.
- Introduced stream_manager.go to handle task streams and manage cancellation signals.
- Created worker_pool.go to manage a bounded pool of workers for executing tasks concurrently.
- Implemented graceful shutdown and cleanup mechanisms for task runners and streams.
- Enhanced error handling and logging throughout the task management process.
2025-08-06 18:29:08 +08:00
Marvin Zhang
3678d14082
feat: implement bounded goroutine pools for task execution and notification handling; enhance task scheduler with graceful shutdown and cleanup routines; update metric component for new time range options
2025-08-06 17:57:37 +08:00
Marvin Zhang
9745129e33
feat: configure test database as master node for testing
2025-07-23 15:28:51 +08:00
Marvin Zhang
cf5ec81250
fix: add nil checks for logger in config initialization and logging methods
2025-07-23 15:07:39 +08:00
Marvin Zhang
a2d13fae36
feat: temporarily disable batch file saving route and implement alternative handler in spider controller
2025-07-23 14:55:04 +08:00
Marvin Zhang
b4288b08a5
fix: Update FsFileInfo to use pointer slices for children and remove redundant getter methods
2025-07-23 09:42:07 +08:00
Marvin Zhang
3c3ff09723
feat: enhance gRPC client with health check functionality and improved connection handling
2025-07-09 14:38:53 +08:00
Marvin Zhang
20ba390cf6
refactor: improve mongo client connection error logging format and remove redundant gRPC server start in MasterService
2025-07-09 14:06:10 +08:00
Marvin Zhang
46c0cd6298
refactor: update gRPC client access patterns to use safe getter methods for improved error handling
2025-07-08 18:08:46 +08:00
Marvin Zhang
8bd3ef0b72
feat: add goroutine count metric to the MetricService and update related files
2025-07-08 14:08:31 +08:00
Marvin Zhang
7e4187f388
chore: add TypeScript declaration files for various components and services
2025-07-08 14:08:13 +08:00
Marvin Zhang
00daa0ed96
fix: enhance gRPC client reconnection logic and add goroutine monitoring for potential leaks
2025-07-08 13:39:39 +08:00
Marvin Zhang
f8e9c45a85
fix: enhance gRPC client connection management with circuit breaker and keep-alive settings
2025-07-08 13:34:43 +08:00
Marvin Zhang
92046a8c2e
fix: improve task cancellation and connection health check logic with timeout handling
2025-06-27 14:02:24 +08:00
Marvin Zhang
9f251f3ebe
fix: enhance task cancellation logic with graceful termination and stuck task cleanup
2025-06-27 13:50:21 +08:00
Marvin Zhang
89514b0154
feat: implement zombie process prevention and cleanup mechanisms in task runner
2025-06-23 13:54:43 +08:00
Marvin Zhang
1008886715
fix: enhance task service resilience with connection health monitoring and periodic cleanup
2025-06-23 11:57:05 +08:00
Marvin Zhang
682efa807d
ci: add version check before publishing to NPM and provide publish summary
2025-06-20 17:53:15 +08:00
Marvin Zhang
5837472de5
fix: update API client constructor and improve error handling for missing API token and URL
...
chore: update package version to 0.1.1-dev.6 and adjust dependencies in pnpm-lock.yaml
refactor: change import paths to relative in tools.ts for consistency
2025-06-20 15:55:50 +08:00
Marvin Zhang
99d59b8e90
chore: add publishConfig to package.json for public access
2025-06-20 15:00:16 +08:00
Marvin Zhang
18137cce0d
fix: update jest test command to allow passing with no tests
2025-06-20 14:58:12 +08:00
Marvin Zhang
b7b7f25451
refactor: remove unused response variables in database, git, node, project, schedule, spider, task modules
2025-06-20 14:55:54 +08:00
Marvin Zhang
4f33460d45
chore: remove .gitattributes file to simplify language detection
2025-06-20 14:45:22 +08:00
Marvin Zhang
13038d4a1a
fix: unable to sync files from master in worker nodes
2025-06-20 14:42:52 +08:00
Marvin Zhang
52b180aaa0
ci: Add GitHub Actions workflow for publishing MCP package
2025-06-20 11:09:02 +08:00
Marvin Zhang
994adc3f71
fix: Update TypeScript ESLint packages and configuration
2025-06-20 10:37:39 +08:00
Marvin Zhang
2d0508a0c2
feat: Add AI tools for LLM provider management and AutoProbe configurations
...
- Implemented new API methods in CrawlabClient for managing LLM providers and AutoProbes.
- Added prompts for setting up AI-powered web scraping with AutoProbe and configuring LLM providers.
- Created a new module for AI tools, integrating various functionalities including listing, creating, updating, and deleting LLM providers and AutoProbes.
- Enhanced error handling and response formatting for better user experience.
2025-06-19 16:31:58 +08:00
Marvin Zhang
3976b93e66
feat: Add database and git tools, enhance node and spider configurations
...
- Implemented database tools for CRUD operations and connection testing.
- Added git tools for managing repositories including create, update, delete, clone, and pull functionalities.
- Enhanced node tools with update, enable, and disable functionalities.
- Updated spider tools to include new parameters for database integration, git repository association, and task execution modes.
- Improved schedule tools with additional parameters for task execution modes and priority settings.
- Introduced system overview tool for comprehensive system health and status reporting.
- Updated task filtering to include new task statuses.
2025-06-19 15:52:09 +08:00
Marvin Zhang
c29e21deec
feat: Implement Crawlab MCP Server with tools and prompts
...
- Added main server file (index.ts) to initialize the Crawlab MCP Server.
- Created prompts for spider analysis, task debugging, spider setup, and system monitoring.
- Developed tools for managing spiders, tasks, nodes, schedules, and system health.
- Implemented a mock client for testing server functionality.
- Added versioning support with version.ts.
- Created a test script (test-server.mjs) to validate tool configurations and server responses.
- Included build validation script (validate-build.mjs) to ensure proper setup and functionality.
- Configured TypeScript settings with tsconfig.json for better development experience.
2025-06-19 15:38:45 +08:00
Marvin Zhang
38c3a7534a
fix: unable to select all options in some dropdown
2025-06-18 22:33:04 +08:00
Marvin Zhang
09cfe37272
fix: unable to sync files to worker nodes when running tasks
2025-06-18 21:50:53 +08:00
Marvin Zhang
22497d22f7
fix: unable to filter tasks issue
2025-06-17 09:28:25 +08:00
Marvin Zhang
c06a9c6052
feat: update AutoProbe service and interfaces to support AutoProbeV2
2025-06-16 22:54:33 +08:00
Marvin Zhang
24452bff23
refactor: extract token retrieval logic into GetAPITokenFromContext utility function
2025-06-16 21:49:27 +08:00
Marvin Zhang
6d65cc13a2
feat: add aggregation methods to ModelService for enhanced data retrieval
2025-06-15 18:03:35 +08:00
Marvin Zhang
ab5e65f2fb
chore: update dependencies to latest versions across modules
2025-06-15 18:03:13 +08:00
Marvin Zhang
c4a469b440
feat: enhance SystemDetail layout and improve customization form with scrollbar
2025-06-15 17:16:32 +08:00
Marvin Zhang
c3230fdbe1
refactor: streamline database handling and remove unused options
2025-06-13 21:53:23 +08:00
Marvin Zhang
335050e72e
refactor: improve schedule service initialization, validation, and error handling
2025-06-13 21:53:08 +08:00
Marvin Zhang
0bd2b62792
feat: add DTOs for models and update project spiders type to array
2025-06-12 19:58:21 +08:00
