108 Commits

Author SHA1 Message Date
Marvin Zhang
db2549e3cd fix: enhance error logging in file log driver and update default task log path
- Improved error messages in the FileLogDriver's cleanup method to include error details for better debugging.
- Updated the default task log path from '/app/logs/tasks' to '/var/log/crawlab/tasks' to ensure consistency across environments.
2025-01-01 14:26:10 +08:00
Marvin Zhang
7b6805a834 feat: enhance task runner with improved logging and dependency support
- Added support for new dependency file types: 'go.mod' and 'pom.xml' in dependency.go.
- Refactored command configuration in runner.go to improve logging and error handling.
- Introduced a new method to configure Node.js paths, enhancing environment setup for tasks.
- Enhanced IPC message handling with detailed logging for better traceability.
- Updated service logging to remove unnecessary prefixes for cleaner output.
- Improved command execution handling in process.go for better compatibility across platforms.
2024-12-31 22:52:21 +08:00
Marvin Zhang
dc59599509 refactor: remove db module and update imports to core/mongo
- Deleted the db module, consolidating database-related functionality into the core/mongo package for better organization and maintainability.
- Updated all import paths across the codebase to replace references to the removed db module with core/mongo.
- Cleaned up unused code and dependencies, enhancing overall project clarity and reducing complexity.
- This refactor improves the structure of the codebase by centralizing database operations and simplifying module management.
2024-12-25 10:28:21 +08:00
Marvin Zhang
3276083994 refactor: replace apex/log with structured logger across multiple services
- Replaced all instances of apex/log with a structured logger interface in various services, including Api, Server, Config, and others, to enhance logging consistency and context.
- Updated logging calls to utilize the new logger methods, improving error tracking and service monitoring.
- Added logger initialization in services and controllers to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Removed unused apex/log imports and cleaned up related code for better maintainability.
2024-12-24 19:11:19 +08:00
Marvin Zhang
99ed4396d1 refactor: improve logging messages and update configuration constants
- Updated logging messages in GrpcClient to provide clearer context, changing "ready" to "client is now ready" and "stopped" to "client has stopped".
- Refactored test setup in runner_test.go to remove unnecessary error checks during gRPC client start for cleaner code.
- Renamed GetDependencySetupScriptRoot to GetInstallRoot and updated related constants for better clarity and consistency in configuration management.
2024-12-23 18:19:08 +08:00
Marvin Zhang
29af5a366b feat: enhance gRPC client with state management and reconnection logic
- Introduced state management in GrpcClient to monitor and handle connection states effectively.
- Added a reconnect channel and a state monitoring goroutine to facilitate automatic reconnections on state changes.
- Updated the connect method to initiate a reconnection loop upon connection loss.
- Enhanced logging for connection state changes and errors during connection attempts.
- Refactored tests to ensure proper initialization of gRPC client and server, improving test reliability and coverage.
2024-12-21 21:41:00 +08:00
Marvin Zhang
c897fb58e4 refactor: streamline error handling and improve HTTP request management
- Removed the print flag from handleError function, simplifying error logging based on the development environment.
- Introduced a new performRequest function for standardized HTTP requests with JSON bodies, enhancing code reusability.
- Updated SendIMNotification and related functions to utilize the new RequestParam type for better clarity and consistency.
- Normalized HTTP request paths in the createHttpRequest method to ensure correct URL formatting.
- Added detailed error logging for JSON unmarshaling failures in syncFiles method.
- Introduced a NewHttpClient function to create HTTP clients with customizable timeouts.
2024-12-21 11:27:58 +08:00
Marvin Zhang
3cb74d76f9 feat: enhance gRPC client functionality and improve logging
- Added WaitForReady method to GrpcClient for blocking until the client is ready.
- Updated WorkerService to utilize WaitForReady for ensuring gRPC client readiness before starting.
- Refactored ModelService to consistently use GetGrpcClient for context management.
- Changed logging level for received metrics in MetricServiceServer from Info to Debug.
- Modified error handling in HandleError to conditionally print errors based on the environment.
- Cleaned up unused GrpcClient references in various services, improving code clarity.
2024-12-20 20:34:04 +08:00
Marvin Zhang
f736b2c58e fix: getting stream error for dependency server 2024-12-18 17:43:41 +08:00
Marvin Zhang
272371d9ce feat: allow set max runners for nodes 2024-12-11 22:05:34 +08:00
Marvin Zhang
1fe74fa8a5 fix: optimized node runners calculation 2024-12-11 20:43:40 +08:00
Marvin Zhang
2764c43322 test: fix issues 2024-11-24 23:20:53 +08:00
Marvin Zhang
24561bcbe0 refactor: optimized code 2024-11-24 23:14:26 +08:00
Marvin Zhang
9560da66b5 test: fix test cases issues 2024-11-22 17:43:59 +08:00
Marvin Zhang
1a460e3a2c refactor: optimized imports 2024-11-22 13:59:41 +08:00
Marvin Zhang
a6e845a9bf test: fix test case issue 2024-11-22 13:59:12 +08:00
Marvin Zhang
9f1d7dd385 test: added more test cases 2024-11-22 13:46:07 +08:00
Marvin Zhang
8357dc6f30 fix: runner cancel issue 2024-11-22 13:32:27 +08:00
Marvin Zhang
eee10ea08c test: fix test cases 2024-11-22 12:48:16 +08:00
Marvin Zhang
aa1b01b0a5 test: fix test cases 2024-11-22 12:45:04 +08:00
Marvin Zhang
110078333e test: fix test cases 2024-11-22 11:32:20 +08:00
Marvin Zhang
da5c3b0ba0 test: fix test cases 2024-11-22 10:51:57 +08:00
Marvin Zhang
d00dd9495c test: fix test cases 2024-11-22 10:43:45 +08:00
Marvin Zhang
b12251d43e test: fix test cases 2024-11-22 10:29:19 +08:00
Marvin Zhang
f64f407735 chore: added debug code 2024-11-22 09:51:34 +08:00
Marvin Zhang
d50383d47a feat: added ipc to task runner 2024-11-21 18:24:29 +08:00
Marvin Zhang
76aabc3692 fix: code warnings 2024-11-20 14:56:36 +08:00
Marvin Zhang
46100ebaf5 ci: updated base-image 2024-11-20 14:55:42 +08:00
Marvin Zhang
98a9c907c9 fix: test case issue 2024-11-19 17:40:44 +08:00
Marvin Zhang
3dc66e48db fix: test case issue 2024-11-19 15:53:40 +08:00
Marvin Zhang
7d2d2413fd refactor: code cleanup 2024-11-18 17:30:14 +08:00
Marvin Zhang
a3b286558b refactor: consolidated configs 2024-11-18 16:48:09 +08:00
Marvin Zhang
6af282058d fix: test issues 2024-11-05 13:54:05 +08:00
Marvin Zhang
e33fcfc150 refactor: renamed files and services 2024-11-05 11:15:27 +08:00
Marvin Zhang
73674832b8 feat: optimized dependency api 2024-11-04 00:16:42 +08:00
Marvin Zhang
66ea5b3ad5 refactor: updated comments 2024-11-01 15:32:41 +08:00
Marvin Zhang
71f0a210ba refactor: fixed dependency errors 2024-11-01 15:19:48 +08:00
Marvin Zhang
68ba84a4e7 refactor: optimized node communication 2024-11-01 15:19:48 +08:00
Marvin Zhang
d9b327de17 refactor: code cleanup 2024-11-01 15:19:48 +08:00
Marvin Zhang
8a5f51de47 refactor: updated grpc services 2024-11-01 15:19:48 +08:00
Marvin Zhang
f0f0aa9814 refactor: updated task related grpc proto 2024-11-01 15:19:48 +08:00
Marvin Zhang
1c03cb3e5c refactor: code cleanup 2024-10-29 12:59:45 +08:00
Marvin Zhang
e1170d5612 test: updated test cases 2024-10-20 17:55:57 +08:00
Marvin Zhang
1b852fb96a refactor: code cleanup 2024-10-18 15:03:32 +08:00
Marvin Zhang
8a74f30a46 feat: updated database related api 2024-10-09 18:28:27 +08:00
Marvin Zhang
4f52936ad8 refactor: integrated database services into task data insert 2024-10-08 18:41:36 +08:00
Marvin Zhang
7f9fac0fc0 feat: updated notification settings and channels 2024-07-23 17:24:40 +08:00
Marvin Zhang
9ffdd3f1cd refactor: Remove unused code and update models and functions for notification channels and settings 2024-07-23 12:22:59 +08:00
Marvin Zhang
8c79d51a5a refactor: Update SendNotification function to handle old and new settings triggers 2024-07-16 17:54:06 +08:00
Marvin Zhang
aca0c0ebce refactor: removed unnecessary code 2024-07-11 12:45:29 +08:00