Commit Graph

5948 Commits

Author SHA1 Message Date
Marvin Zhang
99ed4396d1 refactor: improve logging messages and update configuration constants
- Updated logging messages in GrpcClient to provide clearer context, changing "ready" to "client is now ready" and "stopped" to "client has stopped".
- Refactored test setup in runner_test.go to remove unnecessary error checks during gRPC client start for cleaner code.
- Renamed GetDependencySetupScriptRoot to GetInstallRoot and updated related constants for better clarity and consistency in configuration management.
2024-12-23 18:19:08 +08:00
Marvin Zhang
c3f4c4ae05 feat: enhance gRPC client with structured logging and dependency actions
- Added DependencyActionSync and DependencyActionSetup constants to improve dependency management.
- Refactored GrpcClient to utilize a logger interface for consistent logging across connection states and errors.
- Updated Start, Stop, and connection methods to replace direct log calls with logger methods, enhancing log context and readability.
- Simplified test cases by removing error checks on gRPC client start, ensuring cleaner test setup.
2024-12-23 17:17:21 +08:00
Marvin Zhang
34455dc47c feat: enhance DependencyConfig model with additional fields
- Added TotalDependencies and SearchReady fields to the DependencyConfig struct for improved dependency management.
- Retained the Setup field for backward compatibility while ensuring proper JSON and BSON handling.
2024-12-23 15:18:04 +08:00
Marvin Zhang
ed8fb78c3b feat: expand Logger interface and implement additional logging methods in ServiceLogger
- Added Debug, Info, Warn, Error, and Fatal methods to the Logger interface for comprehensive logging capabilities.
- Implemented corresponding methods in ServiceLogger to facilitate structured logging with service context.
- Enhanced the logging functionality to support various log levels, improving error tracking and debugging.
2024-12-23 14:25:48 +08:00
Marvin Zhang
bdc347aef7 feat: add Fatalf logging method to Logger interface and ServiceLogger
- Introduced Fatalf method in Logger interface for logging fatal messages with formatted content.
- Implemented Fatalf in ServiceLogger to log fatal messages and exit the program, enhancing error handling capabilities.
2024-12-23 13:15:18 +08:00
Marvin Zhang
afe21b7ca0 feat: implement logging interfaces and service logger
- Added Logger interface in core/interfaces/logger.go for standardized logging methods (Debugf, Infof, Warnf, Errorf).
- Introduced ServiceLogger in core/utils/log.go, which prefixes log messages with the service name for better context.
- Implemented logging methods in ServiceLogger to utilize the apex/log package for structured logging.
2024-12-21 23:00:30 +08:00
Marvin Zhang
e44b416e34 feat: enhance model base service with BSON ID normalization
- Added utility function to normalize BSON ObjectId in query parameters for gRPC methods.
- Updated GetOne, GetMany, DeleteOne, DeleteMany, UpdateOne, UpdateMany, ReplaceOne, and UpsertOne methods to utilize the new normalization function.
- Introduced new DependencyConfigSetup model instance in base model definitions for improved structure.
2024-12-21 22:07:54 +08:00
Marvin Zhang
29af5a366b feat: enhance gRPC client with state management and reconnection logic
- Introduced state management in GrpcClient to monitor and handle connection states effectively.
- Added a reconnect channel and a state monitoring goroutine to facilitate automatic reconnections on state changes.
- Updated the connect method to initiate a reconnection loop upon connection loss.
- Enhanced logging for connection state changes and errors during connection attempts.
- Refactored tests to ensure proper initialization of gRPC client and server, improving test reliability and coverage.
2024-12-21 21:41:00 +08:00
Marvin Zhang
75da4fff0f refactor: update gRPC client initialization in model service tests
- Replaced direct gRPC client instantiation with a call to GetGrpcClient().Start() for improved consistency and maintainability across test cases.
- Cleaned up unnecessary imports in model_service.go to streamline the codebase.
- Enhanced test setup by ensuring the gRPC client is properly initialized before executing service methods.
2024-12-21 20:27:02 +08:00
Marvin Zhang
41e973a4d3 fix: misaligned nodes when running tasks from a schedule through enhancement by adding new route and handler for running schedules
- Introduced a new POST endpoint '/:id/run' in the router to trigger the execution of a schedule.
- Implemented the PostScheduleRun function to handle the execution logic, including validation of input parameters and scheduling tasks.
- Refactored the PostSpiderRun function to streamline the scheduling process by directly calling the admin service.
- Updated the FsFileInfo struct to clarify the children field description.
- Modified the Task model to make NodeIds field optional in JSON serialization.
2024-12-21 15:33:07 +08:00
Marvin Zhang
b88b41afe2 feat: add API configuration options for port and path
- Introduced GetApiPort and GetApiPath functions to retrieve API port and path from configuration, with defaults provided.
- Updated GetApiEndpoint to construct the API endpoint URL using the new port and path functions, ensuring proper formatting.
- Added constants for default API port and path to enhance configurability.
2024-12-21 14:05:50 +08:00
Marvin Zhang
c897fb58e4 refactor: streamline error handling and improve HTTP request management
- Removed the print flag from handleError function, simplifying error logging based on the development environment.
- Introduced a new performRequest function for standardized HTTP requests with JSON bodies, enhancing code reusability.
- Updated SendIMNotification and related functions to utilize the new RequestParam type for better clarity and consistency.
- Normalized HTTP request paths in the createHttpRequest method to ensure correct URL formatting.
- Added detailed error logging for JSON unmarshaling failures in syncFiles method.
- Introduced a NewHttpClient function to create HTTP clients with customizable timeouts.
2024-12-21 11:27:58 +08:00
Marvin Zhang
3cb74d76f9 feat: enhance gRPC client functionality and improve logging
- Added WaitForReady method to GrpcClient for blocking until the client is ready.
- Updated WorkerService to utilize WaitForReady for ensuring gRPC client readiness before starting.
- Refactored ModelService to consistently use GetGrpcClient for context management.
- Changed logging level for received metrics in MetricServiceServer from Info to Debug.
- Modified error handling in HandleError to conditionally print errors based on the environment.
- Cleaned up unused GrpcClient references in various services, improving code clarity.
2024-12-20 20:34:04 +08:00
Marvin Zhang
eed451e883 ci: trigger workflow 2024-12-20 15:38:29 +08:00
Marvin Zhang
3932b298bf chore: refine Docker build process and update context paths
- Updated GitHub Actions workflow to specify the Dockerfile location and set the build context to the root of the repository.
- Modified Dockerfile to copy backend and other required modules from their respective directories, improving the build process.
2024-12-20 12:17:15 +08:00
Marvin Zhang
83b81c8353 chore: update Docker configuration and ignore files
- Updated .dockerignore to exclude temporary files, Git directories, and Node modules.
- Modified Dockerfile to copy required modules from the parent directory and build the Go application from the backend directory.
2024-12-20 12:10:23 +08:00
Marvin Zhang
f5631f965d chore: updated deps 2024-12-20 12:00:14 +08:00
Marvin Zhang
4d28421043 chore: updated deps 2024-12-20 11:49:41 +08:00
Marvin Zhang
be93f9d17d feat: added retry for worker node start 2024-12-20 11:40:21 +08:00
Marvin Zhang
5c4099cf48 ci: updated workflow 2024-12-20 11:32:53 +08:00
Marvin Zhang
e04255521b ci: updated workflow 2024-12-20 10:01:43 +08:00
Marvin Zhang
3636151003 Merge branch 'develop' of github.com:crawlab-team/crawlab into develop 2024-12-19 20:48:29 +08:00
Marvin Zhang
f736b2c58e fix: getting stream error for dependency server 2024-12-18 17:43:41 +08:00
Marvin Zhang
5bdcd29a39 fix: getting stream error for dependency server 2024-12-18 17:43:41 +08:00
Marvin Zhang
19b9135238 fix: ci issue 2024-12-17 19:39:26 +08:00
Marvin Zhang
9071378b6e fix: ci issue 2024-12-17 19:39:26 +08:00
Marvin Zhang
f823c7fc14 chore: updated deps 2024-12-17 19:34:48 +08:00
Marvin Zhang
8913cffd01 chore: updated deps 2024-12-17 19:34:48 +08:00
Marvin Zhang
e77d4cdd31 feat: updated dependency management 2024-12-17 19:32:16 +08:00
Marvin Zhang
88426212a1 feat: updated dependency management 2024-12-17 19:32:16 +08:00
Marvin Zhang
c945e093d5 chore: updated deps 2024-12-16 22:28:13 +08:00
Marvin Zhang
2117c6bb69 chore: updated deps 2024-12-16 22:28:13 +08:00
Marvin Zhang
27a8f197c2 feat: updated dependency config setup 2024-12-16 21:48:07 +08:00
Marvin Zhang
14dc8f374f feat: updated dependency config setup 2024-12-16 21:48:07 +08:00
Marvin Zhang
79c1d5d14b feat: updated dependency config setup 2024-12-16 21:44:03 +08:00
Marvin Zhang
53643dad13 feat: updated dependency config setup 2024-12-16 21:44:03 +08:00
Marvin Zhang
c5c08dfba6 feat: added dependency config setup (wip) 2024-12-15 23:09:10 +08:00
Marvin Zhang
0f034e9588 feat: added dependency config setup (wip) 2024-12-15 23:09:10 +08:00
Marvin Zhang
3b54a63bed chore: updated deps 2024-12-14 22:20:30 +08:00
Marvin Zhang
bb66362ac2 chore: updated deps 2024-12-14 22:20:30 +08:00
Marvin Zhang
25580b4694 feat: prepare for dependency lang install/setup 2024-12-14 22:18:22 +08:00
Marvin Zhang
89f93aa08b feat: prepare for dependency lang install/setup 2024-12-14 22:18:22 +08:00
Marvin Zhang
298420b0a6 feat: updated the way syncing pypi projects 2024-12-14 21:47:11 +08:00
Marvin Zhang
694cd40db5 feat: updated the way syncing pypi projects 2024-12-14 21:47:11 +08:00
Marvin Zhang
5293207795 ci: trigger workflow 2024-12-11 22:09:56 +08:00
Marvin Zhang
232b2b813a ci: trigger workflow 2024-12-11 22:09:56 +08:00
Marvin Zhang
2b934076bf chore: updated deps 2024-12-11 22:08:30 +08:00
Marvin Zhang
a08aa0c7ea chore: updated deps 2024-12-11 22:08:30 +08:00
Marvin Zhang
272371d9ce feat: allow set max runners for nodes 2024-12-11 22:05:34 +08:00
Marvin Zhang
eaa834e8f8 feat: allow set max runners for nodes 2024-12-11 22:05:34 +08:00