Commit Graph

214 Commits

Author SHA1 Message Date
Marvin Zhang
bdc347aef7 feat: add Fatalf logging method to Logger interface and ServiceLogger
- Introduced Fatalf method in Logger interface for logging fatal messages with formatted content.
- Implemented Fatalf in ServiceLogger to log fatal messages and exit the program, enhancing error handling capabilities.
2024-12-23 13:15:18 +08:00
Marvin Zhang
afe21b7ca0 feat: implement logging interfaces and service logger
- Added Logger interface in core/interfaces/logger.go for standardized logging methods (Debugf, Infof, Warnf, Errorf).
- Introduced ServiceLogger in core/utils/log.go, which prefixes log messages with the service name for better context.
- Implemented logging methods in ServiceLogger to utilize the apex/log package for structured logging.
2024-12-21 23:00:30 +08:00
Marvin Zhang
e44b416e34 feat: enhance model base service with BSON ID normalization
- Added utility function to normalize BSON ObjectId in query parameters for gRPC methods.
- Updated GetOne, GetMany, DeleteOne, DeleteMany, UpdateOne, UpdateMany, ReplaceOne, and UpsertOne methods to utilize the new normalization function.
- Introduced new DependencyConfigSetup model instance in base model definitions for improved structure.
2024-12-21 22:07:54 +08:00
Marvin Zhang
29af5a366b feat: enhance gRPC client with state management and reconnection logic
- Introduced state management in GrpcClient to monitor and handle connection states effectively.
- Added a reconnect channel and a state monitoring goroutine to facilitate automatic reconnections on state changes.
- Updated the connect method to initiate a reconnection loop upon connection loss.
- Enhanced logging for connection state changes and errors during connection attempts.
- Refactored tests to ensure proper initialization of gRPC client and server, improving test reliability and coverage.
2024-12-21 21:41:00 +08:00
Marvin Zhang
75da4fff0f refactor: update gRPC client initialization in model service tests
- Replaced direct gRPC client instantiation with a call to GetGrpcClient().Start() for improved consistency and maintainability across test cases.
- Cleaned up unnecessary imports in model_service.go to streamline the codebase.
- Enhanced test setup by ensuring the gRPC client is properly initialized before executing service methods.
2024-12-21 20:27:02 +08:00
Marvin Zhang
41e973a4d3 fix: misaligned nodes when running tasks from a schedule through enhancement by adding new route and handler for running schedules
- Introduced a new POST endpoint '/:id/run' in the router to trigger the execution of a schedule.
- Implemented the PostScheduleRun function to handle the execution logic, including validation of input parameters and scheduling tasks.
- Refactored the PostSpiderRun function to streamline the scheduling process by directly calling the admin service.
- Updated the FsFileInfo struct to clarify the children field description.
- Modified the Task model to make NodeIds field optional in JSON serialization.
2024-12-21 15:33:07 +08:00
Marvin Zhang
b88b41afe2 feat: add API configuration options for port and path
- Introduced GetApiPort and GetApiPath functions to retrieve API port and path from configuration, with defaults provided.
- Updated GetApiEndpoint to construct the API endpoint URL using the new port and path functions, ensuring proper formatting.
- Added constants for default API port and path to enhance configurability.
2024-12-21 14:05:50 +08:00
Marvin Zhang
c897fb58e4 refactor: streamline error handling and improve HTTP request management
- Removed the print flag from handleError function, simplifying error logging based on the development environment.
- Introduced a new performRequest function for standardized HTTP requests with JSON bodies, enhancing code reusability.
- Updated SendIMNotification and related functions to utilize the new RequestParam type for better clarity and consistency.
- Normalized HTTP request paths in the createHttpRequest method to ensure correct URL formatting.
- Added detailed error logging for JSON unmarshaling failures in syncFiles method.
- Introduced a NewHttpClient function to create HTTP clients with customizable timeouts.
2024-12-21 11:27:58 +08:00
Marvin Zhang
3cb74d76f9 feat: enhance gRPC client functionality and improve logging
- Added WaitForReady method to GrpcClient for blocking until the client is ready.
- Updated WorkerService to utilize WaitForReady for ensuring gRPC client readiness before starting.
- Refactored ModelService to consistently use GetGrpcClient for context management.
- Changed logging level for received metrics in MetricServiceServer from Info to Debug.
- Modified error handling in HandleError to conditionally print errors based on the environment.
- Cleaned up unused GrpcClient references in various services, improving code clarity.
2024-12-20 20:34:04 +08:00
Marvin Zhang
f5631f965d chore: updated deps 2024-12-20 12:00:14 +08:00
Marvin Zhang
be93f9d17d feat: added retry for worker node start 2024-12-20 11:40:21 +08:00
Marvin Zhang
f736b2c58e fix: getting stream error for dependency server 2024-12-18 17:43:41 +08:00
Marvin Zhang
e77d4cdd31 feat: updated dependency management 2024-12-17 19:32:16 +08:00
Marvin Zhang
79c1d5d14b feat: updated dependency config setup 2024-12-16 21:44:03 +08:00
Marvin Zhang
c5c08dfba6 feat: added dependency config setup (wip) 2024-12-15 23:09:10 +08:00
Marvin Zhang
25580b4694 feat: prepare for dependency lang install/setup 2024-12-14 22:18:22 +08:00
Marvin Zhang
298420b0a6 feat: updated the way syncing pypi projects 2024-12-14 21:47:11 +08:00
Marvin Zhang
272371d9ce feat: allow set max runners for nodes 2024-12-11 22:05:34 +08:00
Marvin Zhang
1fe74fa8a5 fix: optimized node runners calculation 2024-12-11 20:43:40 +08:00
Marvin Zhang
6b78afe95c fix: added default fields 2024-12-11 20:05:03 +08:00
Marvin Zhang
30767754a0 feat: improve api token verification compatibility 2024-12-11 17:52:51 +08:00
Marvin Zhang
19f19939a1 fix: duplicated index recreation 2024-12-11 12:17:25 +08:00
Marvin Zhang
2a54762249 fix: duplicated index recreation 2024-12-11 11:45:08 +08:00
Marvin Zhang
2764c43322 test: fix issues 2024-11-24 23:20:53 +08:00
Marvin Zhang
24561bcbe0 refactor: optimized code 2024-11-24 23:14:26 +08:00
Marvin Zhang
a1275879b3 fix: index creation issue 2024-11-23 11:14:13 +08:00
Marvin Zhang
eed93ae2a4 fix: create index errors 2024-11-23 10:54:21 +08:00
Marvin Zhang
b125ba340e ci: updated base-image 2024-11-23 10:51:46 +08:00
Marvin Zhang
858e5c2b89 fix: unable to start api 2024-11-22 21:19:17 +08:00
Marvin Zhang
7a322ae6c8 fix: unable to start api 2024-11-22 20:58:01 +08:00
Marvin Zhang
9560da66b5 test: fix test cases issues 2024-11-22 17:43:59 +08:00
Marvin Zhang
1a460e3a2c refactor: optimized imports 2024-11-22 13:59:41 +08:00
Marvin Zhang
a6e845a9bf test: fix test case issue 2024-11-22 13:59:12 +08:00
Marvin Zhang
9f1d7dd385 test: added more test cases 2024-11-22 13:46:07 +08:00
Marvin Zhang
36e7f94334 fix: runner cancel issue 2024-11-22 13:40:39 +08:00
Marvin Zhang
8357dc6f30 fix: runner cancel issue 2024-11-22 13:32:27 +08:00
Marvin Zhang
eee10ea08c test: fix test cases 2024-11-22 12:48:16 +08:00
Marvin Zhang
aa1b01b0a5 test: fix test cases 2024-11-22 12:45:04 +08:00
Marvin Zhang
110078333e test: fix test cases 2024-11-22 11:32:20 +08:00
Marvin Zhang
da5c3b0ba0 test: fix test cases 2024-11-22 10:51:57 +08:00
Marvin Zhang
d00dd9495c test: fix test cases 2024-11-22 10:43:45 +08:00
Marvin Zhang
b12251d43e test: fix test cases 2024-11-22 10:29:19 +08:00
Marvin Zhang
f64f407735 chore: added debug code 2024-11-22 09:51:34 +08:00
Marvin Zhang
d50383d47a feat: added ipc to task runner 2024-11-21 18:24:29 +08:00
Marvin Zhang
b3261343b8 refactor: code cleanup 2024-11-20 18:22:27 +08:00
Marvin Zhang
76aabc3692 fix: code warnings 2024-11-20 14:56:36 +08:00
Marvin Zhang
46100ebaf5 ci: updated base-image 2024-11-20 14:55:42 +08:00
Marvin Zhang
bbeb752e17 chore: added debug code 2024-11-19 18:34:59 +08:00
Marvin Zhang
dc9f62dfd0 feat: added health check for worker service 2024-11-19 18:32:50 +08:00
Marvin Zhang
1a93b148e1 fix: unable to connect to grpc server issue 2024-11-19 18:15:17 +08:00