Marvin Zhang
75da4fff0f
refactor: update gRPC client initialization in model service tests
...
- Replaced direct gRPC client instantiation with a call to GetGrpcClient().Start() for improved consistency and maintainability across test cases.
- Cleaned up unnecessary imports in model_service.go to streamline the codebase.
- Enhanced test setup by ensuring the gRPC client is properly initialized before executing service methods.
2024-12-21 20:27:02 +08:00
Marvin Zhang
41e973a4d3
fix: misaligned nodes when running tasks from a schedule through enhancement by adding new route and handler for running schedules
...
- Introduced a new POST endpoint '/:id/run' in the router to trigger the execution of a schedule.
- Implemented the PostScheduleRun function to handle the execution logic, including validation of input parameters and scheduling tasks.
- Refactored the PostSpiderRun function to streamline the scheduling process by directly calling the admin service.
- Updated the FsFileInfo struct to clarify the children field description.
- Modified the Task model to make NodeIds field optional in JSON serialization.
2024-12-21 15:33:07 +08:00
Marvin Zhang
3cb74d76f9
feat: enhance gRPC client functionality and improve logging
...
- Added WaitForReady method to GrpcClient for blocking until the client is ready.
- Updated WorkerService to utilize WaitForReady for ensuring gRPC client readiness before starting.
- Refactored ModelService to consistently use GetGrpcClient for context management.
- Changed logging level for received metrics in MetricServiceServer from Info to Debug.
- Modified error handling in HandleError to conditionally print errors based on the environment.
- Cleaned up unused GrpcClient references in various services, improving code clarity.
2024-12-20 20:34:04 +08:00
Marvin Zhang
e77d4cdd31
feat: updated dependency management
2024-12-17 19:32:16 +08:00
Marvin Zhang
79c1d5d14b
feat: updated dependency config setup
2024-12-16 21:44:03 +08:00
Marvin Zhang
c5c08dfba6
feat: added dependency config setup (wip)
2024-12-15 23:09:10 +08:00
Marvin Zhang
25580b4694
feat: prepare for dependency lang install/setup
2024-12-14 22:18:22 +08:00
Marvin Zhang
298420b0a6
feat: updated the way syncing pypi projects
2024-12-14 21:47:11 +08:00
Marvin Zhang
1fe74fa8a5
fix: optimized node runners calculation
2024-12-11 20:43:40 +08:00
Marvin Zhang
19f19939a1
fix: duplicated index recreation
2024-12-11 12:17:25 +08:00
Marvin Zhang
2a54762249
fix: duplicated index recreation
2024-12-11 11:45:08 +08:00
Marvin Zhang
a1275879b3
fix: index creation issue
2024-11-23 11:14:13 +08:00
Marvin Zhang
eed93ae2a4
fix: create index errors
2024-11-23 10:54:21 +08:00
Marvin Zhang
bbeb752e17
chore: added debug code
2024-11-19 18:34:59 +08:00
Marvin Zhang
dfaf18fbd6
feat: added notification test
2024-11-14 16:44:49 +08:00
Marvin Zhang
4fd1ff8fcf
refactor: optimized field names
2024-11-14 11:22:37 +08:00
Marvin Zhang
86b03307dc
feat: updated controllers
2024-11-13 15:33:47 +08:00
Marvin Zhang
f0bb927b0b
feat: updated permission management
2024-11-12 18:14:12 +08:00
Marvin Zhang
a233075779
refactor: restructured code
2024-11-10 17:23:32 +08:00
Marvin Zhang
a0989d36db
feat: optimized dependency logic
2024-11-05 18:21:52 +08:00
Marvin Zhang
0117794930
fix: test issues
2024-11-05 13:46:22 +08:00
Marvin Zhang
9688026714
fix: test issue
2024-11-05 13:30:49 +08:00
Marvin Zhang
10bb511c5f
feat: updated dependency handler logic
2024-11-05 11:40:12 +08:00
Marvin Zhang
e33fcfc150
refactor: renamed files and services
2024-11-05 11:15:27 +08:00
Marvin Zhang
fbf8e5f9f3
feat: optimizing dependency services including grpc, api
2024-11-04 17:45:34 +08:00
Marvin Zhang
73674832b8
feat: optimized dependency api
2024-11-04 00:16:42 +08:00
Marvin Zhang
71f0a210ba
refactor: fixed dependency errors
2024-11-01 15:19:48 +08:00
Marvin Zhang
68ba84a4e7
refactor: optimized node communication
2024-11-01 15:19:48 +08:00
Marvin Zhang
d9b327de17
refactor: code cleanup
2024-11-01 15:19:48 +08:00
Marvin Zhang
8a5f51de47
refactor: updated grpc services
2024-11-01 15:19:48 +08:00
Marvin Zhang
79ea8a0f88
refactor: updated index related code
2024-10-29 13:18:57 +08:00
Marvin Zhang
1c03cb3e5c
refactor: code cleanup
2024-10-29 12:59:45 +08:00
Marvin Zhang
78912a56b5
chore: updated logs
2024-10-21 12:52:11 +08:00
Marvin Zhang
fd5ebaeb60
ci: updated unit test
2024-10-21 10:22:45 +08:00
Marvin Zhang
d7ac9ebb1f
test: updated test cases
2024-10-20 18:45:54 +08:00
Marvin Zhang
e1170d5612
test: updated test cases
2024-10-20 17:55:57 +08:00
Marvin Zhang
1b852fb96a
refactor: code cleanup
2024-10-18 15:03:32 +08:00
Marvin Zhang
1646df7136
feat: added nav for task detail
2024-10-17 21:10:19 +08:00
Marvin Zhang
2f8f38f145
feat: optimized spider, task pages
2024-10-17 20:11:38 +08:00
Marvin Zhang
eb56598d93
refactor: code cleanup
2024-10-16 20:13:24 +08:00
Marvin Zhang
785424b1ac
refactor: combined system collections
2024-10-14 17:56:39 +08:00
Marvin Zhang
e65b846427
feat: added system collections
2024-10-14 16:16:03 +08:00
Marvin Zhang
045fc42991
feat: added field to spider
2024-10-11 19:06:31 +08:00
Marvin Zhang
4f52936ad8
refactor: integrated database services into task data insert
2024-10-08 18:41:36 +08:00
Marvin Zhang
0ed2150901
feat: added performance monitoring for elasticsearch
2024-10-07 17:19:09 +08:00
Marvin Zhang
e15c3c9b4f
feat: added performance monitoring for database
2024-10-05 14:46:38 +08:00
Marvin Zhang
6986a10246
feat: updated password handling logic
2024-08-07 18:05:35 +08:00
Marvin Zhang
1197c05316
🚀 feat: add IsDefault field to DatabaseV2 model
...
🔨 refactor: modify GetDatabaseList, GetDatabaseById, PostDatabaseTestConnection, and GetDatabaseMetadata functions
🐞 fix: GetDatabaseById and GetMetadata functions in MongoService
🔧 chore: add GetDefaultDatabase and GetDatabaseById functions to utils file
2024-08-06 15:35:36 +08:00
Marvin Zhang
421a06930c
refactor: Update models to use DatabaseV2 instead of DataSourceV2
2024-08-05 16:07:19 +08:00
Marvin Zhang
e57655c67e
refactor: Update models to use DatabaseV2 instead of DataSourceV2
2024-08-05 13:14:57 +08:00
