24 Commits

Author SHA1 Message Date
Marvin Zhang
f441265cc2 feat(sync): add gRPC file synchronization service and integrate end-to-end
- add proto/services/sync_service.proto and generate Go pb + grpc bindings
- implement SyncServiceServer (streaming file scan + download) with:
  - request deduplication, in-memory cache (TTL), chunked streaming
  - concurrent-safe broadcast to waiters and server-side logging
- register SyncSvr in gRPC server and expose sync client in GrpcClient:
  - add syncClient field, registration and safe getters with reconnection-aware timeouts
- integrate gRPC sync into runner:
  - split syncFiles into syncFilesHTTP (legacy) and syncFilesGRPC
  - Runner now chooses implementation via config flag and performs streaming scan/download
- controller improvements:
  - add semaphore-based rate limiting for sync scan requests with in-flight counters and logs
- misc:
  - add utils.IsSyncGrpcEnabled() config helper
  - improve HTTP sync error diagnostics (Content-Type validation, response previews)
  - update/regenerate many protobuf and gRPC generated files (protoc/protoc-gen-go / protoc-gen-go-grpc version bumps)
2025-10-20 12:48:53 +08:00
Marvin Zhang
8c2c23d9b6 feat: Update gRPC service definitions and implement CheckProcess method
- Downgraded protoc-gen-go-grpc and protoc versions for compatibility.
- Added CheckProcess method to TaskService with corresponding request and response types.
- Updated Subscribe and Connect methods to use new generic client stream types.
- Refactored server and client implementations for Subscribe and Connect methods.
- Ensured backward compatibility by maintaining existing method signatures where applicable.
- Added necessary handler for CheckProcess in the service descriptor.
2025-09-17 10:37:03 +08:00
Marvin Zhang
c0e230e5d8 refactor: rename PING code to HEARTBEAT in node service and update related proto files 2025-09-12 14:17:49 +08:00
Marvin Zhang
65aeb3ed8c feat: add PING mechanism for connection health checks; update proto and generated files
- Introduced PING code in TaskServiceConnectCode enum for health checks.
- Updated Runner to use proper PING messages instead of fake log messages for connection health checks.
- Modified TaskServiceServer to handle PING requests and acknowledge them.
- Adjusted generated gRPC files to reflect changes in proto definitions and ensure compatibility.
2025-08-16 17:19:21 +08:00
Marvin Zhang
8bd3ef0b72 feat: add goroutine count metric to the MetricService and update related files 2025-07-08 14:08:31 +08:00
Marvin Zhang
17f8917d0a chore: update dependencies and enhance gRPC services
- Updated various dependencies in go.mod and go.sum files, including cloud.google.com/go/compute/metadata to v0.5.2, google.golang.org/grpc to v1.69.2, and google.golang.org/protobuf to v1.36.1.
- Refactored gRPC service definitions to use the latest protoc-gen-go and protoc versions, ensuring compatibility with the latest gRPC-Go features.
- Introduced a new logger utility in core/utils/logger.go to improve logging capabilities across the application.
- Added a README.md for gRPC setup and compilation guidance, enhancing developer experience.
- Improved the Python installation script to handle version listing more effectively and ensure user-specific environment setup.
2024-12-25 17:46:49 +08:00
Marvin Zhang
e77d4cdd31 feat: updated dependency management 2024-12-17 19:32:16 +08:00
Marvin Zhang
79c1d5d14b feat: updated dependency config setup 2024-12-16 21:44:03 +08:00
Marvin Zhang
c5c08dfba6 feat: added dependency config setup (wip) 2024-12-15 23:09:10 +08:00
Marvin Zhang
25580b4694 feat: prepare for dependency lang install/setup 2024-12-14 22:18:22 +08:00
Marvin Zhang
a0989d36db feat: optimized dependency logic 2024-11-05 18:21:52 +08:00
Marvin Zhang
10bb511c5f feat: updated dependency handler logic 2024-11-05 11:40:12 +08:00
Marvin Zhang
fbf8e5f9f3 feat: optimizing dependency services including grpc, api 2024-11-04 17:45:34 +08:00
Marvin Zhang
71f0a210ba refactor: fixed dependency errors 2024-11-01 15:19:48 +08:00
Marvin Zhang
8a5f51de47 refactor: updated grpc services 2024-11-01 15:19:48 +08:00
Marvin Zhang
f0f0aa9814 refactor: updated task related grpc proto 2024-11-01 15:19:48 +08:00
Marvin Zhang
8c79d51a5a refactor: Update SendNotification function to handle old and new settings triggers 2024-07-16 17:54:06 +08:00
Marvin Zhang
7eb8c08e0e feat: added metric service 2024-07-02 17:59:40 +08:00
Marvin Zhang
023ba27566 fix: unable to sync directories to work nodes 2024-07-01 15:59:20 +08:00
Marvin Zhang
7bdce1af58 feat: added metrics service v2 2024-06-26 23:23:14 +08:00
Marvin Zhang
9c620fa2ee fix: get all spider issue 2024-06-26 17:31:31 +08:00
Marvin Zhang
972713959f feat: updated grpc for dependencies service 2024-06-15 23:25:24 +08:00
Marvin Zhang
e5b20af9a4 feat: updated grpc proto 2024-06-14 19:17:23 +08:00
Marvin Zhang
0b67fd9ece feat: added modules 2024-06-14 15:42:50 +08:00