git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-22 17:31:03 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,176 Commits 6 Branches 35 Tags
ee11cd78eceabb6b4b521b4040d8b5969f4abbef
Commit Graph

4 Commits

Author SHA1 Message Date
Marvin Zhang
f441265cc2 feat(sync): add gRPC file synchronization service and integrate end-to-end 
- add proto/services/sync_service.proto and generate Go pb + grpc bindings
- implement SyncServiceServer (streaming file scan + download) with:
  - request deduplication, in-memory cache (TTL), chunked streaming
  - concurrent-safe broadcast to waiters and server-side logging
- register SyncSvr in gRPC server and expose sync client in GrpcClient:
  - add syncClient field, registration and safe getters with reconnection-aware timeouts
- integrate gRPC sync into runner:
  - split syncFiles into syncFilesHTTP (legacy) and syncFilesGRPC
  - Runner now chooses implementation via config flag and performs streaming scan/download
- controller improvements:
  - add semaphore-based rate limiting for sync scan requests with in-flight counters and logs
- misc:
  - add utils.IsSyncGrpcEnabled() config helper
  - improve HTTP sync error diagnostics (Content-Type validation, response previews)
  - update/regenerate many protobuf and gRPC generated files (protoc/protoc-gen-go / protoc-gen-go-grpc version bumps)
 2025-10-20 12:48:53 +08:00
Marvin Zhang
2a211923da Update core/task/handler/runner_sync.go 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-10-09 11:14:32 +08:00
Marvin Zhang
29ef8d67da feat: implement synchronization and error handling improvements in task reconciliation and file synchronization 2025-09-28 17:42:23 +08:00
Marvin Zhang
784ffc8b52 feat: implement task management service operations, stream manager, and worker pool 
- Added service_operations.go for task management including run, cancel, and execution logic.
- Introduced stream_manager.go to handle task streams and manage cancellation signals.
- Created worker_pool.go to manage a bounded pool of workers for executing tasks concurrently.
- Implemented graceful shutdown and cleanup mechanisms for task runners and streams.
- Enhanced error handling and logging throughout the task management process.
 2025-08-06 18:29:08 +08:00