10 Commits

Author SHA1 Message Date
Marvin Zhang
f441265cc2 feat(sync): add gRPC file synchronization service and integrate end-to-end
- add proto/services/sync_service.proto and generate Go pb + grpc bindings
- implement SyncServiceServer (streaming file scan + download) with:
  - request deduplication, in-memory cache (TTL), chunked streaming
  - concurrent-safe broadcast to waiters and server-side logging
- register SyncSvr in gRPC server and expose sync client in GrpcClient:
  - add syncClient field, registration and safe getters with reconnection-aware timeouts
- integrate gRPC sync into runner:
  - split syncFiles into syncFilesHTTP (legacy) and syncFilesGRPC
  - Runner now chooses implementation via config flag and performs streaming scan/download
- controller improvements:
  - add semaphore-based rate limiting for sync scan requests with in-flight counters and logs
- misc:
  - add utils.IsSyncGrpcEnabled() config helper
  - improve HTTP sync error diagnostics (Content-Type validation, response previews)
  - update/regenerate many protobuf and gRPC generated files (protoc/protoc-gen-go / protoc-gen-go-grpc version bumps)
2025-10-20 12:48:53 +08:00
Marvin Zhang
29ef8d67da feat: implement synchronization and error handling improvements in task reconciliation and file synchronization 2025-09-28 17:42:23 +08:00
Marvin Zhang
13038d4a1a fix: unable to sync files from master in worker nodes 2025-06-20 14:42:52 +08:00
Marvin Zhang
4dace3ce8e refactor: migrate router and controller methods to use Fizz package
- Updated router groups to utilize the new crawlab-team/fizz package for improved routing capabilities.
- Refactored controller methods to accept Fizz router groups, enhancing consistency and maintainability.
- Simplified route registration by incorporating OpenAPI metadata directly into group definitions.
- Improved error handling and response generation in sync controller methods for better clarity.
- Enhanced overall code structure by standardizing route definitions and improving parameter handling.
2025-03-17 12:50:50 +08:00
Marvin Zhang
23cad00d92 refactor: streamline controller methods and enhance parameter handling
- Refactored multiple controller methods to accept structured parameters for improved clarity and maintainability.
- Consolidated error handling and response generation across various endpoints.
- Updated function signatures to eliminate unnecessary context parameters and enhance type safety.
- Improved consistency in response formatting and error handling across controllers.
- Enhanced file handling methods to support multipart file uploads and directory operations more effectively.
2025-03-13 17:37:30 +08:00
Marvin Zhang
a3b286558b refactor: consolidated configs 2024-11-18 16:48:09 +08:00
Marvin Zhang
e33fcfc150 refactor: renamed files and services 2024-11-05 11:15:27 +08:00
Marvin Zhang
4da8976228 refactor: code cleanup 2024-07-06 14:15:14 +08:00
Marvin Zhang
460c8d958a fix: missing relational models issues 2024-06-23 23:20:58 +08:00
Marvin Zhang
0b67fd9ece feat: added modules 2024-06-14 15:42:50 +08:00