git-sync/crawlab
1
0
Fork 0
mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-21 17:21:09 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,159 Commits 6 Branches 35 Tags
f441265cc29fbf33af972ec5cd45de2ff55f29f9
Commit Graph

11 Commits

Author SHA1 Message Date
Marvin Zhang
f441265cc2 feat(sync): add gRPC file synchronization service and integrate end-to-end 
- add proto/services/sync_service.proto and generate Go pb + grpc bindings
- implement SyncServiceServer (streaming file scan + download) with:
  - request deduplication, in-memory cache (TTL), chunked streaming
  - concurrent-safe broadcast to waiters and server-side logging
- register SyncSvr in gRPC server and expose sync client in GrpcClient:
  - add syncClient field, registration and safe getters with reconnection-aware timeouts
- integrate gRPC sync into runner:
  - split syncFiles into syncFilesHTTP (legacy) and syncFilesGRPC
  - Runner now chooses implementation via config flag and performs streaming scan/download
- controller improvements:
  - add semaphore-based rate limiting for sync scan requests with in-flight counters and logs
- misc:
  - add utils.IsSyncGrpcEnabled() config helper
  - improve HTTP sync error diagnostics (Content-Type validation, response previews)
  - update/regenerate many protobuf and gRPC generated files (protoc/protoc-gen-go / protoc-gen-go-grpc version bumps)
 2025-10-20 12:48:53 +08:00
Marvin Zhang
8c2c23d9b6 feat: Update gRPC service definitions and implement CheckProcess method 
- Downgraded protoc-gen-go-grpc and protoc versions for compatibility.
- Added CheckProcess method to TaskService with corresponding request and response types.
- Updated Subscribe and Connect methods to use new generic client stream types.
- Refactored server and client implementations for Subscribe and Connect methods.
- Ensured backward compatibility by maintaining existing method signatures where applicable.
- Added necessary handler for CheckProcess in the service descriptor.
 2025-09-17 10:37:03 +08:00
Marvin Zhang
17f8917d0a chore: update dependencies and enhance gRPC services 
- Updated various dependencies in go.mod and go.sum files, including cloud.google.com/go/compute/metadata to v0.5.2, google.golang.org/grpc to v1.69.2, and google.golang.org/protobuf to v1.36.1.
- Refactored gRPC service definitions to use the latest protoc-gen-go and protoc versions, ensuring compatibility with the latest gRPC-Go features.
- Introduced a new logger utility in core/utils/logger.go to improve logging capabilities across the application.
- Added a README.md for gRPC setup and compilation guidance, enhancing developer experience.
- Improved the Python installation script to handle version listing more effectively and ensure user-specific environment setup.
 2024-12-25 17:46:49 +08:00
Marvin Zhang
e77d4cdd31 feat: updated dependency management 2024-12-17 19:32:16 +08:00
Marvin Zhang
79c1d5d14b feat: updated dependency config setup 2024-12-16 21:44:03 +08:00
Marvin Zhang
c5c08dfba6 feat: added dependency config setup (wip) 2024-12-15 23:09:10 +08:00
Marvin Zhang
25580b4694 feat: prepare for dependency lang install/setup 2024-12-14 22:18:22 +08:00
Marvin Zhang
a0989d36db feat: optimized dependency logic 2024-11-05 18:21:52 +08:00
Marvin Zhang
10bb511c5f feat: updated dependency handler logic 2024-11-05 11:40:12 +08:00
Marvin Zhang
fbf8e5f9f3 feat: optimizing dependency services including grpc, api 2024-11-04 17:45:34 +08:00
Marvin Zhang
71f0a210ba refactor: fixed dependency errors 2024-11-01 15:19:48 +08:00