git-sync/crawlab

mirror of https://github.com/crawlab-team/crawlab.git synced 2026-01-22 17:31:03 +01:00

Files

History

Marvin Zhang f441265cc2 feat(sync): add gRPC file synchronization service and integrate end-to-end

- add proto/services/sync_service.proto and generate Go pb + grpc bindings
- implement SyncServiceServer (streaming file scan + download) with:
  - request deduplication, in-memory cache (TTL), chunked streaming
  - concurrent-safe broadcast to waiters and server-side logging
- register SyncSvr in gRPC server and expose sync client in GrpcClient:
  - add syncClient field, registration and safe getters with reconnection-aware timeouts
- integrate gRPC sync into runner:
  - split syncFiles into syncFilesHTTP (legacy) and syncFilesGRPC
  - Runner now chooses implementation via config flag and performs streaming scan/download
- controller improvements:
  - add semaphore-based rate limiting for sync scan requests with in-flight counters and logs
- misc:
  - add utils.IsSyncGrpcEnabled() config helper
  - improve HTTP sync error diagnostics (Content-Type validation, response previews)
  - update/regenerate many protobuf and gRPC generated files (protoc/protoc-gen-go / protoc-gen-go-grpc version bumps)

2025-10-20 12:48:53 +08:00

..

.github/workflows

feat: disable backend unit tests and document reasons for integration test requirements

2025-10-09 12:35:09 +08:00

feat: enhance API routing and OpenAPI documentation support

2025-03-11 23:45:06 +08:00

feat: add logo printing functionality and update logger usage

2024-12-25 13:08:56 +08:00

fix: add nil checks for logger in config initialization and logging methods

2025-07-23 15:07:39 +08:00

feat: add node disconnection handling and update task statuses accordingly

2025-09-12 15:40:29 +08:00

feat(sync): add gRPC file synchronization service and integrate end-to-end

2025-10-20 12:48:53 +08:00

refactor: renamed files and services

2024-11-05 11:15:27 +08:00

feat: add dependency installer service and enhance task runner with dependency management

2025-01-01 20:51:55 +08:00

fix: Update FsFileInfo to use pointer slices for children and remove redundant getter methods

2025-07-23 09:42:07 +08:00

refactor: code cleanup

2024-11-01 15:19:48 +08:00

refactor: standardize timestamp fields and improve code clarity

2025-04-21 18:13:22 +08:00

fix: Update FsFileInfo to use pointer slices for children and remove redundant getter methods

2025-07-23 09:42:07 +08:00

feat(sync): add gRPC file synchronization service and integrate end-to-end

2025-10-20 12:48:53 +08:00

fix: unable to sync files to worker nodes when running tasks

2025-06-18 21:50:53 +08:00

refactor: extract token retrieval logic into GetAPITokenFromContext utility function

2025-06-16 21:49:27 +08:00

fix: update NotificationRequestDTO to include BSON field names for setting and channel

2025-09-16 15:43:43 +08:00

refactor: improve mongo client connection error logging format and remove redundant gRPC server start in MasterService

2025-07-09 14:06:10 +08:00

chore(node): add timing logs and improve node status diagnostics

2025-10-20 11:14:55 +08:00

fix: update NotificationRequestDTO to include BSON field names for setting and channel

2025-09-16 15:43:43 +08:00

refactor: enhance OpenAPI integration and security handling

2025-03-18 18:01:22 +08:00

feat: implement bounded goroutine pools for task execution and notification handling; enhance task scheduler with graceful shutdown and cleanup routines; update metric component for new time range options

2025-08-06 17:57:37 +08:00

refactor: enhance parameter handling and improve code clarity

2025-04-17 18:03:15 +08:00

refactor: update setting routes and enhance dependency management

2025-01-01 22:37:44 +08:00

feat(sync): add gRPC file synchronization service and integrate end-to-end

2025-10-20 12:48:53 +08:00

refactor: replace apex/log with structured logger across multiple services

2024-12-24 19:11:19 +08:00

feat(sync): add gRPC file synchronization service and integrate end-to-end

2025-10-20 12:48:53 +08:00

.editorconfig

feat: added modules

2024-06-14 15:42:50 +08:00

.gitignore

feat: added modules

2024-06-14 15:42:50 +08:00

BACKEND_TESTS_DISABLED.md

feat: disable backend unit tests and document reasons for integration test requirements

2025-10-09 12:35:09 +08:00

docker-compose.yml

ci: updated unit test

2024-10-20 18:55:02 +08:00

go.mod

fix: unable to filter tasks issue

2025-06-17 09:28:25 +08:00

go.sum

fix: unable to filter tasks issue

2025-06-17 09:28:25 +08:00

LICENSE

feat: added modules

2024-06-14 15:42:50 +08:00

main.go

refactor: code cleanup

2024-10-29 12:59:45 +08:00

README.md

feat: added modules

2024-06-14 15:42:50 +08:00

README.md

crawlab-core

Backend core modules for Crawlab