Commit Graph

  • 5d0c8c580e chore(deps): bump github.com/sirupsen/logrus in /backend dependabot/go_modules/backend/github.com/sirupsen/logrus-1.9.1 dependabot[bot] 2025-12-05 02:36:48 +00:00
  • ee11cd78ec test: add unit tests for IgnoreFileRegexPattern to validate ignored paths develop Marvin Zhang 2025-12-03 15:45:41 +08:00
  • aba7be3b86 fix: update IgnoreFileRegexPattern to exclude .git directory Marvin Zhang 2025-12-03 15:45:18 +08:00
  • 034fbf1a84 fix(git): include path parameter in listDir dispatch for directory listing Marvin Zhang 2025-12-03 15:39:03 +08:00
  • 6085ed0db0 fix(git): update listDir dispatch to include path parameter Marvin Zhang 2025-12-02 17:05:22 +08:00
  • b2ff8baed8 fix(spider): update node selection to use active nodes instead of all nodes fix(spider): optimize form update logic to watch specific fields for changes fix(grpc): adjust sync request ID handling for git and regular spiders Marvin Zhang 2025-12-02 11:42:29 +08:00
  • 0135e2ee3b chore(deps): bump golang.org/x/crypto from 0.25.0 to 0.45.0 in /core dependabot/go_modules/core/golang.org/x/crypto-0.45.0 dependabot[bot] 2025-11-20 00:54:27 +00:00
  • 6c8bd84a0d chore(deps): bump golang.org/x/crypto from 0.23.0 to 0.45.0 in /backend dependabot/go_modules/backend/golang.org/x/crypto-0.45.0 dependabot[bot] 2025-11-20 00:44:01 +00:00
  • 97ab39119c feat(specs): add detailed documentation for gRPC file sync migration and release 0.7.0 Marvin Zhang 2025-11-10 14:07:36 +08:00
  • 18c5eb3956 fix: replace string slicing with filepath.Dir() in gRPC file sync Marvin Zhang 2025-10-30 15:22:53 +08:00
  • bd99899182 fix(core): default to _id descending sort when sort is nil or parsing fails Marvin Zhang 2025-10-28 11:55:00 +08:00
  • 851097dc59 fix(node/service): implement worker client wrapper using local TaskServiceServer Marvin Zhang 2025-10-27 16:26:32 +08:00
  • b8e62c7b6b fix(node/master): add failure counter and grace period to node health checks; increase monitor interval Marvin Zhang 2025-10-23 14:49:30 +08:00
  • ef70312430 fix(grpc/server): add keepalive enforcement and params to match client Marvin Zhang 2025-10-23 10:58:22 +08:00
  • 893cb3cb8a fix(grpc/server): mark node active on Subscribe and notify on status change Marvin Zhang 2025-10-22 22:03:16 +08:00
  • 18fc84afb7 fix(grpc/client): clear reconnecting on failure and requeue reconnection after backoff Marvin Zhang 2025-10-21 22:03:17 +08:00
  • 138bed5c05 fix(grpc/client): wait for full reconnection readiness before clearing reconnecting flag Marvin Zhang 2025-10-21 21:26:57 +08:00
  • ba6d989c7e fix(controllers/health): return after responding OK to avoid falling through; tidy imports Marvin Zhang 2025-10-20 16:34:55 +08:00
  • ec3dd2d077 fix(grpc/client): protect GetGrpcClient with _clientMux lock to avoid race during singleton init Marvin Zhang 2025-10-20 13:43:55 +08:00
  • 2dfc66743b fix(grpc/client,node/task/handler): add RetryWithBackoff, stabilize reconnection, and retry gRPC ops Marvin Zhang 2025-10-20 13:01:10 +08:00
  • f441265cc2 feat(sync): add gRPC file synchronization service and integrate end-to-end Marvin Zhang 2025-10-20 12:48:53 +08:00
  • 61604e1817 fix(task/handler): ensure latest gRPC client is used for task fetch/subscribe Marvin Zhang 2025-10-20 12:22:34 +08:00
  • 4baa5fad59 fix(grpc/client): trigger reconnection on bad conn state and improve connection logging Marvin Zhang 2025-10-20 11:34:41 +08:00
  • 6020fef30b chore(node): add timing logs and improve node status diagnostics Marvin Zhang 2025-10-20 11:14:55 +08:00
  • 49165b2165 refactor(node): reorganize task reconciliation, prioritize worker cache, add periodic cleanup Marvin Zhang 2025-10-20 10:54:32 +08:00
  • 7cbe17723d Merge pull request #1606 from crawlab-team/develop test Marvin Zhang 2025-10-09 12:43:38 +08:00
  • 883c954b4e feat: update Dockerfile to include 'go mod tidy' before installation Marvin Zhang 2025-10-09 12:39:12 +08:00
  • 44fd0809e6 feat: disable backend unit tests and document reasons for integration test requirements Marvin Zhang 2025-10-09 12:35:09 +08:00
  • 5bff8823a8 feat: update test workflows to skip API tests and document controller test status Marvin Zhang 2025-10-09 11:34:26 +08:00
  • 5a0a640900 Merge pull request #1605 from crawlab-team/develop Marvin Zhang 2025-10-09 11:17:07 +08:00
  • 2a211923da Update core/task/handler/runner_sync.go Marvin Zhang 2025-10-09 11:14:32 +08:00
  • 587d9d0960 Merge branch 'test' into develop Marvin Zhang 2025-10-09 11:13:51 +08:00
  • 4c508557e7 feat: parameterize ports in Docker Compose for better configurability Marvin Zhang 2025-09-29 14:31:33 +08:00
  • 29ef8d67da feat: implement synchronization and error handling improvements in task reconciliation and file synchronization Marvin Zhang 2025-09-28 17:42:23 +08:00
  • e80256aa61 feat: add support for Chinese locale in Docker setup Marvin Zhang 2025-09-17 16:00:27 +08:00
  • b6e14a13fe refactor: remove obsolete task reconciliation service tests Marvin Zhang 2025-09-17 11:05:27 +08:00
  • afa5fab4c1 feat: enhance task reconciliation with worker-side status caching and synchronization Marvin Zhang 2025-09-17 11:03:35 +08:00
  • 8c2c23d9b6 feat: Update gRPC service definitions and implement CheckProcess method Marvin Zhang 2025-09-17 10:37:03 +08:00
  • c6834e9964 feat: enhance task reconciliation logic with improved status handling and error messaging Marvin Zhang 2025-09-17 10:18:13 +08:00
  • 8ebdd98f99 feat: enhance ORM functionality with toggle support and UI updates Marvin Zhang 2025-09-16 16:09:33 +08:00
  • bfe40e7c67 fix: comment out AI Assistant toggle button in Header.vue Marvin Zhang 2025-09-16 15:45:56 +08:00
  • 39f83d71b1 fix: update NotificationRequestDTO to include BSON field names for setting and channel Marvin Zhang 2025-09-16 15:43:43 +08:00
  • 196273c423 feat: implement ORM support with toggle functionality and UI updates Marvin Zhang 2025-09-16 15:18:35 +08:00
  • 875ca290b5 fix: update UseORM description to specify supported databases (MySQL, PostgreSQL, SQL Server) Marvin Zhang 2025-09-16 14:04:01 +08:00
  • 293e630f6f feat: add UseORM field to Database struct for ORM support Marvin Zhang 2025-09-16 13:30:50 +08:00
  • 8450b074c0 fix: comment out unused 'models' menu item in SystemDetail.vue Marvin Zhang 2025-09-16 09:33:15 +08:00
  • 56277e47be fix: remove unused build scripts to streamline the build process Marvin Zhang 2025-09-16 09:32:23 +08:00
  • 9ade564d0a Remove unused TypeScript declaration files for task, token, and user components in the Crawlab UI, streamlining the codebase and improving maintainability. Marvin Zhang 2025-09-16 09:31:34 +08:00
  • 7d1a61581e feat: add support for multi-architecture Docker builds with configurable input Marvin Zhang 2025-09-14 16:39:29 +08:00
  • 72177b2728 fix: update node disconnected status styling and behavior Marvin Zhang 2025-09-14 15:20:01 +08:00
  • 437c30b699 fix: ensure worker services depend on healthy master service Marvin Zhang 2025-09-14 15:02:06 +08:00
  • 829fcac3ff feat: add multi-platform support for Docker builds Marvin Zhang 2025-09-14 14:49:53 +08:00
  • 7c33fec784 refactor: remove unused fields from WorkerService struct Marvin Zhang 2025-09-12 18:17:36 +08:00
  • e221e3c640 feat: enhance gRPC client handling with improved reconnection logic and monitoring Marvin Zhang 2025-09-12 18:16:52 +08:00
  • 07bb7f8ba9 fix: enhance node metrics handling by checking state before accessing metrics map Marvin Zhang 2025-09-12 16:34:40 +08:00
  • 316878e129 test: add comprehensive tests for task reconciliation service handling offline nodes Marvin Zhang 2025-09-12 16:10:00 +08:00
  • 60be5072e5 feat: add node disconnection handling and update task statuses accordingly Marvin Zhang 2025-09-12 15:40:29 +08:00
  • 05a71da26e refactor: comment out AI chat features in NormalLayout.vue for future implementation Marvin Zhang 2025-09-12 15:01:29 +08:00
  • b5f10cb6a8 feat: add TypeScript interfaces for Vuex store modules Marvin Zhang 2025-09-12 14:57:42 +08:00
  • 14a94ff798 refactor: enhance error logging in writeLogLines to respect circuit breaker state Marvin Zhang 2025-09-12 14:34:27 +08:00
  • c0e230e5d8 refactor: rename PING code to HEARTBEAT in node service and update related proto files Marvin Zhang 2025-09-12 14:17:49 +08:00
  • d39c265483 feat: add PING message handling for connection health checks Marvin Zhang 2025-09-12 13:58:16 +08:00
  • 333dfd44c0 refactor: implement circuit breaker for log connections to prevent flooding during failures Marvin Zhang 2025-09-12 13:55:44 +08:00
  • 3edd2a1210 refactor: optimize connection health checks to reduce log stream interference; adjust health check intervals and implement non-blocking pings Marvin Zhang 2025-08-16 17:42:07 +08:00
  • 65aeb3ed8c feat: add PING mechanism for connection health checks; update proto and generated files - Introduced PING code in TaskServiceConnectCode enum for health checks. - Updated Runner to use proper PING messages instead of fake log messages for connection health checks. - Modified TaskServiceServer to handle PING requests and acknowledge them. - Adjusted generated gRPC files to reflect changes in proto definitions and ensure compatibility. Marvin Zhang 2025-08-16 17:19:21 +08:00
  • babecc46c0 refactor: update DependencySetupDialog to use language key for tag label; modify useDependencyList to dispatch node retrieval on mount Marvin Zhang 2025-08-08 11:12:05 +08:00
  • 45913ad7e4 refactor: implement health service for master and worker nodes; add health check script and integrate health checks into service lifecycle Marvin Zhang 2025-08-08 00:05:00 +08:00
  • 78f9e0ca8d refactor: update task worker pool to support dynamic max workers and improve queue management; enhance configuration defaults for node runners and task queue size Marvin Zhang 2025-08-07 18:16:23 +08:00
  • 6340a9b880 refactor: Move context initialization for graceful shutdown to appropriate locations Marvin Zhang 2025-08-07 17:27:11 +08:00
  • 6912b92501 refactor: enhance context handling across task runner and service components; ensure proper cancellation chains and prevent goroutine leaks Marvin Zhang 2025-08-07 15:40:48 +08:00
  • e1251d808b refactor: update method receivers to value type for cleanup and connection methods; enhance context usage for task client operations Marvin Zhang 2025-08-07 11:53:42 +08:00
  • f5f83a010f refactor: update metric list structure for improved layout and responsiveness; enhance styling with grid display Marvin Zhang 2025-08-07 11:12:50 +08:00
  • d042bc8cd7 refactor: improve connection readiness check and enhance goroutine management in gRPC client; ensure proper context handling in stream listeners Marvin Zhang 2025-08-07 11:12:46 +08:00
  • 060396af3d refactor: enhance data structure annotations and improve layout responsiveness in Home component Marvin Zhang 2025-08-07 09:58:11 +08:00
  • 44dd68918f refactor: improve goroutine management and context handling in task and stream operations; ensure graceful shutdown and prevent leaks Marvin Zhang 2025-08-07 00:16:46 +08:00
  • 784ffc8b52 feat: implement task management service operations, stream manager, and worker pool Marvin Zhang 2025-08-06 18:29:08 +08:00
  • 3678d14082 feat: implement bounded goroutine pools for task execution and notification handling; enhance task scheduler with graceful shutdown and cleanup routines; update metric component for new time range options Marvin Zhang 2025-08-06 17:57:37 +08:00
  • 9745129e33 feat: configure test database as master node for testing Marvin Zhang 2025-07-23 15:28:51 +08:00
  • cf5ec81250 fix: add nil checks for logger in config initialization and logging methods Marvin Zhang 2025-07-23 15:07:39 +08:00
  • a2d13fae36 feat: temporarily disable batch file saving route and implement alternative handler in spider controller Marvin Zhang 2025-07-23 14:55:04 +08:00
  • b4288b08a5 fix: Update FsFileInfo to use pointer slices for children and remove redundant getter methods Marvin Zhang 2025-07-23 09:42:07 +08:00
  • 3c3ff09723 feat: enhance gRPC client with health check functionality and improved connection handling Marvin Zhang 2025-07-09 14:38:53 +08:00
  • 20ba390cf6 refactor: improve mongo client connection error logging format and remove redundant gRPC server start in MasterService Marvin Zhang 2025-07-09 14:06:10 +08:00
  • 46c0cd6298 refactor: update gRPC client access patterns to use safe getter methods for improved error handling Marvin Zhang 2025-07-08 18:08:46 +08:00
  • 8bd3ef0b72 feat: add goroutine count metric to the MetricService and update related files Marvin Zhang 2025-07-08 14:08:31 +08:00
  • 7e4187f388 chore: add TypeScript declaration files for various components and services Marvin Zhang 2025-07-08 14:08:13 +08:00
  • 00daa0ed96 fix: enhance gRPC client reconnection logic and add goroutine monitoring for potential leaks Marvin Zhang 2025-07-08 13:39:39 +08:00
  • f8e9c45a85 fix: enhance gRPC client connection management with circuit breaker and keep-alive settings Marvin Zhang 2025-07-08 13:34:43 +08:00
  • 92046a8c2e fix: improve task cancellation and connection health check logic with timeout handling Marvin Zhang 2025-06-27 14:02:24 +08:00
  • 9f251f3ebe fix: enhance task cancellation logic with graceful termination and stuck task cleanup Marvin Zhang 2025-06-27 13:50:21 +08:00
  • 89514b0154 feat: implement zombie process prevention and cleanup mechanisms in task runner Marvin Zhang 2025-06-23 13:54:43 +08:00
  • 1008886715 fix: enhance task service resilience with connection health monitoring and periodic cleanup Marvin Zhang 2025-06-23 11:57:05 +08:00
  • 682efa807d ci: add version check before publishing to NPM and provide publish summary Marvin Zhang 2025-06-20 17:53:15 +08:00
  • 5837472de5 fix: update API client constructor and improve error handling for missing API token and URL chore: update package version to 0.1.1-dev.6 and adjust dependencies in pnpm-lock.yaml refactor: change import paths to relative in tools.ts for consistency Marvin Zhang 2025-06-20 15:55:50 +08:00
  • 99d59b8e90 chore: add publishConfig to package.json for public access Marvin Zhang 2025-06-20 15:00:16 +08:00
  • 18137cce0d fix: update jest test command to allow passing with no tests Marvin Zhang 2025-06-20 14:58:12 +08:00
  • b7b7f25451 refactor: remove unused response variables in database, git, node, project, schedule, spider, task modules Marvin Zhang 2025-06-20 14:55:54 +08:00
  • 4f33460d45 chore: remove .gitattributes file to simplify language detection Marvin Zhang 2025-06-20 14:45:22 +08:00
  • 13038d4a1a fix: unable to sync files from master in worker nodes Marvin Zhang 2025-06-20 14:42:52 +08:00
  • 52b180aaa0 ci: Add GitHub Actions workflow for publishing MCP package Marvin Zhang 2025-06-20 11:09:02 +08:00