- add RetryWithBackoff helper to grpc client for exponential retry with backoff and reconnection-aware handling
- increase reconnectionClientTimeout to 90s and introduce connectionStabilizationDelay; wait briefly after reconnection to avoid immediate flapping
- refresh reconnection flag while waiting for client registration and improve cancellation message
- replace direct heartbeat RPC with RetryWithBackoff in WorkerService (use extended timeout)
- use RetryWithBackoff for worker node status updates in task handler and propagate errors
Add svc.getGrpcClient() helper and use it when obtaining TaskClient so task fetch and
subscribe operations don't hold a stale client instance after ResetGrpcClient().
- master: add TIMING logs in setWorkerNodeOnline to mark start and completed DB update
- handler: log node status for reconnection debugging and include active/enabled values in "node not active or enabled" error
- Implemented PING message handling in TaskServiceServer to acknowledge health check pings.
- Updated isConnectionHealthy method in Runner to use a non-blocking approach for health checks, preventing interference with log streams.
- Introduced lastConnCheck timestamp to optimize health check frequency based on recent activity.
- Added PING code to TaskServiceConnectCode enum in proto definition and generated files.
- Updated gRPC client and server interfaces to support new PING functionality.
- Introduced PING code in TaskServiceConnectCode enum for health checks.
- Updated Runner to use proper PING messages instead of fake log messages for connection health checks.
- Modified TaskServiceServer to handle PING requests and acknowledge them.
- Adjusted generated gRPC files to reflect changes in proto definitions and ensure compatibility.
- Added service_operations.go for task management including run, cancel, and execution logic.
- Introduced stream_manager.go to handle task streams and manage cancellation signals.
- Created worker_pool.go to manage a bounded pool of workers for executing tasks concurrently.
- Implemented graceful shutdown and cleanup mechanisms for task runners and streams.
- Enhanced error handling and logging throughout the task management process.
- Updated the error handling in the Run method to log a warning message when file synchronization fails, enhancing visibility into potential issues during task execution.
- Updated timestamp fields across the codebase from `*_ts` to `*_at` for consistency and clarity.
- Renamed constants for node status from "on"/"off" to "online"/"offline" to better reflect their meanings.
- Enhanced validation and error handling in various components to ensure data integrity.
- Refactored test cases to align with the new naming conventions and improve readability.
- Renamed configurePythonPath, configureNodePath, and configureGoPath methods into a single configureEnv method for improved organization and clarity.
- Streamlined the handling of environment variables by using a map to avoid duplicates and facilitate easier manipulation.
- Enhanced logging for environment variable settings, providing better visibility during configuration.
- Ensured all necessary environment variables are set correctly, including Crawlab-specific variables and global system variables.
- Replaced direct manipulation of r.cmd.Env with os.Setenv for setting PYENV_ROOT, PATH, NODE_PATH, and GOPATH, enhancing error handling and clarity.
- Ensured proper initialization of default environment variables in the configureEnv method, maintaining organization and readability in the environment setup process.
- Updated the configurePythonPath and configureNodePath methods to append environment variables directly to r.cmd.Env instead of using os.Setenv, enhancing the clarity and maintainability of the environment setup.
- Ensured the PATH variable is updated correctly by checking for existing paths before appending, preventing duplicates.
- Moved default environment variable initialization to the configureEnv method for better organization.
- Updated the configureNodePath method to properly set the PATH environment variable by concatenating node_bin and node_modules paths, ensuring the correct environment setup for Node.js development.
- Added a new utility function GetNodeBinPath to retrieve the global node_bin path from configuration, with a default fallback.
- Updated configureNodePath method to set the PATH environment variable to include the node_bin path if it's not already present, improving the task runner's environment setup for Node.js development.
- Introduced a new method to configure the global node_bin path in the Runner, ensuring it is included in the system PATH if not already present.
- Added GetNodeBinPath function in utils to retrieve the node_bin path from configuration, with a default fallback.
- Enhanced environment variable management for better integration of Node.js binaries.
- Updated GetHealthFn to return an error for better error handling and clarity.
- Introduced a new test file for schedule management, covering various endpoints including creation, retrieval, updating, and deletion of schedules.
- Added tests for task management, including task creation, retrieval, updating, and cancellation.
- Implemented utility tests for filtering and response generation to ensure consistent API behavior.
- Improved logging in the task scheduler service for better traceability.
- Replaced direct calls to getBaseFileFsSvc with a new method fs.GetBaseFileFsSvc in base_file.go for improved clarity and maintainability.
- Introduced SpiderTemplateService interface and implemented registry service for managing spider templates, enhancing template handling in the spider controller.
- Added template-related fields to the Spider model to support template functionality.
- Created utility functions for string case conversions in utils/string.go to facilitate consistent formatting across the codebase.
- Updated environment configuration to retrieve the Python path dynamically, improving flexibility in the task runner's setup.
- Added configuration initialization in db.go to ensure proper setup of application settings.
- Refactored runner_test.go to streamline IPC message handling by introducing a setupPipe function and an initRunner function for better readability and maintainability.
- Improved synchronization in tests by using channels for signaling readiness and processing, enhancing the reliability of IPC message handling.
- Updated test cases to validate IPC message processing and error handling, ensuring robustness in the task runner's functionality.
- Introduced a new method configureGoPath in runner.go to set the GOPATH environment variable based on the retrieved Go path.
- Updated configureEnv to call configureGoPath, ensuring the Go path is configured alongside Node.js paths.
- Added a new utility function GetGoPath in config.go to retrieve the Go path from configuration, with a default fallback.
- These changes enhance the task runner's environment setup by supporting Go development alongside existing Node.js configurations.
