- Introduced a new DependencyInstallerService interface to define methods for managing dependency installation commands.
- Implemented registry service for managing the DependencyInstallerService instance.
- Enhanced the task runner to install dependencies if available, including command execution and logging for stdout and stderr.
- Improved error handling and logging throughout the task runner's dependency installation process.
- Updated the runner's methods to utilize the new dependency management features, ensuring better integration and functionality.
- Enhanced the IPC message handling in runner_test.go by adding detailed logging for better traceability.
- Refactored the test setup to use channels for synchronization and improved error handling during message processing.
- Updated the runner.go file to rename variables for clarity and streamline the IPC reader implementation.
- Improved the cleanup process in tests to ensure proper resource management and context cancellation.
- Improved error messages in the FileLogDriver's cleanup method to include error details for better debugging.
- Updated the default task log path from '/app/logs/tasks' to '/var/log/crawlab/tasks' to ensure consistency across environments.
- Changed the default task log path from '/var/log/crawlab/tasks' to '/app/logs/tasks' to align with the application's directory structure and improve portability in different deployment environments.
- Added support for new dependency file types: 'go.mod' and 'pom.xml' in dependency.go.
- Refactored command configuration in runner.go to improve logging and error handling.
- Introduced a new method to configure Node.js paths, enhancing environment setup for tasks.
- Enhanced IPC message handling with detailed logging for better traceability.
- Updated service logging to remove unnecessary prefixes for cleaner output.
- Improved command execution handling in process.go for better compatibility across platforms.
- Added 'zip' and 'unzip' utilities to the dependency installation script for improved file handling capabilities.
- Updated usage instructions in the Python installation script to include a new 'setup' command for configuring pyenv, enhancing user experience and script functionality.
- Updated log messages in NodeServiceServer and TaskServiceServer to remove the "[NodeServiceServer]" and "[TaskServiceServer]" prefixes for cleaner output.
- This change enhances log readability and maintains consistency across logging practices in the application.
- Updated the dependencies in the Docker base image installation scripts by removing unnecessary packages and ensuring essential tools are included.
- Simplified the Python installation script by refining the list of required libraries, enhancing clarity and reducing potential installation issues.
- Modified the requirements.txt to include only necessary packages, improving the efficiency of the Python environment setup.
- Modified the Dockerfile to require Python version 3.12 explicitly during the installation process.
- This change ensures that the installation script adheres to the specified version, improving consistency and reliability in Docker environments.
- Modified the Python installation script to remove the default version assignment and enforce version specification during installation.
- Updated usage instructions to reflect the change from a default version to requiring a user-defined version, improving clarity and preventing installation errors.
- This change enhances the robustness of the installation process by ensuring users explicitly define the Python version they wish to install.
- Eliminated the default_version field from the DependencyConfig struct in dependency_config.go to streamline the configuration model.
- This change simplifies the dependency management process and aligns with recent updates in the application structure.
- Introduced a new "setup" command in the Python installation script to facilitate the setup of pyenv.
- This enhancement improves the usability of the script by allowing users to easily configure their Python environment.
- Added logging for error handling in the MasterService when setting a worker node offline, replacing the previous trace.PrintError with a more informative log message.
- Enhanced WorkerService subscription method with debug logs to indicate subscription attempts and status, improving traceability during connection processes.
- Replaced viper calls with utility functions in GetSystemInfo to improve code clarity and maintainability.
- Added a new system.go file with utility functions for retrieving system version and edition information.
- Enhanced PrintLogoWithWelcomeInfo to include detailed system information, improving user experience during server startup.
- Updated output formatting for better readability and consistency in welcome messages.
- Changed the default Python version in the Docker base image installation script from 3.12 to 3.12.8 to ensure the latest patch version is used.
- This update enhances the reliability and security of the Python installation process in Docker environments.
- Updated various dependencies in go.mod and go.sum files, including cloud.google.com/go/compute/metadata to v0.5.2, google.golang.org/grpc to v1.69.2, and google.golang.org/protobuf to v1.36.1.
- Refactored gRPC service definitions to use the latest protoc-gen-go and protoc versions, ensuring compatibility with the latest gRPC-Go features.
- Introduced a new logger utility in core/utils/logger.go to improve logging capabilities across the application.
- Added a README.md for gRPC setup and compilation guidance, enhancing developer experience.
- Improved the Python installation script to handle version listing more effectively and ensure user-specific environment setup.
- Added jq to the dependencies in the Docker base image installation script.
- Updated the Python installation script to support handling of custom requirements via command-line arguments.
- Implemented a new function to manage requirements installation, falling back to a default requirements file if none is provided.
- Improved version check logic to prevent redundant installations of Python versions, enhancing efficiency in Docker environments.
- Refactored the Python installation script to support multiple commands: install, uninstall, switch, and list.
- Added functions for usage printing, dependency installation, pyenv setup, Python version verification, symlink creation, and cleanup.
- Improved error handling for version checks and command execution, enhancing usability and flexibility in Docker environments.
- Default Python version is set to 3.12, with the option to specify a different version during installation.
- Changed the installation script for pyenv to create a user-specific environment file at $HOME/.pyenv-env.sh instead of a system-wide file in /etc/profile.d/.
- Updated the script to source the new environment file, ensuring that the pyenv setup is applied correctly for the user.
- This change improves the flexibility and usability of the pyenv installation in Docker environments.
- Redirected the output of the nginx service start command to /dev/null to prevent cluttering the console with unnecessary messages during server startup.
- This change enhances the user experience by providing a cleaner output when initializing the Docker environment.
- Added a new line after the logo and welcome information prints in the PrintLogoWithWelcomeInfo function to improve output readability.
- This change enhances the user experience by ensuring that the printed information is visually separated, making it easier to read during server startup.
- Added github.com/common-nighthawk/go-figure as a new indirect dependency in both backend and core modules to support logo rendering functionality.
- Removed the github.com/imroc/req dependency from backend and go.sum files, streamlining the dependency list and improving project organization.
- Introduced a new utility function to print a logo and welcome information for the Crawlab server, enhancing user experience during startup.
- Updated logger variable names in the apps package for consistency and clarity.
- Added a new dependency on github.com/common-nighthawk/go-figure to facilitate logo rendering.
- Improved the server command to display the logo when the server starts, provided the user is not using the pro version.
- Removed the db module from the Dockerfile, reflecting the recent refactor that consolidates database functionality into the core/mongo package.
- This change simplifies the Docker image setup by eliminating unnecessary dependencies and improving overall project organization.
- Deleted the db module, consolidating database-related functionality into the core/mongo package for better organization and maintainability.
- Updated all import paths across the codebase to replace references to the removed db module with core/mongo.
- Cleaned up unused code and dependencies, enhancing overall project clarity and reducing complexity.
- This refactor improves the structure of the codebase by centralizing database operations and simplifying module management.
- Removed unused ApiApp and ServerApp interfaces from core/apps/interfaces.go to streamline the codebase.
- Updated the GetApi method in the Server struct to return a pointer to the Api type for better type handling.
- Simplified the GetGinMode function in core/utils/config.go to always return gin.ReleaseMode, removing unnecessary conditional checks for development mode.
- These changes enhance code clarity and maintainability by eliminating redundant code and improving type safety.
- Replaced all instances of apex/log with a structured logger interface in various services, including Api, Server, Config, and others, to enhance logging consistency and context.
- Updated logging calls to utilize the new logger methods, improving error tracking and service monitoring.
- Added logger initialization in services and controllers to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Removed unused apex/log imports and cleaned up related code for better maintainability.
- Updated the deps.sh script to include the 'zip' package in the list of installed dependencies for the Docker base image.
- This addition ensures that the zip utility is available for use in subsequent Docker image builds and applications.
- Removed direct usage of apex/log in favor of a structured logger interface for improved logging consistency and context.
- Updated logging calls in MasterService and WorkerService to utilize the new logger, enhancing error tracking and service monitoring.
- Added logger initialization in both services to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Updated logging messages in GrpcClient to provide clearer context, changing "ready" to "client is now ready" and "stopped" to "client has stopped".
- Refactored test setup in runner_test.go to remove unnecessary error checks during gRPC client start for cleaner code.
- Renamed GetDependencySetupScriptRoot to GetInstallRoot and updated related constants for better clarity and consistency in configuration management.
- Added DependencyActionSync and DependencyActionSetup constants to improve dependency management.
- Refactored GrpcClient to utilize a logger interface for consistent logging across connection states and errors.
- Updated Start, Stop, and connection methods to replace direct log calls with logger methods, enhancing log context and readability.
- Simplified test cases by removing error checks on gRPC client start, ensuring cleaner test setup.
- Added TotalDependencies and SearchReady fields to the DependencyConfig struct for improved dependency management.
- Retained the Setup field for backward compatibility while ensuring proper JSON and BSON handling.
- Added Debug, Info, Warn, Error, and Fatal methods to the Logger interface for comprehensive logging capabilities.
- Implemented corresponding methods in ServiceLogger to facilitate structured logging with service context.
- Enhanced the logging functionality to support various log levels, improving error tracking and debugging.
- Introduced Fatalf method in Logger interface for logging fatal messages with formatted content.
- Implemented Fatalf in ServiceLogger to log fatal messages and exit the program, enhancing error handling capabilities.
- Added Logger interface in core/interfaces/logger.go for standardized logging methods (Debugf, Infof, Warnf, Errorf).
- Introduced ServiceLogger in core/utils/log.go, which prefixes log messages with the service name for better context.
- Implemented logging methods in ServiceLogger to utilize the apex/log package for structured logging.
- Added utility function to normalize BSON ObjectId in query parameters for gRPC methods.
- Updated GetOne, GetMany, DeleteOne, DeleteMany, UpdateOne, UpdateMany, ReplaceOne, and UpsertOne methods to utilize the new normalization function.
- Introduced new DependencyConfigSetup model instance in base model definitions for improved structure.
- Introduced state management in GrpcClient to monitor and handle connection states effectively.
- Added a reconnect channel and a state monitoring goroutine to facilitate automatic reconnections on state changes.
- Updated the connect method to initiate a reconnection loop upon connection loss.
- Enhanced logging for connection state changes and errors during connection attempts.
- Refactored tests to ensure proper initialization of gRPC client and server, improving test reliability and coverage.
- Replaced direct gRPC client instantiation with a call to GetGrpcClient().Start() for improved consistency and maintainability across test cases.
- Cleaned up unnecessary imports in model_service.go to streamline the codebase.
- Enhanced test setup by ensuring the gRPC client is properly initialized before executing service methods.
- Introduced a new POST endpoint '/:id/run' in the router to trigger the execution of a schedule.
- Implemented the PostScheduleRun function to handle the execution logic, including validation of input parameters and scheduling tasks.
- Refactored the PostSpiderRun function to streamline the scheduling process by directly calling the admin service.
- Updated the FsFileInfo struct to clarify the children field description.
- Modified the Task model to make NodeIds field optional in JSON serialization.
- Introduced GetApiPort and GetApiPath functions to retrieve API port and path from configuration, with defaults provided.
- Updated GetApiEndpoint to construct the API endpoint URL using the new port and path functions, ensuring proper formatting.
- Added constants for default API port and path to enhance configurability.
- Removed the print flag from handleError function, simplifying error logging based on the development environment.
- Introduced a new performRequest function for standardized HTTP requests with JSON bodies, enhancing code reusability.
- Updated SendIMNotification and related functions to utilize the new RequestParam type for better clarity and consistency.
- Normalized HTTP request paths in the createHttpRequest method to ensure correct URL formatting.
- Added detailed error logging for JSON unmarshaling failures in syncFiles method.
- Introduced a NewHttpClient function to create HTTP clients with customizable timeouts.
- Added WaitForReady method to GrpcClient for blocking until the client is ready.
- Updated WorkerService to utilize WaitForReady for ensuring gRPC client readiness before starting.
- Refactored ModelService to consistently use GetGrpcClient for context management.
- Changed logging level for received metrics in MetricServiceServer from Info to Debug.
- Modified error handling in HandleError to conditionally print errors based on the environment.
- Cleaned up unused GrpcClient references in various services, improving code clarity.
- Updated GitHub Actions workflow to specify the Dockerfile location and set the build context to the root of the repository.
- Modified Dockerfile to copy backend and other required modules from their respective directories, improving the build process.
- Updated .dockerignore to exclude temporary files, Git directories, and Node modules.
- Modified Dockerfile to copy required modules from the parent directory and build the Go application from the backend directory.
