- Introduced a new method to configure the global node_bin path in the Runner, ensuring it is included in the system PATH if not already present.
- Added GetNodeBinPath function in utils to retrieve the node_bin path from configuration, with a default fallback.
- Enhanced environment variable management for better integration of Node.js binaries.
- Replaced "conditions" parameter with "filter" in various controller methods to unify the filtering approach.
- Updated related functions to utilize the new filter parameter, enhancing consistency in query handling.
- Improved overall code readability and maintainability by aligning naming conventions and refactoring filter-related logic.
- Enhanced error handling in BaseController to return a more descriptive "item not found" message when no documents are found in MongoDB.
- Standardized naming conventions for configuration constants by updating DefaultMcpServerBaseUrl to DefaultMCPServerBaseUrl and introducing DefaultMCPClientBaseUrl for clarity and consistency.
- Ensured alignment with existing coding standards and improved code readability.
- Changed DefaultMcpServerBaseUrl from "http://localhost:9000/sse" to "http://localhost:9000" for improved clarity and consistency in configuration.
- Ensured alignment with other base URL definitions in the config.go file.
- Added github.com/getkin/kin-openapi v0.130.0 to go.mod and updated go.sum for improved OpenAPI integration.
- Introduced DefaultOpenAPIUrl constant in config.go for easier configuration of OpenAPI URL.
- Implemented GetOpenAPIUrl function to retrieve the OpenAPI URL from configuration, enhancing flexibility in API documentation handling.
Added a new configuration utility function GetMcpServerBaseUrl() to:
- Define a default MCP server base URL
- Allow custom configuration via Viper
- Provide a consistent way to retrieve the MCP server base URL
Reverts the previous change that set DefaultInstallEnvs to "node,browser", returning it to an empty default value to maintain configuration flexibility.
- Introduced a new function GetInstallEnvs in config.go to retrieve installation environment variables
- Added DefaultInstallEnvs constant with an empty default value
- Implemented fallback mechanism to split DefaultInstallEnvs if no configuration is found
- Enhances configuration flexibility for installation environments
- Replaced direct calls to getBaseFileFsSvc with a new method fs.GetBaseFileFsSvc in base_file.go for improved clarity and maintainability.
- Introduced SpiderTemplateService interface and implemented registry service for managing spider templates, enhancing template handling in the spider controller.
- Added template-related fields to the Spider model to support template functionality.
- Created utility functions for string case conversions in utils/string.go to facilitate consistent formatting across the codebase.
- Updated environment configuration to retrieve the Python path dynamically, improving flexibility in the task runner's setup.
- Introduced a new method configureGoPath in runner.go to set the GOPATH environment variable based on the retrieved Go path.
- Updated configureEnv to call configureGoPath, ensuring the Go path is configured alongside Node.js paths.
- Added a new utility function GetGoPath in config.go to retrieve the Go path from configuration, with a default fallback.
- These changes enhance the task runner's environment setup by supporting Go development alongside existing Node.js configurations.
- Removed redundant home directory retrieval and nvm checks in the configureNodePath method.
- Introduced a new utility function GetNodeModulesPath to centralize the logic for determining the global node_modules path.
- Updated environment variable setup to use the new utility function, improving clarity and maintainability of the code.
- Improved error messages in the FileLogDriver's cleanup method to include error details for better debugging.
- Updated the default task log path from '/app/logs/tasks' to '/var/log/crawlab/tasks' to ensure consistency across environments.
- Changed the default task log path from '/var/log/crawlab/tasks' to '/app/logs/tasks' to align with the application's directory structure and improve portability in different deployment environments.
- Updated various dependencies in go.mod and go.sum files, including cloud.google.com/go/compute/metadata to v0.5.2, google.golang.org/grpc to v1.69.2, and google.golang.org/protobuf to v1.36.1.
- Refactored gRPC service definitions to use the latest protoc-gen-go and protoc versions, ensuring compatibility with the latest gRPC-Go features.
- Introduced a new logger utility in core/utils/logger.go to improve logging capabilities across the application.
- Added a README.md for gRPC setup and compilation guidance, enhancing developer experience.
- Improved the Python installation script to handle version listing more effectively and ensure user-specific environment setup.
- Removed unused ApiApp and ServerApp interfaces from core/apps/interfaces.go to streamline the codebase.
- Updated the GetApi method in the Server struct to return a pointer to the Api type for better type handling.
- Simplified the GetGinMode function in core/utils/config.go to always return gin.ReleaseMode, removing unnecessary conditional checks for development mode.
- These changes enhance code clarity and maintainability by eliminating redundant code and improving type safety.
- Replaced all instances of apex/log with a structured logger interface in various services, including Api, Server, Config, and others, to enhance logging consistency and context.
- Updated logging calls to utilize the new logger methods, improving error tracking and service monitoring.
- Added logger initialization in services and controllers to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Removed unused apex/log imports and cleaned up related code for better maintainability.
- Updated logging messages in GrpcClient to provide clearer context, changing "ready" to "client is now ready" and "stopped" to "client has stopped".
- Refactored test setup in runner_test.go to remove unnecessary error checks during gRPC client start for cleaner code.
- Renamed GetDependencySetupScriptRoot to GetInstallRoot and updated related constants for better clarity and consistency in configuration management.
- Introduced GetApiPort and GetApiPath functions to retrieve API port and path from configuration, with defaults provided.
- Updated GetApiEndpoint to construct the API endpoint URL using the new port and path functions, ensuring proper formatting.
- Added constants for default API port and path to enhance configurability.
