- Updated router groups to utilize the new crawlab-team/fizz package for improved routing capabilities.
- Refactored controller methods to accept Fizz router groups, enhancing consistency and maintainability.
- Simplified route registration by incorporating OpenAPI metadata directly into group definitions.
- Improved error handling and response generation in sync controller methods for better clarity.
- Enhanced overall code structure by standardizing route definitions and improving parameter handling.
- Updated Go version to 1.23.7 in backend/go.mod.
- Upgraded various dependencies to their latest versions for improved stability and performance.
- Adjusted indirect dependencies for better compatibility.
- Enhanced overall clarity by standardizing import paths across the project.
- Updated various dependencies in go.mod and go.sum for improved stability and performance.
- Replaced references to the old fizz package with the new crawlab-team/fizz package in multiple files.
- Adjusted indirect dependencies to their latest versions for better compatibility.
- Enhanced overall code clarity by standardizing import paths across the project.
- Updated multiple controller methods to return VoidResponse instead of generic Response[any].
- Consolidated error handling to utilize GetErrorVoidResponse for consistent error responses.
- Enhanced parameter handling in export and file management functions for improved clarity and maintainability.
- Refactored health check and login/logout methods to align with new response structure.
- Improved overall consistency in response formatting across various endpoints.
- Renamed filter operation constants for clarity and consistency.
- Introduced short forms for certain filter operations.
- Updated test cases to reflect changes in filter operation handling.
- Enhanced type handling in filter functions to support int64 values for conditions.
- Improved query generation logic to accommodate new constant names and types.
- Updated GetHealthFn to return an error for better error handling and clarity.
- Introduced a new test file for schedule management, covering various endpoints including creation, retrieval, updating, and deletion of schedules.
- Added tests for task management, including task creation, retrieval, updating, and cancellation.
- Implemented utility tests for filtering and response generation to ensure consistent API behavior.
- Improved logging in the task scheduler service for better traceability.
- Refactored export and filter controller methods to utilize structured parameter types for improved clarity and maintainability.
- Consolidated error handling and response generation, ensuring consistent API responses across methods.
- Updated function signatures to eliminate unnecessary context parameters, enhancing type safety and readability.
- Improved handling of query parameters and conditions, streamlining the logic for better performance and maintainability.
- Refactored multiple controller methods to accept structured parameters for improved clarity and maintainability.
- Consolidated error handling and response generation across various endpoints.
- Updated function signatures to eliminate unnecessary context parameters and enhance type safety.
- Improved consistency in response formatting and error handling across controllers.
- Enhanced file handling methods to support multipart file uploads and directory operations more effectively.
- Updated Go version in go.work and backend/go.mod to 1.23.7
- Updated various dependencies in go.sum and backend/go.sum
- Refactored models to remove generic type parameters from BaseModel
- Introduced new utility functions for consistent API responses
- Removed unused utility files from controllers
- Introduced a new OpenAPI wrapper using Fizz for improved API documentation
- Refactored base controller to support more flexible route handling
- Added dynamic route registration with OpenAPI metadata
- Implemented generic response types for consistent API responses
- Updated router initialization to support OpenAPI documentation endpoint
- Improved route and resource naming utilities
- Migrated existing controllers to use new routing and response mechanisms
Added a new configuration utility function GetMcpServerBaseUrl() to:
- Define a default MCP server base URL
- Allow custom configuration via Viper
- Provide a consistent way to retrieve the MCP server base URL
Updated LLMProvider struct to include a new 'DeploymentName' field:
- Added 'deployment_name' tag for JSON and BSON serialization
- Modified IsUnset() method to check for deployment name in Azure OpenAI configuration
- Ensures more comprehensive validation for Azure OpenAI provider setup
Introduced a new method to check if an LLM provider is considered unset:
- Added Unset field to LLMProvider struct
- Implemented IsUnset() method with logic to validate provider configuration
- Handles special cases for different provider types (azure-openai, openai-compatible)
- Checks for empty API key, missing models, and provider-specific configuration requirements
Simplified LLMProvider model:
- Removed description, config schema, default config, and supported features
- Added new field for API version
- Cleaned up and focused on core provider attributes
Restructured LLMProvider and removed LLMModel:
- Added collection tag to LLMProvider
- Renamed and reorganized provider fields
- Removed separate LLMModel struct
- Simplified model and provider attributes
Introduced two new data models for managing Language Learning Models (LLMs):
- LLMProvider: Represents LLM providers like OpenAI, Anthropic
- LLMModel: Represents specific models within a provider
Models include key attributes such as:
- Naming and display information
- Enabled/priority status
- Supported features
- Token pricing
- Configuration schemas
Implemented a new GetTokenList handler in the token controller to:
- Fetch tokens created by the current user
- Support pagination and filtering
- Return tokens with total count
- Handle cases with no documents gracefully
Adds an early return after handling bad request errors in the GetExportDownload function to avoid unnecessary error processing and improve error handling clarity.
- Deleted CHANGELOG-zh.md and CHANGELOG.md
- Added version-specific changelog files in the changelog/ directory for v0.2, v0.3, v0.4, v0.5, and v0.6
- Included both Chinese and English versions for some changelog files
- Organized changelogs by version with detailed feature and bug fix descriptions
Reverts the previous change that set DefaultInstallEnvs to "node,browser", returning it to an empty default value to maintain configuration flexibility.
- Introduced a new function GetInstallEnvs in config.go to retrieve installation environment variables
- Added DefaultInstallEnvs constant with an empty default value
- Implemented fallback mechanism to split DefaultInstallEnvs if no configuration is found
- Enhances configuration flexibility for installation environments
- Added 'apt-get update -y' before package installation to ensure latest package information
- Ensures dependencies are installed from the most recent package lists
- Updated the Spider model to introduce a new SpiderTemplateParams struct for improved template handling.
- Refactored string utility functions in utils/string.go to include a new replaceChars function, streamlining character replacement across multiple functions.
- Enhanced ToSnakeCase and ToKebabCase functions to utilize the new replaceChars function for better maintainability and readability.
- Added splitStringWithQuotes function to facilitate string manipulation with quotes, improving overall utility in string processing.
- Introduced a new function ToKebabCase in utils/string.go to convert strings to kebab-case format.
- The function trims whitespace, converts to lowercase, and replaces spaces and underscores with hyphens, enhancing string manipulation capabilities in the codebase.
- Replaced direct calls to getBaseFileFsSvc with a new method fs.GetBaseFileFsSvc in base_file.go for improved clarity and maintainability.
- Introduced SpiderTemplateService interface and implemented registry service for managing spider templates, enhancing template handling in the spider controller.
- Added template-related fields to the Spider model to support template functionality.
- Created utility functions for string case conversions in utils/string.go to facilitate consistent formatting across the codebase.
- Updated environment configuration to retrieve the Python path dynamically, improving flexibility in the task runner's setup.
- Added configuration initialization in db.go to ensure proper setup of application settings.
- Refactored runner_test.go to streamline IPC message handling by introducing a setupPipe function and an initRunner function for better readability and maintainability.
- Improved synchronization in tests by using channels for signaling readiness and processing, enhancing the reliability of IPC message handling.
- Updated test cases to validate IPC message processing and error handling, ensuring robustness in the task runner's functionality.
- Introduced a new method configureGoPath in runner.go to set the GOPATH environment variable based on the retrieved Go path.
- Updated configureEnv to call configureGoPath, ensuring the Go path is configured alongside Node.js paths.
- Added a new utility function GetGoPath in config.go to retrieve the Go path from configuration, with a default fallback.
- These changes enhance the task runner's environment setup by supporting Go development alongside existing Node.js configurations.
- Changed package manager command from 'apt-get' to 'apt' for improved usability.
- Added additional Python dependencies: 'libbz2-dev', 'libreadline-dev', 'libsqlite3-dev', 'xz-utils', and 'liblzma-dev' to enhance the environment setup.
- Updated 'crawlab-sdk' version in requirements.txt from '0.7.0rc1' to '0.7.0rc5' for compatibility and feature improvements.
- Updated IPC reader initialization in runner_test.go to use a channel for signaling readiness, improving synchronization.
- Added error logging when writing to the pipe to enhance traceability during tests.
- These changes improve the reliability and clarity of the test setup for the task runner.
- Removed redundant home directory retrieval and nvm checks in the configureNodePath method.
- Introduced a new utility function GetNodeModulesPath to centralize the logic for determining the global node_modules path.
- Updated environment variable setup to use the new utility function, improving clarity and maintainability of the code.
- Added a task status update to 'processing' at the start of the Run method in runner.go, improving task tracking.
- Removed redundant task status update from the end of the Run method to streamline the execution flow.
- Updated command execution in process.go to use 'bash' instead of 'sh' for better compatibility across environments.
- Changed route parameter from ':id' to ':key' in settings-related routes for better clarity and consistency.
- Updated GetSetting, PostSetting, and PutSetting functions to use the new ':key' parameter.
- Introduced IsAutoInstallEnabled method in DependencyInstallerService to check auto-installation status.
- Enhanced the task runner to check if auto installation is enabled before proceeding with dependency installation.
- Improved initialization of settings data in the system service, ensuring proper insertion of initial settings.
- Introduced a new DependencyInstallerService interface to define methods for managing dependency installation commands.
- Implemented registry service for managing the DependencyInstallerService instance.
- Enhanced the task runner to install dependencies if available, including command execution and logging for stdout and stderr.
- Improved error handling and logging throughout the task runner's dependency installation process.
- Updated the runner's methods to utilize the new dependency management features, ensuring better integration and functionality.
- Enhanced the IPC message handling in runner_test.go by adding detailed logging for better traceability.
- Refactored the test setup to use channels for synchronization and improved error handling during message processing.
- Updated the runner.go file to rename variables for clarity and streamline the IPC reader implementation.
- Improved the cleanup process in tests to ensure proper resource management and context cancellation.
- Improved error messages in the FileLogDriver's cleanup method to include error details for better debugging.
- Updated the default task log path from '/app/logs/tasks' to '/var/log/crawlab/tasks' to ensure consistency across environments.
- Changed the default task log path from '/var/log/crawlab/tasks' to '/app/logs/tasks' to align with the application's directory structure and improve portability in different deployment environments.
- Added support for new dependency file types: 'go.mod' and 'pom.xml' in dependency.go.
- Refactored command configuration in runner.go to improve logging and error handling.
- Introduced a new method to configure Node.js paths, enhancing environment setup for tasks.
- Enhanced IPC message handling with detailed logging for better traceability.
- Updated service logging to remove unnecessary prefixes for cleaner output.
- Improved command execution handling in process.go for better compatibility across platforms.
- Added 'zip' and 'unzip' utilities to the dependency installation script for improved file handling capabilities.
- Updated usage instructions in the Python installation script to include a new 'setup' command for configuring pyenv, enhancing user experience and script functionality.
- Updated log messages in NodeServiceServer and TaskServiceServer to remove the "[NodeServiceServer]" and "[TaskServiceServer]" prefixes for cleaner output.
- This change enhances log readability and maintains consistency across logging practices in the application.
- Updated the dependencies in the Docker base image installation scripts by removing unnecessary packages and ensuring essential tools are included.
- Simplified the Python installation script by refining the list of required libraries, enhancing clarity and reducing potential installation issues.
- Modified the requirements.txt to include only necessary packages, improving the efficiency of the Python environment setup.
- Modified the Dockerfile to require Python version 3.12 explicitly during the installation process.
- This change ensures that the installation script adheres to the specified version, improving consistency and reliability in Docker environments.
- Modified the Python installation script to remove the default version assignment and enforce version specification during installation.
- Updated usage instructions to reflect the change from a default version to requiring a user-defined version, improving clarity and preventing installation errors.
- This change enhances the robustness of the installation process by ensuring users explicitly define the Python version they wish to install.
- Eliminated the default_version field from the DependencyConfig struct in dependency_config.go to streamline the configuration model.
- This change simplifies the dependency management process and aligns with recent updates in the application structure.
