Adds an early return after handling bad request errors in the GetExportDownload function to avoid unnecessary error processing and improve error handling clarity.
- Deleted CHANGELOG-zh.md and CHANGELOG.md
- Added version-specific changelog files in the changelog/ directory for v0.2, v0.3, v0.4, v0.5, and v0.6
- Included both Chinese and English versions for some changelog files
- Organized changelogs by version with detailed feature and bug fix descriptions
Reverts the previous change that set DefaultInstallEnvs to "node,browser", returning it to an empty default value to maintain configuration flexibility.
- Introduced a new function GetInstallEnvs in config.go to retrieve installation environment variables
- Added DefaultInstallEnvs constant with an empty default value
- Implemented fallback mechanism to split DefaultInstallEnvs if no configuration is found
- Enhances configuration flexibility for installation environments
- Added 'apt-get update -y' before package installation to ensure latest package information
- Ensures dependencies are installed from the most recent package lists
- Updated the Spider model to introduce a new SpiderTemplateParams struct for improved template handling.
- Refactored string utility functions in utils/string.go to include a new replaceChars function, streamlining character replacement across multiple functions.
- Enhanced ToSnakeCase and ToKebabCase functions to utilize the new replaceChars function for better maintainability and readability.
- Added splitStringWithQuotes function to facilitate string manipulation with quotes, improving overall utility in string processing.
- Introduced a new function ToKebabCase in utils/string.go to convert strings to kebab-case format.
- The function trims whitespace, converts to lowercase, and replaces spaces and underscores with hyphens, enhancing string manipulation capabilities in the codebase.
- Replaced direct calls to getBaseFileFsSvc with a new method fs.GetBaseFileFsSvc in base_file.go for improved clarity and maintainability.
- Introduced SpiderTemplateService interface and implemented registry service for managing spider templates, enhancing template handling in the spider controller.
- Added template-related fields to the Spider model to support template functionality.
- Created utility functions for string case conversions in utils/string.go to facilitate consistent formatting across the codebase.
- Updated environment configuration to retrieve the Python path dynamically, improving flexibility in the task runner's setup.
- Added configuration initialization in db.go to ensure proper setup of application settings.
- Refactored runner_test.go to streamline IPC message handling by introducing a setupPipe function and an initRunner function for better readability and maintainability.
- Improved synchronization in tests by using channels for signaling readiness and processing, enhancing the reliability of IPC message handling.
- Updated test cases to validate IPC message processing and error handling, ensuring robustness in the task runner's functionality.
- Introduced a new method configureGoPath in runner.go to set the GOPATH environment variable based on the retrieved Go path.
- Updated configureEnv to call configureGoPath, ensuring the Go path is configured alongside Node.js paths.
- Added a new utility function GetGoPath in config.go to retrieve the Go path from configuration, with a default fallback.
- These changes enhance the task runner's environment setup by supporting Go development alongside existing Node.js configurations.
- Changed package manager command from 'apt-get' to 'apt' for improved usability.
- Added additional Python dependencies: 'libbz2-dev', 'libreadline-dev', 'libsqlite3-dev', 'xz-utils', and 'liblzma-dev' to enhance the environment setup.
- Updated 'crawlab-sdk' version in requirements.txt from '0.7.0rc1' to '0.7.0rc5' for compatibility and feature improvements.
- Updated IPC reader initialization in runner_test.go to use a channel for signaling readiness, improving synchronization.
- Added error logging when writing to the pipe to enhance traceability during tests.
- These changes improve the reliability and clarity of the test setup for the task runner.
- Removed redundant home directory retrieval and nvm checks in the configureNodePath method.
- Introduced a new utility function GetNodeModulesPath to centralize the logic for determining the global node_modules path.
- Updated environment variable setup to use the new utility function, improving clarity and maintainability of the code.
- Added a task status update to 'processing' at the start of the Run method in runner.go, improving task tracking.
- Removed redundant task status update from the end of the Run method to streamline the execution flow.
- Updated command execution in process.go to use 'bash' instead of 'sh' for better compatibility across environments.
- Changed route parameter from ':id' to ':key' in settings-related routes for better clarity and consistency.
- Updated GetSetting, PostSetting, and PutSetting functions to use the new ':key' parameter.
- Introduced IsAutoInstallEnabled method in DependencyInstallerService to check auto-installation status.
- Enhanced the task runner to check if auto installation is enabled before proceeding with dependency installation.
- Improved initialization of settings data in the system service, ensuring proper insertion of initial settings.
- Introduced a new DependencyInstallerService interface to define methods for managing dependency installation commands.
- Implemented registry service for managing the DependencyInstallerService instance.
- Enhanced the task runner to install dependencies if available, including command execution and logging for stdout and stderr.
- Improved error handling and logging throughout the task runner's dependency installation process.
- Updated the runner's methods to utilize the new dependency management features, ensuring better integration and functionality.
- Enhanced the IPC message handling in runner_test.go by adding detailed logging for better traceability.
- Refactored the test setup to use channels for synchronization and improved error handling during message processing.
- Updated the runner.go file to rename variables for clarity and streamline the IPC reader implementation.
- Improved the cleanup process in tests to ensure proper resource management and context cancellation.
- Improved error messages in the FileLogDriver's cleanup method to include error details for better debugging.
- Updated the default task log path from '/app/logs/tasks' to '/var/log/crawlab/tasks' to ensure consistency across environments.
- Changed the default task log path from '/var/log/crawlab/tasks' to '/app/logs/tasks' to align with the application's directory structure and improve portability in different deployment environments.
- Added support for new dependency file types: 'go.mod' and 'pom.xml' in dependency.go.
- Refactored command configuration in runner.go to improve logging and error handling.
- Introduced a new method to configure Node.js paths, enhancing environment setup for tasks.
- Enhanced IPC message handling with detailed logging for better traceability.
- Updated service logging to remove unnecessary prefixes for cleaner output.
- Improved command execution handling in process.go for better compatibility across platforms.
- Added 'zip' and 'unzip' utilities to the dependency installation script for improved file handling capabilities.
- Updated usage instructions in the Python installation script to include a new 'setup' command for configuring pyenv, enhancing user experience and script functionality.
- Updated log messages in NodeServiceServer and TaskServiceServer to remove the "[NodeServiceServer]" and "[TaskServiceServer]" prefixes for cleaner output.
- This change enhances log readability and maintains consistency across logging practices in the application.
- Updated the dependencies in the Docker base image installation scripts by removing unnecessary packages and ensuring essential tools are included.
- Simplified the Python installation script by refining the list of required libraries, enhancing clarity and reducing potential installation issues.
- Modified the requirements.txt to include only necessary packages, improving the efficiency of the Python environment setup.
- Modified the Dockerfile to require Python version 3.12 explicitly during the installation process.
- This change ensures that the installation script adheres to the specified version, improving consistency and reliability in Docker environments.
- Modified the Python installation script to remove the default version assignment and enforce version specification during installation.
- Updated usage instructions to reflect the change from a default version to requiring a user-defined version, improving clarity and preventing installation errors.
- This change enhances the robustness of the installation process by ensuring users explicitly define the Python version they wish to install.
- Eliminated the default_version field from the DependencyConfig struct in dependency_config.go to streamline the configuration model.
- This change simplifies the dependency management process and aligns with recent updates in the application structure.
- Introduced a new "setup" command in the Python installation script to facilitate the setup of pyenv.
- This enhancement improves the usability of the script by allowing users to easily configure their Python environment.
- Added logging for error handling in the MasterService when setting a worker node offline, replacing the previous trace.PrintError with a more informative log message.
- Enhanced WorkerService subscription method with debug logs to indicate subscription attempts and status, improving traceability during connection processes.
- Replaced viper calls with utility functions in GetSystemInfo to improve code clarity and maintainability.
- Added a new system.go file with utility functions for retrieving system version and edition information.
- Enhanced PrintLogoWithWelcomeInfo to include detailed system information, improving user experience during server startup.
- Updated output formatting for better readability and consistency in welcome messages.
- Changed the default Python version in the Docker base image installation script from 3.12 to 3.12.8 to ensure the latest patch version is used.
- This update enhances the reliability and security of the Python installation process in Docker environments.
- Updated various dependencies in go.mod and go.sum files, including cloud.google.com/go/compute/metadata to v0.5.2, google.golang.org/grpc to v1.69.2, and google.golang.org/protobuf to v1.36.1.
- Refactored gRPC service definitions to use the latest protoc-gen-go and protoc versions, ensuring compatibility with the latest gRPC-Go features.
- Introduced a new logger utility in core/utils/logger.go to improve logging capabilities across the application.
- Added a README.md for gRPC setup and compilation guidance, enhancing developer experience.
- Improved the Python installation script to handle version listing more effectively and ensure user-specific environment setup.
- Added jq to the dependencies in the Docker base image installation script.
- Updated the Python installation script to support handling of custom requirements via command-line arguments.
- Implemented a new function to manage requirements installation, falling back to a default requirements file if none is provided.
- Improved version check logic to prevent redundant installations of Python versions, enhancing efficiency in Docker environments.
- Refactored the Python installation script to support multiple commands: install, uninstall, switch, and list.
- Added functions for usage printing, dependency installation, pyenv setup, Python version verification, symlink creation, and cleanup.
- Improved error handling for version checks and command execution, enhancing usability and flexibility in Docker environments.
- Default Python version is set to 3.12, with the option to specify a different version during installation.
- Changed the installation script for pyenv to create a user-specific environment file at $HOME/.pyenv-env.sh instead of a system-wide file in /etc/profile.d/.
- Updated the script to source the new environment file, ensuring that the pyenv setup is applied correctly for the user.
- This change improves the flexibility and usability of the pyenv installation in Docker environments.
- Redirected the output of the nginx service start command to /dev/null to prevent cluttering the console with unnecessary messages during server startup.
- This change enhances the user experience by providing a cleaner output when initializing the Docker environment.
- Added a new line after the logo and welcome information prints in the PrintLogoWithWelcomeInfo function to improve output readability.
- This change enhances the user experience by ensuring that the printed information is visually separated, making it easier to read during server startup.
- Added github.com/common-nighthawk/go-figure as a new indirect dependency in both backend and core modules to support logo rendering functionality.
- Removed the github.com/imroc/req dependency from backend and go.sum files, streamlining the dependency list and improving project organization.
- Introduced a new utility function to print a logo and welcome information for the Crawlab server, enhancing user experience during startup.
- Updated logger variable names in the apps package for consistency and clarity.
- Added a new dependency on github.com/common-nighthawk/go-figure to facilitate logo rendering.
- Improved the server command to display the logo when the server starts, provided the user is not using the pro version.
- Removed the db module from the Dockerfile, reflecting the recent refactor that consolidates database functionality into the core/mongo package.
- This change simplifies the Docker image setup by eliminating unnecessary dependencies and improving overall project organization.
- Deleted the db module, consolidating database-related functionality into the core/mongo package for better organization and maintainability.
- Updated all import paths across the codebase to replace references to the removed db module with core/mongo.
- Cleaned up unused code and dependencies, enhancing overall project clarity and reducing complexity.
- This refactor improves the structure of the codebase by centralizing database operations and simplifying module management.
- Removed unused ApiApp and ServerApp interfaces from core/apps/interfaces.go to streamline the codebase.
- Updated the GetApi method in the Server struct to return a pointer to the Api type for better type handling.
- Simplified the GetGinMode function in core/utils/config.go to always return gin.ReleaseMode, removing unnecessary conditional checks for development mode.
- These changes enhance code clarity and maintainability by eliminating redundant code and improving type safety.
- Replaced all instances of apex/log with a structured logger interface in various services, including Api, Server, Config, and others, to enhance logging consistency and context.
- Updated logging calls to utilize the new logger methods, improving error tracking and service monitoring.
- Added logger initialization in services and controllers to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Removed unused apex/log imports and cleaned up related code for better maintainability.
- Updated the deps.sh script to include the 'zip' package in the list of installed dependencies for the Docker base image.
- This addition ensures that the zip utility is available for use in subsequent Docker image builds and applications.
- Removed direct usage of apex/log in favor of a structured logger interface for improved logging consistency and context.
- Updated logging calls in MasterService and WorkerService to utilize the new logger, enhancing error tracking and service monitoring.
- Added logger initialization in both services to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
