Commit Graph

49 Commits

Author SHA1 Message Date
Marvin Zhang
a585ab16f7 feat: enhance task runner with task status updates and process command execution
- Added a task status update to 'processing' at the start of the Run method in runner.go, improving task tracking.
- Removed redundant task status update from the end of the Run method to streamline the execution flow.
- Updated command execution in process.go to use 'bash' instead of 'sh' for better compatibility across environments.
2025-01-03 16:44:38 +08:00
Marvin Zhang
db2549e3cd fix: enhance error logging in file log driver and update default task log path
- Improved error messages in the FileLogDriver's cleanup method to include error details for better debugging.
- Updated the default task log path from '/app/logs/tasks' to '/var/log/crawlab/tasks' to ensure consistency across environments.
2025-01-01 14:26:10 +08:00
Marvin Zhang
7e7ac621ec fix: update default task log path for consistency across environments
- Changed the default task log path from '/var/log/crawlab/tasks' to '/app/logs/tasks' to align with the application's directory structure and improve portability in different deployment environments.
2025-01-01 11:54:48 +08:00
Marvin Zhang
7b6805a834 feat: enhance task runner with improved logging and dependency support
- Added support for new dependency file types: 'go.mod' and 'pom.xml' in dependency.go.
- Refactored command configuration in runner.go to improve logging and error handling.
- Introduced a new method to configure Node.js paths, enhancing environment setup for tasks.
- Enhanced IPC message handling with detailed logging for better traceability.
- Updated service logging to remove unnecessary prefixes for cleaner output.
- Improved command execution handling in process.go for better compatibility across platforms.
2024-12-31 22:52:21 +08:00
Marvin Zhang
54800974eb feat: refactor system info retrieval and enhance logo output
- Replaced viper calls with utility functions in GetSystemInfo to improve code clarity and maintainability.
- Added a new system.go file with utility functions for retrieving system version and edition information.
- Enhanced PrintLogoWithWelcomeInfo to include detailed system information, improving user experience during server startup.
- Updated output formatting for better readability and consistency in welcome messages.
2024-12-29 19:01:23 +08:00
Marvin Zhang
17f8917d0a chore: update dependencies and enhance gRPC services
- Updated various dependencies in go.mod and go.sum files, including cloud.google.com/go/compute/metadata to v0.5.2, google.golang.org/grpc to v1.69.2, and google.golang.org/protobuf to v1.36.1.
- Refactored gRPC service definitions to use the latest protoc-gen-go and protoc versions, ensuring compatibility with the latest gRPC-Go features.
- Introduced a new logger utility in core/utils/logger.go to improve logging capabilities across the application.
- Added a README.md for gRPC setup and compilation guidance, enhancing developer experience.
- Improved the Python installation script to handle version listing more effectively and ensure user-specific environment setup.
2024-12-25 17:46:49 +08:00
Marvin Zhang
9e67e50c6c fix: add new line after logo and welcome info prints
- Added a new line after the logo and welcome information prints in the PrintLogoWithWelcomeInfo function to improve output readability.
- This change enhances the user experience by ensuring that the printed information is visually separated, making it easier to read during server startup.
2024-12-25 14:22:18 +08:00
Marvin Zhang
a13893b627 feat: add logo printing functionality and update logger usage
- Introduced a new utility function to print a logo and welcome information for the Crawlab server, enhancing user experience during startup.
- Updated logger variable names in the apps package for consistency and clarity.
- Added a new dependency on github.com/common-nighthawk/go-figure to facilitate logo rendering.
- Improved the server command to display the logo when the server starts, provided the user is not using the pro version.
2024-12-25 13:08:56 +08:00
Marvin Zhang
dc59599509 refactor: remove db module and update imports to core/mongo
- Deleted the db module, consolidating database-related functionality into the core/mongo package for better organization and maintainability.
- Updated all import paths across the codebase to replace references to the removed db module with core/mongo.
- Cleaned up unused code and dependencies, enhancing overall project clarity and reducing complexity.
- This refactor improves the structure of the codebase by centralizing database operations and simplifying module management.
2024-12-25 10:28:21 +08:00
Marvin Zhang
a28ffbf66c refactor: simplify interfaces and improve configuration handling
- Removed unused ApiApp and ServerApp interfaces from core/apps/interfaces.go to streamline the codebase.
- Updated the GetApi method in the Server struct to return a pointer to the Api type for better type handling.
- Simplified the GetGinMode function in core/utils/config.go to always return gin.ReleaseMode, removing unnecessary conditional checks for development mode.
- These changes enhance code clarity and maintainability by eliminating redundant code and improving type safety.
2024-12-24 23:05:41 +08:00
Marvin Zhang
3276083994 refactor: replace apex/log with structured logger across multiple services
- Replaced all instances of apex/log with a structured logger interface in various services, including Api, Server, Config, and others, to enhance logging consistency and context.
- Updated logging calls to utilize the new logger methods, improving error tracking and service monitoring.
- Added logger initialization in services and controllers to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Removed unused apex/log imports and cleaned up related code for better maintainability.
2024-12-24 19:11:19 +08:00
Marvin Zhang
99ed4396d1 refactor: improve logging messages and update configuration constants
- Updated logging messages in GrpcClient to provide clearer context, changing "ready" to "client is now ready" and "stopped" to "client has stopped".
- Refactored test setup in runner_test.go to remove unnecessary error checks during gRPC client start for cleaner code.
- Renamed GetDependencySetupScriptRoot to GetInstallRoot and updated related constants for better clarity and consistency in configuration management.
2024-12-23 18:19:08 +08:00
Marvin Zhang
ed8fb78c3b feat: expand Logger interface and implement additional logging methods in ServiceLogger
- Added Debug, Info, Warn, Error, and Fatal methods to the Logger interface for comprehensive logging capabilities.
- Implemented corresponding methods in ServiceLogger to facilitate structured logging with service context.
- Enhanced the logging functionality to support various log levels, improving error tracking and debugging.
2024-12-23 14:25:48 +08:00
Marvin Zhang
bdc347aef7 feat: add Fatalf logging method to Logger interface and ServiceLogger
- Introduced Fatalf method in Logger interface for logging fatal messages with formatted content.
- Implemented Fatalf in ServiceLogger to log fatal messages and exit the program, enhancing error handling capabilities.
2024-12-23 13:15:18 +08:00
Marvin Zhang
afe21b7ca0 feat: implement logging interfaces and service logger
- Added Logger interface in core/interfaces/logger.go for standardized logging methods (Debugf, Infof, Warnf, Errorf).
- Introduced ServiceLogger in core/utils/log.go, which prefixes log messages with the service name for better context.
- Implemented logging methods in ServiceLogger to utilize the apex/log package for structured logging.
2024-12-21 23:00:30 +08:00
Marvin Zhang
b88b41afe2 feat: add API configuration options for port and path
- Introduced GetApiPort and GetApiPath functions to retrieve API port and path from configuration, with defaults provided.
- Updated GetApiEndpoint to construct the API endpoint URL using the new port and path functions, ensuring proper formatting.
- Added constants for default API port and path to enhance configurability.
2024-12-21 14:05:50 +08:00
Marvin Zhang
c897fb58e4 refactor: streamline error handling and improve HTTP request management
- Removed the print flag from handleError function, simplifying error logging based on the development environment.
- Introduced a new performRequest function for standardized HTTP requests with JSON bodies, enhancing code reusability.
- Updated SendIMNotification and related functions to utilize the new RequestParam type for better clarity and consistency.
- Normalized HTTP request paths in the createHttpRequest method to ensure correct URL formatting.
- Added detailed error logging for JSON unmarshaling failures in syncFiles method.
- Introduced a NewHttpClient function to create HTTP clients with customizable timeouts.
2024-12-21 11:27:58 +08:00
Marvin Zhang
c5c08dfba6 feat: added dependency config setup (wip) 2024-12-15 23:09:10 +08:00
Marvin Zhang
272371d9ce feat: allow set max runners for nodes 2024-12-11 22:05:34 +08:00
Marvin Zhang
858e5c2b89 fix: unable to start api 2024-11-22 21:19:17 +08:00
Marvin Zhang
9560da66b5 test: fix test cases issues 2024-11-22 17:43:59 +08:00
Marvin Zhang
8357dc6f30 fix: runner cancel issue 2024-11-22 13:32:27 +08:00
Marvin Zhang
aa1b01b0a5 test: fix test cases 2024-11-22 12:45:04 +08:00
Marvin Zhang
d00dd9495c test: fix test cases 2024-11-22 10:43:45 +08:00
Marvin Zhang
46100ebaf5 ci: updated base-image 2024-11-20 14:55:42 +08:00
Marvin Zhang
1a93b148e1 fix: unable to connect to grpc server issue 2024-11-19 18:15:17 +08:00
Marvin Zhang
98a9c907c9 fix: test case issue 2024-11-19 17:40:44 +08:00
Marvin Zhang
3dc66e48db fix: test case issue 2024-11-19 15:53:40 +08:00
Marvin Zhang
d0caf9f5ad refactor: consolidated configs 2024-11-18 17:25:13 +08:00
Marvin Zhang
a3b286558b refactor: consolidated configs 2024-11-18 16:48:09 +08:00
Marvin Zhang
459d94ad96 feat: optimized dependencies for spider 2024-11-15 18:40:06 +08:00
Marvin Zhang
e1170d5612 test: updated test cases 2024-10-20 17:55:57 +08:00
Marvin Zhang
3c53fde7e2 refactor: code cleanup 2024-10-20 17:22:35 +08:00
Marvin Zhang
4ac92d7eaf refactor: code cleanup 2024-10-20 16:55:57 +08:00
Marvin Zhang
1b852fb96a refactor: code cleanup 2024-10-18 15:03:32 +08:00
Marvin Zhang
08d1acb15e feat: added hash to models 2024-08-26 18:33:46 +08:00
Marvin Zhang
421a06930c refactor: Update models to use DatabaseV2 instead of DataSourceV2 2024-08-05 16:07:19 +08:00
Marvin Zhang
e57655c67e refactor: Update models to use DatabaseV2 instead of DataSourceV2 2024-08-05 13:14:57 +08:00
Marvin Zhang
e2cb99e56a feat: Update models to use DatabaseV2 instead of DataSourceV2
The code changes update the models and related functions to use the new DatabaseV2 struct instead of the deprecated DataSourceV2 struct. This change ensures consistency and clarity in the codebase.
2024-08-04 16:58:46 +08:00
Marvin Zhang
aca0c0ebce refactor: removed unnecessary code 2024-07-11 12:45:29 +08:00
Marvin Zhang
13596436cd refactor: update viper config key for Pro edition check 2024-07-10 18:59:35 +08:00
Marvin Zhang
f4025a982d feat: updated node monitoring tab 2024-07-04 13:01:54 +08:00
Marvin Zhang
023ba27566 fix: unable to sync directories to work nodes 2024-07-01 15:59:20 +08:00
Marvin Zhang
326a8d67d0 fix: missing data source issue 2024-06-26 12:37:24 +08:00
Marvin Zhang
5daeccb87d fix: unable to sync files and save data issues 2024-06-25 14:58:54 +08:00
Marvin Zhang
b757826867 feat: added fields to models 2024-06-19 00:17:36 +08:00
Marvin Zhang
6a60433d25 feat: added modules 2024-06-14 16:37:48 +08:00
Marvin Zhang
dc21bce11f feat: added modules 2024-06-14 15:59:48 +08:00
Marvin Zhang
0b67fd9ece feat: added modules 2024-06-14 15:42:50 +08:00