109 Commits

Author SHA1 Message Date
Marvin Zhang
29ef8d67da feat: implement synchronization and error handling improvements in task reconciliation and file synchronization 2025-09-28 17:42:23 +08:00
Marvin Zhang
060396af3d refactor: enhance data structure annotations and improve layout responsiveness in Home component 2025-08-07 09:58:11 +08:00
Marvin Zhang
44dd68918f refactor: improve goroutine management and context handling in task and stream operations; ensure graceful shutdown and prevent leaks 2025-08-07 00:16:46 +08:00
Marvin Zhang
9745129e33 feat: configure test database as master node for testing 2025-07-23 15:28:51 +08:00
Marvin Zhang
a2d13fae36 feat: temporarily disable batch file saving route and implement alternative handler in spider controller 2025-07-23 14:55:04 +08:00
Marvin Zhang
9f251f3ebe fix: enhance task cancellation logic with graceful termination and stuck task cleanup 2025-06-27 13:50:21 +08:00
Marvin Zhang
13038d4a1a fix: unable to sync files from master in worker nodes 2025-06-20 14:42:52 +08:00
Marvin Zhang
09cfe37272 fix: unable to sync files to worker nodes when running tasks 2025-06-18 21:50:53 +08:00
Marvin Zhang
22497d22f7 fix: unable to filter tasks issue 2025-06-17 09:28:25 +08:00
Marvin Zhang
0bd2b62792 feat: add DTOs for models and update project spiders type to array 2025-06-12 19:58:21 +08:00
Marvin Zhang
79b7e074e1 refactor: update filter parameter in API requests and improve component structure 2025-06-12 00:17:40 +08:00
Marvin Zhang
de69c2b01b refactor: replace activeNodes with allList in dependency dialogs and related components 2025-06-11 13:34:08 +08:00
Marvin Zhang
1f3a9586b3 feat: enhance data model associations and remove unused components 2025-06-09 23:00:47 +08:00
Marvin Zhang
975c601f38 refactor: simplify list retrieval methods and enhance error handling 2025-06-09 10:21:00 +08:00
Marvin Zhang
062077a9bd feat: restructure user parameters for PostUser and simplify email validation; refactor UserAvatar styles and improve user name handling 2025-05-29 22:16:27 +08:00
Marvin Zhang
512f246094 feat: implement AutoProbe components and enhance LLM provider functionality
- Introduced AutoProbeForm and AutoProbeTaskStatus components for managing AutoProbe configurations and task statuses.
- Updated LLMProvider model to include a default model field for improved provider management.
- Enhanced AutoProbeList and AutoProbeDetail components for better user interaction and data display.
- Refactored related Vue components and services to streamline AutoProbe functionality and improve state management.
- Added internationalization support for new AutoProbe features and updated existing translations.
2025-05-12 17:37:30 +08:00
Marvin Zhang
9e25ea9b51 refactor: standardize LLM provider key naming and enhance related components
- Renamed 'Key' fields to 'Type' in various models for consistency across the codebase.
- Updated route naming in router.go for clarity in the context of settings.
- Introduced a new LlmProviderForm component for managing LLM provider configurations.
- Refactored AssistantConsole and related components to utilize providerId instead of provider for improved state management.
- Enhanced UI elements and translations to reflect the new naming conventions and improve user experience.
2025-05-09 15:35:57 +08:00
Marvin Zhang
fdd0595f49 refactor: update route naming for clarity in router.go
- Renamed route from "Get Spider Files" to "List Spider Files" for improved clarity.
- Enhanced description to specify that it lists files given a directory path in the spider project.
2025-04-23 15:50:10 +08:00
Marvin Zhang
f28720ab67 refactor: optimize type assertions and simplify sort option checks in utils.go
- Streamlined type assertions in convertFilterValue function for improved clarity and performance.
- Removed unnecessary nil checks for sorts in GetSortOptionFromString and GetSortsOption functions, enhancing code readability.
2025-04-23 13:58:16 +08:00
Marvin Zhang
c4f2d1e554 fix: restore missing error handling import in utils.go
- Reintroduced the 'github.com/juju/errors' import in utils.go to ensure proper error handling functionality is available in the codebase.
2025-04-23 13:55:04 +08:00
Marvin Zhang
4f57d277e7 refactor: standardize timestamp fields and improve code clarity
- Updated timestamp fields across the codebase from `*_ts` to `*_at` for consistency and clarity.
- Renamed constants for node status from "on"/"off" to "online"/"offline" to better reflect their meanings.
- Enhanced validation and error handling in various components to ensure data integrity.
- Refactored test cases to align with the new naming conventions and improve readability.
2025-04-21 18:13:22 +08:00
Marvin Zhang
34509b8d4c refactor: enhance parameter handling and improve code clarity
- Updated GetListParams to set a default sort option for better query handling.
- Enhanced PostSpiderRunParams to include a default mode and improved error handling for missing spider.
- Added parameters field to ChatMessageContent for more flexible message content management.
- Refactored getNodeIds method to simplify mode handling and removed unnecessary error checks.
- Improved ChatMessageAction component to display parameters and response sections more effectively, enhancing user experience.
2025-04-17 18:03:15 +08:00
Marvin Zhang
336ca5770b fix: improve error handling and enhance setting management
- Updated error handling in GetSetting to use errors.Is for better clarity when checking for no documents.
- Refactored setting retrieval in the Vuex store to ensure a default value is set if the retrieved setting is empty.
- Adjusted API calls in the store to wrap payloads in a data object for consistency.
- Cleaned up SystemDetail.vue by removing unnecessary navigation actions for a more streamlined UI.
- Enhanced conditional rendering in SystemDetailTabDependency to ensure form value checks are safe.
- Replaced el-input with cl-edit-input in SystemDetailTabCustomize for improved input handling and added change event for saving.
2025-04-17 14:50:59 +08:00
Marvin Zhang
f167ff7600 refactor: rename GetSpiderListDir to GetSpiderFiles for clarity
- Updated route name and handler function to better reflect the purpose of retrieving files in a spider directory.
- Renamed parameter struct from GetSpiderListDirParams to GetSpiderFilesParams to maintain consistency in naming conventions.
2025-04-14 23:25:26 +08:00
Marvin Zhang
631f6f96b2 refactor: simplify sorting logic in GetSortsFromString function
- Adjusted the handling of sorting direction by directly checking for descending and ascending prefixes.
- Removed redundant code for handling ascending sorting, streamlining the function's logic.
- Enhanced code clarity and maintainability by simplifying the sorting parameter processing.
2025-04-10 23:25:29 +08:00
Marvin Zhang
0a3c706dca feat: enhance task and spider result handling in API
- Updated the description for the GetSpiderResults route to clarify the type of data returned.
- Added a new route for GetTaskResults to retrieve scraped or crawled results data for tasks.
- Refactored the GetSpiderResults and GetTaskResults functions to improve query handling and error management.
- Ensured consistent usage of MongoDB query methods across the controllers for better maintainability.
2025-04-10 23:18:17 +08:00
Marvin Zhang
95bf247ab1 refactor: enhance response structures and add JSON serialization
- Removed redundant comment for GetListParams struct to improve clarity.
- Updated Response, ListResponse, and VoidResponse types to use omitempty for the Error field, streamlining JSON output.
- Introduced ToJSON method for Response and ListResponse types to facilitate easy serialization of response data.
- Ensured alignment with existing coding standards and improved code maintainability through consistent struct definitions.
2025-04-10 16:04:48 +08:00
Marvin Zhang
c4291e7a93 refactor: update response handling and utility functions
- Changed the summary conversion method in getIDForAction from PascalCase to SnakeCase for consistency with naming conventions.
- Introduced BaseResponse interface and implemented GetData and GetDataString methods for Response, ListResponse, and VoidResponse types, enhancing response handling.
- Simplified the Contains function to specifically handle string slices, improving clarity and maintainability of utility functions.
- Ensured alignment with existing coding standards and added necessary comments for better understanding.
2025-04-07 18:13:49 +08:00
Marvin Zhang
ed29a76566 refactor: restructure stats handling and remove legacy service interface
- Replaced the StatsService interface with direct MongoDB queries in the stats controller, enhancing performance and reducing complexity.
- Introduced structured response types for GetStatsOverview and GetStatsTasks, improving clarity and maintainability of the API responses.
- Removed outdated service implementation files, streamlining the codebase and eliminating redundancy.
- Enhanced error handling for MongoDB operations to ensure robustness in data retrieval processes.
2025-04-03 22:35:22 +08:00
Marvin Zhang
65d344a21e refactor: enhance sorting logic in GetSortsFromString function
- Replaced JSON unmarshalling with string parsing to handle sorting parameters more flexibly.
- Improved handling of sorting direction by allowing both ascending and descending specifications through prefixes.
- Enhanced code readability and maintainability by simplifying the sorting logic and removing unnecessary error handling.
2025-03-27 11:55:22 +08:00
Marvin Zhang
f86173c973 refactor: standardize filter handling across controllers
- Replaced "conditions" parameter with "filter" in various controller methods to unify the filtering approach.
- Updated related functions to utilize the new filter parameter, enhancing consistency in query handling.
- Improved overall code readability and maintainability by aligning naming conventions and refactoring filter-related logic.
2025-03-26 22:08:01 +08:00
Marvin Zhang
863ba153f6 fix: improve error handling and standardize configuration constants
- Enhanced error handling in BaseController to return a more descriptive "item not found" message when no documents are found in MongoDB.
- Standardized naming conventions for configuration constants by updating DefaultMcpServerBaseUrl to DefaultMCPServerBaseUrl and introducing DefaultMCPClientBaseUrl for clarity and consistency.
- Ensured alignment with existing coding standards and improved code readability.
2025-03-25 17:32:40 +08:00
Marvin Zhang
e73ee2e387 fix: improve error handling in the error hook
- Updated the error handling logic in the tonic error hook to unwrap nested errors, ensuring that the most relevant error message is returned.
- Enhanced the clarity of the error response by directly using the unwrapped error message, improving debugging and user feedback.
2025-03-19 22:46:35 +08:00
Marvin Zhang
931a36c8bf refactor: enhance parameter descriptions and standardize model fields
- Added descriptive annotations to various parameters across controllers to improve API documentation clarity.
- Standardized field definitions in models by including descriptions for better understanding of their purpose.
- Updated validation patterns for ID fields to ensure consistency and improve data integrity.
- Enhanced overall code readability and maintainability by aligning naming conventions and adding necessary comments.
2025-03-19 13:39:02 +08:00
Marvin Zhang
13ce1d551b refactor: enhance OpenAPI integration and security handling
- Updated GetOpenAPI function to improve clarity by storing the Fizz instance in a variable.
- Added security middleware to FizzWrapper for API token authentication, enhancing security measures.
- Configured sorting of tags in the OpenAPI generator to maintain consistency in API documentation.
- Improved comments for better understanding of the changes made in the OpenAPI wrapper.
2025-03-18 18:01:22 +08:00
Marvin Zhang
98843fdbac refactor: standardize sorting logic across controllers
- Introduced a consistent approach for sorting in multiple controller methods by utilizing GetSortOptionFromString for improved clarity and maintainability.
- Updated the GetListParams structure to ensure sorting is handled uniformly across different entities.
- Enhanced error handling for sorting parameters to provide clearer feedback on invalid formats.
- Improved comments for better understanding of sorting logic and its integration within the controller methods.
2025-03-17 17:14:08 +08:00
Marvin Zhang
95a6eb4e7b chore: add OpenTelemetry auto SDK dependency
- Introduced go.opentelemetry.io/auto/sdk v1.1.0 to go.mod and updated go.sum accordingly.
- Enhanced observability capabilities by including the OpenTelemetry SDK for automatic instrumentation.
- Ensured consistency in dependency management across the project.
2025-03-17 14:24:07 +08:00
Marvin Zhang
657c73ad2f refactor: improve filter query handling in utility functions
- Added nil return for empty condition strings in GetFilterQueryFromConditionString and GetFilterFromConditionString functions to enhance error handling.
- Improved clarity and maintainability by ensuring consistent behavior when handling empty input conditions.
- Updated comments for better understanding of the changes made.
2025-03-17 14:20:19 +08:00
Marvin Zhang
4dace3ce8e refactor: migrate router and controller methods to use Fizz package
- Updated router groups to utilize the new crawlab-team/fizz package for improved routing capabilities.
- Refactored controller methods to accept Fizz router groups, enhancing consistency and maintainability.
- Simplified route registration by incorporating OpenAPI metadata directly into group definitions.
- Improved error handling and response generation in sync controller methods for better clarity.
- Enhanced overall code structure by standardizing route definitions and improving parameter handling.
2025-03-17 12:50:50 +08:00
Marvin Zhang
bc5e6273b8 chore: update dependencies and refactor imports
- Updated various dependencies in go.mod and go.sum for improved stability and performance.
- Replaced references to the old fizz package with the new crawlab-team/fizz package in multiple files.
- Adjusted indirect dependencies to their latest versions for better compatibility.
- Enhanced overall code clarity by standardizing import paths across the project.
2025-03-17 12:08:35 +08:00
Marvin Zhang
43d1c7692b refactor: standardize response types across controllers
- Updated multiple controller methods to return VoidResponse instead of generic Response[any].
- Consolidated error handling to utilize GetErrorVoidResponse for consistent error responses.
- Enhanced parameter handling in export and file management functions for improved clarity and maintainability.
- Refactored health check and login/logout methods to align with new response structure.
- Improved overall consistency in response formatting across various endpoints.
2025-03-16 22:25:13 +08:00
Marvin Zhang
700c263cfe refactor: update filter constants and improve type handling in filter functions
- Renamed filter operation constants for clarity and consistency.
- Introduced short forms for certain filter operations.
- Updated test cases to reflect changes in filter operation handling.
- Enhanced type handling in filter functions to support int64 values for conditions.
- Improved query generation logic to accommodate new constant names and types.
2025-03-16 20:30:52 +08:00
Marvin Zhang
ce0143ca06 refactor: enhance health check function and add comprehensive test coverage
- Updated GetHealthFn to return an error for better error handling and clarity.
- Introduced a new test file for schedule management, covering various endpoints including creation, retrieval, updating, and deletion of schedules.
- Added tests for task management, including task creation, retrieval, updating, and cancellation.
- Implemented utility tests for filtering and response generation to ensure consistent API behavior.
- Improved logging in the task scheduler service for better traceability.
2025-03-13 18:10:24 +08:00
Marvin Zhang
43691d0d8a refactor: enhance controller parameter handling and error responses
- Refactored export and filter controller methods to utilize structured parameter types for improved clarity and maintainability.
- Consolidated error handling and response generation, ensuring consistent API responses across methods.
- Updated function signatures to eliminate unnecessary context parameters, enhancing type safety and readability.
- Improved handling of query parameters and conditions, streamlining the logic for better performance and maintainability.
2025-03-13 17:43:55 +08:00
Marvin Zhang
23cad00d92 refactor: streamline controller methods and enhance parameter handling
- Refactored multiple controller methods to accept structured parameters for improved clarity and maintainability.
- Consolidated error handling and response generation across various endpoints.
- Updated function signatures to eliminate unnecessary context parameters and enhance type safety.
- Improved consistency in response formatting and error handling across controllers.
- Enhanced file handling methods to support multipart file uploads and directory operations more effectively.
2025-03-13 17:37:30 +08:00
Marvin Zhang
ddff881954 chore: update Go version and dependencies
- Updated Go version in go.work and backend/go.mod to 1.23.7
- Updated various dependencies in go.sum and backend/go.sum
- Refactored models to remove generic type parameters from BaseModel
- Introduced new utility functions for consistent API responses
- Removed unused utility files from controllers
2025-03-12 23:20:06 +08:00
Marvin Zhang
d6badb533d feat: enhance API routing and OpenAPI documentation support
- Introduced a new OpenAPI wrapper using Fizz for improved API documentation
- Refactored base controller to support more flexible route handling
- Added dynamic route registration with OpenAPI metadata
- Implemented generic response types for consistent API responses
- Updated router initialization to support OpenAPI documentation endpoint
- Improved route and resource naming utilities
- Migrated existing controllers to use new routing and response mechanisms
2025-03-11 23:45:06 +08:00
Marvin Zhang
a1f870715f feat: add GET endpoint for retrieving user's token list
Implemented a new GetTokenList handler in the token controller to:
- Fetch tokens created by the current user
- Support pagination and filtering
- Return tokens with total count
- Handle cases with no documents gracefully
2025-02-27 16:39:51 +08:00
Marvin Zhang
6048d4eeb8 fix: missing routes in get me api
https://github.com/crawlab-team/crawlab/issues/1550
2025-02-27 13:33:48 +08:00
Marvin Zhang
3237923f02 fix: prevent unnecessary error handling in export download endpoint
Adds an early return after handling bad request errors in the GetExportDownload function to avoid unnecessary error processing and improve error handling clarity.
2025-02-17 16:27:39 +08:00