Commit Graph

112 Commits

Author SHA1 Message Date
Marvin Zhang
f441265cc2 feat(sync): add gRPC file synchronization service and integrate end-to-end
- add proto/services/sync_service.proto and generate Go pb + grpc bindings
- implement SyncServiceServer (streaming file scan + download) with:
  - request deduplication, in-memory cache (TTL), chunked streaming
  - concurrent-safe broadcast to waiters and server-side logging
- register SyncSvr in gRPC server and expose sync client in GrpcClient:
  - add syncClient field, registration and safe getters with reconnection-aware timeouts
- integrate gRPC sync into runner:
  - split syncFiles into syncFilesHTTP (legacy) and syncFilesGRPC
  - Runner now chooses implementation via config flag and performs streaming scan/download
- controller improvements:
  - add semaphore-based rate limiting for sync scan requests with in-flight counters and logs
- misc:
  - add utils.IsSyncGrpcEnabled() config helper
  - improve HTTP sync error diagnostics (Content-Type validation, response previews)
  - update/regenerate many protobuf and gRPC generated files (protoc/protoc-gen-go / protoc-gen-go-grpc version bumps)
2025-10-20 12:48:53 +08:00
Marvin Zhang
44fd0809e6 feat: disable backend unit tests and document reasons for integration test requirements 2025-10-09 12:35:09 +08:00
Marvin Zhang
5bff8823a8 feat: update test workflows to skip API tests and document controller test status 2025-10-09 11:34:26 +08:00
Marvin Zhang
29ef8d67da feat: implement synchronization and error handling improvements in task reconciliation and file synchronization 2025-09-28 17:42:23 +08:00
Marvin Zhang
060396af3d refactor: enhance data structure annotations and improve layout responsiveness in Home component 2025-08-07 09:58:11 +08:00
Marvin Zhang
44dd68918f refactor: improve goroutine management and context handling in task and stream operations; ensure graceful shutdown and prevent leaks 2025-08-07 00:16:46 +08:00
Marvin Zhang
9745129e33 feat: configure test database as master node for testing 2025-07-23 15:28:51 +08:00
Marvin Zhang
a2d13fae36 feat: temporarily disable batch file saving route and implement alternative handler in spider controller 2025-07-23 14:55:04 +08:00
Marvin Zhang
9f251f3ebe fix: enhance task cancellation logic with graceful termination and stuck task cleanup 2025-06-27 13:50:21 +08:00
Marvin Zhang
13038d4a1a fix: unable to sync files from master in worker nodes 2025-06-20 14:42:52 +08:00
Marvin Zhang
09cfe37272 fix: unable to sync files to worker nodes when running tasks 2025-06-18 21:50:53 +08:00
Marvin Zhang
22497d22f7 fix: unable to filter tasks issue 2025-06-17 09:28:25 +08:00
Marvin Zhang
0bd2b62792 feat: add DTOs for models and update project spiders type to array 2025-06-12 19:58:21 +08:00
Marvin Zhang
79b7e074e1 refactor: update filter parameter in API requests and improve component structure 2025-06-12 00:17:40 +08:00
Marvin Zhang
de69c2b01b refactor: replace activeNodes with allList in dependency dialogs and related components 2025-06-11 13:34:08 +08:00
Marvin Zhang
1f3a9586b3 feat: enhance data model associations and remove unused components 2025-06-09 23:00:47 +08:00
Marvin Zhang
975c601f38 refactor: simplify list retrieval methods and enhance error handling 2025-06-09 10:21:00 +08:00
Marvin Zhang
062077a9bd feat: restructure user parameters for PostUser and simplify email validation; refactor UserAvatar styles and improve user name handling 2025-05-29 22:16:27 +08:00
Marvin Zhang
512f246094 feat: implement AutoProbe components and enhance LLM provider functionality
- Introduced AutoProbeForm and AutoProbeTaskStatus components for managing AutoProbe configurations and task statuses.
- Updated LLMProvider model to include a default model field for improved provider management.
- Enhanced AutoProbeList and AutoProbeDetail components for better user interaction and data display.
- Refactored related Vue components and services to streamline AutoProbe functionality and improve state management.
- Added internationalization support for new AutoProbe features and updated existing translations.
2025-05-12 17:37:30 +08:00
Marvin Zhang
9e25ea9b51 refactor: standardize LLM provider key naming and enhance related components
- Renamed 'Key' fields to 'Type' in various models for consistency across the codebase.
- Updated route naming in router.go for clarity in the context of settings.
- Introduced a new LlmProviderForm component for managing LLM provider configurations.
- Refactored AssistantConsole and related components to utilize providerId instead of provider for improved state management.
- Enhanced UI elements and translations to reflect the new naming conventions and improve user experience.
2025-05-09 15:35:57 +08:00
Marvin Zhang
fdd0595f49 refactor: update route naming for clarity in router.go
- Renamed route from "Get Spider Files" to "List Spider Files" for improved clarity.
- Enhanced description to specify that it lists files given a directory path in the spider project.
2025-04-23 15:50:10 +08:00
Marvin Zhang
f28720ab67 refactor: optimize type assertions and simplify sort option checks in utils.go
- Streamlined type assertions in convertFilterValue function for improved clarity and performance.
- Removed unnecessary nil checks for sorts in GetSortOptionFromString and GetSortsOption functions, enhancing code readability.
2025-04-23 13:58:16 +08:00
Marvin Zhang
c4f2d1e554 fix: restore missing error handling import in utils.go
- Reintroduced the 'github.com/juju/errors' import in utils.go to ensure proper error handling functionality is available in the codebase.
2025-04-23 13:55:04 +08:00
Marvin Zhang
4f57d277e7 refactor: standardize timestamp fields and improve code clarity
- Updated timestamp fields across the codebase from `*_ts` to `*_at` for consistency and clarity.
- Renamed constants for node status from "on"/"off" to "online"/"offline" to better reflect their meanings.
- Enhanced validation and error handling in various components to ensure data integrity.
- Refactored test cases to align with the new naming conventions and improve readability.
2025-04-21 18:13:22 +08:00
Marvin Zhang
34509b8d4c refactor: enhance parameter handling and improve code clarity
- Updated GetListParams to set a default sort option for better query handling.
- Enhanced PostSpiderRunParams to include a default mode and improved error handling for missing spider.
- Added parameters field to ChatMessageContent for more flexible message content management.
- Refactored getNodeIds method to simplify mode handling and removed unnecessary error checks.
- Improved ChatMessageAction component to display parameters and response sections more effectively, enhancing user experience.
2025-04-17 18:03:15 +08:00
Marvin Zhang
336ca5770b fix: improve error handling and enhance setting management
- Updated error handling in GetSetting to use errors.Is for better clarity when checking for no documents.
- Refactored setting retrieval in the Vuex store to ensure a default value is set if the retrieved setting is empty.
- Adjusted API calls in the store to wrap payloads in a data object for consistency.
- Cleaned up SystemDetail.vue by removing unnecessary navigation actions for a more streamlined UI.
- Enhanced conditional rendering in SystemDetailTabDependency to ensure form value checks are safe.
- Replaced el-input with cl-edit-input in SystemDetailTabCustomize for improved input handling and added change event for saving.
2025-04-17 14:50:59 +08:00
Marvin Zhang
f167ff7600 refactor: rename GetSpiderListDir to GetSpiderFiles for clarity
- Updated route name and handler function to better reflect the purpose of retrieving files in a spider directory.
- Renamed parameter struct from GetSpiderListDirParams to GetSpiderFilesParams to maintain consistency in naming conventions.
2025-04-14 23:25:26 +08:00
Marvin Zhang
631f6f96b2 refactor: simplify sorting logic in GetSortsFromString function
- Adjusted the handling of sorting direction by directly checking for descending and ascending prefixes.
- Removed redundant code for handling ascending sorting, streamlining the function's logic.
- Enhanced code clarity and maintainability by simplifying the sorting parameter processing.
2025-04-10 23:25:29 +08:00
Marvin Zhang
0a3c706dca feat: enhance task and spider result handling in API
- Updated the description for the GetSpiderResults route to clarify the type of data returned.
- Added a new route for GetTaskResults to retrieve scraped or crawled results data for tasks.
- Refactored the GetSpiderResults and GetTaskResults functions to improve query handling and error management.
- Ensured consistent usage of MongoDB query methods across the controllers for better maintainability.
2025-04-10 23:18:17 +08:00
Marvin Zhang
95bf247ab1 refactor: enhance response structures and add JSON serialization
- Removed redundant comment for GetListParams struct to improve clarity.
- Updated Response, ListResponse, and VoidResponse types to use omitempty for the Error field, streamlining JSON output.
- Introduced ToJSON method for Response and ListResponse types to facilitate easy serialization of response data.
- Ensured alignment with existing coding standards and improved code maintainability through consistent struct definitions.
2025-04-10 16:04:48 +08:00
Marvin Zhang
c4291e7a93 refactor: update response handling and utility functions
- Changed the summary conversion method in getIDForAction from PascalCase to SnakeCase for consistency with naming conventions.
- Introduced BaseResponse interface and implemented GetData and GetDataString methods for Response, ListResponse, and VoidResponse types, enhancing response handling.
- Simplified the Contains function to specifically handle string slices, improving clarity and maintainability of utility functions.
- Ensured alignment with existing coding standards and added necessary comments for better understanding.
2025-04-07 18:13:49 +08:00
Marvin Zhang
ed29a76566 refactor: restructure stats handling and remove legacy service interface
- Replaced the StatsService interface with direct MongoDB queries in the stats controller, enhancing performance and reducing complexity.
- Introduced structured response types for GetStatsOverview and GetStatsTasks, improving clarity and maintainability of the API responses.
- Removed outdated service implementation files, streamlining the codebase and eliminating redundancy.
- Enhanced error handling for MongoDB operations to ensure robustness in data retrieval processes.
2025-04-03 22:35:22 +08:00
Marvin Zhang
65d344a21e refactor: enhance sorting logic in GetSortsFromString function
- Replaced JSON unmarshalling with string parsing to handle sorting parameters more flexibly.
- Improved handling of sorting direction by allowing both ascending and descending specifications through prefixes.
- Enhanced code readability and maintainability by simplifying the sorting logic and removing unnecessary error handling.
2025-03-27 11:55:22 +08:00
Marvin Zhang
f86173c973 refactor: standardize filter handling across controllers
- Replaced "conditions" parameter with "filter" in various controller methods to unify the filtering approach.
- Updated related functions to utilize the new filter parameter, enhancing consistency in query handling.
- Improved overall code readability and maintainability by aligning naming conventions and refactoring filter-related logic.
2025-03-26 22:08:01 +08:00
Marvin Zhang
863ba153f6 fix: improve error handling and standardize configuration constants
- Enhanced error handling in BaseController to return a more descriptive "item not found" message when no documents are found in MongoDB.
- Standardized naming conventions for configuration constants by updating DefaultMcpServerBaseUrl to DefaultMCPServerBaseUrl and introducing DefaultMCPClientBaseUrl for clarity and consistency.
- Ensured alignment with existing coding standards and improved code readability.
2025-03-25 17:32:40 +08:00
Marvin Zhang
e73ee2e387 fix: improve error handling in the error hook
- Updated the error handling logic in the tonic error hook to unwrap nested errors, ensuring that the most relevant error message is returned.
- Enhanced the clarity of the error response by directly using the unwrapped error message, improving debugging and user feedback.
2025-03-19 22:46:35 +08:00
Marvin Zhang
931a36c8bf refactor: enhance parameter descriptions and standardize model fields
- Added descriptive annotations to various parameters across controllers to improve API documentation clarity.
- Standardized field definitions in models by including descriptions for better understanding of their purpose.
- Updated validation patterns for ID fields to ensure consistency and improve data integrity.
- Enhanced overall code readability and maintainability by aligning naming conventions and adding necessary comments.
2025-03-19 13:39:02 +08:00
Marvin Zhang
13ce1d551b refactor: enhance OpenAPI integration and security handling
- Updated GetOpenAPI function to improve clarity by storing the Fizz instance in a variable.
- Added security middleware to FizzWrapper for API token authentication, enhancing security measures.
- Configured sorting of tags in the OpenAPI generator to maintain consistency in API documentation.
- Improved comments for better understanding of the changes made in the OpenAPI wrapper.
2025-03-18 18:01:22 +08:00
Marvin Zhang
98843fdbac refactor: standardize sorting logic across controllers
- Introduced a consistent approach for sorting in multiple controller methods by utilizing GetSortOptionFromString for improved clarity and maintainability.
- Updated the GetListParams structure to ensure sorting is handled uniformly across different entities.
- Enhanced error handling for sorting parameters to provide clearer feedback on invalid formats.
- Improved comments for better understanding of sorting logic and its integration within the controller methods.
2025-03-17 17:14:08 +08:00
Marvin Zhang
95a6eb4e7b chore: add OpenTelemetry auto SDK dependency
- Introduced go.opentelemetry.io/auto/sdk v1.1.0 to go.mod and updated go.sum accordingly.
- Enhanced observability capabilities by including the OpenTelemetry SDK for automatic instrumentation.
- Ensured consistency in dependency management across the project.
2025-03-17 14:24:07 +08:00
Marvin Zhang
657c73ad2f refactor: improve filter query handling in utility functions
- Added nil return for empty condition strings in GetFilterQueryFromConditionString and GetFilterFromConditionString functions to enhance error handling.
- Improved clarity and maintainability by ensuring consistent behavior when handling empty input conditions.
- Updated comments for better understanding of the changes made.
2025-03-17 14:20:19 +08:00
Marvin Zhang
4dace3ce8e refactor: migrate router and controller methods to use Fizz package
- Updated router groups to utilize the new crawlab-team/fizz package for improved routing capabilities.
- Refactored controller methods to accept Fizz router groups, enhancing consistency and maintainability.
- Simplified route registration by incorporating OpenAPI metadata directly into group definitions.
- Improved error handling and response generation in sync controller methods for better clarity.
- Enhanced overall code structure by standardizing route definitions and improving parameter handling.
2025-03-17 12:50:50 +08:00
Marvin Zhang
bc5e6273b8 chore: update dependencies and refactor imports
- Updated various dependencies in go.mod and go.sum for improved stability and performance.
- Replaced references to the old fizz package with the new crawlab-team/fizz package in multiple files.
- Adjusted indirect dependencies to their latest versions for better compatibility.
- Enhanced overall code clarity by standardizing import paths across the project.
2025-03-17 12:08:35 +08:00
Marvin Zhang
43d1c7692b refactor: standardize response types across controllers
- Updated multiple controller methods to return VoidResponse instead of generic Response[any].
- Consolidated error handling to utilize GetErrorVoidResponse for consistent error responses.
- Enhanced parameter handling in export and file management functions for improved clarity and maintainability.
- Refactored health check and login/logout methods to align with new response structure.
- Improved overall consistency in response formatting across various endpoints.
2025-03-16 22:25:13 +08:00
Marvin Zhang
700c263cfe refactor: update filter constants and improve type handling in filter functions
- Renamed filter operation constants for clarity and consistency.
- Introduced short forms for certain filter operations.
- Updated test cases to reflect changes in filter operation handling.
- Enhanced type handling in filter functions to support int64 values for conditions.
- Improved query generation logic to accommodate new constant names and types.
2025-03-16 20:30:52 +08:00
Marvin Zhang
ce0143ca06 refactor: enhance health check function and add comprehensive test coverage
- Updated GetHealthFn to return an error for better error handling and clarity.
- Introduced a new test file for schedule management, covering various endpoints including creation, retrieval, updating, and deletion of schedules.
- Added tests for task management, including task creation, retrieval, updating, and cancellation.
- Implemented utility tests for filtering and response generation to ensure consistent API behavior.
- Improved logging in the task scheduler service for better traceability.
2025-03-13 18:10:24 +08:00
Marvin Zhang
43691d0d8a refactor: enhance controller parameter handling and error responses
- Refactored export and filter controller methods to utilize structured parameter types for improved clarity and maintainability.
- Consolidated error handling and response generation, ensuring consistent API responses across methods.
- Updated function signatures to eliminate unnecessary context parameters, enhancing type safety and readability.
- Improved handling of query parameters and conditions, streamlining the logic for better performance and maintainability.
2025-03-13 17:43:55 +08:00
Marvin Zhang
23cad00d92 refactor: streamline controller methods and enhance parameter handling
- Refactored multiple controller methods to accept structured parameters for improved clarity and maintainability.
- Consolidated error handling and response generation across various endpoints.
- Updated function signatures to eliminate unnecessary context parameters and enhance type safety.
- Improved consistency in response formatting and error handling across controllers.
- Enhanced file handling methods to support multipart file uploads and directory operations more effectively.
2025-03-13 17:37:30 +08:00
Marvin Zhang
ddff881954 chore: update Go version and dependencies
- Updated Go version in go.work and backend/go.mod to 1.23.7
- Updated various dependencies in go.sum and backend/go.sum
- Refactored models to remove generic type parameters from BaseModel
- Introduced new utility functions for consistent API responses
- Removed unused utility files from controllers
2025-03-12 23:20:06 +08:00
Marvin Zhang
d6badb533d feat: enhance API routing and OpenAPI documentation support
- Introduced a new OpenAPI wrapper using Fizz for improved API documentation
- Refactored base controller to support more flexible route handling
- Added dynamic route registration with OpenAPI metadata
- Implemented generic response types for consistent API responses
- Updated router initialization to support OpenAPI documentation endpoint
- Improved route and resource naming utilities
- Migrated existing controllers to use new routing and response mechanisms
2025-03-11 23:45:06 +08:00