Commit Graph

337 Commits

Author SHA1 Message Date
Marvin Zhang
9745129e33 feat: configure test database as master node for testing 2025-07-23 15:28:51 +08:00
Marvin Zhang
cf5ec81250 fix: add nil checks for logger in config initialization and logging methods 2025-07-23 15:07:39 +08:00
Marvin Zhang
a2d13fae36 feat: temporarily disable batch file saving route and implement alternative handler in spider controller 2025-07-23 14:55:04 +08:00
Marvin Zhang
b4288b08a5 fix: Update FsFileInfo to use pointer slices for children and remove redundant getter methods 2025-07-23 09:42:07 +08:00
Marvin Zhang
3c3ff09723 feat: enhance gRPC client with health check functionality and improved connection handling 2025-07-09 14:38:53 +08:00
Marvin Zhang
20ba390cf6 refactor: improve mongo client connection error logging format and remove redundant gRPC server start in MasterService 2025-07-09 14:06:10 +08:00
Marvin Zhang
46c0cd6298 refactor: update gRPC client access patterns to use safe getter methods for improved error handling 2025-07-08 18:08:46 +08:00
Marvin Zhang
8bd3ef0b72 feat: add goroutine count metric to the MetricService and update related files 2025-07-08 14:08:31 +08:00
Marvin Zhang
00daa0ed96 fix: enhance gRPC client reconnection logic and add goroutine monitoring for potential leaks 2025-07-08 13:39:39 +08:00
Marvin Zhang
f8e9c45a85 fix: enhance gRPC client connection management with circuit breaker and keep-alive settings 2025-07-08 13:34:43 +08:00
Marvin Zhang
92046a8c2e fix: improve task cancellation and connection health check logic with timeout handling 2025-06-27 14:02:24 +08:00
Marvin Zhang
9f251f3ebe fix: enhance task cancellation logic with graceful termination and stuck task cleanup 2025-06-27 13:50:21 +08:00
Marvin Zhang
89514b0154 feat: implement zombie process prevention and cleanup mechanisms in task runner 2025-06-23 13:54:43 +08:00
Marvin Zhang
1008886715 fix: enhance task service resilience with connection health monitoring and periodic cleanup 2025-06-23 11:57:05 +08:00
Marvin Zhang
5837472de5 fix: update API client constructor and improve error handling for missing API token and URL
chore: update package version to 0.1.1-dev.6 and adjust dependencies in pnpm-lock.yaml
refactor: change import paths to relative in tools.ts for consistency
2025-06-20 15:55:50 +08:00
Marvin Zhang
13038d4a1a fix: unable to sync files from master in worker nodes 2025-06-20 14:42:52 +08:00
Marvin Zhang
09cfe37272 fix: unable to sync files to worker nodes when running tasks 2025-06-18 21:50:53 +08:00
Marvin Zhang
22497d22f7 fix: unable to filter tasks issue 2025-06-17 09:28:25 +08:00
Marvin Zhang
24452bff23 refactor: extract token retrieval logic into GetAPITokenFromContext utility function 2025-06-16 21:49:27 +08:00
Marvin Zhang
6d65cc13a2 feat: add aggregation methods to ModelService for enhanced data retrieval 2025-06-15 18:03:35 +08:00
Marvin Zhang
ab5e65f2fb chore: update dependencies to latest versions across modules 2025-06-15 18:03:13 +08:00
Marvin Zhang
335050e72e refactor: improve schedule service initialization, validation, and error handling 2025-06-13 21:53:08 +08:00
Marvin Zhang
0bd2b62792 feat: add DTOs for models and update project spiders type to array 2025-06-12 19:58:21 +08:00
Marvin Zhang
79b7e074e1 refactor: update filter parameter in API requests and improve component structure 2025-06-12 00:17:40 +08:00
Marvin Zhang
622dce51c3 fix: goroutine cleanup and error handling during shutdown 2025-06-11 22:30:37 +08:00
Marvin Zhang
de69c2b01b refactor: replace activeNodes with allList in dependency dialogs and related components 2025-06-11 13:34:08 +08:00
Marvin Zhang
1f3a9586b3 feat: enhance data model associations and remove unused components 2025-06-09 23:00:47 +08:00
Marvin Zhang
975c601f38 refactor: simplify list retrieval methods and enhance error handling 2025-06-09 10:21:00 +08:00
Marvin Zhang
8a00af115c feat: refactor to remove unused 'getAllList' methods and related state properties 2025-06-09 10:20:39 +08:00
Marvin Zhang
8d32d54fe8 feat: add development and index generation run configurations; update SVG files and improve icon styles 2025-06-06 14:59:59 +08:00
Marvin Zhang
062077a9bd feat: restructure user parameters for PostUser and simplify email validation; refactor UserAvatar styles and improve user name handling 2025-05-29 22:16:27 +08:00
Marvin Zhang
512f246094 feat: implement AutoProbe components and enhance LLM provider functionality
- Introduced AutoProbeForm and AutoProbeTaskStatus components for managing AutoProbe configurations and task statuses.
- Updated LLMProvider model to include a default model field for improved provider management.
- Enhanced AutoProbeList and AutoProbeDetail components for better user interaction and data display.
- Refactored related Vue components and services to streamline AutoProbe functionality and improve state management.
- Added internationalization support for new AutoProbe features and updated existing translations.
2025-05-12 17:37:30 +08:00
Marvin Zhang
9e25ea9b51 refactor: standardize LLM provider key naming and enhance related components
- Renamed 'Key' fields to 'Type' in various models for consistency across the codebase.
- Updated route naming in router.go for clarity in the context of settings.
- Introduced a new LlmProviderForm component for managing LLM provider configurations.
- Refactored AssistantConsole and related components to utilize providerId instead of provider for improved state management.
- Enhanced UI elements and translations to reflect the new naming conventions and improve user experience.
2025-05-09 15:35:57 +08:00
Marvin Zhang
73b3c1bb70 refactor: add IgnoreFileRegexPattern constant for directory scanning
- Introduced a new constant `IgnoreFileRegexPattern` to specify patterns for files to ignore during directory scanning.
- Adjusted import order in `file.go` for improved organization and clarity.
2025-05-04 21:44:27 +08:00
Marvin Zhang
8e7f52ab07 chore: update dependencies and improve module configurations
- Added new dependencies: `github.com/crawlab-team/fizz`, `github.com/go-playground/validator/v10`, and `github.com/google/go-cmp`.
- Updated existing dependencies: `github.com/spf13/cobra` to v1.7.0, `github.com/inconshreveable/mousetrap` to v1.1.0, and `golang.org/x/exp` to v0.0.0-20241217172543-b2144cdd0a67.
- Removed outdated dependencies from `go.sum` to streamline the module.
- Adjusted Go version in `go.mod` files to 1.23.0 for improved compatibility.
- Enhanced indirect dependencies for better performance and stability.
2025-04-24 18:31:12 +08:00
Marvin Zhang
fdd0595f49 refactor: update route naming for clarity in router.go
- Renamed route from "Get Spider Files" to "List Spider Files" for improved clarity.
- Enhanced description to specify that it lists files given a directory path in the spider project.
2025-04-23 15:50:10 +08:00
Marvin Zhang
f28720ab67 refactor: optimize type assertions and simplify sort option checks in utils.go
- Streamlined type assertions in convertFilterValue function for improved clarity and performance.
- Removed unnecessary nil checks for sorts in GetSortOptionFromString and GetSortsOption functions, enhancing code readability.
2025-04-23 13:58:16 +08:00
Marvin Zhang
c4f2d1e554 fix: restore missing error handling import in utils.go
- Reintroduced the 'github.com/juju/errors' import in utils.go to ensure proper error handling functionality is available in the codebase.
2025-04-23 13:55:04 +08:00
Marvin Zhang
1aa58f5065 refactor: remove unused task type field in runner test setup
- Eliminated the unused 'Type' field from the Task model in the runner test setup to enhance code clarity and maintainability.
2025-04-22 16:50:56 +08:00
Marvin Zhang
4f57d277e7 refactor: standardize timestamp fields and improve code clarity
- Updated timestamp fields across the codebase from `*_ts` to `*_at` for consistency and clarity.
- Renamed constants for node status from "on"/"off" to "online"/"offline" to better reflect their meanings.
- Enhanced validation and error handling in various components to ensure data integrity.
- Refactored test cases to align with the new naming conventions and improve readability.
2025-04-21 18:13:22 +08:00
Marvin Zhang
34509b8d4c refactor: enhance parameter handling and improve code clarity
- Updated GetListParams to set a default sort option for better query handling.
- Enhanced PostSpiderRunParams to include a default mode and improved error handling for missing spider.
- Added parameters field to ChatMessageContent for more flexible message content management.
- Refactored getNodeIds method to simplify mode handling and removed unnecessary error checks.
- Improved ChatMessageAction component to display parameters and response sections more effectively, enhancing user experience.
2025-04-17 18:03:15 +08:00
Marvin Zhang
336ca5770b fix: improve error handling and enhance setting management
- Updated error handling in GetSetting to use errors.Is for better clarity when checking for no documents.
- Refactored setting retrieval in the Vuex store to ensure a default value is set if the retrieved setting is empty.
- Adjusted API calls in the store to wrap payloads in a data object for consistency.
- Cleaned up SystemDetail.vue by removing unnecessary navigation actions for a more streamlined UI.
- Enhanced conditional rendering in SystemDetailTabDependency to ensure form value checks are safe.
- Replaced el-input with cl-edit-input in SystemDetailTabCustomize for improved input handling and added change event for saving.
2025-04-17 14:50:59 +08:00
Marvin Zhang
1ce6f87ad5 feat: add global node_bin path configuration in Runner
- Introduced a new method to configure the global node_bin path in the Runner, ensuring it is included in the system PATH if not already present.
- Added GetNodeBinPath function in utils to retrieve the node_bin path from configuration, with a default fallback.
- Enhanced environment variable management for better integration of Node.js binaries.
2025-04-15 20:55:47 +08:00
Marvin Zhang
f167ff7600 refactor: rename GetSpiderListDir to GetSpiderFiles for clarity
- Updated route name and handler function to better reflect the purpose of retrieving files in a spider directory.
- Renamed parameter struct from GetSpiderListDirParams to GetSpiderFilesParams to maintain consistency in naming conventions.
2025-04-14 23:25:26 +08:00
Marvin Zhang
67b51fdc83 refactor: update LLMResponseUsage struct and ChatMessage usage aggregation
- Renamed fields in LLMResponseUsage struct to better reflect their purpose, changing InputTokens and OutputTokens to PromptTokens and CompletionTokens.
- Modified GetUsage method in ChatMessage to accept an optional contents parameter, improving flexibility in usage tracking.
- Ensured consistent field naming and improved clarity in token usage aggregation logic.
2025-04-14 16:29:32 +08:00
Marvin Zhang
8273507d20 refactor: update LLMResponseUsage struct and ChatMessage model for BSON compatibility
- Added BSON tags to fields in LLMResponseUsage struct to ensure proper serialization with MongoDB.
- Updated the Usage field in ChatMessage model to exclude it from JSON serialization while retaining BSON compatibility.
- Introduced GetUsage method in ChatMessage to aggregate token usage from contents, enhancing usage tracking capabilities.
2025-04-12 21:25:57 +08:00
Marvin Zhang
2549abdc61 feat: introduce LLMResponseUsage struct and update ChatMessage models
- Added LLMResponseUsage struct to encapsulate token usage details for language model responses.
- Updated ChatMessageContent and ChatMessage models to include a Usage field of type *entity.LLMResponseUsage, enhancing message handling capabilities.
- Improved structure and clarity of models by ensuring consistent field definitions and descriptions.
2025-04-11 22:29:02 +08:00
Marvin Zhang
631f6f96b2 refactor: simplify sorting logic in GetSortsFromString function
- Adjusted the handling of sorting direction by directly checking for descending and ascending prefixes.
- Removed redundant code for handling ascending sorting, streamlining the function's logic.
- Enhanced code clarity and maintainability by simplifying the sorting parameter processing.
2025-04-10 23:25:29 +08:00
Marvin Zhang
0a3c706dca feat: enhance task and spider result handling in API
- Updated the description for the GetSpiderResults route to clarify the type of data returned.
- Added a new route for GetTaskResults to retrieve scraped or crawled results data for tasks.
- Refactored the GetSpiderResults and GetTaskResults functions to improve query handling and error management.
- Ensured consistent usage of MongoDB query methods across the controllers for better maintainability.
2025-04-10 23:18:17 +08:00
Marvin Zhang
95bf247ab1 refactor: enhance response structures and add JSON serialization
- Removed redundant comment for GetListParams struct to improve clarity.
- Updated Response, ListResponse, and VoidResponse types to use omitempty for the Error field, streamlining JSON output.
- Introduced ToJSON method for Response and ListResponse types to facilitate easy serialization of response data.
- Ensured alignment with existing coding standards and improved code maintainability through consistent struct definitions.
2025-04-10 16:04:48 +08:00