Commit Graph

6022 Commits

Author SHA1 Message Date
Marvin Zhang
c62d53576e feat: enhance AutoProbeDetailTabTasks with auto-update and confirmation dialog
- Integrated setupAutoUpdate to automatically refresh task list.
- Added confirmation dialog before running a task to improve user interaction.
- Simplified onBeforeMount lifecycle hook for better readability and performance.
2025-05-13 12:24:18 +08:00
Marvin Zhang
512f246094 feat: implement AutoProbe components and enhance LLM provider functionality
- Introduced AutoProbeForm and AutoProbeTaskStatus components for managing AutoProbe configurations and task statuses.
- Updated LLMProvider model to include a default model field for improved provider management.
- Enhanced AutoProbeList and AutoProbeDetail components for better user interaction and data display.
- Refactored related Vue components and services to streamline AutoProbe functionality and improve state management.
- Added internationalization support for new AutoProbe features and updated existing translations.
2025-05-12 17:37:30 +08:00
Marvin Zhang
ec5bcf9287 feat: introduce AutoProbe module and related interfaces
- Added AutoProbe interfaces to define models for autoprobe functionality, including AutoProbe, AutoProbeTask, and AutoProbeFetchResult.
- Created AutoProbeStore module for managing state, getters, mutations, and actions related to autoprobe.
- Updated existing LLM interfaces to utilize LLMResponseUsage for token tracking.
- Refactored store index to include autoprobe namespace and removed obsolete AiStore module.
2025-05-12 11:25:30 +08:00
Marvin Zhang
a10a05ea80 feat: add AutoProbeList component and related functionality
- Introduced AutoProbeList.vue component for displaying a list layout of extract items.
- Added useAutoProbeList.ts for managing state and interactions with the Vuex store.
- Updated .gitignore to exclude local configuration files.
2025-05-11 22:46:46 +08:00
Marvin Zhang
9e25ea9b51 refactor: standardize LLM provider key naming and enhance related components
- Renamed 'Key' fields to 'Type' in various models for consistency across the codebase.
- Updated route naming in router.go for clarity in the context of settings.
- Introduced a new LlmProviderForm component for managing LLM provider configurations.
- Refactored AssistantConsole and related components to utilize providerId instead of provider for improved state management.
- Enhanced UI elements and translations to reflect the new naming conventions and improve user experience.
2025-05-09 15:35:57 +08:00
Marvin Zhang
73b3c1bb70 refactor: add IgnoreFileRegexPattern constant for directory scanning
- Introduced a new constant `IgnoreFileRegexPattern` to specify patterns for files to ignore during directory scanning.
- Adjusted import order in `file.go` for improved organization and clarity.
2025-05-04 21:44:27 +08:00
Marvin Zhang
8e7f52ab07 chore: update dependencies and improve module configurations
- Added new dependencies: `github.com/crawlab-team/fizz`, `github.com/go-playground/validator/v10`, and `github.com/google/go-cmp`.
- Updated existing dependencies: `github.com/spf13/cobra` to v1.7.0, `github.com/inconshreveable/mousetrap` to v1.1.0, and `golang.org/x/exp` to v0.0.0-20241217172543-b2144cdd0a67.
- Removed outdated dependencies from `go.sum` to streamline the module.
- Adjusted Go version in `go.mod` files to 1.23.0 for improved compatibility.
- Enhanced indirect dependencies for better performance and stability.
2025-04-24 18:31:12 +08:00
Marvin Zhang
d1e6f5a29e refactor: enhance ChatMessage component for improved user experience
- Added loading indicator for streaming messages to inform users during processing.
- Updated token usage display to use an icon for better visual clarity and interaction.
- Refactored CSS styles for token usage to improve layout and hover effects.
- Removed redundant code for displaying message time, streamlining the template structure.
2025-04-24 15:35:34 +08:00
Marvin Zhang
fdd0595f49 refactor: update route naming for clarity in router.go
- Renamed route from "Get Spider Files" to "List Spider Files" for improved clarity.
- Enhanced description to specify that it lists files given a directory path in the spider project.
2025-04-23 15:50:10 +08:00
Marvin Zhang
f28720ab67 refactor: optimize type assertions and simplify sort option checks in utils.go
- Streamlined type assertions in convertFilterValue function for improved clarity and performance.
- Removed unnecessary nil checks for sorts in GetSortOptionFromString and GetSortsOption functions, enhancing code readability.
2025-04-23 13:58:16 +08:00
Marvin Zhang
c4f2d1e554 fix: restore missing error handling import in utils.go
- Reintroduced the 'github.com/juju/errors' import in utils.go to ensure proper error handling functionality is available in the codebase.
2025-04-23 13:55:04 +08:00
Marvin Zhang
1aa58f5065 refactor: remove unused task type field in runner test setup
- Eliminated the unused 'Type' field from the Task model in the runner test setup to enhance code clarity and maintainability.
2025-04-22 16:50:56 +08:00
Marvin Zhang
4f57d277e7 refactor: standardize timestamp fields and improve code clarity
- Updated timestamp fields across the codebase from `*_ts` to `*_at` for consistency and clarity.
- Renamed constants for node status from "on"/"off" to "online"/"offline" to better reflect their meanings.
- Enhanced validation and error handling in various components to ensure data integrity.
- Refactored test cases to align with the new naming conventions and improve readability.
2025-04-21 18:13:22 +08:00
Marvin Zhang
de680d8ebd refactor: streamline conversation initialization and enhance log handling
- Refactored AssistantConsole to utilize a dedicated initializeConversation function for improved clarity and organization.
- Updated DependencyLogsDialog to rename computed property from content to logs for better semantic understanding.
- Enhanced LogsView component by wrapping logs in an el-scrollbar for improved user experience.
- Adjusted ChatInput to call updateSelectedProviderModel on onBeforeMount for better state management.
- Simplified default model handling in ai.ts by introducing a DEFAULT_OPENAI_MODELS constant for consistency.
2025-04-18 16:58:42 +08:00
Marvin Zhang
32f0802f8b style: update AssistantConsole layout and extend store interface
- Added height to the AssistantConsole component for improved layout consistency.
- Extended StoreNamespace interface to include 'dataCollection' for better state management.
2025-04-18 09:40:59 +08:00
Marvin Zhang
34509b8d4c refactor: enhance parameter handling and improve code clarity
- Updated GetListParams to set a default sort option for better query handling.
- Enhanced PostSpiderRunParams to include a default mode and improved error handling for missing spider.
- Added parameters field to ChatMessageContent for more flexible message content management.
- Refactored getNodeIds method to simplify mode handling and removed unnecessary error checks.
- Improved ChatMessageAction component to display parameters and response sections more effectively, enhancing user experience.
2025-04-17 18:03:15 +08:00
Marvin Zhang
36c7d5e3a3 refactor: improve focus handling and simplify content parsing
- Refactored focus handling in useAssistantConsole to use a dedicated function for clarity and adjusted timeout duration.
- Simplified content parsing in ChatMessageAction by directly returning parsed JSON and removing unnecessary mapping logic.
- Updated ChatSidebar to allow a larger maximum width for improved layout flexibility.
- Removed unused interface definitions in llm.d.ts to clean up the codebase.
2025-04-17 16:28:56 +08:00
Marvin Zhang
336ca5770b fix: improve error handling and enhance setting management
- Updated error handling in GetSetting to use errors.Is for better clarity when checking for no documents.
- Refactored setting retrieval in the Vuex store to ensure a default value is set if the retrieved setting is empty.
- Adjusted API calls in the store to wrap payloads in a data object for consistency.
- Cleaned up SystemDetail.vue by removing unnecessary navigation actions for a more streamlined UI.
- Enhanced conditional rendering in SystemDetailTabDependency to ensure form value checks are safe.
- Replaced el-input with cl-edit-input in SystemDetailTabCustomize for improved input handling and added change event for saving.
2025-04-17 14:50:59 +08:00
Marvin Zhang
e0649dc91f chore: simplify Docker workflow by removing e2e test job
- Eliminated the e2e test job from the Docker workflow to streamline the CI process.
- Removed related environment variables and steps associated with e2e testing.
- Updated dependencies for the push_images job to reflect changes in the workflow structure.
2025-04-16 15:28:55 +08:00
Marvin Zhang
526b2c0579 feat: enhance Vite configuration and update metrics display
- Added support for JSX in Vite configuration by including the vueJsx plugin.
- Simplified the metrics display in CurrentMetrics.vue to show only the percentage of used memory and disk.
- Adjusted CSS in CurrentMetrics.vue to prevent wrapping and added a gap between items for better layout.
- Refactored TableCell.vue to remove unused imports, optimizing the component.
- Updated useNodeList.tsx to replace NodeType with ClNodeType and made minor formatting adjustments for consistency.
2025-04-16 14:57:51 +08:00
Marvin Zhang
4e6b00c1b1 feat: add LogsView component for displaying logs
- Introduced a new LogsView.vue component to present logs in a structured format.
- Implemented functionality to scroll to the bottom of the logs view for better user experience.
- Enhanced log processing to trim and format log entries for clarity.
2025-04-16 11:05:00 +08:00
Marvin Zhang
21df22fe24 chore: enhance Dockerfile for improved debugging
- Updated the Dockerfile to include additional commands for listing files in the application directory structure, aiding in the verification of file copying during the Docker build process.
2025-04-16 10:56:10 +08:00
Marvin Zhang
47810fa2bb chore: update Dockerfile for debugging file copy verification
- Added commands to list files and display the contents of LogsView.vue to verify correct copying of application code during the Docker build process.
2025-04-16 10:54:02 +08:00
Marvin Zhang
13fd2d4c4e feat: add preview script and optimize Vite configuration
- Added a new "preview" script to package.json for easier local previewing of the application.
- Refactored Vite configuration by removing unnecessary library build options and external dependencies, streamlining the build process.
- Updated manual chunking logic to improve module handling and included additional dependencies in optimizeDeps for better performance.
2025-04-16 10:51:37 +08:00
Marvin Zhang
ca5113fd33 chore: update Dockerfile and add .dockerignore
- Modified the Dockerfile to change the way application code is added, improving clarity and structure.
- Updated the COPY command to reference the correct path for built artifacts.
- Added a new .dockerignore file to exclude node_modules from the Docker build context, optimizing the build process.
- Removed the unnecessary .Dockerfile.swp file.
2025-04-16 10:40:31 +08:00
Marvin Zhang
864ab25d83 feat: add SVG assets and update component interfaces
- Introduced new SVG assets for various AI models including Anthropic, Azure, DeepSeek, Gemini, Mistral, OpenAI, and Qwen.
- Updated the components index to include the new useAssistantConsole functionality.
- Enhanced type definitions in the i18n and models interfaces to support new AI features, including chat message handling and model configurations.
- Added new properties for better customization and management of AI models and chat conversations.
2025-04-16 10:36:52 +08:00
Marvin Zhang
fac1206a65 refactor: migrated crawlab-ui to crawlab/frontend/crawlab-ui 2025-04-16 10:33:07 +08:00
Marvin Zhang
1ce6f87ad5 feat: add global node_bin path configuration in Runner
- Introduced a new method to configure the global node_bin path in the Runner, ensuring it is included in the system PATH if not already present.
- Added GetNodeBinPath function in utils to retrieve the node_bin path from configuration, with a default fallback.
- Enhanced environment variable management for better integration of Node.js binaries.
2025-04-15 20:55:47 +08:00
Marvin Zhang
f167ff7600 refactor: rename GetSpiderListDir to GetSpiderFiles for clarity
- Updated route name and handler function to better reflect the purpose of retrieving files in a spider directory.
- Renamed parameter struct from GetSpiderListDirParams to GetSpiderFilesParams to maintain consistency in naming conventions.
2025-04-14 23:25:26 +08:00
Marvin Zhang
67b51fdc83 refactor: update LLMResponseUsage struct and ChatMessage usage aggregation
- Renamed fields in LLMResponseUsage struct to better reflect their purpose, changing InputTokens and OutputTokens to PromptTokens and CompletionTokens.
- Modified GetUsage method in ChatMessage to accept an optional contents parameter, improving flexibility in usage tracking.
- Ensured consistent field naming and improved clarity in token usage aggregation logic.
2025-04-14 16:29:32 +08:00
Marvin Zhang
8273507d20 refactor: update LLMResponseUsage struct and ChatMessage model for BSON compatibility
- Added BSON tags to fields in LLMResponseUsage struct to ensure proper serialization with MongoDB.
- Updated the Usage field in ChatMessage model to exclude it from JSON serialization while retaining BSON compatibility.
- Introduced GetUsage method in ChatMessage to aggregate token usage from contents, enhancing usage tracking capabilities.
2025-04-12 21:25:57 +08:00
Marvin Zhang
2549abdc61 feat: introduce LLMResponseUsage struct and update ChatMessage models
- Added LLMResponseUsage struct to encapsulate token usage details for language model responses.
- Updated ChatMessageContent and ChatMessage models to include a Usage field of type *entity.LLMResponseUsage, enhancing message handling capabilities.
- Improved structure and clarity of models by ensuring consistent field definitions and descriptions.
2025-04-11 22:29:02 +08:00
Marvin Zhang
631f6f96b2 refactor: simplify sorting logic in GetSortsFromString function
- Adjusted the handling of sorting direction by directly checking for descending and ascending prefixes.
- Removed redundant code for handling ascending sorting, streamlining the function's logic.
- Enhanced code clarity and maintainability by simplifying the sorting parameter processing.
2025-04-10 23:25:29 +08:00
Marvin Zhang
0a3c706dca feat: enhance task and spider result handling in API
- Updated the description for the GetSpiderResults route to clarify the type of data returned.
- Added a new route for GetTaskResults to retrieve scraped or crawled results data for tasks.
- Refactored the GetSpiderResults and GetTaskResults functions to improve query handling and error management.
- Ensured consistent usage of MongoDB query methods across the controllers for better maintainability.
2025-04-10 23:18:17 +08:00
Marvin Zhang
3b9b3115a3 chore: update Go base image version in Dockerfile
- Changed the base image from golang:1.22.9 to golang:1.23 to leverage the latest features and improvements.
- Ensured alignment with existing coding standards and maintained consistency in the Dockerfile structure.
2025-04-10 17:41:56 +08:00
Marvin Zhang
95bf247ab1 refactor: enhance response structures and add JSON serialization
- Removed redundant comment for GetListParams struct to improve clarity.
- Updated Response, ListResponse, and VoidResponse types to use omitempty for the Error field, streamlining JSON output.
- Introduced ToJSON method for Response and ListResponse types to facilitate easy serialization of response data.
- Ensured alignment with existing coding standards and improved code maintainability through consistent struct definitions.
2025-04-10 16:04:48 +08:00
Marvin Zhang
3372c9a69b refactor: update ChatMessageContent model for improved structure and clarity
- Removed type definitions for ChatMessageContentType and ChatMessageContentActionStatus, replacing them with string fields for flexibility.
- Added a new 'Key' field to ChatMessageContent for enhanced content identification.
- Updated the 'Content' field description in ChatMessage to specify its context for user/system messages.
- Ensured consistency in field naming and descriptions across models for better maintainability.
2025-04-09 22:25:01 +08:00
Marvin Zhang
37307d7c01 feat: introduce ChatMessageContent model for enhanced message handling
- Added ChatMessageContent model to encapsulate message content types and action statuses, improving the structure of chat messages.
- Updated ChatMessage model to include a slice of ChatMessageContent, allowing for multiple content items per message.
- Implemented GetContent method to handle content retrieval and formatting based on content type, enhancing message processing capabilities.
- Ensured alignment with existing coding standards and added necessary comments for clarity and maintainability.
2025-04-08 17:58:43 +08:00
Marvin Zhang
c4291e7a93 refactor: update response handling and utility functions
- Changed the summary conversion method in getIDForAction from PascalCase to SnakeCase for consistency with naming conventions.
- Introduced BaseResponse interface and implemented GetData and GetDataString methods for Response, ListResponse, and VoidResponse types, enhancing response handling.
- Simplified the Contains function to specifically handle string slices, improving clarity and maintainability of utility functions.
- Ensured alignment with existing coding standards and added necessary comments for better understanding.
2025-04-07 18:13:49 +08:00
Marvin Zhang
ed29a76566 refactor: restructure stats handling and remove legacy service interface
- Replaced the StatsService interface with direct MongoDB queries in the stats controller, enhancing performance and reducing complexity.
- Introduced structured response types for GetStatsOverview and GetStatsTasks, improving clarity and maintainability of the API responses.
- Removed outdated service implementation files, streamlining the codebase and eliminating redundancy.
- Enhanced error handling for MongoDB operations to ensure robustness in data retrieval processes.
2025-04-03 22:35:22 +08:00
Marvin Zhang
04d803f7c9 fix: enhance MongoDB client connection error handling
- Added a ping check after establishing a MongoDB client connection to ensure the connection is valid.
- Implemented error logging for ping failures to improve debugging and provide clearer feedback on connection issues.
- Ensured alignment with existing error handling practices in the codebase for consistency and maintainability.
2025-04-03 12:14:20 +08:00
Marvin Zhang
65d344a21e refactor: enhance sorting logic in GetSortsFromString function
- Replaced JSON unmarshalling with string parsing to handle sorting parameters more flexibly.
- Improved handling of sorting direction by allowing both ascending and descending specifications through prefixes.
- Enhanced code readability and maintainability by simplifying the sorting logic and removing unnecessary error handling.
2025-03-27 11:55:22 +08:00
Marvin Zhang
f86173c973 refactor: standardize filter handling across controllers
- Replaced "conditions" parameter with "filter" in various controller methods to unify the filtering approach.
- Updated related functions to utilize the new filter parameter, enhancing consistency in query handling.
- Improved overall code readability and maintainability by aligning naming conventions and refactoring filter-related logic.
2025-03-26 22:08:01 +08:00
Marvin Zhang
863ba153f6 fix: improve error handling and standardize configuration constants
- Enhanced error handling in BaseController to return a more descriptive "item not found" message when no documents are found in MongoDB.
- Standardized naming conventions for configuration constants by updating DefaultMcpServerBaseUrl to DefaultMCPServerBaseUrl and introducing DefaultMCPClientBaseUrl for clarity and consistency.
- Ensured alignment with existing coding standards and improved code readability.
2025-03-25 17:32:40 +08:00
Marvin Zhang
e73ee2e387 fix: improve error handling in the error hook
- Updated the error handling logic in the tonic error hook to unwrap nested errors, ensuring that the most relevant error message is returned.
- Enhanced the clarity of the error response by directly using the unwrapped error message, improving debugging and user feedback.
2025-03-19 22:46:35 +08:00
Marvin Zhang
c713416a91 refactor: standardize sorting direction constants and JSON annotations
- Updated DESCENDING constant from "dsc" to "desc" for clarity and consistency.
- Changed JSON annotation for Direction field in Sort struct from "d" to "direction" to improve readability and maintainability.
- Ensured alignment with existing coding standards and practices across the codebase.
2025-03-19 18:15:17 +08:00
Marvin Zhang
3d11a14fd8 feat: add ChatConversation and ChatMessage models
- Introduced ChatConversation and ChatMessage structs to represent conversation and message data in the application.
- Included relevant fields with JSON and BSON annotations for proper serialization and database integration.
- Enhanced model definitions with descriptions for better understanding of their purpose and usage.
- Ensured alignment with existing coding standards and practices for maintainability.
2025-03-19 16:46:52 +08:00
Marvin Zhang
931a36c8bf refactor: enhance parameter descriptions and standardize model fields
- Added descriptive annotations to various parameters across controllers to improve API documentation clarity.
- Standardized field definitions in models by including descriptions for better understanding of their purpose.
- Updated validation patterns for ID fields to ensure consistency and improve data integrity.
- Enhanced overall code readability and maintainability by aligning naming conventions and adding necessary comments.
2025-03-19 13:39:02 +08:00
Marvin Zhang
13ce1d551b refactor: enhance OpenAPI integration and security handling
- Updated GetOpenAPI function to improve clarity by storing the Fizz instance in a variable.
- Added security middleware to FizzWrapper for API token authentication, enhancing security measures.
- Configured sorting of tags in the OpenAPI generator to maintain consistency in API documentation.
- Improved comments for better understanding of the changes made in the OpenAPI wrapper.
2025-03-18 18:01:22 +08:00
Marvin Zhang
3ca086d969 fix: update DefaultMcpServerBaseUrl for consistency
- Changed DefaultMcpServerBaseUrl from "http://localhost:9000/sse" to "http://localhost:9000" for improved clarity and consistency in configuration.
- Ensured alignment with other base URL definitions in the config.go file.
2025-03-18 15:18:02 +08:00