Commit Graph

6076 Commits

Author SHA1 Message Date
Marvin Zhang
b7b7f25451 refactor: remove unused response variables in database, git, node, project, schedule, spider, task modules 2025-06-20 14:55:54 +08:00
Marvin Zhang
4f33460d45 chore: remove .gitattributes file to simplify language detection 2025-06-20 14:45:22 +08:00
Marvin Zhang
13038d4a1a fix: unable to sync files from master in worker nodes 2025-06-20 14:42:52 +08:00
Marvin Zhang
52b180aaa0 ci: Add GitHub Actions workflow for publishing MCP package 2025-06-20 11:09:02 +08:00
Marvin Zhang
994adc3f71 fix: Update TypeScript ESLint packages and configuration 2025-06-20 10:37:39 +08:00
Marvin Zhang
2d0508a0c2 feat: Add AI tools for LLM provider management and AutoProbe configurations
- Implemented new API methods in CrawlabClient for managing LLM providers and AutoProbes.
- Added prompts for setting up AI-powered web scraping with AutoProbe and configuring LLM providers.
- Created a new module for AI tools, integrating various functionalities including listing, creating, updating, and deleting LLM providers and AutoProbes.
- Enhanced error handling and response formatting for better user experience.
2025-06-19 16:31:58 +08:00
Marvin Zhang
3976b93e66 feat: Add database and git tools, enhance node and spider configurations
- Implemented database tools for CRUD operations and connection testing.
- Added git tools for managing repositories including create, update, delete, clone, and pull functionalities.
- Enhanced node tools with update, enable, and disable functionalities.
- Updated spider tools to include new parameters for database integration, git repository association, and task execution modes.
- Improved schedule tools with additional parameters for task execution modes and priority settings.
- Introduced system overview tool for comprehensive system health and status reporting.
- Updated task filtering to include new task statuses.
2025-06-19 15:52:09 +08:00
Marvin Zhang
c29e21deec feat: Implement Crawlab MCP Server with tools and prompts
- Added main server file (index.ts) to initialize the Crawlab MCP Server.
- Created prompts for spider analysis, task debugging, spider setup, and system monitoring.
- Developed tools for managing spiders, tasks, nodes, schedules, and system health.
- Implemented a mock client for testing server functionality.
- Added versioning support with version.ts.
- Created a test script (test-server.mjs) to validate tool configurations and server responses.
- Included build validation script (validate-build.mjs) to ensure proper setup and functionality.
- Configured TypeScript settings with tsconfig.json for better development experience.
2025-06-19 15:38:45 +08:00
Marvin Zhang
38c3a7534a fix: unable to select all options in some dropdown 2025-06-18 22:33:04 +08:00
Marvin Zhang
09cfe37272 fix: unable to sync files to worker nodes when running tasks 2025-06-18 21:50:53 +08:00
Marvin Zhang
22497d22f7 fix: unable to filter tasks issue 2025-06-17 09:28:25 +08:00
Marvin Zhang
c06a9c6052 feat: update AutoProbe service and interfaces to support AutoProbeV2 2025-06-16 22:54:33 +08:00
Marvin Zhang
24452bff23 refactor: extract token retrieval logic into GetAPITokenFromContext utility function 2025-06-16 21:49:27 +08:00
Marvin Zhang
6d65cc13a2 feat: add aggregation methods to ModelService for enhanced data retrieval 2025-06-15 18:03:35 +08:00
Marvin Zhang
ab5e65f2fb chore: update dependencies to latest versions across modules 2025-06-15 18:03:13 +08:00
Marvin Zhang
c4a469b440 feat: enhance SystemDetail layout and improve customization form with scrollbar 2025-06-15 17:16:32 +08:00
Marvin Zhang
c3230fdbe1 refactor: streamline database handling and remove unused options 2025-06-13 21:53:23 +08:00
Marvin Zhang
335050e72e refactor: improve schedule service initialization, validation, and error handling 2025-06-13 21:53:08 +08:00
Marvin Zhang
0bd2b62792 feat: add DTOs for models and update project spiders type to array 2025-06-12 19:58:21 +08:00
Marvin Zhang
fe79cbb1a5 feat: add custom label template for active navigation item in DetailLayout 2025-06-12 10:31:26 +08:00
Marvin Zhang
79b7e074e1 refactor: update filter parameter in API requests and improve component structure 2025-06-12 00:17:40 +08:00
Marvin Zhang
3552b7805b refactor: remove unused javascript-time-ago types and enhance node sorting logic 2025-06-11 22:30:46 +08:00
Marvin Zhang
622dce51c3 fix: goroutine cleanup and error handling during shutdown 2025-06-11 22:30:37 +08:00
Marvin Zhang
ca069cb570 feat: implement navigation list management in store and detail layout 2025-06-11 18:04:51 +08:00
Marvin Zhang
de69c2b01b refactor: replace activeNodes with allList in dependency dialogs and related components 2025-06-11 13:34:08 +08:00
Marvin Zhang
1f3a9586b3 feat: enhance data model associations and remove unused components 2025-06-09 23:00:47 +08:00
Marvin Zhang
c48d74bf0e feat: add active nodes management and project retrieval functionality 2025-06-09 10:50:33 +08:00
Marvin Zhang
975c601f38 refactor: simplify list retrieval methods and enhance error handling 2025-06-09 10:21:00 +08:00
Marvin Zhang
8a00af115c feat: refactor to remove unused 'getAllList' methods and related state properties 2025-06-09 10:20:39 +08:00
Marvin Zhang
7b3f431ebe Merge remote-tracking branch 'origin/develop' into develop 2025-06-06 15:00:13 +08:00
Marvin Zhang
8d32d54fe8 feat: add development and index generation run configurations; update SVG files and improve icon styles 2025-06-06 14:59:59 +08:00
Marvin Zhang
805ef38a87 feat: add translation for AutoProbe menu item in router 2025-06-03 13:10:45 +08:00
Marvin Zhang
2aefdd3ee1 feat: add AI assistant SVG icon and integrate it into the header with a toggle button 2025-05-30 23:27:47 +08:00
Marvin Zhang
1cb5133fe0 feat: unhide the AutoProbe route in the navigation menu 2025-05-29 22:24:52 +08:00
Marvin Zhang
062077a9bd feat: restructure user parameters for PostUser and simplify email validation; refactor UserAvatar styles and improve user name handling 2025-05-29 22:16:27 +08:00
Marvin Zhang
57348da3f2 feat: implement auto naming for URL input and enhance data fetching with debounce in AutoProbe components 2025-05-28 18:05:23 +08:00
Marvin Zhang
14027f1c0c feat: enhance AutoProbe state management with page pattern and data handling 2025-05-26 11:28:56 +08:00
Marvin Zhang
0f80639a96 feat: add OpenRouter support and refactor LLM provider imports
- Introduced OpenRouter as a new LLM provider in the system.
- Updated LLM provider imports from '@/utils/ai' to '@/utils/llm' for consistency.
- Added new SVG icon for OpenRouter in the assets directory.
- Enhanced AutoProbe interfaces to accommodate new pattern structures.
2025-05-25 23:13:37 +08:00
Marvin Zhang
0022b5ae40 feat: add display configuration options and enhance element filtering in AutoProbeResultsPreview component
- Introduced showLabel and focusMode options for better control over element display.
- Updated page element filtering logic to accommodate new focus mode functionality.
- Added translations for new display configuration options in English and Chinese.
- Included csstype dependency for improved type management in styles.
2025-05-21 00:03:04 +08:00
Marvin Zhang
9e56ae7b00 refactor: rename overlayRef to screenshotRef and update related scaling logic in AutoProbeResultsPreview component 2025-05-20 14:29:55 +08:00
Marvin Zhang
4857d22cd7 feat: add activeNavItem prop and enhance page element filtering in AutoProbe components 2025-05-19 18:07:01 +08:00
Marvin Zhang
50c4bc3750 feat: add viewport selection and run on create options in AutoProbeForm component 2025-05-19 17:32:40 +08:00
Marvin Zhang
da024f7b6f refactor: remove unused props definition in ChatMessageList component 2025-05-19 09:11:09 +08:00
Marvin Zhang
f1f74d8a6a feat: update AutoProbeResultsContainer and related interfaces for improved preview functionality
- Refactored AutoProbeResultsContainer to utilize new previewResult structure and updated API calls.
- Enhanced element styling and dynamic behavior for page elements in the preview.
- Introduced getIconByPageElementType utility for better icon management based on element type.
- Updated AutoProbe interfaces to reflect changes in data structure, including renaming PagePreview to PagePreviewResult and modifying page_elements type.
- Added rowKey functionality in AutoProbeList for improved table data handling.
2025-05-18 23:03:02 +08:00
Marvin Zhang
7afc4ba4c9 feat: enhance internationalization and update AutoProbe interfaces
- Updated Chinese translation for 'AutoProbe' to include its native term.
- Modified AutoProbe interfaces to improve data structure, including changes to page_data and the addition of new properties in AutoProbeTaskResult.
- Enhanced SidebarItem component to conditionally display badge values based on item properties.
2025-05-18 18:53:43 +08:00
Marvin Zhang
07a509c198 feat: implement preview functionality in AutoProbeResultsContainer with enhanced state management and new interfaces
- Added preview feature to display page screenshots and item coordinates.
- Introduced new interfaces for viewport and element coordinates in autoprobe models.
- Improved iframe loading state management and dynamic styling for preview elements.
- Updated AutoProbeDetailTabPatterns to pass activeId prop for better data handling.
2025-05-16 23:38:13 +08:00
Marvin Zhang
c0b0cc9f7e feat: improve iframe loading state management and enhance styling in AutoProbeResultsContainer 2025-05-16 18:17:07 +08:00
Marvin Zhang
2ec4707510 feat: enhance AutoProbe components with improved styling and iframe sandboxing 2025-05-16 17:52:05 +08:00
Marvin Zhang
5d837cca7e feat: add iframe support for URL preview in AutoProbeResultsContainer 2025-05-16 17:17:34 +08:00
Marvin Zhang
6705ce2ded feat: simplify results visibility management in AutoProbeResultsContainer 2025-05-16 13:32:10 +08:00