21 Commits

Author SHA1 Message Date
Marvin Zhang
9f251f3ebe fix: enhance task cancellation logic with graceful termination and stuck task cleanup 2025-06-27 13:50:21 +08:00
Marvin Zhang
22497d22f7 fix: unable to filter tasks issue 2025-06-17 09:28:25 +08:00
Marvin Zhang
0bd2b62792 feat: add DTOs for models and update project spiders type to array 2025-06-12 19:58:21 +08:00
Marvin Zhang
79b7e074e1 refactor: update filter parameter in API requests and improve component structure 2025-06-12 00:17:40 +08:00
Marvin Zhang
1f3a9586b3 feat: enhance data model associations and remove unused components 2025-06-09 23:00:47 +08:00
Marvin Zhang
4f57d277e7 refactor: standardize timestamp fields and improve code clarity
- Updated timestamp fields across the codebase from `*_ts` to `*_at` for consistency and clarity.
- Renamed constants for node status from "on"/"off" to "online"/"offline" to better reflect their meanings.
- Enhanced validation and error handling in various components to ensure data integrity.
- Refactored test cases to align with the new naming conventions and improve readability.
2025-04-21 18:13:22 +08:00
Marvin Zhang
0a3c706dca feat: enhance task and spider result handling in API
- Updated the description for the GetSpiderResults route to clarify the type of data returned.
- Added a new route for GetTaskResults to retrieve scraped or crawled results data for tasks.
- Refactored the GetSpiderResults and GetTaskResults functions to improve query handling and error management.
- Ensured consistent usage of MongoDB query methods across the controllers for better maintainability.
2025-04-10 23:18:17 +08:00
Marvin Zhang
f86173c973 refactor: standardize filter handling across controllers
- Replaced "conditions" parameter with "filter" in various controller methods to unify the filtering approach.
- Updated related functions to utilize the new filter parameter, enhancing consistency in query handling.
- Improved overall code readability and maintainability by aligning naming conventions and refactoring filter-related logic.
2025-03-26 22:08:01 +08:00
Marvin Zhang
931a36c8bf refactor: enhance parameter descriptions and standardize model fields
- Added descriptive annotations to various parameters across controllers to improve API documentation clarity.
- Standardized field definitions in models by including descriptions for better understanding of their purpose.
- Updated validation patterns for ID fields to ensure consistency and improve data integrity.
- Enhanced overall code readability and maintainability by aligning naming conventions and adding necessary comments.
2025-03-19 13:39:02 +08:00
Marvin Zhang
98843fdbac refactor: standardize sorting logic across controllers
- Introduced a consistent approach for sorting in multiple controller methods by utilizing GetSortOptionFromString for improved clarity and maintainability.
- Updated the GetListParams structure to ensure sorting is handled uniformly across different entities.
- Enhanced error handling for sorting parameters to provide clearer feedback on invalid formats.
- Improved comments for better understanding of sorting logic and its integration within the controller methods.
2025-03-17 17:14:08 +08:00
Marvin Zhang
43d1c7692b refactor: standardize response types across controllers
- Updated multiple controller methods to return VoidResponse instead of generic Response[any].
- Consolidated error handling to utilize GetErrorVoidResponse for consistent error responses.
- Enhanced parameter handling in export and file management functions for improved clarity and maintainability.
- Refactored health check and login/logout methods to align with new response structure.
- Improved overall consistency in response formatting across various endpoints.
2025-03-16 22:25:13 +08:00
Marvin Zhang
23cad00d92 refactor: streamline controller methods and enhance parameter handling
- Refactored multiple controller methods to accept structured parameters for improved clarity and maintainability.
- Consolidated error handling and response generation across various endpoints.
- Updated function signatures to eliminate unnecessary context parameters and enhance type safety.
- Improved consistency in response formatting and error handling across controllers.
- Enhanced file handling methods to support multipart file uploads and directory operations more effectively.
2025-03-13 17:37:30 +08:00
Marvin Zhang
d6badb533d feat: enhance API routing and OpenAPI documentation support
- Introduced a new OpenAPI wrapper using Fizz for improved API documentation
- Refactored base controller to support more flexible route handling
- Added dynamic route registration with OpenAPI metadata
- Implemented generic response types for consistent API responses
- Updated router initialization to support OpenAPI documentation endpoint
- Improved route and resource naming utilities
- Migrated existing controllers to use new routing and response mechanisms
2025-03-11 23:45:06 +08:00
Marvin Zhang
dc59599509 refactor: remove db module and update imports to core/mongo
- Deleted the db module, consolidating database-related functionality into the core/mongo package for better organization and maintainability.
- Updated all import paths across the codebase to replace references to the removed db module with core/mongo.
- Cleaned up unused code and dependencies, enhancing overall project clarity and reducing complexity.
- This refactor improves the structure of the codebase by centralizing database operations and simplifying module management.
2024-12-25 10:28:21 +08:00
Marvin Zhang
3276083994 refactor: replace apex/log with structured logger across multiple services
- Replaced all instances of apex/log with a structured logger interface in various services, including Api, Server, Config, and others, to enhance logging consistency and context.
- Updated logging calls to utilize the new logger methods, improving error tracking and service monitoring.
- Added logger initialization in services and controllers to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Removed unused apex/log imports and cleaned up related code for better maintainability.
2024-12-24 19:11:19 +08:00
Marvin Zhang
41e973a4d3 fix: misaligned nodes when running tasks from a schedule through enhancement by adding new route and handler for running schedules
- Introduced a new POST endpoint '/:id/run' in the router to trigger the execution of a schedule.
- Implemented the PostScheduleRun function to handle the execution logic, including validation of input parameters and scheduling tasks.
- Refactored the PostSpiderRun function to streamline the scheduling process by directly calling the admin service.
- Updated the FsFileInfo struct to clarify the children field description.
- Modified the Task model to make NodeIds field optional in JSON serialization.
2024-12-21 15:33:07 +08:00
Marvin Zhang
a3b286558b refactor: consolidated configs 2024-11-18 16:48:09 +08:00
Marvin Zhang
e33fcfc150 refactor: renamed files and services 2024-11-05 11:15:27 +08:00
Marvin Zhang
4da8976228 refactor: code cleanup 2024-07-06 14:15:14 +08:00
Marvin Zhang
dc21bce11f feat: added modules 2024-06-14 15:59:48 +08:00
Marvin Zhang
0b67fd9ece feat: added modules 2024-06-14 15:42:50 +08:00