Commit Graph

75 Commits

Author SHA1 Message Date
Marvin Zhang
c937e0f45f refactor: enhance Spider model and string utility functions
- Updated the Spider model to introduce a new SpiderTemplateParams struct for improved template handling.
- Refactored string utility functions in utils/string.go to include a new replaceChars function, streamlining character replacement across multiple functions.
- Enhanced ToSnakeCase and ToKebabCase functions to utilize the new replaceChars function for better maintainability and readability.
- Added splitStringWithQuotes function to facilitate string manipulation with quotes, improving overall utility in string processing.
2025-01-07 13:21:16 +08:00
Marvin Zhang
8d8b47e474 refactor: streamline file service retrieval and enhance spider template handling
- Replaced direct calls to getBaseFileFsSvc with a new method fs.GetBaseFileFsSvc in base_file.go for improved clarity and maintainability.
- Introduced SpiderTemplateService interface and implemented registry service for managing spider templates, enhancing template handling in the spider controller.
- Added template-related fields to the Spider model to support template functionality.
- Created utility functions for string case conversions in utils/string.go to facilitate consistent formatting across the codebase.
- Updated environment configuration to retrieve the Python path dynamically, improving flexibility in the task runner's setup.
2025-01-06 18:09:45 +08:00
Marvin Zhang
9bdb0c969f refactor: remove default_version field from DependencyConfig
- Eliminated the default_version field from the DependencyConfig struct in dependency_config.go to streamline the configuration model.
- This change simplifies the dependency management process and aligns with recent updates in the application structure.
2024-12-30 15:14:21 +08:00
Marvin Zhang
34455dc47c feat: enhance DependencyConfig model with additional fields
- Added TotalDependencies and SearchReady fields to the DependencyConfig struct for improved dependency management.
- Retained the Setup field for backward compatibility while ensuring proper JSON and BSON handling.
2024-12-23 15:18:04 +08:00
Marvin Zhang
e44b416e34 feat: enhance model base service with BSON ID normalization
- Added utility function to normalize BSON ObjectId in query parameters for gRPC methods.
- Updated GetOne, GetMany, DeleteOne, DeleteMany, UpdateOne, UpdateMany, ReplaceOne, and UpsertOne methods to utilize the new normalization function.
- Introduced new DependencyConfigSetup model instance in base model definitions for improved structure.
2024-12-21 22:07:54 +08:00
Marvin Zhang
41e973a4d3 fix: misaligned nodes when running tasks from a schedule through enhancement by adding new route and handler for running schedules
- Introduced a new POST endpoint '/:id/run' in the router to trigger the execution of a schedule.
- Implemented the PostScheduleRun function to handle the execution logic, including validation of input parameters and scheduling tasks.
- Refactored the PostSpiderRun function to streamline the scheduling process by directly calling the admin service.
- Updated the FsFileInfo struct to clarify the children field description.
- Modified the Task model to make NodeIds field optional in JSON serialization.
2024-12-21 15:33:07 +08:00
Marvin Zhang
e77d4cdd31 feat: updated dependency management 2024-12-17 19:32:16 +08:00
Marvin Zhang
79c1d5d14b feat: updated dependency config setup 2024-12-16 21:44:03 +08:00
Marvin Zhang
c5c08dfba6 feat: added dependency config setup (wip) 2024-12-15 23:09:10 +08:00
Marvin Zhang
25580b4694 feat: prepare for dependency lang install/setup 2024-12-14 22:18:22 +08:00
Marvin Zhang
298420b0a6 feat: updated the way syncing pypi projects 2024-12-14 21:47:11 +08:00
Marvin Zhang
1fe74fa8a5 fix: optimized node runners calculation 2024-12-11 20:43:40 +08:00
Marvin Zhang
dfaf18fbd6 feat: added notification test 2024-11-14 16:44:49 +08:00
Marvin Zhang
4fd1ff8fcf refactor: optimized field names 2024-11-14 11:22:37 +08:00
Marvin Zhang
86b03307dc feat: updated controllers 2024-11-13 15:33:47 +08:00
Marvin Zhang
f0bb927b0b feat: updated permission management 2024-11-12 18:14:12 +08:00
Marvin Zhang
a233075779 refactor: restructured code 2024-11-10 17:23:32 +08:00
Marvin Zhang
a0989d36db feat: optimized dependency logic 2024-11-05 18:21:52 +08:00
Marvin Zhang
10bb511c5f feat: updated dependency handler logic 2024-11-05 11:40:12 +08:00
Marvin Zhang
e33fcfc150 refactor: renamed files and services 2024-11-05 11:15:27 +08:00
Marvin Zhang
fbf8e5f9f3 feat: optimizing dependency services including grpc, api 2024-11-04 17:45:34 +08:00
Marvin Zhang
73674832b8 feat: optimized dependency api 2024-11-04 00:16:42 +08:00
Marvin Zhang
68ba84a4e7 refactor: optimized node communication 2024-11-01 15:19:48 +08:00
Marvin Zhang
1b852fb96a refactor: code cleanup 2024-10-18 15:03:32 +08:00
Marvin Zhang
1646df7136 feat: added nav for task detail 2024-10-17 21:10:19 +08:00
Marvin Zhang
2f8f38f145 feat: optimized spider, task pages 2024-10-17 20:11:38 +08:00
Marvin Zhang
eb56598d93 refactor: code cleanup 2024-10-16 20:13:24 +08:00
Marvin Zhang
785424b1ac refactor: combined system collections 2024-10-14 17:56:39 +08:00
Marvin Zhang
e65b846427 feat: added system collections 2024-10-14 16:16:03 +08:00
Marvin Zhang
045fc42991 feat: added field to spider 2024-10-11 19:06:31 +08:00
Marvin Zhang
4f52936ad8 refactor: integrated database services into task data insert 2024-10-08 18:41:36 +08:00
Marvin Zhang
0ed2150901 feat: added performance monitoring for elasticsearch 2024-10-07 17:19:09 +08:00
Marvin Zhang
e15c3c9b4f feat: added performance monitoring for database 2024-10-05 14:46:38 +08:00
Marvin Zhang
6986a10246 feat: updated password handling logic 2024-08-07 18:05:35 +08:00
Marvin Zhang
1197c05316 🚀 feat: add IsDefault field to DatabaseV2 model
🔨 refactor: modify GetDatabaseList, GetDatabaseById, PostDatabaseTestConnection, and GetDatabaseMetadata functions
🐞 fix: GetDatabaseById and GetMetadata functions in MongoService
🔧 chore: add GetDefaultDatabase and GetDatabaseById functions to utils file
2024-08-06 15:35:36 +08:00
Marvin Zhang
421a06930c refactor: Update models to use DatabaseV2 instead of DataSourceV2 2024-08-05 16:07:19 +08:00
Marvin Zhang
e57655c67e refactor: Update models to use DatabaseV2 instead of DataSourceV2 2024-08-05 13:14:57 +08:00
Marvin Zhang
e2cb99e56a feat: Update models to use DatabaseV2 instead of DataSourceV2
The code changes update the models and related functions to use the new DatabaseV2 struct instead of the deprecated DataSourceV2 struct. This change ensures consistency and clarity in the codebase.
2024-08-04 16:58:46 +08:00
Marvin Zhang
7fe770ae9d feat: Update notification trigger patterns and add alert trigger
The code changes modify the notification constants and models to update the trigger patterns for tasks and nodes. Additionally, a new trigger for alerts is added to the constants. This change allows for more flexible matching of notification triggers.
2024-07-31 15:58:41 +08:00
Marvin Zhang
86a7beb9e2 add: add AlertId field to NotificationSettingV2 model 2024-07-30 17:33:42 +08:00
Marvin Zhang
5b67717ebe feat(models): add HasMetricTarget and LastingSeconds fields to NotificationAlertV2 2024-07-30 15:46:02 +08:00
Marvin Zhang
779d831a6d feat: Add NotificationAlertV2 model
Add NotificationAlertV2 model to handle notification alerts in the system. This model includes fields for name, description, enabled status, metric target ID, metric name, operator, target value, and level.
2024-07-30 12:35:18 +08:00
Marvin Zhang
ea5ec117cc refactor: Remove unused code for SMTP authentication and add Gmail OAuth2 authentication support 2024-07-27 19:29:43 +08:00
Marvin Zhang
e8bdfd228d refactor: Update models to include sender email, name, and mail recipients for notification requests 2024-07-27 16:06:26 +08:00
Marvin Zhang
8d5a0b0c0e refactor: Add Outlook authentication support for sending emails 2024-07-26 18:15:15 +08:00
Marvin Zhang
0e2170f644 refactor: Update models to include sender email, name, and mail recipients for notification requests 2024-07-26 15:33:58 +08:00
Marvin Zhang
96b89d05b8 refactor: Update NotificationChannelV2 model to include Telegram notification settings 2024-07-25 22:58:39 +08:00
Marvin Zhang
70cc86c4d3 refactor: Update NotificationChannelV2 model to include Telegram notification settings 2024-07-25 14:40:37 +08:00
Marvin Zhang
7f9fac0fc0 feat: updated notification settings and channels 2024-07-23 17:24:40 +08:00
Marvin Zhang
9ffdd3f1cd refactor: Remove unused code and update models and functions for notification channels and settings 2024-07-23 12:22:59 +08:00