Commit Graph

62 Commits

Author SHA1 Message Date
Marvin Zhang
a1f870715f feat: add GET endpoint for retrieving user's token list
Implemented a new GetTokenList handler in the token controller to:
- Fetch tokens created by the current user
- Support pagination and filtering
- Return tokens with total count
- Handle cases with no documents gracefully
2025-02-27 16:39:51 +08:00
Marvin Zhang
6048d4eeb8 fix: missing routes in get me api
https://github.com/crawlab-team/crawlab/issues/1550
2025-02-27 13:33:48 +08:00
Marvin Zhang
3237923f02 fix: prevent unnecessary error handling in export download endpoint
Adds an early return after handling bad request errors in the GetExportDownload function to avoid unnecessary error processing and improve error handling clarity.
2025-02-17 16:27:39 +08:00
Marvin Zhang
8d8b47e474 refactor: streamline file service retrieval and enhance spider template handling
- Replaced direct calls to getBaseFileFsSvc with a new method fs.GetBaseFileFsSvc in base_file.go for improved clarity and maintainability.
- Introduced SpiderTemplateService interface and implemented registry service for managing spider templates, enhancing template handling in the spider controller.
- Added template-related fields to the Spider model to support template functionality.
- Created utility functions for string case conversions in utils/string.go to facilitate consistent formatting across the codebase.
- Updated environment configuration to retrieve the Python path dynamically, improving flexibility in the task runner's setup.
2025-01-06 18:09:45 +08:00
Marvin Zhang
47094b8e64 refactor: update setting routes and enhance dependency management
- Changed route parameter from ':id' to ':key' in settings-related routes for better clarity and consistency.
- Updated GetSetting, PostSetting, and PutSetting functions to use the new ':key' parameter.
- Introduced IsAutoInstallEnabled method in DependencyInstallerService to check auto-installation status.
- Enhanced the task runner to check if auto installation is enabled before proceeding with dependency installation.
- Improved initialization of settings data in the system service, ensuring proper insertion of initial settings.
2025-01-01 22:37:44 +08:00
Marvin Zhang
54800974eb feat: refactor system info retrieval and enhance logo output
- Replaced viper calls with utility functions in GetSystemInfo to improve code clarity and maintainability.
- Added a new system.go file with utility functions for retrieving system version and edition information.
- Enhanced PrintLogoWithWelcomeInfo to include detailed system information, improving user experience during server startup.
- Updated output formatting for better readability and consistency in welcome messages.
2024-12-29 19:01:23 +08:00
Marvin Zhang
dc59599509 refactor: remove db module and update imports to core/mongo
- Deleted the db module, consolidating database-related functionality into the core/mongo package for better organization and maintainability.
- Updated all import paths across the codebase to replace references to the removed db module with core/mongo.
- Cleaned up unused code and dependencies, enhancing overall project clarity and reducing complexity.
- This refactor improves the structure of the codebase by centralizing database operations and simplifying module management.
2024-12-25 10:28:21 +08:00
Marvin Zhang
3276083994 refactor: replace apex/log with structured logger across multiple services
- Replaced all instances of apex/log with a structured logger interface in various services, including Api, Server, Config, and others, to enhance logging consistency and context.
- Updated logging calls to utilize the new logger methods, improving error tracking and service monitoring.
- Added logger initialization in services and controllers to ensure proper logging setup.
- Improved error handling and logging messages for better clarity during service operations.
- Removed unused apex/log imports and cleaned up related code for better maintainability.
2024-12-24 19:11:19 +08:00
Marvin Zhang
41e973a4d3 fix: misaligned nodes when running tasks from a schedule through enhancement by adding new route and handler for running schedules
- Introduced a new POST endpoint '/:id/run' in the router to trigger the execution of a schedule.
- Implemented the PostScheduleRun function to handle the execution logic, including validation of input parameters and scheduling tasks.
- Refactored the PostSpiderRun function to streamline the scheduling process by directly calling the admin service.
- Updated the FsFileInfo struct to clarify the children field description.
- Modified the Task model to make NodeIds field optional in JSON serialization.
2024-12-21 15:33:07 +08:00
Marvin Zhang
c897fb58e4 refactor: streamline error handling and improve HTTP request management
- Removed the print flag from handleError function, simplifying error logging based on the development environment.
- Introduced a new performRequest function for standardized HTTP requests with JSON bodies, enhancing code reusability.
- Updated SendIMNotification and related functions to utilize the new RequestParam type for better clarity and consistency.
- Normalized HTTP request paths in the createHttpRequest method to ensure correct URL formatting.
- Added detailed error logging for JSON unmarshaling failures in syncFiles method.
- Introduced a NewHttpClient function to create HTTP clients with customizable timeouts.
2024-12-21 11:27:58 +08:00
Marvin Zhang
6b78afe95c fix: added default fields 2024-12-11 20:05:03 +08:00
Marvin Zhang
858e5c2b89 fix: unable to start api 2024-11-22 21:19:17 +08:00
Marvin Zhang
a3b286558b refactor: consolidated configs 2024-11-18 16:48:09 +08:00
Marvin Zhang
459d94ad96 feat: optimized dependencies for spider 2024-11-15 18:40:06 +08:00
Marvin Zhang
4fd1ff8fcf refactor: optimized field names 2024-11-14 11:22:37 +08:00
Marvin Zhang
c6ca5cb2a9 feat: optimized api response code 2024-11-13 17:31:05 +08:00
Marvin Zhang
86669be84b fix: disallow deleting for root admin users 2024-11-13 17:02:51 +08:00
Marvin Zhang
11e17c8a86 fix: user controller issues 2024-11-13 16:43:33 +08:00
Marvin Zhang
86b03307dc feat: updated controllers 2024-11-13 15:33:47 +08:00
Marvin Zhang
f0bb927b0b feat: updated permission management 2024-11-12 18:14:12 +08:00
Marvin Zhang
a233075779 refactor: restructured code 2024-11-10 17:23:32 +08:00
Marvin Zhang
e33fcfc150 refactor: renamed files and services 2024-11-05 11:15:27 +08:00
Marvin Zhang
73674832b8 feat: optimized dependency api 2024-11-04 00:16:42 +08:00
Marvin Zhang
71f0a210ba refactor: fixed dependency errors 2024-11-01 15:19:48 +08:00
Marvin Zhang
68ba84a4e7 refactor: optimized node communication 2024-11-01 15:19:48 +08:00
Marvin Zhang
d9b327de17 refactor: code cleanup 2024-11-01 15:19:48 +08:00
Marvin Zhang
f0f0aa9814 refactor: updated task related grpc proto 2024-11-01 15:19:48 +08:00
Marvin Zhang
1c03cb3e5c refactor: code cleanup 2024-10-29 12:59:45 +08:00
Marvin Zhang
d7ac9ebb1f test: updated test cases 2024-10-20 18:45:54 +08:00
Marvin Zhang
fa2b024906 fix: delete spider issue 2024-10-20 18:21:06 +08:00
Marvin Zhang
3c53fde7e2 refactor: code cleanup 2024-10-20 17:22:35 +08:00
Marvin Zhang
418af7c215 refactor: code cleanup 2024-10-20 17:17:08 +08:00
Marvin Zhang
1b852fb96a refactor: code cleanup 2024-10-18 15:03:32 +08:00
Marvin Zhang
1646df7136 feat: added nav for task detail 2024-10-17 21:10:19 +08:00
Marvin Zhang
b2f50dbcea fix: results data display issue 2024-10-17 20:54:25 +08:00
Marvin Zhang
2bb697446b feat: optimized spider, task pages 2024-10-17 20:11:30 +08:00
Marvin Zhang
4f52936ad8 refactor: integrated database services into task data insert 2024-10-08 18:41:36 +08:00
Marvin Zhang
1197c05316 🚀 feat: add IsDefault field to DatabaseV2 model
🔨 refactor: modify GetDatabaseList, GetDatabaseById, PostDatabaseTestConnection, and GetDatabaseMetadata functions
🐞 fix: GetDatabaseById and GetMetadata functions in MongoService
🔧 chore: add GetDefaultDatabase and GetDatabaseById functions to utils file
2024-08-06 15:35:36 +08:00
Marvin Zhang
421a06930c refactor: Update models to use DatabaseV2 instead of DataSourceV2 2024-08-05 16:07:19 +08:00
Marvin Zhang
e57655c67e refactor: Update models to use DatabaseV2 instead of DataSourceV2 2024-08-05 13:14:57 +08:00
Marvin Zhang
e2cb99e56a feat: Update models to use DatabaseV2 instead of DataSourceV2
The code changes update the models and related functions to use the new DatabaseV2 struct instead of the deprecated DataSourceV2 struct. This change ensures consistency and clarity in the codebase.
2024-08-04 16:58:46 +08:00
Marvin Zhang
a7da3a0442 feat: Update notification trigger patterns and add alert trigger 2024-08-03 19:43:11 +08:00
Marvin Zhang
aca0c0ebce refactor: removed unnecessary code 2024-07-11 12:45:29 +08:00
Marvin Zhang
f2f298d8f8 refactor: removed unnecessary code 2024-07-11 12:45:20 +08:00
Marvin Zhang
4da8976228 refactor: code cleanup 2024-07-06 14:15:14 +08:00
Marvin Zhang
68307d250e feat: added git diff
https://github.com/crawlab-team/crawlab/issues/1485
2024-06-30 16:10:51 +08:00
Marvin Zhang
1eb81b3f43 feat: finalized git progress for clone, pull, push
https://github.com/crawlab-team/crawlab/issues/1485
2024-06-29 22:29:34 +08:00
Marvin Zhang
8ebfda7377 feat: added multi-spider git
https://github.com/crawlab-team/crawlab/issues/1485
2024-06-29 15:29:24 +08:00
Marvin Zhang
0b3fbcfed8 feat: added multi-spider in git
https://github.com/crawlab-team/crawlab/issues/1485
2024-06-28 18:52:49 +08:00
Marvin Zhang
8836206835 fix: missing notification settings controller 2024-06-26 23:47:27 +08:00