Commit Graph

69 Commits

Author SHA1 Message Date
mtayfur
bc5611a6d0 refactor(memory_system): simplify embedding normalization, memory retrieval, and response decoding logic
Redundant checks and exception handling are removed to streamline code paths, making embedding normalization, user memory retrieval, and LLM response decoding more concise and maintainable while preserving error propagation at higher levels.
2025-11-24 15:59:55 +03:00
mtayfur
3b84f64392 refactor: make skip detection and embedding operations fully async for improved concurrency
Skip detection and embedding-related methods are now asynchronous, allowing
non-blocking execution and better concurrency; embedding function wrappers
and initialization routines are updated to support async/await, and
shared skip detector caching is adapted accordingly. These changes are
necessary to ensure compatibility with async embedding functions, prevent
blocking the event loop, and improve scalability and responsiveness in
high-concurrency environments.
2025-11-24 15:45:03 +03:00
mtayfur
960f8ce4a9 feat(memory_system): introduce "Intermediate" status level for more granular status emission control
Adds an "Intermediate" status level between "Basic" and "Detailed" to provide
summaries and key details, updates all relevant mappings and usages to support
this new level, and adjusts default verbosity for status messages to improve
clarity and reduce noise for users who do not require full diagnostics.
2025-11-15 21:52:25 +03:00
mtayfur
6ae99d4778 refactor(memory): remove redundant valve options and clarify reranking controls
Eliminate unused or redundant valve options such as max_message_chars,
relaxed_semantic_threshold_multiplier, and enable_llm_reranking to
streamline configuration, clarify that llm_reranking_trigger_multiplier
disables reranking when set to 0.0, and update documentation and code to
reflect these changes for improved maintainability and user clarity.
2025-11-09 16:52:56 +03:00
mtayfur
c77237a651 feat(memory_system): make skip category margin configurable via Filter settings
Allowing skip_category_margin to be set in Filter enables dynamic
adjustment of skip detection sensitivity, improving flexibility for
different deployment scenarios.
2025-11-09 15:43:30 +03:00
mtayfur
aaaf848c1e refactor(memory_system): remove "Intermediate" status level and update status emission logic
Eliminates the "Intermediate" status level for message verbosity, consolidating to "Basic" and "Detailed" for clarity and simplicity; updates all related mappings, enums, and status emission calls to reflect this streamlined approach, reducing confusion and improving maintainability.
2025-11-09 14:31:27 +03:00
mtayfur
cfc4dca6fc feat(memory_system): expand skip and personal category descriptions and improve skip state caching logic
Category descriptions are made more comprehensive and precise to improve classification accuracy, the skip margin is adjusted for better thresholding, and skip state caching is now performed before early returns to ensure consistent cache updates regardless of skip conditions. The user message extraction logic is also improved for robustness.
2025-11-07 17:43:58 +03:00
mtayfur
ff6052fd41 feat(memory_system): add configurable status message verbosity levels
Introduce status_emit_level to control status message verbosity, allowing
users to select Basic, Intermediate, or Detailed output; refactor all
status emission calls to respect this setting, improving clarity and
customizability of system feedback for different user needs.
2025-11-07 00:19:50 +03:00
mtayfur
eca65fef43 fix: increase SKIP_CATEGORY_MARGIN for skip category classification to 0.15
The SKIP_CATEGORY_MARGIN constant is raised from 0.10 to 0.15 to make skip
category classification more selective, reducing false positives by requiring
a higher margin above personal similarity.
2025-11-04 01:45:26 +03:00
mtayfur
8997f71f05 refactor(memory_system): remove excessive try/except and input validation, streamline async operations, and add skip state cache
Removes redundant try/except blocks and input validation in several
methods to simplify logic and improve readability. Moves error handling
to higher levels where appropriate. Adds a skip state cache to track
when memory operations should be skipped, improving efficiency by
avoiding repeated skip checks. Cleans up batch operation execution and
cache clearing to include the new skip state. These changes reduce
unnecessary code complexity and improve maintainability, while also
optimizing memory operation flow and cache management.
2025-10-28 17:51:09 +03:00
mtayfur
8ced9aace5 feat(skip-detection): expand and reorganize non-personal category descriptions for improved coverage and clarity
refactor(skip-detection): condense and generalize personal category descriptions for maintainability and easier updates

fix(constants): lower SKIP_CATEGORY_MARGIN from 0.20 to 0.10 to reduce false positives in skip classification

refactor(skip-detection): clarify logging for skip/allow decisions and threshold calculation for better traceability

refactor: use consistent double-quote style for operator and code-ending character lists

fix(filter): ensure embeddings are normalized before use in skip detector to improve classification accuracy

Non-personal category descriptions are now more comprehensive and organized by topic, improving the classifier's ability to detect impersonal content. Personal category descriptions are condensed into broader, clearer categories, making them easier to maintain and update. The SKIP_CATEGORY_MARGIN is reduced to make the skip detector less aggressive, addressing previous false positives. Logging is improved for easier debugging and understanding of skip decisions. Embedding normalization is enforced to ensure consistent similarity calculations.
2025-10-28 17:23:20 +03:00
mtayfur
577f6d6406 Enhance skip detection logic with additional checks for structured content and code indicators v1.1.0 2025-10-28 03:44:26 +03:00
mtayfur
fe3c47f6e4 Update README and logging messages for model configuration clarity 2025-10-28 03:03:11 +03:00
M. Tayfur
551b0c571b ♻️ Skip detection now binary (#4) 2025-10-28 02:35:29 +03:00
M. Tayfur
e6e3f7ab99 Merge pull request #3 from GlisseManTV/dev
Way to use current model instead of dedicated model.
2025-10-27 23:39:06 +03:00
iTConsult4Care
cca8079b94 Fix formatting and add metadata to memory_system.py 2025-10-27 21:14:35 +01:00
GlissemanTV
55a8c70bac add suppport of current chat model 2025-10-27 21:06:35 +01:00
GlissemanTV
e12aa7b776 Merge branch 'dev' of https://github.com/GlisseManTV/openwebui-memory-system into dev 2025-10-27 21:05:26 +01:00
mtayfur
b5a4872096 📝 (memory_system): clarify and strengthen intent filtering and memory consolidation guidelines
Expand and clarify the "Filter for Intent" rule to ensure only direct,
personally significant facts are stored, explicitly excluding messages
where the user's primary intent is instructional, technical, or
analytical. Update processing and decision frameworks to reinforce
selectivity based on user intent. Revise and annotate examples to
demonstrate correct application of the new rules, making it clear that
requests for advice, recommendations, or technical tasks are ignored.
These changes improve the precision and reliability of memory
consolidation, reducing the risk of storing irrelevant or transient
information.
2025-10-27 00:57:26 +03:00
mtayfur
3f9b4c6d48 ♻️ (memory_system): refactor skip detection and add semantic deduplication
- Unify skip detection to a binary classifier (personal vs non-personal)
 for improved maintainability and clarity. Remove multiple technical/
 instruction/translation/etc. categories and consolidate into
 NON_PERSONAL and PERSONAL.
- Adjust skip detection margin for more precise classification.
- Add semantic deduplication for memory operations using embedding
 similarity, preventing duplicate memory creation and updates.
- Normalize and validate embedding dimensions for robustness.
- Add per-user async locks to prevent race conditions during memory
 consolidation.
- Refactor requirements.txt to remove version pinning for easier
 dependency management.
- Improve logging and error handling for embedding and deduplication
 operations.

These changes improve the reliability and accuracy of memory
classification and deduplication, reduce false positives in skip
detection, and prevent duplicate or conflicting memory operations in
concurrent environments. Dependency management is simplified for
compatibility.
2025-10-27 00:27:33 +03:00
mtayfur
bb1bd01222 ♻️ (memory_system.py): reformat code for consistency, readability, and maintainability
- Reorder and group imports for clarity and PEP8 compliance.
- Standardize string quoting and whitespace for consistency.
- Refactor long function signatures and dictionary constructions for better readability.
- Use double quotes for all string literals and dictionary keys.
- Improve formatting of multiline statements and function calls.
- Add or adjust line breaks to keep lines within recommended length.
- Reformat class and method docstrings for clarity.
- Use consistent indentation and spacing throughout the file.

These changes improve code readability, maintainability, and consistency, making it easier for future contributors to understand and modify the codebase. No functional logic is changed.
2025-10-27 00:20:35 +03:00
mtayfur
189c6d4226 🔧 (dev-check.sh, pyproject.toml, requirements.txt): add development tooling and configuration
Introduce a dev-check.sh script to automate code formatting and import
sorting using Black and isort. Add a pyproject.toml file to configure
Black and isort settings for consistent code style. Update
requirements.txt to include Black and isort as development dependencies
and remove version pinning for easier dependency management.

These changes streamline the development workflow, enforce code style
consistency, and make it easier for contributors to run formatting and
import checks locally.
2025-10-27 00:20:05 +03:00
iTConsult4Care
7630259ce1 Delete memory_system_ollama.py 2025-10-26 16:48:26 +01:00
GlissemanTV
a3a627339a Merge branch 'dev' of https://github.com/GlisseManTV/openwebui-memory-system into dev 2025-10-26 16:46:06 +01:00
iTConsult4Care
390747c1d1 Delete memory_system_ollama.py 2025-10-26 16:45:30 +01:00
iTConsult4Care
b6b4c5fde8 Add files via upload 2025-10-26 16:44:24 +01:00
iTConsult4Care
4328e4b79c Add current model usecase 2025-10-26 16:42:36 +01:00
GlissemanTV
fcae27e840 add current model usecase 2025-10-26 16:14:05 +01:00
GlissemanTV
1f779d86ec Merge branch 'dev' of https://github.com/GlisseManTV/openwebui-memory-system into dev 2025-10-26 16:12:12 +01:00
GlissemanTV
d07a853aeb add current model using case 2025-10-26 16:05:35 +01:00
GlissemanTV
89399f57cc add current model workflow with checkbox 2025-10-26 16:01:13 +01:00
mtayfur
c0bfb3927b Refactor memory creation guidelines for improved clarity and conciseness in contextual completeness section. 2025-10-19 05:13:45 +03:00
mtayfur
d05ed8a16e Update semantic retrieval thresholds in Constants class for improved accuracy 2025-10-19 04:57:36 +03:00
mtayfur
7e2209633d Refactor logger initialization in memory_system.py to use module name for better context in log messages. 2025-10-18 19:25:22 +03:00
mtayfur
505c443050 Update README.md to enhance clarity on privacy and cost considerations; restructure sections for better readability and add relevant details. 2025-10-15 14:33:33 +03:00
mtayfur
0726293446 Update README.md for improved clarity and accuracy; revise privacy notice, cache descriptions, and model support details. 2025-10-15 14:13:55 +03:00
mtayfur
e3709fe677 Refactor cache management in Filter class; reduce maximum cache entries and concurrent user caches for improved performance and clarity. Update cache management methods for consistency and better logging. 2025-10-15 14:05:01 +03:00
mtayfur
2deba4fb2c Refactor Filter class to use async for pipeline context setup; implement locking mechanism for shared skip detector cache to enhance concurrency safety. 2025-10-12 23:24:58 +03:00
mtayfur
849dd71a01 Refactor memory selection logic in LLMRerankingService for improved clarity; streamline response handling by directly using response.ids. 2025-10-12 23:03:36 +03:00
mtayfur
158f0d1983 Refactor memory operations in Filter class for improved readability and consistency; utilize statistics.median for score calculation and streamline operation details formatting. 2025-10-12 22:54:18 +03:00
mtayfur
2db2d3f2c8 Refactor SkipDetector to streamline skip detection logic and improve clarity; update method signature for better integration with memory system. 2025-10-12 21:44:51 +03:00
mtayfur
1390505665 Refactor SkipDetector to use a callable embedding function instead of SentenceTransformer; update requirements to remove unnecessary dependencies. v1.0.0 2025-10-10 15:11:41 +03:00
GlissemanTV
08155816ff add memory_system_ollama.py 2025-10-10 09:10:03 +02:00
mtayfur
840d4c59ca Refactor SkipDetector to use a callable embedding function instead of SentenceTransformer; update requirements to remove unnecessary dependencies. 2025-10-09 23:36:27 +03:00
mtayfur
5c0ca1f4ab Update model loading to use 'auto' device for SentenceTransformer, enhancing compatibility across environments. 2025-10-09 21:50:17 +03:00
mtayfur
86de7bad03 Add README.md to document the Memory System for Open WebUI, detailing core features, model support, configuration options, performance optimizations, and memory quality management. 2025-10-09 14:19:57 +03:00
mtayfur
6820ef9dcd Refactor SkipDetector category descriptions for enhanced clarity and specificity, improving technical accuracy and readability across various categories. 2025-10-09 13:43:44 +03:00
mtayfur
e89a9b29b9 Refactor cache configuration constants for improved clarity and consistency; adjust cache entry limits and hash prefix length 2025-10-09 13:10:58 +03:00
mtayfur
74f7ededcd Refactor memory retrieval logging to include retrieval method, enhancing clarity in LLM and semantic decision processes 2025-10-09 11:41:20 +03:00
mtayfur
bb992906ad Refactor LLM analysis logging and status emissions for improved clarity and accuracy in memory retrieval processes 2025-10-09 11:33:50 +03:00