14 KiB
14 KiB
BMAD Memory Storage and Retrieval Methodology
Overview
This methodology defines how to implement memory storage and retrieval operations within different IDE environments. It provides patterns and strategies for efficiently storing and accessing memory data using each platform's available capabilities.
Storage Strategy Methodology
Storage System Selection Framework
Capability Assessment
ide_capability_assessment:
file_system_access:
- can_read_write_files
- supports_directory_creation
- allows_file_watching
- provides_atomic_operations
database_capabilities:
- supports_embedded_databases
- allows_external_connections
- provides_transaction_support
- offers_indexing_capabilities
memory_management:
- available_ram_limits
- supports_caching
- allows_background_processing
- provides_cleanup_mechanisms
api_availability:
- storage_apis
- search_apis
- indexing_apis
- security_apis
Storage System Mapping
storage_system_selection:
working_memory:
primary_options:
- in_memory_variables
- session_storage
- temporary_files
selection_criteria:
- fast_access_required
- session_scoped_lifetime
- automatic_cleanup_needed
short_term_memory:
primary_options:
- workspace_storage
- project_files
- local_database
selection_criteria:
- project_scoped_persistence
- moderate_access_speed
- configurable_retention
long_term_memory:
primary_options:
- persistent_files
- external_database
- cloud_storage
selection_criteria:
- long_term_persistence
- large_storage_capacity
- backup_and_recovery
Multi-Modal Storage Implementation
File-Based Storage Pattern
file_based_storage:
advantages:
- simple_implementation
- human_readable_format
- version_control_friendly
- cross_platform_compatibility
implementation_strategy:
organization:
- use_hierarchical_directories
- implement_consistent_naming
- maintain_index_files
- provide_backup_mechanisms
formats:
- json_for_structured_data
- markdown_for_human_readable
- binary_for_large_content
- xml_for_complex_hierarchies
operations:
- atomic_write_operations
- file_locking_mechanisms
- change_detection_systems
- cleanup_procedures
Database Storage Pattern
database_storage:
advantages:
- efficient_querying
- transaction_support
- concurrent_access
- built_in_indexing
implementation_strategy:
embedded_databases:
- sqlite_for_simplicity
- leveldb_for_performance
- rocksdb_for_scalability
- duckdb_for_analytics
schema_design:
- normalize_for_consistency
- denormalize_for_performance
- index_for_common_queries
- partition_for_scalability
operations:
- use_prepared_statements
- implement_connection_pooling
- handle_transaction_boundaries
- manage_schema_migrations
Hybrid Storage Pattern
hybrid_storage:
strategy:
- use_database_for_metadata
- use_files_for_large_content
- use_memory_for_active_data
- use_cache_for_frequent_access
implementation:
coordination:
- maintain_reference_consistency
- synchronize_updates
- handle_partial_failures
- implement_cleanup_procedures
optimization:
- cache_frequently_accessed
- compress_large_content
- batch_related_operations
- monitor_storage_usage
Retrieval Strategy Methodology
Query Strategy Framework
Query Analysis Pattern
query_analysis:
intent_detection:
direct_lookup:
indicators: ["specific_id", "exact_reference"]
strategy: "id_based_retrieval"
semantic_search:
indicators: ["conceptual_terms", "similarity_requests"]
strategy: "embedding_based_search"
keyword_search:
indicators: ["specific_terms", "tag_references"]
strategy: "text_based_search"
temporal_search:
indicators: ["time_references", "recency_requests"]
strategy: "time_based_filtering"
relationship_search:
indicators: ["connection_terms", "related_requests"]
strategy: "graph_traversal"
Strategy Selection Algorithm
strategy_selection:
decision_tree:
has_specific_id:
true: "direct_lookup"
false: "analyze_content"
analyze_content:
has_conceptual_terms:
true: "semantic_search"
false: "check_keywords"
check_keywords:
has_specific_terms:
true: "keyword_search"
false: "check_temporal"
check_temporal:
has_time_reference:
true: "temporal_search"
false: "relationship_search"
Retrieval Implementation Patterns
Direct Retrieval Pattern
direct_retrieval:
use_cases:
- known_memory_id
- specific_reference
- follow_up_queries
implementation:
steps:
1. validate_memory_id
2. check_access_permissions
3. locate_storage_system
4. retrieve_memory_data
5. format_response
optimization:
- cache_frequently_accessed
- batch_multiple_requests
- preload_related_memories
- validate_data_integrity
Semantic Search Pattern
semantic_search:
use_cases:
- conceptual_queries
- similarity_matching
- knowledge_discovery
implementation:
approaches:
embedding_based:
- generate_query_embedding
- compute_similarity_scores
- rank_by_relevance
- filter_by_threshold
keyword_expansion:
- expand_query_terms
- find_related_concepts
- search_expanded_terms
- merge_result_sets
hybrid_approach:
- combine_embedding_and_keywords
- weight_different_signals
- optimize_for_precision_recall
- learn_from_user_feedback
Temporal Retrieval Pattern
temporal_retrieval:
use_cases:
- recent_memories
- historical_context
- time_based_patterns
implementation:
indexing:
- create_time_based_indices
- use_bucketed_timestamps
- maintain_sorted_lists
- implement_range_queries
querying:
- parse_temporal_expressions
- convert_to_timestamp_ranges
- apply_time_filters
- sort_by_relevance_and_time
Relationship Traversal Pattern
relationship_traversal:
use_cases:
- connected_information
- dependency_chains
- context_expansion
implementation:
algorithms:
breadth_first_search:
- explore_immediate_connections
- expand_level_by_level
- limit_traversal_depth
- avoid_cycles
depth_first_search:
- follow_specific_paths
- explore_deep_connections
- backtrack_when_needed
- maintain_path_history
weighted_traversal:
- consider_relationship_strength
- prioritize_strong_connections
- apply_decay_functions
- optimize_for_relevance
Hybrid Retrieval Methodology
Multi-Strategy Combination
hybrid_retrieval:
combination_strategies:
weighted_fusion:
approach: "combine_scores_with_weights"
implementation:
- execute_multiple_strategies
- normalize_scores
- apply_strategy_weights
- combine_final_scores
rank_fusion:
approach: "combine_rankings"
implementation:
- get_ranked_results
- apply_fusion_algorithm
- merge_rankings
- produce_final_ranking
sequential_filtering:
approach: "filter_progressively"
implementation:
- start_with_broad_strategy
- apply_additional_filters
- narrow_results_progressively
- maintain_result_quality
Result Optimization
result_optimization:
relevance_scoring:
factors:
- query_match_quality
- memory_importance
- recency_factor
- access_frequency
- user_preferences
scoring_formula:
- base_relevance_score
- importance_multiplier
- recency_decay_function
- frequency_boost
- personalization_factor
result_ranking:
primary_sort: "relevance_score"
secondary_sort: "importance"
tertiary_sort: "recency"
result_filtering:
- apply_access_controls
- remove_duplicates
- filter_by_quality_threshold
- limit_result_count
Performance Optimization Methodology
Caching Strategy
Multi-Level Caching
caching_levels:
memory_cache:
purpose: "immediate_access"
implementation:
- use_lru_eviction
- set_size_limits
- monitor_hit_rates
- optimize_for_working_set
query_cache:
purpose: "repeated_queries"
implementation:
- cache_query_results
- use_query_signatures
- implement_ttl_expiration
- invalidate_on_updates
index_cache:
purpose: "fast_lookups"
implementation:
- cache_index_structures
- preload_common_indices
- update_incrementally
- persist_across_sessions
Cache Management
cache_management:
eviction_policies:
lru: "least_recently_used"
lfu: "least_frequently_used"
ttl: "time_to_live"
size: "maximum_size_limit"
invalidation_strategies:
- invalidate_on_memory_update
- invalidate_related_queries
- use_versioning_for_consistency
- implement_lazy_invalidation
monitoring:
- track_hit_rates
- monitor_cache_size
- measure_eviction_frequency
- analyze_access_patterns
Indexing Strategy
Index Types and Usage
indexing_strategy:
primary_indices:
id_index:
structure: "hash_table"
use_case: "direct_lookup"
maintenance: "automatic"
type_index:
structure: "categorized_lists"
use_case: "type_based_queries"
maintenance: "on_creation"
temporal_index:
structure: "time_sorted_lists"
use_case: "time_based_queries"
maintenance: "periodic_cleanup"
secondary_indices:
keyword_index:
structure: "inverted_index"
use_case: "text_search"
maintenance: "on_content_change"
importance_index:
structure: "priority_queue"
use_case: "importance_ranking"
maintenance: "on_score_update"
Index Maintenance
index_maintenance:
update_strategies:
immediate_update:
- update_on_memory_change
- maintain_consistency
- handle_concurrent_access
- ensure_atomicity
batch_update:
- collect_changes
- update_periodically
- optimize_for_throughput
- handle_bulk_operations
lazy_update:
- mark_indices_stale
- rebuild_on_access
- optimize_for_write_performance
- handle_read_latency
optimization:
- compress_large_indices
- partition_by_usage_patterns
- precompute_common_queries
- monitor_index_effectiveness
Resource Management
Memory Management
memory_management:
allocation_strategy:
- set_memory_limits
- monitor_usage_patterns
- implement_garbage_collection
- handle_memory_pressure
optimization_techniques:
- use_object_pooling
- implement_lazy_loading
- compress_inactive_data
- stream_large_results
monitoring:
- track_memory_usage
- identify_memory_leaks
- monitor_gc_performance
- alert_on_thresholds
Storage Management
storage_management:
space_optimization:
- compress_old_memories
- archive_inactive_data
- implement_deduplication
- clean_up_temporary_files
performance_optimization:
- use_ssd_for_hot_data
- implement_read_ahead
- batch_write_operations
- optimize_file_layouts
monitoring:
- track_storage_usage
- monitor_io_performance
- identify_bottlenecks
- plan_capacity_growth
Security and Privacy Implementation
Access Control Methodology
Permission Framework
access_control:
permission_levels:
read_access:
- verify_user_identity
- check_memory_permissions
- apply_privacy_filters
- log_access_attempts
write_access:
- verify_ownership_or_delegation
- validate_modification_rights
- check_business_rules
- audit_changes
share_access:
- verify_sharing_permissions
- validate_recipient_access
- apply_sharing_restrictions
- track_sharing_chains
implementation:
- use_role_based_access_control
- implement_attribute_based_policies
- support_delegation_mechanisms
- provide_audit_trails
Privacy Protection
privacy_protection:
data_minimization:
- collect_only_necessary_data
- limit_retention_periods
- anonymize_where_possible
- provide_deletion_mechanisms
encryption:
- encrypt_sensitive_memories
- use_strong_encryption_algorithms
- manage_encryption_keys_securely
- implement_key_rotation
access_logging:
- log_all_access_attempts
- include_sufficient_detail
- protect_log_integrity
- provide_audit_capabilities
Error Handling and Recovery
Error Handling Patterns
error_handling:
storage_errors:
- implement_retry_mechanisms
- provide_fallback_storage
- maintain_data_consistency
- notify_users_appropriately
retrieval_errors:
- handle_missing_memories
- provide_partial_results
- suggest_alternative_queries
- maintain_search_performance
corruption_handling:
- detect_data_corruption
- attempt_automatic_repair
- restore_from_backups
- prevent_corruption_spread
Recovery Procedures
recovery_procedures:
backup_strategy:
- implement_regular_backups
- test_backup_integrity
- provide_point_in_time_recovery
- maintain_backup_retention
disaster_recovery:
- document_recovery_procedures
- test_recovery_processes
- maintain_recovery_time_objectives
- ensure_data_consistency
This methodology provides comprehensive guidance for implementing memory storage and retrieval systems within any IDE environment while adapting to platform-specific capabilities and constraints.