633 lines
14 KiB
Markdown
633 lines
14 KiB
Markdown
# BMAD Memory Storage and Retrieval Methodology
|
|
|
|
## Overview
|
|
|
|
This methodology defines how to implement memory storage and retrieval operations within different IDE environments. It provides patterns and strategies for efficiently storing and accessing memory data using each platform's available capabilities.
|
|
|
|
## Storage Strategy Methodology
|
|
|
|
### Storage System Selection Framework
|
|
|
|
#### Capability Assessment
|
|
```yaml
|
|
ide_capability_assessment:
|
|
file_system_access:
|
|
- can_read_write_files
|
|
- supports_directory_creation
|
|
- allows_file_watching
|
|
- provides_atomic_operations
|
|
|
|
database_capabilities:
|
|
- supports_embedded_databases
|
|
- allows_external_connections
|
|
- provides_transaction_support
|
|
- offers_indexing_capabilities
|
|
|
|
memory_management:
|
|
- available_ram_limits
|
|
- supports_caching
|
|
- allows_background_processing
|
|
- provides_cleanup_mechanisms
|
|
|
|
api_availability:
|
|
- storage_apis
|
|
- search_apis
|
|
- indexing_apis
|
|
- security_apis
|
|
```
|
|
|
|
#### Storage System Mapping
|
|
```yaml
|
|
storage_system_selection:
|
|
working_memory:
|
|
primary_options:
|
|
- in_memory_variables
|
|
- session_storage
|
|
- temporary_files
|
|
selection_criteria:
|
|
- fast_access_required
|
|
- session_scoped_lifetime
|
|
- automatic_cleanup_needed
|
|
|
|
short_term_memory:
|
|
primary_options:
|
|
- workspace_storage
|
|
- project_files
|
|
- local_database
|
|
selection_criteria:
|
|
- project_scoped_persistence
|
|
- moderate_access_speed
|
|
- configurable_retention
|
|
|
|
long_term_memory:
|
|
primary_options:
|
|
- persistent_files
|
|
- external_database
|
|
- cloud_storage
|
|
selection_criteria:
|
|
- long_term_persistence
|
|
- large_storage_capacity
|
|
- backup_and_recovery
|
|
```
|
|
|
|
### Multi-Modal Storage Implementation
|
|
|
|
#### File-Based Storage Pattern
|
|
```yaml
|
|
file_based_storage:
|
|
advantages:
|
|
- simple_implementation
|
|
- human_readable_format
|
|
- version_control_friendly
|
|
- cross_platform_compatibility
|
|
|
|
implementation_strategy:
|
|
organization:
|
|
- use_hierarchical_directories
|
|
- implement_consistent_naming
|
|
- maintain_index_files
|
|
- provide_backup_mechanisms
|
|
|
|
formats:
|
|
- json_for_structured_data
|
|
- markdown_for_human_readable
|
|
- binary_for_large_content
|
|
- xml_for_complex_hierarchies
|
|
|
|
operations:
|
|
- atomic_write_operations
|
|
- file_locking_mechanisms
|
|
- change_detection_systems
|
|
- cleanup_procedures
|
|
```
|
|
|
|
#### Database Storage Pattern
|
|
```yaml
|
|
database_storage:
|
|
advantages:
|
|
- efficient_querying
|
|
- transaction_support
|
|
- concurrent_access
|
|
- built_in_indexing
|
|
|
|
implementation_strategy:
|
|
embedded_databases:
|
|
- sqlite_for_simplicity
|
|
- leveldb_for_performance
|
|
- rocksdb_for_scalability
|
|
- duckdb_for_analytics
|
|
|
|
schema_design:
|
|
- normalize_for_consistency
|
|
- denormalize_for_performance
|
|
- index_for_common_queries
|
|
- partition_for_scalability
|
|
|
|
operations:
|
|
- use_prepared_statements
|
|
- implement_connection_pooling
|
|
- handle_transaction_boundaries
|
|
- manage_schema_migrations
|
|
```
|
|
|
|
#### Hybrid Storage Pattern
|
|
```yaml
|
|
hybrid_storage:
|
|
strategy:
|
|
- use_database_for_metadata
|
|
- use_files_for_large_content
|
|
- use_memory_for_active_data
|
|
- use_cache_for_frequent_access
|
|
|
|
implementation:
|
|
coordination:
|
|
- maintain_reference_consistency
|
|
- synchronize_updates
|
|
- handle_partial_failures
|
|
- implement_cleanup_procedures
|
|
|
|
optimization:
|
|
- cache_frequently_accessed
|
|
- compress_large_content
|
|
- batch_related_operations
|
|
- monitor_storage_usage
|
|
```
|
|
|
|
## Retrieval Strategy Methodology
|
|
|
|
### Query Strategy Framework
|
|
|
|
#### Query Analysis Pattern
|
|
```yaml
|
|
query_analysis:
|
|
intent_detection:
|
|
direct_lookup:
|
|
indicators: ["specific_id", "exact_reference"]
|
|
strategy: "id_based_retrieval"
|
|
|
|
semantic_search:
|
|
indicators: ["conceptual_terms", "similarity_requests"]
|
|
strategy: "embedding_based_search"
|
|
|
|
keyword_search:
|
|
indicators: ["specific_terms", "tag_references"]
|
|
strategy: "text_based_search"
|
|
|
|
temporal_search:
|
|
indicators: ["time_references", "recency_requests"]
|
|
strategy: "time_based_filtering"
|
|
|
|
relationship_search:
|
|
indicators: ["connection_terms", "related_requests"]
|
|
strategy: "graph_traversal"
|
|
```
|
|
|
|
#### Strategy Selection Algorithm
|
|
```yaml
|
|
strategy_selection:
|
|
decision_tree:
|
|
has_specific_id:
|
|
true: "direct_lookup"
|
|
false: "analyze_content"
|
|
|
|
analyze_content:
|
|
has_conceptual_terms:
|
|
true: "semantic_search"
|
|
false: "check_keywords"
|
|
|
|
check_keywords:
|
|
has_specific_terms:
|
|
true: "keyword_search"
|
|
false: "check_temporal"
|
|
|
|
check_temporal:
|
|
has_time_reference:
|
|
true: "temporal_search"
|
|
false: "relationship_search"
|
|
```
|
|
|
|
### Retrieval Implementation Patterns
|
|
|
|
#### Direct Retrieval Pattern
|
|
```yaml
|
|
direct_retrieval:
|
|
use_cases:
|
|
- known_memory_id
|
|
- specific_reference
|
|
- follow_up_queries
|
|
|
|
implementation:
|
|
steps:
|
|
1. validate_memory_id
|
|
2. check_access_permissions
|
|
3. locate_storage_system
|
|
4. retrieve_memory_data
|
|
5. format_response
|
|
|
|
optimization:
|
|
- cache_frequently_accessed
|
|
- batch_multiple_requests
|
|
- preload_related_memories
|
|
- validate_data_integrity
|
|
```
|
|
|
|
#### Semantic Search Pattern
|
|
```yaml
|
|
semantic_search:
|
|
use_cases:
|
|
- conceptual_queries
|
|
- similarity_matching
|
|
- knowledge_discovery
|
|
|
|
implementation:
|
|
approaches:
|
|
embedding_based:
|
|
- generate_query_embedding
|
|
- compute_similarity_scores
|
|
- rank_by_relevance
|
|
- filter_by_threshold
|
|
|
|
keyword_expansion:
|
|
- expand_query_terms
|
|
- find_related_concepts
|
|
- search_expanded_terms
|
|
- merge_result_sets
|
|
|
|
hybrid_approach:
|
|
- combine_embedding_and_keywords
|
|
- weight_different_signals
|
|
- optimize_for_precision_recall
|
|
- learn_from_user_feedback
|
|
```
|
|
|
|
#### Temporal Retrieval Pattern
|
|
```yaml
|
|
temporal_retrieval:
|
|
use_cases:
|
|
- recent_memories
|
|
- historical_context
|
|
- time_based_patterns
|
|
|
|
implementation:
|
|
indexing:
|
|
- create_time_based_indices
|
|
- use_bucketed_timestamps
|
|
- maintain_sorted_lists
|
|
- implement_range_queries
|
|
|
|
querying:
|
|
- parse_temporal_expressions
|
|
- convert_to_timestamp_ranges
|
|
- apply_time_filters
|
|
- sort_by_relevance_and_time
|
|
```
|
|
|
|
#### Relationship Traversal Pattern
|
|
```yaml
|
|
relationship_traversal:
|
|
use_cases:
|
|
- connected_information
|
|
- dependency_chains
|
|
- context_expansion
|
|
|
|
implementation:
|
|
algorithms:
|
|
breadth_first_search:
|
|
- explore_immediate_connections
|
|
- expand_level_by_level
|
|
- limit_traversal_depth
|
|
- avoid_cycles
|
|
|
|
depth_first_search:
|
|
- follow_specific_paths
|
|
- explore_deep_connections
|
|
- backtrack_when_needed
|
|
- maintain_path_history
|
|
|
|
weighted_traversal:
|
|
- consider_relationship_strength
|
|
- prioritize_strong_connections
|
|
- apply_decay_functions
|
|
- optimize_for_relevance
|
|
```
|
|
|
|
### Hybrid Retrieval Methodology
|
|
|
|
#### Multi-Strategy Combination
|
|
```yaml
|
|
hybrid_retrieval:
|
|
combination_strategies:
|
|
weighted_fusion:
|
|
approach: "combine_scores_with_weights"
|
|
implementation:
|
|
- execute_multiple_strategies
|
|
- normalize_scores
|
|
- apply_strategy_weights
|
|
- combine_final_scores
|
|
|
|
rank_fusion:
|
|
approach: "combine_rankings"
|
|
implementation:
|
|
- get_ranked_results
|
|
- apply_fusion_algorithm
|
|
- merge_rankings
|
|
- produce_final_ranking
|
|
|
|
sequential_filtering:
|
|
approach: "filter_progressively"
|
|
implementation:
|
|
- start_with_broad_strategy
|
|
- apply_additional_filters
|
|
- narrow_results_progressively
|
|
- maintain_result_quality
|
|
```
|
|
|
|
#### Result Optimization
|
|
```yaml
|
|
result_optimization:
|
|
relevance_scoring:
|
|
factors:
|
|
- query_match_quality
|
|
- memory_importance
|
|
- recency_factor
|
|
- access_frequency
|
|
- user_preferences
|
|
|
|
scoring_formula:
|
|
- base_relevance_score
|
|
- importance_multiplier
|
|
- recency_decay_function
|
|
- frequency_boost
|
|
- personalization_factor
|
|
|
|
result_ranking:
|
|
primary_sort: "relevance_score"
|
|
secondary_sort: "importance"
|
|
tertiary_sort: "recency"
|
|
|
|
result_filtering:
|
|
- apply_access_controls
|
|
- remove_duplicates
|
|
- filter_by_quality_threshold
|
|
- limit_result_count
|
|
```
|
|
|
|
## Performance Optimization Methodology
|
|
|
|
### Caching Strategy
|
|
|
|
#### Multi-Level Caching
|
|
```yaml
|
|
caching_levels:
|
|
memory_cache:
|
|
purpose: "immediate_access"
|
|
implementation:
|
|
- use_lru_eviction
|
|
- set_size_limits
|
|
- monitor_hit_rates
|
|
- optimize_for_working_set
|
|
|
|
query_cache:
|
|
purpose: "repeated_queries"
|
|
implementation:
|
|
- cache_query_results
|
|
- use_query_signatures
|
|
- implement_ttl_expiration
|
|
- invalidate_on_updates
|
|
|
|
index_cache:
|
|
purpose: "fast_lookups"
|
|
implementation:
|
|
- cache_index_structures
|
|
- preload_common_indices
|
|
- update_incrementally
|
|
- persist_across_sessions
|
|
```
|
|
|
|
#### Cache Management
|
|
```yaml
|
|
cache_management:
|
|
eviction_policies:
|
|
lru: "least_recently_used"
|
|
lfu: "least_frequently_used"
|
|
ttl: "time_to_live"
|
|
size: "maximum_size_limit"
|
|
|
|
invalidation_strategies:
|
|
- invalidate_on_memory_update
|
|
- invalidate_related_queries
|
|
- use_versioning_for_consistency
|
|
- implement_lazy_invalidation
|
|
|
|
monitoring:
|
|
- track_hit_rates
|
|
- monitor_cache_size
|
|
- measure_eviction_frequency
|
|
- analyze_access_patterns
|
|
```
|
|
|
|
### Indexing Strategy
|
|
|
|
#### Index Types and Usage
|
|
```yaml
|
|
indexing_strategy:
|
|
primary_indices:
|
|
id_index:
|
|
structure: "hash_table"
|
|
use_case: "direct_lookup"
|
|
maintenance: "automatic"
|
|
|
|
type_index:
|
|
structure: "categorized_lists"
|
|
use_case: "type_based_queries"
|
|
maintenance: "on_creation"
|
|
|
|
temporal_index:
|
|
structure: "time_sorted_lists"
|
|
use_case: "time_based_queries"
|
|
maintenance: "periodic_cleanup"
|
|
|
|
secondary_indices:
|
|
keyword_index:
|
|
structure: "inverted_index"
|
|
use_case: "text_search"
|
|
maintenance: "on_content_change"
|
|
|
|
importance_index:
|
|
structure: "priority_queue"
|
|
use_case: "importance_ranking"
|
|
maintenance: "on_score_update"
|
|
```
|
|
|
|
#### Index Maintenance
|
|
```yaml
|
|
index_maintenance:
|
|
update_strategies:
|
|
immediate_update:
|
|
- update_on_memory_change
|
|
- maintain_consistency
|
|
- handle_concurrent_access
|
|
- ensure_atomicity
|
|
|
|
batch_update:
|
|
- collect_changes
|
|
- update_periodically
|
|
- optimize_for_throughput
|
|
- handle_bulk_operations
|
|
|
|
lazy_update:
|
|
- mark_indices_stale
|
|
- rebuild_on_access
|
|
- optimize_for_write_performance
|
|
- handle_read_latency
|
|
|
|
optimization:
|
|
- compress_large_indices
|
|
- partition_by_usage_patterns
|
|
- precompute_common_queries
|
|
- monitor_index_effectiveness
|
|
```
|
|
|
|
### Resource Management
|
|
|
|
#### Memory Management
|
|
```yaml
|
|
memory_management:
|
|
allocation_strategy:
|
|
- set_memory_limits
|
|
- monitor_usage_patterns
|
|
- implement_garbage_collection
|
|
- handle_memory_pressure
|
|
|
|
optimization_techniques:
|
|
- use_object_pooling
|
|
- implement_lazy_loading
|
|
- compress_inactive_data
|
|
- stream_large_results
|
|
|
|
monitoring:
|
|
- track_memory_usage
|
|
- identify_memory_leaks
|
|
- monitor_gc_performance
|
|
- alert_on_thresholds
|
|
```
|
|
|
|
#### Storage Management
|
|
```yaml
|
|
storage_management:
|
|
space_optimization:
|
|
- compress_old_memories
|
|
- archive_inactive_data
|
|
- implement_deduplication
|
|
- clean_up_temporary_files
|
|
|
|
performance_optimization:
|
|
- use_ssd_for_hot_data
|
|
- implement_read_ahead
|
|
- batch_write_operations
|
|
- optimize_file_layouts
|
|
|
|
monitoring:
|
|
- track_storage_usage
|
|
- monitor_io_performance
|
|
- identify_bottlenecks
|
|
- plan_capacity_growth
|
|
```
|
|
|
|
## Security and Privacy Implementation
|
|
|
|
### Access Control Methodology
|
|
|
|
#### Permission Framework
|
|
```yaml
|
|
access_control:
|
|
permission_levels:
|
|
read_access:
|
|
- verify_user_identity
|
|
- check_memory_permissions
|
|
- apply_privacy_filters
|
|
- log_access_attempts
|
|
|
|
write_access:
|
|
- verify_ownership_or_delegation
|
|
- validate_modification_rights
|
|
- check_business_rules
|
|
- audit_changes
|
|
|
|
share_access:
|
|
- verify_sharing_permissions
|
|
- validate_recipient_access
|
|
- apply_sharing_restrictions
|
|
- track_sharing_chains
|
|
|
|
implementation:
|
|
- use_role_based_access_control
|
|
- implement_attribute_based_policies
|
|
- support_delegation_mechanisms
|
|
- provide_audit_trails
|
|
```
|
|
|
|
#### Privacy Protection
|
|
```yaml
|
|
privacy_protection:
|
|
data_minimization:
|
|
- collect_only_necessary_data
|
|
- limit_retention_periods
|
|
- anonymize_where_possible
|
|
- provide_deletion_mechanisms
|
|
|
|
encryption:
|
|
- encrypt_sensitive_memories
|
|
- use_strong_encryption_algorithms
|
|
- manage_encryption_keys_securely
|
|
- implement_key_rotation
|
|
|
|
access_logging:
|
|
- log_all_access_attempts
|
|
- include_sufficient_detail
|
|
- protect_log_integrity
|
|
- provide_audit_capabilities
|
|
```
|
|
|
|
## Error Handling and Recovery
|
|
|
|
### Error Handling Patterns
|
|
```yaml
|
|
error_handling:
|
|
storage_errors:
|
|
- implement_retry_mechanisms
|
|
- provide_fallback_storage
|
|
- maintain_data_consistency
|
|
- notify_users_appropriately
|
|
|
|
retrieval_errors:
|
|
- handle_missing_memories
|
|
- provide_partial_results
|
|
- suggest_alternative_queries
|
|
- maintain_search_performance
|
|
|
|
corruption_handling:
|
|
- detect_data_corruption
|
|
- attempt_automatic_repair
|
|
- restore_from_backups
|
|
- prevent_corruption_spread
|
|
```
|
|
|
|
### Recovery Procedures
|
|
```yaml
|
|
recovery_procedures:
|
|
backup_strategy:
|
|
- implement_regular_backups
|
|
- test_backup_integrity
|
|
- provide_point_in_time_recovery
|
|
- maintain_backup_retention
|
|
|
|
disaster_recovery:
|
|
- document_recovery_procedures
|
|
- test_recovery_processes
|
|
- maintain_recovery_time_objectives
|
|
- ensure_data_consistency
|
|
```
|
|
|
|
This methodology provides comprehensive guidance for implementing memory storage and retrieval systems within any IDE environment while adapting to platform-specific capabilities and constraints.
|