feat(story-5.1): implement markdown telemetry report generation skill
Create bmad-memtrace-telemetry skill with introspection protocol, embedded Markdown template (42 tools across 8 categories), and confinement rules for post-sprint Memtrace usage reporting. Code review patches applied: - Expanded tool catalog to cover all ~42 Memtrace MCP tools - Added history-unavailable fallback guidance - Added objective activation criteria (Memtrace tool call check) - Added output directory creation, collision guard, permission handling
This commit is contained in:
Magal
parent
c902a6f0a4
commit
7d5ffc5280
|
|
@ -0,0 +1,259 @@
|
|||
---
|
||||
name: bmad-memtrace-telemetry
|
||||
description: 'Memtrace usage telemetry report. Use at sprint conclusion or on human request to generate a structured Markdown report detailing tool usage, omissions, errors, and comparative graph-vs-legacy analysis.'
|
||||
---
|
||||
|
||||
# Memtrace Telemetry Report Generation
|
||||
|
||||
## When to Use
|
||||
|
||||
Activate this skill when:
|
||||
- A sprint cycle has completed (all sprint stories are `done`)
|
||||
- A significant development milestone has been reached (e.g., release tag, major feature branch merged to main)
|
||||
- The Human Developer explicitly requests: "generate telemetry report" or "run telemetry"
|
||||
- A sprint or milestone had **at least one Memtrace MCP tool invocation** in the agent's session history (check your tool-call log for any `find_code`, `get_impact`, `find_dead_code`, `index_directory`, or other memtrace tool calls)
|
||||
|
||||
**DO NOT** activate this skill:
|
||||
- Mid-sprint (the report covers completed work, not in-progress)
|
||||
- When zero Memtrace MCP tools were invoked during the sprint period (no telemetry to report)
|
||||
- For individual story completions (wait for the full sprint cycle)
|
||||
|
||||
## Introspection Protocol
|
||||
|
||||
Before writing the report, review your tool-call history to identify every Memtrace MCP tool interaction during the sprint.
|
||||
|
||||
**If tool-call history is unavailable** (session restart, context window cleared, or log truncated): note `history_unavailable` in the Executive Summary. Populate sections with what you can recall from memory. Do NOT fabricate invocation counts or tool names — use approximate counts labeled `~estimated` rather than specific numbers.
|
||||
|
||||
If you cannot recall any tool usage but know the sprint had Memtrace-relevant work, state: "Tool-call history unavailable — report is based on recollection and may be incomplete."
|
||||
|
||||
### Step 1: Inventory Tools Used
|
||||
Scan your conversation and tool-call history for each Memtrace MCP tool invocation. For each tool used, capture:
|
||||
- **Tool name** — exact name from the catalog below
|
||||
- **Invocation count** — how many times it was called
|
||||
- **Primary use case** — what task the tool served (e.g., "blast radius for refactor of auth module")
|
||||
- **Result** — whether each call succeeded, returned mixed results (partial data), or failed
|
||||
|
||||
### Step 2: Identify Tools Omitted
|
||||
Consult the Complete Tool Catalog Reference (Appendix) and identify tools that were available but NOT used. For each omitted tool:
|
||||
- **Category** — which group it belongs to (Discovery, Dependency, Quality, etc.)
|
||||
- **Reason omitted** — why the tool wasn't needed (e.g., "no dead-code analysis performed this sprint", "module was too small for summarization to matter")
|
||||
|
||||
If a tool has zero omissions because ALL tools were used, still document that explicitly: "All 24 Memtrace MCP tools were used this sprint — no omissions."
|
||||
|
||||
### Step 3: Collect Errors & Failures
|
||||
Review your session logs for:
|
||||
- Timeout errors (`MEMTRACE_MCP_ERROR_TIMEOUT`)
|
||||
- Connection drops or refused connections
|
||||
- Stale index warnings (`[FRESHNESS] fresh=false`)
|
||||
- Tool-specific failures (any Memtrace MCP tool that returned an error)
|
||||
- Recovery events (was `npm run memtrace:restart` executed? Did it succeed or fail?)
|
||||
|
||||
### Step 4: Gather Sprint Context
|
||||
- Which epic(s) were worked on this sprint
|
||||
- Which stories were completed
|
||||
- The primary agent model used
|
||||
|
||||
### Step 5: Identify Friction Points
|
||||
For each friction point encountered during the sprint:
|
||||
- Describe the friction (e.g., "query returned too many results before summarization existed")
|
||||
- Note the context (when/where it happened)
|
||||
- Document the workaround used by the agent
|
||||
|
||||
### Step 6: Prepare Comparative Analysis
|
||||
- Contrast graph-based execution (Memtrace) against any legacy text-search heuristics (`grep`, `glob`, `rg`, `find`) that were used
|
||||
- If no legacy tools were used (Memtrace was exclusively relied upon), note that explicitly
|
||||
- Assess the net impact of structural verification on the sprint's outcomes
|
||||
|
||||
### Step 7: Formulate Feature Requests
|
||||
Based on the friction points identified, draft actionable feature requests for maintainers. Each request should include:
|
||||
- A clear description of the desired improvement
|
||||
- Priority (High / Medium / Low)
|
||||
- Context of the situation that motivated the request
|
||||
|
||||
## Template Format
|
||||
|
||||
Generate the report as a Markdown file following this exact structure. Every section must be present. If a section has no data, write "None" or "N/A" rather than omitting the section.
|
||||
|
||||
```markdown
|
||||
# Memtrace Telemetry Report
|
||||
|
||||
**Generated:** {timestamp}
|
||||
**Sprint/Epic:** {epic_identifier}
|
||||
**Agent:** {agent_name}
|
||||
|
||||
## Executive Summary
|
||||
|
||||
{Brief 2-3 sentence summary of Memtrace usage during this sprint: overall reliability, key successes,
|
||||
critical failures, and whether structural verification was consistently achieved.}
|
||||
|
||||
## Sprint Context
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Epic(s) worked | {epic_list} |
|
||||
| Stories completed | {story_count} |
|
||||
| Primary agent | {agent_name} |
|
||||
| Repository | bmad-memtrace |
|
||||
|
||||
## Memtrace Tools Used
|
||||
|
||||
{For each Memtrace MCP tool invoked during the sprint, record:}
|
||||
|
||||
| Tool | Invocations | Primary Use Case | Result |
|
||||
|------|-------------|------------------|--------|
|
||||
| {tool_name} | {count} | {why this tool was needed} | {success/mixed/failure} |
|
||||
|
||||
### Tool Usage Distribution
|
||||
|
||||
{Brief narrative of which tool categories were most used: Discovery, Dependency Analysis,
|
||||
Quality, Temporal, Index Management — and why.}
|
||||
|
||||
## Memtrace Tools Omitted
|
||||
|
||||
{List Memtrace tools that were available but NOT used during this sprint, with justification
|
||||
for why they weren't needed or why the agent chose alternatives.}
|
||||
|
||||
| Tool | Category | Reason Omitted |
|
||||
|------|----------|----------------|
|
||||
| {tool_name} | {category} | {justification} |
|
||||
|
||||
## Errors & Failures
|
||||
|
||||
{Record every Memtrace-related error, timeout, stale index warning, and connection failure
|
||||
encountered during the sprint.}
|
||||
|
||||
### Connection & Recovery Events
|
||||
|
||||
| Timestamp | Error Type | Detail | Recovery Action | Outcome |
|
||||
|-----------|------------|--------|-----------------|---------|
|
||||
| {time} | TIMEOUT / STALE_INDEX / CONNECTION_REFUSED | {detail} | {action taken} | {outcome} |
|
||||
|
||||
### Tool-Specific Failures
|
||||
|
||||
| Tool | Target/Query | Error | Impact |
|
||||
|------|-------------|-------|--------|
|
||||
| {tool} | {query} | {error_message} | {what was blocked} |
|
||||
|
||||
## Friction Points
|
||||
|
||||
{Each friction point encountered. Note: future stories may add severity (1-5) and justification fields.}
|
||||
|
||||
| Friction | Context | Workaround Used |
|
||||
|----------|---------|-----------------|
|
||||
| {description} | {when/where} | {how agent adapted} |
|
||||
|
||||
## Comparative Analysis: Graph vs Legacy
|
||||
|
||||
{Contrast the experiences and outcomes of using Memtrace structural queries against any
|
||||
legacy text-search heuristics used (grep, glob, rg, find).}
|
||||
|
||||
### Graph-Based Execution (Memtrace)
|
||||
|
||||
- **Successes:** {where structural verification provided certainty}
|
||||
- **Limitations:** {where graph queries were insufficient or failed}
|
||||
|
||||
### Legacy Execution (Text-Search)
|
||||
|
||||
- **When used:** {contexts where grep/glob were used instead of or alongside Memtrace}
|
||||
- **Limitations experienced:** {missed dependencies, false positives, manual effort}
|
||||
|
||||
### Net Impact Assessment
|
||||
|
||||
{Was the sprint MORE or LESS efficient with Memtrace? Were there fewer or more regressions?
|
||||
Did structural verification meaningfully prevent breakage?}
|
||||
|
||||
## Feature Requests & Feedback
|
||||
|
||||
{Actionable feature requests or improvement suggestions for the Memtrace maintainers.
|
||||
Note: future stories will add autonomous +1 upvote deduplication.}
|
||||
|
||||
| ID | Request | Priority (H/M/L) | Context |
|
||||
|----|---------|-------------------|---------|
|
||||
| FR-{n} | {description} | {priority} | {situation that prompted this} |
|
||||
|
||||
## Appendix: Complete Tool Catalog Reference
|
||||
|
||||
{Reference list of all Memtrace MCP tools — the agent consults this to populate the
|
||||
"Tools Omitted" section accurately.}
|
||||
|
||||
### Discovery
|
||||
- `find_code` — Semantic + full-text search across indexed codebase
|
||||
- `find_symbol` — Exact/fuzzy symbol lookup by name
|
||||
- `get_source_window` — Bounded source-code read with context lines
|
||||
- `get_directory_tree` — Compact directory structure overview
|
||||
|
||||
### Architecture & Mapping
|
||||
- `get_codebase_briefing` — Repository scale, modules, endpoints, risk summary
|
||||
- `list_communities` — Louvain community clusters (bounded contexts)
|
||||
- `list_processes` — Detected execution processes (HTTP handlers, jobs, CLI)
|
||||
- `get_process_flow` — End-to-end call-chain trace from entry point
|
||||
|
||||
### Dependency Analysis
|
||||
- `get_symbol_context` — 360° view: callers, callees, community, process, cross-repo API
|
||||
- `analyze_relationships` — Callers, callees, class hierarchy, imports, type usages
|
||||
- `get_impact` — Blast radius (upstream/downstream transitive)
|
||||
- `find_dependency_path` — Shortest call/dependency path between two symbols
|
||||
- `get_api_topology` — Cross-repo HTTP call topology (service dependencies)
|
||||
- `find_api_calls` — Outbound HTTP calls made by a service
|
||||
- `find_api_endpoints` — HTTP endpoints exposed by a service
|
||||
- `get_service_diagram` — Mermaid service-dependency diagram
|
||||
|
||||
### Quality Analysis
|
||||
- `find_dead_code` — Zero-caller function/method candidates
|
||||
- `find_most_complex_functions` — Top-N cyclomatic complexity hotspots
|
||||
- `find_bridge_symbols` — High-betweenness architectural chokepoints
|
||||
- `find_central_symbols` — High-PageRank structurally important symbols
|
||||
- `calculate_cyclomatic_complexity` — Approximate complexity of a function
|
||||
|
||||
### Temporal Analysis
|
||||
- `get_evolution` — Change timeline between two timepoints
|
||||
- `get_changes_since` — Incremental diff since a session anchor
|
||||
- `get_timeline` — Version history of a single symbol across episodes
|
||||
- `get_episode_replay` — Full add/modify/remove diff of one episode
|
||||
- `get_cochange_context` — Symbols that historically change together
|
||||
|
||||
### Change Detection
|
||||
- `detect_changes` — Affected symbols from a git diff or file list
|
||||
|
||||
### Index Management
|
||||
- `index_directory` — Parse and index a local directory into the graph
|
||||
- `list_indexed_repositories` — List indexed repos with freshness timestamps
|
||||
- `watch_directory` — Enable live incremental re-indexing on file save
|
||||
- `unwatch_directory` — Stop watching a directory for changes
|
||||
- `list_jobs` — List indexing jobs with status and timestamps
|
||||
- `list_watched_paths` — Currently watched directories
|
||||
- `list_worktrees` — Known worktree overlays
|
||||
- `cleanup_episodes` — Delete historical episode snapshots
|
||||
- `cleanup_stale_records` — Scrub orphan node/edge records
|
||||
- `cleanup_worktrees` — Sweep stale worktree overlays
|
||||
- `replay_history` — Re-run git history replay without re-indexing HEAD
|
||||
- `link_repositories` — Add a typed LINKED_TO edge between repos
|
||||
- `record_external_episode` — Persist externally-authored episodes
|
||||
- `delete_repository` — Remove all nodes, edges, and episodes for a repo
|
||||
|
||||
### Operational
|
||||
- `embed_diag` — Embed pipeline diagnostics snapshot
|
||||
- `embed_reset_breaker` — Reset embed circuit breaker
|
||||
```
|
||||
|
||||
## Output Conventions
|
||||
|
||||
- **Save location:** `_bmad-output/telemetry/` (relative to project root)
|
||||
- **File naming:** `telemetry-YYYY-MM-DD-HHmmss.md` (local timestamp at generation time)
|
||||
- **Format:** Valid Markdown with all required sections present
|
||||
- **Tool names:** Must match the Complete Tool Catalog Reference exactly — no hallucinated tool names
|
||||
- **Directory creation:** If `_bmad-output/telemetry/` does not exist, create it before saving
|
||||
- **File collision:** If a file with the exact timestamp already exists, increment the timestamp by 1 second until unique
|
||||
- **Permission failure:** If the agent cannot write to the output directory, output the report to STDOUT with a clear warning and do NOT lose the report data
|
||||
- **Only create the report file** — do NOT create or modify any other files during this workflow
|
||||
|
||||
## Confinement Rules
|
||||
|
||||
- **ALWAYS** follow the introspection protocol before writing — do not guess tool usage
|
||||
- **ALWAYS** check every tool against the catalog before writing its name in the report
|
||||
- **ALWAYS** include ALL required sections — omit data rather than omit sections
|
||||
- **ALWAYS** save the report to `_bmad-output/telemetry/` with the timestamped naming convention
|
||||
- **NEVER** create Node.js scripts or modify existing code during telemetry generation
|
||||
- **NEVER** make live MCP calls during report generation — introspection only (consult your existing tool-call history)
|
||||
- **NEVER** modify `memtrace-adapter.mjs`, `package.json`, or any existing files
|
||||
- **NEVER** require or rely on internet access — this is an offline introspection workflow
|
||||
Loading…
Reference in New Issue