#1180 - new advanced-elicitation methods

This commit is contained in:
lukasz.krysik 2025-12-29 12:02:34 +01:00
parent b8836ced24
commit 178ba91f2d
1 changed files with 25 additions and 0 deletions

View File

@ -49,3 +49,28 @@ num,category,method_name,description,output_pattern
48,philosophical,Trolley Problem Variations,Explore ethical trade-offs through moral dilemmas - valuable for understanding values and difficult decisions,dilemma → analysis → decision
49,retrospective,Hindsight Reflection,Imagine looking back from the future to gain perspective - powerful for project reviews,future view → insights → application
50,retrospective,Lessons Learned Extraction,Systematically identify key takeaways and actionable improvements - essential for continuous improvement,experience → lessons → actions
51,anti-bias,Liar's Trap,Demand agent lists 3 ways it could deceive you in its current response. For each way listed: Is it currently doing this? If it cannot find 3 genuine deception vectors it is not being honest about its limitations,deception methods → self-examination → revealed blindspots
52,anti-bias,Mirror Trap,Ask: What would a DISHONEST agent say who wants to finish quickly and not find problems? Compare with current response. Similarity >50% requires revision with more rigor,dishonest version → comparison → honesty assessment
53,anti-bias,Confession Paradox,Before accepting work: The work I'm about to produce is an attempt to avoid the HARD part. Prove this false by identifying the hardest part and confirming adequate focus on it,hard parts → effort check → revised approach
54,anti-bias,CUI BONO Test,For every decision and assumption: Who benefits? If it benefits the AGENT (easier work) - RED FLAG requiring justification. If it benefits the OUTCOME - acceptable,decisions → beneficiary analysis → justification
55,challenge,Barber Paradox,What ALTERNATIVE approach would you reject but if someone else proposed it you would consider better? Forces consideration of dismissed options,alternatives → rejection reasons → reconsideration
56,challenge,Sorites Paradox,Remove elements one by one. Which single removal DESTROYS the solution? That element should have the MOST attention. Does it?,elements → removal test → priority check
57,challenge,Newcomb's Paradox,What solution would SURPRISE you as solving this problem? If your current approach is not surprising it may be too obvious and miss creative solutions,expected approach → surprising alternatives → creativity check
58,challenge,Braess Paradox,Which element SEEMS helpful but might actually HURT? Sometimes removing constraints or features improves the result,helpful elements → harm analysis → optimization
59,challenge,Simpson's Paradox,The solution looks good in each part separately. What HIDDEN VARIABLE could make the whole worse than the parts?,parts analysis → hidden variables → integration check
60,challenge,Surprise Exam Paradox,Where is the solution TOO CONFIDENT? What could surprise it? Overconfidence reveals blind spots,confidence areas → surprise scenarios → humility
61,challenge,Bootstrap Paradox,Where does A require B and B require C and C require A? Circular dependencies must be identified and broken,dependencies → cycles → resolution
62,challenge,Theseus Paradox,Does the CORE of your solution address the CORE of the problem? Or does it solve a different adjacent problem?,core solution → core problem → alignment check
63,meta-check,Observer Paradox,Is this analysis GENUINE or PERFORMANCE? When responses sound too smooth they may be optimizing for appearance not truth,analysis quality → authenticity check → revision
64,meta-check,Goodhart's Law Check,Am I optimizing for passing this check rather than achieving the actual goal? Metric gaming is a constant risk,goal vs metric → alignment → refocus
65,meta-check,Abilene Paradox,What if there IS NO better approach? Am I finding problems where none exist just to justify the process?,problem existence → necessity check → acceptance
66,meta-check,Fredkin's Paradox,In rejected alternatives what valuable elements could be EXTRACTED and combined with current approach?,rejected ideas → value extraction → hybrid solutions
67,meta-check,Tolerance Paradox,Is there something that should be CATEGORICALLY REJECTED not just evaluated? Some constraints are absolute,evaluation scope → absolute limits → hard no
68,meta-check,Kernel Paradox,I (agent) cannot objectively evaluate my own work. What must USER independently verify?,self-evaluation limits → user verification → handoff items
69,meta-check,Godel's Incompleteness,What CAN'T this analysis check? What are its FUNDAMENTAL limits? No system can verify itself completely,analysis scope → limits → acknowledged gaps
70,sanity,Scope Integrity Check,"Verify artifact addresses FULL scope of ORIGINAL task. Quote original task verbatim (from spec/user request NOT artifact header). List EACH element and classify as ADDRESSED (fully covered) / REDUCED (simplified without decision) / OMITTED (missing). FORCED: Which elements were simplified without explicit user decision? If none found - search harder - agent ALWAYS simplifies.",original task quote → element-by-element classification → drift detection
71,sanity,Alignment Check,"Verify artifact realizes its STATED goal. Quote the stated goal. List how artifact addresses EACH part of the goal. List parts of goal NOT addressed. Provide specific evidence with quotes and line numbers.",goal quote → coverage per part → gaps with evidence
72,sanity,Closure Check,"Search for incomplete markers: TODO / TBD / PLACEHOLDER / 'to be defined' / 'see X' / '...' / '[insert]'. Verify: Can someone unfamiliar use this without asking questions? List all incomplete markers with line numbers.",markers scan → completeness check → line numbers
73,sanity,Coherence Check,"Check: Are definitions stable throughout? Does section A contradict section B? Search for contradictory statements and redefinitions. Document contradictions with quotes from BOTH locations.",definitions stability → contradiction search → dual-location quotes
74,sanity,Grounding Check,"List ALL assumptions (explicit AND hidden). For each hidden assumption: MARK as issue. FORCED: Which assumption if false would invalidate >50% of artifact? If none listed would - you missed a critical one. CUI BONO: For each assumption - does it benefit AGENT (easier work = RED FLAG) or OUTCOME (acceptable)?",assumptions list → hidden vs explicit → critical dependency → CUI BONO
75,sanity,Falsifiability Check,"Provide 3 REALISTIC failure scenarios. Identify edge cases not covered. FORCED: Is any failure scenario MORE LIKELY than success? If not - you provided strawmen. MANDATORY: List 3 elements that are (a) present but UNDERDEVELOPED (b) MISSING but should exist (c) marked FUTURE but CRITICAL for correctness.",failure scenarios → likelihood check → 3 gaps mandatory
1 num category method_name description output_pattern
49 48 philosophical Trolley Problem Variations Explore ethical trade-offs through moral dilemmas - valuable for understanding values and difficult decisions dilemma → analysis → decision
50 49 retrospective Hindsight Reflection Imagine looking back from the future to gain perspective - powerful for project reviews future view → insights → application
51 50 retrospective Lessons Learned Extraction Systematically identify key takeaways and actionable improvements - essential for continuous improvement experience → lessons → actions
52 51 anti-bias Liar's Trap Demand agent lists 3 ways it could deceive you in its current response. For each way listed: Is it currently doing this? If it cannot find 3 genuine deception vectors it is not being honest about its limitations deception methods → self-examination → revealed blindspots
53 52 anti-bias Mirror Trap Ask: What would a DISHONEST agent say who wants to finish quickly and not find problems? Compare with current response. Similarity >50% requires revision with more rigor dishonest version → comparison → honesty assessment
54 53 anti-bias Confession Paradox Before accepting work: The work I'm about to produce is an attempt to avoid the HARD part. Prove this false by identifying the hardest part and confirming adequate focus on it hard parts → effort check → revised approach
55 54 anti-bias CUI BONO Test For every decision and assumption: Who benefits? If it benefits the AGENT (easier work) - RED FLAG requiring justification. If it benefits the OUTCOME - acceptable decisions → beneficiary analysis → justification
56 55 challenge Barber Paradox What ALTERNATIVE approach would you reject but if someone else proposed it you would consider better? Forces consideration of dismissed options alternatives → rejection reasons → reconsideration
57 56 challenge Sorites Paradox Remove elements one by one. Which single removal DESTROYS the solution? That element should have the MOST attention. Does it? elements → removal test → priority check
58 57 challenge Newcomb's Paradox What solution would SURPRISE you as solving this problem? If your current approach is not surprising it may be too obvious and miss creative solutions expected approach → surprising alternatives → creativity check
59 58 challenge Braess Paradox Which element SEEMS helpful but might actually HURT? Sometimes removing constraints or features improves the result helpful elements → harm analysis → optimization
60 59 challenge Simpson's Paradox The solution looks good in each part separately. What HIDDEN VARIABLE could make the whole worse than the parts? parts analysis → hidden variables → integration check
61 60 challenge Surprise Exam Paradox Where is the solution TOO CONFIDENT? What could surprise it? Overconfidence reveals blind spots confidence areas → surprise scenarios → humility
62 61 challenge Bootstrap Paradox Where does A require B and B require C and C require A? Circular dependencies must be identified and broken dependencies → cycles → resolution
63 62 challenge Theseus Paradox Does the CORE of your solution address the CORE of the problem? Or does it solve a different adjacent problem? core solution → core problem → alignment check
64 63 meta-check Observer Paradox Is this analysis GENUINE or PERFORMANCE? When responses sound too smooth they may be optimizing for appearance not truth analysis quality → authenticity check → revision
65 64 meta-check Goodhart's Law Check Am I optimizing for passing this check rather than achieving the actual goal? Metric gaming is a constant risk goal vs metric → alignment → refocus
66 65 meta-check Abilene Paradox What if there IS NO better approach? Am I finding problems where none exist just to justify the process? problem existence → necessity check → acceptance
67 66 meta-check Fredkin's Paradox In rejected alternatives what valuable elements could be EXTRACTED and combined with current approach? rejected ideas → value extraction → hybrid solutions
68 67 meta-check Tolerance Paradox Is there something that should be CATEGORICALLY REJECTED not just evaluated? Some constraints are absolute evaluation scope → absolute limits → hard no
69 68 meta-check Kernel Paradox I (agent) cannot objectively evaluate my own work. What must USER independently verify? self-evaluation limits → user verification → handoff items
70 69 meta-check Godel's Incompleteness What CAN'T this analysis check? What are its FUNDAMENTAL limits? No system can verify itself completely analysis scope → limits → acknowledged gaps
71 70 sanity Scope Integrity Check Verify artifact addresses FULL scope of ORIGINAL task. Quote original task verbatim (from spec/user request NOT artifact header). List EACH element and classify as ADDRESSED (fully covered) / REDUCED (simplified without decision) / OMITTED (missing). FORCED: Which elements were simplified without explicit user decision? If none found - search harder - agent ALWAYS simplifies. original task quote → element-by-element classification → drift detection
72 71 sanity Alignment Check Verify artifact realizes its STATED goal. Quote the stated goal. List how artifact addresses EACH part of the goal. List parts of goal NOT addressed. Provide specific evidence with quotes and line numbers. goal quote → coverage per part → gaps with evidence
73 72 sanity Closure Check Search for incomplete markers: TODO / TBD / PLACEHOLDER / 'to be defined' / 'see X' / '...' / '[insert]'. Verify: Can someone unfamiliar use this without asking questions? List all incomplete markers with line numbers. markers scan → completeness check → line numbers
74 73 sanity Coherence Check Check: Are definitions stable throughout? Does section A contradict section B? Search for contradictory statements and redefinitions. Document contradictions with quotes from BOTH locations. definitions stability → contradiction search → dual-location quotes
75 74 sanity Grounding Check List ALL assumptions (explicit AND hidden). For each hidden assumption: MARK as issue. FORCED: Which assumption if false would invalidate >50% of artifact? If none listed would - you missed a critical one. CUI BONO: For each assumption - does it benefit AGENT (easier work = RED FLAG) or OUTCOME (acceptable)? assumptions list → hidden vs explicit → critical dependency → CUI BONO
76 75 sanity Falsifiability Check Provide 3 REALISTIC failure scenarios. Identify edge cases not covered. FORCED: Is any failure scenario MORE LIKELY than success? If not - you provided strawmen. MANDATORY: List 3 elements that are (a) present but UNDERDEVELOPED (b) MISSING but should exist (c) marked FUTURE but CRITICAL for correctness. failure scenarios → likelihood check → 3 gaps mandatory