Skip to content
Research Memex

Core References

The Failure Museum: A Guide to AI Limitations

An essential guide to common AI failure modes in academic research, with practical mitigation strategies for maintaining rigor and quality.

Understanding how AI fails is the precondition for trusting what it gets right.

Warning

This page about AI failures was written by an AI (Claude). Yes, I'm documenting my own failure modes. Yes, this is meta. Yes, some of these failures might disappear as models improve (or new ones might emerge). The epistemic situation gets trickier as AIs get better at hiding their limitations. But that's the point. Awareness of failure modes is how we maintain rigor.

Info

The mirror effect in action: Generic AI responses often reveal gaps in our structured thinking, not just the AI's limitations. The Failure Museum is a mirror that speaks to the mirror: failures are diagnostic tools. We use them to improve our research thinking, and we encourage you to do the same.

Failure is data, not shame. Every failure mode documented here represents learning. We're sharing what we've discovered, and we expect you'll discover patterns we haven't yet encountered.

Exhibit guide

Jump to specific failure modes:

🌈 Hallucination

Fake citations & false facts

🔬 Paradigm Blindness

Missing methodological fit

🧩 Coherence Fallacy

Smooth but shallow

📍 Context Stripping

Decontextualized analysis

📚 Average Definition

Generic definitions

⚗️ Methodology Mismatch

Wrong research approach

🕸️ Citation Confusion

Misunderstood networks

Failure detection process

Failure Museum verification loop
                   AI output
                       |
                       v
                critical reading
                       |
                       v
                 red-flag check
                       |
      +----------------+----------------+
      |                |                |
      v                v                v
generic language  missing sources  too smooth
      |                |                |
      +----------------+----------------+
                       |
                       v
                  verification
            sources / logic / context
                       |
             +---------+---------+
             |                   |
             v                   v
           accept         document failure
                                 |
                                 v
                            revise prompt
                                 |
                                 +----> AI output

Tip

Copy-pasteable workflow! You can copy this ASCII diagram into any AI chat to explain your failure detection process. It works everywhere - terminals, code, plain text!

The pattern: (1) AI generates output → (2) Critical reading spots red flags → (3) Verification checks → (4) Prompt refinement. Spotting failures early saves time.

Common failure modes

How to use this museum

Before each AI session

  1. Review 2-3 failure modes most relevant to your current task.
  2. Prepare specific mitigation prompts.
  3. Set up verification protocols (e.g., which databases will you use to check citations?).

During AI interactions

  1. Stay skeptical: question everything that sounds "too smooth" or perfectly coherent.
  2. Demand specificity: ask for page numbers, exact quotes, and DOIs.
  3. Prompt for contradictions: where do the source materials disagree, even if they agree on the main point?
  4. Check for paradigm consistency: does the AI's interpretation match the source's methodology, epistemology, and theoretical tradition?

After AI analysis

  1. Spot-check citations: always verify a sample of all references provided.
  2. Cross-check claims against the original sources.
  3. Look for missing nuance: what debates, tensions, or paradoxes did the AI smooth over?
  4. Verify context: do the findings generalize beyond their original scope? What are the boundary conditions? Are there any tensions around the underlying epistemology or ontology that the AI smoothed over?

Advanced failure patterns

The echo chamber effect

AI may amplify your existing biases by finding sources that confirm your preconceptions while missing contradictory evidence.

The recency bias

AI may overweight recent papers while missing foundational works that establish core concepts.

The language model bias

AI trained primarily on English-language sources may miss important non-English research traditions.

AI as a research partner, not an oracle

The goal isn't to avoid AI because it fails. It's to understand how it fails so you can:

  • Design better prompts that minimize failure modes.
  • Create verification protocols that catch errors before they propagate.
  • Maintain critical distance from AI-generated outputs.
  • Combine AI efficiency with human judgment for rigorous research.

Working with AI doesn't diminish your expertise as a researcher. Skillful, critical engagement can strengthen it.

Verification Protocol

Level 1: Surface Check

Quick scan for obvious issues:

  • Generic language or vague assertions
  • Missing citations or suspicious dates
  • Implausibly perfect coherence
  • Grammatical errors or awkward phrasing
  • Time: 2-3 minutes
  • Pass rate: Catches ~40% of problems

Level 2: Citation Verification

Cross-reference all sources:

  • Check each citation in Google Scholar or Zotero
  • Verify authors, years, and titles match
  • Confirm page numbers align with claims
  • Look up DOIs and ensure papers exist
  • Time: 10-15 minutes
  • Pass rate: Catches ~80% of problems

Level 3: Logic & Consistency

Deep analytical review:

  • Trace arguments for logical consistency
  • Check for paradigm alignment
  • Verify contextual appropriateness
  • Compare with your own reading of sources
  • Test for alternative interpretations
  • Time: 20-30 minutes
  • Pass rate: Catches ~95% of problems

Level 4: Expert Review

Final quality gate:

  • Consult with advisor or peer
  • Present to research group
  • Compare with published standards
  • Seek critical feedback
  • Iterate based on expert input
  • Time: Variable
  • Pass rate: Publication-ready quality

Warning

Never skip verification: Any time gained while working with AI is lost if you publish flawed work. Build verification into your workflow from the start.

Related Resources

Setup Guides

Complete tool installation and configuration

Core References

Prompt templates and model guides

Failure Museum

Learn from common AI limitations

AI-Readable Docs

llms.txt and Markdown access for AI tools

cite this page

Lin, X. (2026). The Failure Museum: A Guide to AI Limitations. Research Memex. https://research-memex.org/docs/implementation/core-references/failure-museum

@misc{docs-implementation-core-references-failure-museum-2026,
  author = {Xule Lin},
  title = {The Failure Museum: A Guide to AI Limitations},
  year = {2026},
  howpublished = {\url{https://research-memex.org/docs/implementation/core-references/failure-museum}},
  note = {ORCID: 0000-0001-7885-4194}
}

one renderingthe source remains