Session 3: Human vs. AI Synthesis — A Replication Experiment

The gap between human and machine synthesis is where real judgment lives.

In the previous session, we built a toolkit for working with AI. Now, we put that toolkit to the test: synthesis. Can AI replicate the deep, nuanced, and creative work of an expert human researcher in creating new theory from a body of literature?

This guide outlines a hands-on "Replication Experiment" designed to explore this question. The goal isn't to find a "winner," but to learn from the gap between human and machine. By critically analyzing where AI succeeds and where it falls short, we develop the judgment needed for effective interpretive orchestration.

Learning outcomes

By the end of this session, you'll be able to:

• Analyze the cognitive processes and judgments that expert researchers use to create novel theoretical syntheses from literature
• Apply advanced, multi-step prompting techniques to guide AI through complex, end-to-end synthesis tasks
• Critically compare AI-driven synthesis with human-authored work, identifying strengths and weaknesses of each approach
• Recognize which synthesis tasks work well with AI assistance (e.g., pattern identification, thematic clustering) and which benefit most from direct human engagement (e.g., theoretical innovation, critical judgment)
• Adapt your research workflow based on practical understanding of AI's current capabilities and limitations

The replication experiment framework

    +-----------------------+
    | Shared literature set |
    +-----------+-----------+
                |
       +--------+--------+
       |                 |
       v                 v
+------+-------+   +-----+--------+
| Human path   |   | AI path     |
+------+-------+   +-----+--------+
       |                 |
 deep reading      batch processing
 pattern insight    pattern detection
 critical judgment  consistency check
       |                 |
       +--------+--------+
                |
                v
       critical comparison
                |
                v
       delegation strategy

The gap between human and AI synthesis shows where each approach is strongest: what AI misses identifies where human expertise matters and where delegation can help.

The human blueprints: case study papers

The foundation for the Replication Experiment is the analysis of expert human work. The two papers below are blueprints for two distinct approaches to research synthesis. We dissect their methodology, then design an AI workflow to replicate it.

The replication experiment: session structure

Part A: analyze the human blueprint

• Deconstruct the methodology: Carefully read the methodology sections of the case study papers. What were the exact cognitive steps the authors took?
• Identify decisions: Where did the authors exercise judgment about inclusion, exclusion, theme naming, or theoretical connections?

Part B: design the AI protocol

• Translate steps into prompts: Convert the cognitive steps you identified into a multi-step "cognitive blueprint" for an AI to follow.
• Use a large context model: Use an AI model with a large context window to process the entire collection of papers from the case study. The AI Model Reference Guide names the current options.

Part C: compare and reflect

• Analyze the gap: Compare the AI's output with the published human synthesis.
• Identify strengths and weaknesses: What did the AI capture well? What nuances or creative leaps did it miss?
• Document failures: Use the Failure Museum template to document the specific ways the AI fell short.

This experiment shows where human expertise remains irreplaceable and how to build a partnership in which AI contributes without displacing human expertise.

Quality Control Checklist

Before accepting any AI-generated research output, verify:

Citation Verification

• All citations have complete metadata (author, year, title, journal)
• DOIs or URLs provided and functional
• Page numbers included for specific claims
• Citations cross-referenced with Zotero library or Google Scholar
• No anachronistic attributions (concepts to wrong time periods)

Logical Consistency

• No internal contradictions in the analysis
• Arguments follow logically from evidence
• Scope and boundary conditions clearly stated
• Limitations and caveats acknowledged
• Alternative interpretations considered

Paradigm Alignment

• Methodology matches epistemological stance
• Language appropriate for paradigm (no "variables" for interpretive work)
• Theoretical tradition respected
• Disciplinary conventions followed

Originality & Depth

• Goes beyond surface-level summary
• Identifies non-obvious patterns or connections
• Provides novel insights or frameworks
• Demonstrates critical engagement with sources
• Avoids generic platitudes

Contextual Appropriateness

• Sample characteristics specified (industry, geography, time period)
• Generalizability limits acknowledged
• Cultural and historical context preserved
• Boundary conditions identified

If any checkboxes remain unchecked: Revise your prompt and regenerate output.

See also: Failure Museum for detailed failure modes

Related Resources

Navigation: Previous: Session 2 • Return to Case Study Overview • Next: Session 4