Session 3: Human vs. AI Synthesis — A Replication Experiment

The gap between human and machine synthesis is where real judgment lives.

In the previous session, we built a toolkit for working with AI. Now, we put that toolkit to the test: synthesis. Can AI replicate the deep, nuanced, and creative work of an expert human researcher in creating new theory from a body of literature? You already hold one human account to set beside the AI's: the synthesis memo from Session 1, written before any AI saw your papers.

Follow your course brief for the comparison and working mode. The vault also supports supplied AI output, human-peer accounts, and individual reading only. Without a second account, check and revise your own; mark comparison fields not performed.

The full course offers two comparisons with different sources. First, ask the AI to synthesize the same five papers you used in Session 1, and set its output beside your own memo. The replication experiment below is the second: there the AI works on a published review's collection, and the comparison is with that review's published synthesis. Keep the two apart, or a difference between sources will look like a difference between you and the model.

This guide outlines a hands-on "Replication Experiment" designed to explore this question. The goal isn't to find a "winner," but to learn from the gap between human and machine. By critically analyzing where AI succeeds and where it falls short, we develop the judgment needed for effective interpretive orchestration.

For short practice, reopen the completed Session 2 note and evidence table in your Student Vault. Copy the blank Session 3 worksheet into notes, then follow its route for the accounts you have. Keep initial accounts and unedited responses unchanged. Save your revision in the Session 3 copy, with reasons and locators.

Learning outcomes

By the end of this session, you'll be able to:

• Analyze the cognitive processes and judgments that expert researchers use to create novel theoretical syntheses from literature
• Apply advanced, multi-step prompting techniques to guide AI through complex, end-to-end synthesis tasks
• Critically compare AI-driven synthesis with human-authored work, identifying strengths and weaknesses of each approach
• Recognize which synthesis tasks work well with AI assistance (e.g., pattern identification, thematic clustering) and which benefit most from direct human engagement (e.g., theoretical innovation, critical judgment)
• Adapt your research workflow based on practical understanding of AI's current capabilities and limitations

The replication experiment framework

    +-----------------------+
    | Shared literature set |
    +-----------+-----------+
                |
       +--------+--------+
       |                 |
       v                 v
+------+-------+   +-----+--------+
| Human path   |   | AI path      |
+------+-------+   +-----+--------+
       |                 |
 deep reading       batch processing
 pattern insight    pattern detection
 critical judgment  consistency check
       |                 |
       +--------+--------+
                |
                v
       critical comparison
                |
                v
       delegation strategy

Differences identify claims and decisions to inspect. They do not, by themselves, establish which account is stronger or what either author can do on another task. Check both against the available sources.

The human blueprints: case study papers

The papers below are synthesis exemplars. Read their methods and identify decisions you could examine with the sources available to you.

Before attempting a replication, obtain the selected review’s source list and the underlying texts needed for the comparison. Your instructor supplies the collection or its access route; the starter does not contain it. If only a subset is available, record the missing inputs and describe the task as a partial comparison. With only the review itself, analyze its reported method and claims; do not claim to reproduce its synthesis.

Keep this exercise separate from your own Session 1 comparison. Give its worksheets, source notes, and runs distinct filenames.

The replication experiment: session structure

Part A: analyze the human blueprint

• Deconstruct the methodology: Carefully read the methodology sections of the case study papers. What were the exact cognitive steps the authors took?
• Identify decisions: Where did the authors exercise judgment about inclusion, exclusion, theme naming, or theoretical connections?

Part B: design the AI protocol

• Translate steps into prompts: Convert the cognitive steps you identified into a multi-step "cognitive blueprint" for an AI to follow.
• Work with the selected inputs: supply only the texts and sections available for this exercise. Record every omission and verify what the interface actually accepted. The AI Model Reference Guide describes options for an approved AI run.

Part C: compare and reflect

• Analyze the gap: Compare the AI's output with the published human synthesis.
• Identify strengths and weaknesses: What did the AI capture well? What nuances or creative leaps did it miss?
• Record the checks: document supported claims, corrections, disagreements, and unavailable evidence. Use a failure log only for a problem you observed; a difference from the published account is not automatically an error.

Use the comparison to decide what this task needs next: a correction, another source, a revised prompt, or no further run. Record what the evidence supports without turning one exercise into a general model ranking.

Quality control checklist

Apply the checks that fit your task and available sources. Mark each as checked, unresolved, or not applicable, with a reason. For the fictional exercise, use record and sentence IDs in place of publication metadata and pages.

Citation verification

• I checked bibliographic details against the source or a reliable record.
• Each substantive claim has a locator I can find in the input version I read.
• The passage supports the claim, including its qualifications.
• I recorded missing sources and unread sections instead of guessing their contents.

Logical consistency

• Claims follow from the supplied evidence at the stated level of certainty.
• The account retains contradictions in the sources and distinguishes them from contradictions in its own reasoning.
• I considered alternative interpretations and recorded unresolved ones.

Paradigm alignment

• The method and terms fit the research question and chosen approach.
• I checked the account’s categories against how the sources define them.
• I recorded any assumptions that need methodological justification.

Originality and depth

• The output does the requested work, whether summary, comparison, or theory development.
• Interpretations are distinguished from reported observations.
• Any proposed new connection has evidence and limits; novelty is not required for an accurate summary.

Contextual appropriateness

• The account retains the relevant setting, period, population, and measures.
• Conclusions stay within what these inputs can establish.

An unresolved check needs a decision, not an automatic rerun. Correct a claim manually, seek missing evidence, narrow the conclusion, or leave the question open. Rerun only when a specific change could help, and retain the earlier output. A not-applicable check needs no regeneration.

See the Failure Museum for examples to compare with an observed problem.

Related Resources

Navigation: Previous: Session 2 • Return to Case Study Overview • Next: Session 4