Skip to content
Research Memex

AI Environment

PDF to Markdown Conversion Guide

A guide to converting PDF research papers into clean Markdown text using OCR tools, a crucial step for making your literature accessible to AI models.

Overview

Converting PDF documents to text formats is essential for effective AI workflows in research. Raw PDFs often contain complex layouts, images, and formatting that can interfere with AI processing, embeddings, and analysis. This guide covers multiple methods for converting PDFs to clean, AI-friendly text formats.

Key Benefits:

  • Better AI Processing: Clean text without layout artifacts or formatting issues
  • Cost Efficiency: Pre-convert PDFs once instead of processing them repeatedly
  • Token Optimization: Text formats use fewer tokens than PDF processing
  • Embedding Quality: Consistent text extraction improves embedding accuracy
  • Workflow Integration: Text files work seamlessly with all AI tools and models

Common Use Cases:

  • Research literature analysis
  • AI-assisted content summarization
  • Embedding creation for semantic search
  • Large language model context preparation

PDF Conversion Methods

Method 1: MinerU MCP (Recommended)

Best for: Claude Code workflows, batch processing, high accuracy

Info

Integrated workflow: Parse PDFs directly within Claude without context-switching. See the full MinerU MCP guide for detailed setup and usage.

MinerU MCP integrates document parsing directly into your AI workflow:

  • 90%+ accuracy with VLM mode for complex layouts
  • Batch processing up to 200 documents at once
  • 109 languages supported via OCR
  • Table and formula recognition

Quick Setup:

claude mcp add mineru-mcp -e MINERU_API_KEY=your-key -- npx mineru-mcp

Then simply ask Claude: "Parse this PDF with VLM mode: [URL]"

Pros: Integrated workflow, high accuracy, batch capable Cons: Requires API key from mineru.net

Method 2: MinerU Web Interface (Free Tier)

Best for: Quick one-off conversions, testing before MCP setup

Use MinerU without MCP setup via their web interface:

  1. Visit MinerU or GitHub
  2. Use the web interface for quick conversions
  3. Download converted markdown

Free Tier: Limited daily conversions Pros: No setup required, good quality Cons: Daily limits, manual download process

Method 3: Mistral OCR Script (Batch Offline)

Best for: Very large offline batch jobs, scripted workflows

For bulk processing outside of Claude (100+ papers at once), use the Mistral OCR script approach.

Trade-off: MinerU MCP is better for integrated Claude workflows. Mistral script is better for massive offline batch jobs.

Method 4: Manual Copy-Paste (Fallback)

Best for: Emergency single documents, when other methods unavailable

  1. Open PDF in a PDF reader
  2. Select and copy text from each page
  3. Paste into a text editor or Markdown file
  4. Save with .md or .txt extension

Limitations: Time-consuming, layout issues, manual errors. Use MinerU instead.

Mistral OCR API Setup (Optional)

The case study provides a ready-to-use script (batch_ocr.py) that leverages Mistral's OCR API for high-quality PDF conversion.

Step 1: Get Mistral API Key

  1. Visit Mistral AI Console
  2. Create an account or sign in
  3. Navigate to API Keys section
  4. Click "Create new key"
  5. Copy and save your API key securely

Pricing: ~$0.001 per page (varies by document complexity)

Step 2: Environment Setup

Set your API key as an environment variable:

export MISTRAL_API_KEY="your_api_key_here"

Step 3: Using the Batch OCR Script

The case study provides batch_ocr.py for automated conversion:

# Basic usage
python batch_ocr.py input_folder output_folder

# Example
python batch_ocr.py readings/pdfs readings/markdown

Script Features:

  • ✅ Recursive folder processing
  • ✅ Maintains original folder structure
  • ✅ Batch processing for efficiency
  • ✅ Automatic cleanup of temporary files
  • ✅ Markdown output with page breaks

File Size Limits: Default 36MB per PDF (configurable)

Alternative API Options

Google Document AI

Best for: Google ecosystem integration

  1. Visit Google Cloud Console
  2. Enable Document AI API
  3. Create a processor for OCR
  4. Use Python client library for batch processing

Azure Form Recognizer

Best for: Enterprise environments

  1. Visit Azure Portal
  2. Create Cognitive Services resource
  3. Use Form Recognizer service
  4. REST API or SDK integration

Best Practices and Tips

File Organization

your-project/
├── pdfs/           # Original PDFs
│   ├── session-1/
│   ├── session-2/
│   └── articles/
└── markdown/       # Converted text files
    ├── session-1/
    ├── session-2/
    └── articles/

Quality Control

  • Spot Check: Review converted files for accuracy
  • Complex Layouts: Some academic PDFs may need manual review
  • Images/Tables: OCR may not capture complex visual elements
  • Languages: Ensure API supports your document languages

Cost Optimization

  • Batch Processing: Convert all PDFs at once rather than individually
  • File Size Limits: Be aware of API size restrictions
  • Free Tiers: Use free options for small projects
  • Caching: Store converted files to avoid re-processing

Workflow Integration

  • Version Control: Track both original PDFs and converted text
  • Backup: Keep original PDFs as source of truth
  • Naming: Maintain consistent file naming conventions
  • Metadata: Preserve citation information alongside converted text

Troubleshooting

Common Issues

"API Key Not Found"

  • Verify environment variable is set: echo $MISTRAL_API_KEY
  • Restart your terminal/command prompt
  • Check for typos in variable name

"File Too Large"

  • Default limit is 36MB per PDF
  • Split large documents or use --max-size parameter
  • Consider alternative conversion methods for very large files

"Processing Failed"

  • Check PDF file integrity
  • Some PDFs may have copy protection
  • Try alternative conversion methods

"Rate Limits Exceeded"

  • APIs have rate limits (requests per minute/hour)
  • Implement delays between requests
  • Consider paid plans for higher limits

Integration with Case Study Workflow

Recommended Usage Pattern

  1. Initial Setup: Convert all PDFs at project start
  2. Ongoing: Convert new PDFs as they're added
  3. Processing: Use converted Markdown files for all AI workflows
  4. Storage: Maintain both original PDFs and converted text

Case Study References

  • AI-powered literature analysis
  • Human-AI synthesis workflows
  • Agentic workflow design
  • API Keys Guide: For Mistral setup
  • Model Reference Guide: For compatible AI models

Advanced Options

Next Steps

  1. Get Started: Choose your preferred conversion method
  2. Test: Convert a few sample PDFs
  3. Scale: Process your full document collection
  4. Integrate: Use converted files in your AI workflows
  5. Monitor: Track conversion quality and costs

Warning

Remember: Converting PDFs once saves time and costs in the long run compared to processing them repeatedly in AI workflows.