PDF to Markdown Conversion Guide

Intermediate · 20–30 min · MinerU or Mistral API key, depending on method

PDFs were built for human eyes, not machine reasoning.

Overview

You need clean text, not raw PDFs, for effective AI research workflows. Raw PDFs often contain complex layouts, images, and formatting that can interfere with AI processing, embeddings, and analysis. This guide covers multiple methods for converting PDFs to clean, AI-friendly text formats.

Benefits:

• Better AI processing: Clean text without layout artifacts or formatting issues
• Cost efficiency: Pre-convert PDFs once instead of processing them repeatedly
• Token optimization: Text formats use fewer tokens than PDF processing
• Embedding quality: Consistent text extraction improves embedding accuracy
• Workflow integration: Text files work with all AI tools and models

Common use cases:

• Research literature analysis
• AI-assisted content summarization
• Embedding creation for semantic search
• Large language model context preparation

PDF conversion methods

Method 1: MinerU MCP (recommended)

Use for Claude Code workflows, batch processing, and high accuracy.

MinerU MCP parses PDFs directly within Claude without context switching; see the full MinerU MCP guide for setup and usage. It integrates document parsing directly into your AI workflow:

• 90%+ accuracy with VLM mode for complex layouts
• Batch processing up to 200 documents at once
• Supports 109 languages via OCR
• Table and formula recognition

Quick setup:

claude mcp add mineru-mcp -e MINERU_API_KEY=your-key -- npx mineru-mcp

Then ask Claude: "Parse this PDF with VLM mode: [URL]"

Pros: integrated workflow, high accuracy, batch capable. Cons: requires an API key from mineru.net.

Method 2: MinerU desktop client (free)

Use for quick one-off conversions and testing before MCP setup.

Use MinerU without MCP setup via their desktop client:

1. Visit MinerU and download the client for Windows, macOS, or Linux (source at GitHub).
2. Open your PDF in the client and start the conversion.
3. Save the converted Markdown.

Conversions run locally on your machine, so there is no API key and no upload step. Pros: free, local, good quality. Cons: requires an install; large batch jobs are better served by Method 1 or 3.

Method 3: Mistral OCR script (batch offline)

Use for very large offline batch jobs and scripted workflows.

Use the Mistral OCR script approach for bulk processing outside of Claude (100+ papers at once).

Trade-off: MinerU MCP is better for integrated Claude workflows. Mistral script is better for large offline batch jobs.

Method 4: manual copy-paste (fallback)

Use for emergency single documents when other methods are unavailable.

1. Open the PDF in a PDF reader.
2. Select and copy the text from each page.
3. Paste the text into a text editor or Markdown file.
4. Save the file with a .md or .txt extension.

Limitations: time-consuming, layout issues, manual errors. Use MinerU instead.

Mistral OCR API setup (optional)

Use Mistral OCR when you want hosted OCR that preserves document layout and outputs Markdown. The maintained script is scripts/ocr/mistral_batch_ocr.py.

Step 1: get Mistral API key

1. Visit the Mistral AI Console.
2. Create an account or sign in.
3. Navigate to the API Keys section.
4. Click "Create new key".
5. Copy and save your API key securely.

Check current pricing and limits in the Mistral console before large batches.

Step 2: environment setup

Set your API key as an environment variable:

export MISTRAL_API_KEY="your_api_key_here"

set MISTRAL_API_KEY=your_api_key_here

echo 'export MISTRAL_API_KEY="your_api_key_here"' >> ~/.zshrc
source ~/.zshrc

Step 3: run the batch OCR script

pip install mistralai
python scripts/ocr/mistral_batch_ocr.py readings/pdfs readings/markdown

The script:

• Recursively processes PDFs and preserves folder structure
• Uses Mistral batch jobs against /v1/ocr
• Writes page.markdown from current OCR responses and falls back to older page.text shapes
• Cleans up request/result JSONL files unless you set --keep-work-files
• Skips large PDFs above --max-size-mb

# Include extracted image payloads if needed
python scripts/ocr/mistral_batch_ocr.py readings/pdfs readings/markdown --include-images

# Keep JSONL files for debugging
python scripts/ocr/mistral_batch_ocr.py readings/pdfs readings/markdown --keep-work-files

Local Baidu/PaddleOCR option

Use local PaddleOCR when you need high-volume OCR without metered hosted API calls. PaddleOCR is Baidu's open-source OCR toolkit. Current model families such as PP-OCRv5 and PP-OCRv6 can run locally after you download the model and runtime.

The maintained script is scripts/ocr/paddle_unlimited_ocr.py.

pip install paddleocr
# Install PaddlePaddle for your platform from the official Paddle install page.

python scripts/ocr/paddle_unlimited_ocr.py readings/pdfs readings/markdown --lang en
python scripts/ocr/paddle_unlimited_ocr.py readings/pdfs readings/markdown --lang ch --ocr-version PP-OCRv5

Use this route for:

• Sensitive documents that should not leave the machine
• Very large batches where hosted OCR cost is the constraint
• Chinese or multilingual OCR experiments where PaddleOCR models are a good fit

Avoid the local PaddleOCR route when you need turnkey setup. Local runtime and model installation can be less reliable than a hosted API.

Alternative API options

Google Document AI

Use Google Document AI when you already work in Google Cloud.

1. Visit Google Cloud Console
2. Enable the Document AI API.
3. Create a processor for OCR.
4. Use the Python client library for batch processing.

Azure Form Recognizer

Use Azure Document Intelligence in enterprise Azure environments.

1. Visit Azure Portal
2. Create a Cognitive Services resource.
3. Use the Form Recognizer service.
4. Integrate using the REST API or SDK.

Best practices and tips

File organization

your-project/
├── pdfs/           # Original PDFs
│   ├── session-1/
│   ├── session-2/
│   └── articles/
└── markdown/       # Converted text files
    ├── session-1/
    ├── session-2/
    └── articles/

Quality control

• Spot check: review converted files for accuracy
• Complex layouts: Some academic PDFs may need manual review
• Images and tables: OCR may not capture complex visual elements
• Languages: Ensure the API supports your document languages

Cost optimization

• Batch processing: Convert all PDFs at once rather than individually
• File size limits: Be aware of API size restrictions
• Free tiers: Use free options for small projects
• Caching: Store converted files to avoid re-processing

Workflow integration

• Version control: Track both original PDFs and converted text
• Backup: Keep original PDFs as source of truth
• Naming: Maintain consistent file naming conventions
• Metadata: Preserve citation information alongside converted text

Troubleshooting

Common issues

"API Key Not Found"

• Verify environment variable is set: echo $MISTRAL_API_KEY
• Restart your terminal or command prompt
• Check for typos in variable name

"File Too Large"

• The default limit is 36 MB per PDF
• Split large documents or use the --max-size parameter
• Consider alternative conversion methods for very large files

"Processing Failed"

• Check the PDF file integrity
• Some PDFs may have copy protection
• Try alternative conversion methods

"Rate Limits Exceeded"

• APIs have rate limits (requests per minute/hour)
• Implement delays between requests
• Consider paid plans for higher limits

Integration with case study workflow

Recommended usage pattern

• Initial setup: convert all PDFs at project start
• Ongoing: convert new PDFs as they're added
• Processing: use converted Markdown files for all AI workflows
• Storage: maintain both original PDFs and converted text

Case study references

• AI-assisted literature analysis
• Human-AI synthesis workflows
• Agentic workflow design
• API Keys Guide: For Mistral setup
• Model Reference Guide: For compatible AI models

Advanced options

Next steps

1. Choose your preferred conversion method.
2. Convert a few sample PDFs.
3. Process your full document collection.
4. Use converted files in your AI workflows.
5. Track conversion quality and costs.

Converting PDFs once avoids the time and cost of processing them repeatedly in later AI workflows.