# PDF to Markdown Conversion Guide

URL: https://research-memex.org/docs/implementation/ai-environment-setup/ocr-pdf-conversion-guide
Description: Convert PDF research papers into clean Markdown so AI tools can read, quote, and analyze them more reliably.



<GuideMeta>
  Intermediate · 20–30 min · MinerU or Mistral API key, depending on method
</GuideMeta>

*PDFs were built for human eyes, not machine reasoning.*

## Overview [#overview]

You need clean text, not raw PDFs, for effective AI research workflows. Raw PDFs often contain complex layouts, images, and formatting that can interfere with AI processing, embeddings, and analysis. This guide covers multiple methods for converting PDFs to clean, AI-friendly text formats.

Benefits:

* Better AI processing: Clean text without layout artifacts or formatting issues
* Cost efficiency: Pre-convert PDFs once instead of processing them repeatedly
* Token optimization: Text formats use fewer tokens than PDF processing
* Embedding quality: Consistent text extraction improves embedding accuracy
* Workflow integration: Text files work with all AI tools and models

Common use cases:

* Research literature analysis
* AI-assisted content summarization
* Embedding creation for semantic search
* Large language model context preparation

## PDF conversion methods [#pdf-conversion-methods]

### Method 1: MinerU MCP (recommended) [#method-1-mineru-mcp-recommended]

Use for Claude Code workflows, batch processing, and high accuracy.

MinerU MCP parses PDFs directly within Claude without context switching; see the full [MinerU MCP guide](/docs/toolkit/mineru-mcp) for setup and usage. It integrates document parsing directly into your AI workflow:

* 90%+ accuracy with VLM mode for complex layouts
* Batch processing up to 200 documents at once
* Supports 109 languages via OCR
* Table and formula recognition

Quick setup:

```bash
claude mcp add mineru-mcp -e MINERU_API_KEY=your-key -- npx mineru-mcp
```

Then ask Claude: "Parse this PDF with VLM mode: \[URL]"

Pros: integrated workflow, high accuracy, batch capable.
Cons: requires an API key from [mineru.net](https://mineru.net).

***

### Method 2: MinerU desktop client (free) [#method-2-mineru-desktop-client-free]

Use for quick one-off conversions and testing before MCP setup.

Use MinerU without MCP setup via their desktop client:

1. Visit [MinerU](https://mineru.net/) and download the client for Windows, macOS, or Linux (source at [GitHub](https://github.com/opendatalab/MinerU)).
2. Open your PDF in the client and start the conversion.
3. Save the converted Markdown.

Conversions run locally on your machine, so there is no API key and no upload step.
Pros: free, local, good quality.
Cons: requires an install; large batch jobs are better served by Method 1 or 3.

***

### Method 3: Mistral OCR script (batch offline) [#method-3-mistral-ocr-script-batch-offline]

Use for very large offline batch jobs and scripted workflows.

Use the Mistral OCR script approach for bulk processing outside of Claude (100+ papers at once).

Trade-off: MinerU MCP is better for integrated Claude workflows. Mistral script is better for large offline batch jobs.

***

### Method 4: manual copy-paste (fallback) [#method-4-manual-copy-paste-fallback]

Use for emergency single documents when other methods are unavailable.

1. Open the PDF in a PDF reader.
2. Select and copy the text from each page.
3. Paste the text into a text editor or Markdown file.
4. Save the file with a `.md` or `.txt` extension.

Limitations: time-consuming, layout issues, manual errors. Use MinerU instead.

## Mistral OCR API setup (optional) [#mistral-ocr-api-setup-optional]

Use Mistral OCR when you want hosted OCR that preserves document layout and outputs Markdown. The maintained script is [scripts/ocr/mistral\_batch\_ocr.py](/scripts/ocr/mistral_batch_ocr.py).

### Step 1: get Mistral API key [#step-1-get-mistral-api-key]

1. Visit the [Mistral AI Console](https://console.mistral.ai/).
2. Create an account or sign in.
3. Navigate to the **API Keys** section.
4. Click &#x2A;*"Create new key"**.
5. Copy and save your API key securely.

Check current pricing and limits in the Mistral console before large batches.

### Step 2: environment setup [#step-2-environment-setup]

Set your API key as an environment variable:

<CodeGroup>
  ```bash macOS/Linux
  export MISTRAL_API_KEY="your_api_key_here"
  ```

  ```powershell Windows
  set MISTRAL_API_KEY=your_api_key_here
  ```

  ```bash Persistent (macOS/Linux)
  echo 'export MISTRAL_API_KEY="your_api_key_here"' >> ~/.zshrc
  source ~/.zshrc
  ```
</CodeGroup>

### Step 3: run the batch OCR script [#step-3-run-the-batch-ocr-script]

```bash
pip install mistralai
python scripts/ocr/mistral_batch_ocr.py readings/pdfs readings/markdown
```

The script:

* Recursively processes PDFs and preserves folder structure
* Uses Mistral batch jobs against `/v1/ocr`
* Writes `page.markdown` from current OCR responses and falls back to older `page.text` shapes
* Cleans up request/result JSONL files unless you set `--keep-work-files`
* Skips large PDFs above `--max-size-mb`

```bash
# Include extracted image payloads if needed
python scripts/ocr/mistral_batch_ocr.py readings/pdfs readings/markdown --include-images

# Keep JSONL files for debugging
python scripts/ocr/mistral_batch_ocr.py readings/pdfs readings/markdown --keep-work-files
```

## Local Baidu/PaddleOCR option [#local-baidupaddleocr-option]

Use local [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) when you need high-volume OCR without metered hosted API calls. PaddleOCR is Baidu's open-source OCR toolkit. Current model families such as PP-OCRv5 and PP-OCRv6 can run locally after you download the model and runtime.

The maintained script is [scripts/ocr/paddle\_unlimited\_ocr.py](/scripts/ocr/paddle_unlimited_ocr.py).

<Aside label="WHAT 'UNLIMITED' MEANS">
  No per-page API meter. Your machine, the PaddlePaddle runtime, and model downloads still cap throughput.
</Aside>

```bash
pip install paddleocr
# Install PaddlePaddle for your platform from the official Paddle install page.

python scripts/ocr/paddle_unlimited_ocr.py readings/pdfs readings/markdown --lang en
python scripts/ocr/paddle_unlimited_ocr.py readings/pdfs readings/markdown --lang ch --ocr-version PP-OCRv5
```

Use this route for:

* Sensitive documents that should not leave the machine
* Very large batches where hosted OCR cost is the constraint
* Chinese or multilingual OCR experiments where PaddleOCR models are a good fit

Avoid the local PaddleOCR route when you need turnkey setup. Local runtime and model installation can be less reliable than a hosted API.

## Alternative API options [#alternative-api-options]

### Google Document AI [#google-document-ai]

Use [Google Document AI](https://docs.cloud.google.com/document-ai/docs) when you already work in Google Cloud.

1. Visit [Google Cloud Console](https://console.cloud.google.com/)
2. Enable the Document AI API.
3. Create a processor for OCR.
4. Use the Python client library for batch processing.

### Azure Form Recognizer [#azure-form-recognizer]

Use [Azure Document Intelligence](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/) in enterprise Azure environments.

1. Visit [Azure Portal](https://portal.azure.com/)
2. Create a Cognitive Services resource.
3. Use the Form Recognizer service.
4. Integrate using the REST API or SDK.

## Best practices and tips [#best-practices-and-tips]

### File organization [#file-organization]

```
your-project/
├── pdfs/           # Original PDFs
│   ├── session-1/
│   ├── session-2/
│   └── articles/
└── markdown/       # Converted text files
    ├── session-1/
    ├── session-2/
    └── articles/
```

### Quality control [#quality-control]

* Spot check: review converted files for accuracy
* Complex layouts: Some academic PDFs may need manual review
* Images and tables: OCR may not capture complex visual elements
* Languages: Ensure the API supports your document languages

### Cost optimization [#cost-optimization]

* Batch processing: Convert all PDFs at once rather than individually
* File size limits: Be aware of API size restrictions
* Free tiers: Use free options for small projects
* Caching: Store converted files to avoid re-processing

### Workflow integration [#workflow-integration]

* Version control: Track both original PDFs and converted text
* Backup: Keep original PDFs as source of truth
* Naming: Maintain consistent file naming conventions
* Metadata: Preserve citation information alongside converted text

## Troubleshooting [#troubleshooting]

### Common issues [#common-issues]

"API Key Not Found"

* Verify environment variable is set: `echo $MISTRAL_API_KEY`
* Restart your terminal or command prompt
* Check for typos in variable name

"File Too Large"

* The default limit is 36 MB per PDF
* Split large documents or use the `--max-size` parameter
* Consider alternative conversion methods for very large files

"Processing Failed"

* Check the PDF file integrity
* Some PDFs may have copy protection
* Try alternative conversion methods

"Rate Limits Exceeded"

* APIs have rate limits (requests per minute/hour)
* Implement delays between requests
* Consider paid plans for higher limits

## Integration with case study workflow [#integration-with-case-study-workflow]

### Recommended usage pattern [#recommended-usage-pattern]

* Initial setup: convert all PDFs at project start
* Ongoing: convert new PDFs as they're added
* Processing: use converted Markdown files for all AI workflows
* Storage: maintain both original PDFs and converted text

### Case study references [#case-study-references]

* AI-assisted literature analysis
* Human-AI synthesis workflows
* Agentic workflow design
* API Keys Guide: For Mistral setup
* Model Reference Guide: For compatible AI models

## Advanced options [#advanced-options]

<AccordionGroup>
  <Accordion title="Custom Script Modifications" icon="code">
    You can customize the `batch_ocr.py` script for specific needs:

    * Change output format (currently Markdown)
    * Modify file size limits
    * Add custom metadata extraction
    * Integrate with other APIs
  </Accordion>

  <Accordion title="Local OCR Alternatives" icon="computer">
    For offline processing or sensitive documents:

    * [Tesseract OCR](https://tesseract-ocr.github.io/): Free, open-source
    * [OCRmyPDF](https://ocrmypdf.readthedocs.io/): PDF-specific OCR tool
    * [PyMuPDF](https://pymupdf.readthedocs.io/): Python PDF processing library
  </Accordion>

  <Accordion title="Commercial Solutions" icon="dollar-sign">
    * Adobe Acrobat: High-quality OCR
    * ABBYY FineReader: Enterprise OCR solution
    * Readiris: User-friendly OCR software
  </Accordion>
</AccordionGroup>

## Next steps [#next-steps]

1. Choose your preferred conversion method.
2. Convert a few sample PDFs.
3. Process your full document collection.
4. Use converted files in your AI workflows.
5. Track conversion quality and costs.

Converting PDFs once avoids the time and cost of processing them repeatedly in later AI workflows.