Mistral AI vs ChatGPT: reliable OCR?

At Koncile, we are always looking for the latest advances in the field of visual language models (VLM) and we regularly put these new technologies to the test to better understand their limits in real conditions. It is in this dynamic that we have developed our own OCR software powered by AI, in order to offer a more accurate and reliable solution for extracting complex data.

Today, Mistral AI unveiled its brand new OCR model, which they present as being at the cutting edge of technology (SOTA), based on as yet unpublished benchmarks. As is often the case, excitement quickly took over the internet. The model found itself at the top of discussions on Hacker News, and many users immediately claimed that extracting text from PDFs was now a problem solved once and for all.

‍

It is with this in mind that we chose to evaluate Mistral OCR, by comparing it with Chat GPT , another major player in the world of artificial intelligence. Although Mistral claims 94.9% accuracy for its OCR and other reports suggest that ChatGPT achieves similar scores (89.77%), our tests revealed a significant gap between this theoretical performance and the real results obtained on our own data set.

Performance of Mistral.Ai on invoices

We analyzed a typical invoice using Mistral's new OCR model  a common scenario in Invoice OCR, where accurate data structuring is essential.

Here is the data extraction legend:

• Types of errors: This column describes the various categories of errors that the tool made when extracting data from the invoice. A distinction is made between:
  • Missing data: This is information that should have been extracted from the document but was not detected by the tool.
  • Misplaced data: This refers to data that has been extracted but assigned to the wrong category or location in the tool output.
  • Incorrectly transcribed data: This category includes errors where the tool extracted data but transcribed it incorrectly (for example, misrecognized numbers or letters).

The results are shown below.
‍

Here is the legend of the reliability table:

• Number of errors: This column shows the number of times each type of error was encountered during the analysis of the invoice.
• Percent error (%): This represents the percentage of each type of error in relation to the total number of data to be extracted.
• Reliability (%): This column indicates the reliability of the tool, that is, the percentage of data that was extracted correctly.

In summary, this legend gives us a clear overview of the types of errors the tool makes, how common they are, and how they impact overall reliability.

Mistral.Ai performance chart on invoices

📌 Overall reliability rate: 63.75%

ChatGPT performance 4.5 On Invoices

We also evaluated a standard invoice using the ChatGPT model.

The results highlight familiar issues often encountered in intelligent document processing, especially when dealing with varied formats and unstructured data.

ChatGPT performance chart on invoices

📌 Overall reliability rate: 57.5%

Mistral AI vs. ChatGPT: Performance Below Expectations

Despite promising claims, our tests reveal that neither Mistral AI (63.75% reliability) nor ChatGPT (57.5%) truly deliver on their OCR capabilities.

📌 Mistral AI excels in pure transcription with 98.75% accuracy, but struggles with 27.5% missing data.

📌 ChatGPT, while better at positioning data, loses even more essential information, with 42.5% missing data.

🔍 The verdict is clear: neither model guarantees reliable and complete data extraction, especially for complex documents like invoices.

Koncile: The Next-Gen OCR Alternative

At Koncile, we’ve developed a next-generation OCR that combines high-precision extraction with intelligent document understanding. Our optimized AI drastically reduces errors and ensures accurate data extraction, even from non-standardized documents.

💡 Why choose Koncile OCR?

Higher reliability – Our model is designed to minimize errors

Fewer missing data & better information structuring

Adapted for complex documents – Perfect for invoices, contracts, and reports

For businesses that rely on precise and structured data extraction, Koncile OCR is the superior alternative.

‍