OCR Benchmark Methodology in 2024

AIMultiple aims to help enterprises identify the right OCR for their business. These enterprises should expect to process a high volume (i.e. at least tens of thousands of pages per month) of documents and images.

What will be the guiding principles?

AIMultiple’s benchmark methodology is designed for an objective and transparent assessment. It also explains participation requirements.

What will be benchmarked?

Extraction of text in English from documents and images.

Dataset is expected to include 500 pages:

• 300 pages of long-form PDF documents (e.g. technical manuals, whitepapers, contracts) which include text in image form. PDFs of varying legibility will be used. PDFs will be collected online.
• 100 pages of transactional documents (e.g. invoices and receipts). They will be collected online and selected from AIMultiple’s and its partners’ documents.
• 100 pages of handwritten documents (e.g. receipts, insurance claims forms). They will be collected online and selected from AIMultiple’s and its partners’ documents.

In certain documents, parts of the document will be digitally altered to protect PII.

How will AIMultiple perform the benchmark?

First, AIMultiple will run a test batch with a few documents and share its results with the participating vendors to ensure that AIMultiple correctly uses vendors’ services. Then, AIMultiple will run the entire benchmark.

Which criteria will be used?

AIMultiple’s OCR benchmark aims to closely match the preferences of OCR buyers. They want a flexible, cost-effective solution. Therefore, AIMultiple will measure these metrics:

Accuracy

It will be measured by cosine similarity. We will not use Levenshtein distance because different products output texts in different orders especially in case of multi-column text. While Levenshtein distance takes these positional differences into account, we are interested in how accurate the text is detected but not where it is located.

Speed

Average response time and distribution of response times will be measured. A maximum of 5 seconds of data processing and transfer time will be allowed per page.

Scalability

The same metrics may be tested with a fixed number of simultaneous connections. This metric may be similar for all providers (i.e. simultaneous connections may not slow down processing). In such a case, AIMultiple may not publish the results for this metric.

Cost

Public cost data published by the vendors will be used to calculate the cost of the benchmark. Vendors’ pricing models will also be shared to help buyers compare prices of different loads.

Customer service

Reviews on B2B review platforms will be analyzed to assess customer satisfaction.

How will the results be published?

They will be published on AIMultiple.com and will feature graphs that users can leverage to find the right vendor for their business. Top vendors in each of the above categories will be presented.

Each participant will receive

• their detailed results for each document and page along with timestamps
• the average results for each document and page
• the dataset

Please note that AIMultiple is in the design phase of the benchmark and changes will be made as AIMultiple gets end user feedback and finalizes the benchmark.

Reach out to AIMultiple team via info@aimultiple.com if you would like to participate in the AIMultiple OCR benchmark.