Businessware Technologies regularly tests large language AI models for business automation tasks. These tests evaluate the effectiveness of models in extracting data from complex documents such as invoices, drawings, and spreadsheets. The datasets used include digital documents with different formats and languages, which allows the models to work in conditions close to real projects.
Key requirements for testing
When evaluating intelligent document processing models, the AI benchmark focuses on recognition accuracy, which reflects the model’s ability to extract data from documents, including headers, field values, formatting, and text blocks. An important parameter is the processing time, which shows the average time required for the model to process one document, as well as the cost, which includes processing 1,000 pages and possible additional costs.
Testing models for invoice processing showed that Amazon Analyze Expense API provided high recognition accuracy with minimal processing time and a cost of $10 per 1,000 pages. Azure AI Document Intelligence demonstrated slightly lower accuracy and longer processing time at a similar cost. GPT-4o achieved high accuracy with third-party OCR, but the processing time was significantly longer and the cost was $8.8 per 1,000 pages. Google Document AI showed relatively low accuracy, especially for document elements, but the processing speed was still fast and the cost was $10 per 1,000 pages.
When testing models for extracting tables from construction drawings, different trends emerged. Azure Prebuilt Layout achieved good accuracy and moderate processing time and the cost was $10 per 1,000 pages. Amazon boto3 Textract showed slightly higher accuracy, but with a slight increase in processing time and a cost of $15 per 1,000 pages.
These results highlight the need for a holistic approach when choosing an IDP model, considering the balance between recognition accuracy, processing time, and cost effectiveness. For business problems where speed and moderate accuracy are important, solutions like Amazon Analyze Expense API or Azure AI Document Intelligence may be suitable. For complex documents requiring maximum precision, more specialized models are preferable, despite the increased costs and processing time.