Azure Document Intelligence eliminates the need to review documents and manually enter data into other systems.

Overview

Microsoft’s Azure Document Intelligence is a set of cloud-based services that use machine learning to automate the extraction of data from electronic documents, including forms, structured or unstructured documents, and text-based images at scale with AI and OCR. You can also leverage pre-trained models or build custom models to help speed up app development with the SDK and APIs.

When do we use it?

Retrieving data from documents:

• Convert a scanned document into raw text, even if the document is hand-written.
• Identify vendors and pull out line-item data from vendor invoices.
• Extract key information from custom forms (e.g. Tax forms, mortgage applications, etc.)
• Pull out certification information from within vendor shipping documents.

These services help you retrieve data from within a document and use that data inside a custom-built application and workflow. This eliminates the need to have someone review documents and manually enter data into other systems.

How does it work?

Training a model – Machine learning requires a “model” to be trained, a process that teaches the system to find patterns and make decisions.  Microsoft has pre-trained the Document Intelligence models to perform functions like:

• Convert images to text.
• Extract key/value pairs like “Name:” and “John Doe.”
• Identify tables and extract the data from the tables.
• Identify form types like check boxes, dates, text, bar codes, etc.

Pre-built models – Microsoft also provides a set of pre-built models:

• Invoices – extracts the invoice ID, customer, vendor, ship to, bill to, etc. from an invoice document.
• Receipts – extracts date & time of the transaction, merchant info, etc., from scanned receipts.
• Identity Documents – extracts name, expiration date, etc., from passports and driver’s licenses.

Custom models

Creating a custom model – If the pre-built models don’t give you the necessary capability, you can build and train a custom model.   The process of training your own model is done within the Microsoft Document Intelligence Studio.  This tool is a wrapper around the Azure Document Intelligence APIs and guides you through setting up projects and training custom models.  It requires an Azure subscription and takes a bit of configuration to get set up.  See the Document Intelligence Quick Start guide here.

Uploading training documents – Once the studio is set up:

1. Upload several examples of the documents from which you want to extract data.
2. Identify the data within the document you are interested in by labeling fields.
3. Highlight in the document where the field is located. You can also define areas within the document where subsets of data exist (e.g., sections of a form or a table of data). 

Example – In the vendor certificate below, we’ve identified a “Vendor Part #” and “TDS Issue Date,” and three areas of the form where we want to extract additional data. See the color-coded brackets: Physical Constants, Application, and Properties:

Training the model – After identifying the data you want to extract, “train” the model using the documents uploaded to the project.  This teaches the model where to locate and extract data from the form.  If the forms are uniform and do not vary, then training the model may only take a few examples. If the forms vary, however, it may take additional training with more examples to get the desired accuracy.

Test the model – Once trained, you can test it by uploading additional forms and viewing the results.  For example, here is a test done against the model that was defined above:

The accuracy results show how the model extracted “TDS Issue Date” as 8-14-20 and several data items in the “Properties” section (e.g., Pencil Hardness, Gloss Max/Min, etc.)

Connecting to your application

From within your application:

1. Connect to the Azure Document Intelligence API endpoints and call functions to identify the model you want to use.
2. Upload the form to the service, then return the analysis results.
3. The results come in a JSON document, a standard text-based data format widely used in many modern applications.

Here is a snippet of the JSON data returned by the model:

The JSON data shows a “TDS Issue Date” of “8-10-20”, along with multiple values in the “Physical Constants” table like “TDS Viscosity Nominal,” Weight Per Gallon,” etc. 

Confidence values – Each value comes with a “Confidence” value of accuracy.  This “Confidence” can be used to set a threshold for manual review.  For example, if the “Confidence” is less than 70%, flag the document for review by a user.

Real World examples

TwinEngines uses Azure Document Intelligence for several customers.  Check out these case studies:

• How First American Resources Corporation used Azure Document Intelligence to automate vendor certificate data entry.
• How Denyse used Azure Document Intelligence for Vendor AP Automation.
• How Amware used Azure Document Intelligence for Vendor AP Automation in an LTL shipping environment.

Implementing your application

Designed for developers – Azure Document Intelligence is a developer’s tool.  A seasoned developer must set up and configure Azure resources, make API calls, and analyze JSON results.  Although the Document Intelligence Studio makes defining and training models relatively easy, integrating these services within your application requires IT expertise. 

Pre-processing required – You may need to perform pre-processing on forms before sending them to the AI service to get the desired results.  For example, vendor documents might include an invoice, bill of lading, and certificate data in one PDF file.  You must identify and extract the individual forms before processing them with the Document Intelligence APIs.  If a vendor sends forms about individual Lots in the same file – you might need to extract the individual lots into separate documents to analyze them independently. 

On multiple occasions, we have built outside solutions to identify and extract documents before processing them with the Document Intelligence APIs.

Bottom line

The key benefit is avoiding the manual labor of combing through documents, locating the desired data, and re-entering it into another system.  AI can eliminate this time-consuming and error-prone process. 

TwinEngines is committed to delivering on artificial intelligence’s promise by developing Real-World AI applications for manufacturers, suppliers, and specialty contractors using Microsoft’s leading-edge suite of Azure AI Services.

To learn how we can automate manual processes in your workflow, schedule an online meeting with a TwinEngines Solution Architect.