Cognitive Document Automation: How Do We Measure Success?
Welcome to part four of our six-part series that takes you on a journey through the latest concepts in multichannel document capture and intelligent OCR. We’re focused on how AI has transformed what’s possible in making your documents and data work for you—and not against you.
In part one, we looked at how RPA marked a revolution in empowering businesses to solve problems associated with manual, data-centric tasks, yet was historically ineffective in automating document processing. In part two, we examined the emergence of Cognitive Document Automation (CDA), which does the “head work” of understanding what the document or email is about, what information it contains, and what to do with it. And in part three of our series, we did a deeper dive into what you should look for in a CDA solution (Hint: You’ll need much more than just OCR functionality).
Read More: RPA Is Just the Beginning of Your Intelligent Automation Journey (Part 1)
In part four, we tackle this question: How do we measure the success of CDA?
Cognitive Document Automation: How Do We Measure Success?
First, let’s acknowledge that the objective of any CDA project is to see the benefits of greater visibility, lower costs, faster processes, fewer errors and improved customer engagement. When it comes to gauging the success of CDA, the single largest indicator is user productivity—the degree to which people become more productive because CDA helps them get work done better and faster.
User productivity is made up of two components: OCR accuracy and user efficiency. Let’s take a closer look at each of these:
Perhaps the most common question after watching a CDA technology demonstration is: What level of OCR accuracy can I expect to achieve?
The short answer is: it depends and varies widely across use cases. OCR accuracy, and more generally classification and extraction accuracy, is determined by several factors, such as:
- Scanner hardware
- Scan resolution
- Image quality
- Document type (form, invoice, letter)
- Document language
- Font type and character spacing
- Field boxes/shadings
- Ability to database match or check for checksums and other rules
Some of these factors will be discussed in greater depth later in our blog series. But for now, just remember this: the higher the accuracy, the more classification and extraction automation; the lower the accuracy, the less automation and more manual labor.
Because accuracy varies so much, it’s recommended to perform benchmark testing for classification and extraction accuracy rates on the business’s actual real-world samples. Use these results to optimize project settings for each document type and field, and thereby increase accuracy and automation. Benchmark testing should record, for each document type and field, its confidence (yes/no) and correctness (yes/no). Our goal is to maximize true positives, minimize false negatives and true negatives, and completely eradicate any false positives from being exported to downstream people, processes and systems.
A related term with which you might be familiar, “straight through processing” (STP), is also used as a metric to describe CDA results. This is the measure of the percentage of documents run through the CDA “acquire, understand, integrate” process untouched by a human (see part 2 of the blog series for more information). The STP rate will never be higher than the lowest OCR accuracy field on the document. To maximize the STP rate, focus on the lowest-accuracy fields being extracted on the document, and adjust settings for those fields.
Read More: What Makes Cognitive Document Automation So Smart? (Part 2)
OCR accuracy is just one side of the coin of user productivity. The other side is user efficiency for exceptions. Documents and fields that don’t pass through untouched (known as “low confidence”) must be reviewed by a human to ensure they’re classified and extracted correctly.
To put it simply, user efficiency is all about how quickly a user can review a low-confidence document or field, make a decision on what needs to be corrected/confirmed, and then execute that decision. So the human-validation interface must be designed for the most efficient use of eyes and hands during the document classification review and data validation process.
Here are some examples of user efficiency features in leading CDA solutions:
- Jumping to the field that needs to be validated, skipping over confident fields
- Highlighting that field on the actual image for context
- Displaying an image snippet of the field in question next to the data entry area
- Custom positioning of panels to each user’s liking
- Correcting a single character in the field rather than re-entering the entire field
- Hitting a hotkey to call a database lookup for a field
- Auto-complete of the field based on the document type list or full-page OCR
- Completing a table’s worth of data by simply highlighting it
The effort spent on user efficiency and user experience will produce ten times the user productivity compared to the same effort spent on improving field OCR accuracy. That’s why it’s best to maximize a human’s work speed processing these data exceptions via effective user engagement and minimal keystrokes and mouse movements.
This brings us to user productivity, which is the combination of OCR accuracy and user efficiency—and represents the single most important metric for a CDA project’s success.
Basically, user productivity is the number of documents per hour/day/week/month each staff member can process with an acceptable level of data quality. For example, consider a mortgage application form. Some form fields will be more important than others, so an “acceptable level of data quality” will vary depending on the field. Benchmarking the CDA project to understand per-field OCR accuracy is necessary to optimize extraction rates for high-priority fields (such as social security number and annual income).
When configured effectively based on the success metric of maximizing user productivity, CDA solutions will deliver an attractive ROI and payback—frequently between six and 18 months from system launch.
Read More: Cognitive Document Automation – Your “Must-Haves” Checklist (Part 3)