The publication of Attention Is All You Need is regarded as the launch of the Transformer model movement, setting AI on its current trajectory.
The Transformer network introduced a novel architecture for handling sequential data, particularly in tasks like machine translation. Unlike earlier models that used recurrent neural networks (RNNs) or convolutional neural networks (CNNs), the Transformer relies solely on attention mechanisms. This allows the model to weigh the importance of different parts of the input sequence (encoder states) when processing or generating each token in the output sequence (decoder states).
The Transformer model represented a paradigm shift in how sequential data is processed in neural networks, leading to significant advancements in natural language processing, text classification, language modeling and image processing.
Turning the corner
With the rapid rise of generative AI and large-language models (LLMs) over the last year, it seems as if new models and AI products hit the market weekly. However, not all models are created equal, nor are they task-interchangeable.
For example, while the Transformer architecture unlocks incredible potential in a machine’s ability to surpass the Turing Test (which aims to determine if a machine can demonstrate human-like intelligence), it comes at a high cost. Massive amounts of training data are needed for the machine to learn and remember relational element connections at a vast scale.
This requirement acts as a barrier to entry on multiple fronts — such as obtaining valid samples, managing computing expenses, finding available talent, and covering the costs of R&D experimentation, which many organizations may struggle to meet. However, for organizations with sufficient training content and investment, such as Dark Matter Technologies, harnessing the Transformer model has led to incredible progress in previously “unsolvable” problems.
Dark Matter has been ahead of the curve for the past decade, having invested substantially in its Artificial Intelligence Virtual Assistant® (Aiva®) solution for the U.S. mortgage originations market. Aiva® Docs demonstrates continual improvements in the challenging use case of intelligent document processing. A growing number of clients using Aiva Docs automate over 70 – 80% of document indexing tasks, with select clients achieving performance above the 90% threshold.
In the past three years alone, Dark Matter has processed over 356 million pages in over 105 million documents and has extracted over 148 million data labels through Aiva. As of March 2024, Aiva Docs is automating around 400 document types and 370 labels into the Empower® loan origination system (LOS) at or above 99% accuracy across all clients (individual client performance can vary depending upon the quality of document and image inputs).
Highly structured, mutually exclusive, consistent documents perform well with the Transformer model. However, other common document types that are highly variable, such as pay stubs, are more challenging. These variable document classes require more samples and are best trained by organizations with access to sufficiently large training sets to generalize the nationwide market of employers. Even in these classes, Aiva has seen significant performance lifts, including achieving F1 accuracy in the >= 95% range for some clients.
The importance of attention modeling
While LLMs, Generative AI, Transformers and reinforcement learning are powerful tools in our development toolkit, they are not the only ones.
Attention modeling excels at extracting context when data points are inconsistent, and it is the preferred model for jumping between contexts while incorporating meaningful relationships from an entire document. This complicated context blending has necessitated the ability to discern and quantify spatial relationships along with language meaning.
Utilizing attention modeling to process the language and spatial context of each part of a document’s text is necessary for intelligent document understanding, an often-missing piece of AI solutions. This method has been a critically important tool to incorporate into Aiva’s current production extraction engine.
Aiva adjusts to contextual nuance
Clearly, one model will not rule them all. Complex problems require an ecosystem of methodologies that can complement and compete with one another when necessary. Intelligent, AI-driven automation must be able to recognize data points and have a mechanism to change contextual perspective. Not only does Aiva depend on an array of models to solve various document automation challenges, but it also utilizes governing models to control and learn which methods may be best suited contextually for a given response.
With the ability to consume available market models through technology vendors, open source, or internal proprietary approaches, Aiva can act as an “AI market microcosm” that puts its best foot forward when identifying document artifacts.
We think this flexibility is its secret power and one you’ll want to see in action in intelligently processing your own documents at speeds and accuracy no human can match.
Brian Swanson is the Director of AI/ML at Dark Matter Technologies, and Boaz Reisman is an AI specialist.