Every UBL invoice a freight company receives carries one field that decides everything: the transport reference, the tracking number that ties the bill to the shipment it pays for. Find it and the invoice reconciles itself. Miss it and a person goes hunting through the document by hand. At the volume a logistics operator runs, that hunt is a full-time job for a lot of people, and it's the kind of work nobody signed up to do.
The usual fix is to prompt a large language model once and hope. Show it the invoice, ask for the reference, take what comes back. That holds up until the format drifts, the field moves, or the document runs long, and in logistics all three happen constantly. A single prompt has no memory of the last thousand invoices it read. So it repeats the same mistake on the thousand-and-first.
MATRIX is our answer. The name unpacks to Memory-Augmented agent Training through Reasoning and Iterative eXploration. Here's the short version a whiteboard would give you: teach the agent from the invoices it has already seen, then let it read the next batch with that experience in hand.
MATRIX runs in two phases, and the first one is training. It takes a batch of invoices, makes its best guess at the references, then checks each guess against the correct answer. Where it got something wrong, it writes itself a note: what the document looked like, what it missed, how to catch it next time. Those notes go into a long-term memory module that persists across batches.
Then it does it again. Batch after batch, the memory fills with how these invoices are really built: where the reference tends to sit, how a given supplier lays out a field, which look-alike numbers to ignore. The agent isn't relearning the task from zero each round. It's carrying forward what the last batch taught it.
Inference is the payoff. When a new invoice arrives, the agent reads it with that accumulated memory alongside. Fewer guesses. Fewer repeated mistakes. Figure 1 in the paper walks the whole loop end to end, from a raw document coming in to a final answer going out.
The result that matters: MATRIX beats single-prompt extraction by more than 30% on accuracy, shown in Figure 2. That's not a rounding-error gain. It's the gap between an agent you can leave alone with the queue and one a person has to check behind.
It's also cheaper to run. Each attempt at an invoice costs API calls, the metered requests to the model that do the reading. MATRIX makes fewer of them per document. Across a full month's queue that difference is real money, and fewer calls keeps the system quick even when a single invoice runs long.
Most agent benchmarks test on documents someone made up. This one doesn't. We built the dataset with Kuehne+Nagel, a global logistics operator, out of their actual Universal Business Language invoices, the UBL format the industry files in.
That hands MATRIX the thing academic tests skip: the mess of production. Real suppliers, real layout drift, real edge cases that a clean synthetic set never shows you. It's also, as far as we know, the first open-source agent benchmark for UBL-format invoices. The next team working this problem gets a shared yardstick instead of a private one.
The paper is Memory-Augmented Agents for Business Document Understanding, and the work is joint. The authors are Jiale Liu, Yifan Zeng, Malte Højmark-Bertelsen, Marie Normann Gadeberg, Huazheng Wang, and Qingyun Wu, the last a co-author of AG2, the agent framework formerly called Autogen. The academic side comes from Oregon State University and Penn State University. The invoices come from Kuehne+Nagel.
It's out in the open: arXiv 2412.15274v1 in the cs.CL section, under a CC BY 4.0 license, so anyone can read it, build on it, and ship it. We think the memory loop reaches well past invoices, to any document task where the same patterns keep recurring and a person keeps solving them from scratch. The invoice is just where we could prove it first.