Today we're introducing On-Device Field Detection, a new capability in the Veryfi Lens SDK that validates receipts the moment they're captured, not after they've been uploaded, processed, and potentially rejected. It's a small shift in when validation happens, and it changes a lot about what happens next.
PicWish
@marianna_babayan1 will it work if I capture a billboard
@mohsinproduct Certainly not in any useful manner because the on device model has been trained to identify fields from a receipt such as vendor name, date, total, etc. and thus showing it an image of a billboard would probably lead it to give no result or rather incorrectly attempt to make sense out of any text available by fitting it into the above mentioned fields. It would definitely be interesting to see what it does in such a scenario, does it give a graceful fail and simply displays that No fields detected or tries guessing?
Validating fields on-device at capture, before anything is uploaded, is the right place to catch a bad receipt scan; the rejection round-trip is exactly the pain when you process after upload. The part I'd test first: does the extraction run fully offline with the receipt's PII never leaving the device until I choose to sync, and roughly what model footprint does the Lens SDK add to an app bundle? And on a failed field, do I get per-field confidence back to drive my own retry UI, or just pass/fail?
@hi_i_am_mimo Appreciate the thoughtful questions, those are exactly the kinds of implementation details we're happy to walk through directly. Feel free to Contact us and we can go deeper there. Thanks!
@marianna_babayan1 Totally fair, will reach out for the deep dive. One quick one that decides fit for us though: does the on-device field detection run fully offline at capture, or does the validation step still make a network round-trip to your API before it clears the receipt?
The capture-time validation is the right call. The piece I'd pin down is threshold calibration. To fit an app bundle the on-device model is almost certainly quantized, and int8 quantization shifts the confidence distribution enough that an accept/reject cutoff tuned on your server model reads differently on-device. When we shipped a quantized field extractor, a 0.8 confidence cutoff that was safe on the full-precision model started waving through borderline scans. Do you recalibrate the on-device thresholds against the quantized model specifically, or share one cutoff across both?
@dipankar_sarkar Thanks for the thoughtful question, calibration between on-device and server models is definitely something we account for internally. Appreciate you raising it.
the calibration discussion in this thread covers the false-confidence direction well, on-device says it's fine and the server disagrees later. curious about the other direction though: the lighter on-device model flagging a field as missing/unclear and asking someone to retake a receipt that the full server model would've actually accepted just fine. that's a worse user experience than the original problem in a way, since now you're annoying someone who did nothing wrong instead of just catching genuine bad scans. is that failure mode rarer in practice, or is retake-friction just the accepted tradeoff for catching the real rejects earlier
Makes sense. The choice I'd be most curious about is which error direction you bias the on-device model toward. A false reject just asks for a retake while someone's still holding the receipt, but a false accept ships bad data your server catches days later, which is the exact round-trip you built this to kill. We ended up setting the on-device cutoff a notch conservative and ate a few extra retakes. Did you land the same way?
validation at capture is right — the sharp edge is the on-device check disagreeing with the heavier server extractor later. late rejection becomes silent disagreement; keeping the two models aligned is the real maintenance cost.
@qifengzheng Really good callout, that's exactly the tension we had to design around. The on-device model is deliberately lighter-weight for speed and offline use. Thanks!
What about receipts not written in English and Latin script languages, is the localization of the field detection models based on regions, or is the accuracy significantly poorer for receipts in other languages than English?