Vision AI for Restaurants: Understanding Interactions | Palona
By Palona AI | November 4, 2026
Suppose a camera shows an employee standing beside a restaurant table. What should the system conclude?
The employee could be taking an order, delivering food, cleaning the table, checking in with guests, or simply walking past. The people and objects in the scene may look similar in each case. What matters is the interaction.
This is the starting point for our Vision AI work at Palona.
We are building models that understand interactions relevant to restaurant operations, turning existing camera footage into useful operational information.
We have written before about why we believe interaction understanding is a missing layer for Physical AI. Since building our first interaction-understanding model in July, we have continued improving both the models and the engineering process around them.
Seeing is not the same as understanding
Modern vision-language models are increasingly good at describing images and video.
But in our early experiments, we found that describing a scene was not enough to reliably answer the operational questions our customers cared about.
In our first interaction-understanding scenario, general-purpose VLMs achieved less than 50% accuracy.
That result was important for us.
The challenge was not simply recognizing that a person or table was present. The system needed to understand what was actually happening between people, objects, and the surrounding environment.
That motivated us to build a specialized interaction-understanding model.
From experiments to a stronger engineering process
Our first model in July showed that this direction could work.
Since then, we have gone through multiple iterations of the system. With each iteration, we have tightened the broader engineering process around model development: how we define operational interactions, build and evaluate models, test them against increasingly difficult real-world situations, and integrate them into the product.
The details continue to evolve, but the principle is simple:
Each iteration teaches us where the system needs to become more robust, and that learning feeds directly into the next version.
Over time, this has allowed us to expand the range of interactions we can recognize while steadily improving model quality.
Measure, learn, improve
Our internal evaluations reflect that progression.
| Metric | 1st iteration | 2nd iteration Balanced sampling |
3rd iteration Enhanced coverage |
|---|---|---|---|
| Date | July 2026 | September 2026 | November 2026 |
| Accuracy | 83.64% | 86.18% | 93.73% |
| Store | Pokeworks | Chica | Chubby |
| Events | Cashier waiting | Table occupancy Table touch |
Table cleaning |
| Frames | 9,000 | 10,800 | 36,000 |
Reported internal evaluations, July–November 2026.
Because these evaluations cover different interactions, stores, and datasets, they should not be interpreted as a controlled comparison between model versions.
What they do show is the progression of the engineering effort.
Over several months, we expanded the range of interactions we could recognize, increased the scale and difficulty of our evaluations, and continued improving model performance.
The November evaluation reached 93.73% accuracy across 36,000 evaluated frames for table cleaning.
For us, the more important story is not any single number. It is the steady improvement of the system as we learn from each new use case and each new deployment.
Make recognition useful during service
The product has to connect this work to the restaurant’s operating routine. Palona Vision AI uses existing security cameras to analyze configured visual events, provide operational insights, and notify managers when attention is needed. It is available through select pilots.
A table-cleaning example illustrates the sequence: the camera provides a view, the model recognizes the defined activity, and the resulting information helps the team understand what happened. The manager still applies the restaurant’s standards and decides what action to take.
This is why deployment matters as much as the model. A well-defined task, a usable camera view, and information that arrives in a useful form all have to work together.
The aim is to give a busy team better visibility into service without requiring them to spend more time interpreting video.
If your team repeatedly returns to camera footage to answer the same operational question, that is a good place to begin a conversation with us. Define the interaction, examine the evidence, and test whether recognizing it helps the restaurant.