Researchers develop robot that can recreate recipes by watching cooking videos

Researchers at the University of Cambridge have developed a robotic ‘chef’ capable of learning to cook various dishes by observing cooking videos.

Initially programmed with eight basic salad recipes, the robot could then identify and replicate a demonstrated recipe after watching a video. By the experiment’s conclusion, the robot had independently created a ninth recipe, illustrating the potential of video content as source for automating food preparation.

The Cambridge team’s approach to this challenge involved using a publicly accessible neural network to train their robot, which was already programmed to recognise various objects, including the fruits and vegetables featured in the salad recipes. Applying computer vision techniques, the robot analysed each video frame, identifying objects and features such as a knife, ingredients, and the human demonstrator’s arms, hands, and face.

The recipes and videos were converted into vectors, enabling the robot to perform mathematical operations to assess the similarity between a demonstration and a recipe vector. The robot’s ability to correctly identify ingredients and the human chef’s actions allowed it to determine which recipe was being prepared. For instance, the robot could infer that a carrot held with a knife would be chopped.

The robot exhibited a high level of accuracy, correctly identifying the recipe in 93% of the 16 videos it watched, despite only detecting 83% of the human chef’s actions. It could also recognise variations in recipes, such as double portions or minor errors, distinguishing them from entirely new recipes. When presented with a demonstration of a novel salad, the robot successfully added it to its cookbook and prepared the dish.

The researchers highlighted the robot’s nuanced understanding of recipes, even simple ones involving chopped fruits and vegetables. However, they noted limitations in the robot’s capability to identify ingredients obscured by the demonstrator’s hand or in fast-cut, visually complex social media videos.

While several companies have developed prototype robot chefs, none are yet available commercially, and their skills fall short of human chefs. Traditional methods of teaching robots to cook are both expensive and time-consuming, in contrast to humans who can learn new recipes by simply watching others or videos.

Overall, the project demonstrates significant progress in robotic learning and food preparation automation. The researchers envision a future where robotic chefs, improving in their ability to discern ingredients in videos, might utilise platforms like YouTube to learn a broad spectrum of recipes.

There’s plenty of other editorial on our sister site, Electronic Specifier! Or you can always join in the conversation by commenting below or visiting our LinkedIn page.