Using virtual reality to train pruners

‘The VR training tool would allow workers to make mistakes within a virtual world before making real cuts on vines’

A research project in New Zealand has come up with a way to train pruners in a classroom using virtual reality headsets. And when they get out into the vineyard, augmented reality glasses or smartphones will ensure they make the correct cuts.
The VR product has now been passed on to Tauranga-based robotics company RoboticsPlus, which is looking at developing the system for commercial purposes. The speed of the development depends on interest from potential buyers.
“The more demand they see for it, the higher a priority they may make it,” says Henry Williams, who has been developing the VR tool as part of the MaaraTech Project.
This is a large-scale, transdisciplinary project collaboratively designing robotic and human assist technologies with AI capabilities for vineyards, apple orchards and blueberry orchards in Aotearoa, New Zealand. The project is based out of the University of Auckland under the leadership of Professor Bruce MacDonald, and is funded primarily by the New Zealand Ministry of Business, Innovation and Employment, with co-funding from industry partners.
The project “sharpened its focus” on vine pruning when Covid-19 travel restrictions caused a shortage of skilled workers in the New Zealand wine industry.
Dr William describes grapevine pruning as a high stakes situation – making precisely the right cuts can have years-long consequences for the productivity of the vine and therefore the profitability of the harvest. The window of opportunity for real-world training opportunities is only open for a small period every year and getting it wrong is not much of an option.
“That is why the industry has relied on recruiting a skilled workforce from overseas for the time they are needed every year. Experienced workers have a ‘sixth sense’ and know exactly what to do,” he points out. But during the pandemic, there have been problems recruiting these skilled workers – which has increased interest in training homegrown workers and also in automated pruning.
The MaaraTech Project and RoboticsPlus are both working on fully automated robots. The same training methods used in VR and AR could be adapted to get robots to prune vines accurately in the future.

This video shows how the headsets work

Karly Burch, a research fellow at the University of Otago’s Centre for Sustainability, adds: “The technologies are also being designed to support growers with other tasks of interest, such as identifying disease and estimating yield.”
Regarding the introduction of these new technologies, she says: “According to my colleagues, the VR training tool and potentially the AR headset could be commercialised and adopted within the wine sector in the next five years. The fully autonomous pruning robot will need more time to develop.”
In the OIV’s report on digital trends in the wine sector, available here, she explains the benefits of virtual reality training: “The purpose of the VR headset is to support with training agricultural workers to complete these tasks. One of the most difficult aspects of training workers is that training takes place using real wine-grape vines, meaning that mistakes made during the training process have actual material consequences on vineyards. The VR training tool would allow workers to make mistakes within a virtual world before making real cuts on vines, something that is very important to growers. We have received a lot of interest from industry partners, growers and agricultural trainers who would like support with streamlining and standardising their training processes and, thus, see value in this technology.”
As cybersickness is a potential problem with VR headsets – and is more common among women than men – the team is also looking at ways to conduct the training without the headset, for example through a touchscreen device.
Horticulture consultant Craig Hornblow says: “The part of the VR prototype people have been most amazed by is how realistic the vine and the pruning actions are. It will reduce training times and allow further upskilling because training in this setting can be done anytime such as pre-season and on wet days.”