How an airline is using AR to improve operations
Technologies sometimes take a surprisingly long time to establish themselves, despite their obvious potential. Augmented reality (AR) is a good example. Although we’ve had the means to support visual information overlays for nearly a decade (think Google Glass), it’s only now that companies are beginning to understand how to take full advantage of its capabilities.
Consider how the landscape changes. Currently, thousands of experiments in what might be called augmented operations are underway in companies around the world. One of the biggest takes place at China Southern Airlines, where the company’s technical arm team, China Southern Technic, have weaved together augmented reality, artificial intelligence, cloud computing, the Internet of Things and 5G connectivity in a single app that can expand human capabilities, enhance security, and improve performance.
Safety inspection is one of the first processes to benefit from CSA’s adoption of augmented reality. (The recent crash of a Boeing 737 operated by China Eastern Airlines only heightens the need to use all available safety inspection tools.)
After every landing of any passenger aircraft anywhere in the world, a maintenance, repair, and operations (MRO) engineer must perform a thorough inspection of the aircraft. An inspection on a Boeing 737 typically takes over 100 steps, and an Airbus 320 takes over 200 steps. This is a fundamental yet crucial part of airline management, an urgent task that front-line MRO engineers must perform every day, often multiple times. CSA is no exception. As the world’s third-largest airline, the airline repeats this procedure more than 2,500 times a day, a task that takes CSA MRO engineers around 1,000 man-hours.
At most airports around the world, engineers check off each inspection item on a bulky paper job card, a pad of more than 20 sheets of paper that they must enter as they go through the task. Until recently, CSA MRO engineers also worked this way, doing this work while juggling paper, pens, walkie-talkies and the job card. But now, at 22 airports where CSA flies, most of the information, archiving and communication tools are integrated into a single AR screen. This display puts a whole range of resources at the service of engineers – not only text, but also images, video, graphics and voice, in any combination useful to engineers.
While AR glasses are expected to reduce those 1,000 daily hours by 6%, we have found in our research into integrating this technology into CSA that the benefits of AR glasses go far beyond the labor dividend. It’s not just a new way to get information, it’s a whole new way to work.
CSA’s AR glasses allow engineers to edit and rearrange their to-do list, change what information they see and how they want it displayed. Their displays can be adjusted based on aircraft, season and even individual preferences. They provide engineers with step-by-step multimedia support and immersive experiences while performing tasks, including AI object recognition and collaboration with a remote expert.
“Combined with some [artificial intelligence], AR glasses can really make our job easier,” said an MRO engineer. “I can now point my fingers at a place, for example a lubricating oil cap, and it automatically recognizes the object or key parts and tells me it’s open but needs to be closed. He can also show me, in a picture or a short video, what the object looked like in a normal state or when it was last used. When the task is complete, engineers can even sign off using voice or even gesture, if there is too much noise on the tarmac to use a voice command.
Rather than lugging bulky manuals like full-length dictionaries or spending valuable time walking to an office to consult one, engineers can instantly access the information they need through the glasses. “I no longer need to pick up the service manual, which could take an hour to get there and back. The manual now comes to me, before my eyes! an engineer told us. AR glasses even allow experts to advise mechanics on the tarmac in real time and provide them with images, videos, voice guidance and graphics.
Lenses also encourage more standardized performance. “It knows where I am in the process and tells me where I need to go next. Everyone goes through the same process in the same order,” another engineer explained.
Welcome to Augmented Operations
Awakened engineers, better compliance, a visual log of every component’s life, and ultimately safer flights are all benefits of this unique pilot project in the 850-plane airline. AR glasses optimize performance not only by bringing knowledge closer to the machines, but by keeping the MRO’s eyes on the prize. Like most previous forms of digitization, CSA’s experience suggests that augmented operations are less likely to supplant people than to augment their capabilities – a win for businesses, employees and travellers.
Today, the first operational augmented CSA system is still in progress, not so much in its ability to transfer data to or from the individual – although this presents challenges – but in adapting the technology to meet the capabilities of human cognition. AR smart glasses must meet industry safety standards, as well as important goals for privacy, comfort, display, connectivity, ergonomics, battery life, noise reduction, multimedia interactivity, immersive experience combined with transparency, required infrastructure (5G, edge computing) and a knowledge graph that can provide deeper AI-based assistance.
That the beginning
And that’s just one application in one industry — imagine the many other ways the technology could be used. Already, thousands of businesses around the world are experimenting with various aspects of AR technologies. And we believe that number will increase significantly once we learn more about the best ways to manage the user interface on all these smart glasses, and as awareness of this highly adaptable new technology increases. It’s not unlike when something called a website appeared on our desktop computers or a decade later when it became clear that apps were the smart phone’s “killer app.”
When enterprise use of AR technologies has its own Netscape moment, we believe many industries will see the dawn of a massive new opportunity. Airlines, for example, will be able to understand their cost structure in much greater detail than they currently do, down to the coin. Ultimately, this cognitive shift could alter the balance of power within the airline business, from sales and the front office to the back office and the maintenance hangar (especially as the carbon footprint becomes more integrated into the price).
And that’s just the beginning. As the CSA project demonstrated, virtualization has no limits. Any person or object in flight operations, from mechanics to planes or the entire airport, can be virtualized, given enough data and enough modeling. By creating a virtual representation of a physical object, along with a continuous stream of new information about its status, digital twins of physical objects and even people can give airlines an unprecedented ability to see how something is working. . at present and simulate or predict how it could perform in the future.
The success of CSA suggests that AR is finally part of our working reality. But there are still many unanswered questions. In our work for CSA, for example, we asked many questions about how to get the best out of people. When do people need reminders? What are the signs that their attention is starting to falter? What’s the most effective way for mechanics to communicate through their glasses with an expert, who can guide them through a complex repair? At the moment, the questions are constantly multiplying but, fortunately, the answers too.