Adaptation of Augmented Reality in Field Service by Comarch

For most people engaged in new technologies or gaming, augmented and virtual reality (AR and VR) are still some kind of novelty to play with or use to attract attention. However, due to the growing complexity of services, cooperation challenges for employees and the limited availability of well-qualified staff, AR is mentioned by experts as one of the key future solutions for today’s problems. This is especially true for field service management, where workforces are physically spread out and communication must be effective even over long distances.

Capabilities of Different Approaches to Artificial Reality

Taking into consideration our observation that around 10% of all tasks require additional support from more advanced experts (of whom there is a very limited number), we decided to invest in technologies utilizing different forms of artificial reality. However, before we started our journey, we analyzed the current technological capabilities of artificial reality.

We discovered that these can be divided into three types:

• Augmented reality (AR) – where virtual elements are generated and overlaid on real objects or the physical environment around the user. Functionality of this element can be strictly informative, and interaction can be very limited (for example, read only information). The devices required are simpler than for other scenarios. Usability includes navigation/ routing and context support, for example during shopping or object identification.
• Virtual reality (VR) – a more advanced approach is to generate a virtual environment in which the user can operate and interact with objects. This requires much more computing power and more advanced devices. Use cases cover simulations, games or training sessions.
• Mixed reality (MR) – the most advanced area, where the real environment is mixed with virtual objects with which the user can interact, and which themselves interact with the surroundings (such as a virtual lamp on a real table). Possibilities range from training scenarios with holograms and interactive context information to cooperation with real devices. This is a type of extended augmented reality.

Comarch FSM and Augmented Reality: the choice of device and development platform

Based on Comarch’s years of market experience, in cooperation with the User Experience team, we carried out research to create models, interfaces and scenarios for the proof of concept application. The main goal was to support technicians in the field with remote expertise, a context knowledge base and operation instructions without overwhelming them with information. To address most of scenarios and gain as much knowledge from the implementation as possible, the Comarch FSM development team decided to use Microsoft HoloLens device together with the Unity Engine.

Microsoft HoloLens is one of the most advanced products available, and is an independent device that doesn’t need to pair with a computer or mobile device. What’s more, communication with the interface can be based on gestures, so additional controllers are not required. Another benefit is an option to stream audio and video from the built-in camera and microphone through WiFi, enabling remote communication and integration with the engineer’s environment. It is also worth considering positional tracking, which refers to a device’s capacity to detect and track the precise position of the head-mounted display. Many other solutions offer “3 degrees of freedom”, which means that the user can move forward or backward, look up, down, left or right. This is the simplest approach, presented in most phone-based VR headsets (such as Google Cardboard and Samsung Gear VR).

However, while this enables you to experience the digital environment from one spot, the viewpoint is fixed. To provide a more realistic and natural digital experience, the device should offer “6 degrees of freedom”, otherwise known as positional tracking or room scale. In this case, the headset is able to track not only the user’s head movements but also their position (additional degrees of freedom to handle pitch, yaw and roll). Such tracking possibilities are available in, for example, Microsoft HoloLens, PlayStation VR and Asus MR.

To limit the dependence of one device, we have chosen the Unity Platform as an environment to visualize two and three-dimensional interactive materials. The most important factors were the platform’s popularity, the devices that could be supported by it, and ease of compiling the same code for different platforms.

Proof of concept: FSM app for Microsoft HoloLens

We have prepared a PoC that can operate in two complementary modes: with a device or with a hologram.

Device mode can interact with a real device, supporting the technician with additional information presented alongside that device and offering hints for each action. Thanks to WiFi and Bluetooth connections, the existing solution can be extended to include real-time communication with the device, work step recording and synchronization with the cloud. Hands-free voice control is provided via on-board speakers and microphone, enabling video conferences and remote support.

The second mode generates a hologram of the device to emulate interaction with the virtual object for training purposes. As the hologram screens and navigation process are identical to those on the real device, technicians can use this mode to prepare themselves before going into the field to manage a real issue.

To learn more, watch our video and read our white paper about mixed reality for field service.

Conclusions from PoC

The implemented application was presented during many events and product demos, attracting large audiences with positive feedback received. We also gathered a lot of practical knowledge about A/V/M reality and the implementation of such products. Based on this, we can conclude that current possibilities of devices are perfect for controlled environments such as training centers, but they are still not ready for production use. Among the most serious drawbacks, the performance of such devices is limited (comparable with a higher end smartphone), the devices are quite heavy (each is effectively a whole computer with sensors, cameras and batteries mounted in the headset), the battery lifetime is around two hours (far below the time spent by technicians in the field), and the quality of rendered objects is not even close to real. Taken together, these drawbacks mean that investment in such a device would not be justified.

The described PoC has been implemented for the first version of MS HoloLens, with some improvements for the second generation. Although the first impression is that the second version of the device is much lighter, although both weigh almost the same. What differs in this case is the location of center of gravity, which gives much better weight balance on the head. The second improvement is the much better responsiveness of UI. Here, the recognition of additional gestures reduces the time needed to get used to the interface. However, the rendering quality is still far from “real life”, and the battery lifetime has been only slightly improved. Let’s hope the next devices will solve these problems too, with lighter batteries of greater capacity, and improved computing power and rendering quality.

The second PoC: augmented reality for mobile devices

With quite good feedback from demos, and having seen the benefits of AR solutions, we were keen to solve the most serious problem: the price of the device. A decision was taken to move the application to devices such as tablets and smartphones, which our customers already possess, as field service staff use both Android and iOS applications for their daily routines. The newest versions of Android and iOS perfectly support augmented reality with advanced features such as surface recognition enabling users to generate, for example, a virtual object falling off a real table.

The second PoC for smartphones and tablets integrates with the Knowledge Database (Comarch FSM feature) to provide fast and easy access to stored documents. Devices registered in FSM can be labelled with QR codes, so the field worker enabling the camera in the mobile app can scan labels and get a context summary of the device. When clicking on the generated AR, a pop-up application presents a dedicated screen with all information and documents related to the equipment, such as status attributes and document links. The second approach raised similar interest during demos, and allowed a more affordable price and development flexibility.

Augmented reality app for field service on tablets and smartphones

Summary: augmented reality for field service: hype or a real solution?

Once the market had been flooded with augmented and virtual reality solutions, with most applications being for entertainment or training, the real business value for service organizations wasn’t tangible and proven in any way, even for field service management software providers. The development of PoCs enabled us to discuss all pros and cons with our clients, and redefine our initial assumptions. The newest PoC for tablets and smartphones seems to meet more expectations of service organizations, especially regarding remote support for less experienced workers, at more affordable price and with the development flexibility that companies need. We believe that, together with more testing and suggestions from clients, the AR solutions can add true value to companies when they are adjusted to business reality. We are happy that Comarch is rising to this challenge.