Where immersive technology immerses us
Hello, Habr! This is Sergey Lukashkin, and today we’ll talk about immersive technologies.
The term "immersive" has now become commonly used, it can be found in reports of such eminent research companies (strategic future search) as Gartner (immersive experience) or CBInsights .
Immersive technology is also called augmented reality technology. Their list includes virtual and augmented reality, as well as 360 ° video. They provide the effect of full or partial presence in an alternative space and thereby change the user experience in completely different areas. In this post we will talk about at what level these technologies are today, how they are implemented and used for business.
First, let's define the terms and abbreviations that you can see in reports, articles and presentations:
Now we will analyze in more detail what is hidden under these abbreviations.
Cameras have been available for quite some time, since 2018 they have been supported by Facebook. In fact, these are sequential sets of pictures stitched using algorithms. You can make them as a single camera, or special 360 ° cameras. This is a very convenient format with which you, for example, can go on a virtual tour of the Pushkin Museum.
How is it implemented. 360 ° cameras shoot the surrounding area, after which the resulting videos are stitched in special programs. Seamless solutions exist, but they are more costly for the business. Sometimes additional graphics are added to the finished video.
Now also widespread "panoramic" online broadcasts. When you have several points with a panoramic view, just install the equipment and start shooting. But this format has rather high requirements for hardware, and also requires careful tuning of the sound.
Where used. This technology is widely used in the demonstration of real estate. A simple Google search gives you a lot of B2B agencies that make virtual tours for the needs of realtors and developers: 360 Real Estate , 360walkthruproperty , Forj and others.
Panoramic videos are also used in the media. Above, I already gave a link to the RT 360 project , which gives the viewer the opportunity to "be present in the moment" - be it the New Year celebration in Berlin or the international superbike cup . In addition to being posted on the Internet, such immersive materials are also on display at the permanent exhibition of the UN Office for Outer Space Affairs in Vienna.
Like many innovative technologies, augmented reality appeared in the middle of the last century for military developments. It was planned and then successfully started to be used to project important instrument readings on the face mask or the windshield of a fighter, in order to reduce the time for receiving information and increase the reaction speed of pilots.
Today, several areas of augmented reality are developing. The most common is the AR in the smartphone . Apple has been developing its AR kit for several years. Today, there are more than 5 billion smartphones in the world, and potentially this is a very large coverage for products with AR. A striking example of success is the Pokemon Go game, which has thundered around the world. There are marketing projects in which, for example, coins are scattered in AR, and the user must search and collect them in order to get a discount. Coins are scattered so that on them like in crumbs the user comes to the threshold of the store. Also worth mentioning is the recently launched Google Street project in the US, which helps with navigation. Simpler scenarios are also in demand: for example, in 2017 Positive Technologies released a merch - calendar and diary on which pictures came to life when the phone was hovering.
Other devices for working with augmented reality are augmented reality glasses . Google Glass is a not-so-successful project. Hololens, a Microsoft project, is an expensive solution, but its advantage is that it is immediately integrated with the company's ecosystem. In addition, there are many other augmented reality glasses on the market - from Ericsson, Huawei and many other large companies and startups.
Separate solutions are implemented for the industry according to the subscription model, which includes consulting and equipment repair.
How is it implemented. Projecting a virtual object into the real world can be done using a telephone. Often used tags, projection onto the glass of a car, airplane, glasses - for example, Microsoft Hololens glasses. If you use virtual reality technology on the street, you need to pay special attention to lighting, since in too bright a light AR objects can look dull. In general, although this is not the most advanced technology, even it allows you to make entertainment applications.
Where used. Holograms, overlaying pictures on the air of television channels, the use of interactive monitors - these are the incarnations of AR that seem familiar to us. However, there are more unexpected examples of the use of technology.
For example, Pepsi Max used AR technology to create its own commercial . The company installed special augmented reality monitors at one of the London stops and filmed the reaction of random passers-by.
Another example is IKEA interior planning:
Augmented reality is, as I said, one of the secrets of the success of PokemonGo, also a feature of the Argon4 application on the AR Browser SDK, with which you can, for example, provide reference information about the sights when you point the camera on your smartphone.
As for AR glasses, this area is still less developed, since current models are quite bulky and expensive. But every year, glasses are becoming easier and prettier. Another direction of AR is projection onto contact lenses. Its drawback so far is that the grid is visible on the lens. But at the same time, there are many advantages of this method.
How is it implemented. Virtual reality is primarily visual and sound content. Sound in this case is of key importance: it complements virtuality and creates the effect of being in an unrealistic location by simulating the reflection and direction of sound waves.
You can get into an alternative, virtual reality, for example, wearing special glasses. Dividing the picture before the eyes into two parts, they create a stereoscopic effect. If there is tracking for body positions, the virtual space will also take into account the movements of the head and body.
There are other ways to get into virtual reality:
Almost all major manufacturers of phones and laptops have released their virtual reality helmets: HTC, Lenovo, Xiaomi, Samsung, Oculus and other companies have them. Some are trying to create an ecosystem: for example, HP is promoting a line of laptops and helmets with manipulators and external tracking.
For such decisions, another name is used - mixed reality . In fact, it is more marketing, appeared with the release of Windows Mixed Reality - a platform for application development. At its core, it is a constructor with 3D models, a specific set of assets in the kit. In terms of the quality of the product being created, the Windows solution is much weaker than the Unreal Engine and Unity engines. They, unlike the Windows Mixed Reality platform, have not only the ability to design and create virtual objects, work with textures and with light, but also create specific business logic.
Where used. An interesting project in the field of VR was made by IKEA. The company invited its customers to wear special glasses and be in the virtual kitchen. Here, with the help of a joystick, they could read a recipe book, cook meatballs, send food for processing.
German carrier Deutsche Bahn is also experimenting with virtual reality. For four years now, the company has been offering its potential employees to put on VR glasses and demonstrate their abilities in simulating a real working day.
Actiview Israeli technology company has developed a platform for remote hiring of employees using VR-technologies. “Simulation in virtual reality allows us to control what the user sees, hears, feels. We see his behavior and can collect this data, ”comments Roy Elishkov, vice president of strategy and business development.
VR companies in various fields are also involved in Russian companies. For example, Modum Lab creates in VR projects for training and other scenarios.
We at VTB are also working on several “virtual” cases. The first of them is a VR voting service for shareholders in virtual reality format, integrated with a consultant robot. We used this product as part of the bank’s annual shareholders meeting in June 2019. The service is easily transformed: we expect that it can be adapted for voting by shareholders in any large organizations comparable in scale to VTB. And it was designed for a subsidiary of VTB Registrar, a member of the VTB Group.
The second case, which is now at the pilot project stage, is VR investments, the main purpose of which is onboarding clients for brokerage services. Here, VR is a way to remotely deliver the initial automated consultation in a quest format. In 5 minutes of a VR session, the user is completely “immersed” in the investment world, chooses the goal of compiling the portfolio, and learns what opportunities VTB brokerage products provide. After that, the manager helps the client open a brokerage account and transfer funds to it. From a business point of view, this case allows you to optimize the cost of training employees and the number of managers needed to sell brokerage services.
VR has great potential for the mortgage business of the bank. We launched the VTB VR-mortgage platform, a platform uniting developers and customers. It provides a service of remote consultation and remote viewing of real estate. The platform allows you to create a virtual space with the possibility of choosing the configuration of the apartment, a mortgage calculator and built-in sales tools, as well as connecting a consultant who will consult potential buyers remotely from another region.
AR can become a universal tool for broadcasting personalized marketing offers to our customers. However, while users as a whole are still far from, for example, pointing their smartphone cameras at an interested poster in a bank branch. So we should keep this in mind when developing an AR implementation strategy and offer customers non-trivial game scenarios, prompting them to a new interaction format.
Finally, I will try to summarize the industry experience and suggest what will happen next. Striving for a realistic image, we will face limitations in graphics and information processing speed. Hybrid systems with the participation of clouds can be a way out. At the same time, the current “cartoony” image still allows you to make “immersive” content, since this is enough for the brain. Try walking on a VR board over a VR abyss.
The penetration of VR helmets is now small, so we create VR installations in our offices and invite consumers there. It is clear that this approach has the prospect of selling expensive and complex products - real estate, yachts, cars, sightseeing tours. The key here is the transition from adhoc solutions, that is, applications that implement a specific case, to entire XR platforms that will allow buyers and sellers to find each other.
In the field of immersive technologies, sharp monetization is not yet visible, players act as if by touch. The reason is that not many people use Augmented Reality in everyday life. To try them, you need to at least download an additional application - this complicates the way consumers get technologies.
In all these developments there is no significant gamification; on the contrary, there is a desire for realism. Creating tools for easy interaction is just the main task for developers, the development of content is now secondary to the development of UX / UI. Image resolution is still low and not photorealistic, so technology has room to grow and develop. Improving user experience is associated with processor performance, with the speed of data transfer and processing.
Perhaps hybrid solutions will appear soon: for example, image processing will be performed on the server online and then it will be transmitted via radio, Wi-Fi or Bluetooth channel to the user's mobile helmet. However, so far these are only speculations. In any case, we are sure that immersive technologies will find application in many industries, including in our financial one.
The term "immersive" has now become commonly used, it can be found in reports of such eminent research companies (strategic future search) as Gartner (immersive experience) or CBInsights .
Immersive technology is also called augmented reality technology. Their list includes virtual and augmented reality, as well as 360 ° video. They provide the effect of full or partial presence in an alternative space and thereby change the user experience in completely different areas. In this post we will talk about at what level these technologies are today, how they are implemented and used for business.
First, let's define the terms and abbreviations that you can see in reports, articles and presentations:
- Immersive (eng. Immersive - immerse) - technologies for full or partial immersion in the virtual world or various types of mixing real and virtual reality.
- RR (real reality) - literally "real reality" or objective reality in which we are located and which we perceive with our senses. It is noteworthy that the reality has to be determined. But you need to understand what we are confusing virtual reality with.
- VR (virtual reality) - virtual reality - fully simulated reality using modern technology. This is not only 3D or 360 scenes, it is also sound, tactile sensations and even smells. More on this later.
- AR (augmented reality) - augmented reality - augmented should be translated more like "added." That is, we add elements of virtual, simulated reality to our real reality (RR).
- MR (mixed reality) - mixed reality - the term appeared due to the launch of Windows Mixed Reality. On the one hand, the term itself allows, in my opinion, to rebuild from competitors. On the other hand, it speaks of some specific type of devices and methods of mixing reality. Although in fact it is VR with some additions of RR. Or AR using Hololens.
- XR (extended reality) - augmented reality is the common name for AR- and VR-technologies.
- 360-photo, video - content consisting of one 360 ​​° - or several stitched photos and videos. 360 ° broadcasts are also common. Russia Today makes a lot of such content , you can see it on a cardboard with any more or less powerful phone.
Now we will analyze in more detail what is hidden under these abbreviations.
Panoramic and 360 ° photo or video
Cameras have been available for quite some time, since 2018 they have been supported by Facebook. In fact, these are sequential sets of pictures stitched using algorithms. You can make them as a single camera, or special 360 ° cameras. This is a very convenient format with which you, for example, can go on a virtual tour of the Pushkin Museum.
360 ° video
How is it implemented. 360 ° cameras shoot the surrounding area, after which the resulting videos are stitched in special programs. Seamless solutions exist, but they are more costly for the business. Sometimes additional graphics are added to the finished video.
Now also widespread "panoramic" online broadcasts. When you have several points with a panoramic view, just install the equipment and start shooting. But this format has rather high requirements for hardware, and also requires careful tuning of the sound.
Where used. This technology is widely used in the demonstration of real estate. A simple Google search gives you a lot of B2B agencies that make virtual tours for the needs of realtors and developers: 360 Real Estate , 360walkthruproperty , Forj and others.
Panoramic videos are also used in the media. Above, I already gave a link to the RT 360 project , which gives the viewer the opportunity to "be present in the moment" - be it the New Year celebration in Berlin or the international superbike cup . In addition to being posted on the Internet, such immersive materials are also on display at the permanent exhibition of the UN Office for Outer Space Affairs in Vienna.
Augmented Reality
Like many innovative technologies, augmented reality appeared in the middle of the last century for military developments. It was planned and then successfully started to be used to project important instrument readings on the face mask or the windshield of a fighter, in order to reduce the time for receiving information and increase the reaction speed of pilots.
Today, several areas of augmented reality are developing. The most common is the AR in the smartphone . Apple has been developing its AR kit for several years. Today, there are more than 5 billion smartphones in the world, and potentially this is a very large coverage for products with AR. A striking example of success is the Pokemon Go game, which has thundered around the world. There are marketing projects in which, for example, coins are scattered in AR, and the user must search and collect them in order to get a discount. Coins are scattered so that on them like in crumbs the user comes to the threshold of the store. Also worth mentioning is the recently launched Google Street project in the US, which helps with navigation. Simpler scenarios are also in demand: for example, in 2017 Positive Technologies released a merch - calendar and diary on which pictures came to life when the phone was hovering.
Other devices for working with augmented reality are augmented reality glasses . Google Glass is a not-so-successful project. Hololens, a Microsoft project, is an expensive solution, but its advantage is that it is immediately integrated with the company's ecosystem. In addition, there are many other augmented reality glasses on the market - from Ericsson, Huawei and many other large companies and startups.
Separate solutions are implemented for the industry according to the subscription model, which includes consulting and equipment repair.
How is it implemented. Projecting a virtual object into the real world can be done using a telephone. Often used tags, projection onto the glass of a car, airplane, glasses - for example, Microsoft Hololens glasses. If you use virtual reality technology on the street, you need to pay special attention to lighting, since in too bright a light AR objects can look dull. In general, although this is not the most advanced technology, even it allows you to make entertainment applications.
Where used. Holograms, overlaying pictures on the air of television channels, the use of interactive monitors - these are the incarnations of AR that seem familiar to us. However, there are more unexpected examples of the use of technology.
For example, Pepsi Max used AR technology to create its own commercial . The company installed special augmented reality monitors at one of the London stops and filmed the reaction of random passers-by.
Another example is IKEA interior planning:
Augmented reality is, as I said, one of the secrets of the success of PokemonGo, also a feature of the Argon4 application on the AR Browser SDK, with which you can, for example, provide reference information about the sights when you point the camera on your smartphone.
As for AR glasses, this area is still less developed, since current models are quite bulky and expensive. But every year, glasses are becoming easier and prettier. Another direction of AR is projection onto contact lenses. Its drawback so far is that the grid is visible on the lens. But at the same time, there are many advantages of this method.
The virtual reality
How is it implemented. Virtual reality is primarily visual and sound content. Sound in this case is of key importance: it complements virtuality and creates the effect of being in an unrealistic location by simulating the reflection and direction of sound waves.
You can get into an alternative, virtual reality, for example, wearing special glasses. Dividing the picture before the eyes into two parts, they create a stereoscopic effect. If there is tracking for body positions, the virtual space will also take into account the movements of the head and body.
There are other ways to get into virtual reality:
- A smartphone with a special VR application that fits into a special case with lenses. The easiest solution is Google Cardboard Cardboard. If you own a Samsung Galaxy device, then you can purchase Gear VR - an advanced case, complemented by gyroscopes and controls.
- Tracking systems. Allows you to move the user into virtual space. Costumes are also being developed that convey sensations from virtual reality - for example, aromas - but so far this is an expensive and not very perfect product.
- Special gloves instead of the usual joystick. This element is needed for human hands to naturally interact with the virtual world. Now gloves are inferior to joysticks in the quality of calibration, but their quality is growing.
- Mobile VR helmets with built-in monitors (HTC Vive, Oculus Go and others) . Such devices are more optimized for working in VR compared to cases for smartphones, they have high-quality graphics, have integrated sound and a joystick for control.
- “Heavy VR” helmets (like Oculus Rift). The processed graphics are transmitted to the helmet by wire from a gaming computer with a powerful video card. The need to communicate with a PC creates restrictions for use, but at the same time, heavy VR helmets have better graphics and a richer potential user experience. Tracking cameras record the position of the joystick and the position of a person, immersing him in virtual reality more realistically. Also included with such helmets are controllers.
Almost all major manufacturers of phones and laptops have released their virtual reality helmets: HTC, Lenovo, Xiaomi, Samsung, Oculus and other companies have them. Some are trying to create an ecosystem: for example, HP is promoting a line of laptops and helmets with manipulators and external tracking.
For such decisions, another name is used - mixed reality . In fact, it is more marketing, appeared with the release of Windows Mixed Reality - a platform for application development. At its core, it is a constructor with 3D models, a specific set of assets in the kit. In terms of the quality of the product being created, the Windows solution is much weaker than the Unreal Engine and Unity engines. They, unlike the Windows Mixed Reality platform, have not only the ability to design and create virtual objects, work with textures and with light, but also create specific business logic.
Where used. An interesting project in the field of VR was made by IKEA. The company invited its customers to wear special glasses and be in the virtual kitchen. Here, with the help of a joystick, they could read a recipe book, cook meatballs, send food for processing.
German carrier Deutsche Bahn is also experimenting with virtual reality. For four years now, the company has been offering its potential employees to put on VR glasses and demonstrate their abilities in simulating a real working day.
Actiview Israeli technology company has developed a platform for remote hiring of employees using VR-technologies. “Simulation in virtual reality allows us to control what the user sees, hears, feels. We see his behavior and can collect this data, ”comments Roy Elishkov, vice president of strategy and business development.
VR companies in various fields are also involved in Russian companies. For example, Modum Lab creates in VR projects for training and other scenarios.
VTB Development
We at VTB are also working on several “virtual” cases. The first of them is a VR voting service for shareholders in virtual reality format, integrated with a consultant robot. We used this product as part of the bank’s annual shareholders meeting in June 2019. The service is easily transformed: we expect that it can be adapted for voting by shareholders in any large organizations comparable in scale to VTB. And it was designed for a subsidiary of VTB Registrar, a member of the VTB Group.
The second case, which is now at the pilot project stage, is VR investments, the main purpose of which is onboarding clients for brokerage services. Here, VR is a way to remotely deliver the initial automated consultation in a quest format. In 5 minutes of a VR session, the user is completely “immersed” in the investment world, chooses the goal of compiling the portfolio, and learns what opportunities VTB brokerage products provide. After that, the manager helps the client open a brokerage account and transfer funds to it. From a business point of view, this case allows you to optimize the cost of training employees and the number of managers needed to sell brokerage services.
VR has great potential for the mortgage business of the bank. We launched the VTB VR-mortgage platform, a platform uniting developers and customers. It provides a service of remote consultation and remote viewing of real estate. The platform allows you to create a virtual space with the possibility of choosing the configuration of the apartment, a mortgage calculator and built-in sales tools, as well as connecting a consultant who will consult potential buyers remotely from another region.
AR can become a universal tool for broadcasting personalized marketing offers to our customers. However, while users as a whole are still far from, for example, pointing their smartphone cameras at an interested poster in a bank branch. So we should keep this in mind when developing an AR implementation strategy and offer customers non-trivial game scenarios, prompting them to a new interaction format.
General trends
Finally, I will try to summarize the industry experience and suggest what will happen next. Striving for a realistic image, we will face limitations in graphics and information processing speed. Hybrid systems with the participation of clouds can be a way out. At the same time, the current “cartoony” image still allows you to make “immersive” content, since this is enough for the brain. Try walking on a VR board over a VR abyss.
The penetration of VR helmets is now small, so we create VR installations in our offices and invite consumers there. It is clear that this approach has the prospect of selling expensive and complex products - real estate, yachts, cars, sightseeing tours. The key here is the transition from adhoc solutions, that is, applications that implement a specific case, to entire XR platforms that will allow buyers and sellers to find each other.
In the field of immersive technologies, sharp monetization is not yet visible, players act as if by touch. The reason is that not many people use Augmented Reality in everyday life. To try them, you need to at least download an additional application - this complicates the way consumers get technologies.
In all these developments there is no significant gamification; on the contrary, there is a desire for realism. Creating tools for easy interaction is just the main task for developers, the development of content is now secondary to the development of UX / UI. Image resolution is still low and not photorealistic, so technology has room to grow and develop. Improving user experience is associated with processor performance, with the speed of data transfer and processing.
Perhaps hybrid solutions will appear soon: for example, image processing will be performed on the server online and then it will be transmitted via radio, Wi-Fi or Bluetooth channel to the user's mobile helmet. However, so far these are only speculations. In any case, we are sure that immersive technologies will find application in many industries, including in our financial one.
All Articles