What is Augmented Reality (AR)

Human perception is naturally constrained by the physiological limits of our sensory organs. For thousands of years, we’ve been using a variety of instruments to enhance our perception of the world, allowing us to simplify activities in our work and personal lives, discover the world around us, and entertain ourselves.

On some level, optical devices with rulers and scales, spyglasses, sextants, microscopes and telescopes augment our perception of reality. Still, we use the term augmented reality technology to designate computer processed information about the world, layered with virtual objects based on identified images.

[ht_termine termine="Augmented reality (AR)" transcript="is a combination of the user’s view of the real world with superimposed computer-generated virtual objects. Augmented reality can enhance visual, auditory and sensory perception."]

Computing technology advancement in the second half of the XX century presented an opportunity to create systems that could identify real objects and track their movements. These first experiments in the field of computer vision were already advanced enough to superimpose simple geometric figures onto the real world and interact with them.

Augmented reality technology

The very first AR applications had to be viewed using special bulky equipment, glasses and helmets. Recent developments in computing technology allow for compact solutions using regular smartphones to interact with augmented reality environments.

One of the most commonly used approaches in AR is based on integrating special tags into real objects that can be read by scanning devices such as smartphones or AR glasses. QR codes, NFC tags or RFID can all serve as tags.

Another AR approach uses computer vision to superimpose virtual objects onto any kind of environment. A special app uses a camera to scan the user’s location and places AR objects in this environment. This approach is more versatile but may require costly computations.

Virtual and augmented reality

There is often some confusion between the two technologies — VR and AR — although it’s not too hard to tell them apart.

VR is a technology that creates a completely simulated environment. The user employs special gear to receive visual, auditory and sensory information describing an artificially constructed location.

Augmented reality technology superimposes virtual objects onto a real environment. AR also requires special gear, but instead of constructing an environment from scratch it simply overlays the real environment with additional elements.

The history of augmented reality technology development

One of the first virtual reality devices was a simulator called Sensorama, developed by cinematographer and inventor Morton Heilig in 1957. The device showed a first-person film to its viewer, accompanied by vibrations and even smells. Although Sensorama functioned as a virtual reality device, the invention contributed to the development of augmented reality technology as well.

The first invention that superimposed virtual objects onto the real world appeared in 1968. Computer scientist Ivan Sutherland joined forces with his student Bob Sproull to create an AR system they called «The Sword of Damocles». The user would wear a special device consisting of a pair glasses with a built-in display, camera and eye tracking system. Due to its large weight, the contraption had to be attached to a mechanical arm suspended from the ceiling, which is what gave the device its name. The system displayed simple outputs from a computer program in the stereoscopic display: a geometric figure that changed with the user’s head and eye movements.

The Sword of Damocles system

1974 saw the invention of a device that allowed users to work with AR without using glasses or helmets. The Videoplace virtual reality laboratory was a set of equipment, including cameras, projectors, etc. The system projected various interactive images, including human silhouettes and virtual objects.

In 1981, Canadian engineer Steve Mann developed the very first wearable AR device — EyeTap. The system included a pair of glasses equipped with a camera and display and a backpack holding a portable computer and battery pack. Since then, Steve Mann has modified his invention several times to make it more portable and functional.

Steve Mann with the first prototype of EyeTap

In 1982, TV weather forecasts started using augmented reality technology. Movements of air masses were tracked using special radars and cameras based in outer space.

In the late 1980s, the US Air Force started using augmented reality technology to develop special AR helmets for combat pilots.

The 90s saw the rapid development of AR technologies. The term «augmented reality» was first coined by Tom Caudell, a researcher at Boeing. At the time, the company was developing AR helmets for its engineers that would allow them to get rid of expensive and inconvenient diagrams written on plywood.

The microprocessor revolution allowed scientists to create progressively more compact solutions.

In 1993, the KARMA AR helmet was created, including interactive printer maintenance instructions.

In 1995, Navicam — a gadget constructed like a modern tablet, including a camera at the back of its body — was developed to identify colored tags on objects. Navicam could rapidly process the video stream it received from its camera and display information about each object on its screen.

A more advanced system appeared in 1996, called CyberCode. Virtual objects were scanned by using two-dimensional barcodes, similar to modern QR codes. When the user scanned these CyberCode tags with the camera, the system displayed virtual objects.

In 1998, the NFL started using AR in live broadcasts of football games. Additional technical lines and the current score were overlaid onto the broadcasts.

In 1999, NASA added AR to the dashboard of its X-38 spacecraft. This new system made it possible to see what is happening on Earth in any weather conditions.

Augmented reality technology development. Programming tools

The first open library for AR applications appeared in 1999. The developer responsible for creating this library, Hirokazu Kato, called it ARToolKit. This library made it possible to write programs capable of identifying camera position and orientation in real-time.

Creators of modern development tools, or SDK for AR applications, prioritize development for mobile platforms — Android and iOS. Most development environments also support desktop applications.

Working with certain AR platforms requires a specialized toolkit: helmets, infrared sensors, or several cameras. But developers soon realized that augmented reality will only gain widespread popularity when AR applications become accessible to owners of regular smartphones featuring a single camera.

For major IT companies, delayed reactions to new technologies might cost them leading positions in the market of the future. Quite often, the company’s presence in a certain industry is defined less by products, and more by available development tools. That’s why tech giants like Google and Apple have their own SDK for creating augmented reality applications.

ARCore is a set of instruments for developing AR applications for Android. Google released the first version on March 1, 2018. One of the advantages of ARCore is its ability to support other APIs working with augmented reality.

The core features of the framework that allow developers to unite the real and virtual worlds include tracking the position of a user’s phone in relation to its surroundings, identification of real objects and their size, and positioning of different surfaces. Apple algorithms also measure lighting conditions in order to make virtual objects look more realistic.

Essentially, applications written using ARCore can track a phone’s position in space in real-time, and integrate virtual objects into a real environment. Users can use their smartphones to interact with augmented reality. Some real apps created using ARCore include Just a Line, ARuler, IKEA Place.

ARCore is absolutely free and supports Android above version 7.0 and iOS above version 11.

ARKit is an Apple toolkit created to develop AR applications and games for iPhone and iPad.

The first version of ARKit was only equipped with the option to create static 2D images. Starting with ARKit 2.0, the algorithms were upgraded to track moving two-dimensional images and identify static 3D objects.

ARKit can create a polygon face model. Apps can track the intensity, temperature and direction of light, as well as language and the intensity of the user’s gaze, which allows them to identify emotions.

The platform is free and supports iOS above version 11. Developers interested in further development of their AR programs can purchase a yearly subscription for $99.

Many developers specializing in creating AR toolkits offer expensive frameworks. Popular solutions include Vuforia, Wikitude and EasyAR.

Applications of AR

The very first AR developments were essentially scientific experiments. But every new milestone created new areas for practical application of this technology. Education and technological processes were the first areas where developers could test their inventions. Augmented reality simplified visualization of complex ideas and mechanisms. In the production line, AR reduced complexity and costs for technological processes — Boeing’s AR helmets serving as an excellent illustration of this fact.

The current level of electronics and computer science development opens virtually unlimited opportunities for AR applications. Scientists, doctors, actors, game manufacturers, and marketing experts alike can benefit from this technology.

Augmented reality in medicine

AR solutions can aid in training new specialists in the medical field. For example, an app called ARnatomy helps students learning about the human skeleton. German app PalpSim AR teaches medical students palpation techniques.

Other developments assist in controlling the patient’s condition during operations without using a complicated sensor system.

Furthermore, augmented reality technologies illustrate various procedures — such as sonograms — with much needed clarity.

Augmented reality can be beneficial to patients as well. VA-ST is an AR device that paints in the features of a conversation partners for visually impaired patients.

Augmented reality in education

AR enables its users to simultaneously take in different types of information with their eyes and interactively control virtual objects, enabling a lot of applications in education. This technology can help students learning almost any subject. Mathematics students will better understand how geometrical constructions are built, while history students will be able to reconstruct historical events, combining them with maps. AR can help visualize complex chemical formulas. Naturally, using augmented reality in education must be accompanied by well crafted and structured textbooks.

New Horizon — an AR app created by Japanese developers — serves as an excellent example of AR applications in education. When the user’s smartphone camera scans a page in their textbook, the app shows them animated character corresponding to the material on that page.

Augmented reality in marketing

AR offers numerous applications in promoting and advertising products and services. For example, AR tags included on billboards would enable users to scan for information about the advertised product and instantly purchase it. Virtual stores are gaining traction in China, with virtual store counters that can be accessed using augmented reality, allowing the user to purchase real items in a virtual environment.

American company Matterport develops AR solutions for the real estate market.

Some magazines and other publications post effective ads on their pages, allowing users to examine 3D models of cars, furniture, gadgets or any other objects in great detail.

Augmented reality in entertainment

Today, AR technology is most widely used in the entertainment industry. Major players such as Lego, Disney and Apple dedicate a lot of resources to developing entertaining AR applications.

AR shooter games represent one of the most captivating and popular genres of augmented reality games.

Founded in 1997, Wearable Computer Lab ported Quake, the very first 3D game, into AR. Similarly to the original game, ARQuake allowed users to fight monsters, except in this case they became a part of the real world.

Father.IO is an excellent example of a first-person shooter in augmented reality. Players attach a special device, that allows them to identify enemies at a distance of 50 meters, to their phones.

The game is set in 2040. According to the plot, some humans have evolved and become very different from regular Homo sapiens. Artificial intelligence, infected by a virus, wants to kill all humans that haven’t mutated. Two teams of players fight each other in augmented reality — one of them consists of mutants, the other one only includes regular people. The map of the game is divided into hexagons, with each one controlled by a single team.

Players can stream the game on YouTube or Facebook. The game is free to download, but you need to purchase a special gadget for your smartphone in order to play it.

Pokemon Go is a game that has given a powerful boost to the development of AR technology in the entertainment industry. Millions of users were drawn into this game that allowed them to move their avatar along a virtual map and find Pokestops. The game’s popularity contributed to a sharp increase in profits and capitalization for Niantic (the developing studio), Nintendo, and several other companies connected to the game.

Modern augmented reality devices

Google Glass

Google presented a prototype of their smart glasses, Google Glass, back in 2013. The gadget didn’t have much in common with augmented reality technology, because the virtual elements did not interact with the real environment.

Some of the company’s services were integrated into the glasses: Google Now, Hangouts, Maps, Search and Translator. The glasses could also be used for taking photos or videos, which would then be uploaded to the cloud, as well as sending messages, finding out the weather or flight information, or getting first aid instructions. Google Glass was controlled by voice commands. For example, users could use the phase ‘Ok, Glass, take a picture’ to take a photo.

Google’s invention did not become popular, but it served as a signal to other companies, and a new market and competition emerged.

In 2015, Google ceased production of Glass.

Microsoft HoloLens

In 2016, Microsoft presented its HoloLens gadget, which works with a special version of Windows 10. The company also developed a special processor to handle large amounts of data.

HoloLens is still unavailable to regular users, because the product is primarily geared towards business clients — NASA, Volvo, Disney и Universal.

The gadget allows users to work with the regular Windows interface in augmented reality, creating new windows in space. Furthermore, several games have been developed for HoloLens, as well as travel app HoloTour, 3D model editor HoloStudio, and visual anatomy aid HoloAnatomy. There are also versions of Paint 3D и Skype available for use with this gadget.

Future prospects

Many large companies and startups are currently working on their own concepts for AR glasses and helmets, including players like Vuzix, Sony, ODG, Solos, Magic Leap and others.

However, the market for mobile apps with practical AR features is currently much more active. Expensive and complex gadgets may spark enthusiasm, but they aren’t accessible — or very useful — for the mass user.

Some apps that are already in high demand include:

AirMeasure — a virtual tape measure that lets its users measure distances and dimensions in 3D.
The mobile version of Google Translate that comes with the feature of real-time text translation.
Sun Seeker — an app that identifies the trajectory of the sun at any time, on all terrains.
Google Sky Map — an app that offers the opportunity to study a map of the stars.

One of the companies with the most interest in AR development is Facebook. The company is currently working on integrating AR masks and other effects into its products. Facebook developers are also working on developing algorithms that can not only track faces, but bodies as well, which would allow their apps to identify silhouettes and poses.

Today, the majority of successful AR developments are focused in the entertainment industry, which means that most developers are also interested in this field. But as the technology further develops, it’s only a matter of time and investment until it is integrated into science, healthcare, security systems and users’ everyday lives. So in the nearest future, we can expect augmented reality technology to spread across all kinds of products and services.