Authors: Edward Caulfield, Gilberto Picareta
Release Date: June 6, 2017
Since the release of Pokémon Go, Augmented Reality seems to have turned into one of the most discussed topics in technology. The reality is that Augmented Reality has been a serious topic for many years, just under the radar of the average consumer or technophile. We are only now seeing the fruits of Augmented Reality labor that has been going on for well over a decade.
Apple, Google and Microsoft have made announcements regards to their Augmented Reality efforts and this post will attempt to take apart the different approaches used and discuss their relative merits. For the sake of convenience, I will not distinguish between Augmented and Mixed Reality, as the application is the deciding factor, not the platform. This post will only discuss publicly available information, being aware that this can change at any moment.
It has been rumoured that Microsoft has invested somewhere near $1B in Augmented reality. They are the first, and thus far only company, to develop a processor specifically for holographic computing. In 2016 they released HoloLens to the world, a $3,000 head mounted Windows 10 computer with a holographic display. In 2017 they released Windows Creator Update, which has a few nice features tied to Augmented Reality. Further improvements are planned for an additional release targeted for September, 2017. Thus far, the Augmented Reality relevant features I am seeing are:
- Paint 3D – to help you create objects for use in your Augmented Reality production.
- Holographic Shell – an environment that allows you to don a Virtual Reality headset and run standard Windows applications as well as 3D and Mixed Reality applications. Kiss your monitor goodbye!!
- View 3D (to be released in September) – according to some web based articles, View 3D will “Use simple controls to see your 3MF, PLY, OBJ, FBX, and STL files from all angles”. I have also seen examples that allow you to overlay our 3D objects on top of a camera display, as shown on the image below.
I think it is fair to say that the HoloLens wasn’t created to be sold, rather as a teaser to show the world what was possible. This year numerous big name vendors (Acer, Asus, Dell, HP, Lenovo) are scheduled to release Mixed Reality head sets based upon the HoloLens foundation. Because these devices are targeted to sell for about $300 each, I expect that they will become an immediate hit. Although they are much more expensive than a typical computer monitor, I can easily see lots of people snapping these up because of the ability to open and place application windows just about anywhere you want in 3 dimensions. Personally, I’ve been wanting this for decades…
What else does Microsoft have up their sleeves that benefits Augmented Reality? Well, they have recently announced a Cognitive Services platform and CNTK, which can be of great help when it comes to Object Recognition in Augmented Reality. Object Recognition is one of the “still difficult to solve” problems in Augmented Reality that becomes much easier with Deep Learning tools supplied by Machine Intelligence services.
What I am not seeing is serious content development tools. While Microsoft has Visual Studio, I don’t see work being done to implement Augmented Reality capability into this. Perhaps they plan on leaving this to Unity and Unreal?
What are Microsofts’ strengths? Windows, windows and windows. If you run a Windows environment, then Holographic Shell is an immediate productivity win. In spite of the proliferation of Smartphones, most companies still have Windows based computing infrastructure and this will fit right in. Considering that Augmented Reality is primarily an Enterprise focused market right now, this should play very well.
What’s the down side? Windows, windows, and windows. Smartphones own the Consumer market and Microsoft has a very low, single digit, share of the Smartphone market. As Smartphones are both Consumer and Enterprise, lack of Android & iOS support will probably cause some friction. That said, considering Microsoft’s recent purchase of Xamarin and their (theoretical?) drive to be a multi-platform player, I wouldn’t rule out broader platform support just yet.
I’ve no clue how much Google has invested into Augmented Reality, however it cannot be a trivial sum. I think we all remember Google’s first foray into the Smart glass market many years back and how that didn’t meet expectations. Actually, for the average consumer it wouldn’t be difficult to walk away with the impression that Google has absolutely nothing currently going on in Augmented Reality, although that would be a completely incorrect impression.
Let’s start with Google’s work to create a 3D capture device – Tango. Tango can be thought of as a capture engine whose primary goal is to create a 3D point cloud of an environment. Tango isn’t a product as much as a technology that Google is making available to Smartphone manufacturers. By itself, Tango is pretty much useless. However, combined with applications that know what to do with a point cloud, Tango is exceptionally powerful.
Tango’s primary drawback is that there are currently only two handsets, the Lenovo Phab 2 Pro and the Asus ZenFone AR, but absolutely no Smart glasses that deliver Tango functionality. I think that similar to the HoloLens, it will take another year, hopefully less, before additional products are announced that have Tango support.
Additionally, Google has been working on WebAR to help make Augmented Reality scanners a thing of the past. Currently, just about every Augmented Reality experience is presented in a proprietary scanner. This means that almost every experience needs its own scanner and that simply doesn’t scale. Google is working on Chrome to allow it to be the center of the Augmented Reality experience. As Chrome is open source, there is nothing to stop the work done here from being ported to Firefox or any other browser. In the long run, I suspect every browser will natively support WebAR, just like every browser supports HTML5.
While WebAR is currently tied to Tango technology, a point cloud is a point cloud and can be created using any number of devices or technologies. As a result, WebAR’s exclusive tie to Tango is a matter of convenience, not necessity.
For the Machine Intelligence portion, Google has released TensorFlow over a year ago. I experimented with the platform and quickly found that it has Object Recognition capabilities as one of the key capabilities.
While Apple has talked up Augmented Reality recently, they have been largely quiet about what they will do to enable it. It has long been rumoured that the iPhone 8 will have some Augmented Reality capabilities, but nothing concrete has surfaced yet. At Apple’s WWDC this year ARKit, a SDK for Augmented Reality, was announced.
ARKit relies on Visual Inertial Odometry (VIO), a coupling of images and motion sensors, to develop an understanding of the environment. At WWDC they claimed that ARKit enables every iPhone and iPad to become an Augmented Reality device, but this is really nothing new. Augmented Reality apps have been supporting the iPad and iPhone from day one – just not using Apple’s own Augmented Reality development tools.
Because ARKit relies on the combination of images and motion sensors, it is sensitive to lighting conditions and will have accuracy issues. It is very difficult to determine precise 3D characteristics using a single camera, even with the help of VIO. Although Apple states that they will be able to have “Scaling estimates”, this is nothing like what Tango can do and far from reliable unless you have an iPhone 7 with dual cameras.
In the realm of Machine Intelligence, Apple has announced Core ML, a Machine Learning library which uses some of the same technology as TensorFlow for Object Recognition.
Head to Head
So where does that leave us? Apple, Google and Microsoft are all working towards the same goal – to make Augmented Reality ubiquitous – each approaching the task from a different mindset.
Frankly, at the developer level the situation is frustrating because each vendor is attempting, more or less, to create a captive environment. Apple wants you to develop for iOS using ARKit, which means forget about reusing your work for Android and Windows. Microsoft wants you to develop for HoloLens, which means forget iOS and Android. With their WebAR efforts, Google is the only vendor developing with a view towards cross-platform capabilities, even if temporarily tied to Tango. Unity and Unreal attempt to be IDEs that allow for multi-platform development, but I’ve no clue if that really works as elegantly as promoted. I have my doubts.
Market Push – Currently, only Microsoft appears to be pushing the market hard for platform adoption. Google didn’t even help Lenovo promote the Tango enabled phone and Apple is playing coy about their long term plans. Microsoft’s efforts are clearly visible and give them a solid advantage.
Office Productivity – This is Microsoft’s home turf, so it is no surprise that they have an inherent advantage here. I can see Holographic Shell combined with affordable headsets being an instant hit, no matter how geeky it looks. In two or three product generations, I expect design will overtake functionality as the primary selection criteria, just like it has with eyeglasses and smartphones.
Gaming & Entertainment – I think that momentum from legacy applications, such as X-Box, give Microsoft a leg up in this market. Take all my old favourites and put them in a HMD – Cool!! Sure, Pokémon Go has proven that we’ll play location based games on hand held devices, but the same game on a headset is all that much more compelling.
Hands-Free Requirements – There are already 30 headsets available on the market already, most of them Android based. With Remote Assistance being one of the most compelling use cases for hands-free applications, you don’t really need all the benefits that HoloLens and Tango bring. I think raw Android currently has the lead here.
Anything but Windows – Microsoft immediately loses their advantage as soon as Windows drops from the equation. That said, the overwhelming majority of businesses run on a Windows centric infrastructure, so using a Windows based headset shouldn’t cause much grief. The only real rub I see is an annoying lack of interoperability with Android, iOS and OS X.
Consumer – Consumers are overwhelmingly mobile and mobile is overwhelmingly Android and iOS. But consumers aren’t the current Augmented Reality market, enterprise is.
Microsoft has done as they always do – they have created an end-to-end, shrink wrapped ecosystem. They want you to buy into the ecosystem and Augmented Reality is an opportunity for them to snare you like they did with Windows 25 years ago.
Google has done as they always do – come up with compelling technologies and set them loose in the wild. How may vendors will pick up on Tango? How long until the first Tango enabled Smart glasses are available? Someone, not Google, will put together the disparate pieces of technology and make something wonderful. Or maybe not. Google Glasses died an ignoble death, so maybe Tango and TensorFlow will also.
Apple seems to be largely inactive in Augmented Reality, unless the iPhone 8 has some really stupendous features.
Today, I would say that Microsoft has the clear lead in Augmented Reality. Let’s see who picks up Tango and makes wonderful things. Let’s see what Apple does this fall with the iPhone 8.