The Web as a Platform for Augmented Reality Experiences

Blair MacIntyre
Principal Research Scientist, Mozilla
Professor, School of Interactive Computing, Georgia Tech
@blairmacintyre /

What is AR?

(or MR or Holographic Computing?)

Mixing media with a
person's perception of the world
registered in 3D, in real-time

Other approaches to context-based media:

  • Heads-up-Displays (Glass)
  • Map mashups (Ingress, Pokemon GO)
  • Geofencing (alerts)

Ivan Sutherland, "The Ultimate Display", mid-1960's

It's an Exciting Time for AR

Head Mounted AR Displays

Microsoft's Hololens

(Perhaps HMD's will dominate consumer use when
they look like this, but that's a long way off)

Traditional (Handheld and Desktop) Displays

Google's Tango

Projection Displays

Can we use the Web for AR?

Leverage all this hardware, and all things webby!

Simple AR is Possible

WebRTC getUserMedia + JS tracking

But Is This Approach Good Enough?

See-through or Video-Mixed

Ron Azuma. "A Survey of Augmented Reality" Presence (1997)


Ron Azuma. "A Survey of Augmented Reality" Presence (1997)

Video-Mixed = VR?

Ron Azuma. "A Survey of Augmented Reality" Presence (1997)

Video-Mixed = VR + World?

Ron Azuma. "A Survey of Augmented Reality" Presence (1997)

The Challenge of AR

Must display in real time (akin to VR)

Can only display based on what we
already know or can sense about the world
relative to the display

Core problems are displays, sensing, and services for world knowledge.

Need to leverage unique capabilities
of each platform!

Simple Approach is Not Enough

Very Little World Knowledge,
Tightly Coupled to Platform

Julian Oliver "Levelhead" 2008

Learn from WebVR

People Tried to do Web+VR before WebVR

in 2013

The Argon Project

coming soon!

argon.js and (soon) Argon4 are Open Source

This Presentation is Running in Argon4

on an iPhone

Using argon.js + reveal.js + aframe.js + argon-aframe.js

Stuff around the room

Computer vision AR w/ Vuforia

Planetary scale geographic AR

Custom Reality (Panorama)

Support for Other Browsers

First-class notion of "Reality"

Any "representation of Reality" can be used, such as:

360 images or video, Google Streetview, 3D Models of the World,

See-thru (video-mixed or optical)

As long as Reality can provide

View of reality

Camera pose for viewer

Camera frustum for viewer

Decouple apps from "Reality"

Platform independence

Leverage platform capabilities

Overlay multiple apps on reality AND each other

User in Control

Time and Place (user chooses reality)

Platform (HMD, handheld, ...)

Content (multiple apps at once)

Mode (AR, VR, 2D)

Potential for More Privacy (AR without sharing all sensing)
Native apps currently get full access to all sensors

Many not want to give web pages full sensor access

Argon4 and argon.js are Webby!

Leveraged aframe.js to add AR to this reveal.js presentation

Add argon and aframe scripts up top

<script src="resources/js/aframe.js"> </script> 
<script src="resources/js/argon.js"> </script>
<script src="resources/js/argon-aframe.js"> </script> 

Add a simple AFrame scene down below


Adjust the CSS a bit, add some Javascript and we're off...

Simple Declarative 3D AR Content

<a-box position="0 3 -10" radius="0.25" color="gold" 
       rotation="0 0 45">
  <a-animation attribute="rotation" from="0 0 45" to="0 360 45" 
	       dur="1000" easing="ease-in-out"

A-Frame markup to create a spinning gold diamond

Geospatial frames of reference

<ar-geopose id="GT" lla="-84.39453 33.77250" userotation="false">
  <a-entity fixedsize="20" billboard>
    <a-plane rotation="0 0 0" width="2.9" height="4"  
             src="#buzzpin" transparent="true" ></a-plane>
    <a-entity css-object="div: #gtdiv" scale="0.02 0.02 0.02" 
        position="0 4 0" 
        showdistance="Tech Tower @ GT<br>Atlanta, GA<br>It is ">

A-Frame markup to put a pin at Georgia Tech

Simple Vuforia Setup and Use

<ar-scene vuforiakey="#vuforiakey"
    <a-asset-item id="vuforiakey" src="key.txt"></a-asset-item>

  <ar-frame id="frame" trackvisibility="true" visible="false"

A-Frame markup to put content on a visual target

ISS Maintenance w/ Traclabs

Web Ecosystem is Rich and Diverse

Many tools, from the simple to the elaborate

Mashups may suggest new ways of creating 3D!





Bring AR to the Web

Build on Advanced Web Tech: WebVR, Servo, WebAssembly, ...

Create new Web Tech: computer vision, sensor fusion, geospatial data, ...

Need new services for object or location-based search and discovery,
world knowledge, object recognition and tracking, ...

What Might WebAR Look Like?

Web Will Make Hybrid AR/VR/2D
Experiences Possible

Start exploring now at


Contact me: and @blairmacintyre

To try some of this yourself

This talk available at

Thanks to everyone who worked on Argon, and to support from Alcatel Lucent, Verizon, Qualcomm, Mozilla, NSF, GEM, GVU Center, and IPaT