Ali Spittel - Web Ready Augmented Reality in 47 Lines of Code - View Source 2019


ALI: Awesome. I'm the difficult one that even my own computer because, demos.

So, I'm Ali, I'm a software engineering instructor. I teach other people how to code. Today I'm going to be talking about augmented reality in JavaScript. So, a little bit of what we're gonna be talking about. So, we're gonna talk about what augmented reality is. And then we're going to play with a ton of different demos. After that, we will talk about the different types of augmented reality there are out there.

Look at some my code. I wrote Snapchat's filters in 37 lines of JavaScript. Talk about that. And talk about the future, what augmented reality will look like in a couple of years. So, to define augmented reality, you're taking the real world and then you're adding something to it. You may have heard of virtual reality. That's creating a new world in 3D.

I think augmented reality is really powerful because you can take the real world and then add something to it. And that's really powerful. It can mean for any sense, but the one that you see most often for now is visual. So, you'll have the real world and have some visual element added to it. So, I am talking about augmented reality not because it's something I do in my day job every day, but because I like art and code. And I have a blog.

And so, one of my blog posts, about two years ago, was about how to build augmented reality apps in JavaScript. And it turns out that a lot of people thought that I would be interested in giving talks about it, which I am. But I am not somebody, again, who writes augmented reality apps all day. At that point I didn't feel like I was an expert at it. That's the reason I'm talking about it.

I was so intimidated by the this idea of augmented reality, I thought you would have to know all this math and wild code and then did more research and it turns out you can do it in JavaScript. It's basically just DOM manipulation is my whole entire application. I think this is really, really cool. At the time I saw a lot of apps coming out using Python or C++ and they were command lines that you could run and filter off of that.

And that's good, but you have to be a programmer to run those applications. You have to download them and have Python running. And if you have built something for the web, anyone can use it just by going to a URL. Mobile apps are cool, but again, you have to download something, and the development is a little bit higher barrier to entry. JavaScript makes development accessible and also makes them accessible.

That's why I wanted it use it to build an augmented reality app. There are so many different applications for augmented reality already. Art, which is one that's really interesting to me. And I think augmented reality is going to be a really huge forefront there. There is gaming as well. You may have seen something like Pokemon Go, we'll talk about that a little bit.

And then businesses are starting to use it as well. So, it can be a pretty powerful thing for marketing or just trying out elements as well there. And then there's also more too. So, we'll talk about some that have. So, one of the most common questions that I get asked about augmented reality is, when will we use augmented reality every day? And I know for the average person in the audience, maybe you don't use augmented reality every day.

But for a different demographic, mostly teenage and girls and young women, it is something that they're using every day. And my demographic is. And so, I want to talk about that and how powerful it already is. There's going to be even more of a future for it. But it's already massive. So, let's look at some demos. So, the first two are Snapchat and Instagram. Instagram is so culturally dominant right now. Not just the app itself, but the world around it.

Like, I go to events and they have Instagram installations so that you take pictures of it and market the event for them. It's a huge piece of it. And also, people buying ads on Instagram. So, that one is absolutely massive. They have in the stories you can put on filters on your face there. And then Snapchat who they pretty much just knocked off to do it. So, Snapchat lets you take images and put them on your face and alter your appearance then.

Those are the original ones. The fun part of this talk is you get to see my face a lot. Which is super fun because I'm especially jet lagged. So, that's fine. One that you may have seen on social media is Face App. What it does is takes your face and does some machine learning on it and spits it out in different ways. So, the one on the that side, the woman one, is a tweaked version of my face. So, I don't know what they changed about it. But they made it way prettier and it's super impressive. And the other side is the male version of my face, apparently. I'm glad that didn't turn out that way for me.

There's also face tune, which is a visual editing software for your phone. And you can essentially take what people made careers off of, airbrushing people back in the day, and you can do it in a couple swipes. Which is really powerful, but also terrifying. You can try on makeup. So, Sephora's website lets you try on all sorts of different makeup. I don't own blue lipstick, but I was wearing some today. IKEA, you can take the app and look at what it will look like dimension wise in your own using this augmented reality application.

Tesla, they're using augmented reality for manufacturing. They're using it to test the cars before putting them out into the wild. They're also experimenting for the dashboard on your computer. Trying to have different panels on there using augmented reality as well. This is one of my favorite AR applications from 2010. So, it was a while ago. Zach Lieberman is an incredible artist. And hopefully this will load. oh, no. Let me try this. Sorry. Looks like I got disconnected. Oh, no! There we go. Okay. Well, I guess that is not working. Sorry about that. So, also museums. Museums are doing really, really cool things with augmented reality.

So, both for extending the artwork in their museums, but also for displaying their content in different ways. For example, walking tours. It used to be that you had like headsets and would type in numbers at different pieces of art and they would tell you about it. Now, using augmented reality, you can walk to places and based on your location, you will hear information about the artwork. That's incredible as well.

The Louvre also made it so that you could project their artwork no matter where you were in the world. So, in Los Angeles, they had a warehouse and converted it into the Louvre by doing projections there. That's really cool as well. Pokemon Go, you may have played this. I didn't really, but my mom was into it even though she didn't know Pokemon. That was really fun. You could capture these little animals that were out in your world and that game was huge for a while. And there are other games like it as well now. Rock climbing.

Rock climbing is one of my big hobbies and people are using augmented reality to create climbing routes and climbing competitions which used to be static. You used to have a route setter and put tape up and that's how it is now, but you can use augmented reality, allowing for quicker and dynamic route settings. That's really cool as well. And then it even goes beyond that. They're using augmented reality for medical purposes as well. In order to increase contrast for some people who have visual impairments.

So, you can see the different lighting that may help some people see certain settings better. So, it goes beyond this artistic and marketing world as well. So, there are a couple different types of augmented reality. The first one was marker based. So, marker based augmented reality was the very first type.

And you would have something that was almost like a QR code or this serial marker here and you would project something based on top of that. So, that was pretty limited at that time, but a lot of augmented reality is still based off of markers. So, something in reality that you're adding something to. Usually your face. But there are other things as well that you may add augmented reality on top of. Markerless augmented reality is similar, but just no markers.

So, you're adding something to the world without an item that you're adding it on top of. So, something like Pokemon Go, something that's location based would be markerless augmented reality. There's projection based augmented reality. Something like that climbing route where you're projecting something out into the real world using augmented reality. Superimposition based. You have something added.

This is a combination with marker and markerless. Something like IKEA, projecting the couches into the real world would be superimposition based. I want to talk about the ways that you can build augmented reality apps now. So, one way that you can is through something called A Frame. And ARjs on top of it. The really cool thing about it is that you can build augmented reality apps without using JavaScript. You can write HTML code and have augmented reality apps built out.

You have to use a marker, like the Hiro marker like I showed you before. It's a little bit limited in that respect. Not as fancy as the face based augmented reality. But you can build something like that where it projects a cube on top of the Hiro marker and 10 lines of HTML is the only code you have to write. It's pretty powerful.

And then I have been using tracing JS. So, tracking.js is a computer vision library for JavaScript. And what it allows you to do is detect things in the real world and then you can use the coordinates of the thing in the real world and build on top of it. So, I did face recognition. But they also have color based recognition or parts of your face or your hands or something along those lines. And it's pretty lightweight. It's in the massive, massive library by any means. And it's super accessible. They have great documentation.

And it's not impossible to write by any means. You don't have to do a ton of the math yourself. And so, we're gonna go ahead and look at my app in my code. So, I'm gonna switch over now. So, hopefully this works on stage and it's giving me the flower, it's doing that for anyone on the projection. I can switch it over to bunny ears as well. And it gets my face there. And this whole application was 47 lines of JavaScript.

So, I'll show you that code now. So, starting off with the HTML, it's pretty basic. The most difficult part of this whole project was actually stacking a Canvas on top of a video element that turned out to be a little bit tricky. But the fact that the CSS was the hardest part says a lot, I think. So, again, I have a video element which is how my video is taken from my webcam. And then I have a canvas on top of that. The canvas is where the actual filters are projected on top of. So, I have those two elements there.

And then my JavaScript. So, the at beginning, it's just a bunch of grabbing things from the DOM that I'm going to be interacting with. And then I'm creating a new tracker for the faces on the screen. And then I'm grabbing the flower crown and the button. I'm taking the image. It's gonna be an image object so that I have all the information about it. That's going it store the crown itself and the bunny face as well.

And then you have the XY width and height. I have to scale those to fit better. I have ability to change pictures. And then I have the ability to change to the flower crown and the bunny ears as well. These coordinates here are just scaling for the images. I found two random PNGs on the Internet that were the flower crown and that bunny face. And so, I had to edit them a little bit in order to be the proper size.

And so, this is just scaling them. So, they're X and Y coordinates and they're width and height. By default, they fit on somebody's head. This was the best for this because I was just tweaking that. But, again, not a huge amount of code here. And then some more tracking code. The first one is set initial scale.

This is how big the box around your face is. Behind the scenes tracking JS is making a box around where your face is. This is saying that the initial scale is going to be 4. If it's 10 instead, then it's a lot big and the crown goes way outside my head, that's what it's doing there. The step size is what it's going to track to see the faces on the screen. This one is too, it checks for new faces pretty frequently on that video.

But you could speed it up, so it tracks faster. It's a little bit less performant in that way. Or slow it down so it's only looking for faces every couple seconds. And the last one is how picky the model is. It's behind the scenes. It's a binary classifier and I will talk about the algorithm later. And this is picking how picky that model is.

Turned this town, less than .1, it's going to get false positives and thinks that things that are not faces are faces. So, I built this code in my instructor's lounge which has whiteboard walls. There were pictures on the walls everywhere. And at first before I set this edge density, it kept thinking that all the drawings behind me were faces. And they were not. Then if I make it way more picky, it won't see my face in some lighting like this or backstage or something along those lines.

So, that's just something to play with to decide how picky you want the model to be. I'm making it track the actual video element on that page. So, that's how I set that up. And then here's the actual tracking. So, it's an event. Listens for the tracking event. When that happens, it's going to clear my canvas that I already have there. I'm gonna make it blank. Then for every face that's in that event data, it is going to draw an image on it.

So, that's all that I had to do. Most of this isn't even code that is related to tracking JS. And I have this fully working AR app. I think I reformatted it so it's 51 lines of code. But it's normally 47. I added the three comments. That's why. I didn't even code golf this. Anybody use Code Golf? So, code golf is when you write code that's as few letters as possible to pass challenges. I have fun doing that. I didn't do that, I have full variable names and multiple things online. I could have made it less than 47 lines of code if I really wanted to.

But not a huge amount of code for a pretty cool app. So, the algorithm. So, tracking JS uses the Viola Jones facial detection. Facial detection and  facial detection is face, not space. And facial recognition is whose face is. This is a binary classifier, it’s a face, this is not a face. What it looks for is people's eye region are further back so it's darker than the rest of your face. And the nose and forehead region, cheekbones, are brighter. It's looking for that essentially.

It sums up the pixel values in any rectangle and uses that pixel value of dark and lightness in order it see if something is a face or not. And then yeah. So, that's the Viola Jones algorithm right there. It's used by a lot of facial detection software. Not just tracking.js. So, that is tracking.js. And it has a lot of other classifiers as well.

It has ones for different colors. It has things for your eyes or your nose or something along those lines. If you wanted to make your filters even better. I tried to add the dog filter at some point. But that got too difficult and like the nose and eye gap was too hard to implement. So, definitely would recommend using those pickier models if you were trying to do even more complicated facial filters. There are also a lot of other libraries out there as well for doing this.

Other JavaScript libraries. And the reason I didn't go with them is just because the documentation was worse. And so, that's a vague call to action for open source maintainers to make sure that your documentation is good. But what comes next? I think augmented reality is going to be huge in the future for a lot of different purposes. I think that the artistic applications are really powerful. In Washington, D.C., where I used to live, there's this art museum called the Art Tech House. And they've had tons of augmented reality exhibits there.

And so, definitely a lot going on in that world. And then, as far as the code goes, I think that it's going to become more and more accessible. So, Mozilla is coming out with mixed reality APIs that are built straight into the browser. So, you don't need to download another library or anything like that. You can build augmented reality and virtual reality apps straight with this browser API. In addition, there's also the face detection API.

That one's implemented in some versions of Chrome. And that detects whether something is a face or not already just in JavaScript. And just in the browser itself. So, again, you don't have to download anything else. It's really incredible how fast this is evolving and how much is already getting built into the browser. And it's even gonna go further, I think, in mobile applications as well. The thing with websites is that you're really limited by webcams and webcams aren't necessarily the best harnesser of information in the world.

And so, there's definitely going to be more with mobile apps than anything. In addition, the marketing context is going to be really interesting as well. Trying on clothes, trying furniture, all sorts of different things right into your browser.

So, that's all these are more resources as well. If you're interested in reading up about any more of the heavy mass side or anything like that, I've got all of these linked. My slides are online, Ali Spittel they will be up online as long as history is going. Unless I delete GitHub or something. They are online.

You can look through all these resources on getting started with augmented reality. There's the Viola Jones paper based off of that mathematical algorithm that tracks. And my article on using tracking.js to make the Snapchat filters. Awesome command new bee, if you saw Saron doing the podcast, she has another. Zach Lieberman, my favorite augmented reality artist. I have his Instagram link there. And I have some of the libraries that you can use as well.

And keep in touch if you want to reach out. I will be at this event tonight and tomorrow as well. But you can also find me online if you want to talk more about augmented reality. And thank you all so much.

Discover the best of web development

Sign up for the Mozilla Developer Newsletter