Building Jarvis – by Mark Zuckerburg

My personal challenge for 2016 was to build a simple AI to run my home — like Jarvis in Iron Man.

My goal was to learn about the state of artificial intelligence — where we’re further along than people realize and where we’re still a long ways off. These challenges always lead me to learn more than I expected, and this one also gave me a better sense of all the internal technology Facebook engineers get to use, as well as a thorough overview of home automation.

So far this year, I’ve built a simple AI that I can talk to on my phone and computer, that can control my home, including lights, temperature, appliances, music and security, that learns my tastes and patterns, that can learn new words and concepts, and that can even entertain Max. It uses several artificial intelligence techniques, including natural language processing, speech recognition, face recognition, and reinforcement learning, written in Python, PHP and Objective C. In this note, I’ll explain what I built and what I learned along the way.

Diagram of the systems connected to build Jarvis.

Getting Started: Connecting the Home

In some ways, this challenge was easier than I expected. In fact, my running challenge (I also set out to run 365 miles in 2016) took more total time. But one aspect that was much more complicated than I expected was simply connecting and communicating with all of the different systems in my home.

Before I could build any AI, I first needed to write code to connect these systems, which all speak different languages and protocols. We use a Crestron system with our lights, thermostat and doors, a Sonos system with Spotify for music, a Samsung TV, a Nest cam for Max, and of course my work is connected to Facebook’s systems. I had to reverse engineer APIs for some of these to even get to the point where I could issue a command from my computer to turn the lights on or get a song to play.

Further, most appliances aren’t even connected to the internet yet. It’s possible to control some of these using internet-connected power switches that let you turn the power on and off remotely. But often that isn’t enough. For example, one thing I learned is it’s hard to find a toaster that will let you push the bread down while it’s powered off so you can automatically start toasting when the power goes on. I ended up finding an old toaster from the 1950s and rigging it up with a connected switch. Similarly, I found that connecting a food dispenser for Beast or a grey t-shirt cannon would require hardware modifications to work.

For assistants like Jarvis to be able to control everything in homes for more people, we need more devices to be connected and the industry needs to develop common APIs and standards for the devices to talk to each other. Read the whole article here: Building Jarvis

This entry was posted in Technology. Bookmark the permalink.

One Response to Building Jarvis – by Mark Zuckerburg

  1. alfy says:

    I knew Jarvis back in the seventies, and thought he was dickhead, even then. I am not sure I would trust his modern manifestation to successfully flush the loo, leave alone control every aspect of my home.

Leave a Comment (email & website optional)