Join Newsletter

Game Engines and Machine Learning: Training a Self-Driving Car Without a Car?

YOW! Data 2019

Are you a scientist who wants to test a research problem without building costly and complicated real-world rigs? A self-driving car engineer who wants to test their AI logic in a constrained virtual world? A data scientist who needs to solve a thorny real-world problem without touching a production environment? Have you considered AI problem solving using game engines?

No? This session will teach you how to solve AI and ML problems using the Unity game engine, and Google’s TensorFlow for Python, as well as other popular ML tools.

In this session, we’ll show you ML and AI problem solving with game engines. Learn how you could use a game engine to train, explore, and manipulate intelligence agents that learn.

Game engines are a great place to explore ML and AI. They’re wonderful constrained problem spaces, tiny little ecosystems for you to explore a problem in. Here you can learn how to use them even though you’re not a game developer, with no game development experience required!

In this session, we’ll look at:

  • how video game engines are a perfect environment to constrain a problem and train an agent
  • how easy it is to get started, using the Unity engine and Google’s TensorFlow for Python
  • how to build up a model, and use it in the engine, to explore a particular idea or problem
  • PPO (proximal policy optimisation) for generic but useful machine learning
  • deep reinforcement learning, and how it lets you explore and study complex behaviours

Specifically, this session will:

  • teach the very basics of the Unity game engine
  • explore how to setup a scene in Unity for both training and use of a ML model
  • show how to train a model, using TensorFlow (and Docker), using the Unity scene
  • discuss the use of the trained model, and potential applications
  • show you how to train AI agents in complicated scenarios and make the real world better by leveraging the virtual

We’ll explore fun, engaging scenarios, including virtual self-driving cars, bipedal human-like walking robots, and disembodied hands that can play tennis.

This session is for non-game developers to learn how they can use game technologies to further their understanding of machine learning fundamentals, and solve problems using a combination of open source tools and (sadly often not open source) game engines. Deep reinforcement learning using virtual environments is the beginning of an exciting new wave of AI.

It’s a bit technical, a bit creative.

Paris Buttfield-Addison


Secret Lab


Paris Buttfield-Addison is cofounder of Secret Lab, a game development studio based in beautiful Hobart, Australia. For eleven years (so far!), Secret Lab has built games and game development tools, including the multi-award-winning ABC Play School iPad games, the BAFTA- and IGF-winning Night in the Woods, the iAward-winning Qantas airlines Joey Playbox games, and the Yarn Spinner narrative game framework. Previously, Paris was mobile product manager for SF Bay Area company, Meebo (acquired by Google). Paris particularly enjoys game design, statistics, law, machine learning, and human-centered technology and research, and writes technical books on mobile and game development (more than 20 so far) for O’Reilly Media. Recently, Paris has helped out international open-data hackathon, GovHack, as Program Manager. He is on the board of the AUC and the ACS (Tasmania), and holds a degree in medieval history and a PhD in computing. You can find him online at and on Twitter @parisba.

Mars Geldard


University of Tasmania


Mars Geldard is a Honours-year Computing student from Down Under in Tasmania. Entering the world of technology relatively late as a mature-aged student, she has found her place in the world: an industry where she can apply her lifelong love of mathematics and optimisation.

When she is not busy being the most annoyingly eager student ever at the University of Tasmania (UTAS)--or mouthing off about why everyone should love data science as much as she does--she compulsively volunteers at industry events, tutors, hikes around in the wilderness, dabbles in research, builds Game of Thrones in Minecraft, and serves on the Executive Committee for her state's branch of the Australian Computer Society (ACS) as well as the national Executive Council of the AUC. In the time she has left she is writing a book on AI with Swift for O'Reilly Media.