The massive computing resources needed to train state-of-the-art artificial intelligence systems mean that well-heeled tech companies are leaving the academic teams in the dust. This was also shown by a 2018 OpenAI study which found that the computing power used to train the most powerful AI increases at an extraordinarily fast rate, doubling every 3.4 months. Now a University of Southern California team with Intel Labs has built a way to train deep-reinforcement learning (RL) algorithms on hardware commonly available to academic researchers. This new method for training AI algorithms may reduce the computational demands, providing underfunded academic researchers with an economical way to train models.

The team explains how they could use a single high-end computer to train a model with state-of-the-art results on the first-person shooter videogame Doom, in a paper presented at this year's International Conference on Machine Learning (ICML).

Instead of running all three major computation jobs at once (simulating environment, deciding on next action based on rules, using results to update the model), the system, called Sample Factory, divides them into separate pieces, allocating resources as required. “From my experience, a lot of researchers don’t have access to cutting-edge, fancy hardware,” says Petrenko. “We realized that just by rethinking in terms of maximizing the hardware utilization you can actually approach the performance you will usually squeeze out of a big cluster even on a single workstation.”

They also tackle DeepMind Lab, a suite of 30 unique 3D challenges for RL benchmarking, using a fraction of the normal computing power!

The researchers were able to process approximately 140,000fps (twice the next best approach) using a single computer equipped with a 36-core CPU and only one GPU while working on Atari videogames and Doom. The proposed architecture combines a highly efficient, asynchronous, GPU-based sampler with off-policy correction techniques, allowing it to achieve throughput higher than 100,000 environment frames per second on non-trivial player control in 3D without sacrificing sample efficiency. As shown by the researchers in their paper, Sample Factory can also be extended to support self-play and population-based training and apply those techniques to train highly capable agents for a multiplayer first-person shooter game.

The paper notes that they chose the most challenging scenario that exists in first-person shooter games – a duel. Unlike multiplayer deathmatch, which tends to be chaotic, the duel mode requires strategic reasoning, positioning, and spatial awareness. Despite the fact that the agents trained with Sample Factory were able to convincingly defeat scripted in-game bots of the highest difficulty, they are not yet at the level of expert human players. One of the advantages human players have in a duel is the ability to perceive sound. An expert human player can hear the sounds produced by the opponent (ammo pickups, shots fired, etc.) and can integrate these signals to determine the opponent’s position. While other recent work has been beaten human players in certain parts of such games, but the task of defeating expert competitors in a full game, as played by humans, would require fusing information from multiple sensory systems.

The code for Sample Factory is available on GitHub.