The controller trained with deep reinforcement learning steers the plasma through multiple phases of an experiment. On the left, there is an inside view in the tokamak during the experiment. On the right, you can see the reconstructed plasma shape and the target points we wanted to hit. Credit: DeepMind & SPC/EPFL

Controlling a substance as hot as the Sun

Tokamaks form and maintain plasmas through a series of magnetic coils whose settings, especially voltage, must be controlled carefully. Otherwise, the plasma could collide with the vessel walls and deteriorate. To prevent this from happening, researchers at the SPC first test their control systems configurations on a simulator before using them in the TCV tokamak. “Our simulator is based on more than 20 years of research and is updated continuously,” says Federico Felici, an SPC scientist and co-author of the study. “But even so, lengthy calculations are still needed to determine the right value for each variable in the control system. That’s where our joint research project with DeepMind comes in.”

3D model of the TCV vacuum vessel containing the plasma, surrounded by various magnetic coils to keep the plasma in place and to affect its shape. Credit: DeepMind & SPC/EPFL

DeepMind’s experts developed an AI algorithm that can create and maintain specific plasma configurations and trained it on the SPC’s simulator. This involved first having the algorithm try many different control strategies in simulation and gathering experience. Based on the collected experience, the algorithm generated a control strategy to produce the requested plasma configuration. This involved first having the algorithm run through a number of different settings and analyze the plasma configurations that resulted from each one. Then the algorithm was called on to work the other way – to produce a specific plasma configuration by identifying the right settings. After being trained, the AI-based system was able to create and maintain a wide range of plasma shapes and advanced configurations, including one where two separate plasmas are maintained simultaneously in the vessel. Finally, the research team tested their new system directly on the tokamak to see how it would perform under real-world conditions.

Range of different plasma shapes generated with the reinforcement learning controller. Credit: DeepMind & SPC/EPFL

The SPC’s collaboration with DeepMind dates back to 2018 when Felici first met DeepMind scientists at a hackathon at the company’s London headquarters. There he explained his research group’s tokamak magnetic-control problem. “DeepMind was immediately interested in the prospect of testing their AI technology in a field such as nuclear fusion, and especially on a real-world system like a tokamak,” says Felici. Martin Riedmiller, control team lead at DeepMind and co-author of the study, adds that “our team’s mission is to research a new generation of AI systems – closed-loop controllers – that can learn in complex dynamic environments completely from scratch. Controlling a fusion plasma in the real world offers fantastic, albeit extremely challenging and complex, opportunities.”

A win-win collaboration

After speaking with Felici, DeepMind offered to work with the SPC to develop an AI-based control system for its tokamak. “We agreed to the idea right away, because we saw the huge potential for innovation,” says Ambrogio Fasoli, the director of the SPC and a co-author of the study. “All the DeepMind scientists we worked with were highly enthusiastic and knew a lot about implementing AI in control systems.” For his part, Felici was impressed with the amazing things DeepMind can do in a short time when it focuses its efforts on a given project.

The collaboration with the SPC pushes us to improve our reinforcement learning algorithms.— Brendan Tracey, senior research engineer, DeepMind

DeepMind also got a lot out of the joint research project, illustrating the benefits to both parties of taking a multidisciplinary approach. Brendan Tracey, a senior research engineer at DeepMind and co-author of the study, says: “The collaboration with the SPC pushes us to improve our reinforcement learning algorithms, and as a result can accelerate research on fusing plasmas.”

This project should pave the way for EPFL to seek out other joint R&D opportunities with outside organizations. “We’re always open to innovative win-win collaborations where we can share ideas and explore new perspectives, thereby speeding the pace of technological development,” says Fasoli.

Reference: “Magnetic control of tokamak plasmas through deep reinforcement learning” by Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Marc Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis and Martin Riedmiller, 16 February 2022, Nature.DOI: 10.1038/s41586-021-04301-9