DeepMind is using a deep reinforcement learning system to control superheated plasma inside a tokamak reactor.
Fusion power is a potentially transformative energy-producing technology—but making it a viable power source has proven to be a difficult endeavor. The folks at DeepMind, the subsidiary of Google focused on artificial intelligence (AI), have taken an innovative approach to finding a solution: using AI to improve control over the nuclear fusion reaction.
In nuclear fusion, the atomic nuclei of hydrogen atoms are forced together to create heavier atoms such as helium. The process generates a massive amount of energy relative to a very small quantity of fuel, making fusion a very efficient power source. It’s also significantly cleaner than fossil fuels and safer than conventional fission-generated nuclear power (which smashes atoms together to create energy).
The Tokamak: Where the Magic Happens
The fusion process takes place in an experimental reactor known as a tokamak, where the energy created by the fusion of atoms is absorbed as heat by the walls of the tokamak’s donut-shaped vacuum chamber. Extreme heat and pressure are applied to the hydrogen atoms, converting them into an electrically charged soup called plasma—which is hotter than the sun’s core. The plasma is shaped and controlled by huge magnetic coils placed around the vacuum chamber, which keep the superheated plasma from touching the chamber walls—which would cool the plasma and potentially damage the chamber.
What is a tokamak?
Controlling the fusion process has proven to be incredibly tricky, though. The plasma, which is inherently unstable, needs to be held in place long enough to extract energy from it. This requires constant monitoring and manipulating the magnetic field with huge magnetic coils at a rate of thousands of times per second.
DeepMind has been developing a solution to this obstacle: using AI to control the magnetic coils. DeepMind collaborated with the Swiss Plasma Center at École polytechnique fédérale de Lausanne (EPFL) to develop a deep reinforcement learning (RL) system that would learn how to autonomously control the coils and successfully sustain and shape the plasma.
Using AI to Control the Plasma
In a paper published in Nature, DeepMind describes how it built and ran controllers on the Variable Configuration Tokamak (TCV) in Lausanne, Switzerland. DeepMind ran its deep RL learning architecture in a powerful EPFL simulated environment that models the dynamics of a tokamak chamber. The simulation needed to be accurate enough to describe the properties of the plasma and its reaction to electric currents while remaining computationally cheap enough to enable learning. The RL algorithm used the data collected from the simulator to determine a framework that would best control the magnetic coils to sculpt the plasma. Those results were then validated on the real TCV.
The results have been promising. DeepMind was able to successfully control the plasma and sculpt it into specific shapes. Not only does this demonstrate that the plasma can be controlled but it also enables researchers to study how the plasma reacts under different conditions—which could provide further understanding and improve how fusion reactions could be created and sustained long enough to produce usable power.
Conventional plasma control systems are sophisticated. For example, TCV uses a nested control architecture where each of its 19 magnetic coils requires its own separate controller. Each controller uses algorithms to independently estimate the plasma’s properties in real time and adjust the magnet’s voltage in response. However, DeepMind’s system streamlines the process by using a single computationally inexpensive neural network to control all the coils simultaneously. The system automatically learns directly from sensors which voltages to apply to best achieve a plasma configuration.
DeepMind’s approach not only enabled the successful control and shaping of the superheated plasma but also gave researchers increased flexibility in manipulating the plasma. Researchers were able to sculpt the plasma into a conventional D-shaped elongated form as well as advanced configurations. These include a negative triangularity (imagine the D shape, but reversed to reduce reaction with the tokamak’s walls) and a snowflake shape whose legs could help spread the energy to specific points in the chamber. DeepMind even successfully sustained two separate plasmas simultaneously in the chamber.
“Our approach achieves accurate tracking of the location, current, and shape for these configurations,” claims the report. “This represents a significant advance for tokamak feedback control, showing the potential of reinforcement learning to accelerate research in the fusion domain, and is one of the most challenging real-world systems to which reinforcement learning has been applied.”
Why Reinforcement Learning?
Reinforcement learning is a unique AI technology that could be ideally suited for this work, according to Kambiz Kayvantash, senior director of ML/AI solutions for Design and Engineering at the Manufacturing Intelligence Division of Hexagon. Compared to other machine learning (ML) methods, RL “learns” on the fly without much need of a priori information about the system it’s trying to learn. The RL control policy interacts directly with the system and learns via a reward/punishment system in a somewhat autonomous manner. This enables the system to learn how to play chess and video games, or how to control robots so they can perform specific tasks.
For the fusion project, the reward was based on basic successes such as stabilizing the position of the plasma, as well as complex outcomes such as achieving a precise shape and location of the plasma for a specified period. The agent was penalized for reaching undesired outcomes.
RL is already used in smart production facilities for scheduling, ordering, resource management as well as robotics and control problematics. It’s also been deployed in virtual environments such as testing self-driving cars and designing experiments via the adaptive selection of learning databases. The technology has also been used to address engineering problems that include computational fluid dynamics (CFD) applications such as turbulence modeling, control of a rigid body in a flow, and airfoil shape optimization.
But while RL may not require much information about complex systems, it is very data-hungry—and it needs an extensive amount of training time to reach human-level performance.
According to Kayvantash, RL’s strengths lie in fields where sampling is not expensive and where data can be acquired easily. In contrast, physics-based learning is ideal for computer-aided engineering (CAE) applications due to the small number of samples required and the high cost of sampling, which is comparable to human learning in terms of the order of magnitude.
Still, RL is particularly useful in the fusion energy sector because it can operate with such limited knowledge of a complex system. In fact, researchers are increasingly turning to simulators to conduct their experiments because tokamaks are in high demand. While there are dozens of tokamaks around the world, they are expensive to operate and researchers often share need to share the tokamak with other scientists. The experiments can be time-consuming as well. For example, TCV can sustain plasma for an experiment for only about three seconds, after which it needs 15 minutes to cool down and reset before performing the next experiment.
What is Deep Reinforcement Learning?
Are We One Step Closer to Fusion Power?
“Tokamak magnetic control is one of the most complex real-world systems to which RL has been applied,” note the study’s authors. “This is a promising new direction for plasma controller design, with the potential to accelerate fusion science, explore new configurations and aid in future tokamak development.”
DeepMind’s successful demonstration of AI-powered tokamak control showcases the potential of AI to help researchers get closer to the elusive goal of fusion energy. And if reinforcement learning can be used in a tokamak, it could potentially have a bright future in helping control a wide variety of complex machines. In fact, DeepMind is leveraging machine learning in collaboration with expert communities in fields such as quantum chemistry, pure mathematics, material design and weather forecasting—with the goal of using AI to solve complex real-world problems.
Read more about the difficulties in making fusion energy a reality at Why is Fusion Power is Always 50 Years Away?
