Oak Ridge Leadership Computing Facility

Researchers harness ORNL supercomputers to understand solar storm/magnetosphere interaction

If the sun is anything, it is reassuring. It rises, sets, and rises again, allowing us to grow crops, get tan, and power homes, just to name a few of humanity’s most important life-sustaining functions. No wonder it was considered a deity by countless ancient civilizations.

Like many other things, however, our sun is prettier at a distance. Turns out the sun is a violent place where magnetic fields and fusion energy spew plumes of radiation into outer space and at Earth, a phenomenon referred to by space physicists as space weather.

Fortunately for us, Earth’s magnetic dipole creates a type of shield known as the magnetosphere. Unfortunately, though, it’s not perfect. The radiation unleashed by the sun in the form of an ionized gas known as plasma can occasionally sneak past Earth’s defenses in a process known as magnetic reconnection, creating the northern lights and wreaking havoc on electronics and our daily lives. These solar storms can knock out whole power grids, rendering entire regions without electricity and, ironically, light. So far more than $4 billion in satellite damage from space weather has been documented. And more airlines are beginning to fly closer to the poles, where there is maximum solar radiation and a potentially greater risk to passengers. While ancient civilizations often worshipped the sun, modern humans occasionally have reason to rue it.

When a solar storm occurs, the radiation can take anywhere from 1 to 5 days to reach Earth. That’s the good news. If we earthlings knew a storm was headed our way, preemptive measures could be taken, preventing power and communication disturbances and the large financial investments needed to recover from such outages.

“When a storm goes off on the sun, we can’t really predict the extent of damage that it will cause here on Earth. It is critical that we develop this predictive capability,” said Dr. Homa Karimabadi, a space physicist at the University of California–San Diego (UCSD). Karimabadi’s team, in close collaboration with Dr. William Daughton at Los Alamos National Laboratory, is currently using the Oak Ridge Leadership Computing Facility’s (OLCF’s) Cray XT5 Jaguar supercomputer, one of the most powerful in the world with a peak performance of 2.33 petaflops, to better understand the processes giving rise to space weather.

Specifically, Karimabadi has used about 30 million hours on the Department of Energy’s premier high-performance computing system to study how plasma spewed from the sun interacts with Earth’s magnetosphere, investigating exactly what happens when the two meet and precisely what gets through and why.

“One of the surprising outcomes of our research is the ubiquity and nature of turbulence in the magnetosphere,” said Karimabadi. “This is important since turbulence implies more efficient mixing of the plasma and fields, and after all, space weather arises because the plasma and fields emanating from the sun can penetrate and mix with the plasma and fields of Earth’s magnetosphere.”

Initially, the team used the National Science Foundation-funded, University of Tennessee-managed Cray XT5 system known as Kraken to model the magnetic reconnection process that is the major cause of space weather, research that was published in the April 10, 2011, edition of Nature Physics. Their follow-up work using Jaguar has included simulations nearly three times as large as those on Kraken and motivated several observational studies to search for the newly discovered effects from the most recent simulations.

Extracting meaningful knowledge from these simulations poses its own challenges. A single run can generate over 200 terabytes of multi-dimensional, highly complex data. To meet this challenge Karimabadi has been working with Burlen Loring of Lawrence Berkeley National Laboratory on the development of specialized visualization techniques. “By color coding the magnetic field lines in our visualizations of the solar wind and the magnetosphere, we can track the level of mixing,” said Karimabadi.

The complex entanglement and mixing of the field lines results from the formation of a magnetic flux rope generated by the magnetic reconnection process. (Courtesy: H. Karimabadi and B. Loring)

Thanks to the speed and power of systems such as Kraken and Jaguar, this complex mixing phenomenon is finally becoming understandable. While Karimabadi has been at this research for 15 years, only since the advent of petascale computing have the simulations begun coming sufficiently close to reality.

Earlier global modeling of Earth’s magnetopshere was based on magnetohydrodynamics simulations, which substituted capability for accuracy, neglecting critical effects in the reconnection process. Essentially the plasma was treated as a fluid, or a single body, an approach that missed critical physical processes but was necessary because of the limitations of yesteryear’s computational capabilities.

With petascale computers, though, it has now become possible to model magnetic reconnection, including kinetic effects—the individual particles that make up the ionized gas called plasma ejected by the sun. “If you want to model the plasma correctly,” said Karimabadi, “you have to model and resolve the orbits of the electrons and ions in a plasma.” Needless to say, the particles are numerous, hence the need for systems such as Jaguar.

“With petascale computing we can now perform 3D global particle simulations of the magnetosphere that treat the ions as particles, but the electrons are kept as a fluid,” said Karimabadi. “It is now possible to address these problems at a resolution that was well out of reach until recently.”

For example, in 2006 the team’s largest simulations contained around 1 billion individual particles. With Jaguar it is now modeling in the neighborhood of 3.2 trillion particles, an increase in resolution that is beginning to reveal the reconnection phenomenon in a whole new light.

Turns out there is a lot going on; the team is observing a great deal of electron- and ion-scale structures and turbulence. Essentially the team zooms in on a particular region of the magnetosphere for which they understand the existing conditions. By simulating the region, “you study the magnetic reconnection in its full glory retaining full electron effects,” said Karimabadi. “If you’re looking at Saturn with your naked eye, you would see it as a fuzzy ball, but with a powerful telescope, you could see its rings and much finer structures.”

Jaguar is one heck of an analogous telescope, and when coupled with Karimabadi’s use of two individual codes, H3D and VPIC, the sun’s stormy effect on our planet becomes clearer by the simulation. While the two applications are indeed different (H3D is a global code developed at UCSD that treats the ions as particles and the electrons as a fluid, while VPIC, developed at Los Alamos National Laboratory, defines both electrons and ions as particles), they are complementary. H3D gives the team a look at the big picture, while VPIC serves as the magnifying glass, revealing all the gritty, noisy details.

3D global hybrid simulation of Earth’s magnetosphere. Magnetic field lines are color coded based on their origin and termination point .

“Our codes are very efficient at using all the available computational power of Jaguar,” said Karimabadi, adding that as soon as the fastest hardware comes out, his software is ready to take full advantage. The current simulations on Jaguar use more than 200,000 of the machine’s 224,000-plus compute cores, no small feat for even the most aggressive applications. Despite Jaguar’s raw computing power, however, there is still plenty of room for improvement.

“The problem in itself is exascale,” said Karimabadi, noting that even with today’s fastest computers, researchers are still unable to model the magnetosphere in 3D treating ions and electrons as particles, a phenomenon that might be tough to sufficiently simulate even when high-performance computing arrives at its next milestone—the exascale.

And while the exascale era isn’t exactly right around the corner, another great machine is. The OLCF is currently standing up Titan, an upgrade to Jaguar that will clock in at a peak speed of somewhere between 10 and 20 petaflops, meaning space physicists such as Karimabadi will soon have a chance to take advantage of even more computing muscle, a union that will certainly produce an unprecedented picture of space weather’s effect on Earth’s magnetosphere and, by extension, our lives. In the meantime we will continue to weather the sun’s storms—literally.

Related publication:

W. Daughton, V. Roytershteyn, H. Karimabadi, L. Yin, B. J. Albright, B. Bergen, & K. J. Bowers. 2011. Nature Physics 7, 539-542. doi:10.1038/nphys1965

(Paper reflects work performed on Kraken)