ESnet6 leverages AMD adaptive networking technology

AMD worked closely with ESnet since 2018 to integrate powerful adaptive computing for the smart and programmable network nodes of ESnet6. ESnet6’s extreme scale packet monitoring system uses AMD Alveo™ U280 FPGA-based network-attached accelerator cards at the core network switching nodes. This will enable high-touch packet processing and help improve the accuracy of network monitoring and management to enhance performance. The programmable hardware allows for new capabilities to be added for continuous innovation in the network.

In order to customize AMD Alveo U280 2x100Gb/s accelerators as network interface cards (NIC) for ESnet6, the OpenNIC overlay – developed by AMD – was used to provide standard network-interface and host-attachment hardware, allowing novel and experimental networking functions to be implemented easily on the Alveo card. OpenNIC has since been open-sourced after being successfully used and evolved by ESnet, as well as various leading academic research groups. Also important for rapid innovation by non-hardware experts was the use of the AMD VitisNetP4 development tools for compiling the P4 packet processing language to FPGA hardware.

AMD Alveo U280 cards, using OpenNIC and VitisNetP4, are being deployed on every node of the ESnet6 network. The high-touch approach based on FPGA-accelerated processing allows every packet in the ESnet6 network to be monitored at extremely high transfer rates to enable deep insights into the behavior of the network, as well as helping to rapidly detect and correct problems and hot spots in the network. The Alveo U280 card with OpenNIC platform also supplies the adaptability to allow the continuous roll-out of new capabilities to the end user community as needs evolve over the lifetime of ESnet6.

The benefits of AMD adaptive computing at demanding network line rates are also applied to the edges of the ESnet6 network. Globally distributed scientific instruments are the primary sources of the ‘big data” science flows that ESnet must support. The Alveo card with OpenNIC can be used for customized data filtering and shaping before transmission over ESnet. For supercomputing facilities, which are among the destinations for these large data sets, the Alveo card with OpenNIC can be used for load balancing across compute and storage servers. 

“AMD is honored to provide ESnet with leading-edge technology and expertise needed to help bring the new network into service,” said Ivo Bolsens, senior vice president and CTO of Adaptive and Embedded Computing Group, at AMD. “AMD FPGA-based adaptive networking capabilities make up a core building block of the ESnet6 extreme-scale packet processing system and provide the underpinning for the DOE’s future vision of distributed e-science.”

High-capacity, high-performance networking

Headquartered at Lawrence Berkeley National Laboratory (Berkeley Lab), ESnet serves as the “data circulatory system” for the DOE, connecting all of its national laboratories, and tens of thousands of DOE-funded researchers. This includes DOE’s premier scientific instruments and supercomputer centers – including the world’s first exascale system, the Frontier supercomputer powered by AMD EPYC™ CPUs and AMD Instinct™ Accelerators.

“ESnet6 is built to support the DOE’s multi-billion-dollar investment in scientific research that leads to breakthroughs that impact our everyday lives. It represents a transformational change in the way networks are built for scientific research,” said Inder Monga, executive director of ESnet. “AMD’s technology helps us enhance the resiliency, flexibility and performance of the network.”

With over 46 Terabits per second of bandwidth, ESnet6 features a significant increase in bandwidth over prior generations to support unique needs of scientific research. With this boost in capacity, along with new service capabilities, scientists can more quickly process, analyze, visualize, share, and store the mountains of research data produced by experiments, modeling and simulations. Data rapidly moves between the interconnected sites and collaborators, accelerating time to discovery.