From Streams to Rivers: NREL Upgrades Data Exchange for High-Use, Real-Time Applications
If research data at the Energy Systems Integration Facility (ESIF) were visible, visitors would be overwhelmed by a turbulence of crisscrossing data streams rushing in and out of laboratories, the Insight Center’s visualization room, and the Eagle supercomputer. A recent upgrade by the National Renewable Energy Laboratory’s (NREL’s) Data Analysis and Visualization Group (DAV) has brought some order to that data flow with a platform that makes data streaming and access much easier for ESIF researchers and partners.
“We’re creating a streaming-first mentality in the labs,” said Matthew Eash, the NREL architect of the data streaming capability. “This new platform is focused on availability—the idea is to get the data out in consolidated flows and with a common methodology.”
For every project or partner at the ESIF there is significant data exchange. Researchers generate data with laboratory devices (e.g., solar inverters, electrolyzers); they analyze data alongside simulated systems; they use data in visualizations; and they share those data with partners. The upgrade by NREL’s team transforms this complicated connectivity into a standardized river of data, where users can simply tap in as consumers or send their data as publishers.
“This platform is a middleman for the diverse experiments and capabilities we have at NREL,” Eash said. “From the same data flow, researchers could hook up visualizations, or run machine learning methods, or interconnect hardware across the laboratory, or blend in partner data.”
Such versatility can be attributed to Apache Kafka, an open-source streaming platform optimized for high-throughput, low-latency feeds. Eash and team installed Kafka on a six-node cluster in the ESIF data center, which interfaces with the ESIF network, including Eagle, as well as private laboratory subnets and the entire Flatirons and South Table Mountain campuses of NREL.
An important outcome of this data streaming capability is its use in hardware-in-the-loop experiments. In one example, an NREL project is leveraging the capability to connect building technologies across facilities. Some assets are in the ESIF—such as smart thermostats and energy storage equipment—and others are across the campus, where HVAC equipment is located. Data is now easily shared between the facilities, supporting NREL’s Commercial Buildings Research Infrastructure by connecting the inputs and outputs from simulation and hardware at different facilities.
Another example involves data from an ESIF laboratory where computer simulations communicate back and forth with diverse energy devices in real time. The Kafka upgrade benefits the data-centric, AI-driven projects within the laboratory.
“The DAV streaming platform has made a very complex communication problem simple for us,” said Bethany Sparn, a senior research engineer at NREL. “We are working on a project where we need to pass data back and forth between physical equipment in our lab and simulations running on Eagle. When we did something similar a few years ago, we had to design a solution from scratch. The new data streaming platform in the ESIF is exactly what we need, with a great team that provides excellent support.”
At the ESIF, the Kafka upgrade will vastly improve the speed and simplicity of how researchers interact with data. The ESIF Research Engineering team at NREL has helped integrate the Kafka platform with the ESIF Research Data Acquisition network to ensure ESIF capabilities are incorporated, like utility SCADA infrastructure and hydrogen electrolyzer test beds. As the capability continues to be built out, research at the ESIF will benefit from greater data connectivity between resources and end uses.
In place of turbulent data streams, the DAV group has engineered reliable data rivers to supply the most cutting-edge capabilities across NREL.