NEFTSec: Networked federation testbed for cyber ‐ physical security of smart grid: Architecture, applications, and evaluation

As today's power grid is evolving into a densely interconnected cyber ‐ physical system (CPS), a high fidelity and multifaceted testbed environment is needed to perform cybersecurity experiments in a realistic grid environment. Traditional standalone CPS testbeds lack the ability to emulate complex cyber ‐ physical interdependencies between multiple smart grid domains in a real ‐ time environment. Therefore, there are ongoing research and development (R&D) efforts to develop an interconnected CPS testbed by sharing geographically dispersed testbed resources to perform distributed simulation while analysing simulation fidelity. This paper presents a networked federation testbed for cybersecurity evaluation of today's and emerging smart grid environments. Specifically, it presents two novel testbed architectures, including cyber federation and cyber ‐ physical federation, identifies R&D applications, and also describes testbed building blocks with experimental case studies. It also presents a novel co ‐ simulation interface algorithm to facilitate distributed simulation within cyber ‐ physical federation. The resources available at the PowerCyber CPS security testbed at Iowa State University (ISU) and the US Army Research Laboratory are utilised to develop this platform for performing multiple experimental case studies pertaining to wide ‐ area protection and control applications in power system. Finally, experimental results are presented to analyse the simulation fidelity and real ‐ time performance of the testbed federation.

Development of an interconnected networked federation testbed (NEFTSec) assists in performing sophisticated, high fidelity, and attack-defence module-based cybersecurity experiments in the smart grid.It is based on the notion of pooling different testbed resources across geographically dispersed research centres in a distributed manner through a common network like the internet or intranet and development of an integrated real-time testbed environment to emulate the real-grid infrastructure.The recent survey [2] on smart grid CPS testbeds presents a systematic study on existing testbeds across the globe.It also discusses how the interconnection of multiple testbeds using virtual private networks (VPN) can enhance individual testbed capabilities while supporting several research areas.Further, the National Institute for Standards and Technology [3] highlighted the need for modular and composable testbeds with remote federation functionality that are interoperable with remote facilities across the nation.

| Motivation
Developing a federated testbed is a non-trivial task as it requires a comprehensive understanding of grid network topology, key applications, and a strong commitment and consensus between federating participants to support multiple interfaces with a minimum of failure points.In [4], the conceptual architecture of sharing the phasor measurement unit (PMU) data using the industry-grade North American Syn-chroPhasor Initiative (NASPI) is presented, which is a collaborative effort between the U.S. Department of Energy and North American Electric Reliability Corporation, for sharing phasor data among utilities and industry vendors.However, the authors of [5] have discussed challenges related to fidelity, Quality of Services (QoS), and cyber-physical security of the interconnected networks of control centres and utilities in the NASPI architecture.Further, there is little advancement on developing implementable architectures, CPS benchmark models, and cybersecurity threat analyses to validate the conceptual NASPI architecture in grid networks.For the above-discussed reasons, it is imperative to develop a NEFTSec to address the evolving cybersecurity challenges in power systems.
In our previous work, we have shown the performance evaluation of machine learning-based anomaly detector for wide-area protection scheme (WAPS) using cyber federation testbed [6].In this work, we present several applications of federation testbeds, discuss different conceptual architectures, propose a co-simulation interface algorithm (CIA) for distributed simulation with a detailed analysis, and provide a quantitative performance evaluation of different federation architectures during real-time simulation.

| Contributions
The key contributions of this paper include: (1) It is one of the first efforts to discuss several applications of NEFTSec, explore its conceptual architectures, and present its experimental design and implementation in a cyber-physical testbed environment.(2) CIA is proposed to perform federation-based distributed simulation using geographically dispersed testbeds in real time.
(3) Experimental testing and evaluation are performed to analyse network packets (latency and bandwidth) and simulation fidelity during the distributed simulation.(4) Quantitative comparison of different federation architectures using simulation parameters during a real-time HIL simulation.
The rest of this paper is organised as follows: Section 2 presents an overview of relevant literature that describes different federated testbeds.Several applications, architecture, and design of NEFTSec are presented in Section 3. Section 4 presents the proposed CIA for the distributed cyber-physical federation.Section 5 and Section 6 present various case studies and their experimental evaluation in the real time.Finally, Section 7 provides the conclusions with a brief description of future works.

| RELATED WORKS
Existing challenges in the design and development of a federated testbed are extensively discussed in other research communities that focus on Internet of Things (IoT), networking, and cyber systems [7]; however, there exist few research efforts that encompass federation-related works in the context of smart grid cybersecurity.To develop a high fidelity, economical, and scalable testbed, researchers from several universities and research laboratories are working together to develop federated testbeds across hundreds or thousands of miles for specific applications in a power system domain.
Table 1 presents a condensed synthesis of key contributions related to the existing federated testbeds that are developed for specific applications in power systems.In [8], the National Renewable Energy Laboratory (NREL) and Pacific Northwest National Laboratory federated to develop a power hardware-in-the-loop-based co-simulation platform for testing multiple solar inverters (hardware) with two largescale distribution systems: the IEEE 123 node and 8500 node test feeders.This federation platform applied Java-Script Object Notation (JSON)-based data exchange protocol and User Datagram Protocol (UDP) protocol is used at the transport layer while ignoring missing and delayed packets during the distribution simulation at a timestep of 1 s.Further, the NREL and Idaho National Laboratory (INL) federated their testbeds to demonstrate their capabilities and computed a real-time latency of 28 milliseconds between their networks [9].The authors of [10] presented the DETER-WAMS-ExoGENI testbed by federating a realtime digital simulator (RTDS)-phasor measurement unit (PMU) testbed at the North Corolina State University with the Defence Technology Experimental Research (DETER) testbed at the University of Southern California (USC) to perform an impact analysis of cyberattacks on wide-area damping control (WADC).The PowerCyber testbed at Iowa State University (ISU) also performed cyber federation with the DETER testbed at the Information Science Institute of USC to emulate substation and control centre networks in power grids.In this study, the authors only performed physical-level impact analysis of cyberattacks in wide-area protection and control [11]; however, a networklevel analysis is also critical to support reliable communication and data exchange between federated testbeds.
The geographically dispersed real-time distributed simulation (GD-RTDS) is one of the key applications of networked federation that facilitates the integration of distant real-time simulators to simulate complex power systems by partitioning them into multiple subsystems without abstracting individual system components.One of the earliest developments of a virtual environment to support the GD-RTDS is reported in 2005 [12] to perform a remote model validation using a LabVIEW virtual instrument and virtual testbed interaction.In Ref. [13], the authors demonstrated the GD-RTDS platform by accurately performing thermo-electric co-simulation using two distant simulators.The authors of Ref. [14] presented a VILLASframework for performing distributed simulation of a transatlantic high-voltage direct current interconnection between the transmission systems of the United States and European grids.This platform is developed through the leading collaboration of INL and Rheinisch-Westfälische Technische Hochschule Aachen University and was joined later by other US national laboratories and institutions in United States, Germany, and Italy.
Since the decoupling point and the selection of the decoupling methods affect the stability and performance of a distribution simulation, a transmission line modelling (TLM)-based decoupling technique is proposed to perform GD-RTDS using the MATLAB-Simulink environment [15] and real-time simulators [16].In [17], the authors showed the application of real-time OPAL-RT simulators to perform the electromagnetic transient (EMT), three-phase transient stability (TS), and hybrid TS-EMT simulations for a small distributed power network.Further, the advancements in GD-RTDS have been presented by introducing interface algorithms for system decoupling, time delay compensation, and superior simulation fidelity while exchanging data over the internet [18,19].
Phasors-based CIA [19] is well-recognised to be the current state-of-the-art solution to facilitate the geographically distributed co-simulation as it exhibits a higher degree of simulation fidelity than other methods.The previously proposed CIAs [19][20][21][22] are mostly focussed on interfacing transmission and distribution systems that exhibit slow transients and less complexity as compared to performing co-simulation at multiple transmission systems.
A review of literature indicates that none of these efforts address the challenges of developing a robust platform and implementable architecture to support the federation-based cybersecurity experiments in the smart grid domain.Also, it is imperative to develop a CIA and analyse its performance during the GD-RTDS for transmission systems using the federation platform.The rest of this paper discusses several applications, implementable design of NEFTSec, and experimental evaluations for multiple case studies.

| Research applications
Federating existing smart grid CPS testbeds enhances collaboration and productive investments without revealing proprietary information between participants.It provides educational and research-integrated platforms for large-scale experimentation with diverse network topology and devices.In general, it can be applied for a comprehensive set of applications that are represented in Figure 1 and elaborated in greater details below.
(1) Scalability: Addresses this challenge by leveraging computational, hardware, and software resources from CPS testbeds.In particular, the scalability in power systems is resolved by performing distributed simulation and co-simulation using distant real-time simulators.Further, the scalability in cyber systems is accomplished by leveraging computational resources, such as server racks from other CPS testbeds, and modelling virtual sensors and actuators to emulate physical devices.(2) Grid Emulation (Fidelity): Emulates substation and control centre networks with industry-grade communication protocols while simulating complex power system dynamics in a HIL environment.This provides a high fidelity representation of geographically dispersed grid elements for industrial, academic, and broader public users.Figure 2 shows the conceptual architecture of NEFTSec that emulates different types of cyber-physical interactions between substation and control centre networks while simulating the power system.The power grid topology is represented through balanced authority areas (BA 1 …BA z ) that are interconnected through the transmission network.Specifically, it is classified into two broad categories: 1) Cyber (network)-based federation (CF) and 2) Cyber-physical-based federation (CPF).

| Cyber (network)-based federation
This architecture, as shown in Figure 2a, emulates the widearea communication between a group of substations and control centres using cyber and physical components of geographically dispersed CPS testbeds.In this architecture, the substation topology, including power system simulators, field network, sensors, actuators, etc., is emulated in one testbed (i.e.Testbed 1), while other testbeds (i.e.Testbed 2… Testbed P) are emulating local or regional control centres that receive grid measurements to support wide-area monitoring, protection, and control (WAMPAC) applications in real time.

| Cyber-physical-based federation
This architecture emulates geographically dispersed multiple substations and control centres (local and regional) and supports bi-directional data exchange between them.Based on the grid topology, it is classified into two broad categories: (1) Parallel CPF (PCPF) and (2) Distributed CPF (DCPF).
(1) Parallel CPF (PCPF): In this architecture, the distant power system simulators and emulators, synchronised with a Global Positioning System (GPS) clock signal, are running identical power system models with the same time step in parallel; however, grid measurements are collected from substation zones and are forwarded to control centres that are emulated in different CPS testbeds.For example, as shown in Figure 2b, Testbed 1 emulates substation 1 to substation m − 1 as substation zone 1 and Testbed 2 emulates substation m to substation n as substation zone 2 while executing the same power system model in real time.This architecture is highly suitable for cyberattack analysis on wide-area measurements during a steady state.Note that the proper analysis of system models, network traffic, and synchronisation of control signals is required to accurately capture transient responses in multiple simulators during injected disturbances.(2) Distributed CPF (DCPF): This architecture facilitates distributed simulation in transmission systems and cosimulation of transmission and distribution systems by splitting a complex power system into multiple subsystems and executing the decoupled subsystems in distant simulators that are synchronised to a GPS clock signal.Figure 2c presents the DCPF in a transmission system where Testbed one executes decoupled subsystem 1 that includes BA 1 to BA k and Testbed two executes decoupled subsystem 2 that includes BA k+1 to BA z .System variables are exchanged at communication points using interface models (i.e.interface models 1 and 2).Further, it emulates geographically dispersed substations and control centres, similar to the PCPF.Although it exhibits high computational efficiency, there exist particular challenges in terms of model partitioning and synchronisation, time delay, and data exchange rates that may affect the stability and accuracy of the DCPF during the real-time simulation.

| Federated testbed design
Figure 3 presents a high-level design architecture of NEFTSec for implementing the aforementioned conceptual architectures and performing cybersecurity-related experiments.We have explored the NASPI network (NASPInet) architecture [4] to develop this industry-grade infrastructure to support the bidirectional sharing of data in secure, reliable, and robust ways.Cyberattack Module: It includes a library of software components that are grouped together as a module to implement cyberattacks in a federated testbed environment.This module includes IT-based attacks (ping, nmap scanning, and OpenVAS scanning), single cyberattacks (line tripping attack and denial of service attack), stealthy cyberattacks (Man-in-the-Middle (MITM) attack and malware installation), etc., and coordinated cyberatttacks as discussed in previous works [6,23,24].
Detection (Defence) Module: It includes state-of-the-art detection and defence solutions, such as rule, model, and machine learning-based anomaly detection systems, moving target defence, etc., as developed in earlier research efforts [23,24], which can be utilised to test, validate, and evaluate system performances with a detailed network analysis using this NEFTSec platform.

| Proposed co-simulation interface algorithm
As a time-domain signal is not appropriate for sending instantaneous values through the packet-based communication [16] in DCPF, the CIA is proposed that couples the decoupled subsystems, simulated in the electromagnetic transient (EMT) domain at distant locations.Figure 5 shows the proposed CIA that facilitates the bi-directional communication between two decoupled subsystems, subsystem X and subsystem Y while assuming a balanced three-phase system.It consists of several steps that are discussed in greater details here.The overall proposed CIA is summarised in Algorithm 1.
Step 1 (Apply TLM): This decoupling technique transforms the given system into its Norton equivalent form and voltage sources are connected at the boundary buses, which interchange their respective values at each time step to determine the parameters of the companion model.The voltages at subsystem X, Vxs abc (t k ), and subsystem Y, Vys abc (t k ), at a time step, t k , are computed using Equations ( 1) and ( 2) for all three phase sources.The impedance, Z l , for an inductor is L/T, where T is a simulation time step and L is an inductance.Note that this TLM technique is selected because of its high efficiency and accuracy of decoupling the given system into several subsystems [25].
Step 2 (Estimate phasor values): Convert the time domain signal, Vxs abc (t k ), coming from the subsystem X, into a phasor domain by decomposing the instantaneous three-phase voltage signals (Vxs a , Vxs b , and Vxs c ) into the dq0 frame by computing direct (Vxs d ), quadrature (Vxs q ), and zero components (Vxs 0 ) using Park's transformation, by substituting (3) into (4), and later computing the fundamental frequency ( f o ) and angle (ω o t k ) of three-phase voltages using the synchronous reference-frame phase-locked loop [26].Finally, the decomposed components are converted into a phasor domain, (Vpxs (t k ) = [Vmx, θ ix , T gps ]), by computing voltage magnitude (Vmx) and phase angle (θ ix ) with a time stamp (T gps ) of the fundamental frequency of a given signal, Vxs abc (t k ), over a running window of one cycle based on the Fourier analysis of a periodic voltage signal, as shown in Equations ( 5)- (8).The magnitude and phase angle are computed as V mx ¼ ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi ffi a 2 þ b 2 p and θ ix = arctan(b/a).Same sequence of steps is followed to compute Vpys(t k ) from Vys abc (t k ). -20 Algorithm 1 Proposed CIA for GD RTDS Step 3 (Compute down-sampling rate (T c )): This step computes T c by performing a closed-loop testing between two distant simulators as needed for reliable data exchange over the VPN tunnel, and later re-scaling (up-sampling) to the original simulation time step (T s ) at the receiving end to facilitate the EMT simulation.
Step 4 (Perform time-delay compensation and signal reconstruction): This step includes voltage signals reconstruction by computing Vxd abc (t k ) at the subsystem Y and Vyd abc (t k ) at the subsystem X.For computing Vxd abc (t k ), the time delay is compensated by adding a phase shift [27], (2πf o Δt k ), as shown in Equation (9).In this equation, Δt k is the computed time difference at the simulation time t k between the current GPS time stamp of the received signal and the sent one.Note that the GPS time clock also provides a standard time reference-based synchronisation to minimise model error due to out-of-step starting times between distant simulators.
Step 5 (Interchange phasor values): Continue interchanging system variables (Vpxs & Vpys) in the phasor domain at a lower sampling rate (T c ) between two distant simulators at communication end points.If the partitioned models converge, continue with the GD-RTDS, otherwise stop the simulation.

| Cyber federation case study
The cyber federation relies on an internet-based communication medium that is subject to various delays, such as network processing delay, transmission delay, etc., as well as packet loss.This platform is utilised to analyse the wide-area communication network during bi-directional data exchange between substation and control centre networks for the EMS-based WAMPAC applications in real time.

| Distributed cyber-physical federation case study
This platform is utilised to perform GD-RTDS on transmission systems in the EMT domain.In this context, we have developed a TLM-based CIA to maintain simulation fidelity during GD-RTDS.We have utilised this CPF platform for cybersecurity-based experimental testing, signal analysis, and validation of the proposed CIA in real time.

| Experimental setup
Figure 6 presents a HIL experimental setup to perform CFand DCPF-related cybersecurity experiments in the EMT domain.During the CF, the substation network is operating in the PowerCyber testbed, where the modified IEEE 39 bus system is running as a test case in the ARTEMiS-SSN (eMEGASIM) solver using the OPAL-RT simulator.The simulator is also mapped to physical relays with modelled transmission lines using the IEC-61850 (Generic Object Oriented System-wide Events) protocol.The virtual PMU drivers are also modelled to send phasor measurements to the ARL control centre to support synchrophasor-based WAMPAC During the DCPF-based experiment, a first-order system is decoupled at the transmission line partition into load and source subsystems.A voltage source with an impedance is connected in series to represent the missing subsystem.The partitioned load and source subsystems are running in the PowerCyber and ARL testbeds in the OPAL-RT real-time simulators at a smaller time step of 50 μs (T s ) that are located at a geographical distance of around 1025 miles.Further, these simulators are synchronised to a GPS time clock while exchanging bi-directional data over the internet-based IPSec VPN tunnel.For computing the RTT delay, ping scanning is performed by pinging the ARL control centre every 0.5 s. Figure 7a shows the ping latency distribution, where the maximum RTT is computed around 87 ms and the minimum around 35 ms.Further, the experimental analysis reveals that there is a 0% packet loss during the ping testing in real time as shown in the red bar of bar graph in Figure 7b.It ensures a reliable communication network between two separate testbeds.We have also computed RTT of synchrophasor packets while varying the sampling rate from 60 to 20 frames/sec as shown in Table 2, and the average RTT has increased from 16.5 to 37 ms.
Figure 8 shows the RTT of synchrophasor packets at a sampling rate of 60 frames/sec that has a minimum value of 6.15 ms and a maximum value of 25.9 ms.(b) Communication Bandwidth: We have computed the PMU bandwidth requirements for different sampling rates, varying from 60 frames/sec to 20 frames/sec, also shown in Table 2.For a sampling rate of 60 frames/sec, the average value of computed bandwidth is approximately 15,250 byte/sec with a minimum value of 15,000 byte/sec and a maximum value of 15,300 byte/sec for the modelled phasors in the IEEE 39 bus system.Note that although the fast PMU measurements provide better grid visibility, the communication bandwidth also increases with an increase in the sampling rate.(c) PMU Communication Delay: Since the communication between the ISU and ARL testbeds is a major latency factor, we have computed the communication latency for the incoming live-streaming PMU data.It is computed as the time delay from when the PMU measurements leave the local PDC of ISU substation and is received at the super PDC of ARL control centre.Figure 9 shows the variation in a data delay for different numbers of PMU packets.We have observed that the average value of computed delay is 16.6 with a minimum value of Inter-tie (PDCI) in the North American Western Interconnection [30,31].Table 3 shows a list of different WAMPAC applications with their operational time (sec) that varies from 150 ms (WAVC) to 300 sec (Real-Time Contingency Analysis).Hence, this CF platform can be utilised to validate these applications in real time.Real-time contingency analysis (RTCA) [32] 60-300 Abbreviations: RTCA, Real-Time Contingency Analysis; WADC, wide-area damping control; WAPAC, wide-area protection and control; WAPS, wide-area protection scheme; WAVC, wide-area voltage controller.

| Experimental evaluation during DCPF
We have applied the proposed CIA by initially performing a closed-loop test to compute T c , analyse the communication latency, and the nature of the incoming signal.Further, we have analysed the simulation fidelity of the GD-RTDS through the model validation.

| Close-loop testing
After the detailed investigation and rigorous testing, closedloop testing is successfully performed using real-time simulators at the PowerCyber and ARL testbeds by sending a continuous sine wave of frequency 10/2π Hz from the ISU testbed's simulator to the ARL testbed's simulator using the ethernet-based communication interface with a Transmission Control Protocol/Internet Protocol end point connection at a time step of 30 ms. Figure 10 shows the comparison between generated continuous and returned discrete sine waves between two distant simulators with an average RTT latency of 90.6 ms, a maximum of 180 ms, and a minimum value of 60 ms with a zero packet loss.

| Simulation fidelity during DCPF
Previous experimental results reveal a reliable data exchange rate (T c = 30 ms) for operating client-server communication interfaces during the GD-RTDS using a first-order two-bus system.To assess the simulation fidelity while applying the proposed CIA, we have performed the model validation by considering three cases as original, decoupled, and federated (GD-RTDS) systems.We have first performed voltage sag, as a physical disturbance, by reducing the magnitude of source voltate (V) from 120 to 108 V. Second, a load tripping-based cyberattack is performed by maliciously tripping a relay directly connected to load 1. -207 Table 4 presents a quantitative performance measures in terms of mean absolute error (MAE), mean absolute percentage error (MAPE), and maximum absolute error (MxAE) of the voltage magnitude during 3 s of dynamic simulation.In this case, we observe a higher voltage magnitude error in the federated system compared to the decoupled system in both server and client sides.The computed signal errors are high at the point of disturbance during the transient simulation as highlighted through the zoomed portions in Figure 11a,b and ovals in Figure 11c,d with the MAE of 0.6792, MAPE of 0.8587%, and MxAE of 5.327 from the client side, and the MAE of 0.3949, MAPE of 0.4505%, and MxAE of 2.2977 from the server side in federated system.
Figure 12 presents voltage plots of all three cases during a load tripping attack from the ISU client and ARL server sides.We also observe a similar trend in this scenario where the federated system is relatively exhibiting a high signal error compared to the decoupled system.In this case, we observe an MAE of 0.3136, MAPE of 0.3434%, and MxAE of 6.787 from the ISU client side, and the MAE of 0.1227, MAPE of 0.1263%, and MxAE of 3.7364 from the ARL server side with respect to the original system (Table 5).Our experimental analysis reveals that the computed voltage error is dissimilar for different scenarios as the computed voltage error is higher during the voltage sag compared to the load tripping attack for decoupled and federated systems.Note that the proposed federated system shows negligible error during the steady state that is illustrated in the reconstructed voltage signals in Figure 11a,b and Figure 12a,b.It clearly illustrates that the proposed CIA algorithm ensures simulation fidelity by compensating time delays as the reconstructed phase A voltage compares closely with decoupled and original systems during the real-time simulation for the steady state.In both cases, the computed signal error gets amplified during transient states due to injected disturbances before restoring back to the normal steady state.
We have also computed the norm-2 error of phasor magnitudes as a simulation fidelity metric using (10) over a moving window of 10 samples to adequately characterise the error in federated and decoupled systems from client and server sides during the load tripping attack.In this equation, v is the phasor voltage magnitude of decoupled/federated systems and v r represents the phasor voltage magnitude of original systems.
Figure 13 indicates that the 2-norm error of the federated system from the ISU client side is relatively large with the computed value of 0.077 compared to the ARL server side that shows the value of 0.0529 at the point of disturbance.We also observe a higher error of margin between the federated and decoupled systems at the ISU client side as compared to the ARL server side during disturbances.In both cases, the 2-norm errors eventually reduce to the initial minimum value after disturbances are removed.

| Real-time simulation analysis
We have also performed the quantitative comparison of CF, PCPF, and DCPF architectures using simulation parameters, as shown in Table 6, during the real-time execution of the firstorder two-bus system for 500 s.In this case, we observe 228 instances of overrunning during the CF, 13 instances of overrunning in the PCPF, and 11 instances of overrunning in the DCPF, where the given system did not simulate in the specified time step, that is, 50 μs.Higher overrunning during CF and PCPF simulations reveals higher computational power requirements compared to the DCPF simulation.Also, the computation usage during the simulation is low in DCPF compared to other federation architectures.We observe a similar trend in other performance parameters, including average major computation, minor computation, and average execution cycle where their values are high in CF and low in DCPF.Average computation time represents the average time taken by the model to perform block calculations that include discrete and continuous states' calculation and algebraic calculation for a major time step (average major computation) and minor time step (average minor computation).Average execution time includes the average time taken to compute all model tasks, such as major and minor computation, and simulator overhead and services.Figure 14 shows the computed parameters (execution cycle (a) and major and minor computation times (b and c)) with respect to the number of samples where the DCPF shows the consistent performance for all parameters.Higher values of these parameters during the CF and PCPF put restrictions on the system size and indicate that a larger number of cores is required to perform real-time simulation compared to the DCPF while supporting HIL testing and communication data exchange.

| CONCLUSION AND FUTURE WORKS
In this paper, we presented the conceptual architecture, testbed design, and several applications of the networked federation testbed in the context of smart grid cybersecurity.We described the implementation of proposed federation architectures by integrating Iowa State University's PowerCyber Laboratory and the US ARL to conduct CF and DCPF experiments for different case studies.Further, we introduced the CIA to ensure simulation fidelity while performing the DCPF experiments by applying the TLM-based decoupling technique, estimating phasor values, and interchanging system variables at a down-sampled rate.For detailed case studies, we utilised the IEEE 39 bus system and performed a network packet analysis to validate the application of CF platform to support WAPS-related cybersecurity experiments.During the DCPF, we decoupled a first-order two-bus system to perform GD-RTDS and analysed the simulation fidelity using several performance measures while validating decoupled models during the load tripping attack.We observed that the proposed CIA efficiently preserves the simulation fidelity during the steady state; however, the model error gets amplified during the transient state that varies at client and server sides.We also performed the quantitative comparison of different federation architectures during the real-time simulation and observed that the DCPF exhibits lower overrunning and computation usage as compared to CF and PCPF.For future works, we plan to explore the efficacy and feasibility of federated testbeds for a broader case of wide-area monitoring, protection, and control applications of the smart grid.Further, this research opens up several avenues for further research, which include:

( 3 )
Remote Access: Provides remote access capability to multiple users using internet-based network interfaces to conduct multiple and coordinated cybersecurity-based attack-defence experiments [20].(4) Testing, Validation, and Evaluation: Since the current operational systems cannot be utilised to perform cyber-physical security-related experiments, the NEFT-Sec provides a platform to test, validate, and evaluate the performances of state-of-the-art research and engineering solutions in real time.(5) Demonstration and Training: Provides a unique demonstration platform by simulating realistic cyberattacks and possible defence measures [23, 24].In addition, it serves as a powerful training ground to provide hands-on cybersecurity training and hacking exercises in grid security.(6) Quality of Service (QoS): Supports QoS assessments by analysing network packets for data latency, packet loss, and communication bandwidth for a reliable communication network between CPS testbeds.(7) Heterogeneity and Modularity: Incorporates heterogeneity by simulating different components of power systems utilising phasor and transient simulations as well as performing hybrid simulations.It also supports modularity by integrating several modules from different CPS testbeds, including physical and cyber system modules, communication modules, and cybersecurity attackdefence modules.(8) Concurrency and Repeatability: Supports concurrency by performing multiple CPS experiments in a parallel or distributed manner.Further, it supports repeatability by allowing federating participants to perform the same experiment multiple times.(9) Data and Models Sharing: Since the availability of real-world data capturing cyber-physical properties is limited and sensitive to the utility's operation, the NEFTSec assists in sharing data and models to multiple participants simultaneously in a distributed manner to facilitate multiple CPS experiments and use cases without affecting real-world operations.(10) Interoperability: Supports interoperability by providing a real-time platform for communication protocol testing, attack-defence analysis, and experimental evaluation of interconnected testbed network interfaces.
Figure3presents a high-level design architecture of NEFTSec for implementing the aforementioned conceptual architectures and performing cybersecurity-related experiments.We have explored the NASPI network (NASPInet) architecture[4] to develop this industry-grade infrastructure to support the bidirectional sharing of data in secure, reliable, and robust ways.Figure4presents the HIL federated testbed that facilitates the supervisory control and data acquisition (SCADA)

F I G U R E 5
Proposed co-simulation interface algorithm (CIA) for geographically dispersed real-time distributed simulation in transmission systems SINGH ET AL.

6. 1 |
Network packet analysis during CFSince data latency and packet loss are two key challenges for performing CF-level experiments due to the internet-based communication medium, we have performed the network packet analysis in terms of network latency, packets drop, and communication bandwidth.

( a )
Communication Latency: We have computed the widearea communication latency in terms of round trip time (RTT) and communication delays.

F I G U R E 6
Experimental setup using Iowa State University (ISU) PC and Army Research Laboratory (ARL) cybersecurity testbeds SINGH ET AL. -205 23983396, 2022, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cps2.12033,Wiley Online Library on [14/12/2022].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License1.9ms and a maximum value of 26.7 ms.Note that the computed maximum PMU delay complies with the designated communication latency (38 ms) of WAPS in the Southern California Edison [28], 60-100 ms latency of wide-area voltage controller (WAVC) in the Bonneville Power Administration [29], and 100-200 ms latency of wide-area damping controller (WADC) in the Pacific DC

F I G U R E 7
Network packet analysis during ping scanning T A B L E 2 Computed round trip time (RTT) and bandwidth of synchrophasor network packets RTT (ms)

F I G U R E 8 F I G U R E 1 1 F I G U R E 1 0
Round trip time (RTT) for synchrophasors at 60 frames/sec F I G U R E 9 Latency for phasor measurement unit (PMU) packets delay during live-streaming 206 -SINGH ET AL.Reconstructed voltage signals (a and b) and phasor magnitude values (c and d) during the voltage sag Generated and returned sinewaves at the Iowa State University (ISU) testbed's OPAL-RT simulator T A B L E 3 Operating time of different wide-area monitoring, protection, and control (WAMPAC) applications

Figure 11
presents the reconstructed voltage signals (a, b) and phasor voltage magnitudes (c, d) of all three cases during the voltage sag from the ISU client and ARL server sides.SINGH ET AL.

( 1 )
Development of CIA for the unbalanced three-phase system and performing the detailed analysis while also considering high-frequency components in the voltage profile.(2)Design a real-time predictor model to improve model accuracy during the GD-RTDS at DCPF level and develop NASPI-inspired network interfaces to federate multiple CPS testbeds.
23983396, 2022, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cps2.12033,Wiley Online Library on [14/12/2022].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License system topology and to model virtual PMUs and relays that interact with software-based PDCs and RTUs [23].Control Centre Network: The PowerCyber control centre network at ISU includes Siemens and General Electric energy management system (EMS)/SCADA systems (Siemens Power, GE e-terraplatform) with WAMPAC applications, data servers (open-PDC, Open Platform Communications and MySQL servers), and a real-time visualisation dashboard.Similar sets of open-source software are operating at the ARL control centre network to support grid communication and EMS-based WAMPAC applications.
SINGH ET AL. 23983396, 2022, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cps2.12033,Wiley Online Library on [14/12/2022].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License 204 - Model validation error during the voltage sag Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cps2.12033,Wiley Online Library on [14/12/2022].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License T A B L E 4 E 1 3 2-norm error of phasor magnitudes during the load tripping attack T A B L E 6 Quantitative comparison of different federation architectures Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cps2.12033,Wiley Online Library on [14/12/2022].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License