| Internet-Draft | HP-WAN STATE OF ART | November 2026 |
| King, et al. | Expires 6 May 2027 | [Page] |
High Performance Wide Area Networks (HP-WANs) provide infrastructure for the modern global Research and Education (R&E) community, facilitating collaboration across national and international boundaries. These networks include global education and research networks, such as GÉANT, Internet2, Janet, ESnet, CANARIE, CERNET, and others, and also refer to large-scale commercial dedicated networks built by hyperscalers and operators. They are designed to support the transfer of data generated by scientific research, high-performance computing, distributed AI-training, and large-scale simulations.¶
This document provides an overview of the terminology and techniques used for existing HP-WANs. It also explores the technological advancements, operational tools, and future directions for HP-WANs, including their use in scientific research, AI training, and large-scale R&E data analysis.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 6 May 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
High Performance Wide Area Networks (HP-WANs) are the backbone of global Research and Education (R&E) infrastructure. They support large data transfers and scientific collaborations worldwide. These networks are designed to meet the demanding requirements of data-intensive research fields, including high-energy physics, climate modelling, genomics, and artificial intelligence.¶
HP-WANs have evolved with the growth in scientific datasets and international research collaboration. General-purpose WAN services did not always meet the capacity requirements of research institutions. As datasets grew from terabytes to petabytes, these institutions needed networks that could carry the data between sites.¶
Specialised (inter)National R&E Networks (NRENs), such as ESnet in the United States, GÉANT in Europe, Janet in the UK, and CERNET in China, have evolved to support the unique needs of the scientific community, while also carrying more generalised and less demanding research and education traffic. These networks are designed by their operators to provide high bandwidth, low latency, reliability, and security for applications like real-time data analysis, distributed computing, and remote instrumentation.¶
This evolution has been made possible by the design decisions made by NREN operators and campus operators who build the localised network and systems infrastructures. HP-WANs require an end-to-end engineering approach: NREN operators provide capacity, while local campus operators engineer their network architectures and data transfer and compute systems to use that capacity effectively, e.g., by adopting the Science DMZ approach described in this document.¶
Today, HP-WANs are foundational to the research community and are leading the way in demonstrating how advanced networking technologies can be applied to other sectors. They serve as testbeds for networking innovations that eventually trickle down to broader commercial applications. Emerging technologies such as quantum computing and the Internet of Agents (IoA) may place further demands on these networks.¶
This document explores the current state of the art in HP-WANs, examining the technological advancements, operational challenges, and emerging trends shaping the future of networks built for research, education, massive data analysis, and collaborative AI training at scale and speed.¶
High Performance Wide Area Networks (HP-WANs) evolved as specialised networks initially designed to facilitate scientific research requiring high-speed data transfer, high reliability, and minimal latency. ESnet, Janet, GÉANT, and CERNET emerged in response to the increasing data volumes generated by scientific and educational institutions, extending the capabilities of traditional WANs.¶
HP-WAN is an end-to-end capability, architected through collaboration among NREN operators, campus IT operators, and research communities. These networks have become integral to research and education, supporting distributed scientific collaborations, large-scale simulations, and intensive data analysis. Their capabilities have been continually enhanced to meet rising demands, laying foundations for future networking technologies.¶
This section provides terminology related to high-performance WANs.¶
HP-WAN development has become synonymous with large-scale research and experimentation, big data, and AI. HPC, and therefore HP-WAN, is driven by the continuous need to move large data between HPC facilities, facilitating the following industries:¶
The data rates required by HPC applications vary significantly based on the application type and data scale.¶
Scientific simulations, such as climate modelling and molecular dynamics, typically demand data rates from 10 Gbps to over 100 Gbps due to the large volumes of data processed and moved between nodes and storage systems.¶
In high-energy physics, such as experiments at CERN, data rates can reach hundreds of gigabits per second, with aggregate peaks between sites currently exceeding 1 Tbps and predicted to rise to 10 Tbps during intensive data processing.¶
Healthcare, Genomics, and Life Sciences might typically operate at rates between 1 Gbps and 40 Gbps. These applications require high throughput to handle large datasets efficiently, often through parallel data streams.¶
AI learning and tasks, particularly those involving deep learning, require data rates from 10 Gbps to 100 Gbps to ensure efficient data movement and keep GPUs and other accelerators fully utilised.¶
These varying data rates underscore the high demands of HPC applications, which are expected to grow as the field evolves and datasets become larger, along with the growing need to move large datasets between sites (including data centres).¶
HP-WANs are specialised wide area networks designed to connect supercomputers, scientific instruments, data repositories, and other high-performance computing resources across sites. They enable collaboration on computational tasks that require significant processing power, memory, and data storage, and they support large-scale scientific research, complex simulations, and data-intensive tasks that exceed the capabilities of standard enterprise networks.¶
The following sub-sections outline typical characteristics and requirements for HP-WANs. These technical requirements ensure wide-area interconnects can meet the demands of distributed HPC environments, enabling researchers and scientists to collaborate effectively worldwide.¶
Not all HP-WAN deployments rely on backbone controllers. In many NRENs, capacity is provisioned ahead of demand on private dedicated infrastructure, with no QoS or bandwidth-on-demand systems in the core.¶
Some HP-WAN providers may choose, or need, to use specific resource and/or network controllers to deliver that functionality.¶
Resource Controllers monitor and configure individual network resources, such as routers and switches, to manage their utilisation and performance.¶
Network Controllers maintain global visibility of network topology, resource availability, and status, essential for path computation, resource reservation, and dynamic reconfiguration to meet stringent performance demands.¶
End-to-End Orchestration translates user and application requirements into network operations for automated, policy-driven management.¶
HPC networks can be broadly categorised into intra-site networks, which connect components within a single HPC site, such as a data centre, and inter-site networks, which link multiple HPC sites across different geographical locations. The intra-site (HP-WAN) networks typically use high-speed, low-latency non-Internet interconnects like InfiniBand or high-speed Ethernet. In contrast, inter-site networks rely on dedicated, high-capacity wide area networks (WANs) to support distributed computing and data sharing at regional and global scales.¶
Each NREN operator, e.g., Jisc in the case of Janet in the UK, will build and operate the NREN infrastructure for its research and education users. This may typically take the form of a well-provisioned backbone, with regional access networks extending to the end sites (campuses, research organisations, etc). The NREN demarcation is typically at the campus edge. In some countries the regional networks are operated separately.¶
The NRENs then typically have interconnects to other NRENs, forming a worldwide RE network infrastructure. In Europe, GÉANT provides connectivity between the European NRENs and then wider connectivity to the rest of the world. NRENs will also have other interconnects to non-RE networks, e.g., via one or more national IXs, direct peerings to content providers (including the big cloud providers) and then "catch-all" commodity connectivity via one or more Tier 1 ISPs.¶
Dedicated infrastructure is commonly used in HPC environments where performance, security, and reliability are paramount. In these cases, the network infrastructure is built exclusively for HPC applications, including dedicated fibre-optic connections, private data centres, and specialised network transport like RDMA over Converged Ethernet (RoCE) and InfiniBand nodes. The primary benefits of dedicated infrastructure include optimised performance for HPC tasks, stronger security by preventing unauthorised access, and consistent reliability by avoiding congestion or performance issues caused by other network traffic.¶
Usually, the responsibility for networking within an end site or campus lies with that organisation (e.g., a university IT department), while an HPC facility may have dedicated (separate) staff to operate it. With the additional administrative domains of the NRENs and inter-NREN backbones like GÉANT, end-to-end traffic may pass through many networks operated by different organisations. To achieve optimal end-to-end performance, everyone on the end-to-end path needs to implement best practices.¶
The technical requirements for wide area interconnects between HPC sites are stringent, given the unique demands of distributed high-performance computing. High bandwidth is a primary requirement, as these interconnects must support the rapid transfer of large datasets between sites without data movement becoming a bottleneck in computational workflows. HPC data flows might typically consume 1 Gbit/s to beyond 400 Gbit/s.¶
Low latency is needed for applications that require real-time or near-real-time data processing. Latency requirements for inter-DC locations will be in the low-millisecond range.¶
Network-intensive applications like networked storage or cluster computing need a network infrastructure with high bandwidth and low latency.¶
These interconnects may need to support specialised communication protocols designed for HPC environments, such as Remote Direct Memory Access (RDMA) [RFC5040] and [RFC7306], which optimises the performance of distributed HPC applications by reducing overhead and improving data transfer efficiency.¶
InfiniBand (IB) is another computer networking communications standard used in high-performance computing that features very high throughput and very low latency. InfiniBand is also used as a direct or switched interconnect between servers and storage systems, as well as between storage systems.¶
The advantages of RDMA and IB over other network application programming interfaces include lower latency, lower CPU load, and higher effective bandwidth. The downside of these specialised protocols is that all interfaces and nodes must support the technique along the end-to-end path.¶
iWARP is a computer networking protocol that implements remote direct memory access (RDMA) for efficient data transfer over Internet Protocol networks. Several IETF techniques are used for iWARP:¶
The scaling of HPC applications, especially across a WAN between multiple sites, requires the ability to route massive traffic volumes. Specifically, this requires network infrastructure to provide several routing and forwarding characteristics, which are detailed below.¶
Efficiently handling these elephant flows is important in HPC as they can otherwise saturate network links, leading to congestion and reduced performance for other network traffic. Strategies to manage elephant flows effectively, such as prioritising these flows or segmenting network traffic, help maintain overall network performance and prevent large data transfers from hindering the execution of other critical tasks within the HPC environment.¶
HPC transport options include IP (UDP and TCP) and emerging mechanisms such as QUIC. However, each transport technology provides strengths and weaknesses. In all cases, the goal is to transfer large data sets at high throughput with low latency, jitter, and packet loss.¶
Packet loss and path failures can interrupt HPC data transfers or increase their completion time. Operators need to assess recovery behaviour as well as steady-state throughput, particularly for long-running jobs.¶
Where the transport uses retransmission, operators can monitor packet loss and retransmissions to identify impaired paths. Testing path failover under load helps establish how quickly transfers resume and whether the alternate path has sufficient capacity.¶
High availability and redundancy support continuous operation during long-running HPC tasks. Networks also need security measures, including encryption and secure access controls, to protect sensitive or classified data in transit.¶
Depending on provisioning and resource contention, the network may need to support Quality of Service (QoS) mechanisms to prioritise traffic so critical HPC tasks receive the necessary bandwidth and low-latency performance.¶
Applications may need a way to request specific bandwidth or latency guarantees for high-priority tasks. These guarantees are hard to provide over shared multi-domain infrastructure.¶
Differentiated Services (Diffserv) offers a flexible method to manage traffic prioritisation without the need for an explicit request-and-grant process. Diffserv marks packets with different priority levels, allowing the network to prioritise and protect capacity for critical tasks. This approach may be useful in HPC environments where dynamic traffic patterns require adaptive resource management.¶
Congestion control adjusts the offered load to the capacity available along a path. Operators need to distinguish congestion from other causes of packet loss and check whether sustained transfers build queues or leave capacity unused. Where several transfers share a bottleneck, throughput also depends on how they share that capacity.¶
Depending on the transport technology used in the HPC environment, several congestion control schemes may be used:¶
End-to-end performance measurement and monitoring across multi-domains and network infrastructures are important in HPC environments. They provide a way to diagnose and troubleshoot network performance issues that can affect data-intensive applications and distributed computing tasks common in HPC.¶
perfSONAR is a network measurement toolkit commonly used in this space. It is designed to provide federated coverage of network paths. It provides an interface for scheduling measurements, storing data, and generating visualisations.¶
Data transfer applications should log their transfers such that monitoring tools can assess their end-to-end, disk-to-disk performance. e.g., FTS does this for WLCG data transfers.¶
The network needs to accommodate additional sites or increased capacity as computational needs grow, without significant reconfiguration. Interoperability across the different hardware, software, and protocols used at HPC sites is also necessary.¶
The scale of HP-WANs makes energy consumption an operational consideration. Network component efficiency and power management affect both operating costs and the environmental impact of data transfers.¶
Operators can consider reducing power consumption during periods of lower utilisation, subject to performance and availability requirements. Energy use needs to be measured to assess the effect of such changes.¶
Some HP-WAN environments may also require service and resource scheduling to coordinate data transfer windows, reserved capacity, and compute availability across administrative domains. This is especially relevant when large transfers are planned in advance and must align with application workflows.¶
HPC and AI workload schedulers commonly know job timing, placement, data-movement, and completion requirements. Network reservation and orchestration platforms need a common way to consume the network-relevant subset of that information without adopting each scheduler's native data model. The emerging workload-to-network models described in Section 6.2 address this gap.¶
The following sub-sections highlight examples of HP-WANs and their technical specifications.¶
The GÉANT network is a pan-European data network dedicated to research and education, providing high-speed, high-capacity connectivity across Europe, between European NRENs and to other worldwide NRENs. It is an essential infrastructure for HPC applications, enabling collaboration and data sharing among research institutions, universities, and HPC centres across the continent and beyond.¶
The core of GÉANT operates at speeds of up to 600 Gbps, using Dense Wavelength Division Multiplexing (DWDM) technology. This provides connectivity suitable for HPC applications, particularly those involving large-scale simulations, scientific research, and real-time data processing. Reliability is provided by using multiple optical underlay paths for data to travel between GÉANT nodes. The alternate paths support continued operation following a path failure.¶
The GÉANT network integrates perfSONAR for real-time network performance monitoring and reporting of IP performance metrics [RFC6703], allowing HPC users to detect and troubleshoot potential issues that could impact data transfer and overall performance. These measurements help operators assess whether the network meets HPC application requirements.¶
GÉANT provides specialised services for specific HPC projects, such as the LHC Optical Private Network (LHCOPN) and LHC Open Network Environment (LHCONE), which are critical for supporting the data-intensive needs of the Large Hadron Collider (LHC) at CERN. These services offer dedicated, high-bandwidth connections that are optimised for the massive data flows generated by LHC experiments.¶
The GÉANT network connects over 50 million users across more than 10,000 institutions in 40 countries. This extensive reach supports a wide range of HPC applications by connecting geographically dispersed research facilities. Beyond Europe, GÉANT connects to other major research and education networks, including Internet2 in the United States and CANARIE in Canada, allowing for global HPC collaborations and data exchanges.¶
The Janet network is the UK NREN, operated by Jisc. First established in 1984, backbone links now run at up to 800Gbps, with a growing number of sites connected at 100Gbps, in some cases with multiple 100G links. A typical university site will have multiple 10G links.¶
Janet connects to other RE networks via a 400G resilient link to GÉANT. It has a presence in multiple IXes, predominantly LINX, connects/peers directly to many content and cloud providers, and has commodity connectivity via Tier1 ISPs. The total aggregate external capacity is around 4-5 Tbit/s.¶
Some private, dedicated links are used by Janet sites, e.g., the CERN to RAL (UK Tier 1 site) LHCOPN link, which is a dedicated 200G path. While LHCONE is a L3VPN that may be on its own circuit or as an overlay on general RE paths. Otherwise, Jisc seeks to provide sufficient capacity to its science communities ahead of demand, and works with them (via its network performance team) to provide advice and guidance on how sites can optimise their use of the Janet network, e.g., by following Science DMZ principles.¶
Google Effingo is a managed data copy service that transfers data between Google's clusters. It supports Google's internal services and moves data on behalf of Google Cloud customers. The service uses Google's private WAN for large-scale data movement.¶
Effingo separates copy request orchestration from data transfer. It adjusts transfer parallelism to the available resources and shares capacity between competing copies. Workloads include batch transfers, replication, and latency-sensitive copies.¶
Google's IETF 121 presentation describes operation across more than 100 data centre sites and a daily transfer volume of about one exabyte [EFFINGO].¶
The Energy Sciences Network (ESnet), established in 1986, serves scientific research within the United States and is operated by the U.S. Department of Energy (DOE). It connects national laboratories, supercomputing centres, universities, and research institutions for data-intensive applications and distributed HPC tasks.¶
ESnet uses a fibre-optic backbone and dynamic circuit provisioning services to provide high-capacity, low-latency connectivity. Data transfer rates range from tens to hundreds of gigabits per second for scientific workflows such as high-energy physics experiments, climate modelling, and genomic research.¶
The On-Demand Secure Circuits and Advance Reservation System (OSCARS) allows researchers to reserve guaranteed-bandwidth paths for their projects. SENSE provides intent-driven orchestration and automated network-resource management.¶
ESnet uses monitoring and diagnostic tools such as perfSONAR to measure end-to-end performance across institutional boundaries. Operators can use these measurements to identify performance bottlenecks.¶
ESnet interconnects with GÉANT, Janet, Internet2, and CANARIE for international research data exchange.¶
Internet2 is a networking consortium established in 1996 to serve the United States research and education community. Its infrastructure supports collaboration among educational institutions, government laboratories, and industry partners.¶
Its optical backbone supports multi-terabit speeds and low-latency connections. Internet2 also supports dynamic bandwidth allocation, network monitoring, and federated identity management.¶
Established in 1993, CANARIE connects universities, research centres, healthcare institutions, and government laboratories across Canada. Canada's national research and education network also has international interconnections with GÉANT, Internet2, and ESnet.¶
CANARIE operates a high-capacity fibre-optic backbone with dedicated high-speed links, automated resource allocation, and integrated identity and access management. Its Digital Accelerator for Innovation and Research (DAIR) service supports cloud-based research and development.¶
CERNET (China Education and Research Network) links universities, research institutions and HPC centres across China. It provides high-speed, secure connectivity for research data sharing and HPC applications, with connections to international research networks.¶
CERNET's backbone operates at up to 100 Gbps (with ongoing upgrades) using DWDM and SDN for HPC workloads such as scientific simulations, climate modelling and high-throughput data processing. Its redundant topology provides alternate paths to limit disruption following a failure.¶
CERNET offers customised services for major national projects, including dedicated high-bandwidth connections for the China Spallation Neutron Source (CSNS) and national supercomputing centres (e.g., Sunway TaihuLight). These support data-intensive workloads and data exchange.¶
CERNET covers 31 Chinese provinces and connects over 200 million users across more than 2,000 universities, institutes, and schools. International peers include GÉANT (Europe) and Internet2 (U.S.).¶
China Mobile Cloud Dedicated Network is a high-performance dedicated service for enterprises, governments, and industrial users. It has built a large-scale SRv6 policy-based cloud dedicated network, offering secure, low-latency connectivity between on-premises data centres, branches, and its public cloud, supporting applications such as real-time industrial control and large-scale data analytics.¶
It covers 31 provinces and over 300 cities, with nearly 50 Tbps of inter-provincial bandwidth and more than 800 SRv6 policy-enabled devices. Its backbone uses Segment Routing (SR); some backbone links support 400 Gbps OTN to handle high-bandwidth demands. The network uses a multi-path redundant architecture.¶
It integrates a management platform for real-time monitoring of latency, jitter, and bandwidth, and connects tens of thousands of enterprise users domestically to ease inter-office collaboration and cloud migration. Internationally, it links to global cloud providers and overseas data centres for secure cross-border data transmission.¶
Scientific research, distributed AI, and industrial data analytics place further demands on HP-WAN capacity and service scheduling. The following sections discuss approaches to coordinating applications and network resources.¶
While these capabilities may not yet be widely deployed on RE infrastructures, the NREN operator community is clearly interested in tracking their development and readiness for production deployment.¶
Resource controllers and network controllers can coordinate capacity reservations with application schedules. Sharing resource availability and timing requirements allows them to allocate bandwidth when an application needs it and release it when the transfer completes.¶
For example, a typical trend for HP-WANs is to satisfy the needs of distributed or decentralised AI, using so-called "scale-across" networks between multiple large data centres over long distances. The integration of network resources and applications may need to consider underlying resources, such as the optical transport layer [HIC-OTN], routing policies configured by network controllers, and service requests. This integration requires coordination among the optical resource management platform, the network controller, and the service orchestrator.¶
The analysis of current HP-WAN technologies and operational practices identified a standards gap between the platform-specific information held by HPC and AI workload schedulers and the network mechanisms used to realise an HP-WAN service. A set of complementary models has been proposed to provide a common information flow from scheduler and job metadata, through technology-independent service intent, to network tunnel realisation.¶
The HPC/AI Scheduler Job Metadata Model [I-D.xkk-rtgwg-hpc-scheduler-job-metadata] normalises the network-relevant job, workload, scheduler, tenant, timing, and task metadata available from heterogeneous workload managers and orchestration platforms.¶
The HPC/AI Service Intent Model [I-D.xkk-rtgwg-hpc-service-intent] uses that context to express endpoints, communication patterns, timing, performance, data-movement, policy, and admission requirements as a technology-independent desired network outcome.¶
The HPC/AI Service Intent Tunnel Realization Model [I-D.xkk-teas-hpc-tunnel-realization] correlates an admitted service intent with its network realisation state, including tunnels, paths, policies, resources, lifecycle, protection, admission outcome, and performance monitoring. Together, the models form the following information flow:¶
scheduler/job metadata -> HP-WAN service intent -> tunnel realization¶
Operational platforms such as ESnet's SENSE and OSCARS, and comparable systems operated by other NRENs, illustrate the scheduling, reservation, and orchestration environments in which common models could be used. Such platforms could consume, produce, or map to these models to exchange network-relevant workload information and correlate requested service outcomes with their realisation. This does not imply that any named platform has implemented these models or demonstrated interoperability with them.¶
Intent-based networking (IBN) and automation technologies are increasingly used to manage and orchestrate HP-WANs. IBN allows network administrators to define desired network states or outcomes, with automated systems translating these intents into actionable network configurations. Platforms such as ESnet's SENSE demonstrate intent-driven orchestration for resource allocation.¶
As the scale and complexity of HP-WAN deployments grow, efficient signalling mechanisms become increasingly critical, especially when running HP-WAN services over shared public infrastructure.¶
Applications may want to signal their desired bandwidth to the network, enabling more precise rate negotiation and collaborative congestion control to achieve a targeted completion time for the data transfer.¶
Therefore, efficient and scalable signalling approaches are vital for dynamic resource allocation in HP-WAN environments. Effective protocols must support rapid dissemination of resource states and swift propagation of requests between network components, minimising latency and overhead.¶
Signalling mechanisms need to be extensible across different technologies, with low overhead and timely delivery of requests and resource updates. They also need to handle failures reliably.¶
This document makes no requests for action by IANA.¶
HPC networks, particularly those connecting data centres, need to protect the integrity, confidentiality, and availability of sensitive data and computational resources. Security requirements depend on the data being processed, for example in defence and pharmaceutical research or climate science.¶
This document surveys existing networks and techniques and does not define a protocol or data model. It introduces no security considerations beyond those of the technologies it describes.¶
This document was partly motivated by discussions on the IETF hp-wan@ietf.org mailing list.¶
The authors thank Gorry Fairhurst and Zahed Sarker for their reviews and suggestions.¶
The following authors contributed significantly to this document:¶
Nicholas Race, Lancaster University, United Kingdom. Email: n.race@lancaster.ac.uk¶