Isode Strategy for HF, STANAG 5066 and Applications over HF
This whitepaper sets out Isode's strategy for HF Radio. It summarizes the importance of HF Radio, why HF has implications from Antenna to Application and how this is addressed by Isode's "Everything above the Modem" strategy and partner approach to providing a complete solution. The paper looks at Isode's product set for HF, with particular focus on innovative capabilities, research and Isode led standardization. As well as current Isode product functions, it gives a Road Map for future products and capabilities.
Isode whitepapers are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Why HF is Special
HF Radio has unique characteristics for Beyond Line Of Sight (BLOS) communications. HF Transmissions can bounce of the Ionosphere to provide very long distance communication. HF Ground Wave can also follow the earth's curvature to provide BLOS communication, which is particularly effective over water.
While Satcom is the dominant choice for BLOS communication, HF provides an important alternative. There are rarely other viable communication options.
Where HF is Needed
There are several situations where HF is needed:
- Satcom-denied. It is likely that satellite communication will be prevented, in a symmetric-warfare situation. HF provides an important back-up communication mechanism.
- Satcom communication not viable. There are locations where use of Satcom is not viable, including valleys, fjords, jungle, and polar regions.
- Satcom too expensive. There are situations where cost is the primary reason to choose HF.
- Satcom not practical. For example, the necessary equipment is too large.
Problems to Address in an HF Solution
HF Radio has desirable BLOS characteristics and a number of difficulties:
- Propagation is highly variable, and links cannot always be made.
- Achievable speeds are highly variable and can be very low.
- Latency is high due to overheads in setting up links and the need to make transmissions reasonably long.
Systems communicating over HF need to address this by:
- Making use of higher speeds (including Wideband HF when available) while being able to fall back to very slow speeds (down to 75bps) when needed.
- Adaptation to high latency which impacts all communication layers and applications.
- Prioritization, to ensure that the most urgent and important things get through when capacity is constrained.
Isode HF Strategy: “Everything above the Modem”
Isode's HF Strategy is “Everything above the Modem”, which reflects:
- Isode does not provide HF modems, ALE (Automatic Link Establishment) and other hardware necessary for HF communications.
- HF characteristics impact every layer involved, all the way up to the applications. To address HF communications, Isode provides capabilities from link layer all the way up to the application.
- There is good NATO and US standardization of Modem and below. There is work needed above modem layer, and this paper describes Isode work, including:
- Research to better understand HF behaviour.
- Innovation of product capabilities.
- Standardization to address necessary functionality.
Partner Products & Integration
There are several components of an HF system (e.g., antennae) that Isode does not supply. The three HF components that integrate with Isode products are:
- Modems. This is the primary integration point and Isode works with modem partners (currently RapidM and Rockwell Collins). Note that some modern modems are integrated with Software Defined Radios, rather than being separate products. Modem control interfaces are proprietary, and so vendor-specific support is generally needed to control speed and other parameters. Any modem can be supported for fixed speed operation.
- Address Link Establishment (ALE). ALE is used to select and negotiate an HF frequency to use, and bandwidth of link. ALE interfaces are proprietary, and so support is needed for each ALE unit and ALE type (2G/3G/4G).
- Synchronous Serial. HF security is provided by Crypto boxes that generally require use of synchronous serial interfaces. So, support for synchronous serial hardware is necessary in the Isode product set. Isode supports synchronous serial products from Sea-level and has a framework to add support for other synchronous serial products as needed.
Isode has a number of products to support HF, which are described in more detail in this paper. These are:
- Icon-5066. This provides STANAG 5066 HF Link Layer services.
- M-Switch. M-Switch products provide ACP127, CFTP and ACP142 messaging protocols operating over STANAG 5066.
- Sodium Sync. This provides bandwidth-efficient directory synchronization operating over messaging using M-Switch.
- M-Link. M-Link products provide XMPP real time chat and group communication over STANAG 5066.
- Icon PEP. This future product will provide optimized TCP and UDP operation for other applications over STANAG 5066.
STANAG 5066: The HF Link Layer
STANAG 5066 "Profile for High Frequency (HF) Radio Data Communication" is a NATO specification to enable applications to communicate efficiently over HF Radio. STANAG 5066 provides peer protocols that operate above an HF Modem and below the application level. STANAG 5066 includes the (mandatory) SIS (Subnet Interface Service) protocol that enables an application to connect to an HF Modem through a STANAG 5066 server over TCP/IP. This enables a clean separation between application and modem. An introduction to STANAG 5066 is provided in [STANAG 5066: The Standard for Data Applications over HF Radio].
Why STANAG 5066 is Key
An HF Modem provides basic transfer of data over HF. A link layer is needed over this to provide reliable data exchange and to allow multiple applications to share the HF channel. A specialized protocol is needed to address the special characteristics of HF. STANAG 5066 provides a good solution, and there are no realistic standardized alternatives. Key capabilities:
- Decent efficiency.
- Client access for both reliable (acknowledged) and un-acknowledged data. This is achieved by the SIS protocol.
- Allow multiple clients to efficiently share a channel. STANAG 5066 provides a general framework and two specific protocols to achieve this.
STANAG 5066 is central to Isode's HF strategy.
S5066-EPs and Enhancing STANAG 5066
While STANAG 5066 is a good basis for HF communication, there are some deficiencies, in particular problems relating to use of STANAG 5069 (the new Wideband HF Standard). Isode has led community specification of a series of extensions to STANAG 5066 (STANAG 5066 Extension Protocols). An index of these specifications is available as [S5066-EP1]. These specifications have been made available to NATO, so that they can be incorporated into a future edition on STANAG 5066. Icon-5066 makes use of some of these specifications which can be turned off to achieve strict compliance to STANAG 5066 Ed3.
A list and short summary of all the EPs is given below, and subsequent sections describe how Isode makes use of some of these extensions.
|S5066‑EP2||STANAG 5066 Padding DPDU||This uses “spare” transmission space to enable improved resilience and EOW communication|
|S5066‑EP3||Pipelining the CAS 1 Linking Protocol||This reduces communication latency.|
|S5066‑EP4||Data Rate Selection in STANAG 5066 for Autobaud Waveforms||This provides a modern approach to Data Rate Selection, enabling optimization for both Latency and Throughput. It also provides support for STANAG 5069.|
|S5066‑EP5||STANAG 5066 Large Windows Support||This addresses ARQ window exhaustion, which will impact ARQ usage with STANAG 5069 and at faster narrowband HF speeds.|
|S5066‑EP6||Slotted Option for STANAG 5066 Annex K||This improves performance and resilience of Annex K (CSMA) for small numbers of nodes.|
|S5066‑EP7||Advertising Extended Capabilities||This facilitates interoperable use of the various S5066-EPs.|
|S5066‑EP8||Block Based EOTs||This enables more accurate timing of end of transmission to give faster turnaround.|
|S5066‑EP9||Compact Acknowledgement||This reduces the overhead of end to end acknowledgement.|
|S5066‑EP10||Extension DPDU||This supports protocol extensibility.|
|S5066‑EP11||Variable C_PDU Segment Size||This gives better link utilization.|
Icon-5066 is Isode's STANAG 5066 link layer product. A product description of Icon-5066 is given on the Icon-5066 page. The rest of this section looks selected current and planned capabilities, particularly those that are tied to Isode research or differentiate Icon-5066 from other STANAG 5066 products.
HF has traditionally been operated over 3kHz channels, which gives a practical throughput limit of around 9600 bps for one channel in good conditions. There are two key developments to increase throughput over HF:
- WBHF. This uses up to 16 adjacent channels to obtain a channel up to 48 kHz with maximum speed of 240,000 bps. This technology is ready to deploy:
- Specified in STANAG 5069 with identical specification standardized in US MIL-STD-188-110D.
- At least three implementations of STANAG 5069.
- 4G ALE to support WBHF is standardized in US MIL-STD-188-141D.
- Necessary STANAG 5066 extensions are specified in S5066-EP4 and S5066-EP5.
- HFXL. This uses up to 16 non-contiguous 3kHz channels to obtain the same maximum throughput as WBHF. HFXL is specified in STANAG 4539 Annex H. A key benefit of HFXL is more flexible use of spectrum to find “free slots”. Readiness concerns on HFXL:
- There is only one known implementation.
- Supporting specifications (ALE, Data Link) are immature.
Support of Wideband HF (both WBHF and HFXL) is a key Isode target. Isode's initial focus is on WBHF, as this is more mature technology.
Modems & Supporting Multiple Modem Vendors
Icon-5066 is modem-independent, whereas many STANAG 5066 servers are associated with a specific modem. Modem control interfaces are specific to the modem vendor, and so Icon-5066 needs a different control interface for each modem family supported. Icon-5066 uses a driver architecture so that modem drivers (and other components) are decoupled from the core. The drivers are written in Lua (a high-performance scripting language) which makes it straightforward for Isode and Isode partners to develop and maintain drivers. This approach key to maintaining modem independence and provides flexible support for Isode's modem partners
Duplex & Broadcast
Most deployed STANAG 5066 operates using ARQ (Automatic Repeat reQuest) to provide acknowledged/reliable communication between a pair of nodes with simplex communication (nodes transmitting in turn) to exchange a limited amount of data. This is an important mode of operation, but Icon-5066 is also oriented to support other modes of operation that will be of increasing importance:
- Multicast. Multicast is an important way to optimize use of the broadcast nature of HF. ACP 142, which is supported by M-Switch and described below, is the standardized multicast protocol operating over STANAG 5066. This uses non-ARQ support in Icon-5066.
- Broadcast. Most deployed broadcast systems, such as BRASS operate without use of a link layer such as STANAG 5066. STANAG 5066 defines a non-ARQ broadcast mode. Using this will allow applications to share a broadcast link and provides data integrity. New applications will need to be developed to take advantage of this.
- Duplex. HF Duplex transmission can be achieved with good separation of transmitters and receivers, which is straightforward for fixed shore stations and possible on larger ships. Duplex transmission can significantly improve performance and reduce latency. STANAG 5066 has a duplex mode, which Icon-5066 implements and can use with either one modem (in duplex mode) or a pair of modems (one for transmit and the other for receive).
Automatic Link Establishment (ALE) is an important HF technology that allows sharing of frequencies by a set of nodes and selection of a frequency that is available and propagates well. ALE versions 2G, 3G and 4G are standardized. ALE is important for automated deployment in many environments, and 4G ALE will generally be needed in support of STANAG 5069 WBHF deployment.
Icon-5066 supports use of ALE in conjunction with STANAG 5066, by establishing an ALE link prior to starting STANAG 5066 communication. Icon-5066 does not implement any ALE protocols; rather it provides a generic ALE driver framework so that ALE provided by modem or radio vendors can be driven. This allows choice of appropriate ALE for a deployment.
The basic ALE model assumes that each application is a single ALE station. This works well for mobile units with a single modem. It can also work for a fixed system supporting multiple mobile units, where there is a modem assigned for each mobile unit. If the fixed station needs to share a pool of modems with a larger number of mobile units a different approach is needed. Isode's planned approach is described in the HFIA presentation Using ALE with STANAG 5066.
Monitoring & Analysis
A simple view of STANAG 5066 is that it is a “protocol stack” and can essentially be buried between application and modem. While this can make sense, there are important reasons why the STANAG 5066 layer needs to be made visible.
Icon-5066 provides operator-oriented Web monitoring of activity of STANAG 5066 and Modem systems. This addresses a simple requirement of checking that the HF subsystem is working correctly. Monitoring at the application level, while often appropriate for faster networks, is not sufficient for HF. It can be difficult to distinguish between the HF layer working normally (and slowly) and the HF layer not working. Good STANAG 5066 monitoring makes this straightforward.
A broader issue is that HF provides a difficult and variable channel and optimizing link layer and applications for that channel is not particularly well understood. For those testing and trialling systems, it is important to clearly understand operational performance. This understanding can lead to performance improvements, either by configuration or product change. Monitoring is the first part of this analysis. It is important to be able to observe how a system behaves in order to assess its performance.
Real time analysis is important but is only one element of performance analysis. Often runs will be impractical to observe as they happen; Data needs to be analyzed retrospectively. Also, it is often preferable to analyze using a statistical, rather than looking at individual instances. To achieve this, Icon-5066 plans to record all data in a Time series database (TSDB). This can support visualization of historical data and gives a framework for more general analysis of what happened. This data will enable product and configuration tuning.
Data Rate Selection
When an HF transmission is made over STANAG 5066, a number of decisions need to be made at the start of the transmission:
- Interleaver (block size).
- Length of Transmission.
- Maximum C_PDU Segment size (largest block of data to be transmitted in single D_PDU).
These choices all affect throughput and latency at STANAG 5066 level, which reflects up to the applications using STANAG 5066. Isode research has considered this in the Isode whitepaper [Optimizing STANAG 5066 Parameter Settings for HF & WBHF]. Further research is anticipated, as this area is not well understood.
S5066-EP4 (“Data Rate Selection in STANAG 5066 for Autobaud Waveforms”) is an approach based on this research, which is implemented in Icon-5066. The 2018 HFIA presentation Experience with STANAG 5066 Data Rate Selection supports this approach which suggests it is a good direction. Isode expects to continue work on this area, which is key to optimizing STANAG 5066 and HF performance.
Multi-Node: Annex K
It is often useful to have several nodes share a single HF frequency. This might be achieved by a fixed frequency configuration or use of ALE to establish a frequency shared by a set of nodes. When a pair of nodes are doing ARQ communication, this will ensure alternating transmission. Otherwise the original STANAG 5066 did not say anything about controlling when nodes transmit. This is an important issue for multi-node systems, as when multiple nodes transmit at the same time, there is significant performance impact. Several STANAG 5066 systems developed ad hoc approaches to address this.
STANAG 5066 Ed3 developed a framework to address this, specified in Annex J. Annex K adds a CSMA (Carrier Sense Multiple Access) approach to address this, also known as Listen Before Transmit, based on one vendor approach. This uses a jitter mechanism, where a node waits a randomized time before transmitting. The exact parameters of the jitter are based on size of network and anticipated usage pattern. The jitter approach is a good one for a network with a large number of nodes.
S5066-EP6 (“Slotted Option for STANAG 5066 Annex K”) provides important clarifications for Annex K usage and defines an alternative mechanism based on assigned time slots relative to end of previous transmission. This gives improved performance and resilience for networks with a small number of nodes.
Icon-5066 supports both Annex K and S5066-EP6.
Multi-Node: Annex L
While Annex K provides a simple robust approach, it is relatively slow at switching between nodes, which makes it unsuitable for a network with high utilization. Annex L addresses this with a Wireless Ring Token Protocol that enables managed switching of control between multiple nodes.
Although the details of Annex L are quite complex, the principle it works on is very simple. There is a token, which is held by one node at a time. Only the node holding the token is allowed to transmit. When the node has finished transmitting, it will pass the token on to another node. The token provides a simple mechanism to ensure that two nodes will not transmit at the same time. Annex L allows the sender full choice of transmission parameters, which means that it can provide excellent performance and channel utilization.
Isode plans to add Annex L support to Icon-5066. This is an important capability.
Isode also plans to look at two capabilities, which will likely lead to evolution of Annex L:
- Variable speed. Annex L is specified for fixed speed. It is highly desirable to adapt to varying conditions and modify speed and other parameters to appropriate values.
- Improved Ring Recovery. There is one known Annex L implementation (SPAWAR HFIP). Anecdotal reports suggest that it works well when it works, but in the event of token loss, ring recovery is very slow. Isode believes that work will need to be undertaken to minimize risks of token loss and speed recovery in the event of token loss.
STANAG 5066 Security
STANAG 5066 has a very simple architecture for encrypting data. A stream cryptographic device is placed between modem and STANAG 5066. All nodes in the network have a common secret key. This is a simple and well understood architecture, but has some problems; in particular:
- When used with modern block-oriented modems, it can lead to long turnaround times. This is significantly compounded by most cryptos which use synchronous serial interfaces. This problem can be addressed but is complex for the implementation. Icon-5066 plans to add capabilities to address this. A detailed discussion is in the Isode whitepaper [Reducing Turnaround Times in STANAG 5066].
- When doing ARQ it would be significantly preferable to do this below the crypto. Correcting data before encryption is handled would improve resilience and give wider choice of crypto mechanism. It would also eliminate problems associated with crypto sync data being corrupted. This problem is fundamental to the current architecture.
The way to improve things is to move the crypto boundary upwards, as is done for communication at many other frequencies. HF BLOS propagation means that HF transmissions are straightforward to intercept and there are many systems that use this to determine location of HF transmissions. A consequence of this is that care needs to be taken about what is transmitted un-encrypted, and in particular there are strong security reasons to not transmit any addressing information in the clear. This puts a significant constraint on any changes that can be made.
Technically, it is possible to address some of the constraints of the current approach in a secure manner. A potential direction is noted in the Isode HFIA presentation A new COMSEC Architecture for HF .
Crypto Bypass & XML Guards
In the STANAG 5066 Crypto architecture all data to be transmitted over the air goes through a crypto. STANAG 5066 sits red side; modem sits black side; and crypto separates them. While limited operation can be achieved with full red/black separation, improved operation can be achieved by having information and control flow across the red/black boundary. In particular:
- Optimal speed selection. This requires modem speed control to flow red to black and modem SNR information to flow black to red.
- ALE. ALE units sit black side. For STANAG 5066 to use ALE, control information needs to flow in both directions. ALE allows selection of frequency and for STANAG 5069 WBHF it also selects bandwidth.
Although none of this information is transmitted over the air, this information does need to cross the red/black boundary, creating what is known as “crypto bypass”. This raises security accreditation concerns, particularly for red to black flow which may create a covert channel.
The benefits of crypto bypass for ALE and Speed Selection are significant, particularly for WBHF. For these reason, modern HF systems will generally prefer to use crypto bypass. While it will sometimes be acceptable to make a network connection across the red/black boundary, it will generally be required to have an accredited solution. The industry standard for this sort of boundary is an XML Guard. Icon-5066 will add capability to make use of XML guards to connect across the red/black boundary. This is described in more detail in the Isode HFIA presentation: Using XML Guards to Support STANAG 5066 Crypto Bypass.
EMCON (Emission Control) is a state for a site of not transmitting data. It is often presented as a product capability, but the concept is broader than a product. EMCON is an operational state that will often be implemented at multiple levels. For example, a radio transmitter may be turned off (physically enforcing EMCON) and a messaging application will be set into EMCON state so that it does not send any messages. Icon-5066 has planned support for EMCON, as well as associated support in M-Switch ACP 142, which does the following:
- A local EMCON state may be set, which enforces that no data is sent.
- It can be recognized that a peer entity is in EMCON state and that no data will be received from the peer.
- Operator can set EMCON status locally and for peers.
- EMCON state can be communicated. For example, if EMCON state is set for Icon-5066, EMCON state will also be set for M-Switch ACP 142 and communicated to an attached radio.
This provides a way of managing EMCON state across the Isode product set and with connected components.
Applications over STANAG 5066
Isode strongly recommends that critical applications communicating over HF do so using STANAG 5066. This section explains why and gives an overview of Isode applications with HF support.
Why Operate Direct over STANAG 5066
In order to directly use STANAG 5066, application protocols or profiles need to be defined and the application will connect directly to a STANAG 5066 server using the STANAG 5066 SIS protocol. Doing this directly has a number of benefits:
- The application can make use of STANAG 5066 ARQ and non-ARQ services with a minimum of protocol overhead.
- End to end handshakes, which may significantly degrade performance, are minimized.
- The application has direct access to SIS flow control, which enables link utilization to be optimized.
These are all important when trying to maximize performance over a difficult channel.
While this approach requires special protocols to be used over HF, this can be completely transparent to the end systems. For example, messaging communication can use standard SMTP for most communication, and then convert to HF optimized protocols for communication over the HF channel. This application relay approach means that operation over the channel can be optimized while most systems can simply use standard protocols.
Isode provides messaging products for military and general purpose messaging. The two most important are:
- Harrier, which is a Web Military Messaging client.
- M-Switch which is a Message Transfer Agent (MTA) that supports a number of messaging protocols. Providing protocol conversion enables Harrier and other servers and clients to communicate over HF.
M-Switch supports three protocols for operation over HF.
- ACP 127. ACP 127 is an old and basic military messaging protocol widely used for HF communications often as part of a BRASS system. ACP 127 operates over STANAG 5066 using Character Oriented Stream Service (COSS) which emulates a serial line.
- ACP 142. ACP 142 is a military specification for multicast messaging that can be operated over STANAG 5066. It supports flexible and efficient modern messaging with two variants:
- STANAG 4406 Annex E. This is the NATO standard for messaging over constrained bandwidth.
- MULE (Multicast Email) defines operation of SMTP based messaging over ACP 142. This can be used for both email and military messaging.
- CFTP (Compressed File Transfer Protocol). This is a basic protocol defined in STANAG 5066 to support SMTP email over HF. It is also known as Battleforce Email (BFEM). While inferior to ACP 142/MULE, it may be useful for interoperability.
Directory Services are often vital to support operations in the sort of systems that use HF. Client/server access to directory services over HF would generally be too slow, so a preferred architecture is to maintain separate directory servers and keep them synchronized with updates. Isode provides a solution for this using Isode's Sodium Sync product, exchanging incremental update over email, described in [Directory Replication by Email and over 'Air Gap'].
XMPP (eXtended Messaging and Presence Protocol) is the open standard for real time messaging, being adopted by many organizations including NATO. Isode M-Link supports operations over HF described in M-Link: Constrained Networks. This includes operation over STANAG 5066 as standardized in XEP-0365: Server to Server communication over STANAG 5066 ARQ, which provides optimized point to point communication. Group communication is optimized using Federated Multi-User Chat (FMUC) standardized in XEP-0289: Federated MUC for Constrained Environments.
Supporting IP Applications
Best performance will be obtained by operating applications directly over STANAG 5066 and this approach is recommended for all mission critical applications. It is impractical to do this for all applications, so this section considers generic support of IP applications.
The Problems with STANAG 5066 F.12
STANAG 5066, in Annex F Section 12, defines a mechanism to operate IP over STANAG 5066. This enables a subnet to be provided using STANAG 5066, as illustrated above. This follows the standard IP subnet architecture and is a clean approach. However, use of this mechanism has significant problems:
- Decision Point. The choice to send or not send an IP packet is made by applications which this architecture decouples from the STANAG 5066 subsystem. This decoupling makes it hard to ensure that the HF channel is used to full capacity, but not overloaded (leading to IP discard).
- TCP. This decoupling is a particular problem for TCP, which is used by many applications. Under load the window and retransmission algorithms of TCP interact very badly with typical HF subnet characteristics.
- Overhead. IP has large packet header. TCP has large headers and uses extra packets for acknowledgements and other functions. This overhead is an impact at HF speeds.
- Priority. When there are multiple applications using a slow HF link, it is highly desirable to control use of the link by priority. The F.12 architecture does not enable this.
More detailed analysis of the issues with STANAG 5066 F.12 are given in the Isode whitepaper [Performance Measurements of Applications using IP over HF Radio].
It is useful to understand how optimized applications protocols can improve performance. This is shown in the diagram above, with the proxy using a protocol optimized for HF (e.g., for messaging ACP 142/MULE) with transparent relay to the end application using a standard protocol (e.g., SMTP).
This architecture gives maximum efficiency where (proxy) protocols have STANAG 5066 mappings.
Middle Layer Proxy
The Middle Layer Proxy architecture shown above provides an intermediate architecture. The protocols operating over IP are terminated by a proxy and then use an optimized protocol over HF. This is shown for TCP and UDP, which are the most popular protocols used over IP. The architecture could be applied to any protocol running over IP. The application protocol runs end to end.
This architecture optimizes TCP and UDP performance over the link. For TCP, this architecture is often called Performance Enhancing Proxy (PEP). Benefits of this architecture:
- Minimizes TCP and UDP overheads on the HF link using a special protocol.
- Uses the HF link efficiently and maximizes utilization.
- Enables prioritization based on application type and endpoints.
This will give good performance for a wide range of applications, noting that:
- Applications with lots of end to end handshaking will not perform well due to high HF latency.
- Applications with short time-outs may have problems.
Icon PEP is a planned Isode product. It will support the middle layer proxy architecture set out above, as well as STANAG 5066 F.12. Icon PEP will facilitate efficient deployment of IP applications over HF, and will support a wide range of HF and WBHF speeds.
Isode HF Road Map
This whitepaper has set out Isode's HF vision. It has described Isode's existing product capabilities, including Icon-5066, Messaging, XMPP and Directory applications for operation over HF. Road Map capabilities are also described, with the major elements:
- Addition of Icon PEP to provide support for IP applications not provided by Isode.
- Addition of new functionality to Icon-5066:
- Annex L (Wireless Token Ring) support.
- XML Guard support for supporting modem control and ALE over red/black separation.
- EMCON support.