Messaging and Directory Server software used around the world in the Government, Military, Aviation and Commercial sectors.

isode.com / whitepapers /

Performance Measurements of Messaging Protocols over HF Radio

Summary: This white paper sets out and analyses the results of a measurements of various messaging protocols over HF Radio. HF Radio has unusual performance and reliability characteristics, which has led to specific application protocols being developed. This paper finds that three of the four protocols analysed perform well. It concludes with a discussion of the best choice of messaging protocol for various types of deployment.

Publish Date: 05 Aug 09 Modify Date: n/a

| More

Messaging Protocols Compared

This paper looks at messaging protocols specifically developed for use over HF Radio. A future paper will look at use of standard protocols operating over IP (over HF Radio). All of these protocols operate over STANAG 5066, described in the Isode white paper "STANAG 5066: The Standard for Data Applications over HF Radio".

The four protocols are:

1. HMTP (HF Mail Transfer Protocol) defined in STANAG 5066 Annex F
2. CFTP (Compressed File Transfer Protocol). CFTP is a file transfer mechanism, and STANAG 5066 Annex F defines its use for message transfer.
3. ACP 142 is a CCEB Combined Communications-Electronics Board – AU, CA, NZ, US, UK) Protocol for reliable multicast. It was defined for use with STANAG 4406 Annex E, and described in the Isode white paper Military Messaging over HF Radio and Satellite using STANAG 4406 Annex E.
4. Connection Oriented ACP 142 (CO-ACP 142) is a protocol defined by Isode based on ACP 142 that is optimized for point to point (ARQ) communication. CO-ACP 142 is described in the Isode white paper Messaging Protocols for HF Radio, which also describes operation of Internet mail (using BSMTP (Batch SMTP)) over ACP 142 and CO-ACP 142).

All of the testing was done with Internet mail, as only the latter two protocols support use of STANAG 4406 (X.400). The results are broadly applicable to X.400, which is discussed separately.

The CO-ACP 142 measurements are described first, and detailed analysis is given. Reasons for this choice:

• CO-ACP 142 has the best performance of the four protocols.
• CO-ACP 142 shares capabilities with both ACP 142 and CFTP, and so is a natural reference point.
• CO-ACP 142 (and ACP 142) are Isode’s implementation, and so we were able to measure at more points of the stack than for CFTP and HMTP.

Testing Framework

The above diagram shows the test architecture. A message generation script injects a number of messages into the sending server using SMTP. The protocol under test then transfers to the receiving system, where the received messages are retrieved using POP. Message performance analysis looks at the time information in the message to determine start and end times. There is also an option (for the RapidM servers) to determine end time by POP polling. For the multicast tests, two receivers are used. For the ACP 142 and CO-ACP 142 tests, some additional information was obtained from the messaging server logs.

Messages were given a payload that is random binary data, so that minimal compression of this data is expected. This type of data was chosen, as the tests are primarily seeking to measure data transfer, and not the effectiveness of compression algorithms. Payload sizes used are: 0; 1 kByte; 10 kByte; 100 kByte. Message numbers were varied, using 1, 10 and 100 messages in test runs.

All tests used the STANAG 4539 Waveform and maximum MTU size for STANAG 5066 (2048 bytes). A base set of tests was done at 1200 bits per second. This was chosen as a typical mid-range HF speed and one where STANAG 5066 has been measured to perform well. Tests were done with all combinations of payload size and number of messages, where the run was expected to complete within six hours. A subset of tests were done at 75 bits per second and 9600 bits per second, which are the lower and upper bounds of coded HF data rates. The goal of this mix was to test a wide range of message loads.

The underlying STANAG 5066 system used for these tests is described in the STANAG 5066 Performance White Paper. This uses three RapidM RM6 HF Modems, connected with audio links (“Perfect HF Radio”) and using the RapidM RC66 STANAG 5066 Servers (one per modem).

CO-ACP 142

CO-ACP 142 (Connection Oriented ACP 142) is described in the Isode white paper Messaging Protocols for HF Radio, The protocol stack choices provided by Isode and implemented in the M-Switch ACP 142 Channel are shown below:

These tests used the stack option: BSMTP; Compression; ACP 142; CO-ACP 142; RCOP; STANAG 5066. The deployment configuration is shown below.

The following tables show the results of the tests run, grouped by link speed. The following data points are shown:

1. Number of messages.
2. Size of payload – the random binary data carried in the message.
3. Size of submitted messages gives the size of the RFC 2822 message created by the test generator for submission.
4. Size of the BSMTP file, gives the size of the file that includes message and envelope data provided for transfer.
5. Size of ACP 142 Transfer file shows the size of file passed to the ACP 142 process, which is essentially the BSMTP file after the STANAG 4406 Annex E compression.
6. First message arrival time is the delivery time of the first message, relative to submission time of the first message.
7. Final message arrival time is the delivery time of the last message, relative to submission time of the first message.
8. Time from final message arrival to last ack. The acknowledgement of the final message back to the sending system will always happen after the final message delivery. This measures how much after final message delivery this acknowledgement happens.
9. First message throughput measures the effective throughput of the first message. Link utilization measures this as a percentage of the link speed. This measurement is calculated from the ACP 142 file size, and so the throughput and utilization numbers are measuring CO-ACP 142, RCOP and STANAG 5066 layers. This does not look at message encoding overheads and gains from compression, which are looked at later.
10. Where more than one message is sent, the same analysis is done for all messages (excluding the first one). This gives a figure that excludes the overhead of link establishment.
11. Message rate gives an effective throughput in messages per hour, which gives an alternate view that may be helpful to consider performance.

Results for 1200 bits/second

Size of Payload (kBytes)	0			1			10			100
Size of Submitted Message (bytes)	386			1867			14025			135604
Size of BSMTP File (bytes)	700			2206			14521			137679
Size of ACP 142 Transfer File (bytes)	351			1448			10825			104421
Number of Messages	1	10	100	1	10	100	1	10	100	1	10
First Message Arrival Time	21	19	23	30	34	37	97	107	101	796	858
Final Message Arrival Time		56	346		148	1182		837	8268		7993
Time From Last Message Arrival to Last Ack	6	7	41	3	4	27	5	4	5	4	8
First Message Throughput (bits/sec)	134	148	122	386	341	313	893	809	857	1049	974
Link Utilization for First Message	11%	12%	10%	32%	28%	26%	74%	67%	71%	87%	81%
Subsequent Message Throughput (bits/sec)		683	861		915	1002		1068	1050		1054
Link Utilization for Subsequent Messages		57%	72%		76%	83%		89%	87%		88%
Message Rate (message per hour)			1103			311			44		5

Results for 9600 bits/second

Size of Payload (kBytes)	0		10		100
Size of Submitted Message (bytes)	386		14025		135604
Size of BSMTP File (bytes)	700		14521		137679
Size of ACP 142 Transfer File (bytes)	351		10825		104421
Number of Messages	1	10	1	10	1	10
First Message Arrival Time	18	16	24	21	127	146
Final Message Arrival Time		21		130		1182
Time From Last Message Arrival to Last Ack	11	4	6	5	12	8
First Message Throughput (bits/sec)	156	176	3608	4124	6578	5722
Link Utilization for First Message	2%	2%	38%	43%	69%	60%
Subsequent Message Throughput (bits/sec)		5054		7150		7257
Link Utilization for Subsequent Messages		53%		74%		76%
Message Rate (message per hour)		6,480		297		31

Results for 75 bits/second

Size of Payload (kBytes)	0		1		10
Size of Submitted Message (bytes)	386		1867		14025
Size of BSMTP File (bytes)	700		2206		14521
Size of ACP 142 Transfer File (bytes)	351		1448		10825
Number of Messages	1	10	1	10	1
First Message Arrival Time	91	86	254	226	1647
Final Message Arrival Time		619		2102
Time From Last Message Arrival to Last Ack	14	61	13	13	13
First Message Throughput (bits/sec)	31	33	46	51	53
Link Utilization for First Message	41%	44%	61%	68%	70%
Subsequent Message Throughput (bits/sec)		47		56
Link Utilization for Subsequent Messages		63%		74%
Message Rate (message per hour)		61		17	2

Some observations on these numbers:

1. Broadly the numbers are in line with what might have been expected from the previous measurements of the underlying STANAG 5066 subsystem, and the performance that should be expected from a messaging protocol making efficient use of the STANAG 5066 layer.
2. Performance of the messaging layer is consistent and good at all speeds (noting that the STANAG 5066 layer is slightly more efficient at 1200 than and 75 or 9600).
3. For large messages, link utilization is good, with minimal overhead.
4. For small messages, once link is established, that ACP 142/RCOP/STANAG 5066 layer leads to an overhead of around 30% for small messages. This overhead is discussed below.
5. To send any message, there is a delay of 10-20 seconds as a base value. For short messages, this delay dominates the effective throughput of the first message. This is a cost of STANAG 5066 link setup.
6. For the higher speeds, final acks will typically arrive 4-6 seconds after the final message at 9600 bits per second single message transfer it takes longer, which we believe is due to an unexplained STANAG 5066 level characteristic. At 75 bits per second, the time to transfer the ack data has a noticeable effect. For large numbers of messages at 1200, the final ack is noticeably later. This is a consequence of the RapidM STANAG 5066 implementation choice (described in the STANAG 5066 performance paper) to limit the volume of data on the return channel. For small messages, this leads to a buildup of acks, which only get cleared after all of the messages have been sent.
7. There was a quite significant variation in numbers on repeat runs, with differences in 10s of seconds (although broadly consistent with the published numbers). A possible explanation for this is given in the ACP 142 section.

CO-ACP 142 Utilization Analysis

We now look in more detail a two results:1 kByte and 100 kByte payload at 1200 bits per second, in order to analyse protocol performance for large and small payload. In the following analysis, messaging protocol overhead is determined by difference between the payload size and the ACP 142 file size. RCOP and CO-ACP 142 overheads are determined by theoretical analysis (which we believe will be accurate) and STANAG 5066 overhead is the balance.

100 kByte Payload Analysis

1 kByte Payload Analysis

The performance of each of the layers is considered in turn.

STANAG 5066

The 12-13% overhead for STANAG 5066 for payloads in 1-100 kByte range is in line with the measurements in the STANAG 5066 Performance White Paper. This overhead seems reasonable, given the services provided, and significant improvement would be difficult.

For messages without payload, the overhead increases to 23%. This reflects the measured overhead for sending smaller PDUs. STANAG 5066 is not optimized for transferring a sequence of small PDUs between a pair of destinations. If this type of traffic is important, it would make sense to optimize at the STANAG 5066 layer, by including a concatenation capability within STANAG 5066.

RCOP

The RCOP (STANAG 5066 Reliable Connection Oriented Protocol) overhead is very low, and seems reasonable.

CO-ACP 142

The CO-ACP 142 overhead for a large message is negligible. The 3% overhead for a 1 kByte payload seems reasonable for the service provided (message transfer).

Message Overhead

The 4% message overhead for the 100 kByte pay load is mostly due to the 7 bit encoding of the binary payload. The subsequent compression removes most, but not all of this encoding overhead. For CO-ACP 142 and for ACP 142 (but not for CFTP or HMTP) this overhead can be avoided by use of a binary encoding (either BINARYMIME or 8BITMIME).

For the smaller message, the analysis is more complex and analysed in the rest of this section. It should be noted that 1 kBtye is a small message payload, and that for larger messages the per message overhead is a smaller fraction of the total. Although some gains are possible, the benefits of the gains will depend on target payload. For very small messages, use of an alternate approach (such as XMPP which is designed for small messages) may be appropriate.

The processes involved are illustrated by example. Here is one of the test messages used:

Content-Type: multipart/mixed; boundary="===============1547036784=="
MIME-Version: 1.0
Date: Thu, 23 Jul 2009 10:25:10 -0000
Subject: test 0
From: pop.user@dhcp-122.isode.net
To: pop.user@dhcp-236.isode.net

--===============1547036784==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

This is a text body
--===============1547036784==--

The “Content Type” and “boundary” lines are not needed for a small message, but are a consequence of using a message designed to carry a payload, and this is the message with zero payload (385 bytes).

Prior to transfer, this gets wrapped into a BSMTP file (Isode defined Batch SMTP) shown below:

<pop.user@dhcp-122.isode.net>
<pop.user@dhcp-236.isode.net>

Received: from dhcp-122.isode.net ([172.16.1.122]) by dhcp-122.isode.net
(smtp internal)
via TCP with ESMTP id <Smg6hgAJqTgA@dhcp-122.isode.net> for
<pop.user@dhcp-236.isode.net>;
Thu, 23 Jul 2009 11:25:10 +0100
Content-Type: multipart/mixed; boundary="===============1547036784=="
MIME-Version: 1.0
Date: Thu, 23 Jul 2009 10:25:10 -0000
Subject: test 0
From: pop.user@dhcp-122.isode.net
To: pop.user@dhcp-236.isode.net

--===============1547036784==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

This is a text body
--===============1547036784==--

The first address in the BSMTP file is the return address for errors, and Subsequent addresses are recipients for the message, along with any delivery parameters. Note that trace information (Received:) has been added by M-Switch, and this is included in the measured BSMTP overhead. A more complex example showing a test message with a payload is:

<pop.user@dhcp-122.isode.net>
<pop.user@dhcp-236.isode.net>

Received: from dhcp-122.isode.net ([172.16.1.122]) by dhcp-122.isode.net
(smtp internal)
via TCP with ESMTP id <Smg=MAAJqXcB@dhcp-122.isode.net> for
<pop.user@dhcp-236.isode.net>;
Thu, 23 Jul 2009 11:45:04 +0100
Content-Type: multipart/mixed; boundary="===============1760449971=="
MIME-Version: 1.0
Date: Thu, 23 Jul 2009 10:45:04 -0000
Subject: test 0
From: pop.user@dhcp-122.isode.net
To: pop.user@dhcp-236.isode.net --===============1760449971==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit This is a text body
--===============1760449971==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

F1xsK6AmS0CZUe6nWAtxfYK8iss9AmgvVnBTqqXMo68O+
RF2yDGleSdj/JogKpQ6U8LGyZL311SbbLShrSKWC5YVjY7
6GTrepKUrymezYZfCDONsh57NM9aEZIylQ64LBvurWrWe
JCkbUu9hdx/NFHnbqHS0sM6ADeIuoao/0ykNOoD9b5QlB
CviVyhRP2VDwUyf1mjdXmRzlWH2ml3Xk966n+/29749s/
bo8VESIhG7Qw/ukertbKliScQdsuuU4D7rvJDAPgR6cH8U
+kuEPG4lhJc4Sd/VetMJvWVr2gujo3zl Y4boQkMj7/Lg2g22j
D8RXGWdyLJK1D5/d8T41otoMLThQl7j+D+KnvPQP55rnjn
khloM1X+1e41c /yjUBSyPsWmMgCj0pcSCfhRJMDRAoeou
wgoaqNVZZ+GD1Jq+4GajiiIsZ7NmkBBWOcFmtrrInD1U+A
kws4gG51Q7v27TSB34NBZ397aPgmgJJP1c9zEdvUDRDYaT6
txKMOVHTk4QViBNzuEkepWYSX3P 212Z8heJXpWDMymx+
EqySJZ99lTEpEz3/mvh99Ol1dRgvjAf9J478nRn1EIpA2MTb2vx
6sL3CMlA g6TIIXL5uFf/7ALHJV4ehgOjbZXl7fsGZa2JFi5ME54lh
B3a/y1D0mTS8eXHMLqs8uxlwv64I5Ed Jw/9DO9YpQhl+FZOx
NxLac44Zd0Dm8/zz5bw/AaES5DMrLTNNdEX7L+ncIbKJjPpgBL
hWV+8L81o SNxRH1hcgZcA2cMuLO38rzm1v8z6Tbhknlkoudkx
+XAS+6kZsxOYswAChL15X+iOv/3mnXSq5euN 59v4mQnh
jpf/+wtu+2Y4ugGTi5YJjC3EUqqlEiaysQzBTPPx2WtOx/dJkPhJrsT
zE15S0ufOx1g7 iMVYKZaHg57Dp3PcCtdLGee4efsvv8v48/EdX
HuTNETOMht/Nn72ym80rktLM/Wt9hIMiS/A2r4V 25BxyNL71
werVd8rjB8xJl9M4KWsAf90CeRQN3ggJ1CbEJKIMLo9KgGlFN
bqNmtZ+I0mqtjorK3i quvNT0iCOtgW2BFyTPlZeOLUEJ59Ctk
9jCQICmPwUrq8Po+XzZ2zp8guN0iamkCE46Aftsdj6M+o 2uKq
5ztsO4pdvGPV7YO9DFlmqxkAyBNS6vUA8wTO/z9CqEwAutY
GI1WTqiPt2Ad8QTyEJ2R0KLh7 oyA9Fq7DFNqYo56+cMYHh
AOp+1d8nNTb3Wf5e8KPiz+S4xat6OwOAQ5vxYvbqpaxJQ
51WntO4bxe Bcb+c+zRXzrYhE5ELM7j7HonIz2z6n7VswJ2Wh
/yuQ==

--===============1760449971==--

This file is then compressed using the output format defined in STANAG 4406 Annex E. The message overhead is the difference in size between this final file and the message payload. The factors affecting this overhead are now considered.

The absolute overheads here are 350-450 bytes in the tests. Overheads would be likely to be a little higher with messages from many message clients. This overhead will be acceptable in many situations. For lower link speeds or to support scenarios where many small messages are transferred, it may be desirable to consider optimization.

Message Compression

Message compression uses the DEFLATE algorithm, which is a good general purpose algorithm. It is discussed in the Isode White Paper Measuring MMHS Performance over HF Radio and Satellite: STANAG 4406 Annex E Encoding and Compression.

The output encoding is compact. It provides for use of custom compression, which we believe will be important flexibility that gives potential for significant performance improvement. The base format and approach seem optimal.

BSMTP

<pop.user@dhcp-122.isode.net>
<pop.user@dhcp-236.isode.net>

The BSMTP overhead (66 bytes in the example) is necessary to control message transfer. This is compact and functional, and there seems little scope for reducing this overhead. Note that this data is also compressed before it is transferred.

Received: from dhcp-122.isode.net ([172.16.1.122]) by dhcp-122.isode.net
(smtp internal)
via TCP with ESMTP id <Smg6hgAJqTgA@dhcp-122.isode.net> for
<pop.user@dhcp-236.isode.net>;
Thu, 23 Jul 2009 11:25:10 +0100

M-Switch adds trace information to the message (245 bytes). This could be stripped, although removal of this information could make problem diagnosis harder. Where a message has transferred a number of hops prior to the HF transfer, this could give a larger data volume reduction.

Message Formats

Message headers can often contain more information than the test message. Here is a basic message generated by Microsoft Outlook:

From: "Steve Kille" <Steve.Kille@isode.com>
To: "'Steve Kille'" <Steve.Kille@isode.com>
Subject: Demo Message
Date: Wed, 22 Jul 2009 11:27:17 +0100
Message-ID: <007601ca0ab6$fcd4f1a0$f67ed4e0$@Kille@isode.com>
X-Mailer: Microsoft Office Outlook 12.0
Thread-Index: AcoKtvxuTVHHnhpxTSCe5mnx37q5xA==
MIME-Version: 1.0
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit
Content-Language: en-gb

This is 429 bytes. In simple transfer scenarios the From: and To: duplicate information that will be in the BSMTP file. Other information may not be needed for some mission critical functions. Performance could be optimized by sending simpler headers.

Adding file attachments using MIME structures has an overhead of 100 bytes or so. Binary data is usually encoded as 7bit data, as many systems cannot handle 8bit. Most of this encoding is reverted by compression, but the total effect is an overhead of around 4%. In general it would be desirable to avoid this overhead by use of BINARYMIME or 8BITMIME.

STANAG 4406 (X.400)

The measurements at the CO-ACP 142 / ACP 142 Level and below are directly applicable to use with STANAG 4406 messaging. The only difference is the file format transferred. A detailed analysis is given in the Isode White Paper Measuring MMHS Performance over HF Radio and Satellite: STANAG 4406 Annex E Encoding and Compression. STANAG 4406 will give some improvement performance over Internet mail because the formats are more compact. The base ACP 142 file size in the paper is 232 bytes, as opposed to 351 bytes in the test message./ Adding capabilities is efficient, and attachments use an 8bit encoding by default.

Use of STANAG 4406 is a sensible option where it is important to optimize transfer sizes.

ACP 142

ACP 142 is a CCEB Combined Communications-Electronics Board – AU, CA, NZ, US, UK) Protocol for reliable multicast. It was defined for use with STANAG 4406 Annex E, and described in the Isode white paper Military Messaging over HF Radio and Satellite using STANAG 4406 Annex E. The Isode white paper Messaging Protocols for HF Radio, describes operation of Internet mail (using BSMTP (Batch SMTP)) over ACP 142.

These tests used the stack option: BSMTP; Compression; ACP 142; UDOP; STANAG 5066.

ACP 142 Two Node

Although ACP 142 is designed primarily for multicast messaging, the first set of tests was made with just a single recipient. This gives a direct comparison with the CO-ACP 142 measurements. The deployment configuration for two node testing is the same as for CO-ACP 142.

It was planned to test at 75, 1200 and 9600. Tests were only done at 1200, due to a bug in the RC66 software (which we are told will be fixed soon). This bug led to data loss on reception which affected about 3% of packets at 1200. The problem did not show in the STANAG 5066 tests, as it was not triggered by one way data flow. We were able to make a realistic analysis of the test results at 1200, despite the data loss. Data loss at 75 and 9600 was higher, and we are not confident to publish the results. The results at these speeds appear to be consistent with our expectations based on the results at 1200. We believe that this bug also affects ARQ configurations (but was hidden by the ARQ correction), and was a factor in the variability we saw in test run results for CO-ACP 142, CFTP and HMTP.

Size of Payload (kBytes)	0			1			10		100
Size of Submitted Message (bytes)	386			1867			14025		14025
Size of BSMTP File (bytes)	700			2206			14521		137679
Size of ACP 142 Transfer File (bytes)	351			1448			10825		104421
Number of Messages	1	10	100	1	10	100	1	10	100
First Message Arrival Time	6	7	6	14	15	15	101	195	840
CO-ACP 142 First Message Arrival Time (Comparison)	21	19	23	30	34	37	97	107	796
Final Message Arrival Time		55	407		142	1235		880
Time to send acks		5	44		8	112		9
Time from last message to ack start	41	43	40	43	40	23	43	39	42
First Message Throughput (bits/sec)	468	401	468	827	772	772	857	444	994
Link Utilization for first message	39%	33%	39%	69%	64%	64%	71%	37%	83%
Subsequent Message Throughput (bits/sec)		527	693		821	940		1138
Link Utilization for subsequent messages		44%	58%		68%	78%		95%
Subsequent Message Throughput with Ack Allowance (bits/sec)		477	625		772	861		1123
Link Utilization for subsequent messages		40%	52%		64%	72%		94%
Message rate (messages per hour)			730			244		43	4
CO-ACP 142 Message Rate (Comparison)			1108			311		44	5

To interpret these results, it is important to understand how the RapidM non-ARQ implementation of STANAG 5066 behaves. This is described in the STANAG 5066 Performance White Paper. Because of the long turnaround time for non-ARQ transfer, data is sent until the queues are empty. For ACP 142, this means that when multiple messages are sent, no acks will come back until the last message has been sent. This will cause the ACP 142 systems to retransmit. For the tests we configured ACP 142 values of RE-TRANSMISSION_TIME, BACKOFF_FACTOR, and ACK-PDU_TIME to take this into account. Some retransmission did still occur, and this is noted in the Subsequent analysis. Information on setting ACP 142 parameters is given in the Isode deployment note ACP 142 Parameters for Radio and Satellite Networks.

The values in the table are mostly the same as for CO-ACP 142. Notes on new lines in the table are made where appropriate in the analysis below.

1. The time to transfer the first message is fast, and significantly faster than CO-ACP 142 for short messages (a comparison line is given). This is because with non-ARQ transfer there is no link setup needed, and so the delay to start data transfer is minimized.
2. In terms of getting messages from sender to recipient, ACP 142 is fast because there is no return traffic. It can be seen that 94% data link utilization is achieved, which results in very efficient transfer of the messages.
3. It should be noted that these first two benefits are only achieved if there is no data loss, as the non-ARQ data service is unacknowledged. So while this is very good if there is no loss, in the event of data loss application level recovery is needed, and this will happen with delays in line with the acknowledgements discussed below.
4. Because all of the acks come back at the end, it is possible to measure the transfer time for the acknowledgements. This is done as a new line.
5. A second data rate calculation is done for “subsequent messages” that adds in the ack transfer time into the calculation.
6. None of the utilization calculations include the turnaround time. This is done to make clear the basic message efficiency. We also expect future non-ARQ systems to provide much improved turnaround times. STANAG 5066 Edition 3 is seen as important technology.
7. The measurements on turnaround time give a consistent 40 seconds, between the time a message arrives and when the acks start to be sent back. This ties in with the RapidM collision avoidance algorithm.
8. The messages per hour calculation assumes that there will be a full turnaround every 15 minutes lasting 80 seconds (an overhead of around 9%).
9. This leads to a throughput in messages per hour that is comparable with CO-ACP 142, but less.

Multicast

Multicast testing was undertaken with a similar setup to the previous test, but with two receiving systems and a message sent to one recipient on each system. The results from these tests are shown below

Size of Payload (kBytes)	0		1		10
Number of Messages	1	10	1	10	1	10
First Message Time	7	6	15	15	88	98
Final Message Time		58		126		877
Time Acks Complete to Node 1 After Transfer	39	41	42	51	43	54
Time Acks Complete to Node 1 after Node 1 Acks	43	46	80	84	78	91
Effective Link Utilization for 1 Node		38%		73%		82%
Effective Link Utilization for 2 Nodes		36%		69%		81%

The basic setup is and measurements are similar to single recipient testing, and the figures for message transfer are similar. These tests show that there is minimal additional overhead in the transfer phase when transferring to multiple systems. At the end of data transfer, acks need to come back from each system. With the RapidM slotted collision avoidance system, one node will send its acks back first (after a delay) and then there will be a second delay before the second system sends back its acks. So, dealing with acks takes time, as each system gets its turn. This would also add significant delay to any retransmissions needed.

There is a calculation of the use efficiency, based on time to transfer messages and acks (this analysis can only be made in multiple message tests, because of the way data is recorded). The analysis shows that there is minimal data transfer overhead for handling multiple messages.

This suggests that multicast is well worth doing. You can transfer data to multiple nodes without significant data transfer overhead, noting that each system needs to get a turn to transmit acks. It also means that message transfer to each recipient is low latency, as there is no need to transmit to each node in turn.

The ACP 142 level works well. It would be good to reduce turnaround times to something closer to that achieved with ARQ, and STANAG 4406 edition 3 promises to achieve this.

CFTP

CFTP (Compressed File Transfer) is a mechanism to transfer Internet Mail, specified in Annex F of STANAG 5066. It is also known as Battle Force Email (BFEM). RapidM provide this as part of their RC66 suite, which also includes their STANAG 5066 Server. This provides access with POP and SMTP, so can be used directly with Isode test tools.

Because this is treated as a black box, message analysis is done at the submitted message level. This can lead to throughput of greater that 100% due to compression. Messages transferred by CFTP did not always arrive in order, so it was not helpful to separate out “first message” analysis.

The following data for 1200, 9600 and 75 gives direct comparison with the equivalent numbers taken from the CO-ACP 142 tests. For the Deltas, the positive numbers are where CO-ACP 142 is better, and the negative numbers are where CFTP is better.

1200 Bits Per Second Table

Size of Payload (kBytes)	0			1			10		100
Number of Messages	1	1-	100	1	10	100	1	10	1	10
CFTP Utilization	15%	44%	62%	52%	785	101%	89%	109%	113%	114%
CFTP Time	17	59	417	24	160	1235	105	861	799	7899
CO-ACP 142 Utilization	12%	46%	74%	41%	84%	105%	96%	112%	114%	113%
CO-ACP 142 Time	21	56	346	30	148	1182	97	837	796	7993
Delta Utilization	-3%	2%	13%	-10%	6%	5%	7%	3%	0%	-1%
Delta Time	-4	3	71	-6	12	53	8	24	3	-94

9600 Bits Per Second Table

Size of Payload (kBytes)	0		10		100
Number of Messages	1	10	1	10	1	10
CFTP Utilization	2%	9%	32%	87%	82%	97%
CFTP Time	18	35	37	135	137	1165
CO-ACP 142 Utilization	2%	15%	49%	90%	89%	96%
CO-ACP 142 Time	18	21	24	130	127	1182
Delta Utilization	0%	6%	17%	3%	6%	-1%
Delta Time	0	14	13	5	10	-17

75 Bits Per Second Table

Size of Payload (kBytes)	0		1		10
Number of Messages	1	10	1	10	1
CFTP Utilization	34%	55%	57%	81%	94%
CFTP Time	121	750	350	2450	1593
CO-ACP 142 Utilization	45%	67%	78%	95%	91%
CO-ACP 142 Time	91	619	254	2102	1647
Delta Utilization	11%	12%	22%	14%	-3%
Delta Time	30	131	96	348	-54

 

Looking at these numbers show that the performance is broadly very similar, particularly for large messages. This is not surprising as both protocols use a BSMTP format with DEFLATE compression running over RCOP.

Our interpretation of these results is that CO-ACP 142 has slightly better performance, which can be seen most clearly in the tests with large numbers of small messages. This reflects a lower per message overhead. As noted earlier, the values obtained on individual runs do vary quite significantly. We believe that this affected runs of both protocols. We see that CO-ACP 142 performs better in most tests. Where it does not, it is either a short test with a small time difference or a longer test with a small percentage difference. This ties in with a theoretical protocol analysis, that suggests that CO-ACP 142 will perform slightly better.

HMTP

HMTP (HF Mail Transfer Protocol) is also specified in Annex F of STANAG 5066. The RapidM RC66 supports HMTP, and the same test setup was used as for CFTP. Positive values of the delta are where CO-ACP 142 performs better.

1200 Bits Per Second Table

Size of Payload (kBytes)	0			1			10		100
Number of Messages	1	10	100	1	10	100	1	10	1	10
HMTP Utilization	6%	11%	14%	23%	33%	42%	73%	68%	83%	81%
HMTP Total Time	46	234	1901	53	373	2942	128	1379	1090	11114
CO-ACP 142 Utilization	12%	46%	74%	41%	84%	105%	96%	112%	114%	113%
CO-ACP 142 Total Time	21	56	346	30	148	1182	97	837	796	7993
Utilization Delta	7%	35%	61%	18%	51%	63%	23%	44%	31%	32%
Time Delta	25	178	1555	23	225	1760	31	542	294	3121

9600 Bits Per Second Table

Size ofPayload (kBytes)	0		10		100
Number of Messages	1	10	1	10	1	10
HMTP Utilization	2%	2%	20%	30%	43%	45%
HMTP Total Time	18	164	58	386	262	2528
CO-ACP 142 Utilization	2%	15%	49%	90%	89%	96%
CO-ACP 142 Total Time	18	21	24	130	127	1182
Utilization Delta	0%	13%	29%	60%	46%	51%
Time Delta	0	143	34	256	135	1346

75 Bits Per Second Table

Size of Payload (kBytes)	0		1		10
Number of Messages	1	10	1	10	1
HMTP Utilization	11%	28%	37%	48%	72%
HMTP Total Time	369	1476	533	4165	2069
CO-ACP 142 Utilization	45%	67%	78%	95%	91%
CO-ACP 142 Total Time	91	619	254	2102	1647
Utilization Delta	34%	39%	41%	47%	19%
Time Delta	278	857	279	2063	422

 

HMTP performance is poor. One factor is that HMTP does not provide compression. The tests were favourable to HMTP, as the user data (random binary) will get minimal compression. However, the 7bit MIME encoding does have a significant effect. Even allowing for this HMTP performance is poor, particularly for multiple small messages.

Analysis

Impact of “Real HF” and Errors

Apart from HMTP, all of the messaging protocols analysed perform well over STANAG 5066. These are lab tests, and it would be good to repeat them in an operational situation with HF connections that have errors.

We believe that the RCOP based protocols (CO-ACP 142 and CFTP) will deal efficiently with error which will be handled by STANAG 5066 ARQ, that will effectively insulate the applications from HF errors.

ACP 142 deals efficiently with errors at the application level, which showed up clearly in some test runs. Errors in a system with the current collision avoidance algorithms will lead to significant additional delays. STANAG 5066 ed 3 promises to address this.

Forward Error Correction in ACP 142

ACP 142 provides an option to use application level forward error correction (FEC), although unfortunately does not standardize any algorithms to use. This would enable the application to add additional FEC packets, so that message transfer would be tolerant to some packet loss.

The the low latency small transfer and multicast benefits of ACP 142 are highly desirable characteristics, which will be important for many deployments. This FEC capability could usefully help address the negative impact of packet loss by providing a configurable level or application resilience to packet loss.

Which Messaging Protocol to Chose

Apart from not using HMTP, it is useful to conclude with some guidance on choice of protocol. Some deployment requirements will force a choice:

1. If there is a requirement to support multicast or Emcon (Emission Control) then ACP 142 should be used.
2. If there is a need to support STANAG 4406 then ACP 142 or CO-ACP 142 should be used.

For other situations, where point to point transfer of Internet mail is the only requirement, both CFTP and CO-ACP 142 offer reasonable performance.

CFTP is a standardized protocol, and this is inherently desirable.

CO-ACP 142 is technically a better protocol, and the following points are of note:

• Performance is slightly better, borne out by measurements here.
• Support of 8 bit message formats (BINARYMIME and 8BITMIME). This is inherently desirable for a modern internet messaging infrastructure, and will also gain about 4% performance improvement for large messages with attachments.
• Support of per message notification requests, and in particular the ability to request a delivery report for a message.
• Support for Message Priority, to allow higher priority messages to overtake lower priority messages.
• Extensible compression will allow future adoption of optimized compression techniques.

For more information on this comparison see the Isode white paper Messaging Protocols for HF Radio.

Conclusions

This paper has measured the performance of four messaging protocols designed to operate over HF Radio, and has found that three of them work well: CFTP; ACP 142; CO-ACP 142.