Messaging and Directory Server software used around the world in the Government, Military, Aviation and Commercial sectors.

isode.com / whitepapers /

Directory Replication by Email and over 'Air Gap'

Published on: 03 September 2008

Purpose

Directory replication is an important feature of most directory services, and is commonly achieved by use of directory protocols as discussed in Replicating and Synchronizing Data Between Directory Servers. There are a number of situations where using directory protocols to perform replication does not work well. These include:

• HF Radio and other constrained links.
• System boundaries, where there is only email traffic allowed.
• Directory gateways performing security checks.
• Tactical directories with irregular network access.

This paper looks at these scenarios, and shows how directory replication over email and 'air gap' can address them. Then, the architecture and key features of Isode's solution are described.

Directory Replication by Email Overview

The diagram above shows a high level overview of Isode's approach to performing efficient directory replication using email. Directory data is replicated from one directory server (the supplier) to another (the consumer). Isode's Sodium Sync product will access the supplier directory servers using LDAP or X.500 DAP, and determine changes made since the last update. These changes are written out to an LDIF (LDAP Data Interchange Format) file. Sodium Sync will then attach the file to an email and send it to one or more recipients. At the consumer directory server end, another Sodium Sync will extract the LDIF file from the email. It will then check that the update is the 'next in sequence' and apply the changes to the consumer directory server.

It can be seen that this provides a flexible mechanism for directory replication, and that the volume of data to be transferred is minimized.

Scenarios

Directory replication by email, and similar approaches using different transfer mechanisms are more complex than use of direct server to server replication such as that provided by X.500 DISP (Directory Information Shadowing Protocol). This section looks at scenarios where direct server to server replication is not possible or desirable.

HF Radio

HF Radio is an important communications technology for military communication. Directory is critical infrastructure, and so replication across the slow HF links is important. Direct server to server replication would be inefficient, because of the nature of HF Radio. It would not work when a receiver is in radio silence (EMCON - Emission Control), and does not take advantage of the broadcast nature of radio. It would be possible to define a new directory replication protocol working over STANAG 5066, but this would be complex.

Messaging can be efficiently supported over HF Radio, both Internet Mail and STANAG 4406 formal messaging (see the Isode whitepaper Messaging Protocols for HF Radio). The messaging infrastructure provides:

• Reliable Data Transfer
• Compression
• Support of multicast and EMCON
• Security - in particular Message Origin Authentication and Content Integrity

Use of email infrastructure to support directory replication is a natural fit.

Satellite

Although standard directory replication can be used over satellite, it does not take advantage of the multicast nature of satellite. The same messaging infrastructure used for HF Radio, but operating over IP rather than STANAG 5066, can be used to provide multicast directory replication. This can be important for supporting servers that have copies of the same data.

Email only Boundary

Security boundaries often restrict access and functionality available over the boundary. Many military networks provide secure email gateways, which may follow ACP 145, but do not permit directory protocols.

Where email is allowed, and directory protocols are prohibited, directory replication by email is a natural approach. The military ACP 137 specification "Griffin Directory Services Technical Architecture" defines a mechanism for directory replication by email that is specifically designed for this scenario.

Directory Gateway

A related scenario to the email only boundary is a security boundary where there are stringent security requirements on data flow. In this scenario, there are requirements to check replicated data for things such as:

• Leaks of sensitive information to a lower security environment.
• Checks for malware and security threats within the replicated data.

A system performing such checks would generally need to be security accredited, according to Common Criteria or similar measure. Use of 'air gap' replication, or a product such as the Tenix Data Diode [pdf] provides a straightforward way to achieve this.

Although the above architecture is quite complex, it gives a key benefit of making the Security Accredited process that checks the directory data a very simple one. Essentially it needs to perform security checks and copy data across the security boundary. It enables the bulk of the directory replication functionality to not require (expensive) security accreditation. In keeping the accredited process simple, it also increases the overall security of the system.

Tactical Directory

In a tactical environment, directory data is often replicated to many similar directory servers that operate on tactical units that may have irregular network access. Rather than having a supplier directory push updates to each server, email replication can be used to send updates to an email server. Then each tactical unit can retrieve these updates from the email server at a convenient time, and apply the updates locally. If the email server also holds "total updates", this approach can also be used to restore broken directory servers or to provision new ones. The real benefit of using email here is that it decouples the consumer directory server from the supplier ones, and allows multiple consumes to easily share a single supplier.

The Isode Solution

We now look at how Isode's Sodium Sync product provides the capabilities described, in order to address the above scenarios.

Architecture

The basic architecture of Isode's solution separates the directory synchronization component from the underlying transport. The reason for this is to enable this system to be used with a variety of transports, including:

• Internet email, using SMTP submission and transport with IMAP or POP delivery.
• X.400 (STANAG 4406) message transport.
• File copying for 'air gap' approaches.
• Custom transport.

To achieve this, Isode provides a clean separation between synchronization and transport, and allows use of Isode provided or customer provided transport mechanisms. This is shown below.

The diagram above shows how sending works. Sodium Sync will access the supplier directory using LDAP or X.500 DAP and determine if any changes have been made, by efficiently comparing data in the directory to a reference copy of the data. If changes have been made, Sodium Sync will generate an LDIF (change) file representing these changes, and place it into a file system directory. Then it will (optionally) run a 'transport process' that will cause the LDIF file to be transferred (e.g., sent by email) to one or more consumer systems. The transport process will log activity, and on successful operation (e.g., correct submission of messages to transfer) will rename the LDIF. This means that if there is a temporary failure, that the file can be picked up next time the transport process runs.

The diagram above illustrates the reverse process. Sodium Sync runs a consumer transport process which places LDIF files into a directory (e.g., by retrieving email messages using an access protocol and extracting the LDIF files). Sodium Sync then looks in the directory to find the 'next' LDIF change file, based on sequence numbering within the LDIF file name. The overall system needs to deal with:

• Duplicates (e.g., a message being sent twice. Message transfer has 'at least once' and not 'exactly once' semantics). The current release flags duplicates for operator attention. A future version will check that if duplicate updates (i.e., two with the same sequence number) are identical, and if this is the case will ignore the duplicate.
• Out of sequence. Messages may not be delivered in order. Sodium Sync will not process a change until the previous one has been processed.
• Missing updates. Sodium Sync will report this, and the operator will observe that processing is 'stuck'. A future version may automatically request resending an update.

Updates are simply applied to the consumer directory, and this keeps the consumer directory up to date.

Monitoring & Logging

Sodium Sync provides a monitoring view, so that an operator can check on progress. This is shown above and includes:

• All configured synchronization agreements and schedule.
• Time each sync was last run and success/failure status.
• Success/failure status of the transport process.

Sodium Sync provides logging of all activity, that may be used for both diagnostic and audit purposes.

Scheduling & Total Updates

Updates may be scheduled to occur at specific times each day, or at an interval (specified in minutes) since the previous update. An LDIF update is always provided, even if there are no changes. This allows a consumer to watch for scheduled updates, and enables it to quickly detect if a scheduled update is not sent. The first synchronization will contain all of the data (a total update).

Theoretically, this initial transfer is the only time that a total update is needed. In practice it is often desirable to use total updates from time to time. For example, a user or operator might delete an entry in the consumer directory. Sodium Sync allows for total updates to be sent at intervals. This is configured in terms of the number of scheduled incremental updates. This can also be used to send a single total update when needed.

Filtering and Mapping

Sodium Sync includes sophisticated capabilities for filtering, mapping and transforming information synchronized. These are all available for use with directory replication by email, as part of the supplier process. Two are of particular importance:

• Filtering information, such as removing specific attributes, or entries. This is important to minimize transfer of data.
• Moving information to a different part of the directory information tree.

Future Synchronization Work

The directory synchronization elements of directory synchronization by email are broadly complete in Isode R14.3. Future enhancements under consideration are:

1. Support of transports providing GUI configuration, enabling configuration from within Sodium Sync.
2. Support of the ACP 137 transfer format.
3. Optimized support for CRL (Certificate Revocation List) updates. CRLs are often large and frequently updated, but with the new CRL very similar to the old one. Sodium Sync could transfer delta information to enable CRL update without transfer of the whole CRL.
4. Option to not send an update when no changes have occurred.

Please contact Isode if any of these features are important to you.

Email transport

Sodium Sync provides a general transport mechanism. It is intended that this is for use both by Isode provided transports and by customer provided transports. In R14.3, Isode has provided a simple transport for use with SMTP and IMAP (implemented using a Perl script). This is suitable for:

• Testing and demonstration.
• Use in situations not requiring email security.
• A model for customers to provide transports.

Isode plans to provide further transport capabilities in future releases, including:

• SMTP and X.400 (STANAG 4406) email transport products.
• GUI configuration of email transports.
• Message Security based on digital signature. This will include Message Origin Authentication and Content Integrity, and may include Content Confidentiality.
• Message splitting, to handle very large updates.

Isode would like to receive input on product requirements for email transports.

Conclusions

Directory replication over email or air gap can be used to address a number of important specialized scenarios, in particular directory support over HF Radio, tactical directory, and various boundary replication scenarios. Isode's Sodium Sync product implements directory synchronization over email.