Messaging and Directory Server software used around the world in the Government, Military, Aviation and Commercial sectors.

isode.com / whitepapers /

IMAP IDLE: The best approach for 'push' email.

27th June 2006

The concept of "push email" has been widely marketed as a desirable feature of mobile email services, to enable users to get immediate notification of and access to new messages. This paper looks at various approaches to meeting user requirements, and concludes that the Internet Standard IMAP (Internet Message Access Protocol) IDLE command is the best way to achieve this service.

 

What are the User Requirements?

"Push Email" is an unfortunate choice of terminology, as it implies a specific technical solution to a more generic user requirement. The primary user requirement described by "push email" is:

"Whenever a new message is delivered to my mailbox, I want to be notified 'immediately' that the message has arrived."

'Immediately' in the context of store and forward messaging would typically be interpreted as "a few seconds", rather than hours and minutes. There are some associated secondary requirements that should be addressed by a good solution:

1. The user should have control over message download to the mobile device. A typical choice would be to automatically download small messages, and to download larger messages under user control.
2. The user should be able to turn off notifications when desired.
3. The user should have control over which messages lead to notifications (e.g., only messages from the boss!).

 

Polling is a Poor Solution

The basic approach used by many email devices is to connect to the server to access new messages. This is a good model for many uses of mobile email, where access to email is under user control – the user checks email when the user wants to.

In order achieve automatic notification of new messages, a simple approach is to use 'polling' where the mobile client automatically connects to the server at intervals to check for new messages. However, there are two main problems with this approach:

1. Frequent polling is an inefficient use of network and mobile device resources, increasing the cost to the user.
2. New mail notification is only as frequent at the polling, and not 'immediate'.

Polling is a poor solution for a user needing immediate notification.

What is IMAP IDLE?

IMAP (Internet Message Access Protocol) is the best open standard for accessing mobile email. It handles immediate notification as part of general operations and by the IDLE command, which is a widely implemented standard extension to the core IMAP protocol.

How IDLE Works

IMAP works by the software on the mobile device (the client) issuing commands to the server. An IMAP server provides two things in response to a client command:

1. An answer to the request.
2. Information on any new messages.

This means that where a client is actively doing things with an IMAP server, it will be told immediately about new messages. The client can then get summary information on the message to present to the user, and can (automatically) download the message when appropriate.

This means that an active client will always be kept up to date. The IDLE command deals with the situation where the client has no more requests to make. The server responds to the idle command when there is a new message (or messages) which indicates to the client that there are new messages.

When the user is inactive, and does not wish to receive notifications, the client simply stops using IDLE, which is very efficient.

Performance

The basic network use of the IDLE command is very small, and so it makes very efficient use of bandwidth. In practice things are made more complex by the problem of timeouts occuring when there is no activity keeping the connection open. The main timeouts that will occur are:

1. IMAP server timeout: Typically occurs after 30 minutes with no activity.
2. NAT Gateway timeout: Most mobile devices access the Internet through a device operated by the mobile service provider called a NAT (Network Address Translation) gateway. These will typically time out an idle connection after 15 minutes.

The solution to this is for the IMAP client to issue a NOOP (No Operation) command at intervals, typically every 15 minutes. This will exchange a few bytes of data, and keep everything active. The impact of holding an IMAP connection open on the client, server and intermediate components should be considered:

• IMAP Server. A good IMAP server will have minimal overhead for an Idle connection, and should be able to support 10’s or 100’s of thousands of connections.
• IP Routers and other network components. Negligible impact.
• Phone. For older phones there could be an issue of increased battery usage due to holding the connection open. This is unlikely to be a problem on a modern phone.

Another practical problem is that current phone networking technology will lose IP network connectivity from time to time, and this will need to be automatically re-established, and the IMAP connection re-established if this is lost due to a long network failure.

In summary, the overall IMAP IDLE architecture has good performance.

 

'True' Push

An alternative to the IMAP IDLE approach is for a mechanism whereby the server pushes something to the client when a new message arrives, without there being an open connection from the client to the server. This section looks at this approach.

Alert vs Message Send

There are two variants of the 'true' push approach:

1. Push the new message.
2. Push a short generic message alert, prompting the client to connect to the server in order to retrieve the message.

Using approach 1 leads to three problems:

• The mechanism will need to deal with security and data confidentiality, which leads to a lot of additional complexity.
• The data being pushed becomes larger, which reduces the options for sending the data (e.g., SMS could not be used).
• There is no client control on the choice to download.

For these reasons, the second approach is generally better, and this is the one considered here.

Implementation Options

A clean way to send data from the server to the mobile device would be for the server to open a TCP connection. This would give a lot of flexibility in protocol choice and deployment. Unfortunately, this is impractical because most mobile devices do not have registered IP addresses to which a server can connect. They are also generally connected through a NAT gateway that will prevent connections being made to the phone. This means that use of a TCP connection is not generally a viable option.

This means that another mechanism needs to be used to do the 'push'. There are various options to do this. SMS is a good candidate, as it is widely supported as a data listening mechanism on most mobile devices. SMS is used as an example interconnect mechanism in this paper. The use of SMS as the mechanism to carry messaging alerts leads to two integration problems:

1. Phone. SMS is a general purpose service, not specific to email. There are two integration approaches:
   1. Use a "you’ve got mail" message to the human user, who will then connect with the email application. This crude approach would only be suitable for very basic use.
   2. Standardize how SMS is used, so that phones can detect email notifications and pass them to the email client for automatic processing.
2. Messaging server. There are two deployment scenarios:
   1. Messaging server deployed by a mobile operator. In this scenario integration with SMS is straightforward.
   2. Messaging server deployed by the end organization or independent service provider. Typically such a deployment will rely on Internet access. Integration with SMS would provide both technical problems (how to make it happen) and commercial problems (who pays for the SMS message, and how to prevent abuse).

These problems are not insurmountable, but will be a barrier to widespread adoption.

Performance

The response time and data use of this push approach are contrasted to IMAP IDLE,
A messaging server offering both approaches would be able to send the push notification and IDLE response at the same time. The IDLE response is immediate, and it will initiate the client to deal with the new messages. The push notification will have two delays:

1. Time for the SMS message to reach the phone. This may be a few seconds, but could take longer.
2. Time for the client to connect to the server. This will typically be a few seconds.
   True push will be somewhat slower than IMAP IDLE, but in practice this is not likely to be a big problem.

Data usage for IMAP IDLE is essentially the 15 minute NOOP to keep the connection alive, plus a small amount of data to do the notification. The true push will have the cost of the SMS notification. The data for connection establishment is more significant, typically including TCP Connection; TLS (for data confidentiality); client authentication; client (re) synchronization.

It can be seen that for frequent message arrival, that IMAP IDLE is more efficient and that for longer intervals between notifications that true push has better data efficiency. The details will depend on many parameters. A rough calculation suggests that a typical break-even point would be around two days. This suggests that for a typical user receiving and getting notifications for 10 messages per day, that IMAP IDLE has significantly better data performance.

When the user does not want to receive notifications, there is a need to change server configuration (which causes extra complexity and network activity).

 

Conclusions

IMAP + IMAP IDLE is a good approach for providing the immediate email notification and delivery service of "push email". It has substantial implementation, deployment and performance advantages over a "true push" approach.