Apple's iPhone & Making Mobile Email work well using IMAP
Summary: Apple's iPhone has an excellent email client which forms an important part of the iPhone package. Unlike Blackberry, the iPhone uses IMAP (Internet Message Access Protocol) to access messages. The iPhone's sophisticated use of IMAP (in contrast to other widely available IMAP clients) is one reason that the iPhone email client works so well.
Isode believes that Open Standards in general and IMAP in particular are essential to a good mobile email experience and we've outlined the benefits of IMAP for mobile email in a series of whitepapers available from the Isode website. This whitepaper looks at how iPhone uses IMAP to meet mobile messaging requirements, contrasts that with other email clients and looks at other IMAP capabilities that could be used to support mobile email.
Share this whitepaper
Mobile Email Requirements
There are two primary hardware platforms used for handling email on the move:
- A "smart phone" sized device, using an email client usually developed specifically for that platform.
- A laptop, using the same email clients as found on desktop computers.
The requirements and approaches discussed in this whitepaper apply both to desktop email clients and to "smart phone" email clients.
This paper does not consider situations where WiFi and wired Broadband internet connections are available but is instead focused on situations where mobile phone networks (2G and 3G) are used.
The data rates and latency characteristics of modern mobile phone networks mean that the transfer of a few kBytes of data rarely cause performance issues that would impact the user experience, leading to the conclusion that there is no major requirement for optimization of data volumes at this level of granularity.
However, for data volumes of the order of 100 kBytes or 1Mbyte, the way that a typical mobile network handles data will adversely affect overall performance. Although an increasing number of data networks provide unrestricted volumes nationally such provision is not universal, especially in developing nations and international roaming costs remain a factor for all markets.
IMAP Clients Examined
Apart from specialized email clients such as Mulberry, most email clients supporting IMAP make quite basic use of the protocol. This includes: Microsoft Outlook; Thunderbird; the Sony Ericsson UIQ email client (used here as an example client running on the Symbian platform) and the 'Modest' email client used by Nokia's Internet Tablet devices.
The Windows Mobile client's handling of IMAP is somewhat better, although not as good as the iPhone. This paper looks at how the clients listed use IMAP. We are not aware of any mainstream clients that have significantly better IMAP usage than the examples given.
As we have already established, dealing with large messages is an important issue. While most messages are small, many email users regularly receive a small number of larger (often multi MByte) messages. When operating over a mobile network, it's important that the user has control over how these messages are treated.
Microsoft Outlook presents the user with a number of controls for handling large messages (as in the screenshot below).
The key control is "download headers only" versus "download complete item including attachments". For mobile use, "download header only" needs to be selected in order to prevent automatic download of large messages.
With 'download header only' selected, when Outlook connects over IMAP it will download the message headers, clearly indicating which messages have been downloaded, as well as key message summary information including message size.
The user then has two strategies for reading mail:
- Select messages as desired and they are downloaded "on demand".
- Select all messages to be downloaded, download them, and then go offline.
The first strategy works well for LAN and WiFi but less well for mobile use as:
- If a large message is accidentally selected, there is no way to cancel the download.
- Downloading individual messages can cause the UI to become "sticky" as IMAP commands are issued for each download, and delays can occur due to network latency and gaps in network availability.
In practice, the second strategy works much better for using Outlook on a mobile link. This is all workable, but far from ideal. It is also sometimes hard to decide from the message listing whether or not to download.
Thunderbird also provides a mechanism for selecting messages (or folders) for offline use. In addition Thunderbird uses IMAP's ability to download large messages in 'chunks' to enable the user to cancel the download of messages before download is complete.
The Sony Ericsson UIQ email client downloads message headers, to give the user a listing, but does not show message size. Thus the user does not know how large a message is before selecting to download it. When the user requests that the message be downloaded, it will be downloaded in full. The user can configure a maximum size of message to download (default 100 kByte).
Modest presents a list of message headers, together with message sizes. An option exists to restrict the maximum message size to download, but this option does not appear to be active. When a message is selected for download, it is always downloaded in full.
Apple’s iPhone provides an approach to handling large messages that does not need configuration or user intervention. The first basic strategy is to fully download all messages of 20kbytes or less (60 kBytes if connected by WiFi). This means that for most users, the vast majority of messages are simply downloaded and available locally.
For larger messages, a technique referred to as "email metadata analysis" is used. A message contains a number of attachments, which can include other messages that in turn will have attachments. First the core message, including the basic message text is downloaded. Then IMAP is used to determine the message structure. This means that the email client can display the core message and give a summary and list of all attachments (including name, type and size). When the iPhone is connected over WiFi, attachments are automatically downloaded if less than 1 Mbbyte, as shown below.
The user is prompted to download attachments larger than 1MByte in size:
Like Thunderbird, the iPhone email client uses IMAP to download the data in chunks, which means that the user can cancel the download.
From a user perspective, the iPhone email client "does the right thing" and does not need special configuration to cope with different usage scenarios.
Windows Mobile 6.1 has a similar strategy to iPhone, although the details are less sophisticated (in particular it does not adapt to different connectivity speeds) and needs user configuration. The user can configure how much of a message to download (header only; full message; up to a certain size). If part of the message is downloaded, the user is prompted to download the rest. The user can also configure whether or not to download attachments, including configuring a maximum size of attachments to download. Attachments are shown to the user with type and size, so an intelligent decision can be made about whether or not to download a message.
Handling Many folders
A key advantage of the IMAP protocol, particularly for small mobile devices, is that personal and shared folders can be held on the server and accessed when needed. Organizations and users that take advantage of this will often end up with large numbers of folders on the server. Because of this, a good IMAP client should support configurations with many folders.
Some IMAP clients will synchronize the full folder list each time they connect. This approach makes the client unusable over a mobile connection. Clients that suffer from this problem include the Sony Ericsson UIQ client and the Windows Mobile 6.1 client.
The iPhone client, Modest, Outlook and Thunderbird only synchronize on demand, and so provide reasonable support for large numbers of folders.
By default Outlook enables the option “get folder unread count for subscribed messages” something that, over a mobile link, could be quite expensive in terms of IMAP protocol and network use.
Handling Large Folders
High data volumes can arise from dealing with large numbers of messages in a folder, as well as from large individual messages or a large numbers of folders. Although many users keep inboxes small, there are many reasons that large folders occur:
A "Google style" approach to email encouraging users to leave messages in place and to locate by search.
- Some users delete by marking (a useful IMAP technique), so that deleted messages can be searched if needed. This leads to folders that appear small, but actually contain lots of deleted messages.
- Folders other than inbox are often naturally large, especially where they are used to store a 'history' of communication about a subject or with an organization/individual.
Where an email (IMAP) client accesses a folder, there is a need to perform a "Folder Synchronization". The reason for this is that other clients may access the same folder, and so any cached state that the client has may be incorrect. For each message in the folder on the server being synchronized, there needs to be one of two actions:
- If the client has cached information, the client needs to check the message flags of the message on the server to verify that the cache is correct and update flags if needed.
- If the client does not have cached information, it needs to download appropriate per-message information as described earlier.
The second action requires quite a lot more processing per message. A smartphone client is much less likely than a desktop client to have a cached copy (because of lower storage capacity on the device) and so this second case is more likely scenario.
A basic strategy is to synchronize the whole folder. This is what Outlook and Thunderbird do. This is a reasonable standard approach for a desktop client, but it will have a performance cost for larger folders (1,000 messages or more), particularly over a mobile link.
The Sony Ericsson UIQ client and Windows Mobile 6.1 client allow a choice between whole folder sync, and a configurable number of the most recent messages. The user has the choice of taking the performance hit of a potentially very large sync, or restricting access to only the most recent messages. Neither is ideal.
Modest also allows a choice between whole folder sync and a configurable number of the most recent messages. When there are more messages, it will ask the user if the rest should be downloaded.
Extended IMAP and LEMONADE
Although the iPhone makes very good use of IMAP, there is more that can be done, in particular by use of the LEMONADE extensions to IMAP. Although these extensions are not supported by all IMAP servers, IMAP clients can still use them effectively without restricting which servers can be supported. This is because capabilities are negotiated for each connection, and so the client will be aware when a server does not support a specific advanced capability.
The iPhone approach to large messages is excellent, but two more things could be done:
- The CONVERT (RFC 5259) capability allows the client to request a server to convert a body part. This would be useful to facilitate efficient preview. For example a large photograph could be converted to a much smaller (phone sized) preview image.
- The self descriptive LEMONADE Forward without Download capability could be used to send a large message to an alternate recipient, for example to someone back in the office for review.
Full folder synchronization gives a real performance overhead for large folders. The iPhone approach is reasonable for a smart phone, but not appropriate for a mobile laptop.
Where a mobile laptop is the primary client, full synchronization is desirable. As noted, the basic synchronization done by most IMAP clients requires the flags of every message to be checked on connection. This has a high overhead for large folders, which is a particular nuisance for mobile operation. For the common case where the client is reasonably well synchronized to the server (ie., a small percentage of messages in the folder have changed since the last connection), substantial optimization is possible by use of IMAP CONDSTORE (RFC 4551) and IMAP Quick Mailbox Resynchronization (RFC 5162). Modest takes advantage of both of these extensions.
An alternate solution that is particularly useful for infrequently accessed folders is to synchronize on demand, so that in effect synchronization is only done for messages when the listing is displayed to the user (“synchronize on view”). The IMAP ESEARCH (RFC 4731) facilitates this style of operation.
We've seen in this whitepaper that, in a number of common mobile email usage scenarios, the iPhone makes superior use of IMAP to provide a good user experience. IMAP client authors would benefit from examining the approach to mobile email that Apple have adopted.
We have also seen that there is still scope for improvement in the way that email clients handle IMAP in the mobile environment, in particular the adoption of some LEMONADE features could provide real competitive advantages via an improved user experience.