InfiniTec - Henning Krauses Blog

Don't adjust your mind - it's reality that is malfunctioning

Getting the body of an Email with a FindItems request

The FindItem operation (or the corresponding ExchangeService.FindItems method) does not return the body of an email by default. And when trying to explicitly request them via a custom propertyset, the call fails. Consider this method, which uses the EWS Managed API to execute a FindItems method to get every message from the inbox folder of a mailbox, fetching only the item id and the body:

   1: private static void GetAllItems(ExchangeService exchangeService)
   2: {
   3:     var offset = 0;
   4:     const int pageSize = 100;
   5:  
   6:     FindItemsResults<Item> result;
   7:     do
   8:     {
   9:         var view = new ItemView(pageSize, offset)
  10:                        {
  11:                            PropertySet = new PropertySet(BasePropertySet.IdOnly)
  12:                                              {
  13:                                                  ItemSchema.Body
  14:                                              }
  15:                        };
  16:  
  17:         result = exchangeService.FindItems(WellKnownFolderName.Inbox, view);
  18:  
  19:         foreach (var item in result)
  20:         {
  21:             ProcessItem(item);
  22:         }
  23:         offset += pageSize;
  24:     } while (result.MoreAvailable);
  25: }

When executed, the ExchangeService instance throws a ServiceValidationException stating “The property Body cannot be used in FindItem requests”. The official workaround proposed by Microsoft is to use a FindItem request to get the item ids of the items in a folder and afterward issue a GetItem request containing all the item ids and request the body property. However, there is another solution: The body properties can be fetched by requesting the MAPI properties containing the body:

   1: private static ExtendedPropertyDefinition TextBodyProperty = new ExtendedPropertyDefinition(0x1000, MapiPropertyType.String);
   2: private static ExtendedPropertyDefinition HtmlBodyProperty = new ExtendedPropertyDefinition(0x1013, MapiPropertyType.Binary);

The first property definition can be used to fetch the text body of a mail. The second one fetches the Html body. The new GetAllItemsMethod now looks like this:

   1: private static void GetAllItems(ExchangeService exchangeService)
   2: {
   3:     var offset = 0;
   4:     const int pageSize = 100;
   5:  
   6:     FindItemsResults<Item> result;
   7:     do
   8:     {
   9:         var view = new ItemView(pageSize, offset)
  10:                        {
  11:                            PropertySet = new PropertySet(BasePropertySet.IdOnly)
  12:                                              {
  13:                                                  HtmlBodyProperty
  14:                                              }
  15:                        };
  16:  
  17:         result = exchangeService.FindItems(WellKnownFolderName.Inbox, view);
  18:  
  19:         foreach (var item in result)
  20:         {
  21:             object body;
  22:             if (item.ExtendedProperties.TryGetValue(HtmlBodyProperty, out body))
  23:             {
  24:                 Console.Out.WriteLine("item.Body = {0}", Encoding.UTF8.GetString(Convert.FromBase64String((string)body)));
  25:             }
  26:         }
  27:         offset += pageSize;
  28:     } while (result.MoreAvailable);
  29: }

The HtmlBody is requested with the addtion of the HtmlBodyProperty to the ItemView in line 13. Since the Html body is stored in binary form and returned in Base64 encoded format, it needs to be decoded before it can be displayed. This is done in line 24. If the plaintext body is requested, the value of the body property in line 24 can simply be converted to a string.


Posted by Henning Krause on Tuesday, June 9, 2009 9:00 PM, last modified on Wednesday, April 20, 2011 6:19 PM
Permalink | Post RSSRSS comment feed

FindItems and SyncFolderItems performance

By default, Exchange returns a rather large set of properties with each item during a FindItem or SyncFolderItems request. If the query returns a large set of items, this slows down the entire process: On one hand, Exchange needs to get all properties from the store and secondly, all the data needs to be sent over the wire to the calling application. The calling application can specify which items to fetch for a FindItem or SyncFolderItems request by specifying it in the PropertySet parameter. The default property set looks like this:

   1: var PropertySet = new PropertySet(BasePropertySet.FirstClassProperties)

To request only the property id of the items, use this declaration instead:

   1: var PropertySet = new PropertySet(BasePropertySet.IdOnly)

I’ve measured the FindItems call with the Exchange server at my workplace from home. So the call went over the Internet using a 2 MBit connection (on both sides). The inbox folder of my mailbox currently contains approximately 4500 items and I have repeated the measurement two times for each of the propertysets. When querying only for the id of the items, the entire process took 46 seconds to complete. On the other hand, when querying the default set of properties, the whole process took between 1:37 minutes and 1:41 minutes to complete.

But item id is seldom the only property needed. Luckily, the additional properties can be specified on the property set:

   1: var PropertySet = new PropertySet(BasePropertySet.IdOnly)
   2:                   {
   3:                      ItemSchema.Subject,
   4:                      ItemSchema.DateTimeReceived
   5:                   }

Executing this query on my mailbox, took about 50 seconds to complete. This is a significant improvement of the default property set.


Posted by Henning Krause on Sunday, June 7, 2009 12:56 PM, last modified on Saturday, November 27, 2010 6:35 AM
Permalink | Post RSSRSS comment feed

Processing items in an Exchange folder using EWS Managed API

A requirement I often hear is something like this: “We have a mailbox that receives emails of some kind that need to be processed somehow”. What options are there to fulfill the requirement? As always, it depends. When the items need to be processed as soon as possible and the client processing the items can be directly reached by the Exchange server, push notifications are certainly a good choice. If, on the other hand, item processing is not time critical a suitable method of processing those items is polling. It’s much simpler to use than push notifications. A simple example might look like this:

   1: private static void ProcessItems(ExchangeService exchangeService)
   2: {
   3:     var offset = 0;
   4:     const int pageSize = 100;
   6:     FindItemsResults<Item> result;
   7:     do
   8:     {
   9:         var view = new ItemView(pageSize, offset);
   5:  
  10:  
  11:         result = exchangeService.FindItems(WellKnownFolderName.Inbox, view);
  12:  
  13:         foreach (var item in result)
  14:         {
  15:             ProcessItem(item);
  16:         }
  17:         offset += pageSize;
  18:     } while (result.MoreAvailable);
  19: }

This is a very naïve implementation as it always returns every item from the inbox folder of the mailbox. But at least it uses paging. This breaks the processing down from one very large request into many small requests (in this case 100 items are returned per request).

This method is suitable if you delete the processed items from a mailbox after they are processed. Of course, the items should not be removed until the every item has been processed. Otherwise items may be skipped since the offsets of the individual items change when an item is removed from the folder.

If the items are not removed from the store, the calling application must distinguish new items from items already processed. An obvious way to do this is to mark each item once it has been processed by marking it as read. This changes the requirement for the ProcessItems method: It should only process unread items. This modifications has been incorporated into the following example:

   1: private static void ProcessItems(ExchangeService exchangeService)
   2: {
   3:     var offset = 0;
   4:     const int pageSize = 100;
   5:  
   6:     FindItemsResults<Item> result;
   7:     do
   8:     {
   9:         var view = new ItemView(pageSize, offset)
  10:         {
  11:             SearchFilter = new SearchFilter.IsEqualTo(EmailMessageSchema.IsRead, false)
  12:         };
  13:  
  14:         result = exchangeService.FindItems(WellKnownFolderName.Inbox, view);
  15:  
  16:         foreach (var item in result)
  17:         {
  18:             ProcessItem(item);
  19:         }
  20:         offset += pageSize;
  21:     } while (result.MoreAvailable);
  22: }

The only change in this example is in line 11: A search filter has been added to filter for unread items. What’s missing here is the modification of the Unread status of each message. And that is a considerable drawback of the whole solution. To touch each processed item, the UpdateItems method must be called. It is sufficient to call the UpdateItems method and pass all item ids from the last resultset to the method. But this adds a significant burden on the Exchange server and slows overall processing down. Furthermore, if someone accesses the mailbox (either with Outlook or Outlook Web Access) and accidently marks one or more items as read, those items will not be processed by the client application.

Next idea: Find all items that were received after the last time the mailbox was checked. A ProcessItems method that implements this behavior might look like this:

   1: private static void ProcessItems(ExchangeService exchangeService, DateTime lastCheck)
   2: {
   3:     var offset = 0;
   4:     const int pageSize = 100;
   5:  
   6:     FindItemsResults<Item> result;
   7:     do
   8:     {
   9:         var view = new ItemView(pageSize, offset)
  10:                        {
  11:                            SearchFilter = new SearchFilter.IsGreaterThanOrEqualTo(ItemSchema.DateTimeReceived, lastCheck)
  12:                        };
  13:  
  14:         result = exchangeService.FindItems(WellKnownFolderName.Inbox, view);
  15:  
  16:         foreach (var item in result)
  17:         {
  18:             ProcessItem(item);
  19:         }
  20:         offset += pageSize;
  21:     } while (result.MoreAvailable);
  22: }

As with the second example, the difference to the first one is line 11. A search filter has been added that restricts the FindItems call to those items received after a specific time. This method removes the requirement to mark each processed item on the server. But it adds another caveat: A mail that is received during a ProcessItems call will be missed if no additional checks are performed. This can be a little tricky.

Luckily, the Exchange WebServices offer a more suitable solution for the whole requirement: The SyncFolderItems method. This method not only solves the problems mentioned above but also returns deleted items, if such a processing is necessary. A ProcessItems method that uses this API now looks like this:

   1: private static string SyncItems(ExchangeService exchangeService, string syncState)
   2: {
   3:     const int pageSize = 100;
   4:  
   5:     ChangeCollection<ItemChange> changeCollection;
   6:     do
   7:     {
   8:         changeCollection = exchangeService.SyncFolderItems(new FolderId(WellKnownFolderName.Inbox),
   9:                                                            new PropertySet(BasePropertySet.FirstClassProperties), null, pageSize,
  10:                                                            SyncFolderItemsScope.NormalItems, syncState);
  11:  
  12:         foreach (var change in changeCollection)
  13:         {
  14:             if (change.ChangeType == ChangeType.Create)
  15:             {
  16:                 ProcessItem(change.Item);
  17:             }
  18:         }
  19:         syncState = changeCollection.SyncState;
  20:  
  21:     } while (changeCollection.MoreChangesAvailable);
  22:     return syncState;
  23: }

This method only processes newly created items and ignores all other item changes or deletions. The application calling this method needs to store the synchronization state between each call. If an empty sycnState is provided, Exchange will return every item from the mailbox as “Created” item. This makes it possible to process all existing items once and then, with the same logic, every changed item. The only drawback with this method is the fact that the synchronization state can become quite big (in a folder with ~4500 items, the syncstate has a size of approx. 60kb).


Posted by Henning Krause on Sunday, June 7, 2009 12:54 PM, last modified on Monday, November 29, 2010 9:21 PM
Permalink | Post RSSRSS comment feed