InfiniTec - Henning Krauses Blog

Don't adjust your mind - it's reality that is malfunctioning

The case of the hanging WCF service [Updated]

Recently a co-worker was complaining about a windows service which took ages to start. Because this exact code runs on a large number of servers I suspected the the fault to be on the machine rather than the code. The task manager yielded no interesting information (memory consumption was ok and so was the CPU usage). So I downloaded and installed the Windows Debugging tools, configured the symbol server and attached the debugger to the process in question. To enable debugging of managed applications, the SOS extension has to be loaded:

.loadby sos mscorwks

The next step was to look what all the threads are doing:

!eestack –ee

The !eestack command lists the stacktraces of all running threads and the –ee option restricts the output to managed methods. This makes the trace much more readable. Anyway, most threads where just sitting around doing nothing, but one got my attention:

   1:  Thread   6 
   2:  *** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v2.0.50727_64\System\65f46521e7fca2cd2d216162175f2fd6\System.ni.dll 
   3:  *** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v2.0.50727_64\System.ServiceModel\e000a1cd822ffb6f6483426a67622d75\System.ServiceModel.ni.dll 
   4:  *** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v2.0.50727_64\Netatwork.Common\73da8de102292417562adaf43872eec6\Netatwork.Common.ni.dll 
   5:  *** ERROR: Module load completed but symbols could not be loaded for C:\Windows\assembly\NativeImages_v2.0.50727_64\Netatwork.Common\73da8de102292417562adaf43872eec6\Netatwork.Common.ni.dll 
   6:  Child-SP         RetAddr          Call Site 
   7:  0000000001f9e900 000007fef1b64304 System_ni!DomainBoundILStubClass.IL_STUB(UInt32, System.Security.Cryptography.SafeLocalAllocHandle, System.Security.Cryptography.OidGroup)+0x77 
   8:  0000000001f9ea00 000007fef1b2b402 System_ni!System.Security.Cryptography.CAPI.c(UInt32, System.Security.Cryptography.SafeLocalAllocHandle, System.Security.Cryptography.OidGroup)+0x94 
   9:  0000000001f9eac0 000007fef1b2b28c System_ni!System.Security.Cryptography.X509Certificates.X509Utils.FindOidInfo(UInt32, System.String, System.Security.Cryptography.OidGroup)+0x152 
  10:  0000000001f9eba0 000007fef1b282ec System_ni!System.Security.Cryptography.Oid..ctor(System.String, System.Security.Cryptography.OidGroup, Boolean)+0x2c 
  11:  0000000001f9ebe0 000007fef1b27f03 System_ni!System.Security.Cryptography.X509Certificates.X509Certificate2.get_PublicKey()+0x8c 
  12:  0000000001f9ec40 000007feee6c5425 System_ni!System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()+0xb3 
  13:  0000000001f9eca0 000007feee6c538d System_ServiceModel_ni!System.ServiceModel.Security.SecurityUtils.EnsureCertificateCanDoKeyExchange(System.Security.Cryptography.X509Certificates.X509Certificate2)+0x55 
  14:  0000000001f9ed20 000007feee6c5349 System_ServiceModel_ni!System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateServerX509TokenProvider()+0x2d 
  15:  0000000001f9ed60 000007feee6c52e1 System_ServiceModel_ni!System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateLocalSecurityTokenProvider(System.ServiceModel.Security.Tokens.RecipientServiceModelSecurityTokenRequirement)+0x49 
  16:  0000000001f9edc0 000007feef00fcd9 System_ServiceModel_ni!System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateSecurityTokenProvider(System.IdentityModel.Selectors.SecurityTokenRequirement)+0x41 
  17:  0000000001f9ee10 000007feef010a0f System_ServiceModel_ni!(System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateTlsnegoServerX509TokenProvider(System.ServiceModel.Security.Tokens.RecipientServiceModelSecurityTokenRequirement)+0xd9 
  18:  0000000001f9ee60 000007feee6c5811 System_ServiceModel_ni!System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateTlsnegoSecurityTokenAuthenticator(System.ServiceModel.Security.Tokens.RecipientServiceModelSecurityTokenRequirement, Boolean, System.IdentityModel.Selectors.SecurityTokenResolver ByRef)+0x23f 
  19:  0000000001f9eee0 000007feef624a7a System_ServiceModel_ni!System.ServiceModel.Security.ServiceCredentialsSecurityTokenManager.CreateSecurityTokenAuthenticator(System.IdentityModel.Selectors.SecurityTokenRequirement, System.IdentityModel.Selectors.SecurityTokenResolver ByRef)+0x2a1 
  20:  0000000001f9ef40 000007feee77dc27 System_ServiceModel_ni!System.ServiceModel.Security.SymmetricSecurityProtocolFactory.OnOpen(System.TimeSpan)+0xe9fc7a 
  21:  0000000001f9efb0 000007feee6ebe43 System_ServiceModel_ni!System.ServiceModel.Security.WrapperSecurityCommunicationObject.OnOpen(System.TimeSpan)+0x17 
  22:  0000000001f9efe0 000007feef18bf45 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Open(System.TimeSpan)+0x233 
  23:  0000000001f9f0d0 000007feef1881cb System_ServiceModel_ni!System.ServiceModel.Security.SecurityListenerSettingsLifetimeManager.Open(System.TimeSpan)+0x65 
  24:  0000000001f9f140 000007feee6ebe43 System_ServiceModel_ni!System.ServiceModel.Channels.SecurityChannelListener`1[[System.__Canon, mscorlib]].OnOpen(System.TimeSpan)+0xcb 
  25:  0000000001f9f1b0 000007feee727aa7 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Open(System.TimeSpan)+0x233 
  26:  0000000001f9f2a0 000007feee6ebe43 System_ServiceModel_ni!System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(System.TimeSpan)+0x57 
  27:  0000000001f9f2f0 000007feee6b936e System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Open(System.TimeSpan)+0x233 
  28:  0000000001f9f3e0 000007feee6ebe43 System_ServiceModel_ni!System.ServiceModel.ServiceHostBase.OnOpen(System.TimeSpan)+0x6e 
  29:  0000000001f9f450 000007fef09bec81 System_ServiceModel_ni!System.ServiceModel.Channels.CommunicationObject.Open(System.TimeSpan)+0x233 
  30:  

For administrative reasons, this server exposes a number of encrypted WCF endpoints. During startup the WCF runtime examines the certificate and checks whether it is valid. During this operation, it tries to access the private key and calls the Win32 native function CryptFindOidInfo. As it turns out, this function may contact a domain controller to fetch certain information. This “feature” can be disabled by passing the CRYPT_OID_DISABLE_SEARCH_DS_FLAG to the function, but the .NET framework doesn’t do this. Apparently, the function had difficulties contacting its domain controller. A common cause for this is an incorrect DNS configuration, so I checked this first. As it turned out, the DNS configuration was quite complicated and a conditional forwarder for the domain the machine was on was missing. Once this was fixed, the startup speed of the service was back to normal.

Update

Alex from Decrypt my World has an article related to this problem and offers a different solution: http://blogs.msdn.com/b/alejacma/archive/2010/12/20/big-delay-while-calling-envelopedcms-constructor.aspx


Posted by Henning Krause on Tuesday, November 30, 2010 7:33 PM, last modified on Monday, December 20, 2010 2:33 PM
Permalink | Post RSSRSS comment feed

Troubleshooting Push notifications failures

It can sometimes be tricky to get Exchange push notifications working. The development of a listener is only part of the solution. The other part is getting the notifications from the Exchange server to the listener, especially when the listener expects the notifications on a secure channel. Exchange is of no particular help here. The only error message that is generated when Exchange can’t send a notification message to a listener is something like this:

image

This message is not helpful at all. Ok – Exchange could not send a notification a specific subscription. But why?

That’s why I wrote a small utility that should be able to diagnose the three most common problems:

  • The Exchange server cannot reach the notification listener because a firewall is blocking the access.
  • The host specified in the callback address can not be resolved to an IP address
  • If the callback address of the listener is secured using TLS/SSL, Exchange will not send notifications if the server certificate used by the listener is not trusted.

Instructions:

Copy the program on the Exchange server. Open a command prompt and start the program. It expects the address of the listener as the first parameter. The execution of the program can take up to one minute.

Download the program here:

   (105kb)

The source code can be downloaded from CodePlex.


Posted by Henning Krause on Sunday, June 21, 2009 12:14 AM, last modified on Sunday, June 21, 2009 10:59 AM
Permalink | Post RSSRSS comment feed

InfiniTec.Exchange.Notifications updated to 1.5.0.0

I have just published a new version of the my notification component on CodePlex. The new version has some breaking changes to the previous version but it should be simpler to use. Additionally, I have added a small sample application that shows how to use the component. It’s a small WPF application that allows a user to subscribe to calendars of multiple users. The application uses the new Managed API which made the whole thing a whole lot easier to write. Here is a class diagram of the component:

ClassDiagram

If you have worked with the component before you’ll notice that the Subscription class has lost some of its members. I have removed those methods to avoid confusion on how the component is utilized. The only way to create a new subscription now is to use the SubscriptionCollection class. Here is an example on how to use the component:

   1: using System;
   2: using System.ComponentModel;
   3: using System.IO;
   4: using System.Net;
   5: using InfiniTec.Exchange.Notifications;
   6: using InfiniTec.Threading;
   7:  
   8: namespace NewMailNotificationExample
   9: {
  10:     internal class Program
  11:     {
  12:         private const string StateSaverFilename = "notificationstate.bin";
  13:  
  14:         private static void Main()
  15:         {
  16:             // Ignore any certificate errors
  17:             ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => true;
  18:  
  19:             // Setup the adapter which will be used to call into the Exchange WebService  
  20:             using (var adapter = new ExchangeWebServicesAdapter(new Uri("https://w2k3x64/ews/exchange.asmx"), 
  21:                 new NetworkCredential("administrator", "password", "contoso")))
  22:             using (var listener = new PushNotificationListener())
  23:             {
  24:                 Console.Out.WriteLine("Starting Notification Service...");
  25:                 listener.Start();
  26:  
  27:                 SubscriptionCollection subscriptionCollection;
  28:  
  29:                 if (File.Exists(StateSaverFilename))
  30:                 {
  31:                     using (var stream = File.OpenRead(StateSaverFilename))
  32:                     {
  33:                         subscriptionCollection = SubscriptionCollection.Deserialize(stream, adapter, listener);
  34:                         subscriptionCollection.SubscriptionRestartCompleted +=
  35:                             SubscriptionCollection_OnSubscriptionRestartCompleted;
  36:                         subscriptionCollection.SubscriptionRestartProgressChanged +=
  37:                             SubscriptionCollection_OnSubscriptionRestartProgressChanged;
  38:  
  39:                         subscriptionCollection.RestartAsync();
  40:                     }
  41:                 }
  42:                 else
  43:                 {
  44:                     // Create a new subscription collection to manage all the subscriptions  
  45:                     // Register for a NewMail notification on the inbox of the administrator
  46:                     subscriptionCollection = new SubscriptionCollection(adapter, listener)
  47:                                                  {
  48:                                                      {new[] {new FolderReference(WellKnownFolderId.Inbox)}, EventTypes.All}
  49:                                                  };
  50:                 }
  51:  
  52:                 Console.Out.WriteLine("Creating subscription");
  53:                 foreach (var subscription in subscriptionCollection)
  54:                 {
  55:                     // Write a line to the console for each new mail received  38:  
  56:                     subscription.NewMail += (sender, e) =>
  57:                                             Console.Out.WriteLine(string.Format("{0}: New Mail arrived in your inbox", e.Timestamp));
  58:                     subscription.Start();
  59:                 }
  60:                 Console.Out.WriteLine("Waiting for notifications... Hit [Enter] to quit...");
  61:                 Console.ReadLine();
  62:  
  63:                 Console.Out.WriteLine("Saving the current state of the notification listener...");
  64:                 using (var stream = File.OpenWrite(StateSaverFilename))
  65:                 {
  66:                     subscriptionCollection.Serialize(stream);
  67:                 }
  68:                 Console.Out.WriteLine("State saved to {0}", Path.GetFullPath(StateSaverFilename));
  69:             }
  70:         }
  71:  
  72:  
  73:         private static void SubscriptionCollection_OnSubscriptionRestartProgressChanged(object sender, ProgressChangedEventArgs args)
  74:         {
  75:             Console.Out.WriteLine("Subscription restart {0}% complete.", args.ProgressPercentage);
  76:         }
  77:  
  78:         private static void SubscriptionCollection_OnSubscriptionRestartCompleted(object sender, AsyncCompletedEventArgs<SubscriptionRestartErrorSummary> args)
  79:         {
  80:             Console.Out.WriteLine("Subscription restart is complete. {0} subscriptions could not be restarted.", args.Result.Errors.Count);
  81:         }
  82:     }
  83: }

Posted by Henning Krause on Saturday, May 23, 2009 12:38 PM, last modified on Saturday, May 23, 2009 12:40 PM
Permalink | Post RSSRSS comment feed

Push-Notifications – Surviving application restarts

This is the fourth article about the .NET component I published at CodePlex recently. To all related articles, click here.

If you subscribe to a folder on an Exchange mailbox or public folder, the server will try to send notifications to the registered endpoint of yours upon every modification. If your application is shutdown, Exchange will try to reach it for a certain amount of time. Once this time span has elapsed, it will delete the subscription. If the application is restarted and a new subscription is created, it will be notified about all subsequent events. But for certain use cases, it is imperative that no events are missed – even events which occurred between application downtime must be catched. One example for this is a synchronization applications which keeps a sql server database (like a CRM application) and Exchange folders in sync.

For these types of scenarios, all Exchange events carry a watermark with them. This watermark is opaque to the client, but for the Exchange server it contains enough information to reconstruct changes made since the watermark was received. To use this features, an application has to save each watermark it gets. The watermark can be sent to the Exchange server along with the subscription request for a folder. Exchange will then replay the events which happened since that watermark has been generated.

I’ve implemented this feature in my Push Notification component in the SubscriptionCollection class. It can be serialized to either a System.IO.Stream or System.Xml.XmlWriter. The latter one uses the System.Runtime.Serialization.DataContractSerializer to perform the serialization. And the way it is used, it requires .NET Framework 3.5 Service Pack 1. If you cannot rely on having this version installed on your clients machines, you should probably use the binary serialization instead.

Here is a small code sample on how to use the serialization feature (this is an example only, so the exception handling is far from optimal…)

   1: using System;
   2: using System.ComponentModel;
   3: using System.IO;
   4: using System.Net;
   5: using InfiniTec.Exchange.Notifications;
   6: using InfiniTec.Threading;
   7:  
   8: namespace NewMailNotificationExample
   9: {
  10:     internal class Program
  11:     {
  12:         private const string _StateSaverFilename = "notificationstate.bin";
  13:  
  14:         private static void Main()
  15:         {
  16:             // Ignore any certificate errors
  17:             ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) => true;
  18:             
  19:             // Setup the adapter which will be used to call into the Exchange WebService  
  20:             var adapter = new ExchangeWebServicesAdapter(new Uri("https://w2k3x64/ews/exchange.asmx"),
  21:                                                          new NetworkCredential("administrator", "password", "contoso"));
  22:  
  23:  
  24:             // Setup a listener that listens on port 80 on the local computer          
  25:  
  26:             using (var listener = new PushNotificationListener())
  27:             {
  28:                 Console.Out.WriteLine("Starting Notification Service...");
  29:                 listener.Start();
  30:  
  31:                 SubscriptionCollection subscriptionCollection;
  32:  
  33:                 if (File.Exists(_StateSaverFilename))
  34:                 {
  35:                     using (var stream = File.OpenRead(_StateSaverFilename))
  36:                     {
  37:                         subscriptionCollection = SubscriptionCollection.Deserialize(stream);
  38:                         subscriptionCollection.SubscriptionRestartCompleted += 
  39:                             SubscriptionCollection_OnSubscriptionRestartCompleted;
  40:                         subscriptionCollection.SubscriptionRestartProgressChanged +=
  41:                             SubscriptionCollection_OnSubscriptionRestartProgressChanged;
  42:  
  43:                         subscriptionCollection.RestartAsync(listener, adapter);
  44:                     }
  45:                 }
  46:                 else
  47:                 {
  48:                     // Create a new subscription collection to manage all the subscriptions  
  49:                     // Register for a NewMail notification on the inbox of the administrator
  50:                     subscriptionCollection = new SubscriptionCollection(adapter)
  51:                                                  {
  52:                                                      {new[] {new FolderReference(WellKnownFolderId.Inbox)}, EventTypes.NewMail}
  53:                                                  };
  54:                 }
  55:  
  56:                 Console.Out.WriteLine("Creating subscription");
  57:                 foreach (var subscription in subscriptionCollection)
  58:                 {
  59:                     // Write a line to the console for each new mail received  38:  
  60:                     subscription.NewMail += (sender, e) =>
  61:                                             Console.Out.WriteLine(string.Format("{0}: New Mail arrived in your inbox", e.Timestamp));
  62:                     subscription.Start(listener);
  63:                 }
  64:                 Console.Out.WriteLine("Waiting for notifications... Hit [Enter] to quit...");
  65:                 Console.ReadLine();
  66:  
  67:                 Console.Out.WriteLine("Saving the current state of the notification listener...");
  68:                 using (var stream = File.OpenWrite(_StateSaverFilename))
  69:                 {
  70:                     subscriptionCollection.Serialize(stream);
  71:                 }
  72:                 Console.Out.WriteLine("State saved to {0}", Path.GetFullPath(_StateSaverFilename));
  73:             }
  74:         }
  75:  
  76:         private static void SubscriptionCollection_OnSubscriptionRestartProgressChanged(object sender, ProgressChangedEventArgs args)
  77:         {
  78:             Console.Out.WriteLine("Subscription restart {0}% complete.", args.ProgressPercentage);
  79:         }
  80:  
  81:         private static void SubscriptionCollection_OnSubscriptionRestartCompleted(object sender, AsyncCompletedEventArgs<SubscriptionRestartErrorSummary> args)
  82:         {
  83:             Console.Out.WriteLine("Subscription restart is complete. {0} subscriptions could not be restarted.", args.Result.Errors.Count);
  84:         }
  85:     }
  86: }

 

This essentially is the sample from my first blog post in this series, but enhanced to take advantages of the restart capabilities of the SubscriptionCollection class. The program checks to see if there is a saved state from a previous run available. If found, it is deserialized and restarted (lines 35 to 43). Otherwise a new instance is created. Finally, the events are wired to the individual subscriptions (line 57 to 62). And once the application stops, the current state is serialized to a file.

Since the subscriptions are restarted asynchronously in the background, the SubscriptionRestartCompleted and SubscriptionRestartProgressChanged events are used to keep track of the deserialization process. The progress changed event handler is raised for each subscription restarted. Once complete, the SubscriptionRestartCompleted handler is raise. The event args this event handler contain information about all failed restart attempts.


Posted by Henning Krause on Monday, January 12, 2009 11:36 PM, last modified on Monday, January 12, 2009 11:36 PM
Permalink | Post RSSRSS comment feed

Push notifications with WCF – Security considerations

This is the third article about the .NET component I published at CodePlex recently. To all related articles, click here.

The PushNotificationListener I created for that component uses a WCF endpoint to receive notifications. This means that it needs to open an TCP endpoint of some sort. Thanks to WCF, all the hard stuff is done by the WCF infrastructure. There are, however, two issues which must be resolved before you can receive notifications from your Exchange Server: The Windows Firewall, which blocks all incoming traffic by default and the WCF permission system. To deal with the former, you need to setup an exception for a specific port or for your executable. If you are using Windows Installer Xml, you can use the Firewall Extension to create such an exception during the setup of your application. If you want to do this directly from your application, you will need administrative rights and perform some interop stuff (see the links at the end of the article).

To open an WCF endpoint, you need administrative permissions too, by default. Because this is a very nasty requirement, the system allows you to create so-called reservations which can be tied to specific users. You can use nethsh.exe to manipulate these permissions. You can also use the httpcfg.exe tool. Or, you save you the headache that comes with these tools and head over to Paul Wheelers blog and take a look at this blog post of him: AddressAccessDeniedException: HTTP could not register URL http://+:8080/<…>. He has published a small tool (full source included) to enumerate and manage WCF port reservations.

Here is a screenshot of the reservations on my machine:

image

The really only real interesting endpoint here is the second from the bottom: http://+:80/Temporary_Listen_Addresses. This is a reservation in the form of an UrlPrefix String (more about that here on MSDN) that can be used by everyone (and Dominick Baier on leastprivilege.com has a nice post about the security ramifications here). However, the reservation is there and we can use it for the PushNotificationListener. This is the reason why the default port is 80 and the RelativePath property is set to a folder below the Temporary_Listen_Addresses. If you need to use another relative path or even another port, you’ll have to create a reservation for it. You can take a look at the source code of the HttpNamespaceManager tool from Paul Wheelers Blog on how to do this. Again, this is probably best done during setup. Again, if you are using Windows Installer Xml, you can use a managed custom action (using the DTF Framework) to create the necessary reservation during the setup of your application.

Controlling the Windows Firewall with C#

Here are a few links I just found on the internet. I have tested none of them, but they might give you a hint:

http://www.shafqatahmed.com/2008/01/controlling-win.html

http://www.codeproject.com/KB/winsdk/WinXPSP2Firewall.aspx


Posted by Henning Krause on Thursday, January 1, 2009 9:24 PM, last modified on Thursday, January 1, 2009 9:24 PM
Permalink | Post RSSRSS comment feed