http://www.infinitec.de/InfiniTec - Henning Krauses Blog - .NET Tools2005-02-12T12:25:00+00:00Don't adjust your mind - it's reality that is malfunctioning Henning KrauseBlogEngine.Net Syndication Generatorhttp://www.infinitec.de/opml.axdHenning KrauseDon't adjust your mind - it's reality that is malfunctioning en-USInfiniTec - Henning Krauses Blog 0.0000000.000000http://www.infinitec.de/post/2005/02/12/SmartMemoryManagement.aspxSmartMemoryManagement2005-02-12T11:26:00+00:00hkrause<h2 xmlns="http://www.w3.org/1999/xhtml">Description</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>The .NET Runtime has four different heaps where it stores objects. There are many good articles on how the runtime manages object on these heaps so I won’t into great details about the first three. Here are a two good links to blogs with more information on this topic:</div>
<ul>
<li>
<a href="http://infinitec.local/link.aspx?id=http://weblogs.asp.net/clyon/archive/2004/09/14/229477.aspx">A Few Good GC Links</a>
</li>
<li>
<a href="http://infinitec.local/link.aspx?id=http://blogs.cathedral-of-hate.org/tony/archive/0001/01/01/159.aspx">.net Garbage Collector Tips and Tricks </a>
</li>
</ul>
<h3>The Large Object Heap</h3><div>
<div>Most of the time, the GC does a well job managing the memory of your programs. </div>
<div>But there are situations, where it performs sub-optimally. One of these situations are so-called <span style="font-weight:bold;">Large Objects</span>. All objects with a size greater than 85000 bytes fall within this category. These objects are stored on a separate heap, called Large Object Heap. This heap will neither be defragmented, nor is it collected often, as the GC processes this heap along with the Generation-2 heap.</div>
<div>To reproduce the problem, try to read a large file (say 70MB or so) from a file stream and store it in a MemoryStream. You will see that your memory usage will climb to about <span style="font-weight:bold;">250 MB</span>. Now, destroy your Memorystream and read the file again into another MemoryStream. Now, depending on your memory pressure, the GC will do a Gen-2 Collection (remember.. this is expensive) and reclaim that memory, or you end up with about <span style="font-weight:bold;">500MB</span> of consumed memory. </div>
<div>Now you will probably argue that 250MB are way to much because you could preallocate one large buffer of 70MB for the file via MemoryStream.SetLength(), so you’ll end up with a smaller memory consumption. But there are two things with this argument:</div>
<ol>
<li>On the second read of the file you memory usage will increase anyway, albeit not to 500 MB but perhaps to about 150MB.</li>
<li>You might not always know in advance how much memory you will need. Think of a server application that reads data over a TCP connection, there might be no hint on how large the amount of received data is. So you end up with increasing your buffer every now and then.</li>
</ol>
</div>
<h3>The Solution</h3><div>
<div>The classes you can download below will give you the ability to reuse previously allocated chunks of memory. And once you are done with the memory, you can return it to the pool.</div>
<div>Note that I used weak references within the pool, so the GC can reclaim the buffers within the pool if memory pressure comes up.</div>
<div>
</div>
<div>Additionally, I have developed a new MemoryStream that uses this object pooling technique.</div>
</div>
</div><h2 xmlns="http://www.w3.org/1999/xhtml">Downloads</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>
<a href=file.axd?file=smartmemorymanagement/SmartMemoryManagement.zip>Source and documentation</a>
</div>
</div>2005-02-12T11:26:00+00:00hkrauseThese classes implement a buffer manager for large .NET byte arrays as well as a new MemoryStream class that uses buffers from this pool for memory allocations.http://www.infinitec.de/pingback.axdhttp://www.infinitec.de/post.aspx?id=f6b957b1-4d76-476e-a1d0-dc09a5fa93320http://www.infinitec.de/trackback.axd?id=f6b957b1-4d76-476e-a1d0-dc09a5fa9332http://www.infinitec.de/post/2005/02/12/SmartMemoryManagement.aspx#commenthttp://www.infinitec.de/syndication.axd?post=f6b957b1-4d76-476e-a1d0-dc09a5fa9332http://www.infinitec.de/post/2005/02/07/WebDAVLayer-10173512520.aspxWebDAVLayer 1.0.1735.125202005-02-07T20:04:00+00:00hkrause<h2 xmlns="http://www.w3.org/1999/xhtml">More Information</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>The database that is used by Exchange 2000/2003 is called WebStorageSystem. Each element within this store is accessible via an URL using the WebDAV protocol. This protocol extends the HTTP protocol by exposing several new verbs like <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/e2k3/e2k3/_webdav_propfind.asp">PROPFIND</a>, <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/e2k3/e2k3/_webdav_proppatch.asp">PROPPATCH</a> or <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/e2k3/e2k3/_webdav_search.asp">SEARCH</a> to name a few. Additionally, the Exchange implementation of the WebDAV protocol has some proprietary commands for batch operations: <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/e2k3/e2k3/_webdav_bmove.asp">BMOVE</a> and <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/e2k3/e2k3/_webdav_bcopy.asp">BCOPY</a>. The parameters of these commands are expressed within the body of the HTTP/WebDAV request as XML.</div>
</div><h2 xmlns="http://www.w3.org/1999/xhtml">Features</h2><div xmlns="http://www.w3.org/1999/xhtml">
<ul>
<li>Encapsulation of most of the WebDAV commands.</li>
<li>Easy access to folders, elements and properties within the store</li>
<li>Support for various authentication schemas: Basic, Digest, NTLM, Kerberos and even the Form-based authentication of Exchange 2003</li>
</ul>
<div>Unfortunately, there is currently no documentation available. But on the other hand: You can use it free of charge! </div>
</div><h2 xmlns="http://www.w3.org/1999/xhtml">Downloads</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>
<a href=file.axd?file=webdavlayer/WebDAVLayer.1.0.1735.12520.zip>Full Source code</a>
</div>
</div>2005-02-07T20:04:00+00:00hkrauseThis package simplifies the access to Exchange 2000/2003 by encapsulating the WebDAV Protocol into a set of classes that can be used from any .NET language.http://www.infinitec.de/pingback.axdhttp://www.infinitec.de/post.aspx?id=1a00947a-df56-4b63-a25e-456f0b1f41f82http://www.infinitec.de/trackback.axd?id=1a00947a-df56-4b63-a25e-456f0b1f41f8http://www.infinitec.de/post/2005/02/07/WebDAVLayer-10173512520.aspx#commenthttp://www.infinitec.de/syndication.axd?post=1a00947a-df56-4b63-a25e-456f0b1f41f8http://www.infinitec.de/post/2005/01/02/StableEvents.aspxStableEvents2005-01-02T13:10:00+00:00hkrause<h2 xmlns="http://www.w3.org/1999/xhtml">Description</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>The events within the .NET framework have a severe drawback: If multiple objects have subribed to an event and one subscriber throws an exception, all subsequent subribers are not notified of the event. This is especially true for events used within the remoting framework: If a connection is broken between a event provider and a subscriber, a <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/cpref/html/frlrfsystemreflectiontargetinvocationexceptionclasstopic.asp">TargetInvocationException</a> is thrown when the event is invoked.</div>
<div>This class solves this problem by manually invoking each subscriber. All failed notifications are returned to the sender, which in turn can remove the failed subscribers from the list.</div>
</div><h2 xmlns="http://www.w3.org/1999/xhtml">Downloads</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>
<a href=file.axd?file=stableevents/InfiniTec.SystemUtilities.StableEvents.zip>Sourcecode</a>
</div>
<div>
<a href=file.axd?file=stableevents/InfiniTec.SystemUtilities.StableEvents.chm>Documentation</a>
</div>
<div>
</div>
</div>2005-01-02T13:10:00+00:00hkrauseThis class simplifies the usage of events, especially with the .NET remoting framework, where connection-drops occur frequently.http://www.infinitec.de/pingback.axdhttp://www.infinitec.de/post.aspx?id=446bbc30-f28b-48b7-938b-9496f2b84a250http://www.infinitec.de/trackback.axd?id=446bbc30-f28b-48b7-938b-9496f2b84a25http://www.infinitec.de/post/2005/01/02/StableEvents.aspx#commenthttp://www.infinitec.de/syndication.axd?post=446bbc30-f28b-48b7-938b-9496f2b84a25http://www.infinitec.de/post/2005/01/02/MagicPackets.aspxMagicPackets2005-01-02T12:45:00+00:00hkrause<h2 xmlns="http://www.w3.org/1999/xhtml">Description</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>MagicPackets are special IP packets that are recognized by Wake-On-LAN capable network adapters. If such a packet is received by a network adapter, it will boot the computer.</div>
<div>The packet contains the MAC address of the destination computer and is sent either sent to the global broadcast address (255.255.255.255) or the broadcast address of the destination subnet (e.g. 192.168.1.255).</div>
</div><h2 xmlns="http://www.w3.org/1999/xhtml">Downloads</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>
<a href=file.axd?file=magicpacket/InfiniTec.SystemUtilities.MagicPacket.zip>Source</a>
</div>
<div>
<a href=file.axd?file=magicpacket/InfiniTec.SystemUtilities.MagicPacket.Example.zip>Example program</a>
</div>
<div>
<a href=file.axd?file=magicpacket/InfiniTec.SystemUtilities.MagicPacket.chm>Documentation</a>
</div>
<div>
</div>
</div>2005-01-02T12:45:00+00:00hkrauseThis class allows the sending of so-called Magic Packets.http://www.infinitec.de/pingback.axdhttp://www.infinitec.de/post.aspx?id=137a8715-28b7-49df-9e0b-685aa06d98dc0http://www.infinitec.de/trackback.axd?id=137a8715-28b7-49df-9e0b-685aa06d98dchttp://www.infinitec.de/post/2005/01/02/MagicPackets.aspx#commenthttp://www.infinitec.de/syndication.axd?post=137a8715-28b7-49df-9e0b-685aa06d98dchttp://www.infinitec.de/post/2005/01/02/Advanced-Thread-Pool.aspxAdvanced Thread Pool2005-01-02T11:47:00+00:00hkrause<h2 xmlns="http://www.w3.org/1999/xhtml">Description</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>The existing <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/cpref/html/frlrfsystemthreadingthreadpoolclasstopic.asp">threadpool class</a> has several drawbacks:</div>
<ul>
<li>The number of available threads can hardly be changed (See <a href="http://infinitec.local/link.aspx?id=http://www.csharphelp.com/archives3/archive487.html">Changing the Default Limit of 25 Threads of ThreadPool Class</a>).</li>
<li>Deadlocks can occur when a method is called from the threadpool which itself calls methods on the threadpool. </li>
</ul>
<div>One example for the latter point is the <a href="http://infinitec.local/link.aspx?id=http://msdn.microsoft.com/library/en-us/cpref/html/frlrfsystemnethttpwebrequestclasstopic.asp">HttpWebRequest Class</a>, which still goes further and throws an exception when less than two threads are available on the threadpool. (See my article <a href="http://infinitec.local/link.aspx?id=software/dotnetarticles/it000002">HttpWebRequest fails when several connections are made concurrently</a>).</div>
<div>These drawbacks are evaded with this implementation: Within one process more than one instance can be instantiated and the number of available threads can be adjusted per instances. Additionally, the number of used threads is dynamic, so unused threads will starver over time and will be recreated when load goes up.</div>
</div><h2 xmlns="http://www.w3.org/1999/xhtml">Downloads</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>
<a href=file.axd?file=advancedthreadpool/advancedthreadpool.zip>Source and documentation</a>
</div>
</div><h2 xmlns="http://www.w3.org/1999/xhtml">References</h2><div xmlns="http://www.w3.org/1999/xhtml">
<div>Some of the ideas for this implementation were taken from the CodeProject article <a href="http://infinitec.local/link.aspx?id=http://www.codeproject.com/csharp/SmartThreadPool.asp">Smart Thread Pool</a> by Ami Bar.</div>
</div>2005-01-02T11:47:00+00:00hkrauseThis class implements a custom threadpool. While there is already an implemention of a threadpool, that implementation has a few drawbacks.http://www.infinitec.de/pingback.axdhttp://www.infinitec.de/post.aspx?id=23b3f153-79ba-4d05-9d95-cc3adb1019130http://www.infinitec.de/trackback.axd?id=23b3f153-79ba-4d05-9d95-cc3adb101913http://www.infinitec.de/post/2005/01/02/Advanced-Thread-Pool.aspx#commenthttp://www.infinitec.de/syndication.axd?post=23b3f153-79ba-4d05-9d95-cc3adb101913