November 7 2014

nHibernate and Type Mappings

While diagnosing random poor performance on live servers recently, we discovered that the issue lay with some inefficient query performance in sql server. Upon examining the queries we discovered the issue was due to search parameters being sent as nvarchar (unicode) while the lookup column value was a varchar (ansi). Its a subtle difference with some major impacts.

To do a type comparison, sql server must convert the data types to the same type. In this case, it converts the varchar to an nvarchar (widening) before comparison. When you think about it this has to be the case. When sql server compares the column value to the search parameter it needs to do a bit for bit comparison. If it was to simply narrow the parameter, bits might be lost and the value would be altered rendering the query invalid. This being the case, it has no option but to widen the lookup value of every row in the table. Not only does this cause high cpu usage but it also causes a large number of reads consuming more memory in the process. Not a good state of affairs but at least the fix is easy enough, we need to make the parameter value a varchar instead of an nvarchar. This allows the server to use a seek with no type conversion and a small number of reads giving a drop in resource usage and a signifigant performance gain.

Given we are using nHibernate, it was time to examine some type mappings to see what was causing this behaviour. What I found was this …

Xml: <property Name="myPropName" Column="myLookupColunm" Type="String" />
Fluent: Map(x => x.myPropName);

All looks well so far … or does it? A search for c sharp to sql type mappings for nHibernate reveals that string will map to an nvarchar. There is another type mapping that will instruct nHibernate to use varchar …

Xml: <property Name="myPropName" Column="myLookupColunm" Type="AnsiString" />
Fluent: Map(x => x.myPropName).CustomType("AnsiString");

It seems so simple … and it is, when you know what the problem is. The issue here is that sql server is a little too tolerant of our oversights. It does such a good job of just converting the types and getting on with it that you only notice the issue under some hefty load on large tables if you’re running on production grade metal. Once you’re at this stage, that code was likely written long ago and long forgotten and diagnosing the problem can take some time.

It should be noted that this applies to any and all datatype mismatches, not just strings. If the parameter data type is wider than the column data type, this issue will occur. Remember the bit for bit comparison thing … all data types must be equal!

Happy Coding! :)

Category: DotNet, Sql | 3 Comments
December 9 2012

Wcf Services and https Load Balancers

I was recently required to write a small WCF service to run over https. Simple enough. I configured a simple basicHttpBinding with transport security and all was well. The service was run through the usual set of test and acceptance environments and was given the green light to go to production. On deployment day we ran into a somewhat unexpected problem. The service was throwing an error along the lines of …

“Could not find a base address that matches scheme https for the endpoint with binding basicHttpBinding. Registered base address schemes are [http]:”

This error occurs when the IIS server doesn’t have a binding to the scheme you have specified … in our case https (if you get this on a dev machine you need to read up on creating a self signed certificate so you can test https locally). As it turns out our live servers do not have an https binding in IIS. All of the https traffic and certificates are handled by load balancers and the traffic is forwarded as http from that point. Ok then … so we can just set up WCF without transport security and it should be fine right? No … you need the wsdl to contain the address to the load balancer … ie the urls should start with https, but since the service is running without transport security it generates the wsdl with http urls.

One simple solution would be to create an https binding on each of our servers and let the service run with transport security. Our systems guys weren’t keen to say the least. I’m not a systems guy but there are a number of servers that need to be configured and, just as it makes sense in software, it makes sense to centralise the certificates etc. So that wasn’t an option.

I did a *lot* of searching over the web for articles that might resolve this issue. There is another problem related to this to do with message security but in my case I was only using transport security so they didn’t apply. It would seem that problem is much more common than mine as I couldn’t find much at all about it. I eventually stumbled across the ‘externalMetadataLocation’ attribute that can be added to the mex endpoint and had an idea. I could simply download and edit the wsdl to contain load balancer urls then use this attribute to tell the service to publish the edited wsdl.

It turned out to be *almost* as simple as that … I also had to download the auto generated xsd files and edit them also. So here’s what to do …

1) Configure your service to run on basicHttpBinding with no transport security
2) Browse to the wsdl in the browser on your dev machine and copy it to a text editor.
3) Look in the wsdl file for the urls that end in xsd=xsd0 and browse to these in the browser and copy them to the text editor.
4) *Carefully* edit all urls in the files to point to the load balancer urls.
5) Don’t forget to edit the xsd file urls in the wsdl to where they will be hosted in production.
6) Deploy these files to a server.
7) Configure your mex endpoint with the externalMetadataLocation attribute (in your dev and test environments just leave this attribute out and use the automatic generation)

<behavior name="">
    <serviceMetadata httpGetEnabled="True" externalMetadataLocation="https://MyServiceUrl/Wsdl/theWsdl.wsdl"/>             
</behavior>

Your service will be available on the http schema but will respond to mex requests with the wsdl file containing the https urls to the load balancer. When a request is made the load balancer will decrypt the request then forward it to the service on http. The response will return to the load balancer over http which it will then encrypt and send over https … all is well.

The downside to this approach is, of course, that when new methods are added to the service or changes are made to the signatures of existing methods the wsdl and xsd files must be maintained manually by repeating the process used to create them. Whilst this isn’t too horrible it might catch new developers on the system by surprise so its worth sticking a giant README file in there.

Category: DotNet | 2 Comments
September 27 2012

XML Serlializer Memory Leaks

A little while ago at work we received a complaint from the systems team that one of our web applications was increasing in memory usage until it crashed the process … we had a memory leak. Its easy to think that with all the garbage collecting wizardry baked into .NET that a memory leak is not possible … but oh how it is.

We looked through our code for the usual suspects … web requests/responses not being disposed, datareaders, etc. We did find a few issues and cleaned them up and released the “fixed” version. Memory usage slowed slightly but still continued to climb despite our efforts. The plot thickens …

After some digging we found some quite interesting information about, our friend, the XmlSerializer. We had been using an overload of the constructor which allowed us to specify the name of the element when serialising a collection of objects as we could not use the default element. As it turns out, this overloaded constructor was the cause of our memory leaks. To understand why this was the case we need to dig into the inner workings of the XmlSerializer a little but I’ll try to keep this high level.

When you create an XmlSerializer for a given type, the .NET framework generates dlls that contain classes that allow serialization and deserialization of the objects. These dlls are then dynamically loaded into memory. Yes that’s right … they are not generated on disk but loaded straight into memory. So surely, you say, once the dll is no longer required the garbage collector will clean it up. Actually no … the garbage collector works on objects. The dll generated by the XmlSerializer is an assembly which is loaded using the assembly loader … these are out of the control of the garbage collector. Once a dll is loaded it stays in memory until the application stops.

To avoid memory leaks the XmlSerializer caches these assemblies so if one has already been loaded the same one is used over and over. But this only applies for two of the nine constructors that you can use. The other seven constructors will load a new assembly into memory every time they are called. The serializer instance will be disposed but the assembly it generated in memory will persist. This is actually by design … it seems Microsoft did not expect the seven constructors that do not use caching would be called all that much so holding dlls for these serializers in the cache might result in a large cache of objects they may not need any more … so they left them out.

So how do we fix it? Firstly we need to know which constructors are safe to use … these two do not cause memory leaks and can be called repeatedly without concern …

 
XmlSerializer(Type)
XmlSerializer(Type, String)

We were using the XmlSerializer(Type, XmlRootAttribute) version with the memory leak issue. To achieve our same goal of naming the element that collections of our objects are serialised to while still using one of the safe constructors we had to make a few simple changes.

Code that looked like this …

 
var people = new List<Person>();
 
//unsafe constructor
var serializer = new XmlSerializer(typeof(List<Person>), new XmlRootAttribute("people");

Was replaced with code like this …

 
[XmlRoot("people")]
public class PeopleCollection : List<Person>
{
}
 
//safe constructor
var serializer = new XmlSerializer(typeof(PeopleCollection));

If you absolutely must use one of the unsafe constructors you need to override the caching and implement it yourself to prevent the memory leaks as shown here on msdn.

Tess Ferrandez goes into much detail about the inner working of the XmlSerializer and the causes of the memory leaks in her blog post.

Category: DotNet | LEAVE A COMMENT
May 18 2012

Make the XmlSerializer Your Friend

Over the years I have had to do a number of integrations with third party xml web services. These have included credit card payment gateways, government agencies, hotel booking engines and even game providers. Before all the cool kids jump in and say we should be using json, yes we should in a perfect world where we get to rewrite our software every time that the latest and greatest cool thing comes out. For those of us in the real world, these things still exist and need to be maintained and integrated with … enough said.

There are a few ways to consume xml, some good, some not. Lets have a look …

1) You can use XmlWriter and XmlReader classes to manually parse and write the xml. This method is error prone, time consuming, inefficient and difficult to read and maintain. Each time you modify the model object you also need to modify the serializer and deserializer. The one and only upside is that it gives you complete, utter control over the serialization and you can map to whatever object you like in one go. It is the most infinitely customisable way of consuming xml. I have never actually found the need to do it this way … I suggest you don’t either ;)

2) You could use an XmlDocument object or even LinqToXml. This can be handy if dealing with a simple document or you only want one or two nodes out of a complex xml structure. In general this isn’t much better than manually parsing xml if you are trying to deserialize an entire object and is only a little less verbose. Still not a great idea except for the case where it just a node or two.

3) My preferred option … use the built in XmlSerializer given to you by Microsoft. Its not difficult to use and is easily customised using attributes on your model objects. At the end of the day, the xml is a model of data which can easily be translated into model objects.

Lets start with a sample xml that we want to deserialize. Its the standard cookie cutter person-with-an-address structure but its enough to give the basic idea …

 
<?xml version="1.0" encoding="utf-16"?>
<people>
    <person dateadded="2007-09-24T08:00:00+02:00">
      <firstname>Leonard</firstname>
      <surname>Hofstadter</surname>
      <address>
        <housenumber>12</housenumber>
        <street>Some Street</street>
        <suburb>Los Angeles</suburb>
        <state>California</state>
      </address>
  </person>
  <person dateadded="2007-09-24T07:00:00+02:00">
    <firstname>Sheldon</firstname>
    <surname>Cooper</surname>
    <address>
      <housenumber>12-B</housenumber>
      <street>With Leonard</street>
      <suburb>Los Angeles</suburb>
      <state>California</state>
    </address>
  </person>
</people>

The nice touch about using the serializer is that it almost totally abstracts the fact that you are dealing with xml by mapping to a set of objects that model the data. Here are those models … we have a People class that contains a list of Person classes that each contain an Address class …

 
    [XmlRoot("people")]
    public class PeopleCollection : List<Person>
    {
    }
 
    [XmlType("person")]
    public class Person
    {
        [XmlAttribute("dateadded")]
        public DateTime DateAdded { get; set; }
 
        [XmlElement("firstname")]
        public string FirstName { get; set; }
 
        [XmlElement("surname")]
        public string LastName { get; set; }
 
        [XmlElement("address")]
        public Address Address { get; set; }          
    }
 
    [XmlType("address")]
    public class Address
    {
        [XmlElement("housenumber")]
        public string HouseNumber { get; set; }
 
        [XmlElement("street")]
        public string Street { get; set; }
 
        [XmlElement("suburb")]
        public string Suburb { get; set; }
 
        [XmlElement("state")]
        public string State { get; set; }
    }

Nothing really that special … just a simple set of objects that represent the data contained in the xml. The decorators on each property and class tell the serializer how to serialize and deserialize the values. Does it output the property value as an attribute, an element or simply ignore it altogether? The decorators are optional .. if they are omitted the serializer will expect the xml to have the same elements (case sensitive) as the property names and all properties will be serialized as elements.

In this case I have purposely changed the cases of the xml and the property names to demonstrate the use of decorators to customise the serializer. You may note that in the Person class I have mapped the surname element to the LastName property by using the decorator and I have also specified that DateCreated should be emitted as an attribute. More information about the many decorators that can be used to control serialization can be found here.

From this point the serialization code is straightforward …

 
    public class Serializer<T> where T : class 
    {
        public string Serialize(T obj)
        {
            var serializer = new XmlSerializer(typeof (T));
            var namespaces = new XmlSerializerNamespaces();
            namespaces.Add(string.Empty, string.Empty);
 
            var sb = new StringBuilder();
            using (var writer = XmlWriter.Create(sb))
            {
                serializer.Serialize(writer, obj, namespaces);
            }
 
            return sb.ToString();
        }
 
        public T Deserialise(string xml)
        {
            var serialiser = new XmlSerializer(typeof (T));
            using (var stream = new MemoryStream(Encoding.Unicode.GetBytes(xml)))
            {
                return serialiser.Deserialize(stream) as T;
            }
        }
    }

The full example solution can be downloaded from GitHub
Happy coding :)

Category: DotNet | LEAVE A COMMENT
January 13 2012

Code Re-use With Abstract Classes

An often overlooked feature of C# (and OO programming in general) is the use of abstract classes to maximise code re-use. Put simply, if you have code in related classes that is the same but needs to call implementation specific methods, you can probably use an abstract class to consolidate this code. Lets dive right in with a simple example.

This is production code that has been stripped back to highlight the use of abstraction. In my scenario I had a number of user controls that had to implement the same functionality to each display three different views. The initialisation code ran through the same method calls in each control but the implementations of the called methods were different. Enter the abstract class …

public abstract class ControlBase 
{
    public int ViewMode { get; set; }
 
    protected abstract void SetSaveButtonEnabled();
    protected abstract void SetCollapsedView();
    protected abstract void SetSummaryView();
    protected abstract void SetExpandedView();
    protected abstract void HideAll();
 
    protected virtual void InitControl()
    {
        SetViewMode();
        SetSaveButtonEnabled();
    }
 
    protected virtual void SetViewMode()
    {
        HideAll();
        switch (ViewMode)
        {
            case 1:
                SetCollapsedView();
                break;
            case 2:
                SetSummaryView();
                break;
            case 3:
            case 4:
                SetExpandedView();
                break;
        }
    }
}

The InitView() and SetViewMode() methods are the same across all controls. Obviously the actual code to render the correct view is specific to the implementation of the control itself. So how does it work? Firstly, an abstract class can never be instantiated … it can only be inherited from. Any methods specific to the implementation are declared as abstract. In the abstract base class we know what the method signature should be but we can’t implement it at this level. Once the signature is provided and declared abstract we can call that method in the abstract class code as demonstrated in the SetViewMode() and InitControl() methods. At runtime this will result in the implementation of the method in the child class being called.

Our inherited class would look something like this …

 
public class Control1 : ControlBase
{
    protected override void SetSaveButtonEnabled()
    {
        //control specific code here 
    }
 
    protected override void SetCollapsedView()
    {
        //control specific code here 
    }
 
    protected override void SetSummaryView()
    {
        //control specific code here 
    }
 
    protected override void SetExpandedView()
    {
        //control specific code here 
    }
 
    protected override void HideAll()
    {
        //control specific code here 
    }
}

In the child class we override the abstract methods. Failing to override all the abstract methods on the base class will result in a compile time error unless your child class is also abstract.

That’s really all there is to it. By using this technique I saved having the InitControl() and SetViewMode() methods being implemented with the exact same code in all of my child controls. This would at best have been copy-paste coding which is horrible and to be avoided at all times. A bug in one of those methods would have resulted in needing to update all the implementations to fix it. Add the complexity of working in a team and another team member may not realise that the code is repeated in multiple controls. By using the abstract class there is only one implementation to maintain making maintenance much simpler while allowing a much more object oriented design.

More info on MSDN

Category: DotNet | LEAVE A COMMENT
December 12 2011

Radio Buttons and Repeaters in Aspx

If you’re using a Repeater control and have a requirement to have radio buttons in the repeated rows you may find yourself in a head scratching mess. You may find that checking one radio button does not result in all other buttons being unchecked. The radio buttons will essentially start to behave like a bunch of check boxes … not cool.

With normal html radio buttons we group them together using the ‘name’ attribute. All radio buttons with the same value in the name attribute become mutually exclusive resulting in the familiar behaviour where only one radio button in that group can be checked at any one time. In an asp:RadioButton control the name attribute maps to the GroupName parameter which normally works fine.

Things start to go astray when the radio buttons are added to a repeater however. You may be aware that asp.net will insert its own unique id for each control which is not the same as the id you originally gave it in code or markup. It does this to ensure that any elements that have the same id end up with a unique id once rendered. In the case of the repeater this is very necessary as you are taking the same elements with the same id and emitting them multiple times. Unfortunately a repeater also mangles the name attribute of the controls. Normally this can be safely ignored but radio buttons rely on this attribute to work out which group they belong to. All of a sudden each radio button belongs to its own little group and so we end up with check box behaviour.

This is a known issue explained in detail here, however, Microsoft have not yet fixed this and give no indication of if or when it will be fixed.

So now what? Well … there are a couple of work-arounds, none of which are particularly elegant.

1) Avoid using the repeater or radio buttons and see if you can solve the problem with other controls.

2) If you’re stuck with repeaters and radio buttons the most elegant solution is to use javascript to de-select other radio buttons using the click event.

This goes in the <head> tag …

 
 //I use jquery to make life easier but you don't have to 
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js"></script> 
 
<script type="text/javascript">
    function MakeRbExclusive(rb) {
        var rbs = $('#rbData :radio');
        for (var i = 0; i < rbs.length; i++) {
            if (rbs[i] != rb) rbs[i].checked = false;
        }
    }
</script>

And our repeater …

<div id="rbData">
    <asp:Repeater ID="repeater" runat="server">
        <ItemTemplate>
            <!-- I have bound to simple Dictionary<string, string> created in page load for example purposes -->
            <asp:Label ID="lbl" runat="server" AssociatedControlID="radio"><%#Eval("key")%></asp:Label>
            <asp:radioButton ID="radio" runat="server" value=<%#Eval("value")%> onclick="MakeRbExclusive(this);"/>
            <br />
        </ItemTemplate>
    </asp:Repeater>
</div>



3) Avoid the use of a repeater completely and add your controls dynamically from code behind in a loop to achieve the same result. I *really* don’t recommend this. It is finicky at best and takes very careful coding. Dynamically added controls are not automatically tracked in viewstate and can create big problems if not carefully coded. I have had to do this in the past but it is definitely an option I’d only consider once I have exhausted all others.


Happy coding! :)

Category: DotNet | 3 Comments
June 9 2011

Generic Singelton

We are often required to implement the singleton pattern in our code. While doing this yet again I thought there must be a generic way to do this and I came up with the following code …

public static class Singleton<T> where T : class, new()
{
   private static volatile T _instance;
   private readonly static object _lockObj = new object();
 
   public static T Instance
   {
       get
       {
           if (_instance == null)
              lock (_lockObj) if (_instance == null) _instance = new T();
 
           return _instance;
       }
  }
}



This allows you to use any class as a singleton …

var mySingleton = GenericSingleton<MySingleton>().Instance;



It’s not perfect … there is nothing stopping the developer from creating a new instance of the class which breaks the singleton pattern. Doing it the normal way makes it impossible to create more than one instance of a class but this way allows the flexibility to use a class as a singleton even if it’s not implemented that way and also allows a mix of singleton and instances of the same class.

In any event it makes interesting use of generics and I just thought it was cool so I thought I’d share :)

Category: DotNet | LEAVE A COMMENT
June 8 2011

Insert Multiple Rows With a Single Insert

I often have to insert multiple rows of (usually) test data into tables. The simplest but most tedious way is to rewrite the INSERT statement over and over with different values …

INSERT INTO User VALUES ('Penny', 'Penny')
INSERT INTO User VALUES ('Sheldon', 'Cooper')
INSERT INTO User VALUES ('Leonard', 'Hofstadter')
INSERT INTO User VALUES ('Howard', 'Walowitz'
INSERT INTO User VALUES ('Rajesh', 'Koothrappali')


Its not a big deal but as a developer I hate writing the same statement over and over … it just goes against my grain. Its also more of an overhead to repeatedly call insert for each row. Here’s a better way …

INSERT INTO User
SELECT 'Penny', 'Penny' UNION ALL
SELECT 'Sheldon', 'Cooper' UNION ALL
SELECT 'Leonard', 'Hofstadter' UNION ALL
SELECT 'Howard', 'Walowitz' UNION ALL 
SELECT 'Rajesh', 'Koothrappali'



This inserts the same data as the first example but uses only one insert so has less overhead. If you happen to be running SQL Server 2008 there’s a new syntax to achieve the same although the above example will still work. In SQL Server 2008 you can use this syntax …

INSERT INTO User
VALUES ('Penny', 'Penny')
, ('Sheldon', 'Cooper')
, ('Leonard', 'Hofstadter')
, ('Howard', 'Walowitz') 
, ('Rajesh', 'Koothrappali')
Category: Sql | LEAVE A COMMENT
May 15 2011

Viewstate Is Not Evil

Viewstate is often touted to be heavy and slow and better off disabled in most aspx applications. When used properly veiwstate can be quite a useful tool with not too much overhead. Use it improperly and all the horror stories you’ve heard start to come true …

If you view page source in an aspx application that has viewstate enabled site wide you will see the hidden __VIEWSTATE field will have a long encoded string of data. This is quite heavy and takes some time to trasmit betweeen the server and client. Our aim is to minimise the length of this data load to the point where overhead is no longer a concern.

First of all viewstate is enabled by default across the entire project so the first thing we need to do is turn it off. The idea is to opt in to viewstate rather than opt out. We only want to turn it on for the controls where we specifically want to maintain state.

In each masterpage or at the top of each page you can place this code in the page directive to disable the viewstate.

<%@ Page EnableViewState="False"  ViewStateMode="Disabled"%>



Or you can disable the viewstate across the whole application by putting this in the Pages tag in web config.

<Pages EnableViewState="false"  />



Now that Viewstate is off by default … we can choose which controls we want to maintain state for. Most text fields and simple controls will be fine without viewstate. As a general rule leave it off until you have a specific reason to turn it on. The give away will be when you try to access the value of a control in code behind and get the default value no matter what value was set. In this case its time to turn on viewstate for that control only. We do this by using the same code as we did to turn it off …

<asp:DropDownList runat="server" ID="myDdl" EnableViewState="True"  ViewStateMode="Enabled"/>



You can add this to any control and access its value from a postback. If you view the page source again you will see the __VIEWSTATE field is much smaller … naturally the larger number of controls you enable viewstate for the larger the field will be.

Use it with caution but don’t write it off as bad practice.

Category: DotNet | LEAVE A COMMENT
May 7 2011

Bad Examples

All over the web I see the code you see below. While it will work it is wrong … don’t do it! A web response stream is a stream like any other, you need to ensure that it is closed when you are finished with it or it will not release its resources. In the case of a web response (or request for that matter), failing to close it could mean the server will run out of connections. In a high traffic web site this won’t take all that long.

Let’s take a look at the code … what would happen if the call to Write throws an exception? Execution would stop right there and the exception will get bubbled up the call stack. The call to Close will never happen and the response stream will remain open reducing the server’s connection capability by one.

 var responseStream = response.GetResponseStream();
 responseStream.Write(byteData, 0, byteData.Length);
 responseStream.Close();



We can avoid this by putting the call to Close in a finally block as below. This would guarantee that the call to Close will be made even if an exception is thrown. This code is perfectly valid, but cumbersome to have to do every time you need to ensure a resource is released.

var responseStream = response.GetResponseStream();
try
{
    responseStream.Write(byteData, 0, byteData.Length);
}
finally
{
    responseStream.Close();
}



The code can be rewritten as below to produce the same behaviour. The ‘using’ keyword automatically calls Dispose (as good as calling Close) on the resource as soon as it falls out of scope resulting in the same behaviour as our finally block. Any time you use a stream, db connection or any resource that needs to be released, use a ‘using’ statement and you can’t go wrong.

using (var responseStream = response.GetResponseStream())
{
    responseStream.Write(byteData, 0, byteData.Length);
}
Category: DotNet | LEAVE A COMMENT