Wcf Services and https Load Balancers

I was recently required to write a small WCF service to run over https. Simple enough. I configured a simple basicHttpBinding with transport security and all was well. The service was run through the usual set of test and acceptance environments and was given the green light to go to production. On deployment day we ran into a somewhat unexpected problem. The service was throwing an error along the lines of …

“Could not find a base address that matches scheme https for the endpoint with binding basicHttpBinding. Registered base address schemes are [http]:”

This error occurs when the IIS server doesn’t have a binding to the scheme you have specified … in our case https (if you get this on a dev machine you need to read up on creating a self signed certificate so you can test https locally). As it turns out our live servers do not have an https binding in IIS. All of the https traffic and certificates are handled by load balancers and the traffic is forwarded as http from that point. Ok then … so we can just set up WCF without transport security and it should be fine right? No … you need the wsdl to contain the address to the load balancer … ie the urls should start with https, but since the service is running without transport security it generates the wsdl with http urls.

One simple solution would be to create an https binding on each of our servers and let the service run with transport security. Our systems guys weren’t keen to say the least. I’m not a systems guy but there are a number of servers that need to be configured and, just as it makes sense in software, it makes sense to centralise the certificates etc. So that wasn’t an option.

I did a *lot* of searching over the web for articles that might resolve this issue. There is another problem related to this to do with message security but in my case I was only using transport security so they didn’t apply. It would seem that problem is much more common than mine as I couldn’t find much at all about it. I eventually stumbled across the ‘externalMetadataLocation’ attribute that can be added to the mex endpoint and had an idea. I could simply download and edit the wsdl to contain load balancer urls then use this attribute to tell the service to publish the edited wsdl.

It turned out to be *almost* as simple as that … I also had to download the auto generated xsd files and edit them also. So here’s what to do …

1) Configure your service to run on basicHttpBinding with no transport security
2) Browse to the wsdl in the browser on your dev machine and copy it to a text editor.
3) Look in the wsdl file for the urls that end in xsd=xsd0 and browse to these in the browser and copy them to the text editor.
4) *Carefully* edit all urls in the files to point to the load balancer urls.
5) Don’t forget to edit the xsd file urls in the wsdl to where they will be hosted in production.
6) Deploy these files to a server.
7) Configure your mex endpoint with the externalMetadataLocation attribute (in your dev and test environments just leave this attribute out and use the automatic generation)

<behavior name="">
    <serviceMetadata httpGetEnabled="True" externalMetadataLocation="https://MyServiceUrl/Wsdl/theWsdl.wsdl"/>             

Your service will be available on the http schema but will respond to mex requests with the wsdl file containing the https urls to the load balancer. When a request is made the load balancer will decrypt the request then forward it to the service on http. The response will return to the load balancer over http which it will then encrypt and send over https … all is well.

The downside to this approach is, of course, that when new methods are added to the service or changes are made to the signatures of existing methods the wsdl and xsd files must be maintained manually by repeating the process used to create them. Whilst this isn’t too horrible it might catch new developers on the system by surprise so its worth sticking a giant README file in there.

Movember Moustache

So this year I decided to join my collegues and participate in Movember. I have never grown a beard or moustache of any sort but I have a fairly thick beard so I figure this shouldn’t be a problem. Anyway … I’ll post up photos of my progress as I go. If you think my mo is worthy … please donate a small amount to my movember page.

Movember Week 1

Some more mogress has been made …

Movember Week 2

Figured it could use a little streamlining …

Movember Week 2

Moments before it was unceremoniously removed …

Movember Week 4

In all we did really well .. I personally managed to raise €344 with my team of colleagues raising a whopping €7959 and coming first in the country for the team with the highest donation tally. Not a bad effort MoBros :)

XML Serlializer memory leaks

A little while ago at work we received a complaint from the systems team that one of our web applications was increasing in memory usage until it crashed the process … we had a memory leak. Its easy to think that with all the garbage collecting wizardry baked into .NET that a memory leak is not possible … but oh how it is.

We looked through our code for the usual suspects … web requests/responses not being disposed, datareaders, etc. We did find a few issues and cleaned them up and released the “fixed” version. Memory usage slowed slightly but still continued to climb despite our efforts. The plot thickens …

After some digging we found some quite interesting information about, our friend, the XmlSerializer. We had been using an overload of the constructor which allowed us to specify the name of the element when serialising a collection of objects as we could not use the default element. As it turns out, this overloaded constructor was the cause of our memory leaks. To understand why this was the case we need to dig into the inner workings of the XmlSerializer a little but I’ll try to keep this high level.

When you create an XmlSerializer for a given type, the .NET framework generates dlls that contain classes that allow serialization and deserialization of the objects. These dlls are then dynamically loaded into memory. Yes that’s right … they are not generated on disk but loaded straight into memory. So surely, you say, once the dll is no longer required the garbage collector will clean it up. Actually no … the garbage collector works on objects. The dll generated by the XmlSerializer is an assembly which is loaded using the assembly loader … these are out of the control of the garbage collector. Once a dll is loaded it stays in memory until the application stops.

To avoid memory leaks the XmlSerializer caches these assemblies so if one has already been loaded the same one is used over and over. But this only applies for two of the nine constructors that you can use. The other seven constructors will load a new assembly into memory every time they are called. The serializer instance will be disposed but the assembly it generated in memory will persist. This is actually by design … it seems Microsoft did not expect the seven constructors that do not use caching would be called all that much so holding dlls for these serializers in the cache might result in a large cache of objects they may not need any more … so they left them out.

So how do we fix it? Firstly we need to know which constructors are safe to use … these two do not cause memory leaks and can be called repeatedly without concern …

XmlSerializer(Type, String)

We were using the XmlSerializer(Type, XmlRootAttribute) version with the memory leak issue. To achieve our same goal of naming the element that collections of our objects are serialised to while still using one of the safe constructors we had to make a few simple changes.

Code that looked like this …

var people = new List<Person>();
//unsafe constructor
var serializer = new XmlSerializer(typeof(List<Person>), new XmlRootAttribute("people");

Was replaced with code like this …

public class PeopleCollection : List<Person>
//safe constructor
var serializer = new XmlSerializer(typeof(PeopleCollection));

If you absolutely must use one of the unsafe constructors you need to override the caching and implement it yourself to prevent the memory leaks as shown here on msdn.

Tess Ferrandez goes into much detail about the inner working of the XmlSerializer and the causes of the memory leaks in her blog post.

Make the XmlSerializer Your Friend

Over the years I have had to do a number of integrations with third party xml web services. These have included credit card payment gateways, government agencies, hotel booking engines and even game providers. Before all the cool kids jump in and say we should be using json, yes we should in a perfect world where we get to rewrite our software every time that the latest and greatest cool thing comes out. For those of us in the real world, these things still exist and need to be maintained and integrated with … enough said.

There are a few ways to consume xml, some good, some not. Lets have a look …

1) You can use XmlWriter and XmlReader classes to manually parse and write the xml. This method is error prone, time consuming, inefficient and difficult to read and maintain. Each time you modify the model object you also need to modify the serializer and deserializer. The one and only upside is that it gives you complete, utter control over the serialization and you can map to whatever object you like in one go. It is the most infinitely customisable way of consuming xml. I have never actually found the need to do it this way … I suggest you don’t either ;)

2) You could use an XmlDocument object or even LinqToXml. This can be handy if dealing with a simple document or you only want one or two nodes out of a complex xml structure. In general this isn’t much better than manually parsing xml if you are trying to deserialize an entire object and is only a little less verbose. Still not a great idea except for the case where it just a node or two.

3) My preferred option … use the built in XmlSerializer given to you by Microsoft. Its not difficult to use and is easily customised using attributes on your model objects. At the end of the day, the xml is a model of data which can easily be translated into model objects.

Lets start with a sample xml that we want to deserialize. Its the standard cookie cutter person-with-an-address structure but its enough to give the basic idea …

<?xml version="1.0" encoding="utf-16"?>
    <person dateadded="2007-09-24T08:00:00+02:00">
        <street>Some Street</street>
        <suburb>Los Angeles</suburb>
  <person dateadded="2007-09-24T07:00:00+02:00">
      <street>With Leonard</street>
      <suburb>Los Angeles</suburb>

The nice touch about using the serializer is that it almost totally abstracts the fact that you are dealing with xml by mapping to a set of objects that model the data. Here are those models … we have a People class that contains a list of Person classes that each contain an Address class …

    public class PeopleCollection : List<Person>
    public class Person
        public DateTime DateAdded { get; set; }
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public Address Address { get; set; }          
    public class Address
        public string HouseNumber { get; set; }
        public string Street { get; set; }
        public string Suburb { get; set; }
        public string State { get; set; }

Nothing really that special … just a simple set of objects that represent the data contained in the xml. The decorators on each property and class tell the serializer how to serialize and deserialize the values. Does it output the property value as an attribute, an element or simply ignore it altogether? The decorators are optional .. if they are omitted the serializer will expect the xml to have the same elements (case sensitive) as the property names and all properties will be serialized as elements.

In this case I have purposely changed the cases of the xml and the property names to demonstrate the use of decorators to customise the serializer. You may note that in the Person class I have mapped the surname element to the LastName property by using the decorator and I have also specified that DateCreated should be emitted as an attribute. More information about the many decorators that can be used to control serialization can be found here.

From this point the serialization code is straightforward …

    public class Serializer<T> where T : class 
        public string Serialize(T obj)
            var serializer = new XmlSerializer(typeof (T));
            var namespaces = new XmlSerializerNamespaces();
            namespaces.Add(string.Empty, string.Empty);
            var sb = new StringBuilder();
            using (var writer = XmlWriter.Create(sb))
                serializer.Serialize(writer, obj, namespaces);
            return sb.ToString();
        public T Deserialise(string xml)
            var serialiser = new XmlSerializer(typeof (T));
            using (var stream = new MemoryStream(Encoding.Unicode.GetBytes(xml)))
                return serialiser.Deserialize(stream) as T;

The full example solution can be downloaded from GitHub
Happy coding :)

Sexism in IT

There has been a spate of tweets, articles and blog posts in recent times about sexism in the IT industry. They have ranged from the daily niggles of working with predominantly male co-workers to full blown internet lynchings of companies for the use of scantily clad women in advertising or as service staff at events. These are not anything new, these things have been happening in the IT industry for years and are only coming to the surface now because of the rise in numbers of female IT staff in traditionally male fields of expertise. There’s power in numbers and the more women in IT the less tolerated these behaviours will be. And there’s the rub … the solution to sexism in IT is more women in IT and this leads me to the motivation for writing this post.

It’s a well known fact that bad news makes for good reading. This is every bit as true for the online world as it is for traditional media channels. Any women doing any kind of research on the internet as to a career in IT would surely be dissuaded from such a choice. We have to be careful to not discourage women wishing to join the IT industry with horror stories of discrimination and objectification of women. For every company that has objectionable behaviour towards women, there must be hundreds that don’t. There might even be a few that treat their female (and male) staff very well. But somehow these are unworthy of tweets, articles and blog posts that may show the IT industry as being a place where a woman may find herself a nice career.

I have spent the last ten years working for multiple companies in multiple locations around the world and continue to enjoy my career immensely. I feel like the IT industry is in such a dynamic state at the moment with all the new devices and technologies that have emerged in recent times and it’s a volatile and exciting time to be in this field. I would encourage any person with an aptitude for computers to pursue a career in IT regardless of their race, age or gender and I think it’s a shame that some of these people may be discouraged by reading all negativity on the internet.

I am not, in any way, suggesting that the companies that step over the line should not appear in articles, blogs or be tweeted into submission. I am, however, suggesting that we should endeavour to present a more realistic image of the IT industry as a whole. Lets add some balance by also highlighting the companies that treat their female staff really well or even go the extra mile somehow. Lets tweet about what it is that makes us turn up to work every day or to spend our personal time pouring away over open source projects. If you are a female in IT, make yourself known by writing a blog and sharing your experiences, the good ones as well as the bad ones, and why you joined the IT industry in the first place.

In my time I’ve been lucky enough to have worked with a handful of very talented female developers. I even fell in love with one. We need more of these women. We need more of them now and we need more of them in the future. Eventually I really hope that the cause of the recent rants is a thing of the past and we can get back to what we love … coding!

Code Re-use With Abstract Classes

An often overlooked feature of C# (and OO programming in general) is the use of abstract classes to maximise code re-use. Put simply, if you have code in related classes that is the same but needs to call implementation specific methods, you can probably use an abstract class to consolidate this code. Lets dive right in with a simple example.

This is production code that has been stripped back to highlight the use of abstraction. In my scenario I had a number of user controls that had to implement the same functionality to each display three different views. The initialisation code ran through the same method calls in each control but the implementations of the called methods were different. Enter the abstract class …

public abstract class ControlBase 
    public int ViewMode { get; set; }
    protected abstract void SetSaveButtonEnabled();
    protected abstract void SetCollapsedView();
    protected abstract void SetSummaryView();
    protected abstract void SetExpandedView();
    protected abstract void HideAll();
    protected virtual void InitControl()
    protected virtual void SetViewMode()
        switch (ViewMode)
            case 1:
            case 2:
            case 3:
            case 4:

The InitView() and SetViewMode() methods are the same across all controls. Obviously the actual code to render the correct view is specific to the implementation of the control itself. So how does it work? Firstly, an abstract class can never be instantiated … it can only be inherited from. Any methods specific to the implementation are declared as abstract. In the abstract base class we know what the method signature should be but we can’t implement it at this level. Once the signature is provided and declared abstract we can call that method in the abstract class code as demonstrated in the SetViewMode() and InitControl() methods. At runtime this will result in the implementation of the method in the child class being called.

Our inherited class would look something like this …

public class Control1 : ControlBase
    protected override void SetSaveButtonEnabled()
        //control specific code here 
    protected override void SetCollapsedView()
        //control specific code here 
    protected override void SetSummaryView()
        //control specific code here 
    protected override void SetExpandedView()
        //control specific code here 
    protected override void HideAll()
        //control specific code here 

In the child class we override the abstract methods. Failing to override all the abstract methods on the base class will result in a compile time error unless your child class is also abstract.

That’s really all there is to it. By using this technique I saved having the InitControl() and SetViewMode() methods being implemented with the exact same code in all of my child controls. This would at best have been copy-paste coding which is horrible and to be avoided at all times. A bug in one of those methods would have resulted in needing to update all the implementations to fix it. Add the complexity of working in a team and another team member may not realise that the code is repeated in multiple controls. By using the abstract class there is only one implementation to maintain making maintenance much simpler while allowing a much more object oriented design.

More info on MSDN

Radio Buttons and Repeaters in Aspx

If you’re using a Repeater control and have a requirement to have radio buttons in the repeated rows you may find yourself in a head scratching mess. You may find that checking one radio button does not result in all other buttons being unchecked. The radio buttons will essentially start to behave like a bunch of check boxes … not cool.

With normal html radio buttons we group them together using the ‘name’ attribute. All radio buttons with the same value in the name attribute become mutually exclusive resulting in the familiar behaviour where only one radio button in that group can be checked at any one time. In an asp:RadioButton control the name attribute maps to the GroupName parameter which normally works fine.

Things start to go astray when the radio buttons are added to a repeater however. You may be aware that asp.net will insert its own unique id for each control which is not the same as the id you originally gave it in code or markup. It does this to ensure that any elements that have the same id end up with a unique id once rendered. In the case of the repeater this is very necessary as you are taking the same elements with the same id and emitting them multiple times. Unfortunately a repeater also mangles the name attribute of the controls. Normally this can be safely ignored but radio buttons rely on this attribute to work out which group they belong to. All of a sudden each radio button belongs to its own little group and so we end up with check box behaviour.

This is a known issue explained in detail here, however, Microsoft have not yet fixed this and give no indication of if or when it will be fixed.

So now what? Well … there are a couple of work-arounds, none of which are particularly elegant.

1) Avoid using the repeater or radio buttons and see if you can solve the problem with other controls.

2) If you’re stuck with repeaters and radio buttons the most elegant solution is to use javascript to de-select other radio buttons using the click event.

This goes in the <head> tag …

 //I use jquery to make life easier but you don't have to 
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js"></script> 
<script type="text/javascript">
    function MakeRbExclusive(rb) {
        var rbs = $('#rbData :radio');
        for (var i = 0; i < rbs.length; i++) {
            if (rbs[i] != rb) rbs[i].checked = false;

And our repeater …

<div id="rbData">
    <asp:Repeater ID="repeater" runat="server">
            <!-- I have bound to simple Dictionary<string, string> created in page load for example purposes -->
            <asp:Label ID="lbl" runat="server" AssociatedControlID="radio"><%#Eval("key")%></asp:Label>
            <asp:radioButton ID="radio" runat="server" value=<%#Eval("value")%> onclick="MakeRbExclusive(this);"/>
            <br />

3) Avoid the use of a repeater completely and add your controls dynamically from code behind in a loop to achieve the same result. I *really* don’t recommend this. It is finicky at best and takes very careful coding. Dynamically added controls are not automatically tracked in viewstate and can create big problems if not carefully coded. I have had to do this in the past but it is definitely an option I’d only consider once I have exhausted all others.

Happy coding! :)

Barcellona to Morella, Spain

We’re touring through Spain for five weeks avoiding motorways like the black plague. This is the town of Morella in the background … as you can see the bike is fully laden … three box luggage kit, tank bag and we’re two up … you could say we’re making use of the ‘tourer’ aspect of the bike’s design …

Morella, Spain

This video is of us having a little fun on a nice little stretch of road between Barcelona and Morella … I have to give the bike credit here for still handling given the weight its got on it :)

Generic Singelton

We are often required to implement the singleton pattern in our code. While doing this yet again I thought there must be a generic way to do this and I came up with the following code …

public static class Singleton<T> where T : class, new()
   private static volatile T _instance;
   private readonly static object _lockObj = new object();
   public static T Instance
           if (_instance == null)
              lock (_lockObj) if (_instance == null) _instance = new T();
           return _instance;

This allows you to use any class as a singleton …

var mySingleton = GenericSingleton<MySingleton>().Instance;

It’s not perfect … there is nothing stopping the developer from creating a new instance of the class which breaks the singleton pattern. Doing it the normal way makes it impossible to create more than one instance of a class but this way allows the flexibility to use a class as a singleton even if it’s not implemented that way and also allows a mix of singleton and instances of the same class.

In any event it makes interesting use of generics and I just thought it was cool so I thought I’d share :)

Insert Multiple Rows With a Single Insert

I often have to insert multiple rows of (usually) test data into tables. The simplest but most tedious way is to rewrite the INSERT statement over and over with different values …

INSERT INTO User VALUES ('Penny', 'Penny')
INSERT INTO User VALUES ('Sheldon', 'Cooper')
INSERT INTO User VALUES ('Leonard', 'Hofstadter')
INSERT INTO User VALUES ('Howard', 'Walowitz'
INSERT INTO User VALUES ('Rajesh', 'Koothrappali')

Its not a big deal but as a developer I hate writing the same statement over and over … it just goes against my grain. Its also more of an overhead to repeatedly call insert for each row. Here’s a better way …

SELECT 'Penny', 'Penny' UNION ALL
SELECT 'Sheldon', 'Cooper' UNION ALL
SELECT 'Leonard', 'Hofstadter' UNION ALL
SELECT 'Howard', 'Walowitz' UNION ALL 
SELECT 'Rajesh', 'Koothrappali'

This inserts the same data as the first example but uses only one insert so has less overhead. If you happen to be running SQL Server 2008 there’s a new syntax to achieve the same although the above example will still work. In SQL Server 2008 you can use this syntax …

VALUES ('Penny', 'Penny')
, ('Sheldon', 'Cooper')
, ('Leonard', 'Hofstadter')
, ('Howard', 'Walowitz') 
, ('Rajesh', 'Koothrappali')
Return top