Beddets Blog

Code and other things

  • WMI.NET Series Post
  • Twitter
  • LinkedIn Profile

Restoring a database from SQL Azure

Posted by Beddet on December 16, 2015
Posted in: Programming. Tagged: azure, sql. 2 Comments

A while ago I had to restore a backup from SQL Azure on my local machine – I got the backup made (a .bacpac file) and started my usual restore procedure.

I usually use the restore UI when restoring a new backup locally but it’s a bit different when restoring an azure backup.

Right click on “Databases” in the object explorer and choose “Import data-tier application” to bring up the wizard.

Find the backup and choose the database and file paths and start.

Just as simple as restoring a regular backup, however – after a while I got the following error message.

Exception of type ‘System.OutOfMemoryException’ was thrown. (mscorlib)

in the “importing database” step.

Hmm, that was weird. But I could ofcourse have run out of memory, but I had plenty of memory available and my instance was set to use an unlimited amount of memory.

I tried a few more times and kept getting the same error, after a bit of googling I found something called sqlpackage.

I opened up a command line and found the sqlpackage.exe – which is bundled when installing sql server.

In my case it was located here “C:\Program Files (x86)\Microsoft SQL Server\120\DAC\bin”.

Running the following command with a couple of parameters started up the process to import the database, and after a bit it had succeeded

SqlPackage.exe /a:Import /sf:C:\backup.bacpac /tsn:localhost /tdn:databaseName /p:storage=file

/tsn: targetservername
/tdn: target database name
/a action
/sf sourcefile

The parameters are self explanatory, but the documentation is available here https://msdn.microsoft.com/en-us/library/hh550080(v=vs.103).aspx

If sqlpackage is not installed, you can get it via the SQL Server Data Tools download.

Some thoughts and experiences about IEnumerable

Posted by Beddet on October 7, 2015
Posted in: Programming. Tagged: .NET. Leave a comment

Case study of the first time I encountered an issue with IEnumerable usage.

IEnumerable ConsolidateOrders(List orders)
{
	var items = new List();

	//we may have several orders for the same customer in the input list
	foreach(var order in orders) 
	{
		if(order.Customer == null) continue;

		if(!items.Contains(order.Customer)) items.Add(new ConsolidatedOrder(order....)); //
		else
		{
			var item = items.Get(order.Customer);
			item.UpdateStuff();
		}
	}


	var ordersWithoutCustomer = orders.Where(x => x.Customer == null).
		Select(x => new ConsolidatedOrder(order ...., new Customer("empty customer")));

	var result = items.Union(ordersWithoutCustomer);
	result.ForEach(x => x.CalculateInternalMembers());

	return result;
}

This was the case I had and a weird bug had occurred where the internal members where not calculated correctly. The apt reader might have noticed the error, and after a bit of debugging and looking through the code multiple times, I found the error as well.

The problem lies in the type of result, which is an IEnumerable, and in conjunction with the used LINQ expressions, result is lazily evaluated. This means the CalculateInternalMembers is never actually called as result is just a union of a list and another IEnumerable, which returns a lazy evaluated collection.

The last few lines of code is only actually “called” when we start enumerating the object.

If we change the declaration of result to this.

var result = items.Union(ordersWithoutCustomer).ToList();

We now get the expected result.

This is all by design, and also both the good as well as the bad reason for using IEnumerable in your code.

The interface, IEnumerable, only provides a method for getting an Enumerator, which really means all you can expect to do is iterator over it. This is a sequence and can potentially never stop.

Consider the following code.

void Main()
{
	var ints = Get();
}

IEnumerable Get()
{
	while(true) yield return 0;
}

This runs without any problems whatsoever, even though calling Get would never actually stop returning zero. But because it’s in the context of an IEnumerable, it will only evaluate when needed (when MoveNext() is called on the Enumerator from the object).

Next up, add the following – so the main method will look like this.

var ints = Get();
var firstInt = ints.FirstOrDefault();
Console.WriteLine(firstInt);

This will write out “0” to the console, as the enumeration stops after the first element (if the enumerable returns any elements).

If however, you do a .Count(), it will fail with an OverFlowException because it never returns.

Most of the time it’s not going to be a problem, because you’re probably just throwing lists around in your code, so it will have an end, but it’s important to know the ups and downs of IEnumerable. It can really be a problem if the above Get method does something like this, actually returning a finite amount, but having an expensive action.

IEnumerable<Tuple<int, int>> GetCustomers()
{
	var i = 0;
	const int NumberOfCustomers = 10;
	while(i < NumberOfCustomers)
	{
		i++;
		using(var con = new SqlConnection(""))
		{
			con.Open();
			var cmd = con.CreateCommand();
			cmd.CommandText = string.Format("select count(*) from orders where customerid = {0}", i);
			var result = cmd.ExecuteScalar();
			yield return Tuple.Create<int, int>(i, (int)result);			
		}		
	}	
}

Hopefully you’ll never see something like this, but it’s plausible. This presents the next issue, multiple enumerations.

Let’s imagine a case, where we want to do something for each customer that has any orders.

void UpdateCustomers(IEnumerable<Tuple<int, int>> customers)
{
	if(customers.Count() == 0) return;

	foreach(var customer in customers)
	{
		//do stuff
	}

	//notify of changes to customers
}

void Main()
{
	var customers = GetCustomers();
	UpdateCustomers(customers);
}

This doesn’t seem suspicious, we start by checking if there actually are any customers to avoid notifying at the end of the method. Afterwards we loop through the customers and do some stuff (update a status for example).

Because our GetCustomers() method is an IEnumerable and yield returns every result, we will actually do a database lookup a total of 20 times.
First of, this won’t perform as well and the underlying data may actually change throughout the method.

Count() vs Any().

Always use if(.Any()) over (.Count() > 0).

Any will stop as soon as it finds an element and return, .Count will enumerate the entire collection and return. If we replace the .Count() == 0 with a call to .Any() instead, we’ll end up with 11 calls to the database. Definitely better than the 20 we had before, lowering the time also reduces the chance of having the data change.

In this case, we probably know a database connection is being used, and that’s fine, we just don’t want more lookups than necessary. We could obviously change the sql query and method a bit to only do one lookup. But changing the UpdateCustomers method to not accept an IEnumerable, but require a list instead, solves the problem. The caller is now forced to decide if they want the entire collection (by calling .ToList()) or to only take a limited number of elements.

I’ve previously used IEnumerable everywhere, but after experiencing this issue I’ve been more careful. I like to use the lowest common denominator for my dependencies, this includes the collection type, often I just want to loop over the collection once, because IList is a bigger interface and has methods for adding and removing items, it may not be needed and just hinders the usage of my code.

But as we’ve seen, this can cause some problems. So I’m probably going to use ICollection or simply IList a lot more to avoid any possible problems where I’m not limiting the input.

It does however also come with a perk, if you don’t need the entire collection or just want to know if it’s empty or not. Having an IEnumerable is great as you’ll only get what you need. If you need everything, just use a list.

The moral of the story is, be mindful of what you expect.

Why I don’t use things like automapper

Posted by Beddet on August 23, 2015
Posted in: Programming. Tagged: code, opinion. Leave a comment

First off, to clarify – this is by no means a way to bash on a various libraries and I’m just using automapper as a reference as I briefly talked about this with a colleague and some new interns.

I’ve talked about ORM’s before and this post will have some similarities.

As I will be using automapper as a reference in this post, I’ll just give a very brief description about what it does.

In LoB applications, like the one I work on, we have a couple of layers where we move objects around (using typical customer, order system).
We have a wpf client, here we either use DTO’s directly or wrap objects in view models, we then have a webservice that talks to the database etc. The setup looks a bit like this – for a list of customers, going from server to client.

CustomerEntity -> CustomerDTO -> CustomerViewModel and back again.

To view and save a customer we have to do 4 separate mappings (excluding the database layer).

Mapping each property of the entity to a dto and then to a viewmodel and back is boring work which could clearly be automated.
You can obviously cut off some work by making sure the code to map is in a factory of sorts, if it’s ever needed in more than one place, but it’s still a lot of work – this becomes annoying and error prone when new properties are added.

This is where automapper comes in. You can setup a Map like so and then do the mapping in that class, but we still get a new class with manual mappings. If the scenario is as simple as the one I’m using here, properties will have the same type and name, everything can then be done by conventions and reflection (magic) – thus saving time.

There are probably more things it can do, I don’t really use it and know what it really can do, but all in all it’s pretty smart – no doubt about that.

I wrote the word “magic” in there, a word that, whenever I see it, I think to myself, hmm.

If we use the convention based setup (which is what we actually gain from), what happens when a property some where is renamed? It stops working, and will only fail when the code is actually hit. I like having a compiler, it stops me from doing idiotic things, a compiler won’t help you here.

A “solution” is to have a test to make sure every property is mapped correctly, writing some reflection yourself.

I put solution in quotation because tests need to be run and may not always be possible and if possible I’d rather get my errors exposed when I hit compile than when I run a test – very subjective.

Next, it’s harder to know which properties are used, if only used indirectly – I’ve seen the same problems with IoC containers where registrations have been deleted or even the implementation and things stop working (still compiling).

Removing complexity may be an argument, but I disagree, it merely moves it to another place.

Saving time, yeah sure it’s boring to write this code, but it’s not like it’s hard or takes a lot of time. Take a class with 100 properties you need to map, will take a couple of minutes with copy paste and find/replace. And of course every minute spent on boilerplate, is a minute that could be used for something else, but it’s not like you’re going to spend many hours doing this in every project.

I want to know how things work, at least at the surface level. Introducing frameworks to help with anything, especially minor things, just wastes my time. I’ve never actually setup any automapper stuff, only added a property or two to existing setups, so I don’t actually know how it works, what it can do to save my time. This is a problem with any framework, you need to know how it works to use it, at least some of it. Automapper is as far as I know not a huge framework, and quite simple as well, but in principle there are lots of things you need to know to be able to use it (big ORM’s is a prime example).

Then we have the dependency hell, this is a potential problem with any dependencies (https://en.wikipedia.org/wiki/Dependency_hell), especially with many small projects and several solutions referencing the same projects or other versions of same packages.

This was a bit of a bashing, but just my thought. I try to stay away from a lot of these types of libraries, at work it’s obviously harder but I’d much rather be able to solve the problem in a simple way, that can be used in every language/framework. If I were to switch to a whole other language than C#, it’d be more tedious if I relied too heavily on specific libraries.

Edit: I completely forgot about this earlier, I knew I was missing something but I couldn’t for the life of me remember what. But performance.

I’m not saying this performs badly, but it’s impossible for any sort of reflection to be as fast as handwritten and compiled code. I know it’s a very minor thing, but why write something you know won’t perform as well as something else, which is just as easy to write?

SQL Rally Nordic 2015

Posted by Beddet on March 5, 2015
Posted in: Programming. Tagged: conference, sql, t-sql. Leave a comment

Wauw, back after three days in Copenhagen, and my mind is still trying to get some decent structure of everything I’ve heard these last couple of days.

Sunday, the day we went to Copenhagen seemed quite long, I knew I had to get ready but I’m a procrastinator by heart and soul, so things were basically ready but not yet packed. My brother was going to pick me up shortly after 5PM, he just needed to pick up a colleague first. Alright, so my deadline was set, however I had an appointment at 2, which I thought would only take 60-90 minutes, but at 4.17 we were done. Hurry to get home, pack the last things and get ready. Got to the hotel, checked in, watched a bit of TV (haven’t done that in over a year, and to no surprise, there wasn’t really anything worth watching), early to bed as well so I would be ready for a day in the company of Itzik Ben-Gan and his Practical T-SQL – Efficient Solutions precon session.

I met one of my colleague when checking in, but didn’t really know when they were going to setup our stand and all that, so I asked and christ that was early 😦

We met at 6.45 at the congress center, which luckily was the same building as the hotel (yay!), but I got up and went to meet them and help setup a few things before heading back to the hotel for some breakfast, a quick shower and a last minute nap.

Off to the first day of sessions, I’d brought my pc in case we were going to need it, but that wasn’t the case, anyway I think there were around 30 people to listen to Itzik.

Links:
APPLY operator with Itzik
resources from Itzik

I won’t go much into the details of his sessions here as most of it is available at the link above. But in essence, he was talking about the APPLY table operator and sequences. I also attended his session dedicated to sequences.

This covered little over half the day, but the rest started to seem quite too theoretical for my taste and knowledge of databases, set math, trees and so on so I didn’t see the whole session, instead I just went back to reflect on what I’ve heard.

That was it for the first day, lots if new things to think about. All in all, a good day.

I’ll write a bit about the other two days as well as more technical stuff in a coming days / weeks.

Why I will never use an ORM unless my boss tells me to

Posted by Beddet on November 24, 2014
Posted in: Programming. Tagged: .NET, code, nhibernate, opinion, orm, sql. Leave a comment

Alright, this post is somewhat of a rant of the title is a bit misleading, in this circumstance, my boss, can mean project lead, client requirement etc.

Also, as a colleague just pointed out, this is not a rant about micro ORMs like Dapper, but only regarding the big ones like nhibernate or entityframework.

My colleagues will be able to testify to this, I whine a lot about ORMs and I thought it’d might be a good idea to get these thoughts out in the open.

First off, a couple of good things, well at least one good thing. After the initial setup, working with an ORM is extremely efficient (from a developer point of view), the setup can be a bit daunting and the first issue I have is that some frameworks tend to play by convention over configuration rules, I am strongly against this. Conventions can be good, if they are widely understood, accepted and followed, however what basically happens whenever conventions kick in, is magic. As a developer, I dislike magic, we’ve all heard and seen magic numbers throughout code and how terrible that is. Even if by chance no one ends up ruining something, it’s just harder to see and understand what is going on.

if(statusId == 2) ...

What does that do? No one knows, a comment might be present but shouldn’t be necessary to understand code, for business logic, sure, but not for the code itself. What does the number 2 mean?

Okay, that was getting a bit off topic. But the solution to this is quite easy, throw that number into a constant or an enum (if there is more than one value).

But not everything is as easy to solve or understand as that, especially when it’s a framework doing this.

At work we’re using nhibernate for the project I’m on and how it works is we make a mapping between a C# class and a table in the database, this is done using an XML file, the first (and worst) problem with an xml file is, it’s harder to check for usages and refactor, particular renaming of properties. This has of course been solved and in enter fluent nhibernate, which allows us to write our mappings in C#, strongly typed etc. This leads to many a line like this.

Map(x => x.ProductId, "ProductId");
Map(x => x.ProductName, "ProductName");

Can’t there be a less verbose way of doing this? Certainly, we can setup a convention which essentially does that work for us, behind our back. This works due to, well conventions, so as long as our property name and column name matches, we’re good to go. In general this is not a problem and just saves the developer a lot of time writing boring boilerplate stuff, but when shit hits the fan (or new developers join the project), we’re in for quite a ride.

My biggest concern is when writing queries to get data out of the database, it seems that we (developers) are lazy as fuck and fear doing a bit of work, especially in something is new and crazy as SQL. It must be so much better to use a third party framework that can and will change syntax and behavior “every” version. Most people working with software, or have studied it, know SQL, at least enough to write a simple SELECT … FROM … JOIN.

Let’s see an example of a typical boring Customer, Order, OrderLine system, the system is up and running and everything is well, but we need to add another simple view, only showing a list of every product a given customer has ever ordered.

Given this signature let’s try a very simple (bad) nhibernate implementation.

public List<ProductDto> GetProducts(int customerId)
{
    var results = new List<ProductDto>();
    var ordersFromCustomer = Session.Get<Customer>(customerId).AllOrders;
    foreach(var order in ordersFromCustomer)
    {
        foreach(var orderLine in order.OrderLines)
        { 
            var productId = orderLine.Product.ProductId;
            if(!results.Any(x => x.Id == productId))
                results.Add(new ProductDto(){ Id = productId, Name=orderLine.Product.ProductName }); 
        } 
    }
    return results;
}

This looks innocent enough, right? But depending on how the mappings are written, as a customer makes more and more orders and we add more properties we could potentially load everything from the database, and in the process also spend a lot of time hydrating the properties on our objects.

No matter how bad that performs, it works and if it’s possible to change things within a couple of days if the performance is too bad, sure, go ahead and throw it in production. That didn’t take more than a couple of minutes to write and that was not in an IDE.

Alright, we’ve discovered, that this doesn’t perform as well as we’d hoped, but it has run for a short while until we’ve gotten more data, so now we need to optimize it, we can start looking at cache, lazy loading etc, but that could potentially bring other issues to the table, it’s better to only optimize this one method.

I see a couple of ways to do this. First we can simply rewrite it using nhibernate criterias, projections and all sorts, which basically ends up writing a regular join and only selecting the needed properties. Don’t get me wrong, this works and it works really well, but unless you’re really proficient in the given framework (which may be changed at any time) it will take a while to write. I can’t do it by hand and it will probably take me 2-3 hours to get it to work. This option is also very error prone if the framework or structure changes, but let’s not worry about that, we live in a perfect world 🙂

The next thing we could do is create a SQL view, which does the simple join, something like this.

SELECT DISTINCT p.ProductId, p.ProductName, o.CustomerId FROM OrderLines AS ol
INNER JOIN Orders AS o ON ol.OrderId = o.OrderId
INNER JOIN Products AS p ON ol.ProductId = p.ProductId

Now we have a simple view, create a new entity specific for this scenario and map it to this view, now let’s see how the method would look.

public List<ProductDto> GetProducts(int customerId)
{
    var results = new List<ProductDto>();
    var productsFromCustomer = Session.QueryOver<UniqueProducts>().Where(x => x.CustomerId == customerId).List();
    foreach(var product in productsFromCustomer)
    {
         results.Add(new ProductDto(){ Id = product.ProductId, Name=product.ProductName }); 
    }
    return results;
}

Alright, now it’s a lot simpler and we’ve let our database handle the data, doing the join and only finding distinct products.

If we use nhibernate, this is the approach I would personally take. The last option is to forgo the entire nhibernate layer and manually wiring up a sql connection, sending a command etc, either using a view with an added WHERE o.CustomerId == customerId or a stored procedure.

It will certainly be tricker to do regular debugging if all of the work is in the database, however I am of the persuasion that the database should handle the data queries and let the application take care of the business logic. Most developers know SQL, and I think the above SQL statement would work on most SQL engines, not just Microsoft SQL Server. There is a very small chance that we will change the underlying database for the same data, but a framework like nhibernate might be swapped out for something else, as of writing this the development on nhibernate has stopped and we’ve already had several issues where fluent nhibernate simply does not support our entity structure.

The last point, that I can think of at this very moment is that not everybody has used a specific framework before. Hiring a .NET developer means that they will know how to write code in .NET, C# for example. If they have some form of education they will have worked with SQL. Bringing in too many frameworks can overload anybody if they’re new to them.

Applying IoC in an ASP.NET application.

Posted by Beddet on December 9, 2013
Posted in: Programming. Tagged: .NET, asp.net mvc, code, daily blog december, ioc. 2 Comments

Disclaimer: I haven’t really done any asp.net in a long time, so my way might not be the way to go.

This is going to be one of the shorter posts.

So we want to build a website using asp.net mvc and if we look at examples found around the internet, many of them just uses database access directly in the controllers, something like this. http://nerddinnerbook.s3.amazonaws.com/Part4.htm

public class DinnersController : Controller 
{
	DinnerRepository dinnerRepository = new DinnerRepository();

	public void Index() 
	{
		var dinners = dinnerRepository.FindUpcomingDinners().ToList();
	}
}

And as a start, this isn’t really an issue, but we have probably learned from other projects that we might want to abstract our services and use DI. This way the controllers aren’t responsible for knowing how to get the data, they just require some implementation of an interface which has the method FindUpcomingdinners().

Great, so we move the repository to an interface and add it to the constructor.

public class DinnersController : Controller
{
	private readonly IDinnerRepository _dinnerRepository;
	
	public DinnersController(IDinnerRepository dinnerRepository) //I omit a null check, so we only have the important parts
	{
		_dinnerRepository = dinnerRepository;
	}
	
	public void Index() 
	{
		var dinners = _dinnerRepository.FindUpcomingDinners().ToList();
	}
}

But how is the controller created? This basically happens behind our back, as it should, at least in my opinion. In a typical desktop application we have a main method which sets everything up and opens a window, we would then have an IoC container available to the main method, which takes care of newing up everything we need.

But where is this magical place you ask, it’s in the global.asax of course!

A fresh mvc website will have a class that looks something like this inside.

public class MvcApplication : System.Web.HttpApplication
{
	public static void RegisterGlobalFilters(GlobalFilterCollection filters)
	{
		filters.Add(new HandleErrorAttribute());
	}

	public static void RegisterRoutes(RouteCollection routes)
	{
		routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

		routes.MapRoute(
			"Default", // Route name
			"{controller}/{action}/{id}", // URL with parameters
			new { controller = "Home", action = "Index", id = UrlParameter.Optional } // Parameter defaults
		);
	}

	protected void Application_Start()
	{
		AreaRegistration.RegisterAllAreas();

		RegisterGlobalFilters(GlobalFilters.Filters);
		RegisterRoutes(RouteTable.Routes);
	}
}

But hey,I don’t see any place the controllers get registered, so how can I possibly plug into this system?

Implement the DefaultControllerFactory, we must.

In here there is one method we should override.

public override IController CreateController(System.Web.Routing.RequestContext requestContext, string controllerName) { }

That method will be used every time asp.net tries to get a controller, before we see how. Let’s make sure we use our own implementation.

In the Application_Start() method we simply add, as the very first thing.

ControllerBuilder.Current.SetControllerFactory(new ControllerFactory());

Now to the more interresting part, the implementation.

public class ControllerFactory : DefaultControllerFactory
{
	public override IController CreateController(System.Web.Routing.RequestContext requestContext, string controllerName)
	{
		string key = controllerName.ToLowerInvariant();
		IController controller = GetController(key);
		return controller ?? base.CreateController(requestContext, controllerName);
	}

	private IController GetController(string key)
	{
		switch(key)
		{
			case "dinners":
				return new DinnersController(new DinnersRepository());
				break;
			default:
				 return new HomeController(); //default homepage with no dependencies
				 break;
		}
	}
}

Alright, now we take care of creating the controllers, and calling the base.CreateController method, we can hand over the control to asp.net again.

Now we have moved the creation of the controller to our own code, which makes us responsible for doing it correctly, but also gives us the opportunity to make good use of DI.

This example isn’t using an IoC container, and in this project we only have one controller with one dependency so it’s ok. But we could simply have an IoC container in this class and resolve dependencies based on the controller name.

Beware when using tools to auto generate test data.

Posted by Beddet on December 5, 2013
Posted in: Programming. Tagged: daily blog december, tools, unit tests. Leave a comment

We all write automated tests of some sort, it’s not always the most entertaining things to do, especially when dealing with legacy code, but oh how awesome it is, once they’ve been written and we’re ready to refactor or add new functionality to a piece of code.

One of the worst things to deal with when creating brand new tests for old code, is setting up all that test data. Especially things we don’t really care about in a specific test.

Let’s look at the case of a shop with customers, products, transactions etc. We want to test the calculation of the price for a transaction. The transaction entity has a method to do this, we just want to test it.

Right, first step is to make a transaction object, oh it has a constructor taking in 10 parameters for different non-nullable objects, which all in return take several other arguments. All we need to do the calculation is the products, eventual discount and the number of each product purchased, but we still need to create a customer, billing info and a lot of other things we don’t really care about.

Actually creating dummy objects for this might end up taking tens of minutes to setup.

Sure we could mock our way through some of these things, but that still requires some work.

What to do? We go grab the nearest tool for automization of course!

Then we might be able to auto-generate the transaction, with a couple of real objects as well, products and discounts in this case.

Call the Calculate method and verify it works. Great, it works and we can go grab another cup of coffee.

Now comes the tricky part, what if something unexpected value gets generated once in a blue moon, that screws up our test? Dates are probably the perfect example of things that can make things go bonkers.

The calculate methods also takes into account which date the transaction was purchased on, so if it was purchased on christmas eve it’s 20% cheaper and we haven’t really thought about that in this case, we’re only testing for regular product price and discount. A simple piece of math says that once every 365.Th run, we will end up with the special christmas discount, and the test fails.

We know about this special offer and we have other tests for that, but to get that date in might not be as simple as sending in a simple DateTime object to our generator.

If it is, then great as soon as we see the test fail and find out why, we fix it. However, if the test needs to be run a couple thousand times on average for the failure to appear, we might never really notice it.

Now we have a basically broken test, because we can’t really count on what it says.

If we can’t do the simple thing, then we can still fix it and just take that extra time to create the whole transaction by hand and we’re good.

My point is, be careful when writing tests and using auto generated data, the more static it is the better as the tests will be consistent. I’m not saying that you should avoid it altogether, just be careful, and if you suspect something like that might happen – try to run the test in a loop and see if you can force the error to occur before it has been around for too long.

Code practices from the functional paradigm.

Posted by Beddet on December 5, 2013
Posted in: Programming. Tagged: .NET, code style, daily blog december, standards. Leave a comment

This post won’t be a “this is a way to solve this specific problem”, but more a general talk about some coding styles, my opinions.

One of the things that are often brought up when talking functional programming opposite typical OOP languages is mutability.

Most of us can probably agree, that having mutable objects where in the object itself changes over time can result in weird bugs happening at seemingly random times. It’s not as much of a problem in a single threaded application, but now a days where multi threading and parallelism is much more commonly used, these things are more likely to occur, unless we pay sufficient heed.

Immutable objects are easy to work with, it will never be changed, no matter what you do. If you think you’re changing the object, it’s just making a new one for you instead.

That’s the simple introduction, now let’s get down to some examples and more interresting things.

Side effects are bad mmkay?

Disclaimer: Everything is being exaggerated, probably even to the extremes, but the points will still hold true.

Example, we have a simple Computer class, containing hardware specs, as disk, motherboard, but also ranks for how power efficient it is etc. These are set via the constructor and can’t be changed directly from the outside, only through given methods.

We can change every piece of hardware via public void methods, such as.

public void ChangeGraphicCard(GraphicCard card);

We’d of course expect that the graphic card property to get changed. But what about the internal rankings?

The internal ranks could be calculated on demand, but if the specs rarely change, might as well calculate it when updating the hardware. And here comes the side effect, when we change a piece of hardware, the rankings change as well. This makes sense, but calling a setter, we don’t really expect anything else to change. The consequences could be dire, although this exact case it’s not that bad.

I once learned, sadly I can’t remember where, that every method should return something, and passing objects to a method for the purpose of changing the input without returning anything seems wrong to me.

I quite often see something like this, especially when creating viewmodels.

while(startDate < endDate)
{
	CreateItems(startDate, myViewModelCollection, myOtherViewmodelCollection, 42, "myMagicString");
	startDate = startDate.AddDays(1);
}

The first time this runs, both of the collections might be empty, but after I’ve called the method, they might have objects in them. Those collections might not even be method scoped, but rather declared as fields used in other places as well. Again, this isn’t directly dangerous in itself, but add in multiple callers to that method, or even remove the collection parameters and just have the method use those automatically, we end up in the case of not being able to predict what happens to those collections.

Wouldn’t it be better to simply return new collections with the newly created objects and after that, update the actual collections with the new items after we’re done and know the output?

“But I need to return two collections then, that’s impossible!”, well yes, but there are ways to handle that, wrap the result in its own object or even use a Tuple (I don’t recommend using Tuples unless neccessary).

In my opinion that’s a much better solution and I strive to do things that way, more immutable and closer to the pure functional programming paradigm. See http://en.wikipedia.org/wiki/Purely_functional

If a function does not change anything else, we can use it freely without the entire system breaking down on us, using return values also help to clarify what is actually being done.

At least this is my opinion, I think I haven’t really gotten my point across as clearly as I’d hoped, so I expect there to be a part two some day, with some better examples as well.

To show an example of how I would’ve done it.

while(startDate < endDate)
{
	var itemsForDate = CreateViewModelsForDate(startDate, 42, "myMagicString");
	
	myViewModelCollection.Add(itemsForDate.PrimaryViewModel);
	myOtherViewModelCollection.AddRange(itemsForDate.OtherViewModels);
	
	startDate = startDate.AddDays(1);
}

private ItemViewModelCreationResult CreateViewModelsForDate(DateTime day, int numberOfItems, string name)
{
	List<OtherViewModel> otherViewModels = new List<OtherViewModel>();
	PrimaryViewModel viewModel = new PrimaryViewModel(name);
	
	for(int i = 0; i < numberOfItems; i++)
	{
		var otherViewModel = new OtherViewModel(viewModel);
		otherViewModels.Add(otherViewModel);
	}
	
	return new ItemViewModelCreationResult(viewModel, otherViewModels);
}

internal class ItemViewModelCreationResult
{
	public readonly PrimaryViewModel PrimaryViewModel;
	public readonly IEnumerable<OtherViewModel> OtherViewModels;
	
	public ItemViewModelCreationResult(PrimaryViewModel primaryViewModel, IEnumerable<OtherViewModel> otherViewModels)
	{
		if(primaryViewModel == null) throw new ArgumentNullException("primaryViewModel");
		if(otherViewModels == null) throw new ArgumentNullException("otherViewModels");
		
		this.PrimaryViewModel = primaryViewModel;
		this.OtherViewModels = otherViewModels;
	}
}

Granted, the data doesn’t really have any meaning, but an example should be simple.

I’ve created an internal class to hold my result in, having readonly fields I make sure I can only set them through the constructor, and here I can make sure I don’t have invalid data.

As the result class only makes sense in this context I make in internal, or even private inside my class. Also the method to create the view models have been renamed to have a clearer meaning, and since I don’t actually need to know about other viewmodels when creating the new ones, there’s no need to pass them on.

I would even go as far as to move the Create method into an entirely new class and also extract the parameters into an object.

AOP with interceptors

Posted by Beddet on December 4, 2013
Posted in: Programming. Tagged: .NET, AOP, daily blog december. 3 Comments

Todays topic is on AOP, more specifically implementing cross cutting concerns.

For those who doesn’t know AOP stands for Aspect Oriented Programming, as a sort of alternative to OOP, here we work with aspects. I won’t really go into AOP, mostly because I haven’t really done anything in this style before, but it basically means that we want to do stuff on aspects instead of an object.

When talking about cross cutting concerns, examples usually circles around one of three typical scenarios, where we want some sort of behavior across the different layers/objects in a system.

The three mostly used examples are:

  • Logging
  • Security
  • Auditing

Depending on the system, we might want, or even need, some sort of security or auditing but we probably always want logging in some form or another.

We all know how annoying implementing logging can be, especially “late” in a project, there are several logging frameworks available that essentially does what you need, but you still have to go through your entire codebase to add logging.

A commonly used framework is the log4net framework, for .NET applications, this can be used by having a static field in your class calling a factory method to get a logger, which you can then use in each of your classes to get consistent logging. Great, but this creates a very hard dependency which we might not want, and what is the primary solution to removing hard dependencies? Abstract it of course!

And as some wise man once said, the only problem in programming that can’t be solved by abstraction, is too many layers of abstraction. By extracting the logging functionality into an internal interface we then need to use that, probably by DI. Now we have a strict dependency as well, this time just not on an external system. This solves some of the problems.

The next issue is having to write code in each of the methods we want logging on, if we use a regular object for logging, we need to call that explicitly each time we want something logged.
By doing this, we also have the problem with inconsistency throughout the codebase, this might not really be an issue, but it makes running through logs a lot easier.

Introducing cross cutting concerns.

aop diagram

So we want some consistent logging through out several layers in our application without doing too much code and with the possibility to control it, for example have a lot of logging for development, and less for production.

It’s a common practice to use an IoC container for DI and having some sort of bootstrapper to handle this, we probably have interfaces for most of the things we want logging for anyways, so we’re already halfway there!

There are frameworks for implementing this in different ways. Spring and Postsharp
I’ve used Castle Windsor as my IoC container before and Castle also provides a dynamic proxy generator, which is what we’re after, together with interceptors.

Interceptors basically intercept method invocations, as the name implies 🙂

I’ve made a LoggerInterceptor like so, implementing a simple interface provided by Castle which has only one method, Intercept.

internal class LoggerInterceptor : IInterceptor
{
    public void Intercept(IInvocation invocation)
    {
        StringBuilder sb = new StringBuilder();

        string methodName = GetMethodName(invocation); //is method is just for providing a formatted version of the method name including the name of the class

        try
        {
            invocation.Proceed(); //call the method
        } catch(Exception e)
        {
            Console.WriteLine("Exception caught while calling {0}, \n {1}", methodName, e.ToString()); //log the error
            throw; //rethrow the error so the rest of the application can handle it.
        }
    }

    private string GetMethodName(IInvocation invocation)
    {
        StringBuilder methodNameBuilder = new StringBuilder();
        methodNameBuilder.Append(invocation.Method.DeclaringType);
        methodNameBuilder.Append(".");
        methodNameBuilder.Append(invocation.Method.Name);
        return methodNameBuilder.ToString();
    }
}

This is a very, very simple interceptor, we just wrap the entire call in a try catch and log any exception thrown. But we have plenty of opportunities to have more advanced logging done. When we’re done writing the interceptor we need to use it somehow, we add this to our IoC container for the interfaces we want logging on and it will create a dynamically generated decorator class.

This of course doesn’t really do much and it will intercept EVERY method on the interfaces this is registered for, but we can select which methods to intercept and even do stuff based on the parameters.

This is where it gets funky, I have also written this little method for arguments passed on to the invocation.

private string FormatMethodParameters(IInvocation invocation)
{
    StringBuilder sb = new StringBuilder();
    ParameterInfo[] parameters = invocation.Method.GetParameters();

    for (int i = 0; i < invocation.Arguments.Length; i++)
    {
        sb.Append(parameters[i].ParameterType.Name);
        sb.Append(" ");
        sb.Append(parameters[i].Name);
        sb.Append(" value: ");
        sb.Append(invocation.GetArgumentValue(i));
        sb.Append(";");
    }

    return sb.ToString();
}

So now we can intercept every method call and we can actually get the values of all of the parameters in each method, this is really nifty. If a method fails occasionally, based on some certain input, for example a null string, we can choose to log the parameters when it fails to find out why, without having to do trial and error.

A null value isn’t much fun, what if it only fails on any odd integer between 20 and 30 , that would be very hard to find out, unless we can see in our logs, that every time this throws an exception the input has a value of an odd number between 20 and 30. Now, we have the input which makes it fail and it’s extremely easy to reproduce.

We can of course check the name of the method, attributes on it or basically what ever we want, to see if we actually want to do anything.

Any number of interceptors can be added, so everything can be chained for maximum customization.

For registering this, I have used a factory method instead of the more traditionally IoC way, but still using a container most things.

public IMyCustomService GetService()
{
    var service = _container.Resolve();
    return new ProxyGenerator().CreateInterfaceProxyWithTarget(service, new IInterceptor[] { new LoggerInterceptor() });
}

When calling GetService I now get a proxy which does whatever my interceptor chooses to do.

Of course, we could also make manual decorators and wrap our logging logic into one class again, even though this would be quite simple and less tight coupled than manual ILogger dependency in each class, we now have everything in one place with basically no work.

And with a bit more setup in the container, we can have logging for every single interface we have, in one way and only one place to edit later on. Mission accomplished.

SQL Server data types

Posted by Beddet on December 3, 2013
Posted in: Programming. Tagged: daily blog december, sql. 2 Comments

Alright, this is the first post in a long time, I really do post way to randomly, oh well this is the first post for my daily-blog-December; where I will try my best to post one new post each day.

This one is dedicated to SQL Server, just to get things started the right way 🙂

So most of us use some sort of database and many of those use SQL Server, as we all know there are different types and sizes for those and it’s a very good idea to spend some time thinking about using the right type for the right job, as it is in every typed language.

Choosing the right type isn’t really that hard, do you need to store an integer, cool use an int. Need to store some text then either varchar or nvarchar is a good choice, but what about the size?

I often use http://technet.microsoft.com/en-us/library/ms187752.aspx this as a good reference tool.

For small systems, sure it doesn’t really matter if you’re using 10mb of database storage or 12mb, but as systems grow older and larger, every optimization is welcome.

Take for example the decimal type (http://technet.microsoft.com/en-us/library/ms187746.aspx), there are different sizes to it. It defaults to be a “decimal(18,0) null”.

This means the column can hold a number with a maximum of 18 digits, precision of 18 and a scale of 0, an example would be.

create table Numbers
( number decimal)

insert into Numbers (number) VALUES (15.3)

The saved number is 15 and not 15.3, so the decimal type isn’t worth much without a scale > 0.

Case:

Imagine a system with a customer, orders and products, we’re designing the database and are working on the products table, we’re currently thinking about the price. Right, so we will probably want to use a decimal, fine, but which size? We need to think about what the absolute highest price ANY future product would have. The system is currently being made for a grocery shop, so as of now, nothing costs more than 100 dollars and we usually have a scale of 2 for currency. So a decimal(5,2) should be efficient. Should we use this? Well it can contain less numbers than decimal(8, 2), so it should be smaller in size, right?

The answer is, no. No matter the content, if a decimal has a between 1 and 9 it will always take up the same amount of space, which is 5 bytes.

Reference: http://technet.microsoft.com/en-us/library/ms187746.aspx

Seeing these numbers, we might think to ourselves, “hey, let’s just use a decimal(9,2) then, this allows us to have even higher prices in the future without having to change the database and it will take up the same amount of storage”. This is a fair assumption, but thinking further down the road, we could make it even bigger and store up to 19 digits for only 4 bytes more per row. Doing a bit of math, this sounds like the plan to go for.

5 bytes * 100,000 rows is roughly half a megabyte and using 9 bytes instead will still keep it less than one megabyte. Do we think that we’ll ever go beyond 100,000 products? Probably not in this system, but that is just for one column, if the price per product takes up that amount, we need an even bigger one for keeping a total per order, and we might also want a higher scale if we want products costing several millions.

Add in indices and several other similar colums and it’s not the same case anymore, and this is just for a grocery shop (granted very expensive food 🙂 ), but what if we want this to be customized to a movie production system where we want to save the cost of actors etc, then it’ll be a lot easier to have a big number and never have to change.

End of case “study”.

As we can see, space wise it doesn’t make that much of a difference, but think of a site like amazon or ebay that has enormous amounts of products and transactions rolling, all of which needs to have enough precision as to not lose anything in the process and it will make a difference.

The next thing to think about is, it’s always easier and safer to scale up than down, and storage is cheap, so it’s much better to chose a small size and then scale up when and if it’s needed.

But, beware the side effects of changing this. Best case scenario, we only have to change it in one place, the given table, worse, we have to change several change scripts, application code, database code, SSIS packages, reports etc and make sure everything is running smoothly after the upgrade, without having bad data in your hand.

As mentioned in the the case above, we should also think about indices and performance. For example, any char/string type with a length higher than 900 can’t be indexed.

Thinking about all of these small things can be useful in certain cases and knowing the bounderies of the change of storage required is great, so your 1tb database doesn’t suddenly go to 2tb just because you’re changing to an type bigger than needed. Also, no need for a 1tb database, if you can cut it down to 700mb. Size does matter after all 🙂

Numbers and strings aren’t the only things to think about of course, if you really really need that extra performance/storage or whatever, then you probably need to look even further into the internals, but I won’t go there.
Dates are things to watch out for as well. If you only need the date, use a date type instead of a datetimeoffset, which is 7 bytes bigger than date (at 3 bytes); both are fixed sizes.

So choose correctly and remember, it’s always easier to make things bigger than shaving off data in the end 🙂

Posts navigation

← Older Entries
  • Search

  • Categories

    General Programming Uncategorized
  • Tags

    .NET .NET 2.0 AOP asp.net mvc azure code code style conference daily blog december dynamic general intellisense ioc learning to code miracle nhibernate no zebra open source opinion orm powershell presenting projects remote school shortcuts sql standards t-sql toolbox tools unit tests win32 windows service WMI wmi.net work wql
  • My tweets

    Tweets by beddet
  • Archive

    • December 2015 (1)
    • October 2015 (1)
    • August 2015 (1)
    • March 2015 (1)
    • November 2014 (1)
    • December 2013 (5)
    • August 2013 (1)
    • May 2013 (1)
    • May 2012 (1)
    • April 2012 (3)
    • March 2012 (1)
    • February 2012 (3)
    • January 2012 (2)
    • November 2011 (3)
    • September 2011 (1)
    • April 2011 (2)
    • December 2010 (1)
    • November 2010 (1)
    • September 2010 (2)
  • Meta

    • Create account
    • Log in
    • Entries feed
    • Comments feed
    • WordPress.com
Create a free website or blog at WordPress.com.
Beddets Blog
Blog at WordPress.com.
Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
  • Subscribe Subscribed
    • Beddets Blog
    • Already have a WordPress.com account? Log in now.
    • Beddets Blog
    • Subscribe Subscribed
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
Loading Comments...
Design a site like this with WordPress.com
Get started