The talk that excited me most this year was an introduction to U-SQL by Michael Rys. U-SQL is a new querying/processing language that acts as a composite of SQL and C#, allowing you to call C# code from within SQL-like queries. You can write C# code in-line in queries, or call it from referenced assemblies. Here’s a sample query from https://azure.microsoft.com/en-gb/documentation/articles/data-lake-analytics-u-sql-get-started/

DROP FUNCTION IF EXISTS Searchlog;

CREATE FUNCTION Searchlog() 
RETURNS @searchlog TABLE
(
            UserId          int,
            Start           DateTime,
            Region          string,
            Query           string,
            Duration        int?,
            Urls            string,
            ClickedUrls     string
)
AS BEGIN 
@searchlog =
    EXTRACT UserId          int,
            Start           DateTime,
            Region          string,
            Query           string,
            Duration        int?,
            Urls            string,
            ClickedUrls     string
    FROM "/Samples/Data/SearchLog.tsv"
USING Extractors.Tsv();
RETURN;
END;

For the most part it looks like standard T-SQL, but you see that call to “Extractors.Tsv()” near the bottom? That’s C# code. In this case it’s an inbuilt function for U-SQL, but it could be any C# you wrote or referenced. You can also see some handy inbuilt features that allow it to easily extract data from files (the demo took them from an Azure Blob Storage account).

U-SQL acts as Data Processing As A Service – you write your U-SQL query, submit it to Azure, then wait for the results. You pay per query, and it handles all infrastructure orchestration. You don’t need to set up machines – it also handles parallelization automatically for you!

Microsoft are positioning this as a Big Data solution designed to be used with the new Azure Data Lake features, but I think it has the potential to be used in a wide variety of scenarios.

It’s very early days, but it’s something I’ll be definitely keeping my eye on.

In the first part of this series I gave a brief overview of Serilog and Seq and what differentiates them from logging you may have done in the past.  One of the most useful aspects of structured logging is the ability to trace the path of any particular call to your site, including key variables along the way. This post will explain how to set that up using Owin and middleware.

A key requirement in entry point logging is standardisation – all entry point logs should be generated by the same code if they are to be universally searchable (as well as being good SOLID practice!).

Example

Here is an example of a static entry point logger we are using, it is very simple code and uses the class SiteAccessDetails (which is added to our ClaimsPrincipal in middleware – see this blog for details on how to do that) to supply a shopName to be used as additional context.

public static class EntryPointLogger
{
	public static IDisposable AddEntryPointContext(ILogger logger, SiteAccessDetails siteAccessDetails, string entryPointName)
	{
		var disposable = LogContext.PushProperty("HttpRequestId", Guid.NewGuid());

		if (!string.IsNullOrWhiteSpace(siteAccessDetails?.ShopName))
			LogContext.PushProperty("Shop", siteAccessDetails.ShopName);

		if (!string.IsNullOrWhiteSpace(entryPointName))
			LogContext.PushProperty("entryPointName", entryPointName);

		logger.Information("Entry Point: {entryPointName} called", entryPointName);

		return disposable;
	}

	public static IDisposable AddEntryPointContext(ILogger logger, string shopNameOrUrl, string entryPointName)
	{
		var userSiteDetails = new SiteAccessDetails(shopNameOrUrl, null, null);
		return AddEntryPointContext(logger, userSiteDetails, entryPointName);
	}

	public static IDisposable AddEntryPointContext(ILogger logger, string entryPointName)
	{
		return AddEntryPointContext(logger, (SiteAccessDetails)null, entryPointName);
	}
}

We can call this manually from entry points in our code (Web API controllers etc) by adding a line like the following

var disposableLogContext = EntryPointLogger.AddEntryPointContext(logger, siteAccessDetails, "/api/customer/info");

to the start of the controller action and

disposableLogContext.Dispose();

to the end, where logger has been defined in the constructor and passed in by IOC  or defined directly in the class as a private variable by using

private readonly ILogger logger = Log.ForContext<CustomerController>();

Using Owin

..but this means remembering to add it to every entry point that you create.  A better way if you are using Owin is to define a piece of middleware that calls it for you. Using the standard template for adding middleware…

public class AppendEntryPointToSerilogContextMiddleware
{
	private readonly Func<IDictionary<string, object>, Task> _next;
	private readonly ILogger logger = Log.ForContext<AppendEntryPointToSerilogContextMiddleware>();

	public AppendEntryPointToSerilogContextMiddleware(Func<IDictionary<string, object>, Task> next)
	{
		_next = next;
	}

	public async Task Invoke(IDictionary<string, object> env)
	{
		var userSiteDetails = ClaimsPrincipal.Current.GetSiteDetails();
		object requestPath;
		env.TryGetValue("owin.RequestPath", out requestPath);

		using (EntryPointLogger.AddEntryPointContext(logger, userSiteDetails, requestPath?.ToString()))
		{
			await _next(env);
		}
}

Which is invoked in the configuration method of your startup.cs class

 public void Configuration(IAppBuilder app)
{
  ...
  app.Use(typeof(AppendEntryPointToSerilogContextMiddleware));
  ...
}

Notice that my code gets the site details from the ClaimsPrincipal as mentioned above and gets the path of the entry point from the Owin  RequestPath.

You will now get every call to your API fully logged with the path and in this case, the shop.

If you have entry points to your code that do not go through a request – we have subscriptions to an Azure Service Bus Topic for instance – you can just add the manual call to the entry point logger at the start.

Try it!

This is an extremely useful method for tracking live code which we now use extensively. If it has given you food for thought, give it a try, it’s really simple to use.

Get Serilog from here, get Seq from here

It used to be that logging was a bit of an after thought where a few statements such as ‘Connecting to the database’ or ‘Invalid user login’ were written to a text file that was rarely looked at.  Well things have moved on a fair bit since then.

Part 1 of 2.  Go here for part 2

Serilog and Seq

Serilog is a modern structured logging solution that allows you to add context to your logs so that you can easily track a particular execution path.  Seq (pronounced ‘seek’) gives you the ability to host, display and search those logs.

For instance, imagine a customer calling and saying that when they log into your swanky portal and go to their profile page they don’t see their address.  With structured logging you could easily highlight all the logged api calls made by that customer, filter to the profile page and follow the execution path through to the address query to see the actual data returned.

So simple!

To add a context to Serilog and log to it, you just use

LogContext.PushProperty("customerName", customerName);

Any further logging will now happen in that context, such as

logger.Information("Getting customer profile");
...
logger.Information("Customer address: {customerAddress}", customerAddress);

These two informational logs will now be tagged with the customer name. Notice also how Serilog uses a format similar to C# 6.0 string interpolation to embed variables. Doing it this way means you can filter by the variable in your logs.

An example

Here is a snapshot of a single log entry on our Seq server.

A seq view of a serilog entry, highlighting the use of a tagged context.

It is from a system we created to track users through online shops and it highlights several things;

• We are logging the entry point of all our API calls
• We have set tagging on Seq to display the entry point name at the beginning of the log entry (in this case it is ‘api/customers’) – this will then be displayed at the beginning of every subsequent log entry created during the execution of that particular api
• We have assigned a unique id to every httpRequest that comes through which can also be tagged or filtered
• We can also tag or filter by the Shop that was being tracked by the system

If I tell Seq to limit the log display to only ‘api/customer’ and also to filter by the httpRequestId I get the following (click to view full size);

Tracking the path of an api call through your code using Serilog and Seq

I can now see that the call to ‘customers/api’ generated three further calls from our server to external apis (to our analytics server). Clicking on any of those will show me the details of the call or the response as we embedded the information in the log.

If I had instead chosen to tag and filter by the shop name I could see all activity related to a particular online shop.

What is great about this is how easy it is to quickly narrow down the exact log entries you are looking for and having done so, inspect the state of whatever variables you chose to embed in your log.

Go try it yourself!

I strongly recommend you to go and try out both Serilog and Seq.  In the second part of this blog I show how we used Owin and middleware to create the entry point logging.

If you want to know even more, take a look this Pluralsight course.

Have fun!

Recently I was working on a web api 2.0 project that used a bearer token for authentication and passed a UserId in the claims for the generated ClaimsPrincipal.

Each controller in the project was accessing the User property from the ApiController base class to get the UserId from the claims and creating various other data to pass to a CQRS style query.  Some (simplified)  sample code follows;

var userId = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == ClaimTypes.NameIdentifier);
var userRegion = GetUserRegionForUser(userId);
var userAccountName = GetUserAccountNameForUser(userId);
... Call an external query with userRegion and userAccountType to assemble data to return

Doesn’t seem too bad, but this kind of code was littered throughout the project and in some cases only the UserId was passed to the external query which was then getting the other data itself.  Clearly we needed a unifying UserData class.

The UserData was only ever going to be three simple pieces of information (UserId, UserRegion and UserAccountName) and seemed like an ideal candidate for adding directly to the ClaimsPrinipal. You might think that a custom ClaimsPrincipal that inherits from a base class and adds UserData would be the way to go, but on his blog, Dominick Baier, suggests that deriviation is not ideal and I tend to agree.

You could simply add the three pieces of information as separate claims and then read them back again in your controller:

var userId = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == ClaimTypes.NameIdentifier);
var userRegion = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == "UserRegion");
var userAccountName = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == "UserAccountName");

but this approach is messy and doesn’t give us much more than the code we are trying to change.

Using extensions

The answer, as suggested in Dominick’s blog, relies on extension methods. Careful use of these allows us to not only get and set the UserData class on the principal but also to check for its existence.

public static class ClaimsPrincipalExtensions
{
  private const string UserRegion = "UserRegion";
  private const string UserAccountName = "UserAccountName";

  public static bool HasUserData(this ClaimsPrincipal principal)
  {
    return principal.HasClaim(claim => claim.Type == UserRegion);
  }

  public static UserData GetUserData(this ClaimsPrincipal principal)
  {
    var userId = principal.Claims.SingleOrDefault(x => x.Type == ClaimTypes.NameIdentifier)?.Value;
    var userRegion = principal.Claims.SingleOrDefault(x => x.Type == UserRegion)?.Value;
    var userAccountName = principal.Claims.SingleOrDefault(x => x.Type == UserAccountName)?.Value;

    return new UserData(userId, userRegion, userAccountName);
  }

  public static void AddSiteDetails(this ClaimsPrincipal principal, UserData userData)
  {
    principal.Identities.First().AddClaim(new Claim(ClaimTypes.NameIdentifier, siteAccessDetails.UserId));
    principal.Identities.First().AddClaim(new Claim(UserRegion, userData.UserRegion));
    principal.Identities.First().AddClaim(new Claim(UserAccountName, userData.UserAccountName));
  }
}

Getting the UserData

Now we can access the UserData class from the principal with

var userData = ((ClaimsPrincipal)User).GetUserData();

or, not relying on the ApiController base class

var userData = ClaimsPrincipal.Current.GetUserData();

Creating the claims transformation

But how do we add the claims in the first place, especially considering we are using Owin middleware? The easiest way is to use the freely available NuGet package IdentityModel Owin ClaimsTransformation (the simple source for which is available here if you want to hand craft it).

Having installed this package, add a claims transformer class:

public interface IClaimsTransformer
{
  Task<ClaimsPrincipal> TransformWithSiteDetails(ClaimsPrincipal principal);
}

public class ClaimsTransformer : IClaimsTransformer
{
  private readonly IUserDataProvider userDataProvider;

  public ClaimsTransformer(IUserDataProvider userDataProvider)
  {
    this.userDataProvider = userDataProvider;
  }

  public Task<ClaimsPrincipal> TransformWithUserData(ClaimsPrincipal principal)
  {
    if (principal.Identity.IsAuthenticated && !principal.HasUserData())
    {
      principal.AddUserData(userDataProvider.GetDataByClaimsPrincipal(principal));
    }
    return Task.FromResult(principal);
  }
}

Here I am passing in a UserDataProvider that can get me the user data from wherever it originates and using that to populate the principal.

Integrating with Owin middleware

Next add the transformer to your Startup.cs class (after your authentication middleware)

app.UseClaimsTransformation(incoming => Container.Resolve<IClaimsTransformer>().TransformWithUserData(incoming));

Note that I am resolving the claimsTransformer from our IOC container, you can do it whichever way you choose.

And that’s it! Your authenticated controllers will now have the UserData contained in the ClaimsPrincipal and can easily be accessed or tested for.

I hope that helps.

In my first article I outlined fluent builders – what they look like and how to put them together. This time I will go a little deeper to show you just how versatile they can be.

Method Naming

You can be as flexible as you like in naming the methods in your builders. A standard starting place is to use ‘With…’ for example WithFirstName(..), WithDateOfBirth(..) etc. I find the best naming convention is one that makes your tests as easy to read as possible.  For instance

Customer testCustomer = new CustomerBuilder().
    .WithName("Peter Perfect")
    .Aged(32)
    .ThatIsAFirstTimeBuyer();

Remember that it is fine to add specific methods such as ‘ThatIsAFirstTimeBuyer()’ above where it could be setting an individual property, a series of properties or calling further methods within the customer class to create that state. You can also add specific tests for your builders to ensure those methods are working as expected.

Adding Lists

Your builder methods can also take in lists or add to lists. Let’s say that our CustomerBuilder allows us to add products they have bought.  It could be created in two ways;

public CustomerBuilder WithProduct(Product product)
{
    this.customer.Products.Add(product);
    return this;
}

or

public CustomerBuilder WithProducts(List products)
{
    this.customer.Products = products;
    return this;
}

In the first case we could use a product builder to add each product to the customer

Customer customer = new CustomerBuilder()
    .WithProduct(new ProductBuilder().WithName("Apple")))
    .WithProduct(new ProductBuilder().WithName("Orange"))

In the second we could create a specific ProductListBuilder()

public class ProductListBuilder
{
    private readonly List products;

    public ProductListBuilder()
    {
        this.products = new List();
    }

    public static implicit operator List(ProductListBuilder builder)
    {
        return builder.products;
    }

    public ProductListBuilder WithProduct(string productName)
    {
        this.products.Add(new Product { Name = productName });
        return this;
    }
}

Which we would use this way

    List<Product> products = new ProductListBuilder().WithProduct("Apple").WithProduct("Orange");
    Customer customer = new CustomerBuilder().WithProducts(products);

Builders for Mocked Objects

Do you often have to mock repositories?  How about using a builder to do it?

public class MockCustomerRepositoryBuilder
{
    private readonly Mock<irepository> mockCustomerRepository;

    public MockCustomerRepositoryBuilder()
    {
        this.mockCustomerRepository= new Mock<irepository>();
    }

    public Customer SavedCustomer { get; private set; }

    public static implicit operator Mock<irepository>(MockCustomerRepositoryBuilder builder)
    {
        return builder.mockCustomerRepository;
    }

    public MockCustomerRepositoryBuilder WhereGetByIdReturns(Customer customer)
    {
        this.mockCustomerRepository.Setup(cfr => cfr.GetById(It.IsAny())).Returns(customer);
        return this;
    }

    public MockCustomerRepositoryBuilder WithSave()
    {
        this.mockCustomerRepository.Setup(cfr => cfr.Save(It.IsAny())).Callback(cfr => { SavedCustomer = cfr; });
        return this;
    }
}

The usage is as follows

    var mockCustomerRepository = new MockCustomerRepositoryBuilder()
        .WithSave()
        .WhereGetByIdReturns(new Customer().WithName("Tom Test"));

What is really cool about this is that you can set up the save method in your mocked repository to save whatever you pass in to a public property. This allows you to assert that your code under test has actually passed the right object through.  You can also direct it to return whatever you like from the GetById() method of this repository so that you don’t have to set it up in each test.

I hope this has helped you to further appreciate the power of these builders.  Please feel free to ask any questions you may have below.  Next time I will show how to test MVC controllers with builders in a fluent way.

Thanks for listening.

No, I don’t mean the kind that come around and drink all your coffee / tea. When you are testing your code you often have to initialise various classes beforehand. There are several ways to do this. Consider this simple api that takes in a ‘Booking’ object and returns the cost of the booking.

public int GetCostOfBooking(Booking booking)
{
    // ... implementation not important here
}

Where the Booking object is

public class Booking
{
    public DateTime FirstDay { get; set; }

    public DateTime LastDay { get; set; }

    public decimal LocationCostPerNight { get; set; }

    public Customer Customer { get; set; }
}

and the Customer object is

public class Customer
{
    public decimal PercentDiscount { get; set; }
}

Your test could be set up as follows

[TestMethod]
public void BookingCostIsCorrect()
{
    var customer = new Customer { PercentDiscount = 10 };
    var booking = new Booking
                      {
                          FirstDay = DateTime.Parse("2015/08/15"), 
                          LastDay = DateTime.Parse("2015/08/20"), 
                          LocationCostPerNight = 10, 
                          Customer = customer
                      };
    var actual = GetCostOfBooking(booking);
    var expected = 45;
    Assert.AreEqual(expected, actual);
}

(please bear with me on these primitive samples and tests!)

Now this doesn’t look too bad, but what if I add more properties to Customer and Booking or force the use of constructors and specific set methods?

public class Booking
{
    public DateTime FirstDay { get; private set; }
    public DateTime LastDay { get; private set; }
    public decimal LocationCostPerDay { get; private set; }
    public Customer Customer { get; private set; }

    public void SetCostPerDay(decimal costPerDay)
    {
        this.LocationCostPerDay = costPerDay;
    }

    public void SetFirstDay(DateTime firstDay)
    {
        this.FirstDay = firstDay;
    }

    public void SetLastDay(DateTime lastDay)
    {
        this.LastDay = lastDay;
    }

    public void SetCustomer(Customer customer)
    {
        this.Customer = customer;
    }
}

public class Customer
{
    public Customer(string firstName, string lastName, DiscountScheme discountScheme)
    {
        this.FirstName = firstName;
        this.LastName = LastName;
        this.DiscountScheme = discountScheme;
    }

    public string FirstName { get; private set; }

    public string LastName { get; private set; }

    public DiscountScheme DiscountScheme { get; private set; }
}

public class DiscountScheme
{
    public decimal PercentDiscount { get; private set; }

    public void SetDiscount(decimal discount)
    {
        this.PercentDiscount = discount;
    }
}

Now my test setup becomes

[TestMethod]
public void BookingCostIsCorrect()
{
     var discountScheme = new DiscountScheme();
     discountScheme.SetDiscount(10);
     var customer = new Customer("Johnny", "BigCheese", discountScheme);
     var booking = new Booking();
     booking.SetCustomer(customer);
     booking.SetFirstDay(DateTime.Parse("2015/08/15"));
     booking.SetLastDay(DateTime.Parse("2015/08/15"));
     booking.SetCostPerDay(10);
     var actual = GetCostOfBooking(booking);
     var expected = 45;
     Assert.AreEqual(expected, actual);
}

Oh, not so nice any more. I have polluted my test with superfluous information, I am relying on calling the right methods to initialise properties and I have made the test that bit more obfuscated.

Instead, how about this

[TestMethod]
public void BookingCostIsCorrect()
{
    var customer = new CustomerBuilder()
                        .WithDiscount(10);

    var booking = new BookingBuilder()
                        .FromFirstDay("2015/08/15")
                        .ToLastDay("2015/08/19")
                        .ForCustomer(customer)
                        .AtCostPerDay(10);

    var actual = GetCostOfBooking(booking);
    var expected = 45;
    Assert.AreEqual(expected, actual);
}

Much nicer! It is easy to see what we are testing and how it is set up. There is no unnecessary information here. Also, thanks to the use of a builder, all the code for creating those objects is in a central place so you don’t have to edit all your tests when there is a change in how they should be initialised.

Great!  How do I do it?

It’s really not that difficult and once you have written a few you start to get the hang of it.  For instance here are the builder classes we have used above

public class BookingBuilder
{
    private Booking booking;

    public BookingBuilder()
    {
        this.booking = new Booking();
    }

    public static implicit operator Booking(BookingBuilder builder)
    {
        return builder.booking;
    }

    public BookingBuilder ForCustomer(Customer customer)
    {
        this.booking.SetCustomer(customer);
        return this;
    }

    public BookingBuilder FromFirstDay(string firstDay)
    {
        this.booking.SetFirstDay(DateTime.Parse(firstDay));
        return this;
    }

    public BookingBuilder ToLastDay(string lastDay)
    {
        this.booking.SetLastDay(DateTime.Parse(lastDay));
        return this;
    }

    public BookingBuilder AtCostPerDay(decimal costPerDay)
    {
        this.booking.SetCostPerDay(costPerDay);
        return this;
    }
}

public class CustomerBuilder
{
    private Customer customer;

    public CustomerBuilder()
    {
        this.customer = new Customer("Test", "Test", new DiscountScheme());
    }

    public static implicit operator Customer(CustomerBuilder builder)
    {
        return builder.customer;
    }

    public CustomerBuilder WithDiscount(decimal discount)
    {
        this.customer.DiscountScheme.SetDiscount(discount);
        return this;
    }
}

The main thing to remember when creating a fluent builder is that your methods return the class itself, that allows you to chain them. Note also the static implicit operator that converts between the builder and the object it is building. Adding that means we can directly use the builder in place of the object being built (ie passing the booking builder into the GetCostOfBooking method.

I mean this blog to be an introduction to fluent builders – you can get much more fancy with them than this and I have written a blog here that shows how to do that.

If you have any questions or want to add your own feedback, please comment below.  We would love to hear from you.

The Null-Conditional Operator

Here is a small piece of code that I inherited recently.  It has a very simple task, get the client’s postcode or return ‘unknown’

public string GetPostCodeOfClientsOffice(Client client)
{
  if ((client == null) || (client.Office == null)
   || (client.Office.Address == null) || (client.Office.Address.PostCode == null))
  {
    return "unknown";
  }
  return client.Office.Address.PostCode;
}

how much nicer it becomes when we use the new null conditional operator

public string GetPostCodeOfClientsOffice(Client client)
{
 return client?.Office?.Address?.PostCode ?? "unknown";
}

Note that the type of client?.Office?.Address?.PostCode is nullable String.

This isn’t restricted to properties, the following method can also be refactored

public int GetNumberOfClientsAwatingFeedback(List clients)
{
 if (clients != null) {
  return clients.Where(client=>client.NeedsFeedback).Count;
 }
 return 0;
}

which is expressed more simply as

public int GetNumberOfClientsAwatingFeedback(List clients)
{
 return clients?.Where(client=>client.NeedsFeedback).Count ?? 0;
}

Nameof Expressions

When we have methods that expect non null properties we might typically see code like this

public MemberParty GetMainClientAffiliation(Client client)
{
 If (client == Null)
  throw new ArgumentNullException("client");
 If (client.MemberParty == Null)
  throw new ArgumentNullException("MemberParty ");
 return client.MemberParty ;
}

Notice here that the method refers to client affiliation and the property it returns is ‘MemberParty ‘. Not a very ubiquitous approach.

When we refactor to use affiliation however, the following occurs

public Affiliation GetMainClientAffiliation(Client client)
{
 If (client == Null)
  throw new ArgumentNullException("client");
 If (client.Affiliation == Null)
  throw new ArgumentNullException("MemberParty ");
 return client.Affiliation;
}

Aargh, the dreaded magic string has caught us out.  C# 6.0 to the rescue

public Affiliation GetMainClientAffiliation(Client client)
{
 If (client == Null)
  throw new ArgumentNullException(nameof(client));
 If (client.Affiliation == Null)
  throw new ArgumentNullException(nameof(client.Affiliation));
 return client.Affiliation;
}

Now we have strongly typed properties and safe refactoring. Not a massive advance but something that has caught me out many times over the years.

Expression bodied properties and functions

Let’s say we have the following code

public class Client
{
 public string FirstName { get; set; }
 public string LastName { get; set; }

 public string FullName { 
  get{ 
   return string.Format("{0} {1}",FirstName,LastName); 
  }
 }
}

we can now rewrite this as

public class Client
{
 public string FirstName { get; set; }
 public string LastName { get; set; }

 public string FullName => string.Format("{0} {1}",FirstName,LastName);
}

Note that these are not lambda expressions, they are just syntactic sugar intended to simplify your code and make it more readable.

String Interpolation

even better, we can change this to

public class Client
{
 public string FirstName { get; set; }
 public string LastName { get; set; }

 public string FullName => $"{FirstName} {LastName}";
}

Much more readable!

Also if we wanted to override the ToString method for this class, we might use several new features together in the following way

 public override string ToString() => $"Full name: {FullName} City: {Address?.City ?? "Unknown"}";

These are a few of the new language features. Like I said at the start, most of the effort this time around went into Roslyn but I really appreciate that the subtle changes they did make add to the beauty of the language.