The talk that excited me most this year was an introduction to U-SQL by Michael Rys. U-SQL is a new querying/processing language that acts as a composite of SQL and C#, allowing you to call C# code from within SQL-like queries. You can write C# code in-line in queries, or call it from referenced assemblies. Here’s a sample query from https://azure.microsoft.com/en-gb/documentation/articles/data-lake-analytics-u-sql-get-started/

DROP FUNCTION IF EXISTS Searchlog;

CREATE FUNCTION Searchlog() 
RETURNS @searchlog TABLE
(
            UserId          int,
            Start           DateTime,
            Region          string,
            Query           string,
            Duration        int?,
            Urls            string,
            ClickedUrls     string
)
AS BEGIN 
@searchlog =
    EXTRACT UserId          int,
            Start           DateTime,
            Region          string,
            Query           string,
            Duration        int?,
            Urls            string,
            ClickedUrls     string
    FROM "/Samples/Data/SearchLog.tsv"
USING Extractors.Tsv();
RETURN;
END;

For the most part it looks like standard T-SQL, but you see that call to “Extractors.Tsv()” near the bottom? That’s C# code. In this case it’s an inbuilt function for U-SQL, but it could be any C# you wrote or referenced. You can also see some handy inbuilt features that allow it to easily extract data from files (the demo took them from an Azure Blob Storage account).

U-SQL acts as Data Processing As A Service – you write your U-SQL query, submit it to Azure, then wait for the results. You pay per query, and it handles all infrastructure orchestration. You don’t need to set up machines – it also handles parallelization automatically for you!

Microsoft are positioning this as a Big Data solution designed to be used with the new Azure Data Lake features, but I think it has the potential to be used in a wide variety of scenarios.

It’s very early days, but it’s something I’ll be definitely keeping my eye on.

In the first part of this series I gave a brief overview of Serilog and Seq and what differentiates them from logging you may have done in the past.  One of the most useful aspects of structured logging is the ability to trace the path of any particular call to your site, including key variables along the way. This post will explain how to set that up using Owin and middleware.

A key requirement in entry point logging is standardisation – all entry point logs should be generated by the same code if they are to be universally searchable (as well as being good SOLID practice!).

Example

Here is an example of a static entry point logger we are using, it is very simple code and uses the class SiteAccessDetails (which is added to our ClaimsPrincipal in middleware – see this blog for details on how to do that) to supply a shopName to be used as additional context.

public static class EntryPointLogger
{
	public static IDisposable AddEntryPointContext(ILogger logger, SiteAccessDetails siteAccessDetails, string entryPointName)
	{
		var disposable = LogContext.PushProperty("HttpRequestId", Guid.NewGuid());

		if (!string.IsNullOrWhiteSpace(siteAccessDetails?.ShopName))
			LogContext.PushProperty("Shop", siteAccessDetails.ShopName);

		if (!string.IsNullOrWhiteSpace(entryPointName))
			LogContext.PushProperty("entryPointName", entryPointName);

		logger.Information("Entry Point: {entryPointName} called", entryPointName);

		return disposable;
	}

	public static IDisposable AddEntryPointContext(ILogger logger, string shopNameOrUrl, string entryPointName)
	{
		var userSiteDetails = new SiteAccessDetails(shopNameOrUrl, null, null);
		return AddEntryPointContext(logger, userSiteDetails, entryPointName);
	}

	public static IDisposable AddEntryPointContext(ILogger logger, string entryPointName)
	{
		return AddEntryPointContext(logger, (SiteAccessDetails)null, entryPointName);
	}
}

We can call this manually from entry points in our code (Web API controllers etc) by adding a line like the following

var disposableLogContext = EntryPointLogger.AddEntryPointContext(logger, siteAccessDetails, "/api/customer/info");

to the start of the controller action and

disposableLogContext.Dispose();

to the end, where logger has been defined in the constructor and passed in by IOC  or defined directly in the class as a private variable by using

private readonly ILogger logger = Log.ForContext<CustomerController>();

Using Owin

..but this means remembering to add it to every entry point that you create.  A better way if you are using Owin is to define a piece of middleware that calls it for you. Using the standard template for adding middleware…

public class AppendEntryPointToSerilogContextMiddleware
{
	private readonly Func<IDictionary<string, object>, Task> _next;
	private readonly ILogger logger = Log.ForContext<AppendEntryPointToSerilogContextMiddleware>();

	public AppendEntryPointToSerilogContextMiddleware(Func<IDictionary<string, object>, Task> next)
	{
		_next = next;
	}

	public async Task Invoke(IDictionary<string, object> env)
	{
		var userSiteDetails = ClaimsPrincipal.Current.GetSiteDetails();
		object requestPath;
		env.TryGetValue("owin.RequestPath", out requestPath);

		using (EntryPointLogger.AddEntryPointContext(logger, userSiteDetails, requestPath?.ToString()))
		{
			await _next(env);
		}
}

Which is invoked in the configuration method of your startup.cs class

 public void Configuration(IAppBuilder app)
{
  ...
  app.Use(typeof(AppendEntryPointToSerilogContextMiddleware));
  ...
}

Notice that my code gets the site details from the ClaimsPrincipal as mentioned above and gets the path of the entry point from the Owin  RequestPath.

You will now get every call to your API fully logged with the path and in this case, the shop.

If you have entry points to your code that do not go through a request – we have subscriptions to an Azure Service Bus Topic for instance – you can just add the manual call to the entry point logger at the start.

Try it!

This is an extremely useful method for tracking live code which we now use extensively. If it has given you food for thought, give it a try, it’s really simple to use.

Get Serilog from here, get Seq from here

It used to be that logging was a bit of an after thought where a few statements such as ‘Connecting to the database’ or ‘Invalid user login’ were written to a text file that was rarely looked at.  Well things have moved on a fair bit since then.

Part 1 of 2.  Go here for part 2

Serilog and Seq

Serilog is a modern structured logging solution that allows you to add context to your logs so that you can easily track a particular execution path.  Seq (pronounced ‘seek’) gives you the ability to host, display and search those logs.

For instance, imagine a customer calling and saying that when they log into your swanky portal and go to their profile page they don’t see their address.  With structured logging you could easily highlight all the logged api calls made by that customer, filter to the profile page and follow the execution path through to the address query to see the actual data returned.

So simple!

To add a context to Serilog and log to it, you just use

LogContext.PushProperty("customerName", customerName);

Any further logging will now happen in that context, such as

logger.Information("Getting customer profile");
...
logger.Information("Customer address: {customerAddress}", customerAddress);

These two informational logs will now be tagged with the customer name. Notice also how Serilog uses a format similar to C# 6.0 string interpolation to embed variables. Doing it this way means you can filter by the variable in your logs.

An example

Here is a snapshot of a single log entry on our Seq server.

A seq view of a serilog entry, highlighting the use of a tagged context.

It is from a system we created to track users through online shops and it highlights several things;

• We are logging the entry point of all our API calls
• We have set tagging on Seq to display the entry point name at the beginning of the log entry (in this case it is ‘api/customers’) – this will then be displayed at the beginning of every subsequent log entry created during the execution of that particular api
• We have assigned a unique id to every httpRequest that comes through which can also be tagged or filtered
• We can also tag or filter by the Shop that was being tracked by the system

If I tell Seq to limit the log display to only ‘api/customer’ and also to filter by the httpRequestId I get the following (click to view full size);

Tracking the path of an api call through your code using Serilog and Seq

I can now see that the call to ‘customers/api’ generated three further calls from our server to external apis (to our analytics server). Clicking on any of those will show me the details of the call or the response as we embedded the information in the log.

If I had instead chosen to tag and filter by the shop name I could see all activity related to a particular online shop.

What is great about this is how easy it is to quickly narrow down the exact log entries you are looking for and having done so, inspect the state of whatever variables you chose to embed in your log.

Go try it yourself!

I strongly recommend you to go and try out both Serilog and Seq.  In the second part of this blog I show how we used Owin and middleware to create the entry point logging.

If you want to know even more, take a look this Pluralsight course.

Have fun!

Recently I was working on a web api 2.0 project that used a bearer token for authentication and passed a UserId in the claims for the generated ClaimsPrincipal.

Each controller in the project was accessing the User property from the ApiController base class to get the UserId from the claims and creating various other data to pass to a CQRS style query.  Some (simplified)  sample code follows;

var userId = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == ClaimTypes.NameIdentifier);
var userRegion = GetUserRegionForUser(userId);
var userAccountName = GetUserAccountNameForUser(userId);
... Call an external query with userRegion and userAccountType to assemble data to return

Doesn’t seem too bad, but this kind of code was littered throughout the project and in some cases only the UserId was passed to the external query which was then getting the other data itself.  Clearly we needed a unifying UserData class.

The UserData was only ever going to be three simple pieces of information (UserId, UserRegion and UserAccountName) and seemed like an ideal candidate for adding directly to the ClaimsPrinipal. You might think that a custom ClaimsPrincipal that inherits from a base class and adds UserData would be the way to go, but on his blog, Dominick Baier, suggests that deriviation is not ideal and I tend to agree.

You could simply add the three pieces of information as separate claims and then read them back again in your controller:

var userId = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == ClaimTypes.NameIdentifier);
var userRegion = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == "UserRegion");
var userAccountName = ((ClaimsPrincipal)User).Claims.SingleOrDefault(x => x.Type == "UserAccountName");

but this approach is messy and doesn’t give us much more than the code we are trying to change.

Using extensions

The answer, as suggested in Dominick’s blog, relies on extension methods. Careful use of these allows us to not only get and set the UserData class on the principal but also to check for its existence.

public static class ClaimsPrincipalExtensions
{
  private const string UserRegion = "UserRegion";
  private const string UserAccountName = "UserAccountName";

  public static bool HasUserData(this ClaimsPrincipal principal)
  {
    return principal.HasClaim(claim => claim.Type == UserRegion);
  }

  public static UserData GetUserData(this ClaimsPrincipal principal)
  {
    var userId = principal.Claims.SingleOrDefault(x => x.Type == ClaimTypes.NameIdentifier)?.Value;
    var userRegion = principal.Claims.SingleOrDefault(x => x.Type == UserRegion)?.Value;
    var userAccountName = principal.Claims.SingleOrDefault(x => x.Type == UserAccountName)?.Value;

    return new UserData(userId, userRegion, userAccountName);
  }

  public static void AddSiteDetails(this ClaimsPrincipal principal, UserData userData)
  {
    principal.Identities.First().AddClaim(new Claim(ClaimTypes.NameIdentifier, siteAccessDetails.UserId));
    principal.Identities.First().AddClaim(new Claim(UserRegion, userData.UserRegion));
    principal.Identities.First().AddClaim(new Claim(UserAccountName, userData.UserAccountName));
  }
}

Getting the UserData

Now we can access the UserData class from the principal with

var userData = ((ClaimsPrincipal)User).GetUserData();

or, not relying on the ApiController base class

var userData = ClaimsPrincipal.Current.GetUserData();

Creating the claims transformation

But how do we add the claims in the first place, especially considering we are using Owin middleware? The easiest way is to use the freely available NuGet package IdentityModel Owin ClaimsTransformation (the simple source for which is available here if you want to hand craft it).

Having installed this package, add a claims transformer class:

public interface IClaimsTransformer
{
  Task<ClaimsPrincipal> TransformWithSiteDetails(ClaimsPrincipal principal);
}

public class ClaimsTransformer : IClaimsTransformer
{
  private readonly IUserDataProvider userDataProvider;

  public ClaimsTransformer(IUserDataProvider userDataProvider)
  {
    this.userDataProvider = userDataProvider;
  }

  public Task<ClaimsPrincipal> TransformWithUserData(ClaimsPrincipal principal)
  {
    if (principal.Identity.IsAuthenticated && !principal.HasUserData())
    {
      principal.AddUserData(userDataProvider.GetDataByClaimsPrincipal(principal));
    }
    return Task.FromResult(principal);
  }
}

Here I am passing in a UserDataProvider that can get me the user data from wherever it originates and using that to populate the principal.

Integrating with Owin middleware

Next add the transformer to your Startup.cs class (after your authentication middleware)

app.UseClaimsTransformation(incoming => Container.Resolve<IClaimsTransformer>().TransformWithUserData(incoming));

Note that I am resolving the claimsTransformer from our IOC container, you can do it whichever way you choose.

And that’s it! Your authenticated controllers will now have the UserData contained in the ClaimsPrincipal and can easily be accessed or tested for.

I hope that helps.

• Part 1: Setting up Gulp with Visual Studio
• Part 2: Getting SASSy
• Part 3: Deployment: minification and concatenation
• Part 4: Code quality: tests and linting
• Part 5: Thoughts on Gulp

In this post I want to talk about JavaScript code quality.

Code quality: tests and linting

JavaScript is a notoriously quirky language –  the tricky “this” operator, lack of block level scoping, the ability to use variables before declaring them, “==” vs “===”, etc etc.

We can do a number of things to tighten up our code quality. We can run tests to ensure our code works as expected, and we can run a linting program to catch any syntatic mistakes before running our code. Combining this with Gulp gives a nice workflow for development.

Linting with JSHint

Linting is a process of running a program to analyse code for potential problems, catching bugs before the program gets executed. Douglas Crockford created JSLint in 2011, and of course there is a gulp plugin for it. However, I prefer to use JSHint – a community offshoot of JSLint that allows for more customisation and is slightly more relaxed in its defaults. You can find a good comparison of the two here.

First, I would seperate out any third-party libraries or frameworks you have into their own folder(s). In my opinion this is good practice anyway, but especially if we are running linting – we don’t want failures on code we haven’t written! Next, we want to add the plugin to our package.json:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-jshint": "~1.11.2",
    // other plugins here
  } 
}

Then, add the Gulp task to our gulpfile:

var sources = ['./Scripts/*.js', '!./Scripts/libs/*.js'];

gulp.task('watch-lint', function () {
    gulp.watch(sources, ['lint']);
});

gulp.task('lint', function () {
    return gulp.src(sources)
        .pipe(plugins.jshint({
            strict: true,
            predef: [""]
        }))
        .pipe(plugins.jshint.reporter('default'))
        .pipe(plugins.jshint.reporter('fail'))
    ;
});

Here’s a line by line breakdown of the important stuff:

• Line 1: Define the JavaScript files we want linting, excluding any third party libraries and frameworks
• Line 3-5: Set up a watch to run the linting task whenever we make a change to our JavaScript
• Line 7: Set up our linting task
• Line 8-12: Send our files to JSHint. JSHint has a number of options you can set to define how lenient or strict you want it to be. You can look at the docs to get an idea of what you can set. Here I’m only using the strict option (here is a good blog explaining strict mode). There is also a section to define any global variables you’re using that should be ignored – for example if you’re using Angular you’ll want to add “angular” here
• Line 13-14: Send the output of the linting to some inbuilt reporters – one to output the results to console and one to raise an error for any failures

Now we can add the watch task to our default task, and we’ll have linting every time we change our code.

Tests with Karma

One of the great things about Gulp is that you can get a Gulp task to run other Node based packages directly. For example, you can run Karma within a task, without having to use a plugin. Combined with PhantomJS, a headless browser, you get a great unit testing framework for your JavaScript. First, add the packages:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "karma-jasmine": "~0.3.6",
    "karma": "~0.13.9",
    "karma-phantomjs-launcher": "~0.2.1"
    // other plugins here
  } 
}

Then, we’ll need to add our karma config file (karma.conf.js) at the root of the project :

module.exports = function (config) {
    config.set({
        files: ['./Scripts/*.js',
                './Scripts/Specs/**/*.js',
        ],

        exclude: ['./Scripts/libs/*.js'],

        frameworks: ['jasmine'],

        autoWatchBatchDelay: 1000,

        reporters: ['progress'],

        browsers: ['PhantomJS']
    });
};

One thing I’ve found using Karma in Visual Studio: often Karma can crash in Visual Studio with an “EBUSY” error. I haven’t found an adequate explanation for this yet – I think it’s because when you modify and save a file Visual Studio writes a temporary file to replace the file you are saving, which doesn’t sit well with Node. This can be mitigated by setting the “autoWatchBatchDelay” to 1000, which waits 1 second after any change before running tests.

Finally, our gulp task:

var Server = require('karma').Server;

gulp.task('runTests', function () {
    new Server({
        configFile: __dirname + '/karma.conf.js',
        singleRun: false
    }).start();
});

Here, we are simply firing up Karma from within a Gulp task and passing in our karma config file. We are also telling Karma to watch our files for any changes and re-run our tests if they change. If you are going to run this task on a build server (and you should), you’ll need to configure the singleRun variable here. This can be done by using yargs to pass in a variable to switch this behaviour.

There we are! There is a commit of all the changes here, which includes a sample test. I highly recommend using both linting and unit tests when writing JavaScript code, and Gulp is great at running these tasks for us.

Next, in the last past I want to share my thoughts on Gulp, specifically within the .NET ecosystem.

• Part 1: Setting up Gulp with Visual Studio
• Part 2: Getting SASSy
• Part 3: Deployment: minification and concatenation
• Part 4: Code quality: tests and linting
• Part 5: Thoughts on Gulp

Small is beautiful

To increase the speed of a site, and to save bandwidth, we want to:

• Have as few requests for the smallest files possible
• Cache these files for as long as possible

We can achieve this by concatenating and minifying the files, then adding a hash of the files contents to the filename so they can be cached indefinitely.

There are a number of different ways of approaching minification and concatenation with Gulp, and a lot of different plugins and plugin combinations. The following is a style that has worked for me, you may need to adjust it to suit your own workflow.

Plugins

There are a number of plugins we will need:

• gulp-useref for concatenation
• gulp-uglify and gulp-minify-css for minifcation of JavaScript and CSS respectively
• gulp-rev and gulp-rev-replace for revisioning
• gulp-filter for filtering files to process them independently

Lets add these to the package.json:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-useref": "~1.3.0",
    "gulp-uglify": "~1.2.0",
    "gulp-minify-css": "~1.2.1",
    "gulp-rev": "~5.1.0",
    "gulp-rev-replace": "~0.4.2",
    "gulp-filter": "~3.0.0"
  } 
}

Referencing scripts and CSS

Now we’ll want to add our CSS and JavaScript files to our pages. These are put in commented build blocks that let gulp-useref know where a section it should concatenate is. So, at the top of the page (here it’s the _Layout.cshtml page) we reference the CSS:

<!-- build:css /dist/site.css -->
    <link href="/Content/bootstrap.css" rel="stylesheet" />
    <link href="/Content/Site.css" rel="stylesheet" />
<!-- endbuild -->    

In the first comment, we define two things:

• The first part, “build:css” tells useref that this be a CSS block
• The second is the location of the concatenated file that will eventually be referenced (more on this later)

We do the same at the bottom with the JavaScript:

<!-- build:js /dist/libs.js -->
    <script src="/Scripts/Libraries/jquery.js"></script>
    <script src="/Scripts/Libraries/bootstrap.js"></script>
    <script src="/Scripts/Libraries/respond.js"></script>
<!-- endbuild -->

We can have multiple build blocks of the same type (e.g. multiple JavaScript blocks) so we can have one for libraries (Angular, jQuery etc) which will change less frequently than the code we write.

The Gulp task

Now for the biggie – the actual task itself. I’ll put it here and go through it in more detail below:

gulp.task('minifyFilesForRelease', function () {

    var cssFilter = plugins.filter('**/*.css', { restore: true });
    var jsFilter = plugins.filter('**/*.js', { restore: true });

    var assets = plugins.useref.assets();
    gulp.src('./**/*.cshtml')
        .pipe(assets)

        //Process JavaScript
        .pipe(jsFilter)
        .pipe(plugins.uglify())
        .pipe(plugins.rev())
        .pipe(assets.restore())
        .pipe(jsFilter.restore)
        
        //Process CSS
        .pipe(cssFilter)
        .pipe(plugins.minifyCss({
            keepSpecialComments: 0
        }))
        .pipe(plugins.rev())
        .pipe(assets.restore())
        .pipe(cssFilter.restore)

        .pipe(plugins.useref())
        .pipe(plugins.revReplace({
            replaceInExtensions: ['.js', '.css', '.html', '.cshtml']
        }))
        .pipe(gulp.dest(function (data) {
            return data.base;
        }
        ));
});

Here’s a line by line breakdown of what’s happening:

• Line 1: Define the task
• Line 3-4: Setup some filters we will use later to separate out the CSS and JavaScript for individual processing
• Line 6: Reference the assets object of useref which will return the build blocks we setup earlier
• Line 7: Get all our cshtml files
• Line 8: Get all our build blocks in these files
• Line 11: Filter our build blocks to just the JavaScript
• Line 12: Minify the JavaScript
• Line 13: Revision the JavaScript
• Line 14-15: Remove the filters, so we have access to all our build blocks again
• Line 18: Filter our build blocks to just the CSS
• Line 19-21: Minify the CSS removing any and all comments
• Line 22: Revision CSS
• Line 23-24: Remove the filters again
• Line 26: Replace the commented, unprocessed build block in our cshtml files with the processed ones
• Line 27-29: Add the revision number to the processed files (note we need to include cshtml files here)
• Line 30-32: Replace the original cshtml files. We need to specify a destination here, and as we have views in multiple  folders we can use this technique by Yiling to overwrite a file regardless of its location

Phew, there’s a lot there! If you run this task each build block in our cshtml files will be replaced with a single, minified, revisioned file.

Before:

After:

Running in Release mode

We only want to run this task when we’re putting together a release, whether that’s manual or (much better!) automated. To do this, add a PreBuildEvent that runs the Gulp task before a release build. Go to the project properties, then Build Events and add the following to “Pre-build event command line”:

if $(ConfigurationName) == Release (
   cd $(ProjectDir)
   npm install
   gulp minifyFilesForRelease
)

This will then run our minifying task only on a Release build.

Lastly, as we are adding the minified files to a “dist” folder that doesn’t exist during development, we’ll need to add this folder after each build. Open up the csproj for the web project, and at the very end add the following:

<Target Name="AfterBuild">
    <ItemGroup>
      <Content Include="dist\*.*" />
    </ItemGroup>
 </Target>

This will include the contents of our dist folder after each build.

Wrapping up

We now have an automated process for minifying and concenating files that doesn’t intefere with the development process. The commit for everything is here.

Happy Gulping!

In my previous blog I showed you how to set up Visual Studio for use with Gulp. Now I want to show you how you can use it in everyday development, including processing SASS, running tests, linting your code and getting ready for deployment.

In this series of posts I’ll talk about how you can use Gulp with Visual Studio to run your common front end tasks:

• Part 1: Setting up Gulp with Visual Studio
• Part 2: Getting SASSy
• Part 3: Deployment: minification and concatenation
• Part 4: Code quality: tests and linting
• Part 5: Thoughts on Gulp

The code I’ll be showing is based off a standard MVC 5 project. If you want the code I’ve created a GitHub project for it here.

Basic Gulpfile + package.json

Throughout these examples I’ll be using gulp-load-plugins – an awesome meta-plugin that takes all your Gulp plugins from your package.json and attaches them to an object. At the start, the package.json file looks like this:

{
  "scripts": {"install": "echo Done"},
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0"
  } 
}

N.B. I like outputting something to the console after installing packages as the default is to not output anything, which can be a little confusing!

The basic gulpfile looks like this:

var gulp = require('gulp');
var plugins = require('gulp-load-plugins')();

gulp.task('default', ['']);

Now, any plugins we add to our package.json will immediately be available on the plugins object.

Processing SASS

I’m a big fan of SASS for developing CSS as it helps keep my CSS clean and DRY. SASS files need processing, and Gulp can do this for you. As a bonus, we can add sourcemaps so that when we inspect the page in Firefox, Chrome or IE 11 we refer back to the original SASS rather than the processed CSS. As a double bonus, we can use LiveReload to reload the page and automatically see our changes whenever we make a change to the SASS (note with LiveReload on Windows you’ll need the Chrome extension for the best experience).

First off, we add the SASS file. For this demo project, based on MVC 5, this is simple – rename the Site.css file to Site.scss. As SASS is a superset of CSS, all valid CSS is valid SASS.

Next, we add the plugins we’ll need to the package.json:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-sass": "~2.0.4",
    "gulp-sourcemaps": "~1.5.2",
    "gulp-autoprefixer": "~2.3.1",
    "gulp-livereload": "~3.8.0"
  } 
}

Next, add the gulp tasks – one to process the SASS into CSS, and another to watch the SASS files and recompile whenever we make a change:

var gulp = require('gulp');
var plugins = require('gulp-load-plugins')();

gulp.task('sass', function () {
    gulp.src('./Content/*.scss')
        .pipe(plugins.sourcemaps.init())
        .pipe(plugins.sass())
        .pipe(plugins.autoprefixer())
        .pipe(plugins.sourcemaps.write())
        .pipe(gulp.dest('./Content'))
        .pipe(plugins.livereload())
    ;
});

gulp.task('watch-sass', function () {
    plugins.livereload.listen();
    gulp.watch('./Content/*.scss', ['sass']);
});

gulp.task('default', ['watch-sass']);

Here’s a line by line breakdown of what is happening within our ‘sass’ task:

• Line 5: Grab all our SASS files from the Content folder. At the moment this is just the Site.scss file, but as we build this up we’ll want to add different SASS files for different parts of the project
• Line 6: Initialise the sourcemaps. Note that this must be done before they get processed
• Line 7: Process the SASS to generate the CSS
• Line 8: Use autoprefixer to add any vendor-specific prefixes to the CSS. The defaults for the plugin target any browser with >1% share, which is eminently sensible
• Line 9: Insert the sourcemaps into the generated CSS
• Line 10: Write the generated CSS back to the Content directory
• Line 11: Notify LiveReload that we’ve made changes

The ‘watch-sass’ task simply tells LiveReload to start listening, watches our SASS files for any changes, and when they are changed runs our ‘sass’ task. This watch has also been added to the default task, so will run automatically. Now, when we make a change to the Site.scss file the css will automatically get generated, and we get all the other goodness from our other plugins too.

Sourcemaps

Below I’ve changed the Site.scss file so that the text in the Jumbtron is blue, and have inspected it in Chrome. You’ll notice the red box in the inspector refers to the Site.scss file (not the generated Site.css file) , thanks to sourcemaps.

LiveReload

If you open the site in Chrome and start the LiveReload extension, any time a change is made to the SASS the web page will automatically reload with the changes. Here the color of the jumbotron text is switching between red and blue, and automatically updating:

Wrapping up

So there we are! Automatic processing of SASS with Gulp, along with a few other goodies. The commit for eveything I’ve shown is here.

One thing to note is that in this task we didn’t do anything minification to the CSS – I’ll come on to that in my next post.

In my first article I outlined fluent builders – what they look like and how to put them together. This time I will go a little deeper to show you just how versatile they can be.

Method Naming

You can be as flexible as you like in naming the methods in your builders. A standard starting place is to use ‘With…’ for example WithFirstName(..), WithDateOfBirth(..) etc. I find the best naming convention is one that makes your tests as easy to read as possible.  For instance

Customer testCustomer = new CustomerBuilder().
    .WithName("Peter Perfect")
    .Aged(32)
    .ThatIsAFirstTimeBuyer();

Remember that it is fine to add specific methods such as ‘ThatIsAFirstTimeBuyer()’ above where it could be setting an individual property, a series of properties or calling further methods within the customer class to create that state. You can also add specific tests for your builders to ensure those methods are working as expected.

Adding Lists

Your builder methods can also take in lists or add to lists. Let’s say that our CustomerBuilder allows us to add products they have bought.  It could be created in two ways;

public CustomerBuilder WithProduct(Product product)
{
    this.customer.Products.Add(product);
    return this;
}

or

public CustomerBuilder WithProducts(List products)
{
    this.customer.Products = products;
    return this;
}

In the first case we could use a product builder to add each product to the customer

Customer customer = new CustomerBuilder()
    .WithProduct(new ProductBuilder().WithName("Apple")))
    .WithProduct(new ProductBuilder().WithName("Orange"))

In the second we could create a specific ProductListBuilder()

public class ProductListBuilder
{
    private readonly List products;

    public ProductListBuilder()
    {
        this.products = new List();
    }

    public static implicit operator List(ProductListBuilder builder)
    {
        return builder.products;
    }

    public ProductListBuilder WithProduct(string productName)
    {
        this.products.Add(new Product { Name = productName });
        return this;
    }
}

Which we would use this way

    List<Product> products = new ProductListBuilder().WithProduct("Apple").WithProduct("Orange");
    Customer customer = new CustomerBuilder().WithProducts(products);

Builders for Mocked Objects

Do you often have to mock repositories?  How about using a builder to do it?

public class MockCustomerRepositoryBuilder
{
    private readonly Mock<irepository> mockCustomerRepository;

    public MockCustomerRepositoryBuilder()
    {
        this.mockCustomerRepository= new Mock<irepository>();
    }

    public Customer SavedCustomer { get; private set; }

    public static implicit operator Mock<irepository>(MockCustomerRepositoryBuilder builder)
    {
        return builder.mockCustomerRepository;
    }

    public MockCustomerRepositoryBuilder WhereGetByIdReturns(Customer customer)
    {
        this.mockCustomerRepository.Setup(cfr => cfr.GetById(It.IsAny())).Returns(customer);
        return this;
    }

    public MockCustomerRepositoryBuilder WithSave()
    {
        this.mockCustomerRepository.Setup(cfr => cfr.Save(It.IsAny())).Callback(cfr => { SavedCustomer = cfr; });
        return this;
    }
}

The usage is as follows

    var mockCustomerRepository = new MockCustomerRepositoryBuilder()
        .WithSave()
        .WhereGetByIdReturns(new Customer().WithName("Tom Test"));

What is really cool about this is that you can set up the save method in your mocked repository to save whatever you pass in to a public property. This allows you to assert that your code under test has actually passed the right object through.  You can also direct it to return whatever you like from the GetById() method of this repository so that you don’t have to set it up in each test.

I hope this has helped you to further appreciate the power of these builders.  Please feel free to ask any questions you may have below.  Next time I will show how to test MVC controllers with builders in a fluent way.

Thanks for listening.

Microsoft’s Power BI is excellent.  It is so easy to get up and running with data coming from a multitude of places. Out of the box you can easily show facts and figures related to your data.  One thing it currently misses though is a ‘percent of total’ aggregation for measures.

Recently I wanted to show browser usage for visitors to a web based questionnaire. The client had asked to see the browsers ordered by popularity with their respective percentage share displayed.

The SQL Server table ‘Dim_Respondents’ had the following columns available: LocalRespondentId (unique integer), Browser (text), DateStarted (Date) plus some foreign key references to other tables I could filter by such as Questionnaire Name, Questionnaire Status etc.  It was easy to get the browser count with a horizontal bar chart as follows

Notice how I have just added Browser to the Axis and ‘Count of LocalRespondentId’ to the Value.  The difficulty comes in trying to get percentages because you can only aggregate your data by sum, average, minimum, maximum and count.  The closest you can come is a 100% stacked bar chart;

but with a lot of browsers that is not a workable option.  Luckily Power BI lets us create our own measures using DAX. To do so , you need to right-click on your data table and select ‘New Measure’.

You can then enter your measure name and a DAX expression. (See ‘DAX basics in Power BI Desktop’ for basic DAX information). For our purposes we would like to know the number of respondents with a particular browser divided by the total number of respondents.  Before I create the measure though we need to consider that it is highly likely that this chart will have been further filtered – maybe by date or something else. Our calculation must take that into account.  The easiest way to express it is to say that we need the number of respondents for a particular browser taking into account all applied filters divided by the number of respondents for all browsers taking into account all applied filters (sorry, I know that is a long-winded statement, but bear with me).

In DAX we can write ‘COUNTA(Dim_Respondents[Browser]) ‘ for the first part which is simple enough (basically counts the number of values in a column after all filters have been applied – and remember that we have Browser on an axis so we are effectively filtering by Browser for each chart position), but the second part is a little more tricky. We can solve it by using ‘CALCULATE(COUNTROWS(Dim_Respondents),ALL(Dim_Respondents[Browser])) ‘ which says calculate the first expression ‘COUNTROWS(Dim_Respondents)’ for the second filtered data set ‘ALL(Dim_Respondents[Browser])’. The ‘ALL(..)’ statement clears the filter for the mentioned column so in our case ignores the axis filter on Browser and gives us a total count. The whole expression is

Browser % = COUNTA(Dim_Respondents[Browser]) / CALCULATE(COUNTROWS(Dim_Respondents),ALL(Dim_Respondents[Browser]))

Note that I haven’t multiplied the answer by 100 because it is better to click your new measure and set its format type to ‘Percentage’ in the Modeling tab.

You can now replace the count of LocalrespondentId directly with your new measure and set your chart to sort by the percentage;

One last nice touch is to put the percentages on the graph by setting Data Labels to ‘on’.

Hope this helps!

What is Gulp?

Gulp is a build process for front end development. Gulp automates all of the repetitive tasks you have to do to prepare your front end files (such as JavaScript and CSS) for development and deployment. Gulp uses small, simple, composable plugins to that let you define exactly what happens to the files in your build.

Common tasks you can get Gulp to automate

• Minifying JavaScript/CSS/images
• Concatenating JavaScipt/CSS files
• Running JavaScript lint tooling
• Compiling your SASS/LESS
• Compiling TypeScript/CoffeeScript
• Running JavaScript tests (eg Karma, Jasmine, Mocha)
• Transpiling ES6 into ES5
• Minifying Angular without breaking Dependency Injection
• …and many more

To install and use Gulp, you’ll need npm. Which begs the question…

What is npm?

npm is the package manager for Node. It’s a front-end equivalent to NuGet – you can use it to install and update packages and their dependencies. We’ll be using it to install Gulp and all its plugins.

How to set up Gulp in Visual Studio

Currently you can set up Gulp in Visual Studio 2013 and 2015. If you have 2015, skip the first 3 points as they come with Visual Studio 2015. Note that the screenshots are taken using Visual Studio 2013 with a standard MVC 5 project template.

1. Install Node/npm.

   You can install Node from here. What we’re really after here isn’t Node, it’s npm (which comes installed with Node).

2. Install Gulp.

   After installing npm, open up a Console window, and enter:

   npm install gulp -g

   This will install Gulp. The “-g” option installs it globally, so you can run Gulp commands anywhere.

3. Install the Grunt Launcher Visual Studio Extension.

   Grunt Launcher provides menu options to install npm packages and run Gulp tasks from within Visual Studio.

4. Add Gulp to your project

   At the root of your web project, add a new JSON file called package.json with the following contents:

   {
       "devDependencies": {
           "gulp":  "3.9.0"
       }   
   }
   

   The package.json is a npm file we use to define metadata about our project. Here we are listing the (front end) dependencies for developing our project – this will include plugins for running Gulp tasks. For now, we just want to add Gulp to our project.

   Now, right-click on the package.json file and click “NPM install packages” (if you’ve installed Grunt Launcher) to add it to our project.

   

   You can ignore those warnings for now – you can add the metadata it is complaining about later.

   This will add a node_modules folder in your project folder, where all our npm packages will be kept. Within that will be Gulp.

   

5. Add a Gulpfile

   Next we want to add a Gulpfile, where we will define the steps for our build process. Add a JavaScript file called gulpfile.js to the root of the project, and add the following code:

   var gulp = require('gulp');
   
   gulp.task('default', function () {
       console.log('Hello world!');
   });
   

    

   
   In line 1 we load Gulp. On line 3 we define a new Gulp task. A task is a step, or a set of steps, that Gulp will execute. A task takes a function that it will run – in this case we simply log to the console. A task also has a name – here it’s called “default” and as you might guess it runs automatically when we run Gulp.

   We can now run this task by right-clicking on the gulpfile.js and clicking the “Gulp” menu item added by the Grunt Launcher extension.

   

    

   Now Visual Studio is running our task, and outputting “Hello world!” to the output window. We’re done! Well, not quite…

Doing something actually useful

Hello World is all well and good, but what about automating something useful? One common task is minifying CSS files, and that’s what we’ll do here.

1. Add the plugins

   Change your package.json file to the following:

   {
       "devDependencies": {
           "gulp": "3.9.0",
           "gulp-uglifycss": "1.0.4",
           "gulp-rename": "1.2.2"
       }
   }
   

   Here we’ve added two new packages, gulp-uglifycss (which minifies css) and gulp-rename (which renames files passed into it). We can now use these in our task.

2. Modify the task

   Change the gulpfile.js to the following:

   var gulp = require('gulp'),
       uglifycss = require('gulp-uglifycss'),
       rename = require('gulp-rename');
   
   gulp.task('default', function () {
       return 
            gulp.src(['./Content/*.css', '!./Content/*.min.css'])
           .pipe(uglifycss())
           .pipe(rename({
               suffix: ".min",
               extname: ".css"
           }))
           .pipe(gulp.dest('./Content'));
   });
   

   Here’s a line by line breakdown of what the task is doing:
   Line 1-3: Add references to the plugins we just added in our package.json
   Line 7: Get the CSS files from our Content directory (excluding any .min files)
   Line 8: Pipe (i.e. pass) the files to gulp-uglifycss, which minifies them
   Line 9-12:  Rename the minified files to include a .min suffix
   Line 13: Save the minified files back to our Content folder

   Now, we can run this task to automatically minify all our CSS files.

   And there we are! We now have a process that will save us from a repetitive boring task, and we can build on this to automate lots of other tasks.

Next steps

Note that for brevity I have excluded everything that isn’t necessary for getting Gulp set up in Visual Studio. There are a few things you will want to do next to get fully set up:

1. Install the Task Runner Explorer extension

   You can install the Task Runner Explorer Visual Studio extension to run tasks automatically on each build or every time the project opens.

2. Install NPM Package Intellisense

   NPM Package Intellisense gives you package intellisense when installing packages and plugins.

3. Add more plugins

   Nearly any task you want to automate will have a plugin here – there are over 1500 at time of writing.

Happy Gulping!

No, I don’t mean the kind that come around and drink all your coffee / tea. When you are testing your code you often have to initialise various classes beforehand. There are several ways to do this. Consider this simple api that takes in a ‘Booking’ object and returns the cost of the booking.

public int GetCostOfBooking(Booking booking)
{
    // ... implementation not important here
}

Where the Booking object is

public class Booking
{
    public DateTime FirstDay { get; set; }

    public DateTime LastDay { get; set; }

    public decimal LocationCostPerNight { get; set; }

    public Customer Customer { get; set; }
}

and the Customer object is

public class Customer
{
    public decimal PercentDiscount { get; set; }
}

Your test could be set up as follows

[TestMethod]
public void BookingCostIsCorrect()
{
    var customer = new Customer { PercentDiscount = 10 };
    var booking = new Booking
                      {
                          FirstDay = DateTime.Parse("2015/08/15"), 
                          LastDay = DateTime.Parse("2015/08/20"), 
                          LocationCostPerNight = 10, 
                          Customer = customer
                      };
    var actual = GetCostOfBooking(booking);
    var expected = 45;
    Assert.AreEqual(expected, actual);
}

(please bear with me on these primitive samples and tests!)

Now this doesn’t look too bad, but what if I add more properties to Customer and Booking or force the use of constructors and specific set methods?

public class Booking
{
    public DateTime FirstDay { get; private set; }
    public DateTime LastDay { get; private set; }
    public decimal LocationCostPerDay { get; private set; }
    public Customer Customer { get; private set; }

    public void SetCostPerDay(decimal costPerDay)
    {
        this.LocationCostPerDay = costPerDay;
    }

    public void SetFirstDay(DateTime firstDay)
    {
        this.FirstDay = firstDay;
    }

    public void SetLastDay(DateTime lastDay)
    {
        this.LastDay = lastDay;
    }

    public void SetCustomer(Customer customer)
    {
        this.Customer = customer;
    }
}

public class Customer
{
    public Customer(string firstName, string lastName, DiscountScheme discountScheme)
    {
        this.FirstName = firstName;
        this.LastName = LastName;
        this.DiscountScheme = discountScheme;
    }

    public string FirstName { get; private set; }

    public string LastName { get; private set; }

    public DiscountScheme DiscountScheme { get; private set; }
}

public class DiscountScheme
{
    public decimal PercentDiscount { get; private set; }

    public void SetDiscount(decimal discount)
    {
        this.PercentDiscount = discount;
    }
}

Now my test setup becomes

[TestMethod]
public void BookingCostIsCorrect()
{
     var discountScheme = new DiscountScheme();
     discountScheme.SetDiscount(10);
     var customer = new Customer("Johnny", "BigCheese", discountScheme);
     var booking = new Booking();
     booking.SetCustomer(customer);
     booking.SetFirstDay(DateTime.Parse("2015/08/15"));
     booking.SetLastDay(DateTime.Parse("2015/08/15"));
     booking.SetCostPerDay(10);
     var actual = GetCostOfBooking(booking);
     var expected = 45;
     Assert.AreEqual(expected, actual);
}

Oh, not so nice any more. I have polluted my test with superfluous information, I am relying on calling the right methods to initialise properties and I have made the test that bit more obfuscated.

Instead, how about this

[TestMethod]
public void BookingCostIsCorrect()
{
    var customer = new CustomerBuilder()
                        .WithDiscount(10);

    var booking = new BookingBuilder()
                        .FromFirstDay("2015/08/15")
                        .ToLastDay("2015/08/19")
                        .ForCustomer(customer)
                        .AtCostPerDay(10);

    var actual = GetCostOfBooking(booking);
    var expected = 45;
    Assert.AreEqual(expected, actual);
}

Much nicer! It is easy to see what we are testing and how it is set up. There is no unnecessary information here. Also, thanks to the use of a builder, all the code for creating those objects is in a central place so you don’t have to edit all your tests when there is a change in how they should be initialised.

Great!  How do I do it?

It’s really not that difficult and once you have written a few you start to get the hang of it.  For instance here are the builder classes we have used above

public class BookingBuilder
{
    private Booking booking;

    public BookingBuilder()
    {
        this.booking = new Booking();
    }

    public static implicit operator Booking(BookingBuilder builder)
    {
        return builder.booking;
    }

    public BookingBuilder ForCustomer(Customer customer)
    {
        this.booking.SetCustomer(customer);
        return this;
    }

    public BookingBuilder FromFirstDay(string firstDay)
    {
        this.booking.SetFirstDay(DateTime.Parse(firstDay));
        return this;
    }

    public BookingBuilder ToLastDay(string lastDay)
    {
        this.booking.SetLastDay(DateTime.Parse(lastDay));
        return this;
    }

    public BookingBuilder AtCostPerDay(decimal costPerDay)
    {
        this.booking.SetCostPerDay(costPerDay);
        return this;
    }
}

public class CustomerBuilder
{
    private Customer customer;

    public CustomerBuilder()
    {
        this.customer = new Customer("Test", "Test", new DiscountScheme());
    }

    public static implicit operator Customer(CustomerBuilder builder)
    {
        return builder.customer;
    }

    public CustomerBuilder WithDiscount(decimal discount)
    {
        this.customer.DiscountScheme.SetDiscount(discount);
        return this;
    }
}

The main thing to remember when creating a fluent builder is that your methods return the class itself, that allows you to chain them. Note also the static implicit operator that converts between the builder and the object it is building. Adding that means we can directly use the builder in place of the object being built (ie passing the booking builder into the GetCostOfBooking method.

I mean this blog to be an introduction to fluent builders – you can get much more fancy with them than this and I have written a blog here that shows how to do that.

If you have any questions or want to add your own feedback, please comment below.  We would love to hear from you.

Debugging Lambda expressions

This has been a long time coming. Ever had to debug a long lambda expression only to find you can’t evaluate it in the Watch windows? Well now you can and more.  Dirk Strauss has written a good blog about it here.

Javascript and HTML Intellisense and Usage Improvements

Intellisense support has been added for Angular, BootStrap CSS and ARIA in the html editor as well as intellisense for ReactJS, AngularJS, RequireJS and ECMAScript 2015 (aka ECMAScript 6) in the javascript editor.  Oh, let’s throw in a Json editor too!

See VisualStudio.com’s article about the HTML editor intellisense here

The Visual Studio blog has more about the javascript intellisense here

Also, Bower and NPM intellisense, helped by the JSON editor. Jalpesh Vadgama writes it up in this blog article

GIT support improvements and additions

I tend to work primarily in Visual Studio now when it comes to source control, however I have to switch out occasionally when I need more info (we use Atlassian’s SourceTree). With the new improvements in Visual Studio 2015, I can stay in VS. VisualStudio.com mention the main features here

Enterprise Single Sign On with Azure Active Directory

If you are creating a website for users that are within your domain and you are using Azure Active Directory you can now add Enterprise Single Sign On at any point in your development by right-clicking on your application and selecting ‘Configure Azure AD Authentication’. This will bring up a wizard that will allow you to the specify the domain from a list of known domains.  The wizard will register your application with that Active Directory and configure your application to prompt for sign-in.

Intellitest – if you have to support or refactor legacy code with no tests, this will help.

I suspect this is going to be a bit of a love it or hate it addition. Personally I love it for the reasons below.
Intellitest grew out of Pex, a Microsoft Research Project. It basically boils down to a way of automatically creating Unit Tests for your code that exercise every line (if it is actually possible to exercise every line!). It does this by auto constructing objects that are required parameters for the method under test and then calling that method with many variations of the parameters.  There seems to be a bit of an initial backlash from some quarters as it is definitely NOT a good way to quickly construct Unit Tests as a box ticking exercise for your shiny new project.  However its great strength lies in legacy projects that need to be refactored / upgraded / fiddled with in some way. I have inherited many of these over the years and trying to work out the behaviour of a particularly messy method is never easy.  Now I can point Intellitest at it and make sure my replacement method mimics the results. Note that this is otherwise known as Characterisation Testing.

MSDN has an article about using Intellitest

Overall a great set of features that should really help save time. You can also see my blog about improvements that came with C# 6.0

Have fun!