• Part 1: Setting up Gulp with Visual Studio
• Part 2: Getting SASSy
• Part 3: Deployment: minification and concatenation
• Part 4: Code quality: tests and linting
• Part 5: Thoughts on Gulp

In this post I want to talk about JavaScript code quality.

Code quality: tests and linting

JavaScript is a notoriously quirky language –  the tricky “this” operator, lack of block level scoping, the ability to use variables before declaring them, “==” vs “===”, etc etc.

We can do a number of things to tighten up our code quality. We can run tests to ensure our code works as expected, and we can run a linting program to catch any syntatic mistakes before running our code. Combining this with Gulp gives a nice workflow for development.

Linting with JSHint

Linting is a process of running a program to analyse code for potential problems, catching bugs before the program gets executed. Douglas Crockford created JSLint in 2011, and of course there is a gulp plugin for it. However, I prefer to use JSHint – a community offshoot of JSLint that allows for more customisation and is slightly more relaxed in its defaults. You can find a good comparison of the two here.

First, I would seperate out any third-party libraries or frameworks you have into their own folder(s). In my opinion this is good practice anyway, but especially if we are running linting – we don’t want failures on code we haven’t written! Next, we want to add the plugin to our package.json:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-jshint": "~1.11.2",
    // other plugins here
  } 
}

Then, add the Gulp task to our gulpfile:

var sources = ['./Scripts/*.js', '!./Scripts/libs/*.js'];

gulp.task('watch-lint', function () {
    gulp.watch(sources, ['lint']);
});

gulp.task('lint', function () {
    return gulp.src(sources)
        .pipe(plugins.jshint({
            strict: true,
            predef: [""]
        }))
        .pipe(plugins.jshint.reporter('default'))
        .pipe(plugins.jshint.reporter('fail'))
    ;
});

Here’s a line by line breakdown of the important stuff:

• Line 1: Define the JavaScript files we want linting, excluding any third party libraries and frameworks
• Line 3-5: Set up a watch to run the linting task whenever we make a change to our JavaScript
• Line 7: Set up our linting task
• Line 8-12: Send our files to JSHint. JSHint has a number of options you can set to define how lenient or strict you want it to be. You can look at the docs to get an idea of what you can set. Here I’m only using the strict option (here is a good blog explaining strict mode). There is also a section to define any global variables you’re using that should be ignored – for example if you’re using Angular you’ll want to add “angular” here
• Line 13-14: Send the output of the linting to some inbuilt reporters – one to output the results to console and one to raise an error for any failures

Now we can add the watch task to our default task, and we’ll have linting every time we change our code.

Tests with Karma

One of the great things about Gulp is that you can get a Gulp task to run other Node based packages directly. For example, you can run Karma within a task, without having to use a plugin. Combined with PhantomJS, a headless browser, you get a great unit testing framework for your JavaScript. First, add the packages:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "karma-jasmine": "~0.3.6",
    "karma": "~0.13.9",
    "karma-phantomjs-launcher": "~0.2.1"
    // other plugins here
  } 
}

Then, we’ll need to add our karma config file (karma.conf.js) at the root of the project :

module.exports = function (config) {
    config.set({
        files: ['./Scripts/*.js',
                './Scripts/Specs/**/*.js',
        ],

        exclude: ['./Scripts/libs/*.js'],

        frameworks: ['jasmine'],

        autoWatchBatchDelay: 1000,

        reporters: ['progress'],

        browsers: ['PhantomJS']
    });
};

One thing I’ve found using Karma in Visual Studio: often Karma can crash in Visual Studio with an “EBUSY” error. I haven’t found an adequate explanation for this yet – I think it’s because when you modify and save a file Visual Studio writes a temporary file to replace the file you are saving, which doesn’t sit well with Node. This can be mitigated by setting the “autoWatchBatchDelay” to 1000, which waits 1 second after any change before running tests.

Finally, our gulp task:

var Server = require('karma').Server;

gulp.task('runTests', function () {
    new Server({
        configFile: __dirname + '/karma.conf.js',
        singleRun: false
    }).start();
});

Here, we are simply firing up Karma from within a Gulp task and passing in our karma config file. We are also telling Karma to watch our files for any changes and re-run our tests if they change. If you are going to run this task on a build server (and you should), you’ll need to configure the singleRun variable here. This can be done by using yargs to pass in a variable to switch this behaviour.

There we are! There is a commit of all the changes here, which includes a sample test. I highly recommend using both linting and unit tests when writing JavaScript code, and Gulp is great at running these tasks for us.

Next, in the last past I want to share my thoughts on Gulp, specifically within the .NET ecosystem.

• Part 1: Setting up Gulp with Visual Studio
• Part 2: Getting SASSy
• Part 3: Deployment: minification and concatenation
• Part 4: Code quality: tests and linting
• Part 5: Thoughts on Gulp

Small is beautiful

To increase the speed of a site, and to save bandwidth, we want to:

• Have as few requests for the smallest files possible
• Cache these files for as long as possible

We can achieve this by concatenating and minifying the files, then adding a hash of the files contents to the filename so they can be cached indefinitely.

There are a number of different ways of approaching minification and concatenation with Gulp, and a lot of different plugins and plugin combinations. The following is a style that has worked for me, you may need to adjust it to suit your own workflow.

Plugins

There are a number of plugins we will need:

• gulp-useref for concatenation
• gulp-uglify and gulp-minify-css for minifcation of JavaScript and CSS respectively
• gulp-rev and gulp-rev-replace for revisioning
• gulp-filter for filtering files to process them independently

Lets add these to the package.json:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-useref": "~1.3.0",
    "gulp-uglify": "~1.2.0",
    "gulp-minify-css": "~1.2.1",
    "gulp-rev": "~5.1.0",
    "gulp-rev-replace": "~0.4.2",
    "gulp-filter": "~3.0.0"
  } 
}

Referencing scripts and CSS

Now we’ll want to add our CSS and JavaScript files to our pages. These are put in commented build blocks that let gulp-useref know where a section it should concatenate is. So, at the top of the page (here it’s the _Layout.cshtml page) we reference the CSS:

<!-- build:css /dist/site.css -->
    <link href="/Content/bootstrap.css" rel="stylesheet" />
    <link href="/Content/Site.css" rel="stylesheet" />
<!-- endbuild -->    

In the first comment, we define two things:

• The first part, “build:css” tells useref that this be a CSS block
• The second is the location of the concatenated file that will eventually be referenced (more on this later)

We do the same at the bottom with the JavaScript:

<!-- build:js /dist/libs.js -->
    <script src="/Scripts/Libraries/jquery.js"></script>
    <script src="/Scripts/Libraries/bootstrap.js"></script>
    <script src="/Scripts/Libraries/respond.js"></script>
<!-- endbuild -->

We can have multiple build blocks of the same type (e.g. multiple JavaScript blocks) so we can have one for libraries (Angular, jQuery etc) which will change less frequently than the code we write.

The Gulp task

Now for the biggie – the actual task itself. I’ll put it here and go through it in more detail below:

gulp.task('minifyFilesForRelease', function () {

    var cssFilter = plugins.filter('**/*.css', { restore: true });
    var jsFilter = plugins.filter('**/*.js', { restore: true });

    var assets = plugins.useref.assets();
    gulp.src('./**/*.cshtml')
        .pipe(assets)

        //Process JavaScript
        .pipe(jsFilter)
        .pipe(plugins.uglify())
        .pipe(plugins.rev())
        .pipe(assets.restore())
        .pipe(jsFilter.restore)
        
        //Process CSS
        .pipe(cssFilter)
        .pipe(plugins.minifyCss({
            keepSpecialComments: 0
        }))
        .pipe(plugins.rev())
        .pipe(assets.restore())
        .pipe(cssFilter.restore)

        .pipe(plugins.useref())
        .pipe(plugins.revReplace({
            replaceInExtensions: ['.js', '.css', '.html', '.cshtml']
        }))
        .pipe(gulp.dest(function (data) {
            return data.base;
        }
        ));
});

Here’s a line by line breakdown of what’s happening:

• Line 1: Define the task
• Line 3-4: Setup some filters we will use later to separate out the CSS and JavaScript for individual processing
• Line 6: Reference the assets object of useref which will return the build blocks we setup earlier
• Line 7: Get all our cshtml files
• Line 8: Get all our build blocks in these files
• Line 11: Filter our build blocks to just the JavaScript
• Line 12: Minify the JavaScript
• Line 13: Revision the JavaScript
• Line 14-15: Remove the filters, so we have access to all our build blocks again
• Line 18: Filter our build blocks to just the CSS
• Line 19-21: Minify the CSS removing any and all comments
• Line 22: Revision CSS
• Line 23-24: Remove the filters again
• Line 26: Replace the commented, unprocessed build block in our cshtml files with the processed ones
• Line 27-29: Add the revision number to the processed files (note we need to include cshtml files here)
• Line 30-32: Replace the original cshtml files. We need to specify a destination here, and as we have views in multiple  folders we can use this technique by Yiling to overwrite a file regardless of its location

Phew, there’s a lot there! If you run this task each build block in our cshtml files will be replaced with a single, minified, revisioned file.

Before:

After:

Running in Release mode

We only want to run this task when we’re putting together a release, whether that’s manual or (much better!) automated. To do this, add a PreBuildEvent that runs the Gulp task before a release build. Go to the project properties, then Build Events and add the following to “Pre-build event command line”:

if $(ConfigurationName) == Release (
   cd $(ProjectDir)
   npm install
   gulp minifyFilesForRelease
)

This will then run our minifying task only on a Release build.

Lastly, as we are adding the minified files to a “dist” folder that doesn’t exist during development, we’ll need to add this folder after each build. Open up the csproj for the web project, and at the very end add the following:

<Target Name="AfterBuild">
    <ItemGroup>
      <Content Include="dist\*.*" />
    </ItemGroup>
 </Target>

This will include the contents of our dist folder after each build.

Wrapping up

We now have an automated process for minifying and concenating files that doesn’t intefere with the development process. The commit for everything is here.

Happy Gulping!

In my previous blog I showed you how to set up Visual Studio for use with Gulp. Now I want to show you how you can use it in everyday development, including processing SASS, running tests, linting your code and getting ready for deployment.

In this series of posts I’ll talk about how you can use Gulp with Visual Studio to run your common front end tasks:

• Part 1: Setting up Gulp with Visual Studio
• Part 2: Getting SASSy
• Part 3: Deployment: minification and concatenation
• Part 4: Code quality: tests and linting
• Part 5: Thoughts on Gulp

The code I’ll be showing is based off a standard MVC 5 project. If you want the code I’ve created a GitHub project for it here.

Basic Gulpfile + package.json

Throughout these examples I’ll be using gulp-load-plugins – an awesome meta-plugin that takes all your Gulp plugins from your package.json and attaches them to an object. At the start, the package.json file looks like this:

{
  "scripts": {"install": "echo Done"},
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0"
  } 
}

N.B. I like outputting something to the console after installing packages as the default is to not output anything, which can be a little confusing!

The basic gulpfile looks like this:

var gulp = require('gulp');
var plugins = require('gulp-load-plugins')();

gulp.task('default', ['']);

Now, any plugins we add to our package.json will immediately be available on the plugins object.

Processing SASS

I’m a big fan of SASS for developing CSS as it helps keep my CSS clean and DRY. SASS files need processing, and Gulp can do this for you. As a bonus, we can add sourcemaps so that when we inspect the page in Firefox, Chrome or IE 11 we refer back to the original SASS rather than the processed CSS. As a double bonus, we can use LiveReload to reload the page and automatically see our changes whenever we make a change to the SASS (note with LiveReload on Windows you’ll need the Chrome extension for the best experience).

First off, we add the SASS file. For this demo project, based on MVC 5, this is simple – rename the Site.css file to Site.scss. As SASS is a superset of CSS, all valid CSS is valid SASS.

Next, we add the plugins we’ll need to the package.json:

{
  "scripts": { "install": "echo Done" },
    
  "devDependencies": {
    "gulp": "~3.9.0",
    "gulp-load-plugins": "~0.10.0",
    "gulp-sass": "~2.0.4",
    "gulp-sourcemaps": "~1.5.2",
    "gulp-autoprefixer": "~2.3.1",
    "gulp-livereload": "~3.8.0"
  } 
}

Next, add the gulp tasks – one to process the SASS into CSS, and another to watch the SASS files and recompile whenever we make a change:

var gulp = require('gulp');
var plugins = require('gulp-load-plugins')();

gulp.task('sass', function () {
    gulp.src('./Content/*.scss')
        .pipe(plugins.sourcemaps.init())
        .pipe(plugins.sass())
        .pipe(plugins.autoprefixer())
        .pipe(plugins.sourcemaps.write())
        .pipe(gulp.dest('./Content'))
        .pipe(plugins.livereload())
    ;
});

gulp.task('watch-sass', function () {
    plugins.livereload.listen();
    gulp.watch('./Content/*.scss', ['sass']);
});

gulp.task('default', ['watch-sass']);

Here’s a line by line breakdown of what is happening within our ‘sass’ task:

• Line 5: Grab all our SASS files from the Content folder. At the moment this is just the Site.scss file, but as we build this up we’ll want to add different SASS files for different parts of the project
• Line 6: Initialise the sourcemaps. Note that this must be done before they get processed
• Line 7: Process the SASS to generate the CSS
• Line 8: Use autoprefixer to add any vendor-specific prefixes to the CSS. The defaults for the plugin target any browser with >1% share, which is eminently sensible
• Line 9: Insert the sourcemaps into the generated CSS
• Line 10: Write the generated CSS back to the Content directory
• Line 11: Notify LiveReload that we’ve made changes

The ‘watch-sass’ task simply tells LiveReload to start listening, watches our SASS files for any changes, and when they are changed runs our ‘sass’ task. This watch has also been added to the default task, so will run automatically. Now, when we make a change to the Site.scss file the css will automatically get generated, and we get all the other goodness from our other plugins too.

Sourcemaps

Below I’ve changed the Site.scss file so that the text in the Jumbtron is blue, and have inspected it in Chrome. You’ll notice the red box in the inspector refers to the Site.scss file (not the generated Site.css file) , thanks to sourcemaps.

LiveReload

If you open the site in Chrome and start the LiveReload extension, any time a change is made to the SASS the web page will automatically reload with the changes. Here the color of the jumbotron text is switching between red and blue, and automatically updating:

Wrapping up

So there we are! Automatic processing of SASS with Gulp, along with a few other goodies. The commit for eveything I’ve shown is here.

One thing to note is that in this task we didn’t do anything minification to the CSS – I’ll come on to that in my next post.

What is Gulp?

Gulp is a build process for front end development. Gulp automates all of the repetitive tasks you have to do to prepare your front end files (such as JavaScript and CSS) for development and deployment. Gulp uses small, simple, composable plugins to that let you define exactly what happens to the files in your build.

Common tasks you can get Gulp to automate

• Minifying JavaScript/CSS/images
• Concatenating JavaScipt/CSS files
• Running JavaScript lint tooling
• Compiling your SASS/LESS
• Compiling TypeScript/CoffeeScript
• Running JavaScript tests (eg Karma, Jasmine, Mocha)
• Transpiling ES6 into ES5
• Minifying Angular without breaking Dependency Injection
• …and many more

To install and use Gulp, you’ll need npm. Which begs the question…

What is npm?

npm is the package manager for Node. It’s a front-end equivalent to NuGet – you can use it to install and update packages and their dependencies. We’ll be using it to install Gulp and all its plugins.

How to set up Gulp in Visual Studio

Currently you can set up Gulp in Visual Studio 2013 and 2015. If you have 2015, skip the first 3 points as they come with Visual Studio 2015. Note that the screenshots are taken using Visual Studio 2013 with a standard MVC 5 project template.

1. Install Node/npm.

   You can install Node from here. What we’re really after here isn’t Node, it’s npm (which comes installed with Node).

2. Install Gulp.

   After installing npm, open up a Console window, and enter:

   npm install gulp -g

   This will install Gulp. The “-g” option installs it globally, so you can run Gulp commands anywhere.

3. Install the Grunt Launcher Visual Studio Extension.

   Grunt Launcher provides menu options to install npm packages and run Gulp tasks from within Visual Studio.

4. Add Gulp to your project

   At the root of your web project, add a new JSON file called package.json with the following contents:

   {
       "devDependencies": {
           "gulp":  "3.9.0"
       }   
   }
   

   The package.json is a npm file we use to define metadata about our project. Here we are listing the (front end) dependencies for developing our project – this will include plugins for running Gulp tasks. For now, we just want to add Gulp to our project.

   Now, right-click on the package.json file and click “NPM install packages” (if you’ve installed Grunt Launcher) to add it to our project.

   

   You can ignore those warnings for now – you can add the metadata it is complaining about later.

   This will add a node_modules folder in your project folder, where all our npm packages will be kept. Within that will be Gulp.

   

5. Add a Gulpfile

   Next we want to add a Gulpfile, where we will define the steps for our build process. Add a JavaScript file called gulpfile.js to the root of the project, and add the following code:

   var gulp = require('gulp');
   
   gulp.task('default', function () {
       console.log('Hello world!');
   });
   

    

   
   In line 1 we load Gulp. On line 3 we define a new Gulp task. A task is a step, or a set of steps, that Gulp will execute. A task takes a function that it will run – in this case we simply log to the console. A task also has a name – here it’s called “default” and as you might guess it runs automatically when we run Gulp.

   We can now run this task by right-clicking on the gulpfile.js and clicking the “Gulp” menu item added by the Grunt Launcher extension.

   

    

   Now Visual Studio is running our task, and outputting “Hello world!” to the output window. We’re done! Well, not quite…

Doing something actually useful

Hello World is all well and good, but what about automating something useful? One common task is minifying CSS files, and that’s what we’ll do here.

1. Add the plugins

   Change your package.json file to the following:

   {
       "devDependencies": {
           "gulp": "3.9.0",
           "gulp-uglifycss": "1.0.4",
           "gulp-rename": "1.2.2"
       }
   }
   

   Here we’ve added two new packages, gulp-uglifycss (which minifies css) and gulp-rename (which renames files passed into it). We can now use these in our task.

2. Modify the task

   Change the gulpfile.js to the following:

   var gulp = require('gulp'),
       uglifycss = require('gulp-uglifycss'),
       rename = require('gulp-rename');
   
   gulp.task('default', function () {
       return 
            gulp.src(['./Content/*.css', '!./Content/*.min.css'])
           .pipe(uglifycss())
           .pipe(rename({
               suffix: ".min",
               extname: ".css"
           }))
           .pipe(gulp.dest('./Content'));
   });
   

   Here’s a line by line breakdown of what the task is doing:
   Line 1-3: Add references to the plugins we just added in our package.json
   Line 7: Get the CSS files from our Content directory (excluding any .min files)
   Line 8: Pipe (i.e. pass) the files to gulp-uglifycss, which minifies them
   Line 9-12:  Rename the minified files to include a .min suffix
   Line 13: Save the minified files back to our Content folder

   Now, we can run this task to automatically minify all our CSS files.

   And there we are! We now have a process that will save us from a repetitive boring task, and we can build on this to automate lots of other tasks.

Next steps

Note that for brevity I have excluded everything that isn’t necessary for getting Gulp set up in Visual Studio. There are a few things you will want to do next to get fully set up:

1. Install the Task Runner Explorer extension

   You can install the Task Runner Explorer Visual Studio extension to run tasks automatically on each build or every time the project opens.

2. Install NPM Package Intellisense

   NPM Package Intellisense gives you package intellisense when installing packages and plugins.

3. Add more plugins

   Nearly any task you want to automate will have a plugin here – there are over 1500 at time of writing.

Happy Gulping!

Debugging Lambda expressions

This has been a long time coming. Ever had to debug a long lambda expression only to find you can’t evaluate it in the Watch windows? Well now you can and more.  Dirk Strauss has written a good blog about it here.

Javascript and HTML Intellisense and Usage Improvements

Intellisense support has been added for Angular, BootStrap CSS and ARIA in the html editor as well as intellisense for ReactJS, AngularJS, RequireJS and ECMAScript 2015 (aka ECMAScript 6) in the javascript editor.  Oh, let’s throw in a Json editor too!

See VisualStudio.com’s article about the HTML editor intellisense here

The Visual Studio blog has more about the javascript intellisense here

Also, Bower and NPM intellisense, helped by the JSON editor. Jalpesh Vadgama writes it up in this blog article

GIT support improvements and additions

I tend to work primarily in Visual Studio now when it comes to source control, however I have to switch out occasionally when I need more info (we use Atlassian’s SourceTree). With the new improvements in Visual Studio 2015, I can stay in VS. VisualStudio.com mention the main features here

Enterprise Single Sign On with Azure Active Directory

If you are creating a website for users that are within your domain and you are using Azure Active Directory you can now add Enterprise Single Sign On at any point in your development by right-clicking on your application and selecting ‘Configure Azure AD Authentication’. This will bring up a wizard that will allow you to the specify the domain from a list of known domains.  The wizard will register your application with that Active Directory and configure your application to prompt for sign-in.

Intellitest – if you have to support or refactor legacy code with no tests, this will help.

I suspect this is going to be a bit of a love it or hate it addition. Personally I love it for the reasons below.
Intellitest grew out of Pex, a Microsoft Research Project. It basically boils down to a way of automatically creating Unit Tests for your code that exercise every line (if it is actually possible to exercise every line!). It does this by auto constructing objects that are required parameters for the method under test and then calling that method with many variations of the parameters.  There seems to be a bit of an initial backlash from some quarters as it is definitely NOT a good way to quickly construct Unit Tests as a box ticking exercise for your shiny new project.  However its great strength lies in legacy projects that need to be refactored / upgraded / fiddled with in some way. I have inherited many of these over the years and trying to work out the behaviour of a particularly messy method is never easy.  Now I can point Intellitest at it and make sure my replacement method mimics the results. Note that this is otherwise known as Characterisation Testing.

MSDN has an article about using Intellitest

Overall a great set of features that should really help save time. You can also see my blog about improvements that came with C# 6.0

Have fun!

Presuming you have already designed your tables and they are present in your project;

• Create a folder in your project called StaticData then add a sql script to generate the data you require, one for each table.  For instance I have a date dimension table called Dim_Date and have created a script called Populate_Dim_Date.sql.
  
• Next create a script called ‘Script.PostDeployment.sql’ in the root of your project by right clicking your project and choosing ‘Add’ then ‘Script’
  
  and choosing ‘user scripts’ in the SQL Server section, followed by ‘Post-Deployment Script’, then clicking the Add button.
  

This will create a script that runs following the full deployment of your project and you can direct it to step through each of your sql creation scripts in the StaticData folder we created earlier. To do this you simply add a SQLCMD read command to the post deployment script for each of the sql files you want to execute. Here I have added a command to read (include) the script I created for the date dimension.

Note that to prevent an infestation of the red squigglies you will need to select the icon that represents placing the editor in SQLCMD Mode, in my image above it is the icon with a blue border.

Now when you publish your project your tables will be populated according to your scripts – easy peasy!

If the database you are referring to is also a part of the solution – for instance if you are creating a data warehouse and you have a supporting staging database, it is very similar to the process of adding an external project reference to a c# project. you just need to select ‘Database projects in the current solution’ and choose your database from the dropdown list.

You can also reference a system database here. Occasionally however you will need to reference a database that is neither of the above.  If the database you need to reference is already a data tier application you can browse for the dacpac but if it is not (for instance it may be a legacy database) you will need to create a data tier application from it first.

To extract a dacpac from an existing database, open the sql server object explorer in Visual Studio, connect to the server, right click the database and choose ‘Extract Data-tier application’.

Choose where to save the dacpac, then select ‘Extract Schema Only’ and decide whether you need the additional options such as user login mappings etc. Once you have created the dacpac you can go back to the add database reference dialog and select the dacpac you just created.

Database Location

In the Database Location section of the Add Database Reference dialog set the location of the referenced database e.g. ‘Different database, same server’, the name of the database (should be pre-filled) and whether you want to add a database variable to embed into your sql instead of a hardcoded path.  Finally (and often very importantly!) you may want to check the ‘Supress errors caused by unresolved references in the referenced project’ option at the bottom of the dialog.  This is especially useful in legacy databases that themselves reference other databases.

After clicking ok you should be good to go. This reference is there just to allow the application to compile, it does not get included in the deployment, but is an excellent way of making sure that your external references are correct.

As a final note, you need to ensure that your references are kept up to date if you are working on a project.  Although you can create a NuGet feed for your created dacpacs, as of Sept 2014 NuGet does not yet support importing to an SSDT project. For now you will have to keep a manual check.

Using Visual Studio and SSDT you can immediately publish a fresh database or update an existing one by simply right clicking the database project and choosing ‘Publish’. This will bring up a dialog that lets you edit the connection string to your database and set various options related to the publish.

At this point you have a choice – whether to register your published database as a Data Tier Application or not. Please do so.  There are two distinct advantages

• You can enable database drift detection.
• You can query the server to see what version database is published there, when and by whom.

Registering a database as a Data-Tier Application stores a copy of the current state of the database schema in system metadata tables as well as associated information relating to the publish itself.  To see quick info you can run

select * from sysdac_instances

against the msdb database on the server you published to.

Database drift is a very useful feature that allows you to check whether somebody has directly altered the database in some way since the last publish. I know this is a problem for many small teams – a query is a little slow so an index is added, a default was applied to a column because an app was throwing exceptions further down the pipeline and it had to be fixed quickly. This allows you to detect that, find the changes and if necessary bring them in to your project or else delete them. I always choose the option to block publish when database has drifted from registered version.

Now that you have selected your server, registered the database as a DTA and prevented drift, save the profile in the form .publish.xml. This isn’t just so that you can double click the profile to immediately publish (although you can of course) this is so that when TeamCity (you ARE using CI aren’t you?) builds your SSDT solution you can tell it which profile to use to automatically publish your database.  How good is that?

In the TeamCity build step that actually builds your solution you can select a target of ‘Build Publish’ and then in your command line parameters add the following

/p:SqlPublishProfilePath=<profilename> /p:DeployOnBuild=true

Where <profilename> is the full name of your profile e.g. myserver.publish.xml.

Done!  An easily updated, version controlled database that watches for people messing about with it on the server and lets you know.  What more could you ask for?

How many times have you found inconsistencies between a test and live database, not known which one is right and been unsure who to ask on your team?  In my experience, especially in smaller teams with no DBA, this happens a lot and people do bad things like edit the live database directly (“Hmmm, according to the query plan I need to add an index.  I’ll just do that now.”).

By creating a SQL Server project using SSDT

• You will know exactly who changed what and when.
• You can fail a publish if database drift is detected (ie the database you are publishing to is different from the last time you published because someone changed something directly).
• You can create unit tests with standardised data that run against a fresh database.
• You can track database refactoring (ie changing a column name) so that tables are not dropped and rebuilt when published.
• You can fail a publish if data loss will occur.
• You can fully automate deployment to multiple servers and bring them all to a standard level regardless of their current version.

So much good stuff!

Here’s how you do it.

First, make sure you have the latest version of SSDT.

Create the project

1. In Visual Studio, select File / New Project.
2. Under Templates. click ‘SQL Server’ and choose ‘SQL Server Database Project’
3. Name your project according to the database you are importing and also name your solution, then click OK.
   

Import your database

1. Right click the project you just created and select Import, then Database
2. Select your database from the Source Database Connection dropdown list, or use the New Connection button to add it.
3. In addition to the default Import Settings that are selected I usually also select ‘Import Database Settings’ so that the correct database version is chosen in the project properties. Otherwise you can usually leave everything else as is.

Check your folders and files

At this point you will have a folder structure based on the database schema – if you only use dbo then you will only see that, otherwise you will see a folder per schema and an additional security folder where sql for creating your schemas has been put. (This is also where you put sql for new schemas) Here you can see that my database has two schemas – dbo and IntegrationTest. SSDT also separates out your functions, stored procs and tables. In the Security folder you can see the sql for creating the IntegrationTest schema.

If there is anything missing at this point you may want to check the import settings. You can always delete all the added files and folders and do the import again.

Build and be happy!

Right click your project and select Build.

• If there are no external references the project will build and create a dacpac. This will be saved in your bin directory in the appropriate configuration folder.
• If there are external references required you will need to add them. I have blogged how to do this here.

You can also add static data such as lookup tables or date dimensions and again I have blogged about this here.

We are now at a stage where you can version control your database and to publish it to a particular server you just need to right click the project and select ‘Publish’. Note that there are several options when publishing and you can create profiles for different servers.  I have blogged about how to do that here.

Good luck!

Here is the outline of how it all hangs together, I will do a few more detailed posts to follow up.

The solution itself consists of three projects. Two database projects and a unit test project.  They are checked into GIT (using BitBucket) and TeamCity watches the repository to trigger a build.  The build creates a dacpac for each database, publishes the changes to the appropriate server, runs the unit tests and saves the dacpacs into a nuget package that is published by TeamCity itself.

Out of the box, TeamCity can handle the build but you need MSTest and SSDT installed on the build server.  That is where your troubles begin.  You can download an MSTest agent as I wrote about here.  You can even install the SSDT tools for VS2012 (though that was a bit hit and miss for me). The real problem is that with VS2013 the SSDT tools are integrated and no longer available as a separate download.  In the end I had to settle for installing VS2013 on the build server.  Microsoft allows your developer license to be used on the build server and at least I can easily update the SSDT version – I’m not particularly happy about having to do it though.

In order to publish your database to the correct server you just need to include appropriate publish profile files and Microsoft have provided the ability in database unit tests to switch config files according to the build server machine name. The publishing part worked fine for me but the config switching seems flaky and if you put a whitespace in the wrong place your config will be ignored.  My workaround here was to resort to Slow Cheetah – a plugin from NuGet that allows config switching for app.config in the same way that you can achieve it with web.config. This works well but remember to restore your nuget packages as the first build step in TeamCity. It’s not so restrictive as a config that switches based on machine name (we only have a single build server so would be unable to test on multiple platforms).

The result is that on every check in, the dacpacs are built, deployed to a test environment, tested and if they pass, made available via a nuget feed for other projects.

It feels good when it all just starts working!

However if you need to run MSTest for your unit tests TeamCity recommends you install the full Visual Studio.  If you already have a licence for this then it is ok to install a second copy for your build server, but what if the build server is at a client premises and they don’t want to buy a license? I came across this stack overflow post that pointed me to an MSTest Agent download for 2010. You can download the Agents for Visual Studio 2013 from here. All you need is the executable so don’t install the agent as a service and you don’t need the controller.

Hope that helps.