<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>data warehouse &#8211; Cognim &#8211; Internet development</title>
	<atom:link href="https://www.cognim.co.uk/tag/data-warehouse/feed/" rel="self" type="application/rss+xml" />
	<link>https://www.cognim.co.uk</link>
	<description>Enterprise system implementation. Making the complex simple</description>
	<lastBuildDate>Mon, 27 Jul 2015 16:59:11 +0000</lastBuildDate>
	<language>en-GB</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	
<site xmlns="com-wordpress:feed-additions:1">91553907</site>	<item>
		<title>How to export a lot of data from on-prem SQL Server to Azure SQL for testing</title>
		<link>https://www.cognim.co.uk/how-to-export-a-lot-of-data-from-on-prem-sql-server-to-azure-for-testing/</link>
					<comments>https://www.cognim.co.uk/how-to-export-a-lot-of-data-from-on-prem-sql-server-to-azure-for-testing/#respond</comments>
		
		<dc:creator><![CDATA[Darren Hall]]></dc:creator>
		<pubDate>Thu, 02 Jul 2015 09:52:06 +0000</pubDate>
				<category><![CDATA[Azure]]></category>
		<category><![CDATA[SQL Server]]></category>
		<category><![CDATA[SSMS]]></category>
		<category><![CDATA[data warehouse]]></category>
		<category><![CDATA[SSIS]]></category>
		<guid isPermaLink="false">http://www.cognim.co.uk/?p=4929</guid>

					<description><![CDATA[Lately we have created several data marts in Azure SQL that will be fed from a data warehouse hosted on premises. The data mart is first created and tested locally so that we can get the structure and required stored procs correct. Whilst we are doing this it is easy to quickly transfer representative data to [&#8230;]]]></description>
										<content:encoded><![CDATA[<p>Lately we have created several data marts in Azure SQL that will be fed from a data warehouse hosted on premises. The data mart is first created and tested locally so that we can get the structure and required stored procs correct. Whilst we are doing this it is easy to quickly transfer representative data to test the efficiency. Every now and then though we need to test everything on Azure so that we can be sure we are still operating at an acceptable speed. This is where the problems begin. Linked servers are a no-go if you are transferring large amounts of data to Azure which will treat every insert as a separate transaction. We could back up the database, copy it to blob storage and restore from there but that takes a while as well and sometimes it is just a single large staging table that we want to populate. We could hand craft an SSIS package which would be fast but seems overkill considering the amount of times we adjust and retest.</p>
<p>The answer is actually quite simple thanks to a little promoted feature of SSMS. Follow the steps below to see how</p>
<ol>
<li>Right click the database you wish to export data from</li>
<li>Select &#8216;Tasks&#8217; then &#8216;Export Data&#8217;<img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter wp-image-4937 size-full" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/Tasks-ExportData.png?resize=516%2C530&#038;ssl=1" alt="Tasks-ExportData" width="516" height="530" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/Tasks-ExportData.png?w=516&amp;ssl=1 516w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/Tasks-ExportData.png?resize=292%2C300&amp;ssl=1 292w" sizes="auto, (max-width: 516px) 100vw, 516px" /></li>
<li>You will see theSQL Server Import and Export Wizard. Click &#8216;Next&#8217; to go past the welcome screen and make sure your source is correctly selected (it will have been if you started by right clicking the database), then click &#8216;Next&#8217; again<img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter wp-image-4938 size-full" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/YourSource.png?resize=566%2C579&#038;ssl=1" alt="YourSource" width="566" height="579" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/YourSource.png?w=566&amp;ssl=1 566w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/YourSource.png?resize=293%2C300&amp;ssl=1 293w" sizes="auto, (max-width: 566px) 100vw, 566px" /></li>
<li>Now you can choose the destination database, note that for Azure you need to select &#8216;Net Framework Data Provider for SqlServer&#8217; in the destination drop down</li>
<li>This will cause connection properties to appear. For Azure you need to set the data source,userid, password andinitial catalog. I would also recommend setting connection encryption as I have below.<img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter wp-image-4932 size-full" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/AzureConnection.png?resize=566%2C1027&#038;ssl=1" alt="AzureConnection" width="566" height="1027" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/AzureConnection.png?w=566&amp;ssl=1 566w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/AzureConnection.png?resize=165%2C300&amp;ssl=1 165w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/AzureConnection.png?resize=564%2C1024&amp;ssl=1 564w" sizes="auto, (max-width: 566px) 100vw, 566px" /></li>
<li>You can now either copy data directly from a table or view or write your own query. Such power! Here I have elected to copy data from a table in the source database, but bear in mind that you could easily write a query to limit the number of rows or transform the data in some way.<img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter wp-image-4935 size-full" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/SelectTable.png?resize=566%2C604&#038;ssl=1" alt="SelectTable" width="566" height="604" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/SelectTable.png?w=566&amp;ssl=1 566w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/SelectTable.png?resize=281%2C300&amp;ssl=1 281w" sizes="auto, (max-width: 566px) 100vw, 566px" /></li>
<li>Choose the table or tables with data you want to copy. The wizard will automatically try to choose a matching table on your destination database or suggest auto creating it if it does not exist. Click &#8216;Edit Mappings&#8217; for each table you are transferring to see where your data will go.<img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter wp-image-4936 size-full" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/SelectTablesToExport.png?resize=566%2C604&#038;ssl=1" alt="SelectTablesToExport" width="566" height="604" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/SelectTablesToExport.png?w=566&amp;ssl=1 566w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/SelectTablesToExport.png?resize=281%2C300&amp;ssl=1 281w" sizes="auto, (max-width: 566px) 100vw, 566px" /><br /> Here I have chosen two tables that exist in both source and destination and have the same structure and a third table that is different on the destination database. By highlighting that row and clicking &#8216;Edit Mappings&#8217; you can choose exactly where your data will go on a row by row basis. Again, the wizard will auto suggest the best fits.<img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter wp-image-4933 size-full" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/Mappings.png?resize=609%2C506&#038;ssl=1" alt="Mappings" width="609" height="506" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/Mappings.png?w=609&amp;ssl=1 609w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/Mappings.png?resize=300%2C249&amp;ssl=1 300w" sizes="auto, (max-width: 609px) 100vw, 609px" /><br /> Here there are no columns that are named the same so the wizard leaves it entirely up to you. You can also choose to preview the data.</li>
<li>Having mapped your source to your destination, click &#8216;Next&#8217;. Thefinal step is to either run the package immediately or save it as a package toSSIS. I usually run immediately if it is just a quick transfer I am after. Here I have selected both.<img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter wp-image-4934 size-full" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/RunOrSave.png?resize=566%2C604&#038;ssl=1" alt="RunOrSave" width="566" height="604" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/RunOrSave.png?w=566&amp;ssl=1 566w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2015/07/RunOrSave.png?resize=281%2C300&amp;ssl=1 281w" sizes="auto, (max-width: 566px) 100vw, 566px" /></li>
<li>Finally you can click &#8216;Finish&#8217; to execute the package. Note that you have created an SSIS package and you will see feedback on the progress as it goes, including number of rows transferred. If it encounters any problems it will stop and tell you.</li>
</ol>
<p>Overall an excellent way to quickly transfer data from one system to another. Especially useful where you need to transform that data, map between two different schemas or send it to Azure.</p>
<p>What are you waiting for? Go Play!</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.cognim.co.uk/how-to-export-a-lot-of-data-from-on-prem-sql-server-to-azure-for-testing/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">4929</post-id>	</item>
		<item>
		<title>Adding tables of static data to an SSDT project</title>
		<link>https://www.cognim.co.uk/adding-tables-of-static-data-to-an-ssdt-project/</link>
					<comments>https://www.cognim.co.uk/adding-tables-of-static-data-to-an-ssdt-project/#comments</comments>
		
		<dc:creator><![CDATA[Darren Hall]]></dc:creator>
		<pubDate>Mon, 15 Sep 2014 15:54:06 +0000</pubDate>
				<category><![CDATA[SQL Server]]></category>
		<category><![CDATA[SSDT]]></category>
		<category><![CDATA[Visual Studio]]></category>
		<category><![CDATA[data warehouse]]></category>
		<guid isPermaLink="false">http://makesharp.wordpress.com/?p=71</guid>

					<description><![CDATA[Quite often when creating a database you will need tables of static data.  These range from simple lookups to date dimensions in a data warehouse.  You can easily create these in an SSDT project. Presuming you have already designed your tables and they are present in your project; Create a folder in your project called [&#8230;]]]></description>
										<content:encoded><![CDATA[<p>Quite often when creating a database you will need tables of static data.  These range from simple lookups to date dimensions in a data warehouse.  You can easily create these in an SSDT project.</p>
<p>Presuming you have already designed your tables and they are present in your project;</p>
<ul>
<li>Create a folder in your project called StaticData then add a sql script to generate the data you require, one for each table.  For instance I have a date dimension table called Dim_Date and have created a script called Populate_Dim_Date.sql.<br /> <a href="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/staticsqlscript.png?ssl=1"><img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter size-full wp-image-76" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/staticsqlscript.png?resize=244%2C258&#038;ssl=1" alt="StaticSQLScript" width="244" height="258" /></a></li>
<li>Next create a script called &#8216;Script.PostDeployment.sql&#8217; in the root of your project by right clicking your project and choosing &#8216;Add&#8217; then &#8216;Script&#8217;<br /> <a href="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/addscript.png?ssl=1"><img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter size-medium wp-image-75" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/addscript.png?resize=300%2C146&#038;ssl=1" alt="AddScript" width="300" height="146" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/addscript.png?w=665&amp;ssl=1 665w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/addscript.png?resize=300%2C147&amp;ssl=1 300w" sizes="auto, (max-width: 300px) 100vw, 300px" /></a><br /> and choosing &#8216;user scripts&#8217; in the SQL Server section, followed by &#8216;Post-Deployment Script&#8217;, then clicking the Add button.<br /> <a href="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdeploymentscript.png?ssl=1"><img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter size-medium wp-image-74" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdeploymentscript.png?resize=300%2C207&#038;ssl=1" alt="PostDeploymentScript" width="300" height="207" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdeploymentscript.png?w=955&amp;ssl=1 955w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdeploymentscript.png?resize=300%2C207&amp;ssl=1 300w" sizes="auto, (max-width: 300px) 100vw, 300px" /></a></li>
</ul>
<p>This will create a script that runs following the full deployment of your project and you can direct it to step through each of your sql creation scripts in the StaticData folder we created earlier. To do this you simply add a SQLCMD read command to the post deployment script for each of the sql files you want to execute. Here I have added a command to read (include) the script I created for the date dimension.</p>
<p><a href="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdepscript.png?ssl=1"><img data-recalc-dims="1" decoding="async" loading="lazy" class="aligncenter size-medium wp-image-77" src="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdepscript.png?resize=300%2C135&#038;ssl=1" alt="PostDepScript" width="300" height="135" srcset="https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdepscript.png?w=776&amp;ssl=1 776w, https://i0.wp.com/www.cognim.co.uk/wp-content/uploads/2014/09/postdepscript.png?resize=300%2C135&amp;ssl=1 300w" sizes="auto, (max-width: 300px) 100vw, 300px" /></a></p>
<p>Note that to prevent an infestation of the red squigglies you will need to select the icon that represents placing the editor in SQLCMD Mode, in my image above it is the icon with a blue border.</p>
<p>Now when you publish your project your tables will be populated according to your scripts &#8211; easy peasy!</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.cognim.co.uk/adding-tables-of-static-data-to-an-ssdt-project/feed/</wfw:commentRss>
			<slash:comments>1</slash:comments>
		
		
		<post-id xmlns="com-wordpress:feed-additions:1">71</post-id>	</item>
	</channel>
</rss>
