SSDT: SQL Project Options

To set the properties for your SQL Project, right-click the Project and select Properties. You'll see a window containing several tabs to define your project.
 
Project Settings
Here you can set your normal target platform (SQL 2005, 2008, 2012, or Azure).
You can enable the T-SQL verification option, but this is mostly for Azure projects.
If you will generally output to a script, choose the "Create script" option. On the whole, just using the dacpac file will likely provide greater flexibility in the long run as it can be adapted to a different target. Generating a script will only be guaranteed to work against the target used to generate the script.

You can also set the dacpac file properties, which sadly do not do much when used against a database not using these as data-tier applications. There is some talk of enhancing the projects to use this version number and description somewhere within the database, but right now they are pretty much only there for you to populate if you wish.

 
You can also set the Database Settings for your project. These options are the normal ones you would set when setting options in SQL Server. You can choose to publish these to your targets so set these to your preferences. (More on that in a future post.)



I'm going to bypass the SQLCLR and SQLCLR Build tabs as I am not familiar with them and don’t currently use either in our system.
 
Build
The Build tab is the place to specify which warnings to ignore, the output path for your files when you build the project, and the name of the file. You may want to specify certain warnings to ignore, especially if you use cross-database projects or any references to system databases. Those can often result in warnings about missing objects, even though they'll work perfectly fine when you publish them.

• Note that the warnings are set per configuration. If you use multiple configurations be sure to set the warnings in each one.

 

 
SQLCMD Variables
SQLCMD variables can be used throughout the project as placeholders for values passed in at publication time. For example, you may choose to change the way your scripts run based on your Development, Customer Acceptance, or Production environments. Define these variables here and use them wherever applicable in the project by inserting them as $(DeployType). SSDT will substitute them with the proper value on publication.

 
Build Events
Build events can be set up to run certain commands before and after the project is built. You can also choose to run the Post-Build event commands only when the project builds successfully or on every build.
 
Debug
Start Action and Target Connection String should make sense for those who need to change them. One item to note is that the debug action defaults to a SQL Express type instance of your database that is released to (localhost)\DBName. This will run within SSDT so you can try out your T-SQL before committing the changes in the project.
** Note that if you use FILESTREAM, you will not be able to use these user instances. You will need to change your debug database to a SQL Server install.
In your Deployment Options, you have a screen that looks something like this:

"Block incremental deployment" is one to be aware of. If you run into any constraints, refactoring, data type changes, NULL to NOT NULL, or similar changes, you could have your debug action stop unexpectedly. It can be really useful to avoid dropping columns that could contain data, but if you are aware of the changes, this option could also get in the way of normal releases. According to the MS documentation, this will only block the change if there is data in the table.
"DROP Objects in target but not in project" can be useful to make sure that everyone is running with what's in the project and that they don't have leftover objects that could interfere with testing. However, it's also possible to lose a bunch of work if you publish a project with this option against your local instance containing the stored procedure you've worked on for the last 2 weeks.
"Advanced" contains a lot of different options about what to compare and how to generate the changes. Some of the more interesting options include:

• Allow incompatible platform - this will let you push your changes to a version of SQL Server other than the one specified in the project properties. The changes will fail if you try to do something not allowed in the target, but useful in case you have your project set to the lowest version supported.
• "Drop xyz not in source" - these options control what will be dropped from the target if it's not defined in the source. The default options are likely good for most users.
• "Ignore xyz" - defines what should be ignored when comparing the project against the target. You may want to ignore details about the partition schemes, or what permissions are set in the target
• "Include Transactional Scripts" will set up the release script as transactions when publishing the database. This is useful when you want to make sure that if one part of the publish action fails, the change script is rolled back.
• "Verify Deployment" - stops the publish actions if the script could run into some problems.

 
Reference Paths
This section is used to modify and maintain the various Server & Database variables used with cross-database references. See the "External Database References" page for more detail.
 
Code Analysis
This option can be useful to check for common issues such as using reserved words, non-standard characters in object names, or using "select *" in your queries. You can also specify whether to show these as errors or warnings. Set appropriately for your environment.

SSDT: Importing an Existing Database

To Import From an Existing Database
Import Database into your project by right-clicking the project name or selecting the Project Menu and selecting the Import -> Database option. Note that you can also import from SQL Scripts or an existing dacpac file. One of those may be necessary if you don't have direct access to your source server.

You'll then be asked to choose your project name, location, folder structure, and what to import. You can choose the defaults if you want. If you have an intricate permissions structure, you may want to import the permissions. If your logins, users, and permissions tend to vary by environment, you may want a different way to handle those. That will be covered in the "Permissions" section.
You'll have several options for folder structure. I'd choose the recommended structure unless you have a reason to choose something else. That will structure all of your objects under their appropriate schema, by object type. If you have a relatively simple database model, you may choose to structure your project by schema alone, object type alone, or just put everything in the root of the project. This only affects your project file structure, not your database structure.
Once you're satisfied with your settings, click the "Start" button to begin populating your project.

 
Importing From a Dacpac File
SSDT supports importing from a dacpac file. This can be useful if you're working on a project for a database to which you do not have direct access in order to import the schema directly. These files also can be used as external database references if you work in an environment where multiple databases interact.
In order to create a dacpac file, you have a couple of options.
You can generate a dacpac file from a SQL Server database using SSMS 2012. Right-click the database you wish to use. Select Tasks, then "Extract Data-tier Application…".

Set the file location and any other options you want to use.

Click next. Verify the settings. Click Next again. The system will now generate a dacpac file in the location you specified.
Note that choosing the "Export Data-tier Application" will create a bacpac file. This is used to backup all schema and data in a database so it can be published to an Azure database. You cannot use a bacpac file to import schema into your project.
 
Generating a dacpac using SQLPackage.exe
You can always use the SQLPackage command line to extract the DB Schema into a dacpac file. For a complete list of parameters, see http://msdn.microsoft.com/en-us/library/hh550080%28v=VS.103%29.aspx.
For a quick extraction from a trusted server, you can run something like the following:

Code Snippet

1. Sqlpackage.exe /a:extract /ssn:localhost /sdn:Adventureworks2008 /tf:Adventureworks2008.dacpac

This command uses an "Action" to Extract the file, with the SourceServerName of localhost, a SourceDatabaseName of Adventureworks2008, and a TargetFile of Adventureworks2008.dacpac

SSDT: Creating a New SQL Project

Download the latest SSDT package if you do not have the product installed already:
http://msdn.microsoft.com/en-us/data/tools.aspx
 
To create a new SQL Project, you can either start by creating a new Project or create a new project from an existing database.
When creating a brand new project, be sure to choose the "SQL Server Database Project".

 
This is not the same as the Database -> SQL Server Project template. Using this will create a usable DB Project, but not the newer SQL Project.

 
Once you've created a new, empty project, you'll need to populate the project in order to be able to use it. You have a couple of options. You can import from an existing database, import from a Dacpac file, or just start creating your objects if this is a completely new project.
 
To Start a New Project from a Connected Database
Open your "SQL Server Object Explorer" window. Connect to the database server containing the database for which you want to create a project. Right-click that database and choose the "Create New Project…" option.

You'll then be asked to choose your project name, location, folder structure, and what to import. You can choose the defaults if you want. If you have an intricate permissions structure, you may want to import the permissions. If your logins, users, and permissions tend to vary by environment, you may want a different way to handle those. That will be covered in the "Permissions" section.
You'll have several options for folder structure. I'd choose the recommended structure unless you have a reason to choose something else. That will structure all of your objects under their appropriate schema, by object type. If you have a relatively simple database model, you may choose to structure your project by schema alone, object type alone, or just put everything in the root of the project. This only affects your project file structure, not your database structure.
Once you're satisfied with your settings, click the "Start" button to begin populating your project.

SSDT: Why Use SQL Projects?

What Are SQL Projects and why use them?
SQL Server has always been lacking in a good solution to control versions of your database. We make changes directly to the database and if we're good, we remember to take backups. If we're really good, we save scripts so we can make those changes elsewhere. Too often, we resort to tools for schema differences or try to save up all of the scripts to replay them later. Both have some advantages. Schema differences guarantee that you can make two environments look the same. Scripts can be replayed in order to keep the systems in sync. However, a schema difference doesn't handle data changes well and can't handle making partial changes when not everything is ready to promote. Sets of scripts can also work well, but fail if you end up needing to push sub-sets of those scripts because certain functionality isn't going to be released yet.
Borrowing from Gert Drapers' excellent discussions on Database Development Challenges:

• Databases are inherently stateful
  • ALTERs are generally run instead of DROP/CREATE
  • Dependencies can greatly complicate the changes and scripts
  • Data needs to be persisted
• Databases are often not integrated well with the application development life cycle
• DB Versions are frequently not persisted or managed well
• SQL Scripts may not handle different SQL Server versions well

Sequential Change Scripts

• Required to run in a set order
  • Script 1, Script 2, Script 3, …
  • Can't do Script 1, Script 4, Script 9
• Validating the end state after script runs vs. an expected end state can be complex
• Could easily miss a script in the sequence without knowing
• Handling a release where scripts need to be "skipped" can cause cascading dependency problems
• Can require extensive use of IF NOT EXISTS statements
• Manual process
• Point in time model
• Used by systems such as DBDiff

Declarative Model

• The model is the desired end state
  • Compare the current state of the target against the desired end state
  • Use the differences to generate an upgrade plan
  • Use the upgrade plan to generate a change script, correctly ordered to bring the target up to the desired end state
• Generated / Programmatic process
• Always current
• Used by Red Gate, Microsoft, DB Ghost

Microsoft and Red Gate have both provided solutions to store versions of database schemas. While Red Gate offers a good product, it also requires some external tools to provide a complete solution for schema control. Until recently, Microsoft's solution wasn't accessible to everyone because it required a copy of Visual Studio to handle even the basics. This improved in Visual Studio 2010, but still required a separate purchase. With SQL Server 2012, Microsoft introduced SQL Server Data Tools and SQL projects.
With SQL projects, developers can store a version of their database objects in a source control system of their choice. Tables, stored procedures, views, and even permissions can be stored in such a way as to provide upgrades to databases and even build a new set of databases from scratch. This could even be used to build and release these databases continuously in an automated fashion to release new code.

Tales of a Lazy DBA–MySQL, SSIS, and “0” Dates

We’ve recently been tasked with converting a lot of MySQL Data into our system from a multi-tenant MySQL Database. We don’t have a fixed schedule to actually execute the imports because they’ll be on a “per customer” basis.  Well, that sounded like a great task for SSIS.  I set up the ODBC driver, connected, set the initial “sql_mode” options for our connections to (‘mssql, allow_invalid_dates’) and started to work.
First problem we ran into with an ADO.NET connection to MySQL and writing a SELECT * from schema.table was when we hit a MySQL “Date” column containing a value of ‘0000-00-00’.  SSIS threw an error, not sure what to do.  Thanks to some others who have solved this problem, I realized that within the MySQL Select statement, we could do something like:

CASE date_created WHEN '0000-00-00' THEN NULL else date_created END as date_created

to pass those dates over as NULL. That solves the implicit conversion to datetime (SQL 2005) and avoids the invalid dates.  I ran something similar for a “Time” column to:

CAST(Time_Created as char(8) ) as Time_Created

 
So that solved one particular table export with about 20 or so CAST and CASE statements.  Needless to say, I wasn’t looking forward to doing this for another 500 tables with a total of almost 6000 columns.
I finally set up a really basic query to generate most of the SELECT statements we would need to pull our MySQL data across without too much pain. Admittedly, a small step and I’d still need to copy/paste when I set up each new ADO.NET source, but it worked reasonably well. I’m adding the code snippet for MySQL here in case anyone else has a similar problem.

select

CONCAT(CASE WHEN ordinal_position = 1 THEN 'SELECT ' ELSE ', ' END,

 CASE Data_Type WHEN 'date'

 THEN CONCAT('CASE ',column_name,' WHEN ''0000-00-00'' THEN NULL else '

    ,column_name,' END as ',column_name)

 WHEN 'time' THEN CONCAT('CAST(',column_name,' AS CHAR(8) ) as ',column_name)

 ELSE column_name END, CASE WHEN Ordinal_Position <> MaxOrd THEN ''

 ELSE CONCAT('

    FROM MySchema.', c.table_name) END

    ) as Select_Column_Name

from information_schema.columns as c

JOIN (select table_name, MAX(ordinal_position) as MaxOrd

        from information_schema.columns

        WHERE Table_Schema = 'MySchema'

        GROUP BY Table_Name) as t1

    ON c.Table_Name = t1.Table_Name

where table_schema = 'MySchema'

order by c.table_name, ordinal_position LIMIT 0, 50000;

I’d love to hear other ideas if anyone has encountered this before and come up with a more elegant solution for translating “0” Date or Time data from MySQL into SQL Server.

Installing SQL Server, Sharepoint, PowerPivot on a single server

I've been trying to set up a single virtual machine for a proof of concept using SQL Server Denali CTP3, Sharepoint 2010, and PowerPivot. I want to get Project "Crescent" running through this as well to prove out some report concepts. I've had a lot of trouble finding the right steps to get everything installed and working correctly on a single box, especially since my Sharepoint knowledge right now is practically non-existent when it comes to BI. Every time I tried to get the Sharepoint / PowerPivot integration working, I hit a roadblock or managed to corrupt my Sharepoint install.

I had the opportunity to speak with Brian Knight ( blog | twitter ) and he pointed me to PowerPivotPro.com which, in turn, led me to PowerPivotGeek.com. They had a page of instructions for single-server installs with a pointer to an MS whitepaper and a private cached file of the whitepaper just in case. This gave me the information needed to get PowerPivot working on my VM.

A couple of notes from my personal experience:

• Install Sharepoint SP1 right after installing Sharepoint 2010. SP1 is required when you're using Denali
• Do NOT configure Sharepoint until the directions tell you to do so.
• Use a Named Instance in order to get PowerPivot working.
• If you plan to use Crescent, make sure that you set up SSAS with the "Tabular Data" option (or whatever the final name ends up being).

Once again, thanks to Brian for pointing me towards the right solution. Thanks to PowerPivotGeek for hosting these files. I imagine the instructions would work for Virtual Server, Virtual PC, VMWare, VirtualBox, or similar Virtual Machine apps. I hope this is helpful to anyone else trying to set up their own VM for testing out the Microsoft BI stack.

Service Broker–WAITFOR and Activated Queues

We’ve implemented Service Broker for handling a small portion of the transactions we want to catch and apply into our Operational Data Store. Mostly we wanted to make sure we accounted for actual DELETE operations and handled those records correctly. We can use Change Data Capture at this time on our source systems and Service Broker seemed to fit the bill. We still handle the remaining inserts and updates through SSIS.

While running Service Broker, we noticed a pretty constant CPU hum on the receiving server. That seemed odd, but we had a lot of trouble tracking it down. Regular Profiler traces didn’t show any running TSQL, “normal” Service Broker traces weren’t showing much, either. We just saw SQL Server running at a pretty constant 20% even with nothing seemingly happening.

I want to give a public thank you to Mark Hill (twitter) for doing a little extra digging and catching the root cause of this. In our tinkering with Service Broker, Queues, Activation, Stored Procs, and such we missed some very important information along the way.

We had written a stored procedure that would be used on the Receiving side of our Service Broker queue to run as the “Activated” proc. Inside the stored procedure, we included a WHILE loop to process anything that was in the queue. If nothing, exit.  That seemed pretty simple – if something comes into the queue, process it and stop when nothing is left.  End of story, right?

Sadly, this is where we missed a small, but important, fact about Service Broker. If you have an activated stored procedure that doesn’t use the WAITFOR command, Service Broker, we execute that stored proc as many times as possible looking for something to process. We looked at it, turned off the Activation on the queue and saw the CPU drop to nothing. We re-activated the queue and saw the CPU shoot up again. We tweaked the stored procedure after that and added a WAITFOR command with a timeout of 60000. After doing this, we saw the stored procedure run to process everything in the queue, then go idle, which was exactly the intended behavior in the first place.

I’m not going to pretend to be an expert on Service Broker. We had tried to code with the intention of being able to use this stored procedure as an Activated stored procedure or as one called from an external process to work through the queue. While that may be possible, it was unnecessary in our actual usage. Adding a WAITFOR command around our queue processing eliminated our extra, non-essential CPU usage and stopped trying to execute a stored procedure for no reason.

Resources

Pro SQL Server 2008 Service Broker ( Amazon | Apress )

WAITFOR (Also see this performance article – it’s the little details that get you)