Showing posts with label SQL. Show all posts
Showing posts with label SQL. Show all posts

Thursday, April 21, 2016

Getting All Permissions for Server

We recently had to work out a way to fetch and display all effective permissions granted to our Production SQL Server. I had been required to get all permissions at the DB level before, but even then that tended to be more granular and only showed to what roles you belonged and what explicit permissions you'd been granted. We wanted to run something through a loop of all users and get that information into a report. It turns out this was easier than I'd expected, especially after my first attempts to dig through the system tables.

We used a script something like the following:

EXECUTE as login = 'DomainName\Login';
select 'DomainName\Login' AS Login, * from fn_my_permissions (NULL, 'SERVER');
REVERT;

With a little work this can be used within PowerShell or SQL to loop through all logins and capture the information in a table or report.

 

The function could also be used to loop through all databases by using the database and then substituting Database for Server above:

USE AdventureWorks2012;
EXECUTE as login = 'DomainName\Login';
SELECT 'DomainName\Login', * FROM fn_my_permissions (NULL, 'DATABASE');
REVERT;

For an interesting take on revamping sp_msforeachdb, check out this script from Edward Pollack (t). It might help you if you need to loop through all of your databases to run something like this for your auditors.

Monday, January 17, 2011

SQL 2008 Merge and PK/FK Constraints

We ran into this issue a while back.  SQL 2008 has a “feature” that will not allow inserts into a table on either side of a Foreign Key relationship.  If you attempt to insert into a table involved in the FK relationship, you get an error something like:
The target table 'TableName' of the INSERT statement cannot be on either side of a (primary key, foreign key) relationship when the FROM clause contains a nested INSERT, UPDATE, DELETE, or MERGE statement. Found reference constraint 'ConstraintName'.
This is documented in Connect 435031. There is one workaround documented in the ticket – drop the FK constraint, run the MERGE query, then re-create the constraint. Happily, this bug is still open as the requirement to drop a constraint just to run a MERGE statement kind of defeats the purpose of putting a constraint on the table in the first place. While dropping the constraints will work, there is another possible workaround. I was able to run this without any issues while leaving my FK Constraint in place.
  1. Create a temp table that matches the definition of the table into which you want to perform your insert.
  2. Instead of inserting into your base table like you normally would in a merge, write your code to INSERT #Tablename at the beginning of the MERGE process. You’ll still use your main table and staging table inside the MERGE INTO … USING query.
  3. After the Merge, insert into the main table using the values in your Temp table.
Here’s a shell of an example. Line 11 below contains the main difference to the MERGE query from a “normal” MERGE.
   1: --1: Create Temp table
   2: CREATE TABLE #MyFactTable(
   3:     ID INT NULL
   4:     , CustomerName VARCHAR(100) NULL
   5:     , SourceID INT NULL
   6:     , OutputAction VARCHAR(100) NULL
   7: );
   8:  
   9: --2: INSERT into the temp table instead of your normal target table
  10: --   Merge query will be the same otherwise
  11: INSERT INTO #MyFactTable (ID, CustomerName, SourceID, OutputAction)
  12: SELECT so.ID, so.CustomerName, so.SourceID, so.output_action
  13: FROM (
  14:     MERGE INTO dbo.MyFactTable AS t
  15:     USING Staging.MyFactTable AS s
  16:     ON ( s.ID = t.ID
  17:         AND s.NewLoad = 0 )
  18:         
  19:     WHEN MATCHED AND ( s.SourceID <> t.SourceID )
  20:             AND s.NewLoad = 0            
  21:     THEN UPDATE 
  22:         SET RecordState = 0
  23:         , UpdatedDate = getdate()
  24:         
  25:     WHEN NOT MATCHED BY TARGET AND s.NewLoad = 0 THEN
  26:         INSERT (ID, CustomerName, SourceID)
  27:         VALUES (s.ID, s.CustomerName, s.SourceID)
  28:         OUTPUT $action AS OutputAction
  29:             , ID
  30:             , CustomerName
  31:             , SourceID
  32:         ) AS so (OutputAction, ID, CustomerName, SourceID)
  33:         WHERE OutputAction = 'UPDATE'  ;
  34:  
  35: --3: Perform the final insert into your target table
  36: INSERT INTO MyFactTable (ID, CustomerName, SourceID)
  37: SELECT DISTINCT ID, CustomerName, SourceID
  38: FROM #MyFactTable ;
  39:  
  40: --4: Clean up your temp objects.
  41: DROP TABLE #MyFactTable ;

I’d love to hear if anyone else has a workaround for this that doesn’t involve dropping and recreating the FK constraint. If this is an issue for you, please vote for the Connect Ticket.

Tuesday, March 16, 2010

Powershell: Writing Text Files

I was recently trying to generate a bunch of data that I needed to pass through one of our in-house DLLs in order to use some custom encryption algorithms. Thanks to the wonderful Powershell community, I found several examples on using custom DLL methods. However, I ultimately wanted this data to end up in SQL Server. My first attempt used an “Invoke-SQLCmd” Cmdlet from the SQLPS snap-in. However, generating several million rows took a while to insert. I knew that I didn’t want to queue all of this data in memory, so was inserting one row at a time. I stopped this process after a while because it was too time-consuming.

My next attempt was to push these values to a text file using a variable pointing to a file and then using add-content to write each line to the file.  This took longer than doing the SQL inserts one at a time.

I remembered coming across an article by Linchi Shea titled Add-Content and Out-File are not for performance. After finding the link to that article, I found that he recommends using System.IO.StreamWriter to write out files in the most performant manner. I definitely recommend that you check out his article for his summary and examples. For my part, I can say that it took me the better part of 6 hours to generate and populate 10 million rows. I just generated 20 million into a file in < 15 minutes!

I’m including a mockup of my code below in case anyone is interested. I’m definitely not a Powershell expert so as always, be careful if you copy any of this code.  My first attempt at directly inserting into SQL Server worked, but threw an error at about 3.5 million rows. As the time was already pretty lengthy by then, I stopped the process and moved on to writing the text file, resuming where I’d left off. That ran for hours until I finally killed it and looked for a better way.  The uncommented code is what finally ended up working in a timely manner.

I started by loading my DLL. I called it’s “EncryptValue” function (against just the integer – that’s a longer story) and populated both out to a table. There are several reasons for doing this, but this gave me a good working set for a test project and definitely lays some groundwork for the future.  If anyone has any ideas on how I can speed this up, feel free to leave me some comments. If I find some, I’ll post an update.

 

[Reflection.Assembly]::LoadFile("C:\MyDLL.dll")

# First attempt – insert directly into SQL Server.
#foreach ($i IN 1..10000000) { invoke-sqlcmd –query '
#
("insert dbo.MyTable VALUES(" + "$i" + ",'" + [MyDLLNamespace]::EncryptValue("$i") + "')") -serverinstance "localhost" }

#Attempt #2 – Write to file with add-content
#$file = New-Item -type File "C:\Values.txt"

#foreach ($i IN 1..10000000) { add-content $file ("$i" + "," + [MyDLLNamespace]::EncryptValue("$i") ) }

$file = New-Object System.IO.StreamWriter "E:\Values.txt";

foreach ($i IN 10000001..30000000) { $file.Writeline("$i" + "," + [MyDLLNamespace]::EncryptValue("$i") ) }

$file.close();

Monday, February 8, 2010

DDL Schema Change Auditing on SQL Server 2005 / 2008

I’ve run across this before so wanted to write up a solution to keeping track of schema changes we’ve used. We trust our team to make DDL changes, additions, drops in order to get their job done. We also trust them to save those scripts into source control as they are run so we can use them when we release software. However, there are times that people make changes and forget to save the script or just think that it’s a temporary change to test and forget about it. With that in mind, we decided to take advantage of SQL Server 2005’s DDL triggers.
First, we created a database called [Audit] on our development server.
Next, we created a table to store the logs.
USE Audit
GO
CREATE TABLE [dbo].[DDL_Audit](
[DDL_Audit_ID] [int] IDENTITY(1,1) NOT NULL,
[Event_Type] [varchar](100) NULL,
[Database_Name] [varchar](100) NULL,
[SchemaName] [varchar](100) NULL,
[ObjectName] [varchar](100) NULL,
[ObjectType] [varchar](100) NULL,
[EventDate] [datetime] NULL,
[SystemUser] [varchar](100) NULL,
[CurrentUser] [varchar](100) NULL,
[OriginalUser] [varchar](100) NULL,
[EventDataText] [varchar](max) NULL
) ON [PRIMARY]

GO
GRANT INSERT ON DDL_Audit TO public
GO



Note in this example, I granted INSERT permissions to public to avoid needing to give our team any other access to that database. I didn’t want them to read/write rows in that table if I could avoid it. After that, I ran a script in SSMS using Text output to step through all of our databases and generate the trigger create code.


sp_msforeachdb 'SELECT ''use ?
GO
SET ANSI_PADDING ON
GO
SET QUOTED_IDENTIFIER ON
GO

CREATE TRIGGER trg_DDL_Monitor_Change
ON DATABASE
FOR DDL_DATABASE_LEVEL_EVENTS
AS
SET NOCOUNT ON
SET ANSI_PADDING ON

declare @EventType varchar(100)
declare @SchemaName varchar(100)
declare @DatabaseName varchar(100)
declare @ObjectName varchar(100)
declare @ObjectType varchar(100)
DECLARE @EventDataText VARCHAR(MAX)

SELECT 
@EventType = EVENTDATA().value(''''(/EVENT_INSTANCE/EventType)[1]'''',''''nvarchar(max)'''')  
,@DatabaseName = EVENTDATA().value(''''(/EVENT_INSTANCE/DatabaseName)[1]'''',''''nvarchar(max)'''')  
,@SchemaName = EVENTDATA().value(''''(/EVENT_INSTANCE/SchemaName)[1]'''',''''nvarchar(max)'''')  
,@ObjectName = EVENTDATA().value(''''(/EVENT_INSTANCE/ObjectName)[1]'''',''''nvarchar(max)'''')
,@ObjectType = EVENTDATA().value(''''(/EVENT_INSTANCE/ObjectType)[1]'''',''''nvarchar(max)'''')   
,@EventDataText = EVENTDATA().value(''''(/EVENT_INSTANCE/TSQLCommand/CommandText)[1]'''',''''nvarchar(max)'''')

insert into Audit.dbo.DDL_Audit (Event_Type, Database_Name, SchemaName, ObjectName, ObjectType
, EventDate, SystemUser, CurrentUser, OriginalUser, EventDataText)
select @EventType, @DatabaseName, @SchemaName, @ObjectName, @ObjectType
, getdate(), SUSER_SNAME(), CURRENT_USER, ORIGINAL_LOGIN()
, @EventDataText
GO
'''


We took the results of that script and removed any databases that did not matter for purposes of tracking changes. We ran that and created the appropriate triggers on all of the databases. That left us with the notification messages. We handled this through a SQL Server Agent job using Database Mail.


IF EXISTS (SELECT 'x' FROM Audit.dbo.DDL_Audit
WHERE Event_Type NOT LIKE '%statist%'
AND SystemUser NOT IN (service account list here)
AND EventDate >= convert(varchar(10),DATEADD(dd, -1, GETDATE()),101))
BEGIN
DECLARE @email_from nvarchar(100)
, @email_address nvarchar(200) 
, @TheSubject nvarchar(255)

SELECT @Email_Address = 'myemail@myemail.com'
SET @email_from = 'DevServer@server.com'
select @email_address as 'To:' ,  @email_from as 'From:'

set @TheSubject = 'Recent Schema changes on ' + RTRIM(@@SERVERNAME)
DECLARE @tableHTML  NVARCHAR(MAX) ;

SET @tableHTML =
N'<H1>DevDB Schema Change</H1>' +
N'<table border="1">' +
N'<tr><th>Database_Name</th><th>SchemaName</th>' +
N'<th>ObjectName</th><th>Event_Type</th><th>ObjectType</th>' +
N'<th>EventDate</th><th>SystemUser</th><th>CurrentUser</th><th>OriginalUser</th><th>EventDataText</th></tr>' +
CAST ( ( SELECT td = Database_Name,       '',
td = SchemaName, '',
td = ObjectName, '',
td = Event_Type, '',
td = ObjectType, '',
td = EventDate, '',
td = SystemUser, '',
td = CurrentUser, '',
td = OriginalUser, '',
td = EventDataText
FROM Audit.dbo.DDL_Audit
WHERE Event_Type NOT LIKE '%statist%'
AND SystemUser NOT IN (serviceaccount)
AND EventDataText not like '%ALTER%INDEX%REBUILD%'
AND EventDate >= convert(varchar(10),DATEADD(dd, -1, GETDATE()),101)
ORDER BY Database_Name, ObjectType, ObjectName, EventDate, Event_Type
FOR XML PATH('tr'), TYPE 
) AS NVARCHAR(MAX) ) +
N'</table>' ;

EXEC msdb.dbo.sp_send_dbmail
@profile_name = 'Default' ,
@recipients=@email_address,
@subject = @TheSubject,
@body = @tableHTML,
@body_format = 'HTML' ;

END



We scheduled this code in a job step that runs daily at 10am. Any changes in that time frame were then sent to a team of people on a daily basis. If no changes were made, no email was sent. Hopefully this will give someone some ideas on one way to audit changes to their DBs. I make no claims that this is the best way, but it worked well for us.

You can download the script file containing all of this code using the link below.

Thursday, September 17, 2009

Data Loads

One of the things I really like about SQL Server Integration Services is that it can load large data sets quickly, especially if you need to look up values along the way.  However, there are times that things behave in strange manners that seem hard to explain.  We were recently trying to load about 64 million rows into a warehouse-type database and kept hitting a bottleneck once we got past around 30-35 million rows.  No matter what I tried, the load would run at about 2 million rows per minute, gradually slowing down until it reached the 10,000 rows per second load speed.  I found several helpful posts, including some pointers to check the MaxRowBufferSize and similar settings.  Tweaking these helped someone who was on a memory-bound machine with a very similar problem.  I tried that and saw my initial load times improve, but still slowed down to a crawl at around the same point.

I learned more about looping through (aka "shredding") a recordset of CustomerID values to try to go through that set.  I set up a FOR Loop to loop through 5 million records at a time, I checked for processes running against the table. Nothing seemed to help.  I had posted information on Twitter, had a co-worker looking at the package with me, and even got some replies from Paul Randal.  (Thank you for the assist, Paul - sorry I wasn't checking my tweets more regularly that time.)

In the process above, I was challenged trying to set the CustomerID in some dynamic fashion on my source query and eventually resorted to treating the SQL Command as an expression and hacking in my variables into the WHERE clause.  It worked, but definitely felt awkward. I am more than open to some suggestions about how to use SSIS variables inside of a FOR loop as part of the OLEDB Source Command.  I also learned a little more about watching variable values - like needing to have a breakpoint set before they'll be available.  And my co-worker pointed me to a wonderful discussion showing that DtExecUI runs only in 32-bit mode. I'd been using that as a quick/easy way to get my parameters plugged in and to keep an eye on the general progress of the load.

About the same time that Paul suggested checking my index fragmentation levels, I remember seeing a very similar behavior pattern several years ago when a co-worker was trying to update a column in the middle of a somewhat wide clustered index on a multi-million row table. The server was thrashing all over the place, trying to re-order millions of rows as this command took place.  I checked the table and sure enough, there was a wide clustered index on the table that would cause exactly that behavior.  I don't know why I didn't check it before other than thinking that I'd just created a test table and no indexes on it.  In retrospect, that was poor planning on my part.  For the future, I'll remember to check clustered indexes before I try to load millions of (unsorted) rows into a new table.

It was a somewhat humbling experience, mostly because one of the basic things I know about loading lots of data was also something that I just completely ignored when doing all of my troubleshooting. Still, a little humility is a good thing and I've now re-learned several programming techniques in the process. The best part about this was seeing a 6 hour load process reduced down to 30 minutes, including dropping and re-creating the clustered index. That's going to make a noticeable difference in our nightly processes.