MSSQLTips.com - your daily source for SQL Server tips

Google
 
Web mssqltips.com

ESSENTIALS: Home | Tips | Search | Categories | Top 10 | Products | Authors | Blogs | Forums | Webcasts | Advertise | About
Designing Tables for Audit Data in SQL Server -

in Search

Designing Tables for Audit Data in SQL Server

Last post 04-11-2008 10:08 AM by admin. 4 replies.
Page 1 of 1 (5 items)
Sort Posts: Previous Next
  • 04-02-2008 12:30 AM

    Designing Tables for Audit Data in SQL Server

    This post is related to this tip: Designing Tables for Audit Data in SQL Server

    http://www.mssqltips.com/tip.asp?tip=1468

  • 04-02-2008 11:56 AM In reply to

    Re: Designing Tables for Audit Data in SQL Server

    I also have to do data change auditing on our database.  Our current implementation has been in place for several years but I am looking at changing it.  The new approach is based on an article by Itzik Ben-Gan a couple years ago.  Below is a summary of the current approach and the proposed approach.  Any comments would be greatly appreciated.

     

    Current Approach

    Currently auditing is done with 3 "INSTEAD OF" triggers (trg<tableName>Insert, trg<tableName>Update, trg<tableName>Delete) on each table and 2 auditing tables (AuditLog and AuditDetail).

    The insert trigger copies the data from Inserted to a temp table, calls a procedure that populates the AuditLog table with the table name, Table PK, Operation, who made the change, the time of the change and what computer the changes was made on. The procedure then inserts a row for each column of the table into the AuditDetail table. This contains the column name, old value (always NULL in this case) and new value. Once the procedure is completed the temp table is dropped and the data from INSERTED is inserted into the base table.

    The Update trigger copies data from Inserted and Deleted into temp tables and calls a procedure that populates the AuditLog table just like the insert trigger and then compares the inserted and deleted temp tables column by column and inserts a row in the AuditDetails table for each of the columns that have changed. The temp tables are then dropped and the base table is updated.

    The Delete trigger copies data from Deleted into a temp table and calls a procedure that populates the AuditLog table and inserts a row for each column of the table into the AuditDetail table and then deletes the record in the base table.

    Pros: Only 2 additional tables required.

    Cons: Performances is terrible in a table with millions of records. Bulk changes to a table take hours.

    New Approach

    What I am considering for the new approach is this.  One trigger on each table and 3 additional tables per audited table. The trigger identifies the type of statement that fired the trigger and the number of rows affected. Uses table variable instead of temp tables which allow roll back of a change with ability to log the attempted change. Has logic to log attempted changes to the data and block the change (e.g. changing the PK column or integrity violations).

    Each audited table will have a <TableName>AuditHeader table that contains the dmltype, date of change, who made the change, application name, what computer the change was made from, a failed flag and comment if the attempted change failed or was blocked.

    Each audited table will also have a <TableName>InsAuditDelDetail table that contain a FK to the header table and a column for each of the columns in the base table. This table will store the full row of data for inserts and deletes.

    The final table is <TableName>AuditUpdDetail. This table contain a FK to the Header table, the name of the column, the old value and new value for each of the columns that changed in the update.

    Pros: Performance is very good. I ran a change of several hundred thousand records and the time went from about 4 minutes to 6 minutes. A comercial package using two audit tables ran for 20 minutes and crashed. Current version ran for several hours and I killed it.

    Avoids hot spots created by all user changes resulting in hitting just 2 tables (the detail in particular).

    Cons: A lot of extra tables. May be more difficult to generate a view of all changes depending on requirements

    Admittedly bulk changes do not happen often but when they do it is a killer and we are seeing performance problems under normal use with the current approach when the log grows too large. Some of this can be mitigated by table and index changes.

    The <TableName>AuditUpdDetail could be eliminate and the <TableName>InsAuditDelDetail used. The before and after values would need to be determined by comparison of the 2 records. I have seen this approach also. Another downside to this is that you are storing an entire record even if only one column changed (a big hit for tables with image data, file attachments, etc.)

     

     

  • 04-04-2008 7:21 AM In reply to

    Re: Designing Tables for Audit Data in SQL Server

    MarkJo_FP,

    Thank you for sharing some of the challenges you are facing today with auditing.

    Although your new solution does have many tables it sounds like the performance is acceptable and is something you are willing to manage.

    How do you think a generic solution either with an XML column or a generic table would fair in that scenario?

    We have implemented both solutions under different circumstances and they have appeared to perform well and meet the overall reporting needs.

    Just interested in your thoughts on the matter as opposed to recommending a different approach.

    Thank you,
    The MSSQLTips.com Team

     

  • 04-08-2008 9:30 AM In reply to

    Re: Designing Tables for Audit Data in SQL Server

    If I am understanding your generic solution, I think it is very similar to out current approach of using 2 tables.  Your implementation has the flexibility to audit selected columns in a table whereas we audit the entire table.  I assume you use triggers to populate these tables when a change is made. 

    You are storing the table and column information in one table whereas we have a header table where we store the table name, rowid, operation, modifiedby, modifieddate and computername and then a child table which stores the column name, old value and new value.

    Since you are storing only one value for the column you are apparently storing either the new value for the column or the old value for the column.  How do you distinguish between inserts, updates and deletes when reporting?  How does it handle failed update attempts? We have an auditlog viewer as part of the product that allows searching the log based on a variety of criteria.  Without storing the old and new values I assume you would need to join on the previous change for that column.  Storing the Old and New values is where we ended up with the temp tables and INSERTED and DELETED tables as I described in my post.

    Have you tried doing a bulk load (or change) of data with your auditing in place.   It is not very common but I do run into it once in a while.

    If I am missing something in my assumptions above let me know.  I am always looking for better ways to deal with the auditing issues. 

    Thanks.

    Mark

     

     

     

     

  • 04-11-2008 10:08 AM In reply to

    Re: Designing Tables for Audit Data in SQL Server

    Mark,

    Thank you for the lengthy reply and feedback.

    Generic tables: 

    • You are correct, only specific columns are being audited with the 2 table generic solution.  The requirement is not to audit the entire row.
    • For the population, some of the data is populated via a trigger and other data is actually populated by calling a stored procedure when a specific set of logic is called in the front end application.
    • We are storing new values and use the start date to determine when the new value went into effect.
    • We do not have a need to distinguish between INSERT, UPDATE and DELETE statements.  I could see how that could be very important in some environments.
    • We are not trying to capture failed attempts to INSERT, UPDATE and DELETE data.  Once again that is not a requirement, but with some data is sometimes more important than a normal change.
    • Reporting is primarily from the front end application or custom reports.

    XML tables

    • This option has much more flexibility and can include many of these items based on the XML schema.
    • Once again may of the items you mentioned were not requirements, but I think this solution has a much easier means to change schema with less development time, but that could change based on implementation details.

    Thank you,
    The MSSQLTips.com Team

Page 1 of 1 (5 items)