This is the final in a series on shrinking and transactions. You can see them all in the Shrinking & Transactions category.

In part 1, we talked about factors that affect transaction log growth. We talked about what a SQL Server transaction is and the various modes SQL interacts with a transaction (Autocommit, Implicit Transactions and Explicit Transactions).

The theme of that post was about managing your transactions so they don’t manage you. In that same theme, let’s continue being in charge of our transactions. Rather than from a physical size perspective, let’s look at it from a performance standpoint.

Database Engine/Administration Factors

When it comes to ensuring that transaction logging and transactions don’t cause performance bottlenecks there are a few things DBAs should remember and look at, I’ll highlight a few of them.

Pay careful attention to the I/O of your transaction log – As I discussed in Part 1, the transaction log record needs to be written first and this is vital to remember. Keep your Transaction log on fast I/O, separate I/O and follow best practices with your log file.

A great blog post by Kimberly Tripp with eight tips here goes through many more details that I have often hinted about in various posts. Some of her tips are best practices for the data file as well.

She mentions things like: Pre-Allocate Your Space!! This is a huge pet peeve of mine. Plan for your capacity and start out with the size you think you will need. This will stop costly autogrowths which can lead to fragmentation and waits (for the growth.)

Or Defragment the physical volume , people spend a lot of time in the database world looking at index fragmentation (at least I hope you do!) What about your physical storage?

When it comes to choosing your RAID, remember the transaction log is going to want fast writes. These writes are sequential. These writes are writes. RAID 5 (Wikipedia article on RAID) may not generally be a horrible solution for your data (.MDF/.NDF) files. For the Log files, it just doesn’t make sense. RAID 5 incurs extra cost for each write (essentially 2 reads and 2 writes for each write because of the maintenance of parity for fault tolerance).

If cost is an issue, look at just mirroring. This will give you some fault tolerance (Remember, the transaction log is key to a happy, consistent database as we talked about in Part 1) and still gives you the same performance as writing to one disk.

If the mirror doesn’t give you great performance for your high end, high throughput applications, consider RAID 10 or RAID 0+1 (both are essentially combining striping and mirroring in various means). These will give you the redundancy with the performance benefit of a stripe.

Don’t get into a shrink/grow. shrink/grow pattern. I have harped on this one. Set it to the size you think you need, manage it’s growth and monitor it. Tibor Karaszi has an excellent post about log shrinking and it’s effect on Virtual Log Files (VLFs).

Pick the right recovery model! As I discussed in Part 1, recovery models are important for log growth. It’s also important to pick the right model for your recovery needs (in fact much more important to pick for recovery needs.. (Remember we do backups for restores in fact it’s the restores that are more important). Let’s presume that you care about point in time recovery and you pick Full Recovery:

That’s great. Now get a strategy to PERFORM LOG BACKUPS. Being in full recovery mode without doing full backups is pointless. This tells me you aren’t performing restore drills (are you?), you are probably truncating and shrinking your log file when you run out of space and you potentially can’t meet SLAs or expectations. This will cause all of the above issues but it will also manifest itself in performance problems. So get with the program and do those log backups on a schedule that meets your required recoverability (one of the “itties” we should all care about) and manages growth/performance best.

These are some of the highlights that quickly come to mind that a DBA can have effect on. There are definitely more, check out Kimberly’s post above and look around her blog, great stuff to keep in mind.

What about:

Things Developers Can Be Mindful Of

KISS – Keep it short stupid – Alright, the stupid part was uncalled for but it worked. The key to remember when developing is if you are holding a transaction open, you are holding locks for the duration of your transaction.

What kind of locks you are holding depends on your operation (Select, DML, DDL, etc) and your transaction isolation level and any lock hints you may be using (Andrew Kelly has a great post about the dangers of NOLOCK or SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED). Remember that when you are in a transaction, whatever lock you had to hold to perform your operation is held until that transaction is committed.

If you perform an update within an explicit transaction (BEGIN TRANSACTION) and you are in the default isolation level of SQL Server (Read Committed) you will be holding some sort of exclusive lock even after your update statement is finished. SQL doesn’t know if you are going to rollback or commit your work so it has to lock either the rows, page or table (depending on the grain of lock chosen) until you tell SQL that you are done. This means that you can block other updates, other reads or other operations until you have committed/rolled back your transaction.

This is excellent behavior out of a DBMS. You want to hold a lock until you are done. Otherwise you might as well just throw data integrity out the window. It’s bad when you hold that poor transaction hostage longer than need be. So, the solution is simple:

• Keep your transactions as short as possible
• Properly handle errors and rollback when necessary
• COMMIT as soon as possible
• Don’t try to solve world peace and hunger within one transaction - I cringe when I see a single “BEGIN TRANSACTION” and then multiple page downs of operations, cursors, select statements, crazy logic, etc before you get to the commit or rollback. This means that all of those locks are held and you are probably causing blocking until you are done. Batch out work.
• Even with implicit transactions, don’t try to do too much work in one operation. If you are updating an entire table, and it is a huge table, try and batch it out, allow the work to commit, allow a checkpoint to fire and keep your log from growing.

You tell those Writers, we just want to read!

Sorry, you hold locks also. When you issue a select you are still holding a potentially blocking lock. This depends on isolation level and lock hints but in the default isolation level you are holding a lock. This is to prevent you from reading data that is about to be rolled back. Readers Can (and often do) block writers. If you are in a busy OLTP system and experience a lot of timeouts on update/insert operations look at your reporting queries, your select queries.

Optimize your code for performance. This holds true for updates/inserts and selects but watch your selects, if it takes you 10 seconds to bring back data to a screen, that is 10 seconds (per select) that you are potentially blocking a writer. Even if it takes 20ms but is executed quite frequently, that is a potentially substantial situation. Look at the performance of every piece of code you release! Query tuning doesn’t have to be difficult. One place to start is this post but a quick search will yield a lot of tips. Check out my blogroll, a lot of excellent advice on those blogs.

Think about performance at the beginning of a project rather than the end and your life will be easier in production. DBAs – make performance be an initiative at the beginning. Yes you are too busy for proactive work right now, make time for it anyway. When you pay later you always pay a lot more.

Again, with Part 1, I am hoping that a lot of this was review and simple. At the same time, if it helped anyone, great. There is plenty of advice out there and some careful thought and consideration to managing your transactions and .LDFs will go a long way.

Related Posts At StraightPath SQL

• Manage Your Transactions (So They Don’t Manage You – Part 1)
• Don’t Touch That Shrink Button!
• Shrinking Is A Popular Topic
  

This is part 3 in a series on shrinking and transaction management. Click here for part 4. You can see them all in the Shrinking & Transactions category.

If you are one of the people who have been reading this blog, you know where I stand on shrinking databases/log files. I don’t like the practice (Post 1, Post 2, SQLServerPedia article.)

That being said, let’s look at a couple of reasons why transaction logs can grow and what you can do about that in a proactive mode. We’ll break this down into a couple posts.

Through answering forum questions and some recent conversation at a Microsoft SQL Tuning class, it strikes me that there are some misconceptions about how transactions work, when one needs to commit/rollback and what the recovery models actually do to your database. Hopefully we can rake through some of that today.

What is a transaction?

This is probably a review, so let’s just hit the quick points:

• An atomic unit of work that either succeeds or fails together as one.
• Classic example is the credit/debit example (Note this doesn’t work with government budgets, think of your checking account)
  • You are online with your bank and initiate a transfer from your checking account to your loan account. This transaction includes a withdrawal and deposit. What happens if there is a failure between operations? Depends on if the work was done in a transaction:
  • In a Transaction – The withdrawal that happened is undone (rollback) and everything is fine.
  • No Transaction – The withdrawal completes (commit) and the deposit never happens. You are out $300 and have it sitting in limbo land someplace.

Implicit/Explicit/Autocommit?

There are three modes that SQL Server can operate in with regards to transactions. Implicit and Explicit transactions or in autocommit mode. Let’s explore those:

• Autocommit Mode – This is default behavior in SQL Server. This is also where a lot of people can get tripped up in their thinking (more on that in a minute…). Basically here, if you were to issue a SQL query that modifies data, once that statement completes (without error) the work you did is automatically committed.
• Explicit Transaction – This is also default behavior in SQL Server. If you want to declare a transaction yourself and specify when to commit or rollback a unit of work you use the BEGIN TRANSACTION, ROLLBACK TRANSACTION or COMMIT TRANSACTION commands. You declare your transaction and then through proper error handling manage when to rollback or  commit based on your business rules. Through this method you can have multiple statements inside the same transaction as opposed to what happens with autocommit.
• Implicit Transaction Mode – This is off by default but you can enable it within a session with this command: SET IMPLICIT_TRANSACTIONS ON. You can read more about it in books online (here). Basically anytime you issue a command that can modify data, SQL Server will implicitly issue a behind the scenes BEGIN TRANSACTION for you. Every statement issued from that same connection is part of that one transaction and you have to manually issue a rollback/commit command to complete that transaction. Forget to do it and now you have one big transaction which has it’s drawbacks (see below).

My Log is still growing! I didn’t do a BEGIN TRAN?!

So you are doing ETL, doing a large delete operation or otherwise modifying a lot of data at once. You are in simple recovery mode (transaction log is overwritten at certain periods, more on recovery models here) because you don’t need to restore to a point in time. Since you haven’t declared a transaction, why on earth is SQL Server’s transaction log growing out of control?

The answer is simple: because it cares. It doesn’t matter which recovery model you have chosen, SQL Server will always write to the transaction log. In order to maintain consistency in the database, before something is considered as having been completed it needs to be in the transaction log. It will eventually write to the physical disk but it has to be in the transaction log before your transaction is complete (see why it’s so important to put your transaction log on a separate drive and great IO??). If something were being written to the physical .mdf file but not complete when someone pulls the power plug from your server, we have a reasonable expectation of SQL to come back up in a consistent state. This is called recovery. Any transactions that are marked as committed in the log file are essentially rolled forward to the data files. Any that were not done but started are rolled back.

The recovery models just tell SQL how some operations get logged and when that log file is marked for reuse. In FULL recovery mode, it is not reusable until you do your log backup. In Simple recovery mode it is reusable once the data is written to the physical disk and no longer needed in the log, essentially.

The consistency and recovery means something to us when combined with the default mode that SQL operates in:

• Every statement is a transaction whether you tell SQL it is or isn’t. This is a great thing for us and our data quality.

So if you think about that statement, it makes sense that one big update or delete statement would cause your log file to grow “out of control” (it’s actually very much in control, just not your expected results ).

Try and think about different methods for handling what you are doing:

• Can you batch your work into smaller units? Issuing several updates,inserts or deletes dealing with a smaller amount of rows each time can help manage log growth. (The log will only be reused only to the point of the longest running transaction, if you have one big query it’s all one transaction).
• When deleting, are you deleting most of the rows? Maybe consider doing a truncate and recreate/repopulate of the table if the rows remaining after the delete will be less than those deleted.
• Consider changing recovery models – Going to Simple or Bulk Logged recovery mode won’t help you if it is all one being done in one operation. It will potentially help the reuse if you batch it out into manageable chunks. If you change your recovery mode, look it up on books online, you will most likely have to perform a full backup to keep your log chain intact and maintain restorability!
• Staying in Full Recovery mode? Perform frequent log backups. Again this won’t help you with the growth of your transaction log during one large operation but it does help with multiple batches.
• Look at using SSIS for ETL work, various options within SSIS can help manage this better and do your loads in a more efficient manner.

While I have your attention –

I mentioned above one good reason to have your transaction log separate. This falls into the “widespread news” category but it’s my experience it isn’t so widely followed. Separate your transaction logs onto a separate drive when you can! Log write performance is incredibly important. If you are going to do RAID at all on it, do RAID 10 or RAID 1 as performance is important (hence no RAID 5) and redundancy is important (hence no RAID 0- striping only).

While we are at it do you put your log backups (are you taking any???) on a separate drive also? If not, what happens if your log file drive crashes and you want to restore a log file backup? Oops.. Separate (Physically, not partitions on the same drive) your usage of drives. Yes budgets come into play but there are some basics you should try and follow where possible…

This is part 2 in a series on Shrinking “Best Practices” and logging. Click here for part 3. You can see them all in the Shrinking & Transactions category.

A while back I posted about not touching the shrink button. I am still finding it fun to see what search queries bring traffic to my blog so I look at the search query log every once in awhile. That shrink post is coming up the most in searches which leads me to believe I should add a few more points.

As I said in that post there are some great comments on shrinking over at Paul Randal’s blog. I suggest you definitely take a visit there, in fact if you had to stop reading and go to his blog, I wouldn’t be offended in the least.

Let’s talk about some of the more common questions that I see or infer from the web searches…

My Transaction Log Has Grown Too Much or My Transaction Log is Out of Space

If you have come here because your transaction log is growing out of control there is a strong chance that you are in Full Recovery mode and are not backing up your log file on a regular basis. Your transaction log continues to grow until you explicitly backup that log (a full backup won’t do). This is the expected behavior since Full Recovery mode means you want the ability to backup to a point in time and as long as your backup is someplace safe minimize your losses to the frequency of your backups.

Solutions:

1.) Setup a log backup schedule that meets your business needs. Search books online and understand recovery models, figure out the SLAs you are supposed to be supporting (do it before it’s too late since you are here reading this and Mr. Murphy likes to attack in these kinds of situations). Once you figure that out get your logs backing up on that schedule. Make sure they are going to a drive other than your mdf/ldf files so they are useful in the event of a failure. Perhaps even to tape directly or after a copy. You should see your log files become more manageable size-wise, hopefully.

2.) Get more space. Maybe you are doing Log backups but you still don’t have enough space. Either your activity is quite high or your allocated space is quite low. If it’s the former maybe a more frequent log backup helps. If it’s the latter or former, more space for your log files may be required.

3.) Switch to Simple Recovery Mode. This is not to be done lightly. You are no longer able to restore to a point in time but can only restore to the last full backup. Maybe that is fine by your SLA. If it is and you have no desire to restore to a point in time, switch to simple. Your log file will now truncate (see below) on certain intervals.

4.) While you are adding that space or setting up your backup look at your growth ratio? Is it the default 10% for a transaction log? How large is your log file? Is 10% really the right amount you want to see it growing by? On that same note has your log file grown a lot larger than it need be because of poor management? Perhaps once you do your first T-Log backup, you should look at setting a reasonable size knowing that it will be truncated on a regular basis. If that is considerably smaller than where you are, maybe one last shrink of the log file is warranted. (Don’t shrink the database! Just the file. Here sis a good reference on DBCC SHRINKFILE so you can see how to just shrink that one file).

What is the Difference Between Truncate and Shrink?

A lot of people get confused. They have truncated their log file yet they still have no free space and the file hasn’t reduced it’s footprint at all. This is because a truncation does nothing to the physical size of the allocated file on the OS. A shrink operation shrinks space from a file and a truncate essentially frees up the used space within that file.

This is why a shrink on a log file that is using all of the space won’t budge the size and why a truncate of a log file won’t budge the size. A truncate would have to happen first to make room available for the shrink to work. I do NOT suggest this, however. Another great post about why not to srink your log file is done by Tibor Karaszi here.

Someone on the newsgroups told me to…

A large portion of the folks responding to questions in the newsgroups and forums are giving great advice. They are taking their own time to help guide you down the right path. That being said, there are at times pieces of bad advice that hopefully aren’t followed in production. I talked about this practice of doing before trying in an earlier post. Gail Shaw also recently posted about this same topic on her blog.

Judging by some of the searches I have seen, it looks like people are getting some bad advice but at least trying to verify it through google (I would hope the next step is understanding the pieces involved and a test in dev/test/sandbox). A couple examples of that advice and my responses:

Stop SQL, or deatch the DB and delete the log file – No. This will most assuredly remove any transactions in your log, can leave your DB in a transactionally inconsistent state meaning a potential for loss of data or worse. If you are stuck without space for further growths, try the log backup, if you must do one last truncate and shrink after making a full backup and heeding the warning to get a real recovery strategy in place.

Setup a nightly job to issue BACKUP LOG WITH TRUNCATE_ONLY – I hopefully don’t need to say much more about this one that I didn’t say here or in my previous shrink post.

If any of this isn’t clear or you are stuck with an uncomfortable situation, send me an e-mail or leave a comment, recovery is important and I hate to see people messing this up because of bad advice or misunderstanding. I won’t charge you for any time for quick help with something like this.

This should be my last post on shrinking

This post is part 1 in a series on shrinking and transaction management. Part 2 is linked at the bottom or here. You can see them all in the Shrinking & Transactions category.

This topic has indeed been done to death. Yet I still often encounter unnecessary database autoshrinks, scheduled shrink jobs and at times a seeming lack of knowledge on just what one should be doing as far as their database sizes.

Update [1-31-09] I wrote a follow-up to this regarding some of the questions that I see Google Search bringing folks to this blog. Check out my my additional post for some of the more common questions here .

What Happens when you Shrink a Database

When you click that shrink button (or leave a DB in autoshrink, or schedule a job to perform shrinks), you are asking SQL Server to remove the unused space from your database’s files. Deallocate that space and let the O/S do what it needs with it. If you have a growing database (as the majority of non-static databases tend to be), this means that that database will grow again. Depending on your autogrowth settings (another pet peeve for another post) this growth will probably be more than necessary and you will end up shrinking again… At best this is just extra work (shrink grow/shrink grow) and the resulting file fragmentation is handled alright by your I/O subsystem. At worse this is causing file fragmentation, interrupting what would have otherwise been contiguous files and potentially causing I/O related performance problems.

What should we do then, leave free space in a file?!

Yes! In fact I find it best to plan for where your data size needs will be, not where they are at the time of initial go live. Set your initial database size to what you expect it to be in the future. If you can, look a year or more into the future. That’s your initial size. Create the database that size and set the autogrowth to a reasonable number in bytes rather than percent. (I hate the default of 10%.. if you have a 1TB DB that means you will be growing 100GB anytime you need to grow a file.. Better in 2005 with Instant File Initialization but still not “responsible”). Monitor your free space and look at size trending over time so you can plan for a large allocation of more space should your planning have been off. That free space in the file is “just sitting there doing nothing” as your SAN team may say, but would you rather have what you expect to need and grow into it or scramble to allocate space at the last minute? I vote for the former and if the budget allows (if you can justify it with the growth expectations, the budget better allow it because it will need to in a year anyway) it makes sense.

Yeah, but my transaction log is always running out of space!

Yet another pet peeve and really for another post but really quick: If you are in Full Recovery Mode on a database that means you intend to be able to recover to a point in time in the event of a failure. This means you plan on using a combination of Full Backups and Transaction Log Backups (and possibly differentials). SQL Server understands your intent, and it will not truncate (free up space within the file.. notice the file stays the same, I didn’t say shrink I said truncate.. Truncate frees space within a file, shrink removes that “free” space to make the physical file smaller) the log file(s) of your database (the .LDF files). Instead, they will continue to grow until such time as you take a transaction log backup. 90% of the time when I am helping someone with an out of control Transaction Log growth problem, it is because they are incurring the “cost” of Full Recovery mode (growing log file, the full logging of qualified events, etc.) but none of the benefit (being able to restore to a point in time from your transaction log backups if your log chain is unbroken and your log files survive)… Simple solution here.. Look at your backup/recovery plan!! Why aren’t you taking Log backups? Is it because you don’t understand them? There are plenty of resources to help. It is relatively simple to begin working on a proper backup/recovery strategy and the face you save may be your own Is it because you don’t need point in time recovery? Well then consider Simple Recovery mode which will truncate the log at certain events. Don’t go right to simple recovery mode though, analyze your situation and learn about recovery models and do what is right for your organization and business unit.

Yeah, that means you shouldn’t be doing backup log with truncate_only all the time…

If this is all new to you.. Check out a couple of resources about these topics:

• Paul Randal has plenty to say about Shrinking Files (and says it much better than I). Check out this blog entry and look at the rest of his blog.
• Recovery Models overview – Based on SQL Server 2005 but concepts apply to most versions of SQL Server

As always, feel free to hit me with any questions specific to your environment. I am happy to exchange quick e-mails with tips/tricks especially when it comes to my “pet peeve” topics.

Related Posts

• Shrinking is a Popular Topic
• Manage your Transactions So they Don’t Manage You (Part 1)
• SQLServer Pedia Article From this Blog and WebCast with Brent Ozar