Which would you rather do, wade through the nuances of SQL Server licensing or a typical tax form?  Some days, I think the latter would be a welcome respite. Let’s just put it this way, I bet Brent Ozar (BrentO on twitter) is happy that licensing wasn’t the biggest part of the MCM grade. (I don’t know this, I just presume it because people actually pass the MCM every once in a while).

Common SQL Server Licensing Myths

The people in this maze might be better off than us.

I am going to talk about some of the more common myths I’ve encountered with licensing throughout my career and consulting. Microsoft has some decent SQL Server licensing resources including some PDFs (some short and some nearing legislation size with flowcharts abound). This quick guide is a good start to look at. Which brings me to a disclaimer:

This blog post does not replace your own research and understanding of licensing guidelines from Microsoft.  The thoughts presented here are my own and represent my views of Microsoft licensing guidelines and my interpretations thereof. My perspective has the potential to be wrong (I’m a husband, so I’m likely wrong, come to think of it ) and could become outdated with future changes. Check with Microsoft and your licensing representative to make any final decisions.

On to the myths I encounter…

You Need Enterprise

Look at the edition feature matrix for your version (2008, 2008 R2, 2005) and see what you need and anticipate needing – it can save you around $15k + per CPU to go with Standard over Enterprise. I have commonly bumped into people believing that they need enterprise licenses when they don’t when it comes to things like:

• Log Shipping (Standard is fine)
• Clustering (You can have a single instance two node cluster on Standard)
• Mirroring (Some types are allowed in Standard)

In fact, this post was inspired recently when I was able to see a project at a company to roll out Enterprise Edition simply because of Log Shipping. No other enterprise features needed. This company had great pricing on Standard and I was able to save them almost $60,000 in Enterprise licenses they didn’t need. Wait. There’s more! Act now and you can save even more because another myth sounds like this:

You Need To License Your Mirrored Copy Also

In most situations (yours may be different, double check), a DR standby (For mirroring or log shipping) doesn’t need to be licensed by default. Now there are certain triggers (pardon me) that will hit you with a license cost (Being failed over to it for more than 30 days or reporting off of a snapshot of the mirror, for a couple examples) but by default, many simple options don’t. This same company had an Enterprise License purchase (single CPU) for the fail over site. Another $19,000 saved with no standard license even needed because that instance can sit there “unlicensed”.

I Can Put SSAS, SSRS and the DB Engine anywhere…

You can. And, as a best practice in busier environments I’d say you should. But you can’t do it for nothing. You can put them all on the same box and they will be licensed as though you just had one (under most circumstances). The moment you split them out (again, generally a best practice for performance) you must license each component separately. While I’ve saved some companies money, here I’ve cost some money… Sorry. I don’t like it either but it is what it is.

Multiple Instances Still Means Multiple Licenses

I hate typing this because just as soon as I do, I am sure some pointy headed boss someplace will say, let’s consolidate every instance onto one machine! Don’t do that. But this myth isn’t true – you pay by the CPU by the machine. If you have two instances on one machine each using 2 CPUs, you only need one 2 CPU license. Add another instance to it? Same thing, no new license, you pay by the machine not the instance. This is great and can really help save some license expenses at smaller shops but just do so carefully as I wrote when I first drafted this SQLServerPedia wiki article on multiple instance decision making.

SQL Server Licenses Are By Core, Like Oracle

Or at least like Oracle was last time I checked… SQL Server (today.. I don’t know of anything changing, but you never know) licenses by the socket. By the actual die that goes into the socket. Not by the core or thread. So you can have a Quad core proc with hyperthreading that looks like 8 CPUs in perfmon but you are paying for a single CPU license. This is good to know. Might make that hardware upgrade more worthwhile. Who knows how long this will last and, for all I know, maybe it is gone by the time you read this, so double check yourself.

Virtual Server Licensing Works Just Like Physical

I’m not going to say anything except to say this isn’t necessarily true. There are all sorts of scenarios and differences depending on if you are using standard or enterprise. The IRS has a hotline (800-829-1040). So does Microsoft, if you aren’t sure it may be worth a call or e-mail.

Developer Edition Works for All Developers

Well that may be true most of the time. Your developers could probably use developer edition but keep a few things in mind: 1.) Did you pay for it? Or is it just a download everyone shares? If you are covered by MSDN (For each developer who touches the instance) or some sort of enterprise agreement then you may be fine. If you paid for the product and a proper license you are fine for the intended use in the agreement. If not, you aren’t. But also – consider this: What is production going to be? If your production environment is standard, do you necessarily want the product developed and tested on developer edition, which is Enterprise Edition by features, Developer edition only be price tag and EULA?? There may be a reason (especially if you have MSDN licenses for each developer, a good investment typically) to make sure an app is developed and tested in something other than Enterprise. Consider the end goal and end edition. It would stink to see Dev and QA using Developer and relying on some enterprise feature that never failed a test only to find a deploy gone wild.

Licensing, Just Like SQL Server, Is Set-It-And-Forget-It

No. No to both. Depending on your agreement with Microsoft the day may come when they come to do a true up of your environment. They will discover your SQL instances with you and verify you are using what you purchased. If you haven’t you’ll be buying more licenses and you are in violation of laws and software agreements. You could face fines as a company or a individual, potentially. An answer? Do your own True Ups. Awhile ago I blogged about using the Microsoft Assessment and Planning Toolkit (MAP) to discover your SQL instances. The report it spits out shows you all sorts of great details (server name, edition, version, CPU count, etc). Do your own true up, no reason you can’t. Are you in line? If not fix it by adding licenses or consolidating and saving the company hard earned cash (“Hey, boss, I just want 1%…”)

Summing It Up

A simple look at a few of the more common myths I bump into around licensing. They can all be summed up with the same advice, do your homework, understand the license agreements with factual information (not someones recollection or interpretation) and always double check. Even if the information comes from a trusted vendor (you know how I feel about that), you could be breaking the law or throwing away company money. Neither are great career advancement moves.

What about you? Do you have a licensing horror story? A tip for someone else out here. Leave it in the comments or blog about it and let us know when your post goes up, I’ll be more than happy to link to any tips to help folks avoid making a mistake up front.

[This is the first in a 4 part series of posts about "Getting Organized" , on track and my use of Evernote.]

Getting Organized Series Outline

• Introduction and License Giveaway
• A Task’s Life
• The Death of a Task, Man
• 451 Degrees – Good For Notebooks

Do you struggle with organization? Note taking? Note Finding? I do and I could have found the fix for me. Leave a comment for the contest at the bottom and I’ll give you a one year premium account to the tool I’m using. Details are at the bottom.

Maybe this blog post is too early. “They” say it takes 3 months of doing something for it to become a habit. Well, I am writing after experiencing something new that is still new to me and being tweaked but it is really helping.

A mockup of a real notebook (even with the doodle)

The Problem Statement

“They” also say a picture is worth 1,000 words. In this case the picture probably represents hundreds of thousands of written words. The picture to the right is an example of what one of my (many) notebooks would look like. Chicken scratch writing, doodles from being on long calls and lost Action Items.

The Problem Statement here could be “I am wasting time writing notes I can’t find or use later, what should I do?”

I normally have a few notebooks in circulation at a time. Some for personal notes, some for blog ideas, some for client work. I have a general idea of where something is but sometimes I start a new one if it isn’t handy. I can’t understand the context sometimes. Maybe that was alright because my note taking used to be more just about writing it once to get it into memory and that worked most of the time. I still did a lot of mental gymnastics trying to recall what a note meant. I also lacked the discipline to go back through all the “Action Item” tags to bring them onto a list. I’ve used many systems (Outlook Tasks, Franklin Covey, Typed To Do Lists, etc.)

Another problem statement? I’m getting sick and tired of paper. I am finding myself hating paper’s clutter a little more each year. (In case my wife is reading – only a little more, that’s why I still have so much but I am getting to where you are, quickly)

Other problems: The notebooks aren’t searchable and aren’t always there; the kids like to cut and draw when they find paper; I look like a fool fumbling through the notebooks; Can you read my writing? I can’t always either; It’s a pain to link thoughts and track them over time… I could keep going.

The inspiration to get organized

An Answer?

No, it isn’t the Mac but the picture shows the simplicity of the solution. It doesn’t require a notebook at all. At work I show up with my Mac Book and phone. At a client I bring the home windows laptop (That over sized, noisy, ugly laptop ) and phone. I always have the windows mobile phone (does that make up for the mac?). If I ever don’t have a device, I probably can get to the internet somehow.

All of these tools allow me to use a new tool in my arsenal: Evernote. They describe their product:

Use Evernote to save your ideas, things you see, and things you like. Then find them all on any computer, phone or device you use. For free.

That’s it. That’s what it does. So far I have used the tool to manage notes, whiteboard images, business cards and it has been enjoyable, usable and makes me want to keep using it. Some tasks I’ve performed:

• Save notes from meetings
• Create Action Items with priority and category (Where are they from? Which “Mike” has to worry about them?)
• Track my spending and eating (why not, I need to be better at both)
• Remember ideas for blog posts (like this series you are reading)
• Do things at home that I normally would have “yes’d” and forgotten about (until gently reminded)
• Started to store links and captures from websites of useful information about a topic of interest
• Search for previous notes and quickly find them
• Upload images (White Boards, Business Cards) and search them with decent  (for OCR) results
• Solve world peace and middle east hunger (Were you paying attention?)

How Have I Done This?

With some tags, folders and typing in Evernote. I keep saying that word, I’m not getting paid to talk about their tool, I just really like it so far. I’ve used the features available in the Mac Client (which is lacking a few things, in my opinion… We’ll get to that in part 2 or 3) during the day. I’ve even downloaded and used the features on the phone, like taking an audio note when I was in need of duct tape and a knife in my truck (it’s not what you think.. I didn’t need a cooler). Evernote is definitely not a task management tool by default and I started using it for searchable notes in “one” place that I won’t clutter my house with. Using the note capabilities, It is working out alright for task management.

I’ll go into more details in the next post where I give you the lowdown on the techniques I am using and some more details.  The plan I have for the next posts:

Part 2 – My Evernote Technique - How I use the product, the tags/folders I use and method. Works for me, maybe can help you design your own. Also how I plan on using it to both make my life more search-able and reduce the paper clutter.

Part 3 – Evernote Wish List – Evernote just unveiled a neat “trunk” of apps and there are a few apps I’d still like to see created with a few features I’d love to see in the Mac client. So a small wishlist.

A Contest

Evernote is free and the free features are probably fine for most casual uses, even serious uses in the free version. With a premium account you can get rid of the advertisements, get some neat features (ability to search PDFs, quicker OCR on the Mac client, import more file types, import more data to name a handful) and feel good about supporting a well done piece of software.

So… I want you to have a free upgrade to premium. I will select a name (based on some method I choose, inspire me with your story, I guess) if two conditions have happened: the comment is made before midnight EST on Friday, 7/23 and I have at least 10 comments by that time. Just tell me about your current system in the comments. How do you keep track of this all? How could a tool like Evernote help you? What is your biggest frustration with time and task management? Would you hire someone who essentially starts out a blog post with “I need to be better at time management, note taking and organization?”

Subscribe to this blog’s feed to see the next posts and the contest winner when announced.

Psst. Hey Vendor – DBAs are secretly plotting against you! We hate what your products do to our environments. Sometimes we even work to get you replaced by someone else who makes a product in the same space but is “DBA-Approved”….

What’s that? You want the DBA stamp of approval?

I write this blog to help folks so I’ll pretend I’m not at work in a meeting railing on about how your product has no indexes or has way too permissive security or doesn’t think to recommend index maintenance. That’s right… I’ll help you. Why? Because I want you to succeed. I want to like working with you. I want you to be “DBA-Approved”

A History

You don’t have to look to hard to find blog posts of DBAs annoyed with vendor gaffes. There are some great vendors who really understand their destination database environment. They go out of their way to make sure their client properly backs up and maintains the environment. They performance test and look at best practices. There are a lot who don’t.

I’ve worked as a consultant for a vendor who wanted to do a better job in helping their customers handle the SQL Server environment they ship their app on. I’ve worked full time for a vendor who didn’t. I can tell you that my experience with the latter was early on in my career as a tech support engineer and I got a lot of painful calls from customers up a creek. Customers who maybe would have been better off with some prescriptive guidance…

But We Don’t Own Their Environment!

I know. But you didn’t tell the client that they really needed a DBA to look over their SQL Server environment did you? You might have even marketed it as “so easy! A few clicks and our installation is complete!”. I’m not saying you should own their environment. I’m not saying you should coddle each customer and play the role of DBA. I am saying that you should….

Care Enough To Do It Right

That’s right. I’m appealing to your heart. Though, if you do a great job it becomes marketing and word of mouth advertising. You’ll also be able to receive less “basic” support calls or deal with upset customers who got in trouble because of database issues. What about the lost business because of clients talking to other potential clients? Or IT staff commiserating with IT staff at user group meetings? Now I’m appealing to your bottom line, I hope?

Some Thoughts on Doing It Right

I’m not asking for much. This list is a good start. We might revisit this post in the future with more but starting with some basics, you’ll be on your way to being a “DBA-Approved” software vendor:

• Cheat – If you are looking to have your software sold and installed at a company that I work at, consult for or have worked at. Be prepared to answer these questions. Maybe all don’t apply, maybe some look like gibberish. Learn about the terms and understand why a DBA would ask. Send me an e-mail or leave a comment if you are confused about why I ask a question
• Learn and Know SQL Server – What kind of SQL Server expertise do you have on staff? You probably have some great developers but do you have someone with a DBA interest and skill set? Do you have someone who stays involved in the SQL Server community? Check out sites like SQLSkills, SQLBlog, SQL Server Central, SQLServerPedia and the SQLPASS organization.
  
• Speaking of PASS – PASS is the Professional Association for SQL Server users. It is a great organization with a lot of resources for us in the SQL Server space. They have a huge annual Summit with a lot of great technical content. Send someone to the conference and have them sit in the sessions. Have them talk to DBAs (we aren’t shy…)
• Documentation? Do you provide anything to your clients (especially those smaller shops that may not have a dedicated DBA, even if I think they should) around recommended best practices for database maintenance, backups, troubleshooting, etc.? You should.
• Best Practice Review – From some of the communities/blogs mentioned above you can find a lot of intelligent and experienced DBAs who can help you review your application from a database performance and best practices point of view. I would hazard to say that even just 12-24 hours of billable time can gain you a lot of customer good will. I know I’ve helped folks out with this in the past and as a DBA on the receiving end, it makes a difference working with a vendor who has Database best practices in mind.
• Security! Please. Don’t. Ask. For. Sysadmin (SA). Rights!!! Please don’t even ask for DBO, if you can avoid it. You should be using role based security with least privilege. It would be nice if you used Active Directory security and integrated with our AD but I’ll even let you use SQL authenticated if you stop asking for SA rights! Let me review your deploys and see the scripts that get run. Do your deploys with the least privilege necessary or heck, let me do it for you… I’m not being a jerk, I’m trying to do my job as a DBA.
• Backup/Restore – It goes with the database maintenance documentation above. I wanted to call it out here separately. Work with your clients on best practices for backup and recovery. Learn about them yourselves first. Don’t just install someplace and expect it all to be fine. Ask your support teams if they’ve dealt with a client who lost more data than they were willing. Ask them how that call went.
• Don’t Go With Defaults – There have been lots of blog posts about this that you should read to get an idea of what I mean -My thoughts, Paul Randal, Aaron Bertrand and plenty of others linked from those  – Back? Alright, learn about how to install SQL Server and include that in the documentation. Talk about recovery models so you don’t end up with huge transaction logs and bad advice being given to your customer’s IT support team from Google and forums.

Was That So Horrible?

I don’t think anything on there is going to make you freeze your code, hire a lot of developers or Project Managers. Sure if you don’t have the expertise on staff, you might engage a consultant and spend $5,000 or less on reviewing your docs and plans but how much will it cost you in lost opportunity to not do that?

Thanks for listening and I really hope you think about some of the points above or some of the points mentioned in my DBA Questions for Vendors list referenced earlier. I want your product to be a success and I want to see us DBAs start writing positive vendor rants (well.. Asking a DBA, a pessimist by trade, to be positive is a bit much. Maybe we can at least stop with the negative rants if you take some steps with us).

My offer stands – shoot me an e-mail or leave a comment below and I’ll help you out with a quick question as time allows, I might be busy consulting for a client of yours having headaches with their SQL Server environment

When I picture Paul Randal, I picture a guy who likes to run up to a bees nest at a crowded picnic and give it a good couple whacks. You know what I mean, it’s that laughing picture in his twitter profile. Or maybe it’s the devil’s advocate style he has. Maybe it’s his controversial facebook thoughts? Either way, he likes to call things out and make people think. That’s a great trait, all kidding aside. He started a blog meme asking what 5 “features” you’d like to see retired in SQL Server. His responses were great and he tagged some folks. I’ve looked at Brent Ozar’s response and was then tagged by Aaron Bertrand.

After reading those three responses so far, I have to say I am not going to be that original so most of these will be a hearty “Amen!” to earlier sentiments.

Auto Shrink

I made my entrance into SQL Blogging with a series of rants about shrinking databases. I still get hits to articles like “Don’t Touch That Shrink Button!” from web searches or forums. The simple fact of the matter is.. Well to put it simply, “raise your hand if you think regular shrinking of production databases is a good idea.” Okay. Look around. Any hands raised? Not many. So why, on earth, would Microsoft make that a stinking setting option?

I would like to see this retired, and doing it in a Service Pack would be alright with me. Or at least rename it to “I’m not dangerous enough by myself, please automatically add to performance frustrations and index fragmentation while keeping my database size tidy, even though it will grow again right after the shrink but then again I never liked right sizing databases that requires too much work – Yes/No” (Yeah, I know… Not the best grammar there, Jeremiah, but…) This was already suggested by Paul and Brent.

Full Recovery by Default (Or something…)

Again I’m stealing ideas (but I’m doing it to help people!) – Brent and Paul both also suggested this. This has been like this for a long time. It still trips countless people up each day. Another hand raising question – how many of you have experienced, helped someone with or laughed at someone experiencing uncontrolled transaction log growth caused by a lack of log backups in full recovery model? All of the hands that were down in the earlier question were just up. We’ve all seen it, hopefully we haven’t really laughed though. It happens All. The. Time.

A good tactical move would be to remove this as a default. Start folks out in Simple.  Or perhaps a strategic option would be a “Recovery Model Wizard” as a required step of installations? It would be nice to not need it but I think a lot of folks could benefit. As long as there was some way to override it with an advanced configuration, I wouldn’t mind a tutorial explaining pros/cons and costs of settings so people could make <gasp> informed decisions about their recovery models. Or maybe that is enabling companies to not hire experienced DBAs? I don’t know, what do you think?

No Simple Way to Truncate Backup History

Why make someone write a script or know to go look in a maintenance plan to find this option? Backup history isn’t critical to restores, sure it makes GUI based restores easier but how many GUI based restores are done going past 30 days? Why not make a default rolling window and let SQL cleanup that history. Provide a configuration option to change this window, even a configuration screen during setup asking other questions that should be asked but never get looked at by a lot of the SQL Server installations out there (backup history, recovery modes, heck even error log recycles)

Auditing – Enterprise Only?

Aaron talked about licensing models and I agree. One area is Auditing. Nowadays, even smaller companies with smaller budgets are being subject to increasing scrutiny and audit standards. In some ways, this can help a larger company that has a budget for Enterprise edition everything and an audit staff. What about the smaller company that is cash strapped? They can get most of the performance they need out of Standard and handle most questions/problems just fine, a $15k difference (or so) in per CPU licensing is a lot to ask for to better handle some of the auditing abilities in SQL Server.

Next, Next, Next, Finish

Aaron talked about the installation process in his response. He was talking about some of the bad defaults like I mentioned when talking about cleaning up backup history. This is perhaps more of a tongue-in-cheek response but it has some sort of a serious underpinning. I’ve always complained (raise your hand one last time if you’ve shared this complaint) that SQL Server is too easy to install. Accepting defaults and clicking next, next, finish gives folks a SQL Server instance that they put all of their most sensitive and important data. It also gives them:

• No backups
  

  Brings back memories of 386DX, "Boost Priority" is not a turbo button

• Full Recovery mode with no log backups
• No thought of DR
• No thought of right sizing a database
• Options that look tempting to select like “boost SQL priority”
• Data and Logs on the same drive (performance concern and recoverability concern)
• False confidence that everything is fine

That list could go on but the point is maybe SQL Server is a little too easy to install? I’ve heard it in a joking manner before like, “why do we need a SQL Server DBA? Any moron can install it!” I’ve seen the results of any old IT person (or business person) installing SQL. It inspired a series of posts on thinking about SQL installations.

I am going to take a cue from Brent and not tag anyone but welcome you to respond in your own post, leave a comment arguing with me (or agreeing) below or leave a comment on any of the blog posts that are sparked from the original post by Paul. I’d love to hear your thoughts, maybe I’ve gone too far? Maybe I missed something critical. I’d love to hear your thoughts. Also feel free to subscribe to the feed here for more posts like this.

Related Posts At StraightpathSQL.Com

• Why I think you focus too much on your backups – As discussed in the presentation, the focus should be on restores. May seem like semantics, but the focus on restore means you are thinking of more questions.
• How Do You Install SQL Server? – The first of a 3 part series where we discuss some tips for installing SQL Server and planning for a successful implementation.
• SQL Server Benchmarking Tips – I introduced the Performance Analysis For Logs tool in this presentation, this post talks about that tool.
• Find All SQL Server Instances On Your Network – I talk about using the Microsoft Assessment and Planning toolkit to locate all of your SQL Server instances. This post describes my discovery and use of this great (and free) tool.

Aaron Nelson is hosting this week’s T-SQL Tuesday. His topic, like mine from last week, has a lot of ways I could go. He basically said to talk about something that has to do with reporting. I decided to take the hodgepodge approach and talk about a couple things.

Disclaimer  – I am not syndicating this post because I am talking about a vendor’s product. I make no money from said vendor but enjoy working with their tools and have a complimentary license from them for use in consulting (mainly because when clients see the tool they want to buy it anyway). I also was invited to, and participated in, their first ever Product Advisory Board meeting, got some free food out of the deal (but not enough to leave my integrity at home). At the end of the day I like their product and get nothing if you select them and nothing less if you don’t but I think it is at least worth a side by side test drive with other such products.

SQL Sentry 5.5 And Letting Reports Save You

I have known about SQL Sentry for a long time, since hearing about them from a former manager of mine who happens to be a SQL MVP (regular readers already know who I am referring to). I have used their products at a couple of places I’ve worked and have enjoyed the experience. One thing I thought was lacking on the performance side was some underdeveloped reporting functionality but that has just changed. I downloaded their new version 5.5 when it was in Beta and I have loved their reports, even saved my bacon a couple of times.

Well, SQL Sentry 5.5 is in general release now and you can download a trial here. You can read about 5.5′s features in a blog post by Greg Gonzalez, their President and CEO.  I am not going to send screen shots of my environment’s reports because I don’t have a dummy environment setup and it would require too much Photoshop cleaning server names/etc but I’ll steal some of the screen shots Greg used in his post. I’ll also describe the importance of these reports in my environment.

While I think SQL Sentry is a great tool and would recommend it to anyone, I don’t really care what tool you use as long as you track history and can report off of it. It can save your butt, show negative trends and make life easier for you. There are lots of tools that can do this for you. You can buy a 3rd party product to handle monitoring, alerting and the reporting. You can play with the Management Data Warehouse and Data Collectors that Microsoft provides with SQL Server 2008 also.

Anyway… What are some questions that I have recently used these reports to answer?

Our New SAP GL system has been growing, we think. Can you tell us how much the SQL Server database grew since last Friday? So before I had SQL Sentry 5.5 I could have prepared for this question in advance by setting up jobs to track the data, configuring a SCOM SQL management pack to collect that data, etc. I didn’t have those things setup and the question caught me off guard, to be honest. I could go back and look at backup sizes and get a decent idea but I had SQL Sentry Eevent Manager/Performance Advisor running and had just recently installed 5.5 at the time. I clicked the Reports drop down knowing new reports were added when I upgraded to the beta version and saw a menu that looked a bit like this:

Some of the new Performance Reports

Nice! I saw one for SQL Server File Utilization – By Database… What the heck, I gave it a shot and sure enough. I selected the database I wanted, the date range I wanted (Friday to the Tuesday they were asking on) and poof! A pretty report showing the growth of used space in each data file in the SAP ECC database and a great graphical representation of how much free space remained. I was able to quickly tell them that each file had grown by about 12GB in that 5 day process. Graphical representation helped me explain the free space/used space concepts, the beginning/end text data helped me do a quick calculation. (The report had a percent change but they wanted the GB changed so I still had to do a little simple math. Maybe an enhancement request to put the MB changed also? Saves me from having to type calc ;-0 )

Database Growth Over Time

“The database server wasn’t running right last night!”

Another use of another report while still in beta mode. Came in to work and folks were complaining about performance the night before in the database server. Application team indicated it was probably a database issue and we should consider ways to improve that. It was an area we have had discussion in the past and there had been some DB issues but I got no alerts from SQL Sentry the night before, no alerts from SCOM and when talking it didn’t make sense that there was any DB issues.

Well before these reports, I could gather the data from a few sources. I could look at SCOM as we are tracking several performance counters, I could look at SQL Sentry’s Performance Advisor tool for that time frame and see the disk activity, TOP SQL commands issued, memory/CPU graphs, etc.

I decided to play with the Reporting menu again and discovered the Windows & SQL Server Performance report. Figured that might be a good way to quickly rule in or out DB server issues for further investigation. Ran the report and pretty much saw a quiet server that had a few spikes that were not correlated. Nice! I exported this report (like I did with the DB file growth report) to .pdf and sent it off with some thoughts. Turns out there were no issues at all (in any portion), just a bit of overreacting..

What’s the Moral?

As fun as it is when first starting out to build your own reports and monitoring processes, there is something nice about having everything you need a few clicks away. Consider looking at collecting your performance data in some form with some tool and have some reports ready to use to save you time in the rabbit hole later.

Mini Rant

I said it was a hodgepodge. This one is about SQL Server Reporting Services (full disclosure: I am in a SSRS 2005 environment, but I don’t believe this was changed for 2008). Now maybe it is only the environment I am in with some reports that need “help” and tuning work (some of which is in progress as time allows the development teams). Why can’t SQL Server write a permanent entry every time a report starts?! Sure there is some data in the Reporting Services TEMPDB but it is difficult, if not impossible, to make usable sense out of the data. Yes, if the report aborts or finishes a record is written to the Execution Log.

That being said, we occasionally have a report execution of one or a couple of our 300-450 reports that just brings the well provisioned SSRS box to its knees. We have no choice but to resort to service restart/iis reset/etc. Mixture of cube and Stored Procedure reports and it could be either. When trying to figure out what is causing the issue and seeing if we can just isolate the responsible dashboard, queries, etc. we can’t see anything. Why? Because there is no execution log entry until the report is done (with success or not).

I should (and will now that I am thinking of it) open a Connect item to suggest that possibly an Execution Log entry with a status of “in progress” be written when a report is FIRST requested. Then when it completes the status can change to completed,aborted,etc.

Is that so wrong? Do you bump into similar issues ever?

We have a plan to get moving to 2008 and that may even reduce the need for this functionality as SQL Server 2008 renders reports much more efficiently than 2005 and we likely won’t have these restart requiring incidents, hopefully.

Yes, he was impeached, by the house and acquitted by the Senate. Not the point of my post today, forget the politics. He wasn’t impeached for all of that business with Monica Lewinski. Was it unethical? Yes. Was it immoral? Well to me it is. Was that activity an impeachable offense? No. Why was he in legal trouble then? Simple: he lied. He, according to the conviction in the House, perjured himself about his affair while under oath. You may also remember his infamous press conference where he did the same thing to the public.

What’s Your Point? This is a SQL Server Blog…

Well my point is simple, often the cover-up or lie is worse (or has worse effects at least) than the act being lied about.

That’s where it relates to my chosen career field. That is where we, as DBAs, Sysadmins, Developers, etc., have to make sure we pay attention to the many lessons we see in public oopses.

Ever Have This Feeling?

One time, I was doing some work on a critical production cluster. I was adding drives and modifying an existing drive. Took the drive offline (shh if you already know how this ends). Took the disk offline… And then… I had that feeling. Please tell me you know the feeling also – my face felt like it was on fire. My fight or flight system was in overdrive as I had an adrenaline dump. Hands got a bit jittery, awareness increased while at the same time a narrow focus started. My stomach? Oh man. That horrible feeling of your stomach being twisted and turned upside down like an engine that just won’t turn over. Yup… I realized what I did when I saw the SQL cluster begin a dance of fail-overs. You bonehead! That drive was a dependency of the SQL Server group.

Yup. I brought the production cluster to a standstill for about 10 minutes while I hustled to repair the damage done. So while fixing that issue I had two choices of how I would proceed:

1. Listen to the voice in my head that says (somehow italics look more sneaky), “quickly fix it, figure out how to cover the tracks and draft an e-mail that starts with something like, ‘a mysterious issue…’ “
2. Listen to the voice in my head that says, “quickly fix it, ping someone in charge and give them a heads up and let them know what you did. Prepare to write a note explaining the event.”

I chose Option 2. I’m not saying I’m better than you if you went with Option 1. I’m not saying I’m perfect. I’m just saying that Option 1 doesn’t work. Early on in my career I may have gone with something between the two options. Even that doesn’t work because it leaves untied ends. The best manager I ever worked for (I blogged about him when I answered the SQL Quiz on Leadership) was really big on this. His philosophy on this more or less was, “Own it”. If you don’t own it on his team you have a problem, probably lost some respect and trust. If you do own it and the mistake or issue wasn’t something really horrible, you’ll end up alright in the long run.

How Do You “Own It”?

It’s simple. You admit you messed up, fix the issue or enlist help in fixing the issue, make a plan to prevent this from happening in the future and move on. You’ll be fine. We are all human, we all mess up and most of us still have our jobs. Heck, even Bill Clinton after his not owning it commands a lot of money to speak.

A Pattern

Now one of my flaws that I am still working on after 10 years in the field is keeping e-mails short and to the point. So take this advice with that caveat in mind. A good quick template of an e-mail I could have sent in that issue above:

Earlier today the cluster service became unavailable while I was doing maintenance on it. This e-mail outlines what happened.

Summary

I was tasked with removing a disk that was no longer being used. As this disk was removed, I realized (too late) that it was a dependency on the SQL Server group in the cluster. This means that the SQL Server failed back and forth between the nodes until it rested in a failed state. I was able to resolve this by breaking the dependency on the unused disk and bring the cluster back online.

How Was This Missed?

I should have checked the dependencies before touching the disk. I did not use a checklist to check for this and I missed this important detail.

What will be done to prevent it?

I have created a document for working with cluster resources like disks. This document links back to Microsoft checklists and includes a warning about resource dependencies with a process to check this. I have also sent a note to the server teams and DBA teams outlining what happened.

That isn’t exactly what I would write but the bolded points are the “pattern”. Describe what happened, why it happened and how it won’t happen again… And then move on. Deal with the repercussions and feel confident knowing that you were honest and up front and you mitigated your own issue. And remember…

You Learned a Good Lesson

So maybe this isn’t one to include in the e-mail or in a mea culpa discussion with your manager (though you should have that mea culpa discussion). Keep this tip for yourself – whatever you did wrong… whatever caused that stomach roller coaster just taught you a lesson you won’t soon forget. Like I said, unless you really, totally ruined the day for your company, you’ll still have a job. You’ll still get respect from your colleagues because you quickly came clean and even talked about how to prevent it. You learned something, though. So eventually you may even have a net positive out of the issue. Just remember to learn from that lesson and don’t let pride get in the way.

Bring Your Own Negative Example

So, I didn’t talk about some negative examples. I can instantly think of the times where I have been involved with projects and teams that had people who listened to voice number 1. I won’t. I don’t need to. You can think of them already, right? How did you feel as the coworker (or victim) of the person who reached for the broom and the carpet corner? Don’t be that person…

Share your story below in the comments. I want to hear about how you cope with a mistake you make on the job. I want to hear about a counter example that you’ve bumped into and how it made you feel.

No, this is not an installation guide

There is documentation for that found in many places. First good place to look is Books Online (2005/2008) but there are plenty of other community resources found at sites like SQL Server Central and SQLServerPedia as well as blogs.

What I mean by this post is to ask, “How do you install SQL Server?” Do you follow any best practices? Do you use a repeatable process and a checklist to the process? Do you review post installation to verify you did everything right? Do you know what is actually needed for the instance you are building?

If you do, then this post might be a review and not worth reading. If you do things kind of quickly, or give every instance every service then maybe some of these tips will help you out. More to follow in Part 2.

Some SQL Server Installation Tips

I used to catch a lot of grief (in good humor) from a Sharepoint developer (pot meet kettle ) about how my job as Database Administrator was a series of left and right clicks. While I enjoy scripting things out, the GUI is there and it is helpful so sure, I do some clicking. One area in particular I caught flack was for the install process (described, almost accurately, as “Next, Next, Next, Finish”).

I have worked in a lot of environments where that was the extent of the SQL Server installation – some quick clicks and poof you are live with defaults and some form of SQL Server services running ready for “prime time”. Giving a little more thought to the installation and setup may save you some invoiced consulting hours in the future.

I recently spoke about some aspects of these issues when sharing on the Quest Connect webcast for Top 10 SQL Server Administration mistakes (I blogged about that here but it is now unavailable – it was a 24 hour virtual conference). I want to zero in on some of the installation/initial setup related items now…

Read the Documentation

Ok so this one sounds like a freebie. You’d be surprised how many people don’t, though. It’s apparent in some questions online and it’s apparent in some basics that are misunderstood when choosing options. If you don’t know or you aren’t 100% sure, look it up. Most of the installation steps have a “help” dialog and Books Online really does go into a lot of detail.

Plan

Figure out what type of usage you should expect. Plan your I/O infrastructure. Some basics rules of thumb (I hate rules of thumb as a rule of thumb… Your mileage will vary but some basic thought processes to go through follow. Part 2 will have some resource links, check these out and look at Books Online thoroughly):

• Memory – the more the merrier. Now you could go overboard and waste money but on a busy database server with a fair amount of data memory is key. The less memory you have, the more frequently you are reading from disk and flushing data to disk.
• Disks – Fast is better. More spindles (physical disk drives) is better. Separation (at least Data, TempDB, Log files and backups each onto their own physical drives is generally a good start) onto the RAID arrays is even better. RAID 5 (Striping with Parity) is generally alright for most data file setups. RAID 10 makes your log and tempdb files happier (even just going with mirroring alone for Log files is fine in a lot of environments also)
• CPU – Multi cores are generally fine. In my mind the verdict on the new hyperthreading on top of the quad cores is still being decided. It used to be that hyperthreading (prior to the dual cores) was bad but it has been souped up and reintroduced. Most SQL Server environments I encounter are disk or memory bound before they are CPU bound but don’t skimp here either for a busy environment.
• 64 or 32 bit? 64 will give you better memory management, it is trusted and well used in the “wild”. Some annoyances with SSIS packages but nothing impossible to work with. Make sure your hardware and O/S and SQL installation are all friendly with one another.

The point here is to understand what makes each of these areas tick (you may grow tired of me saying this but you can find what you need from documentation out there), understand what the needs are for your environment and get a plan for what is needed before you even think about looking for the install media. Heck before you even think about putting an order in for the server.

Don’t just plan the hardware here but actually take a look through the below items, the items in Part 2 and incorporate decisions needed down the line here in the planning stage. I wrote a checklist for questions to ask vendors (Blogged about it here). An installation/pre-installation checklist is also a good idea. I know Microsoft has some for cluster installations…

Install What You Need

What was requested? Is what was requested understood properly by the requester? Depending on the version of SQL, installing Reporting Services (in 2005) means you need to install IIS. Do you want that running on your busy production database instance if it wasn’t needed? Why install Analysis Services or Free text Search if you don’t have a requirement for it? Wastes a little space, uses a little memory and it’s one more item to increase the surface area of your installation for security concerns. (SQL Server has had very few exploits in 2005 and beyond compared to many competitors but security should be a concern of a DBA and why increase even a slight chance of risk if you don’t need the tool?)

Do you want all of the client tools on the server? Maybe SSMS but do you need BI Development studio up there?

(Install What You Need) Where You Need It

So you do need Reporting Services and Analysis Services. How busy will they be? Have you done some planning to understand the capacity, usage and load factors to know what should go where? Analysis Services has a different IO pattern than SQL Server. When you need to load your cubes each night your data source is on the same box if you collocated the tools.

For Reporting Services have you considered putting the front end on an “app” or web server? Memory and CPU are generally more important (there are exceptions and report caching/subscriptions can complicate that blanket statement… See Documentation above )  and the usage of memory and CPU is different and competitive with the Database Engine. Perhaps your reports will be so busy that having the Front End on one box, the data source on one box and the database server for the two databases that get loaded when you install and configure Reporting Services (By The Way – Reporting Services has it’s own databases and they are crucial to reporting performance)

Installation Directories

Don’t just accept the defaults without thinking through them. Do you have a directory setup for your data files? Do you like seeing your application binaries in a path other than C:\Program Files\… ?

In fact while talking about directories, do you have standards for folder naming conventions? For me, I typically like to make a “SQLData” and “SQLLogs” folder, sometimes I incorporate an instance name when going with multiple instances (I wrote a wiki article for SQLServerPedia on some factors in the decision making process there). So if you do have one, implement it before you begin the installation. If you don’t have a standard, maybe now is the time to think about one? There is no wrong answer for a standard (other than not having one or, even worse, having one and not following it consistently).

Enough for Part 1

In Part 2, we’ll talk about Security Considerations, High Availability/Recoverability Considerations and a process for testing your installation. We’ll close with Part 3 at the end of November when we discuss post installation configuration options, maintenance and defaults.

I welcome comments. If I my comments don’t backup your experiences, you have a different take or more to add, please feel free to drop a line below.

There is another quiz going around the SQL Server blogosphere. Great question again, “How did you get started with SQL Server?” I was tagged by Denis Gobo (Blog, Twitter) when he answered this question here. You can hit his blog to track the lineage of this one.

I have been working with SQL Server for just over 10 years now. I guess to paint the start we should go back a little bit to my “pre-techie” life… Maybe I’m going back too far on a blog talking about my professional skills but hey it’s my history and it helepd shape me.

Pre SQL Mike

I worked throughout High School at a hotel in a small Maine tourist town. I worked in a lot of roles (started as a dishwasher and eventually moved to the Front Desk) and assumed the unofficial computer geek role. Flashback a ways and I was always a bit of a computer geek, from my first TRS-80, to the C-64 (blogged about them here for a different quiz). At the hotel, I impressed myself with my notepad HTML abilities as I worked on a project to develop a self service concierge system that was supposed to give area information to guests (Internet wasn’t as widespread, still mostly dialup and no kiosks/etc. in the area… Spent a lot of time answering the same questions and I wanted to put more in the guests hands). Due to various reasons this never made it to the other side of the desk but it was used a little bit behind the desk for some information. Really simple, really ugly.

“Database”  - I didn’t use a database for that information system, data points were hard coded into the notepad html code. At that point in my mind a database was defined as a flat file to store my music collection or some address information. A spreadsheet really (funny, today one of my pet peeves is folks talking about their “excel databases” they want my help with spread throughout the enterprise).

Due to many reasons, I focused more on work and other things and school work trailed off. I lost interest and eventually stopped going to High School. After 6 years at the hotel, I was comfortable and “content”. Met a great gal at work, was really impressed with her but she wanted someone with their life more together: enter motivation. Quite a few things changed in my life through meeting her, her family and the God her family worshipped. Two of those were: Going back to get my diploma and looking for a job that utilized a talent I had developed on my own and taught myself: “computer stuff”. (I married that girl, by the way..)

Contract to Perm Tech Support

Around ’98 I signed up for an e-mail distribution for jobs in the state I was living in at the time (NH) and saw a posting that looked interesting… A software company needed Tech Support help, talked about helping with printing problems (their primary business was check printing software that tied into large ERPs…), computer issues, etc… “I can do this”, crossed my mind so I made my first resume and sent it in. To my surprise they wanted an interview and the interview was largely about troubleshooting skills. Something that is important in every role I have ever held (both with and without computers). One of my first blog posts was about a troubleshooting methodology and I wrote a SQL Server Central article about the topic also.

I got the job! meet Relational Databases

I got the job. First order of business was two weeks of training. Well the first week was about the software, how it worked and what went wrong with it, etc. It was a training class for all new hires. Well the second week really piqued my interest. They started talking about the database behind the application. Talking about the tables that the application used perform various functions in the application. Talking about the configuration tables, the  data for the application, etc.

What? Tables behind the application???

I… I thought a database was just used to store information like what CDs I owned and which ones I liked. Maybe store account information. What the heck are these tables that control an application they are talking about?! I was intrigued and had to learn more. I started digging deeper and learning about relational databases, I got more intrigued. I am a math geek deep inside, I messed up by not applying myself in mathematics in school but this relational theory stuff looked interesting. I really liked the ability to change configuration information without having to rebuild/deploy code. I liked the order of the relational model: no matter how crazy the data points may be, no matter what kind of business logic may exist, the database brought some order (well, most databases do… remember I was still an idealist as I was learning at this phase of the career ) to the chaos.

On The Job

I was still high on this database knowledge. The Application at that time was on Oracle 7 something, SQL Server 6.5 and SQL Server 7.0… I took some computer based training for the Oracle stuff and really started getting into this Structured Query Language stuff. I had an opportunity to take some SQL Server 7.0 development and administration training, I jumped at it. I really picked up all I could, installed SQL at home and started playing with it a lot. In the support role my focus was more on Database Administration (a lot of our SQL Server shops didn’t have DBAs) and helping some bad default setups and SQL Server issues.

I just kept learning, trying and growing and instead of it getting old hat or boring there was always something new. This was exciting and fun to learn. After about 6 months on the support team, I became a go-to guy for database issues. We had an advanced support team for issues we couldn’t take and I hung out with those couple of guys and talked database talk with them. I would try and head off other resources database issues before we had to “bother” them and that was a huge growth.

An amazing opportunity

I had a chance to work for Andrew Kelly at another company. I didn’t know who he was, SQL Server 2000 was still very new and I believe Andy was either a new MVP or about to be one. I remember he was always on those newsgroup things (that’s why he needed a Junior DBA, most likely ). For some reason, he decided to hire this kid with not much experience but a passion for SQL Server. Working for Andy consisted of getting into “large” (They were at the time on SQL 7.0 or the early releases of SQL Server 2000) databases, rolling out maintenance tasks, monitoring performance, learning the vital importance of backup strategies (and testing them), hardware and performance issues (with some basic DTS packages thrown in for good measure.)

The best part about working for Andy was some amazing 1:1 training from someone with such knowledge. He created some classes for me about database administration, new features of 2000 and he patiently explained so many concepts to me that stick with me to today.

He also more or less made me get Kalen’s “Inside SQL Server 2000″ book when it was pretty darn new. That book and I became great friends. I lost my original copy sadly but it was WELL WORN. Binding falling apart, scribble from notes I had to write while on a trip with my (at the time) girlfriend and that was all I had to write on during the train ride to DC (or NYC? I forget). I read and re-read that book. Nah it wasn’t all consuming but it really helped me in my journey to learn more about SQL and I always learned something new (even recently I had to re-read a section for a custom course I delivered for a SQL 2000 site that, for various reasons, can’t upgrade.. Sure enough, I picked up new information, nearly 10 years after the book was published).

Liftoff

That first role that sparked the interest in SQL and the second role that nurtured that growth set me on the path I am on today. I was laid off from that company with Andy, and have worked at various roles since then. Always relying on my DBA and Performance pet peeves/knowledge but getting to wear different hats (some all at the same time): Developer, “Architect”, SQL Server Consultant, SQL Server Blogger who has himself fooled he has a lot of new stuff to offer the blogosphere, unofficial project manager, Performance team lead, etc. etc.

So it’s been over 10 years and I am still very much not bored of SQL Server. Still thoroughly enjoy working with it in all of the above capacities and I still love learning new information and getting to work with new features.

Will I be doing SQL stuff in 10 years? Who knows… Maybe I’ll be supporting legacy non-cloud SQL Server installations for the same kind of money COBOL folks charge today Maybe I’ll be focused on the data within whatever the latest and greatest data storage mechanism is then. Maybe I’ll be a Pastor

I am very happy that I made the discovery of SQL Server and relational databases the way I did. It has been a fun career so far and while every day in the office may not be a fun day, it has overall been a great time.

I am not going to tag anyone since I am a bit late to the game… How about you though? Blog a post and link back here or ping me on twitter. Share your story in the comments.

All of our SQL Server instances need benchmarks!

So the title is somewhat of a trick question… I hear and sense a lack of priority around taking periodic benchmarks in a system with a lot of folks. How often do you take benchmarks? Do you know what “normal” means for your system? Do you know when you may have a new problem before the users complain?

I have started working with a new (to me) tool in the past couple months that has made life a lot easier with benchmarking and it answers some excuses you might be thinking of why you don’t take benchmarks:

• My company won’t invest in a tool that costs money
• It takes too much time
• I don’t really know what to look for
• I don’t know what all the counters mean or what the values should be
• I don’t have the time to make graphs, compare data and make sense of it all.

In walks a PAL –> Performance Analysis of Logs (Literally, PAL). This is a free tool you can download from codeplex.

To run this tool you just need to simply grab the required counters (and most of the time it is at the object level so you can easily create a perfmon log file), run the log file and then run the log through the PAL tool which uses Log Parser (another codeplex download linked from the page above). You tell the PAL tool what threshold file you want to utilize (you can select from several predefined files including one for SQL Server 2000 and one for SQL Server 2005 which works for 2008). These threshold files describe the counters and help determine if there is a potential problem and apply colors (green/yellow/red) to indicate a potential issue. Each section also has great graphs to give a quick view of a particular counter.

This tool has a huge benefit to benchmarking. You just run your counter log (perfmon counter logs are normally pretty low impact to your system with their WMI calls) and parse it through PAL. Poof you have an HTML page with pretty graphs. Do this on a periodic basis and you have the start of a benchmarking tool. Troubleshooting some performance problems? Run this during a day of load and you can help zero in on some potential issues or directions to head down.

How difficult is it to use? Not very, let’ take a look…

First Setup a perfmon log: (Start/Run/Perfmon <enter>) Expand “Performance Logs and Alerts” right click and select “New Log Settings” Give it a name, maybe SQL Benchmark Trace?

Next add Counters & Objects (to find out which ones you will want, install the PAL tool and look at the template file for SQL Server, you will see what counters to capture and what objects they come from). I will capture the following but you can look at tweaking down a bit and not all are required depending on what you are looking for. I get these from the SQL Server 2005 threshold file editing in the PAL tool and a simplification of creating the log:

Here are the objects to capture, tweak down as you see fit to make the file smaller and help lessen any potential monitoring overhead.

Physical Disk

Logical Disk

Process (I will sometimes tweak this down to just the SQL Server process)

Memory

Network Interface (sometimes I will tweak down to just the counters needed and just the NICs involved)

Paging File

Processor

System

SQL Server: Access Methods

SQL Server: SQL Statistics

SQL Server: Buffer Manager

SQL Server: General Statistics

SQL Server: Latches (Often I just look at the two required counters rather than the object)

SQL Server: Locks

SQL Server: Memory Manager

Next Setup a schedule/file location and thresholds for max file/rollover options/etc. Typically I put the files to the C:\ drive or if I have a “utility” drive that is not used for anything critical to operations (no paging file, no SQL Server database files, etc). I also will set a max file size to be sure that no issues arise from a drive running out of space. These options are mostly self explanatory on the Log Files and Schedule tabs. I always log to a binary file (and for PAL you need to do that, I believe) the SQL option is out because I don’t want to incur cost on the SQL instance.

Next wait for the log file to run for your specified threshold, bring the file to the machine (I normally just use my laptop or PC) that you are running PAL on. Launch the PAL tool:

Performance Analysis For Logs First Screen

Click Next and browse to your log file. Restrict to a time period if you want to look at a subset of the data in the file.

Click Next and select your threshold file (I selected SQL Server 2005 in the screenshot below but the file output shown is an anonymous file taken from SQL Server 2000)

On this page you will also be asked a few questions to answer at the bottom. Number of processors (cores), ThreeGBSwitch (enabled or not), 64 bit or 32 bit?, Total Memory on server, KernelDumpOnCdrive flag (there is an alert to indicate a potential flaw of not enough free space for a dump):

What Type of Analysis?

In the the Analysis Interval tab, I typically accept the default (Automatic). Basically the PAL tool will break your file up into slices of time (it takes 30 slices for the period shown). You can change this if you want to have a bit more control over the way the averages and maximums (spikes) and minimums are reported. I typically leave this alone and won’t waste your bandwidth with a screenshot.

The output options tab just asks where you want the file to be saved and if you want the output to an HTML output (default), an XML output or both. I typically work with the HTML but I bet someone could create an import process for the XML to setup some code to analyze baseline data overtime and flag an alert if something gets worse. Again, I’ll spare you a screenshot.

Queue – This tab gives you the ability to store the output from the options selected. It shows you the switches and options for the .VBS command to be executed.

The Execute tab allows you to either immediately execute the options you just selected (what I have chosen every time I use this tool so far) or to simply add the script you just created to a queue so you can add another analysis to the same queue and process multiple at once. Also an option to execute as a low priority. The execution can take quite some time depending on the size of your log file and amount of time in the file.

Once you click execute a command window opens showing you the progress. Ultimately an HTML output file is displayed. This shows graphs of each of the counters looked at for the time period:

Below is a graph from the sample log file provided with the PAL installation:

What We Were Waiting For - Pretty Graphs

Similar looking graphs are shown for each counter specified. You can look at just the graphs as you scroll through to find correlations with each other (for instance did your batch requests/sec jump up just as your disk latency shot up? Did some other process cause memory pressure while your free pages dumped, etc.)

Alerts are also shown when instances of counters go outside of the threshold formula (some thresholds are simple: disk latency for instance. If the avg. latency is over the Microsoft recommended value a yellow will be shown with an alert fired, if it goes above the next number 25ms, red is shown.. If there are high spikes they will also trigger an alert). You can look at these alerts and see if there is a trend (did you have a spike in disk latency during a backup operation or bulk load but the rest of the time yo uare fine? or did you see a mostly acceptable average with a consistent spike in read latency, maybe you need to dig deeper… Did you see free pages going down consistently and full scans up consistently in correlation with these disk latencies? Now you can start looking at potential code/database issues… Maybe you will want to run some traces and analyze them with a tool like ClearTrace, SQL Nexus, SQL Diag, etc… Maybe you want to look at some of the query execution DMVs).

Here are some spikes shown from the sample log file:

Manager Friendly View - I/O Spikes

The other benefits of this tool? It has descriptive information discussing the counter you are looking at that can help you better understand the counter and acceptable limits:

Perfmon Counter Descriptions

The information is all arranged on a simple HTML report with hyperlinks to sections as well as a summary list of counters by time period so you can either look at likely suspects you are looking into or dig into a time period that you care about.

Adding simple tools like this to your repertoire can make your life easier, help you present information to management easier and help you stay on track of your environment. Don’t end up with the users being the main source of information on how your system is performing.

And for just a little bit more money than free you can get some great tools from a number of vendors that provide even more information and most provide pretty graphs and reports that help you keep your environment in check. So you can spend a little time and use a tool like PAL with the DMVs and a little ingenuity or a little time and a little money and buy some tools.

I don’t care how you do it, just do it It’s 2009 and we should be more responsive to our databases by now. Except for the edge cases we really have no excuse now for being surprised by the majority of performance issues that surprise us today.

Like content like this? Subscribe to the StraightPath SQL Server blog feed in your favorite feed reader.

How many times have you heard or uttered those words? You work hard and understand the business requirements, you try and get the logic to work to make the procedure return what was expected or update the data as expected. You do some unit tests in your environment, it may not scream but it comes back in what seams like a reasonable time, the results appear right and you say, “Ship It!” It passes QA (if you are someplace that has a proper QA process) and it gets to production. Then the DBA walks by (or a user calls you) and something is wrong… It isn’t returning, or it is taking far too long or timing out even.

Why? It “Worked fine in development”!!!

If you have experienced that with SQL Server development (and who hasn’t at least once) then I am hopefully writing to you here. I will warn you ahead of time, though: some of the things in this post may seem obvious. I think we need to take a quick back to basics approach here though based on most of the cases I see…

The Basics

• How much data is in dev?-  Seems kind of basic but it is often overlooked. Especially without a production mirroring Test/UAT/Performance environment (Which more environments than you think suffer from). If your development database has a fraction of the data in production you may be in for some surprises.

• Parallelism – Most machines now have more than one (physical or logical) CPU but what if your development environment doesn’t and prod does? (or vice versa which would be kind of odd). What if you don’t mirror your Degree of Parallelism settings? You could go to production and a query that seemed to perform well is now not. One recent example I saw had a query plan that was influenced by parallelism choosing an index scan when it should have been doing a seek. It was reading far too much data but all things were the same between dev and prod except for the number of procs (1 to 8). The fix wasn’t to disable parallelism, the fix was an important missing index. In development the plan was still inefficient but it didn’t “feel” inefficient to the developer.
• You do look at your query plans, right? - As I blogged about here, reading a query plan isn’t rocket science. Once you start looking at them they become clearer over time. The real danger, however, is not looking at them. Even if your query comes back in half a second in dev, look at the plan and make sure you aren’t doing anything silly.

• While you are at it what about IO Statistics or Profiler? – Whenever I am tuning a query, I generally have SET STATISTICS IO ON at the top of that query. I want to see what the reads look like. You will find debates about tuning to reads but from my point of view it is important to keep your reads lower. Not to the point where doing so increases duration because of some crazy logic. Generally it is my finding that the less reads a query does, the less duration it takes. Not always true but look at your reads in dev. If it is high there and in production there are a lot of similar queries and more than the one user in dev you may be asking the production server to flush pages from memory at a faster rate than you would otherwise like. This means more physical I/Os, which are not as happy as logical I/Os. Profiler is a great tool (as Gail blogged about here) to show you the impact of a UDF. Your query plan and I/O Statistics are not going to show you the impact of that UDF you have in your select list because you are a procedural programmer turned SQL Developer. In fact don’t just look at the duration/reads in profiler but look through Gail’s post and look at the statement completed events to see just how many times that UDF code that was so efficient alone is being called.

• You do keep your statistics updated, right? – Sounds like a “No Duh” but it isn’t in a lot of places. How often do you update your statistics in the database? When you refresh dev, do you update statistics? Do you have a job to do so in Prod as well? If your statistics are out of date, your query plans may not make a lot of sense and they could be sub-optimal. The optimizer is a powerful tool but it can only be as good as the information it has when figuring out an approach.

•  How often do you compare schemas? – Your dev environment does look like production, right? What about your indexes? Do you ever run a tool like Visual Studio Team System for Database Professionals or SQL Compare from Redgate (not a paid endorser but this is the tool I personally use for schema comparisons, even with VSTS DB in house which I like for other reasons), etc? You may be surprised but I have seen a lot of cases where a developer has created an index in dev or even in QA/UAT/Performance (Why, oh why, did you give the developers permissions to those environments…) and got quite busy and forgot that they meant to check that the index was scripted and included in a deployment plan.

There are other causes and there are even some strange occurrences that take a bit of time digging into. Generally, though, these are the reasons I tend to uncover in this scenario. Another cause is not meant as a slight against any one role in a project, it is something we all mutually contribute to:

The development is NOT done when the expected results are returned or the expected actions occur!

I have seen overworked developers and DBAs, under funded project managers (resource and time wise), over zealous management (as far as this date will be met, no matter how realistic it is) and a variety of other causes to that rant above. The simple truth is, development is done when it performs the expected action, the unit tests in dev have verified it and the unit tests also looked at performance and the piece of code is performing as best as it possibly can given the circumstances. Low hanging fruit type problems abound in most environments I have encountered. I know the projects never have enough time but sometimes 20 minutes extra on a procedure could have meant that one wouldn’t have showed up on a “worst offenders” list.

I don’t want to put all of the onus on development either. DBAs should setup standards and best practices and have code reviews. They should mentor their teams in performance areas. Senior developers should do the same.

Management needs to put an emphasis on proper testing, including performance testing (either in dedicated environments that mirror production hardware and data as best as possible or in a test environment that is as close as possible). There will always be performance problems that slip past development just like non-performance bugs. Without searching for them before production, it’s your users that will be your bug reporters. That could mean the difference between retaining a client or system adoption.

So I wrote a lot of posts around shrinking (Here, here, here). Some of my posts were turned into a wiki article on SQLServerPedia. I thought I was done posting about this but one more.

I talked about why we shouldn’t do it on a regular basis. Why it should really be reserved for use in dev or extenuating circumstances due to a lack of proper planning. I also talked about why it is bad to let your databases grow in small chunks.

I can’t stress it enough… Right sizing your database is critical. This requires PLANNING. If you don’t plan for performance you will be killed by performance in no time. If your company or your project manager says something like, “We don’t have time to plan for that” then you will pay for it later at least 75% of the time and you will pay more than you would have up front.

The below picture is from a system somewhere that I got involved in after it had been “alive” (persistent vegetative state…) for a couple years. It was set to do a DB shrink each night after a transaction log truncate (Again –> Don’t do this! If you want to recover to a point in time, use FULL Recovery mode and TAKE LOG BACKUPS at a frequency that manages your recovery needs (first) and your transaction log growth (second). This was done on each database on the instance.

Lots of Red and Lots of Fragments

This is with the windows defragment tool. A quick reminder: Blue (there are about 5 or 6 skinny blue lines in the above picture) indicates contiguous space, no physical file fragmentation. Red indicates fragments. No, I didn’t play with the colors. Most other drives remind me of Christmas also.

What does this mean? This means that I/O operations against this drive have an unfair advantage working against them. Reads will not be as well performing as they should. (For instance copying an LDF file off of this – data file drive – to move the .ldf to a log file drive took 4 minutes. No other activity, file was 1GB..)

How did it get this way? Every night the files on this data file drive were shrunk as part of a shrinkdb operation. Every night after this shrink an ETL process loads it. Files then grow in chunks during the ETL (and some of them grow during the day with use, like the report server temp database). These growths and shrinks lead to file fragmentation.

How long did it take? Much like my current status on the scale (14.6 stone when I should be 12.5.. I’ll let you bing, google or ask wolfram alpha if you care) this didn’t happen overnight. It took a period of neglect. Neglect including:

• The Shrinks/growths
• Never planning for the expected space and proper sizing in the first place
• Allocating the bare minimum to the drive (88% full right now) and adding more space from the SAN as “needed”.
• Never running a defragment process during maintenance.

What do we do? Well for the short term we are going to grow the databases to an appropriate amount (hopefully getting more space allocated before all of this obviously), back them up, turn off the SQL Server services, remove any files that shouldn’t be on this drive and defragment the volume.

What do we do in the future? The shrinking operations were killed awhile ago (except for one that was hidden in an ETL process, since removed), I will make sure that a process is put in place to check for fragmentation more frequently. I will keep fighting the fight to properly size a database from the beginning.

Not My Job

Nearly no other phrase can irk me quite like that. You don’t even have to say it, you can just experience it. Now I am guilty here also. Where this case happened, the databases are managed from SQL out by the DBA, the servers/storage/etc are managed by the storage and server team. Put trust in that team and never really looked at fragmentation. OOPS. Now it isn’t my job and normally I go out of my way to worry about things that could affect databases I care about, for some reason I never checked fragmentation here.

When talking to the sysadmin about I/O issues, I started talking about a desire to split some of the report server databases onto their own drives (large reports running, seeing a lot of blocking with the built in report server procs that write/read to the databases) and the reaction was very quickly “No, you have high fragmentation.. let me show you” and then we opened disk management in window and looked at the fragmentation. My reaction was guilt and shame for not checking. Then my second reaction was (internally) “How did you know this and not say anything?!” The conversation basically revealed that this was a known issue for some time. Now before the DBA role was filled there was no one else looking at database performance on the applications team. How did this sysadmin expect project managers and developers without much systems/DBA knowledge play Carnac and figure out the fragmentation?

The attitude that killed here? “I just create the disks and give them to the people who request it, I give them the space they asked for and that’s it.”

You can’t do that. If you want to be a good DBA you need to look above and below your part of the food chain and if you see an area that needs improvement, SPEAK UP. Same goes for sysadmins and storage teams. You are one company with the same customers and the same goal of happy customers.

Reminder –> Are you looking at your physical file fragmentation levels? I am sure you are a great DBA and update your stats, rebuild indexes when appropriate. Do you look at your I/O subsystem? Do you check your fragmentation? Do you get blinded by the simple in search for a complex situation? I do at times. Check your fragmentation levels and run some benchmarks. Fix this headache before it becomes a resume altering situation.

Documentation

(…Or how I learned to stop procrastinating and love tolerate a chore necessary task)

Do you document? Do you document well?

As I told Paul Randal (twitter, blog) today on twitter, this blog post was inspired by an observation in the shower and a recent question by Brent Ozar (twitter, blog). You didn’t need to know that but now you do. I happened to read the back of the body wash and noticed the directions had some minor flaws. It made me think of some of the documentation I have produced or tried to rely on and some common mistakes. Let’s dissect the directions on the soap:

“Directions for Daily use in shower or bath: Squeeze desired amount of product onto wet cloth or cleansing pouf. Work into a lather. Rinse Off”

The Good #1 – The directions exist

A colleague often answers my comments on performance with, “The perfect is the enemy of the good” now we won’t get into perceptions of what “perfect” means and what “good” means but in some regard he is right. If you have documentation and it is useful and not dangerously misinforming people it is a start (a draft) and it’s okay to write and publish a draft. If your documentation will allow you to go on vacation, not have dreams about work and let your backup handle the majority of the tasks well then it’s a great start. Get your favorite document editing tool and at least start something. Not starting documentation because you are afraid of missing something, can’t figure out where to start, etc. is a disservice to you and your organization.

The Bad # 1 – There are assumptions

Assumptions are in two flavors here. An assumption that one already follows a process (“For daily use..” If I didn’t shower daily, I’d have a lot more time for blog posts at night because the couch is uncomfortable for sleeping but what if I didn’t?).

Alright so let’s be serious with the other assumption. They assume the person reading the instructions knows what a “pouf” is (i’ll admit it.. I know what one is… Alright, fine I will admit it – I have and use one…) They assume your idea of the “desired” amount is sufficient.

• The assumptions, definitions, overview and/or introduction sections of a document should be used. Let the audience know what you expect of them, define terms that can be ambiguous at your organization and tell them the goal you had in writing the document.
• You should write to your audience (and think about the future audience). Can someone pick up the document (especially if it is a runbook, deployment guide, etc.) and do what is required without a Vulcan mind meld with you? If not you are assuming too much.
• Test (I know. I harp on testing a lot) your document. Give it to someone who may not be the other DBA on your team but could potentially be in a position to have to rely on the document. Can they follow it? Do you have an environment you can deploy to after dev/integration where a deploy guide is written/unit tested? Someplace where someone other than the author can prove out the steps?

The bad #2 – There are missing steps

This can be lumped with assumptions but I felt it needed a special callout. The piece that caught my eye this morning was “Work into a lather. Rinse off.” If I were to follow the directions literally, I could see myself squeezing the product onto my pouf (Wipe that smirk off your face), and just lathering it up and then rinsing the pouf off. Over time I would have less drive by attacks at my cube but only because of the smell –> I would simply not be clean.

They are assuming (see I told you it could be lumped with assumptions) that the reader knows how to bathe (then why even bother with the directions?). Don’t write a document with obvious information missing because it is obvious to you. 

Ask yourself, “what is the goal of the document?” Hopefully its in the objective/introduction/overview section. Read your document and ask if it meets the goal. Have someone that isn’t always finishing your sentences read it and see if they agree.

Why do I embrace documentation now?

• I am making myself more replaceable – I like uninterrupted vacations, I like letting other people do deployments. Earlier in my career I wanted to always be that go to guy (I still enjoy helping and love gaining and sharing knowledge). I used to love to be that one who saved the day at 2:00AM and led the conference call to do so. Now I like helping others do that. Will I get the “great job!” e-mail from a CIO for it? Probably not but I’ll get to spend time with the family. I can’t think of what kind of fancy thank you e-mail, “we did it” dinner or little plastic trophy beats William or Rachael’s laughter.
• I can say “If you read my maintenance recommendations…” - No, that’s not a great attitude. What I mean is I can let my document be a force multiplier when I can’t get more living, breathing DBAs. I can create templates, suggestions, checklists with steps to create and let “reluctant DBAs” in the infrastructure or deployment teams do some of the menial tasks. They can still call me if there is an issue but if I don’t assume and state the obvious it’s less likely.
• A proposal on paper beats a hallway conversation – Talking is great (I have been know to be a chatty person. I know, you are shocked) but handing a simple document with pros/cons and a suggestion can go a long way. Provide justification for the server maintenance products, provide justification for that indentured servant Jr. DBA, Provide a rationale for going to the PASS Summit, etc.)
• Maybe, just maybe, when that problem arises again, we can solve it quicker – Even just using a blog as a knowledge base, a well organized e-mail folder structure, etc. can help deal with that problem that “happened at year end the past 3 years… what was it again? You had to do something to the readerform switch in the whoozywhatzy table… Next year we’ll be ready for it!!”

The above personality traits sound like bad news. In most areas of life they certainly can be. That being said a DBA needs to have a certain aspect of these to best do their jobs.

As I blogged about in “Checklists, Recipes and Algorithms“, I describe other careers where a keen eye to detail and a mental preparedness for worst case situations are important. I tend to find that the most successful DBAs are those who come across as paranoid. This tends to result in them wanting more control over how their environments (see… “Their Environment” I told you, it comes off as being a control freak.) This in turn results in a worry about the little things that can turn into a resume altering situation.

If you don’t believe me, look at this recent blog post by Paul Randal (@PaulRandal, blog). He is the self professed “most paranoid person on the planet in terms of backups, corruption, HA, etc.” I won’t argue with him and I’ll give myself the label of second most paranoid person in those same terms.

If you are, or want to be a DBA the question you have to ask your self is: Are You?

How can these traits be seen as positive?

Paranoia – Again it’s not the best choice of words. Maybe “realist”, “careful”, “cautious”, “protective” would be better terms. Paranoia implies a lack of rationality and the fears that a DBA should have are entirely rationale:

• Data Breaks –> Be it malicious, accidental, user created, I/O subsystem “created”, corruption, etc. Data can be lost or become unavailable. Like I blogged about in “Where Do I start?!?!” the most important ability of a DBA is recoverability.
• Security Threats Exist –> From within, from without they exist. Over confidence here means compromise and a long walk to the CIOs office.
• Not all code is perfect the first time –> Even if your crack development team wrote it, even if your start vendor sold it. Do you just implement it without question, without preparing for problems? You are setting your organization up for problems if you do.

Having the “This instance can go down” or “that code can cause a problem” mentality (guilty until proven innocent) means you will catch a problem more likely. If you presume everything is A-OK all the time then it could take a large exception to that rule of thumb to teach you a lesson (and give you a new career path).

This leads to a certain “control freak” philosophy. You don’t want the entire organization in the system as System Administrator fixed server role (you don’t even want the entire DBA team to have that level of rights). You don’t want code slammed into production without a proper test plan and documentation trail. You don’t want people trying to help without the right knowledge undoing your efforts.

DBAs always say no… That’s a myth but it’s joked about in most companies (we are also called disk sponges at most places but that’s another post .) A good DBA won’t always say no but they shouldn’t be afraid to say, “Wow! let’s slow it down” or “Not like that”.

Are you assertive? If you aren’t you will be walked over and that can and will backfire. You need to be sure of your standards and enforce them equally. Even to the developer who is older, has been their longer and has that “rules slow us down, we have dates darn it!” attitude.

If you are “paranoid and controlling” you will know why you have the standards. You will know why you should say no and you will know the potential results. That should help you stand your ground and work towards a great solution.

That’s the key –> A good DBA will say “Not yet” but won’t turn around and walk away. Be approachable, work to help people understand how to get to the standards. Teach a brown bag lunch (even if you don’t feel ready the questions you don’t have answers to at the training are all learning opportunities for you, step out and try it). Work side by side with developers. Approach project managers and tell them why involving the DBA earlier will make a difference. Eventually people should start to learn that the paranoia is for their good. They’ll see you as what a DBA should be – an advocate for the helpless, innocent (usually innocent anyway) data of the company.

You are there to be ready to assist when things go bad after the first 40-50 hours of the week. You do this by having that paranoid, controlling attitude for the first 40-50 hours of the week, hopefully reducing the overtime break fix work.

I had submitted an abstract to the SQL PASS 2009 Community Summit in the professional development track. Since I was helping rate abstracts and in that track, I down voted my own session pretty heavily and it missed the mark. It was titled “Iceberg, Dead Ahead!” and I was going to talk about various real life disasters and the attitudes and actions that either made a positive or – far too often – negative impact.

When you look at real world disasters in any field you can far too often see some apathy by some or many involved. People ignoring small issues that multiply and cause catastrophes. People with a very optimistic attitude assuming the best ignoring “acceptable errors” are not behind all disasters but they are behind too many. Sometimes a bit of paranoia and someone being assertive with standards could have made a difference.

Step outside of SQL Server and ask where else can this attitude help?

Ever take a course involving firearms safety? The top 3 lessons hammered into you are: The gun is always loaded, keep your finger off the trigger until ready to fire and only point the muzzle at what you are comfortable with bullet holes in. Even if you just unloaded it, just verified it three times you still follow these rules. Why? Because it’s the comfort and over confidence that leads to mistakes. If you break the rules once you will again and it could be the wrong time.

Do you ever double check that you locked a door? I don’t mean go back and check 75 times but if you presume you could have made a mistake or been distracted and double check you will be that much more likely to leave with a door locked behind you.

Do you presume a fire will never hit your house, office, hotel? If so you don’t know where the nearest exit is, you haven’t thought through what you would do and you are not prepared. Statistics show you are more likely to die in a fire than someone who prepared. Again, I am not saying walk around in a set of turnout gear with breathing air on your back. I am saying look at exits, plan escapes, think about the unlikely (but possible) and think about your reaction to that.

Do you always ignore the safety procedures on your flights? Nose into a book and not even pay attention to what type of plane you are flying on for this trip? Where is the nearest exit to your seat? Where is a second exit? The folks who landed on the Hudson River in January of 2009 would tell you that this information matters. They will even tell you that the instructions on how to use your seat as floatation device aren’t there only because of federal regulations. I bet they all listen intently on every flight now.

So are you fine? Is your environment invincible? Or are you a “paranoid” DBA? Add a healthy “paranoia” to your toolbox!