Yes, he was impeached, by the house and acquitted by the Senate. Not the point of my post today, forget the politics. He wasn’t impeached for all of that business with Monica Lewinski. Was it unethical? Yes. Was it immoral? Well to me it is. Was that activity an impeachable offense? No. Why was he in legal trouble then? Simple: he lied. He, according to the conviction in the House, perjured himself about his affair while under oath. You may also remember his infamous press conference where he did the same thing to the public.
What’s Your Point? This is a SQL Server Blog…
Well my point is simple, often the cover-up or lie is worse (or has worse effects at least) than the act being lied about.
That’s where it relates to my chosen career field. That is where we, as DBAs, Sysadmins, Developers, etc., have to make sure we pay attention to the many lessons we see in public oopses.
Ever Have This Feeling?
One time, I was doing some work on a critical production cluster. I was adding drives and modifying an existing drive. Took the drive offline (shh if you already know how this ends). Took the disk offline… And then… I had that feeling. Please tell me you know the feeling also – my face felt like it was on fire. My fight or flight system was in overdrive as I had an adrenaline dump. Hands got a bit jittery, awareness increased while at the same time a narrow focus started. My stomach? Oh man. That horrible feeling of your stomach being twisted and turned upside down like an engine that just won’t turn over. Yup… I realized what I did when I saw the SQL cluster begin a dance of fail-overs. You bonehead! That drive was a dependency of the SQL Server group.
Yup. I brought the production cluster to a standstill for about 10 minutes while I hustled to repair the damage done. So while fixing that issue I had two choices of how I would proceed:
- Listen to the voice in my head that says (somehow italics look more sneaky), “quickly fix it, figure out how to cover the tracks and draft an e-mail that starts with something like, ‘a mysterious issue…’ “
- Listen to the voice in my head that says, “quickly fix it, ping someone in charge and give them a heads up and let them know what you did. Prepare to write a note explaining the event.”
I chose Option 2. I’m not saying I’m better than you if you went with Option 1. I’m not saying I’m perfect. I’m just saying that Option 1 doesn’t work. Early on in my career I may have gone with something between the two options. Even that doesn’t work because it leaves untied ends. The best manager I ever worked for (I blogged about him when I answered the SQL Quiz on Leadership) was really big on this. His philosophy on this more or less was, “Own it”. If you don’t own it on his team you have a problem, probably lost some respect and trust. If you do own it and the mistake or issue wasn’t something really horrible, you’ll end up alright in the long run.
How Do You “Own It”?
It’s simple. You admit you messed up, fix the issue or enlist help in fixing the issue, make a plan to prevent this from happening in the future and move on. You’ll be fine. We are all human, we all mess up and most of us still have our jobs. Heck, even Bill Clinton after his not owning it commands a lot of money to speak.
Now one of my flaws that I am still working on after 10 years in the field is keeping e-mails short and to the point. So take this advice with that caveat in mind. A good quick template of an e-mail I could have sent in that issue above:
Earlier today the cluster service became unavailable while I was doing maintenance on it. This e-mail outlines what happened.
I was tasked with removing a disk that was no longer being used. As this disk was removed, I realized (too late) that it was a dependency on the SQL Server group in the cluster. This means that the SQL Server failed back and forth between the nodes until it rested in a failed state. I was able to resolve this by breaking the dependency on the unused disk and bring the cluster back online.
How Was This Missed?
I should have checked the dependencies before touching the disk. I did not use a checklist to check for this and I missed this important detail.
What will be done to prevent it?
I have created a document for working with cluster resources like disks. This document links back to Microsoft checklists and includes a warning about resource dependencies with a process to check this. I have also sent a note to the server teams and DBA teams outlining what happened.
That isn’t exactly what I would write but the bolded points are the “pattern”. Describe what happened, why it happened and how it won’t happen again… And then move on. Deal with the repercussions and feel confident knowing that you were honest and up front and you mitigated your own issue. And remember…
You Learned a Good Lesson
So maybe this isn’t one to include in the e-mail or in a mea culpa discussion with your manager (though you should have that mea culpa discussion). Keep this tip for yourself – whatever you did wrong… whatever caused that stomach roller coaster just taught you a lesson you won’t soon forget. Like I said, unless you really, totally ruined the day for your company, you’ll still have a job. You’ll still get respect from your colleagues because you quickly came clean and even talked about how to prevent it. You learned something, though. So eventually you may even have a net positive out of the issue. Just remember to learn from that lesson and don’t let pride get in the way.
Bring Your Own Negative Example
So, I didn’t talk about some negative examples. I can instantly think of the times where I have been involved with projects and teams that had people who listened to voice number 1. I won’t. I don’t need to. You can think of them already, right? How did you feel as the coworker (or victim) of the person who reached for the broom and the carpet corner? Don’t be that person…
Share your story below in the comments. I want to hear about how you cope with a mistake you make on the job. I want to hear about a counter example that you’ve bumped into and how it made you feel.