Haven’t looked into all of the details underneath but at the surface probably a good reminder of where WITH RECOMPILE can be helpful…

Using SQL Server Reporting Services, a workaround was used (this is the part I want to look into more) to basically get a blank report showing when a user first links to a report until they change parameters and run the report themselves. So this first execution gives an false year/month to search back until (190001). The procedure behind this report looks for any “invalid” (per the business rules that means less than 195001) date and issues a “RETURN”, returning control to the color.

The problem was picked up during an execution after a restart. When the Reporting Services query ran asking for 200809 it ran until it timed out, no results coming back. When the query was executed for 200809 from SQL Server Management Studio, it came back in about 30 seconds.

Two issues at play here 1.) The sessions are getting different query plans (topic for another post.. various reasons this can happen even for the same proc) and 2.) The query plan used by SSRS is being compiled first with the first time an SSRS session ever executes the query (190001).

This can be verified by looking at the xml showplan for the execution and seeing what the parameter is compiled for.

In this case, adding WITH RECOMPILE to the stored procedure before the AS, tells SQL to recalculate the plan at each execution instead of cache and reuse a plan for every execution. This resulted in the reporting services query coming back in 35 seconds as opposed to timing out.

Obviously longer term solutions include figuring out why this was designed this way and looking at other possible options. Normally WITH RECOMPILE is best when working with queries that are called with dramatically different parameter values each time that cause differences in the way the plan is figured.

For another post, we’ll talk about how you can have the same procedure use different plans even for the same server and how to minimize/avoid that.

Interesting Issue Today/Yesterday -

A query was using a linked server to grab data from DB2 via an ODBC connection. Normal execution time is about 15 minutes. This had run on for at least 7 hours when it was decided to kill it, appeared to be doing nothing but causing blocks (due to Intent Shared and Intent Exclusive locks preparing to do an update if the data ever came back).

Waited a few hours, still didn’t die so we figured we could wait, if it took 7 hours prior to killing, rollback could potentially take at least that long if not much longer.

Came back today and it was still running.. By this time now about 30 hours since initial query executed and 23 hours after KILL issued. Process state was rollback.

When I looked at the stats for the session (SELECT * FROM SYS.DM_EXEC_REQUESTS), I noticed the CPU counter incrementing so it was at least thinking about doing something if not in the middle of doing anything. Rollback just didn’t make sense to be so long knowing what the query did.

Left me with few options though:

• Restart SQL Service – risk this database coming back up in recovery and not available at all
• Something horrible and drastic – Not a great troubleshooting methodology so put this out of my mind as a serious option… But rebuilding the log would probably have “solved” the immediate issue while potentially leaving a database in a transactionally inconsistent state
• Wait and hope – Safest but due to timelines of necessary processing was not a truly viable option.

So I called MS Support. Great assistance (as normal from SQL support) but confirmed what I was thinking: no help. They had no tools to undo this and had the three options I had above at first. They came back with a 4th I hadn’t considered:

• Take a full backup, restore it with a new name, verify everything appears right and cut over to that database (change name of DB with rollback operation, change name again of the restored DB).

The escalatio engineer had a good point: when you take the full backup that transaction that is rolling back was never comitted and would not be grabbed by that backup so the restore will leave a consistent and ready database.

Before going that route, I searched some more on the issue and after looking at what it was doing (again not a lot of work) and some common repeats of this issue (when using a linked server some folks reported hung queries that on rollback after kill would be in rollback state for as long as 7 days before they gave up and restarted). So I decided to take two of the options to mitigate the risk. Feeling confident that a restart would probably be the end all I:

1. Backed up the offending database to use the 4th option from MSFT PSS if need be
2. Went through the change notification process for the service recycle
3. Restarted SQL Service
4. Held my breath and
5. Presto! The database recovered instantly in a good state
6. DBCC CHECKDB run to ensure DB free of issues that would be picked up here
7. Had users query data to ensure it seemed right
8. Back in business

In keeping with the thought of mitigation, I will be looking into more behind the why, investigate the ODBC drivers and versions being used here, attempt to recreate the same issue in a different environment and remember this blog post is here if I need it…