Tuesday, May 30, 2017

A Quick Look at scalability [Part II]: #SQLServer Checktable without physical_only option

In my last post, we looked at the scalability of 'dbcc checktable... with physical_only' for two tables, from dop 1 to dop 8.

A Quick Look at Scalability [Part I]: #SQLServer Checktable with physical_only

While the operation achieved high utilization of a single vcpu for both tables at dop 1, as dop increased cpu utilization did not keep pace.  Total cpu_ms for the operation was fairly stable.  The results was that elapsed time did not decrease in the same proportion as cpu resources were added by increasing dop.  The main decreases in elapsed time were realized by dop 4, further decreases from dop 5 to 8 were minimal.  The following two graphs summarize these findings well.

All right.   What if we look at full checktable operations, without the physical_only option?  Does that operation scale the same, better, or worse than physical_only?

Here's the results from TableA.  Cool!  Once again, cpu_ms for the operation remained fairly stable.  Rather than continuing to increase as dop increased, idle_ms reached an early peak and remained fairly stable afterward.  The result is decreasing elapsed time with each increase in dop from 1 to 8.  It looks as if elapsed time may continue to decrease beyond dop 8... someday I'll get to test :-) But the gains after dop 8 look as if they'll be smaller and will almost certainly drop off by dop 12.

The results from TableB sent me back to test again, and dig further into the data I gathered.  Check it out.  The cpu_ms is still fairly stable.  Elapsed time decreases with each increase of dop.  But... whoa.  What an unexpected shape from idle_ms.

I thought the 'hill' must be due to a 'one-off' abnormality, not likely to be repeated.  So I repeated tests for dop 2, 3, 4, and 5.  Plugged the results into the graph and... whoa.

Here's one more test run.  This is probably the most characteristic... but so many factors can throw the numbers off.

So the good news for today is that checktable operations without the physical_only option scale farther/better on my test tables than checktable with physical_only.  While with physical_only scaling benefits in elapsed time are primarily seen only to dop 4, without the physical_only option elapsed time benefits to increasing dop extend at least to dop 8.

And we saw that the shape of scalability graphs is pretty volatile :-)  That's largely because modest changes in elapsed time are multiplied by dop in this calculation to arrive at the idle_ms number - that idle_ms number is the one that changes shape most readily.

Friday, May 26, 2017

A Quick Look at Scalability [Part I]: #SQLServer Checktable with physical_only

Ewald Cress and I have been discussing the OLEDB wait for a while - and at this point he's been pulling all the weight in terms of blogging :-)
His post on the nature of OLEDB wait is great - I highly recommend it.

Scheduler stories: OLEDB, the external wait that isn’t preemptive

Before I start digging into the waits experienced by checktable, and in particular OLEDB wait, I want to take a high level look at the scalability of checktable operations, starting with checktable with physical_only.  Below are graphs and additional information showing checktable executions on my 8 vcpu VM (with 64 gb vRAM).  Got results from maxdop 1 to maxdop 8. (Although I actually ran the tests in descending order of maxdop for reasons unknown even to me 😜 )

Lets talk about the difference between "performance" and "scalability".  Performance is a measure of the pace of work that a workload can achieve on a given system - or alternatively the measure of time required to complete a unit of work on that system.  Scalability is the capacity of the workload and system to increase pace of work as additional resources are added.

So we can describe the performance of a checktable operation on a given table and system, at a given degree of parallelism (since the degree of parallelism as well as the number of (v)CPUs on the system may both be limiters of compute resources for the workload).  As the degree of parallelism changes from 1 to the number of (v)CPUs on the system, the performance of checktable may change.  Describing that change in performance will characterize the scalability of checktable on that system.  In the final analysis, a scalability limit may come from the workload(application level coding or database level coding), or it may come from the system.    

All of the numbers below are from perfmon - collected with a 1 second interval, logged to a csv.   The X axis is number of seconds since the start of the checktable(1-indexed rather than 0-indexed).  Don't worry - as I continue I'll be pulling in lots of stuff from DMVs that can't easily be gleaned from perfmon. :-)  But any of the info I grab from DMVs will be summary of time period fgrom start to end opf operation, rather than in 1 second increments.  Observer overhead of the DMV queries is too high for my tastes otherwise.

This system is running SQL Server 2016 SP1.

Here's the operation at DOP 1.  All of the checktable operations are running in a Resource Pool named after me.  In this one you'll see something interesting that I can't explain.  For approximately 60 seconds at the tail end of the activity, CPU utilization hovered near 25%.  My DOP 1 checktable was the only active request in SQL Server, though (confirmed by looking for active requests in the default and internal Resource Pools, the only other Resource Pools in the instance).

Something outside of SQL Server was using 12.5% CPU utilization in the VM.  Oh well... it didn't have enough of a footprint in memory or disk to cause an issue, and my checktable was happily running along on its own vcpu.  Notice that neither the rate of disk read bytes and the rate of logical scan read bytes was able to achieve previous maximum sustained levels, and the CPU usage reported for my resource pool remained at a steady 12.5% throughout.  

At DOP 2, on the 8 vcpu vm the checktable could account for up to 25% of vm-wide cpu utilization.  But wait time appears to have increased substantially, and cpu utilization reaches 25% only briefly near the end.

At maxdop 3, checktable could consume 37.5% cpu.  But it seems to be falling farther from its potential maximum.

Maxdop 4 *could* see up to 50% CPU utilization... doesn't get too close.

62.5% maximum possible at maxdop 5 - but it seems that wait time increases must be outpacing declines in elapsed time.  This checktable does not seem to be scaling very well.

Maxdop 6 would allow for a maximum of 75% CPU utilization.

Maxdop 7 would allow for up to 87.5% CPU utilization.  Even the brief spikes don't get too close.

At maxdop 8 on the 8 vcpu vm, it appears about half of the cpu time is spent in a wait state.

I think its very important to look at time series graphs to understand behavior... hope I didn't lose you to boredom by including all eight of the time series.  There's interesting stuff I hope to return to later in there - especially that always-present peak in CPU utilization in the parallel operations about 2/3 of the way in.

But to compare scalability of checktable across tables, and across options such as physical_only, we'll have to look at things from a higher level.

Each of the schedulers engaged by checktable is either accruing cpu_ms for the checktable (as reported by sys.dm_exec_requests) or not at any given time.  Since these tests are running in isolation on the vm, for now we can assume that time for the schedulers/vcpus that isn't accounted for in cpu_ms is idle_ms.

So we can use this formula when dop, elapsed_ms, and cpu_ms are known:

DOP * elapsed_ms = cpu_ms + idle_ms

That allows the 8 checktable operations to be summarized in this graph.  From DOP1 to DOP 8 the cpu_ms of the operation is extremely steady.  From DOP 1 to DOP 4 there are significant decreases in elapsed time as dop increases.  After dop 4, reduction in elapsed time is slight.  Throughout the tested range, idle_ms increased at a nearly linear rate.

I've got another table - TableB that I've been working with extensively.
Here's how scalability for checktable with physical_only looked on TableB.  Remarkably similar to scalability for TableA.  Hmmm...

Wednesday, May 10, 2017

SQL Server: table & index names from sys.dm_tran_locks

Martin Smith gave a great answer on StackOverflow - as he often does - when someone was looking to get a table name from the information in sys.dm_tran_locks.

Here's my take at an answer - trying to cut down on the rows returned to keep the list short in busy databases.

;WITH tLock_cte(request_session_id, dbName, tableName, indexName, resource_type, request_mode) AS
(SELECT tL.request_session_id,
        db_name(tL.resource_database_id) AS dbName,
        CASE WHEN tL.resource_database_id <> db_id() THEN '[in another database]' 
             WHEN tL.resource_type = 'OBJECT' 
                  THEN object_name(tL.resource_associated_entity_id)
             WHEN tL.resource_type = 'ALLOCATION_UNIT' THEN object_name(sp3.[object_id])
             WHEN au1.type = 0 THEN 'DEFERRED DROP'
             WHEN au1.type = 2 THEN object_name(sp2.[object_id])
             ELSE object_name(sp1.[object_id])
        END       AS tableName,
        si.[name] AS indexName,
 FROM sys.dm_tran_locks               AS tL
 LEFT OUTER JOIN sys.allocation_units AS au1 
 ON au1.container_id = tL.resource_associated_entity_id
 LEFT OUTER JOIN sys.allocation_units AS au2
 ON au2.allocation_unit_id = tL.resource_associated_entity_id
 LEFT OUTER JOIN sys.partitions       AS sp1
 ON sp1.hobt_id = au1.container_id
 LEFT OUTER JOIN sys.partitions       AS sp2
 ON sp2.[partition_id] = au1.container_id
 LEFT OUTER JOIN sys.partitions       AS sp3
 ON sp3.[partition_id] = au2.container_id
 LEFT OUTER JOIN sys.indexes          AS si 
 ON si.[object_id]  = COALESCE(sp1.[object_id], sp2.[object_id], sp3.[object_id])
    AND si.index_id = COALESCE(sp1.index_id, sp2.index_id, sp3.index_id)
 WHERE tL.resource_associated_entity_id IS NOT NULL
 AND   tL.resource_type IN ('OBJECT', 'PAGE', 'RID', 'KEY',
                            'EXTENT', 'ALLOCATION_UNIT', 'HOBT'))
SELECT   request_session_id, dbName, tableName, indexName, resource_type, request_mode
FROM     tLock_cte
GROUP BY request_session_id, dbName, tableName, indexName, resource_type, request_mode;

Let's give it a test.

CREATE TABLE test_locks
(num BIGINT);
CREATE INDEX nci_test ON test_locks(num);
INSERT INTO test_locks SELECT 1;
DELETE FROM test_locks WHERE 1 = 0;


The results...

Good enough for now.

So let's commit the transaction in order to release the locks.  And I'll have more for you another day...