Showing posts with label Statistics. Show all posts
Showing posts with label Statistics. Show all posts

Friday, September 11, 2020

SQL Server snapshot db temporary stats - created or updated

For a great introduction to this topic, please see this post at SQLSkills from Joe Sack.

Temporary Statistics for Database Snapshots in SQL Server 2012

https://www.sqlskills.com/blogs/joe/temporary-statistics-for-database-snapshots-in-sql-server-2012/

Today's test instance is running SQL Server 2019 CU6.  

OK, let's create a test database.

USE [master]
GO

CREATE DATABASE [snap_test_src]
 CONTAINMENT = NONE
 ON  PRIMARY 
( NAME = N'snap_test_src', SIZE = 8192KB , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB 
, FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\snap_test_src.mdf')
 LOG ON 
( NAME = N'snap_test_src__log', SIZE = 8192KB , MAXSIZE = 2GB , FILEGROWTH = 65536KB 
, FILENAME = N'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\snap_test_src__log.ldf')

ALTER DATABASE [snap_test_src] SET AUTO_CREATE_STATISTICS ON
ALTER DATABASE [snap_test_src] SET AUTO_UPDATE_STATISTICS ON
ALTER DATABASE [snap_test_src] SET AUTO_UPDATE_STATISTICS_ASYNC OFF

ALTER DATABASE [snap_test_src] SET ANSI_NULL_DEFAULT OFF
ALTER DATABASE [snap_test_src] SET ANSI_NULLS OFF
ALTER DATABASE [snap_test_src] SET ANSI_PADDING OFF
ALTER DATABASE [snap_test_src] SET ANSI_WARNINGS OFF 
ALTER DATABASE [snap_test_src] SET ARITHABORT OFF
ALTER DATABASE [snap_test_src] SET AUTO_CLOSE OFF 
ALTER DATABASE [snap_test_src] SET AUTO_SHRINK OFF
ALTER DATABASE [snap_test_src] SET CURSOR_CLOSE_ON_COMMIT OFF 
ALTER DATABASE [snap_test_src] SET CURSOR_DEFAULT  GLOBAL 
ALTER DATABASE [snap_test_src] SET CONCAT_NULL_YIELDS_NULL OFF 
ALTER DATABASE [snap_test_src] SET NUMERIC_ROUNDABORT OFF 
ALTER DATABASE [snap_test_src] SET QUOTED_IDENTIFIER OFF
ALTER DATABASE [snap_test_src] SET RECURSIVE_TRIGGERS OFF
ALTER DATABASE [snap_test_src] SET DISABLE_BROKER 
ALTER DATABASE [snap_test_src] SET DATE_CORRELATION_OPTIMIZATION OFF
ALTER DATABASE [snap_test_src] SET TRUSTWORTHY OFF 
ALTER DATABASE [snap_test_src] SET ALLOW_SNAPSHOT_ISOLATION OFF 
ALTER DATABASE [snap_test_src] SET PARAMETERIZATION SIMPLE 
ALTER DATABASE [snap_test_src] SET READ_COMMITTED_SNAPSHOT OFF 
ALTER DATABASE [snap_test_src] SET HONOR_BROKER_PRIORITY OFF 
ALTER DATABASE [snap_test_src] SET RECOVERY FULL 
ALTER DATABASE [snap_test_src] SET PAGE_VERIFY CHECKSUM  
ALTER DATABASE [snap_test_src] SET DB_CHAINING OFF 
ALTER DATABASE [snap_test_src] SET FILESTREAM( NON_TRANSACTED_ACCESS = OFF ) 
ALTER DATABASE [snap_test_src] SET TARGET_RECOVERY_TIME = 60 SECONDS 
ALTER DATABASE [snap_test_src] SET DELAYED_DURABILITY = DISABLED 
ALTER DATABASE [snap_test_src] SET QUERY_STORE = OFF
ALTER DATABASE [snap_test_src] SET READ_WRITE
ALTER DATABASE [snap_test_src] SET MULTI_USER
 

Now let's create a very boring table and put 1000 numbers into each of 2 columns.

 
USE snap_test_src;
CREATE TABLE stats_test(col1 INT NOT NULL, col2 INT NOT NULL);

;WITH num AS 
(SELECT TOP (1000) n = ROW_NUMBER()
                       OVER (ORDER BY (SELECT NULL))
 FROM master..spt_values)
INSERT INTO stats_test
SELECT n, 1001 - n
FROM num;

Because auto_create_statistics is on in this database, a little query will result in creating a statistic on col1.

SELECT * FROM stats_test where col1 < 3;

And here is that statistic.

SELECT ss.name, ss.stats_id, ss.auto_created, ss.user_created, ss.is_temporary
     , sp.last_updated
FROM snap_test_src.sys.stats ss
CROSS APPLY snap_test_src.sys.dm_db_stats_properties(ss.object_id, ss.stats_id) sp
WHERE ss.object_id = object_id('stats_test');

Now let's make that statistic stale 😆 A thousand more numbers oughtta do it.
 
USE snap_test_src;

;WITH num AS 
(SELECT TOP (1000) n = ROW_NUMBER()
                       OVER (ORDER BY (SELECT NULL))
 FROM master..spt_values)
INSERT INTO stats_test
SELECT n, 1001 - n
FROM num;

All right.  Got a table with 2000 rows and a single auto-created statistics object that's now stale.
Let's create a snapshot database!!

CREATE DATABASE snap_test ON
    ( NAME = snap_test_src, 
	  FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\DATA\snap_test_src.ss')
AS SNAPSHOT OF snap_test_src;

OK... now for a query in the snapshot database...

USE snap_test
SELECT * FROM stats_test
WHERE col1 < 3 AND col2 > 998;

Let's check those stats... now there are two stats and both are marked as temporary.

SELECT ss.name, ss.stats_id, ss.auto_created, ss.is_temporary
     , sp.last_updated
FROM snap_test.sys.stats ss
CROSS APPLY snap_test.sys.dm_db_stats_properties(ss.object_id, ss.stats_id) sp
WHERE ss.object_id = object_id('stats_test');

Let's take another look, and compare the stats in the source database and the snapshot database.

(*** update 2020 September 24 ***
this little trick where I reference dm_db_stats_properties in a different database by adding the database name to the reference works in SQL Server 2019 CU6.  But it didn't work on a SQL Server 2016 instance I tried the same trick on - looks like both references to the function resolved to the database context my query was in.)

USE snap_test
SELECT ss.name, source_rows = sp2.rows, source_sample = sp2.rows_sampled
     , snap_rows = sp.rows, snap_sample = sp.rows_sampled
FROM snap_test.sys.stats ss
CROSS APPLY snap_test.sys.dm_db_stats_properties(ss.object_id, ss.stats_id) sp
OUTER APPLY snap_test_src.sys.dm_db_stats_properties(ss.object_id, ss.stats_id) sp2
WHERE ss.object_id = object_id('stats_test');


The source database and the snapshot database both have a stats_test table with 2000 rows, of course.  But the source database has only a statistic on col1, not col2.  And the col1 statistic was last updated when the table had 1000 rows.  The snapshot database has stats on col1, also.  They've been updated - as a temporary statistic - with the row count of 2000 and the sample of all 2000 rows.  That was current as of the time of the snap.

 Consider a snapshot database which is created daily.  The purpose is to provide analytics reporting access while maintenance or data loads take place in the source database.  In the snapshot database, analytics reports have no locking concerns from the activity in the underlying source database.  And the temporary statistics provided by SQL Server, combined with the statistics inherited from the source database, provide a lot of information to the optimizer for query plan selection.

But what if significant fact tables are queried in the snapshot and leave a situation like col2 in stats_test?  A column which generates an auto-created stat in the snapshot, but never gets a statistic created in the source database.  Each day, the cost of creating that statistic and every statistic like it will be paid as part of the workload.  Even if the underlying table is a now-stable dimension. 

If a snapshot database is being used to provide reporting access, it may be valuable to check for temporary statistics that are perpetually being created - or updated - in the snapshot.  In some cases, creating or updating the statistic in the underlying source database may provide a benefit by eliminating redundant work.

q

Thursday, June 28, 2018

Harvesting SQL Server optimizer stats detail from Query Plan XML: Part I Trace Flag 8666

***** Update 6 November 2018 *****

Here's a link to the more efficient type of query that can be used after SQL Server 2016 SP2.  After SQL Server 2016 SP2, trace flag 8666 is no longer needed to get optimizer plan stats details into plan XML.  They are they by default - and in a nicer structure than previously available with T8666.

Here you go...

Harvesting SQL Server optimizer stats detail from Query Plan XML: Part II SQL Server 2016 SP2++

https://sql-sasquatch.blogspot.com/2018/11/harvesting-sql-server-optimizer-stats.html

***** End update *****


Years ago someone said "Hey - why not drop auto-created stats, since the stats you need will just get created again and you'll end up getting rid of those you no longer need."   That *may* be a reasonable step on some systems.  If the risk of bad plans on first execution of a query needed stats that have been dropped is too high, its a bad deal.  If the potential concurrent cost of auto-creating dropped stats is too high, that's a bad deal.  What about analyzing query plans over some period of time to see which stats are actually used in those plans?  Then auto-stats which aren't used in that set of plans could be dropped.

That type of stats analysis could have other uses, too.  Prioritizing manual stats updates in regular maintenance comes to mind.  Or, determining what stats to create/update on an Always On Availability Group primary based on secondary activity.  And troubleshooting problem queries or identifying suspicious "watchlist" stats based on highly variable queries/plans they are involved with.

So I created the following blog post almost 4 years ago... 

Before I drop all those SQL Server auto-generated column stats....
http://sql-sasquatch.blogspot.com/2014/07/before-i-get-rid-of-all-sql-server-auto.html 

And now I'll plead with you to not use the query in that 2014 blog post... it's awful.  If you want to query trace flag 8666 style stats from plan XML, please start from the query in this very 2018 blog post instead - its much more well behaved 😊

In SQL Server 2016 SP2 and after, the optimizer stats consulted in compiling the plan are included in the plan XML by default without trace flag 8666.

Today, let's talk about the world before that.

If trace flag 8666 is set at the session level, a plan retrieved from plan cache will also include details for optimizer stats relevant to the plan in the XML. (Trace flag 8666 also includes other details.) The format is quite different than in SQL Server 2016 SP2 and later. If trace flag 8666 is set globally, plans in the Query Store will also contain the trace flag 8666 details. On a production server, I recommend targeted use of trace flag 8666 for investigation or troubleshooting purposes - please test very carefully for overhead if trace flag 8666 will be used globally.

So here I create a test table, with an integer column and an XML column.  Today I'll insert a single plan for testing.


DROP TABLE IF EXISTS test_plan_XML 
CREATE TABLE test_plan_XML
(plan_id BIGINT,
 test_plan XML);
INSERT INTO test_plan_XML
SELECT 36, '<ShowPlanXML xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/sqlserver/2004/07/showplan" Version="1.481" Build="14.0.3025.34">
  <BatchSequence>
    <Batch>
      <Statements>
        <StmtSimple StatementCompId="2" StatementEstRows="308.313" StatementId="1" StatementOptmLevel="FULL" StatementOptmEarlyAbortReason="TimeOut" CardinalityEstimationModelVersion="140" StatementSubTreeCost="0.999363" StatementText="SELECT 1" SecurityPolicyApplied="false">
          <StatementSetOptions ANSI_NULLS="true" ANSI_PADDING="true" ANSI_WARNINGS="true" ARITHABORT="true" CONCAT_NULL_YIELDS_NULL="true" NUMERIC_ROUNDABORT="false" QUOTED_IDENTIFIER="true" />
          <QueryPlan NonParallelPlanReason="CouldNotGenerateValidParallelPlan" CachedPlanSize="480" CompileTime="142" CompileCPU="126" CompileMemory="5016">
            <InternalInfo LockClassNoHint="0" LockClassIntLockHint="0" LockClassRCIsoHint="0">
              <EnvColl>
                <Recompile>
                  <Field FieldName="wszDb" FieldValue="T8666" />
                  <Field FieldName="wszSchema" FieldValue="sys" />
                  <Field FieldName="wszTable" FieldValue="sysclsobjs" />
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000002_00000040" />
                  </ModTrackingInfo>
                </Recompile>
                <Recompile>
                  <Field FieldName="wszDb" FieldValue="T8666" />
                  <Field FieldName="wszSchema" FieldValue="sys" />
                  <Field FieldName="wszTable" FieldValue="sysnsobjs" />
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000002_0000002C" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000004_0000002C" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                </Recompile>
                <Recompile>
                  <Field FieldName="wszDb" FieldValue="T8666" />
                  <Field FieldName="wszSchema" FieldValue="sys" />
                  <Field FieldName="wszTable" FieldValue="sysscalartypes" />
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc1" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc1" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc2" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc1" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc2" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000004_00000032" />
                  </ModTrackingInfo>
                </Recompile>
                <Recompile>
                  <Field FieldName="wszDb" FieldValue="mssqlsystemresource" />
                  <Field FieldName="wszSchema" FieldValue="sys" />
                  <Field FieldName="wszTable" FieldValue="syscolpars" />
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000002_00000029" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000006_00000029_readonly_database_statistics" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_0000000B_00000029" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000005_00000029_readonly_database_statistics" />
                  </ModTrackingInfo>
                </Recompile>
                <Recompile>
                  <Field FieldName="wszDb" FieldValue="T8666" />
                  <Field FieldName="wszSchema" FieldValue="sys" />
                  <Field FieldName="wszTable" FieldValue="syscolpars" />
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000002_00000029" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000006_00000029" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_0000000D_00000029" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_0000000B_00000029" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000005_00000029" />
                  </ModTrackingInfo>
                </Recompile>
                <Recompile>
                  <Field FieldName="wszDb" FieldValue="T8666" />
                  <Field FieldName="wszSchema" FieldValue="sys" />
                  <Field FieldName="wszTable" FieldValue="sysschobjs" />
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc1" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="clst" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc1" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc2" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc3" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000005_00000022" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000006_00000022" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="nc3" />
                  </ModTrackingInfo>
                  <ModTrackingInfo>
                    <Field FieldName="wszStatName" FieldValue="_WA_Sys_00000008_00000022" />
                  </ModTrackingInfo>
                </Recompile>
              </EnvColl>
              <RX IsRXPlan="0" />
            </InternalInfo>
          </QueryPlan>
        </StmtSimple>
      </Statements>
    </Batch>
  </BatchSequence>
</ShowPlanXML>';


So here's the query...

;WITH XMLNAMESPACES('http://schemas.microsoft.com/sqlserver/2004/07/showplan' AS p)
SELECT DISTINCT plan_id, 
                DbName.DbName_Node.value('@FieldValue','NVarChar(128)')              db,
                SchemaName.SchemaName_Node.value('@FieldValue','NVarChar(128)')      [schema],
                TableName.TableName_Node.value('@FieldValue','NVarChar(128)')        tbl,
                StatsName_Node.value('@FieldValue','NVarChar(128)')                  sts
FROM (SELECT plan_id, test_plan from test_plan_XML) qsp
CROSS APPLY qsp.test_plan.nodes(N'//p:InternalInfo/p:EnvColl/p:Recompile')           AS Recomp(Recomp_node)
CROSS APPLY Recomp.Recomp_Node.nodes(N'p:Field[@FieldName="wszDb"]')                 AS DbName(DbName_Node)
CROSS APPLY Recomp.Recomp_Node.nodes(N'p:Field[@FieldName="wszSchema"]')             AS SchemaName(SchemaName_Node)
CROSS APPLY Recomp.Recomp_Node.nodes(N'p:Field[@FieldName="wszTable"]')              AS TableName(TableName_Node)
CROSS APPLY Recomp.Recomp_Node.nodes(N'p:ModTrackingInfo')                           AS [Table](Table_Node)
CROSS APPLY Table_Node.nodes(N'p:Field[@FieldName="wszStatName"]')                   AS [Stats](StatsName_Node)
OPTION (MAXDOP 1);

And... it works!



Its fairly efficient, too.  The results from "set statistics time on" for this test case are below. If parsing more plans, certainly the CPU time and elapsed time will increase.

I've got an upcoming workshop at TugaIT in July 2018... getting the most bang out of your buck for administrative tasks like stats updates is a topic on the docket!

TugaIT Summer 2018 Edition: Insight-Based Administration
http://sql-sasquatch.blogspot.com/2018/07/tugait-summer-2018-edition-insight.html