Thursday, March 20, 2014

Perfmon was collecting stats... what happened next was...

whackadoodle!

I can't think of any other word to describe it.  Let me know if you've ever seen anything like this from perfmon on a physical windows server.

Perfmon was running locally on a 4 socket, six core per socket server with no Hyper-Threading, logging to tsv file in 15 second intervals.  The results were sent to me for review.

The 24 columns below are for "\Processor(x)\% Processor Time" for 0<=x<=23.

See the zany for yourself below. 


Duplicate times.  Missing per-cpu numbers.  Suspect per-cpu CPU busy reported. And suddenly, 10 minutes later, a return to normal.  I'm thinking major fault at CPU level or maybe a significant memory error.  So I'm hoping to dig details or at least a clue from the system or event log.  But, I'm open to any ideas... because other than that I don't have any.  And I usually have lots of ideas, even if they aren't very helpful :-)

 This isn't what I was expecting to see at all.  I thought I was just going to see some of the disk IO and memory management issues I normally deal with on SQL Server configs.  Maybe some spinlocks.  Nothing that would completely throw me for a loop.  Maybe I just need more coffee.

**** Updated with Exhibit 1 for some context.  SQL Server is expected to be the only significant consumer of CPU, memory, or IO resources from this physical Windows server.  Note the way that the two extraordinary timeperiods are the two deviations from SQL Server CPU consumption strongly correlating with "total server CPU consumption." ****



cpu0 cpu1 cpu2 cpu3 cpu4 cpu5 cpu6 cpu7 cpu8 cpu9 cpu10 cpu11 cpu12 cpu13 cpu14 cpu15 cpu16 cpu17 cpu18 cpu19 cpu20 cpu21 cpu22 cpu23
16:47:14 20 18 11 6 4 2 20 12 6 2 1 2 13 15 6 38 17 30 12 13 20 16 11 13
16:47:29 19 18 11 4 2 1 19 11 8 2 1 0 14 14 5 39 14 31 13 16 18 16 11 10
16:47:44 20 16 10 5 3 1 19 11 6 2 2 0 13 14 6 36 17 26 11 12 27 14 9 12
16:47:44 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18 18
16:47:59 21 17 9 4 2 1 22 12 5 3 1 0 13 17 5 37 15 31 17 14 23 12 6 6
16:47:59 100 46 46 46 46 46 100 46 46 46 46 46 100 100 46 46 46 46 46 46 46 46 46 46
16:48:14 22 20 11 4 2 1 7 16 8 34 21 13 12 15 6 39 19 32 13 14 26 12 8 9
16:48:14 44 100 44
16:48:29 19 19 8 6 4 1 21 14 5 1 1 0 11 14 5 39 17 29 12 14 22 13 5 7
16:48:29 46 46 0 0 0 0 0 46 0 0 0 0 0 46 0 0 46 46 46 46 0 100 0 46
16:48:44 20 19 9 5 2 1 19 12 8 4 2 1 12 14 5 40 16 29 14 10 26 13 9 11
16:48:44 44 100
16:48:59 18 17 9 6 3 1 18 15 7 3 1 0 13 14 8 35 14 22 13 12 24 17 8 8
16:48:59 44 44 100 44
16:49:14 20 18 9 6 4 2 22 13 8 3 0 2 11 12 6 37 19 27 12 13 24 14 5 9
16:49:14 46 46 46 46 46 46 46 100 100 46 46 46 46 46 46 100 46 46 46 46 46 46 46 46
16:49:29 18 16 10 5 2 1 20 12 7 2 2 0 11 14 6 38 18 29 14 12 27 15 7 10
16:49:29
16:49:44 20 15 10 5 2 1 19 9 6 2 3 0 12 14 6 39 20 25 13 13 23 14 5 9
16:49:44 0 0 0 0 0 0 0 0 0 0 0 0 46 46 0 0 0 100 0 46 100 0 0 46
16:49:59 20 14 11 7 2 1 20 15 6 1 0 0 12 14 4 40 14 29 12 14 29 17 8 9
16:49:59 44 44 44 44 44 44 100 44 100 100
16:50:14 25 17 11 5 5 3 19 15 8 3 2 1 12 15 8 37 17 28 13 14 26 16 12 11
16:50:14 46 46 100 0 0 0 100 100 0 0 0 0 0 0 0 46 46 46 100 46 0 0 0 46
16:50:29 20 18 6 6 2 1 23 13 4 2 2 0 13 14 7 38 17 28 13 13 27 13 9 9
16:50:29 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44
16:50:44 21 15 10 3 2 2 20 13 7 2 1 0 13 13 3 38 17 36 11 12 25 16 7 12
16:50:44 0 46 0 0 0 0 0 0 0 0 0 0 0 46 0 100 0 46 46 0 46 46 0 0
16:50:59 14 20 12 10 5 1 18 17 9 4 2 1 6 12 1 52 19 45 8 13 32 25 7 10
16:50:59 44 44 100 44 44 44 44 100 44 44 44 44 44 100 44 44 44 100 44 44 100 44 44 44
16:51:14 15 22 16 12 4 2 18 16 12 6 2 2 7 14 2 52 23 46 3 11 25 32 21 17
16:51:14 44 44 100 44 44 100 44 44
16:51:29 12 24 15 10 4 1 19 17 11 3 2 0 7 14 1 52 22 47 4 11 29 26 12 13
16:51:29 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 46 0 0 46 46 0 0 46
16:51:44 16 21 15 13 4 2 20 17 10 3 1 0 7 14 1 52 20 47 5 13 28 30 17 14
16:51:44 44 44 100
16:51:59 16 20 14 7 5 1 20 16 10 4 2 0 6 15 2 52 22 46 5 14 27 28 13 11
16:51:59 0 0 46 0 0 0 0 0 0 0 0 0 46 0 0 0 46 46 0 0 46 46 0 46
16:52:14 15 22 13 10 4 2 19 18 11 3 1 0 7 15 2 51 23 49 3 9 31 31 16 13
16:52:14 44 44 44 44 44 44 44 44 44 44 44 44 44 44 44 100 44 44 44 44 44 44 44 100
16:52:29 14 22 14 10 4 2 19 16 11 5 3 0 6 14 2 51 23 48 3 11 24 28 17 13
16:52:29 0 0 46 0 0 0 46 0 0 0 0 0 0 0 0 100 0 0 0 0 0 0 0 0
16:52:44 15 23 13 14 8 2 21 18 10 4 2 0 5 12 1 52 20 50 5 15 27 30 15 15
16:52:59 13 21 14 10 6 2 20 16 9 4 4 2 7 12 2 49 25 50 6 13 20 29 19 12
16:52:59 100 100 100 100 100
16:53:14 14 20 16 13 7 4 18 18 11 4 1 1 5 14 1 49 22 50 4 11 24 32 24 20
16:53:14 100 100 100 100 100
16:53:29 13 22 17 11 7 2 20 17 13 6 3 2 5 13 2 51 20 50 3 11 26 30 18 13
16:53:29 100 100 100 100 100
16:53:44 14 25 14 15 9 5 20 14 7 4 1 0 5 15 2 53 24 48 5 13 25 27 18 20
16:53:44 100 100 100 100 100 100 100
16:53:59 12 23 13 13 8 4 20 18 14 6 2 0 4 14 2 51 22 50 5 12 23 31 17 13
16:53:59 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:54:14 19 23 14 9 6 3 20 16 10 6 2 0 7 13 1 50 21 48 6 17 28 28 16 18
16:54:14 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:54:29 14 20 16 10 5 2 19 19 11 5 1 0 6 13 1 53 23 44 4 10 25 32 15 14
16:54:29 100 100 100
16:54:44 15 22 13 9 4 1 19 17 13 7 1 1 8 15 2 51 23 48 7 12 29 30 15 15
16:54:44 100 100
16:54:59 16 23 13 7 3 1 17 16 13 11 3 0 7 14 1 51 24 47 4 10 22 32 15 19
16:54:59 100 100
16:55:14 19 22 19 12 9 8 20 17 12 8 3 1 7 15 2 54 25 45 3 11 25 34 27 17
16:55:14 100 100 100
16:55:29 12 21 16 10 3 1 17 18 15 6 2 1 7 15 2 53 22 48 3 8 24 33 20 20
16:55:29 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:55:44 16 21 14 8 4 2 19 17 13 6 1 1 6 14 4 52 19 50 5 8 28 27 17 18
16:55:44 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:55:59 14 19 15 12 4 1 17 17 13 6 3 1 5 15 2 52 19 48 2 9 24 34 24 21
16:55:59 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:56:14 13 20 17 13 5 3 19 17 13 4 1 1 8 13 2 53 23 47 3 10 28 33 20 14
16:56:14 100 100 100
16:56:29 17 21 13 6 3 1 19 17 9 3 1 0 6 16 2 50 23 45 3 11 29 29 15 12
16:56:29 100 100 100 100 100
16:56:44 17 22 16 8 4 1 17 16 13 4 2 0 9 16 2 53 20 47 4 9 24 29 12 15
16:56:44 100 100
16:56:59 12 21 13 8 5 1 17 19 10 6 2 1 6 13 2 50 21 49 6 13 29 24 9 9
16:56:59 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:57:14 15 21 15 10 4 2 18 17 10 5 2 0 5 16 2 56 19 45 6 12 27 26 13 13
16:57:14 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:57:29 12 21 16 12 5 1 20 18 10 4 1 0 6 15 1 53 24 47 4 9 28 31 16 17
16:57:29 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
16:57:44 16 22 15 9 5 2 18 19 9 5 1 1 7 16 1 52 23 44 3 10 25 31 17 19
16:57:59 13 21 16 9 5 2 21 15 9 4 1 1 5 12 2 49 24 50 4 9 29 31 18 16
1:53:14 21 25 22 3 5 4 14 3 2 4 4 7 45 3 20 18 63 16 12 10 17 10 23 10
1:53:29 20 15 4 2 2 2 7 2 2 3 0 0 22 12 14 6 45 12 14 5 18 5 2 3
1:53:44 16 34 25 19 25 19 28 20 7 21 9 4 19 30 44 15 45 25 11 15 16 29 21 8
1:53:59 15 6 2 0 0 0 2 1 0 0 0 0 30 8 9 13 39 10 12 4 16 0 1 0
1:53:59 46 46 46 46 46 46 46 46 46 46 46 46 46 46 46 100 100 46 46 46 46 46 46 46
1:54:14 17 5 1 1 2 0 6 3 0 0 0 0 10 15 27 23 67 11 14 6 14 2 2 4
1:54:14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 100 0 0 0 100 0 0 100
1:54:29 14 6 2 1 0 1 4 1 0 0 0 0 38 15 12 5 28 8 19 4 10 2 1 1
1:54:29 44 44 44 44 44 44 44 44 44 44 44 44 44 100 44 44 44 44 44 44 44 44 44 44
1:54:44 10 6 1 8 1 0 2 1 0 1 0 0 28 5 10 6 33 3 9 4 16 1 0 0
1:54:44 0 0 0 0 0 0 0 0 0 0 0 0 46 0 0 0 0 0 0 0 0 0 0 0
1:54:59 24 3 5 2 7 0 3 0 0 0 0 0 27 17 16 14 28 2 14 9 14 7 3 0
1:54:59 44 100 100 100 44 44
1:55:14 17 3 12 3 22 0 1 0 0 0 0 0 28 24 18 2 18 1 11 3 16 2 2 4
1:55:14 0 0 0 0 0 0 46 0 0 0 0 0 100 0 0 0 0 0 0 0 0 0 0 0
1:55:29 11 4 2 3 4 1 5 1 4 4 2 0 37 15 16 6 29 4 10 12 19 9 11 4
1:55:29 44 100 100
1:55:44 9 2 1 0 0 0 4 1 1 5 0 0 31 9 11 1 34 0 9 2 12 7 1 1
1:55:44 100 46 46 46 46 46 46 46 46 46 46 46 46 46 100 46 46 46 100 46 46 46 46 46
1:55:59 15 7 6 5 6 1 17 3 5 8 0 0 12 30 15 7 61 16 6 16 22 46 10 12
1:55:59 100 100 100
1:56:14 20 12 21 8 15 13 18 6 30 23 4 2 19 32 28 22 52 27 21 18 26 20 22 9
1:56:14 46 0 0 0 0 0 46 0 0 0 0 0 0 46 0 0 100 0 0 46 100 0 0 46
1:56:29 24 16 19 17 15 8 16 15 10 12 2 2 10 34 23 18 61 25 9 19 33 10 15 7
1:56:29 0 0 46 0 0 0 0 100 0 0 0 0 0 0 0 0 100 100 0 46 0 100 0 0
1:56:44 26 5 8 5 6 0 15 29 5 5 1 0 17 18 31 6 58 7 13 11 16 6 0 3
1:56:44 100 2 2 2 2 2 2 2 2 2 2 2 2 2 100 2 100 2 2 2 100 2 2 2
1:56:59 22 12 21 5 6 9 7 0 0 4 0 0 7 13 52 10 62 18 15 13 14 4 3 4
1:56:59 46 46 46 46 46 46 46 46 46 46 46 46 46 46 46 46 100 46 46 100 46 46 46 46
1:57:14 26 5 7 1 0 0 9 1 0 4 0 0 10 14 17 14 62 10 10 11 16 5 3 8
1:57:14 100 44 100 100 44
1:57:29 38 16 13 3 10 2 14 7 6 11 1 0 8 18 16 8 65 38 16 15 16 3 1 2
1:57:29 46 46 46 100 46 46 46 46 46 46 46 46 46 46 46 46 100 46 46 100 100 46 46 46
1:57:44 30 10 9 2 9 0 9 4 0 8 0 0 31 13 6 7 42 2 9 14 16 1 1 2
1:57:44 44 44 100 100
1:57:59 23 26 9 10 10 12 28 2 9 4 3 2 32 7 18 6 46 20 13 20 17 12 7 4
1:57:59 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48 0 100 0 0 0 100 0 100 48
1:58:14 27 29 16 22 20 24 33 12 20 11 7 4 26 21 26 26 60 29 32 20 32 9 23 10
1:58:14 44 44 100 44 44 44 44 100 44 100 44 44 44 100 44 44 44 44 44 44 44 44 44 44
1:58:29 27 8 10 1 3 1 29 3 1 7 1 1 2 24 17 29 55 26 17 5 15 3 7 4
1:58:29 0 0 0 0 0 0 100 0 0 0 0 0 0 0 46 0 46 46 46 0 0 0 0 0
1:58:44 25 31 19 13 17 19 23 26 15 25 15 11 27 25 16 44 51 27 21 15 35 8 14 13
1:58:44 38 38 38 38 100 100 38 38 100 38 38 38 38 38 38 38 100 38 38 38 100 38 38 38
1:58:59 16 13 16 1 5 1 42 4 5 6 0 3 24 22 18 12 52 6 45 7 16 2 4 9
1:58:59 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
1:59:14 21 9 5 1 1 0 44 6 2 15 2 0 13 12 9 12 55 13 16 6 18 4 5 5
1:59:14 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
1:59:29 16 4 7 1 1 0 20 3 4 5 0 6 19 7 6 16 65 1 15 5 16 6 2 2
1:59:29 100 100
1:59:44 22 7 4 1 0 0 27 4 2 4 4 1 23 10 8 3 61 5 14 5 16 2 2 4
1:59:44 100 100
1:59:59 20 14 5 1 1 0 30 1 0 2 0 0 18 10 19 6 61 5 18 2 11 1 2 1
1:59:59 100 100 100
2:00:14 28 8 16 12 11 12 29 20 16 19 3 5 24 26 16 37 56 16 13 11 33 11 16 29
2:00:14 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:00:29 7 5 37 35 34 40 49 57 55 57 7 13 6 11 7 59 72 3 13 15 24 5 4 4
2:00:29 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:00:44 5 7 38 44 39 44 29 66 63 62 3 4 11 3 6 54 29 2 20 11 15 3 58 8
2:00:44 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:00:59 12 23 43 35 19 19 32 53 50 47 8 8 14 17 9 37 14 3 26 12 20 5 9 5
2:00:59 100 100 100 100 100 100 100
2:01:14 21 32 13 6 6 22 27 7 32 7 22 23 7 42 29 9 3 0 7 28 35 9 15 9
2:01:14 100
2:01:29 23 7 5 1 1 0 24 1 10 0 0 0 5 1 0 0 0 0 8 3 15 8 1 2
2:01:44 28 12 2 0 2 0 3 1 0 0 0 0 5 1 0 2 0 0 8 3 9 2 0 1
2:01:44 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:01:59 21 6 3 0 2 0 4 0 0 0 0 0 6 1 0 0 0 0 15 4 22 8 13 2
2:01:59 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:02:14 10 2 1 1 0 0 1 0 0 0 0 0 2 1 0 0 0 0 10 2 14 2 3 3
2:02:14 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:02:29 6 16 18 7 14 19 12 1 19 13 0 12 3 14 17 5 12 14 7 16 24 19 51 14
2:02:29
2:02:44 7 4 0 1 0 1 2 0 0 0 0 0 4 3 1 0 0 0 16 12 16 13 24 6
2:02:44 100
2:02:59 5 7 2 1 0 0 3 1 0 1 0 0 6 2 0 3 0 0 18 7 11 2 6 6
2:02:59 100
2:03:14 6 0 0 0 0 0 1 0 0 0 0 0 4 2 0 0 0 0 22 15 16 2 13 2
2:03:14 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:03:29 5 1 0 0 0 0 3 1 0 0 0 0 4 1 0 0 0 0 22 13 18 2 8 3
2:03:29 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:03:44 16 11 3 1 6 0 6 1 6 0 0 6 15 2 3 5 0 0 14 27 33 3 24 14
2:03:44 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
2:03:59 19 3 1 0 0 0 12 0 0 0 0 0 14 3 0 0 0 0 1 50 39 4 45 3
2:04:14 17 7 11 2 1 2 20 7 4 1 2 2 19 5 10 6 1 1 5 38 30 8 80 17
2:04:29 2 1 43 2 3 0 2 19 0 9 15 13 10 7 8 0 0 0 6 1 28 2 8 5
2:04:44 9 18 11 5 2 2 12 2 7 9 2 6 5 11 22 11 11 7 8 12 29 5 7 13
2:04:59 34 14 5 19 11 3 10 28 16 55 24 28 31 30 31 8 13 11 19 8 23 27 25 6
Posted by at 1 comment:
Subscribe to: Posts (Atom)