Tuesday, April 9, 2013

late night AIX diagnostics

At 2:30 in the morning, I'll use this stuff to gather 2 hours worth of information.  Because I don't like it when database cache cycles in and out of paging space, and I don't like it when databases are punished for exceeding queue_depth.  And I'll do anything I can to make sure that performance killers like that get addressed.

#if pinned isn't near max numperm_global = 0 may not prevent paging
echo "d=1; while (( \$d < 481 )) do date >> /tmp/vmstat_v_20130410.out; vmstat -v | grep -e pin -e num -e blocked | grep -v ' 0 ' >> /tmp/vmstat_v_20130410.out; sleep 15 ; (( d=\$d+1 )); done" > /tmp/vmstat_v.ksh
chmod 777 /tmp/vmstat_v.ksh
nohup /tmp/vmstat_v.ksh &

#check paging for 4K pages
nohup vmstat -P 4K -t 15 480 >> /tmp/vmstat_4k_20130410.out &

#check paging for 64K pages
nohup vmstat -P 64K -t 15 480  >> /tmp/vmstat_64k_20130410.out &

#check CPU
nohup vmstat -t 15 480  >> /tmp/vmstat_20130410.out &

#check QFULL conditions, IO latency, IO size
nohup iostat -DlRT 15 480 >> /tmp/iostat_20130410.out &

#check for hotspots
nohup lvmstat -l data01 -c100 15 480 >> /tmp/lvmstat_lvdb01_20130410.out &
nohup lvmstat -l index01 -c40 15 480 >> /tmp/lvmstat_lvix01_20130410.out &

1 comment:

  1. That worked ok... much better after I scheduled collection with the 'at' command. And, I didn't have sufficient privileges for the lvmstat command. Privileges for lvmstat, svmon, kdb (to get memory pool sizes and free pages), fileplace, filemon... these are on my wish list.

    ReplyDelete