Why expose memory pool stats in vmstat?

Now that I have put out the request to expose memory pool stats in vmstat* I'm digging up some system stats to support the request.

I also want to lay out the argument that trying to understand memory management behavior in AIX 6.1 and AIX 7.1 without access to memory pool (and maybe even frameset) size and free size is a futile game.

I'm primarily concerned with database servers, and I've seen unexpected paging on lots and lots of IBM Power systems for a long time.  I've seen lots and lots of APARs come out for 64k page use leading to paging, the introduction of numperm_global, and on and on.  I know that there are many administrators for IBM Power database servers running AIX that have more or less given up - from their perspective some amount of paging space use just goes along with AIX and there's no use trying to fight it.  That's too bad.

I'm always trying to eliminate paging, regardless of the OS I am concerned with at the time.

So, when I saw a new set of vmo tunables introduced in AIX 7.1, I wanted to learn as much about them as possible.

Consider the vmo tunable enhanced_affinity_private.

# vmo -h enhanced_affinity_private
Help for tunable enhanced_affinity_private:
Purpose:
Specifies percentage of process private memory allocations that are affinitized by default.
Values:
        Default: 40
        Range: 0 - 100
        Type: Dynamic
        Unit: numeric
Tuning:
This tunable limits the default amount of affinitized memory allocated for process private memory.  Affinitizing private memory may improve process performance.  However, too much affinitized memory in a vmpool can cause paging and impact overall system performance.

OK... I guess 40% is a reasonable default.  How can it be evaluated?  The only way I know of evaluating would involve memory pool monitoring.

Hmmm... in there I also see enhanced_affinity_attach_limit.  What's that?

# vmo -h enhanced_affinity_attach_limit
Help for tunable enhanced_affinity_attach_limit:
Purpose:
Specifies percentage of memory allocation requests with affinity attachments that are affinitized.
Values:
        Default: 100
        Range: 0 - 100
        Type: Dynamic
        Unit: numeric
Tuning:
This tunable limits the amount of affinitized memory allocated for memory allocations that have affinity attachments.  Affinitizing such memory allocations may improve performance.  However, too much affinitzed memory in a vmpool can cause paging and impact overall system performance.

Wait a minute!  It carries the same warning that too much affinitized memory can cause paging... but its default is 100%?  That seems like something I'll want to monitor.

But before I start digging into how to monitor the effects of enhanced_affinity_attach_limit and
enhanced_affinity_private, I run across this:

IBM Power Systems Performance Guide - Implementing and Optimizing

http://www.redbooks.ibm.com/redbooks/pdfs/sg248080.pdf

(3.9 mb pdf)

Page 281

Shared memory is not impacted by the enhanced_affinity_private parameter. This kind of memory allocation is controlled by memplace_shm_named and memplace_shm_anonymous.

I've never paid attention to memplace_shm_anonymous before.  Historically it has determined first-touch or round-robin strategy for affinitized anonymous shared memory allocations.  If I had and wanted to evaluate round-robin vs first-touch... I'd probably do it by monitoring memory pool use.  Then... what if a new value sneaks in? There it is in the AIX 6.1 documentation:

Default placement of anonymous shared memory can be set to first-touch (value of 1) or round-robin across the system (value of 2) or automatic (value of 0), where the system decides the best placement for the memory.

http://pic.dhe.ibm.com/infocenter/aix/v6r1/topic/com.ibm.aix.cmds/doc/aixcmds6/vmo.htm

Truth be told, the new setting was present in the code release before it made its way into the documentation :).

http://www-01.ibm.com/support/docview.wss?uid=isg1IZ74067 

Note that AIX 7.1 on IBM Power7 memplace_shm_anonymous defaults to value 0 - system determined memory allocation.  Well... shoot... the new value is the default?  How does the system determine what strategy it'll use for allocation?  How can I evaluate whether the new "system determined" behavior is responsible for increased paging space traffic that I notice after an upgrade to AIX 7.1?

# vmo -h memplace_shm_anonymous
Help for tunable memplace_shm_anonymous:
Purpose:
Specifies the default memory placement policy for anonymous shared memory.
Values:
        Default: 0
        Range: 0 - 2
        Type: Dynamic
        Unit:
Tuning:
Anonymous shared memory refers to working storage memory, created via shmget() or mmap(), that can be accessed only by the creating process or its descendants. This memory is not associated with a name (or key). Default placement of anonymous shared memory can be set to first-touch (value of 1) or round-robin across the system (value of 2) or auto-affinitized (value of 0), where the system decides the best placement for the memory.

I've been doing this for a while - I like to keep disruptive tuning recommendations (such as changing memory_affinity or vmm_mpsize_support values) to an absolute minimum.  I also don't want to encourage a cycle of problem detection-diagnostics-fix selection-fix apply-repeat with no known end.                               

If memory pool statistics, the fundamental stats driving lrud to steal pages, are exposed, more administrators will be able to track them.  Shoot, maybe they'll even make their way into nmon :)  At any rate, it'll shed more light on the large number of cases of unexpected page outs, and it will make possible evaluating the enhanced_affinity and memplace families of vmo tunables, as well as the behavior of ASO and DSO when used.  I don't know of another method of evaluating those tunables, ASO, or DSO.

*I'll keep saying "vmstat", but I don't really care if its in vmstat or not.  I just want memory pool size, and memory pool free list size, exposed through a tool other than kdb which doesn't require root privileges to execute.