Crowdsourcing: expose IBM AIX Memory Pool total and free frame count in vmstat

Why do page outs to disk paging space occur in AIX, even when vmstat and other tools indicate there is plenty of available server memory?

There are profiles, rules and conditions that determine which server resources end up in an LPAR.  If LPAR resources are dynamically provisioned rather than statically, then there there's the matter of which resources are added or removed over time.  There are lots and lots of rules, and plenty of kernel tunables and environment variables that influence AIX memory allocation and management from LPAR resources.  Then there's the Active System Optimizer daemon which might move stuff around within LPAR resources, the Dynamic System Optimizer (additional licensed module on top of ASO) which might move stuff around within LPAR resources, and the Dynamic Platform Optimizer which might move LPAR resources around within the server.  Put that all together with varying application activity and stress, and understanding the tide, ripple, and break of memory can be extremely complicated.

But paging out to paging space is actually remarkably simple... the magic numbers have just been HIDDEN from you.  And I am hoping that by crowdsourcing some support... grass roots... I can persuade Big Blue to just SHOW us the doggone numbers that make page out operations plain as day.

Here's the deal:  There is one lrud daemon thread for each LPAR memory pool.  The lrud daemon wakes up periodically, and if the number of free frames in the pool is below the low water threshold, page stealing starts.  By default lrud walks its list of memory frames.  Clean filesystem buffer pages can simply be expired and added to the free list.  Computational pages which were paged out once before... paged back in for a READ operation... and still have the same contents in paging space can simply be retired.  Computational pages that have never been paged out before have to be written to paging space.  Once the free frame list in the memory pool is above the high water mark, lrud work in that pool stops.  (***I'll come back later and fill in the names of relevant tunables.***)

That's remarkably simple.  The crazy thing is that there is NO easy way to see the size of the LPAR memory pools, or to see how many free frames there are in each memory pool.  And the only documented way is problematic for two reasons: it requires root privileges, and it uses the kernel debugger.

As someone that often evaluates performance and behavior on other administrators' systems, those are two heavy strikes. I don't want to inconvenience the admins by asking them to run monitors I cobble together... and I don't like asking them to run monitors as root.  And asking them to run a monitor which calls into the kernel debugger?  Doesn't that sound risky???!??

Okay... so maybe this sounds crazy to you.  After all... you got your vmstat... you got your svmon... you got your topas... you got your nmon.  Why is sql_sasquatch asking for something else?  Maybe he just doesn't UNDERSTAND how this really works.

Trust me... I've got a year's worth of scars from figuring this out.

Have you ever worked with IBM's perfpmr package?  When you open a performance PMR with IBM, often you'll be asked to download the script package and take some monitor samples.  I won't post IBM's code here.  But if you've downloaded it, poke around in the scripts.  In the memory monitors you'll see commands somewhat similar to these:

echo "memp *" | kdb
echo "frs *" | kdb

Those use aliases for the mempool and frameset commands to retrieve information including size and free frames.  I like the frameset command because it returns free memory in the frameset in human readable.  My hex ain't that good :)
The lru alias for lrustate command also comes in handy when evaluating page stealing behavior.

Lots more about kdb can be found within this 358 page pdf file.
http://pic.dhe.ibm.com/infocenter/aix/v7r1/topic/com.ibm.aix.kdb/doc/kdb/kdb_pdf.pdf

In the next week I'll post some numbers from one of my investigations that shows how kdb can illuminate paging space usage mysteries.

As much as Big Blue hesitates to agree, sometimes the best thing is to balance the memory pools in size and the number of affinitized logical processors.  That can be accomplished with provisioning server resources to the LPAR in a balanced manner... or by disabling memory_affinity (which has fallen out of favor recently but still may be the best option for large database server LPARs). 

To wrap it all up... these numbers... the size of your memory pools and the amount of free memory in the pools should be consulted when paging space is being utilized even when there seems to be sufficient server memory to avoid paging space writes. Sure, you can do it with root access and calling in to kdb.  But somehow lrud itself is getting the free frame count for the pools - I don't think its using kdb.  At least I hope its not :).  Regardless,  I really, really, REALLY want to see these numbers in a standard monitoring tool that can be used without root privileges.  The vmstat utility seems like the perfect place to put it.  Please?  Pretty please?  I'll be a good sasquatch...

:)

My blog doesn't get a lot of traffic - its not really meant to.  But if I can get some supportive comments from AIX administrators, developers, or other folks that might be persuasive... maybe we can all benefit.