Re: HPUX 11.31 -> Disabling Pseudo-Swap?? (2223 Views)
Reply
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 1 of 26 (2,209 Views)
Accepted Solution

HPUX 11.31 -> Disabling Pseudo-Swap??

I've been reseaching on how to control this pseudo-swap on hpux 11.31. I don't want to turn it off has it seems to be necessary for proper system functionning... (as per http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1229067)

But here's the issue I'm having:

I've got 2 swap disk devices:
[root@NCRCI]:/$ swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 10485760 0 10485760 0% 0 - 1 /dev/vg00/lvol2
dev 27262976 0 27262976 0% 0 - 1 /dev/vg01/swap
reserve - 37586160 -37586160
memory 47841364 7391740 40449624 15%

One is 10G, the other 26G. As you can see, my server has 48G of system memory total.

The server is used to only serve a Sybase Database. Problem is that pseudo-swap sometimes goes to 7Gigs, sometimes to 10Gigs, etc... depending on what's running. (i.e. is sybase started or not).

The DBA's are trying to force a memory allocation of 38Gigs straight to Sybase when starting it... Obviously, if 48Gigs - current OS memory used + pseudo-swap is lower then when they're trying to allocate, then the startup of sybase fails!

1) what can I do to limit the amount pseudo-swap... if it's even possible?

2) If they already started Sybase.. then stop it for whatever reasons.... after a while, pseudo will increase by itself. When they try to start Sybase again with the same settings as previous startup, it will fail because pseudo is taking to much... How can I "reduce it on the fly" ??

Many thanks for clarifying this... :)

Patrick.



Honored Contributor
Jeff Schussele
Posts: 6,795
Registered: ‎02-18-2002
Message 2 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Hey PR (Can I call you that?),

What you're seeing is "runaway" processes in my mind.
IF an app takes memory - by God it should give *all* of it up when it dies.
Unfortunately a lot don't.

Don't *ever* let DBAs tell you how to admin your servers. Just politely tell them to keep their tables in order...;~))

Processes that suck up large amounts of memory & then don't release them properly are the #1 bane of all SAs - PERIOD!!!

There are a lot of kernel monitors out there that will verify these conditions.

Best Rgds,
Jeff
PERSEVERANCE -- Remember, whatever does not kill you only makes you stronger!
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 3 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Haahhh!!! I like how you think Jeff!!! :o)

So basically, there's not a damn thing I can do about it... is what you're saying? And when such a thing happens (i.e. pseudo going up after Sybase was stopped... then sybase not starting), my only option is really to kick the box down to release it.... right?

Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 4 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Also, what would happen if I increased the physical swap? That IS a possibility for me... I mean... shouldn't swap be equal to the same amount as the physical memory? That's how I had learned it in the past... no?

Thanks again. P.
Honored Contributor
Jeff Schussele
Posts: 6,795
Registered: ‎02-18-2002
Message 5 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Hi Patrick,

There is only onen off pseudo-swap.
Set memswap_on to off in kernel parms & reboot.
But frankly, I'd never do that - it tends to come handy...

HTH,
Jeff
PERSEVERANCE -- Remember, whatever does not kill you only makes you stronger!
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 6 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

You mean the swapmem_on boolean right? That's obsolete in V3. No longer available...

Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 7 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

No, your system probably has 55Gb or so. v3 doesn't give per-page metadata to the reservable memory layer, and Memory swap by default should be 90% of the memory given to the reservable memory layer.

"Obviously, if 48Gigs - current OS memory used + pseudo-swap is lower then when they're trying to allocate, then the startup of sybase fails!"

Ok... this isn't obvious to me. Can you elaborate on what you mean here?

First thing to question -- are you talking about a 38Gb *virtual* allocation or is this an allocation which is to be created and faulted/locked in memory? If you're failing to create a 38Gb virtual object -- that is not related to free memory [at all], but rather this is all about swap reservation.

So what may be happening is that you've only got 36Gb of device swap. 158Mb of that has not yet been consumed by reservations. 15% of pseudo-swap has been reserved (could be from User space, could be implicitly by memory locked pages [kernel dynamic is considered such]).

So your remaining reservable swap is 38.5757Gb + 158Mb. I will note that there's a known issue that HP-UX won't properly allow swap reservation that crosses memory swap *and* non-memory swap... so that means you're limited to the larger of what's available for a single virtual object. In this case, 38.5757Gb is the largest single object you could make at that instant.

If what you're trying to say is that if the process of bringing up Sybase (or other processes concurrent) consume the last 158Mb of device swap and more than .5757Gb of memory swap your 38Gb single virtual allocation fails -- yes, yes it will.

As an aside, if you're talking mlock(2) in addition, things get more complex since there are quotas on lockable memory as well that would come into play -- let's leave that alone since it sounds like just a swap problem.

1) It is not possible without treading into undocumented and unsupported methods. Obviously, I'm not going to elaborate.

More to the point -- why do you think lowering the amount of memory swap would help you? As noted above -- only the current available memory swap in your swapinfo output would have a chance to get your object built. You need more memory swap (more RAM, reduce other load) -- not less.

2) Yes, that is likely kernel dynamic memory allocations coming into play. You could reduce it by causing memory pressure (forcing the kernel garbage collector to kick in) or possibly by temporarily lowering filecache_max / filecache_min to force pages and some metadata out of the File Cache, reducing the kernel allocations again.

In practice, though -- it would be much simpler if you added device swap -- when/if Sybase succeeds in coming up, your output implies your system would be running very close to 100% swap reservation. Which means you can easily start getting ENOMEM from such system calls as fork(2), exec(2), malloc, mmap, etc.... the difficulty here is that it may be impossible/very difficult to log in to fix this later if the system gets into this state, or to run the commands required even if you can log in. Adding RAM would work as well -- but in this case, it looks like just swap is all you need.
Frequent Advisor
David P Lavoie
Posts: 33
Registered: ‎10-06-2008
Message 8 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Hello all,

Me and Pat are coworkers and we're currently working on that issue. We're going live in a couple of days and the problem explained by Pat happened yesterday. I think our main question would be :

Sybase loads with a mlock of 35 GB. We want to know if there's a way to lock the system so he doesn't user more than 10 GB. Is it possible? If yes, how.

All values were set to default in Kernel.

Thanks all for your input.
Acclaimed Contributor
Dennis Handly
Posts: 25,089
Registered: ‎03-06-2006
Message 9 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

>I've got 2 swap disk devices: $ swapinfo

Can you please provide the "swapinfo -tam" output when the system is busy.

You have a lot of swap reserved but only 7 Gb of memory used.

>1) what can I do to limit the amount pseudo-swap?

Why? Use of pseudo-swap is a good thing. The only way to reduce it is to cut down on the load.
Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 10 of 26 (2,223 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Thanks -- knowing this is a lockable memory issue instead of a swap issue (per se -- obviously locked memory tries to reserve from memory swap) is helpful data.

If you want to limit Sybase to be able to mlock no more than 10Gb, you should probably look into Process Resource Management (or encapsulating products like WLM/gWLM). That would enable you to put Sybase (and related processes) into a memory silo [as it were] with restricted levels. (Or just configure Sybase to not eat more than 10Gb... surely this is a configuration option at the application level as well?)

Otherwise -- to restrict all of user space lockable memory on v3, you're back in undocumented/unsupported land. PRM is where you want to go.
Honored Contributor
Emil Velez
Posts: 1,450
Registered: ‎05-17-2000
Message 11 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

pseudo swap does 1 thing.. It allows you to have less physical swap than you have memory and it allows you to use all of your memory before you use swap space.

Once all of your physical memory is used then your system starts paging.

Do not pay any attention to the memory line it is not important. The important thing is how much free memory you have. If you get low on free memory get more or shrink your filecache_max.

normally when a process starts it reserves swap in case memory gets low and it needs to page out. Since you have 38 GB of memory but only 20 GB of swap you could only start 20 GB of processes before the OS would say you cannot start a process. Pseudo swap makes the OS think it has more swap. When you fill up memory and need real swap it will use real swap and then you need more memory since you should not be swapping.

That is it.

Frequent Advisor
David P Lavoie
Posts: 33
Registered: ‎10-06-2008
Message 12 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 10240 0 10240 0% 0 - 1 /dev/vg00/lvol2
dev 26624 0 26624 0% 0 - 1 /dev/vg01/swap
reserve - 525 -525
memory 46720 8864 37856 19%
total 83584 9389 74195 11% - 0 -
Acclaimed Contributor
Dennis Handly
Posts: 25,089
Registered: ‎03-06-2006
Message 13 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

>reserve - 525 -525
>memory 46720 8864 37856 19%
>total 83584 9389 74195 11%

This shows you aren't using device swap and almost everything is in memory. You aren't under memory pressure. Can you do this when you have Sybase running?
Frequent Advisor
David P Lavoie
Posts: 33
Registered: ‎10-06-2008
Message 14 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

reserve - 36843 -36843
memory 46720 7389 39331 16%
total 83584 44236 39348 53% - 0 -

This is when Sybase is running. Still not using swap.
Frequent Advisor
David P Lavoie
Posts: 33
Registered: ‎10-06-2008
Message 15 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Forgot to tell you that in this case, the mlock was probably disabled or not very high. When the problem occured, they were trying to lock 37 GB as usual and it failed.
Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 16 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

As mentioned above, mlock'd memory requires memory swap on v3. Hence even your idle amount above (37856 or 36.9Gb) is insufficient to mlock 37Gb.

You have only three options:

1) Figure out what else is using memory swap (around 7Gb) and reduce that. This is almost certainly kernel consumption [implicitly mlock'd], which means reducing filecache_min / filecache_max might do it -- or creating temporary memory pressure to cause Garbage Collection to run to try to flush the kernel is about the only real option. mlock is not an unconditional call, so the system will not aggressively garbage collect in the hopes of meeting a large lock request -- instead the request is just failed as in this case.

Another possibility would be to configure large amounts of memory as Ejectable if this is a vPar (or if you have this as several Cells in a nPar). The kernel does not consume Ejectable in most cases in v3 -- so you'd be implicitly limiting the kernel. I wouldn't recommend this from what I've seen though -- if the mlock succeeds, it looks like very little spare room exists on the system, so you'd be risking kernel memory exhaustion [pingable hang].

2) Add more RAM, therefore giving more memory swap / lockable memory. You probably only need a couple Gb here, you're just at the boundary.

3) As mentioned, use gWLM/WLM/PRM or Sybase configuration to reduce the mlock request to something the system can more feasibly handle.
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 17 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Hey Don.

This might be obvious to you, but not me... a few questions regarding what you said (sorry if this sounds like Unix 101):

=====
1. "Figure out what else is using memory swap..": how would I figure out what's using swap memory here? I have seen howtos like the following to list memory usage by process.. but not sure how to properly interpret this:

UNIX95=1 ps -efo vsz,sz,pid,args | grep -v grep | sort -rnk 1

which will give me an output like :

VSZ SZ PID COMMAND
93312 22304 29794 /sybase/ASE-15_0/bin/dataserver -d/dev/sybase/rmaster -e/sybase/ASE-15_0/install/pptci.log -c/sybase/ASE-15_0/PPTCI.cfg -M/syba
93216 22024 29803 /sybase/ASE-15_0/bin/dataserver -ONLINE:3,0,0x7440a5a2, c000000100000000, 0x8d24d0000, 0x1159
93216 22024 29802 /sybase/ASE-15_0/bin/dataserver -ONLINE:2,0,0x7440a5a2, c000000100000000, 0x8d24d0000, 0x1159
93216 22024 29801 /sybase/ASE-15_0/bin/dataserver -ONLINE:1,0,0x7440a5a2, c000000100000000, 0x8d24d0000, 0x1159
66256 16564 1884 /opt/perf/bin/midaemon
55360 13768 1689 /opt/wbem/lbin/cimprovagt 0 5 10 root SFMProviderModule
46612 3640 1987 /opt/OV/lbin/perf/coda
14196 3524 1687 cimservermain --executor-socket 5
12236 2389 1711 postgres: sysadmin cer [local] idle

=====
2. "reducing filecache_min / filecache_max...": that's where I have a hard time with tunables. Most often, I rather keep the defaults.. unless some particular software (i.e. oracle, sybase, etc.) will implicitly tell me to set a particular value. I just don't know how one can tell what's an "appropriate" value... I'm no engineer here. Just a real junior unix admin... :(

=====
3. "creating temporary memory pressure to cause Garbage Collection to run to try to flush the kernel...": that would force some kind of garbage collection.. freeing up memory swap consumption? That's what you're saying right? How would I do that?

=====
4. "you're just at the boundary...": I had mentioned the fact that more memory would be necessary to be sure to always be able to lock 37G, but if that's not an option, how can I put numbers on this boundary? 30G? 32? 35.. ? How can I ensure the DBAs that mlocking 32G will always work (considering we're not using the server for anything else). What number should I give them?How can I get this number?

So many thanks! We're learning a lot here as we go and even if my head feels like it's gonna blow.... we appreciate so much all responses.

Patrick
Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 18 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

1) pstat is your friend (ps wraps pstat -- but doesn't tell you all the raw API will tell you).

Attached is a program (cc +DD32 -D_PSTAT64 -o object_dump object_dump.c) which will report locked pages consumed for either all processes on the system, a particular PID or a particular GID. That said -- I'd still expect most/all of your memory swap consumption before disk reservation exhaustion to be kernel memory allocations. Non-kernel virtual allocations will always try to reserve from disk/FS swap before memory swap, the kernel always reserves memory swap when it allocates since it is considered memory locked.

2) In this case -- I'm talking about a temporary lowering to force a flush (and release kernel metadata). Set both lower, watch for vhand running (or monitor with Glance/kcusage to see when you get down to the new max) and then you can raise them back. This is just an easy way to kick stuff out of the File Cache. Since the defaults are 5% min, 50% max -- setting max temporarily to min (5%) is likely enough.

3) Create a user application which allocates and mlocks X Gb in order to consume free memory, preferably in small chunks to allow for temporarily low memory swap. I've got something I played around with like that a year or so back -- I'll attach that in another message since multiple attachments isn't allowed here.

And before anyone else asks -- yes, more proactively releasing kernel memory or providing an interface/command to flush kernel caching as much as possible is a known enhancement point.

4) The FREE entry for the memory line from swapinfo -atm is your raw boundary. No single mlock request for a new object will be allowed more than that amount (and 37Gb is just above it). I wouldn't mlock more than 36Gb, personally -- leaving the kernel 1Gb of memory swap for new allocations just in case.

You can't give the DBAs a single raw number unless they/you can agree that the idle state before the DB comes up is relatively consistent. Since you do say this is only used for this purpose but originally you stated that memory swap consumption increases while Sybase is down (causing a problem on restart), what I strongly suspect is that something like a backup or other activity is taking place during the Sybase downtime. Said activity is likely then causing more file I/O, increasing the file cache and the metadata associated with it. If that's the case -- you could work with Sybase to understand if it needs the file cache (raw I/O is often done under the databases) in which case, lowering the filecache tunables either before running Sybase or just in general (if the only other real activity is backup -- that tends to not really need the cache even though it can fill it, read-once/write-once activity and all).

Alternately, you could script a tunable change to flush or use the memory hog to regain a known amount of memory, then release it and start Sybase [with a dedicated box, there's little chance of something snitching what you just released].

A better alternative would be if Sybase can be configured to ask for the 37Gb (fine and dandy) but mlock in smaller increments such that it handles a failure better. (It seems overly optimistic to me to be so "all or nothing" here... certainly mlock'ing 1Gb at a time and just noting that performance might be lowered if it didn't get the last 1 to 2 Gb would make more sense to me...)
Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 19 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Second program (mem_grab.c based on Scott Robertson's memdetails.c) as mentioned. [No points to this response, please].
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 20 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Thanks Don! I'll try to assimilate all this info and look more at what you've given me. I'll bring these up with my colleague...

Thanks so much and I will update this thread as we go :)
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 21 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Don, I'm trying to get your object_dump going, but I can't see how to get a complete listing of all PIDs, or just one. I've tried with 'object_dump v', but it's always giving the same thing :

On system being idle (no Sybase running):

VIRT/PHYS/LOCKED/SWAP summaries in pages.
System page size is 4096 or 0x1000 bytes.
Total physical memory is bfde85 pages (49118 Mb).
Maximum allocatable User memory is b01b9a pages (45083 Mb).
System Wide Mb in Use:
Virt: 8131 Phys: 1044 Swap: 467

and when Sybase is running:

VIRT/PHYS/LOCKED/SWAP summaries in pages.
System page size is 4096 or 0x1000 bytes.
Total physical memory is bfde85 pages (49118 Mb).
Maximum allocatable User memory is b01b9b pages (45083 Mb).
System Wide Mb in Use:
Virt: 192827 Phys: 23315 Swap: 181281

I tried looking at the source code to see what arguments it needed, but it's not working?? Cna you clarify?

Thanks again.
Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 22 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

"object_dump -v" (the dash is important).
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 23 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

Okay. I seem to be getting a few things here.

When we mlock, it doesn't actually "lock" the memory. It's more that it's going to reserve swap, which would prevent other apps from using memory that can't reserve it's swap for... It first does so by reserving all it can in the physical swap devices, then will go to the pseudo pool if there's not enough in the physical.

I'm still waiting on HP to call me back and explain a few things though...

From my findings (trial & error), I quote myself:

"Basically, we need to understand why is it that if I've got only one 10Gb swap device with 48 physical RAM, I'm able to mlock 36Gbs, but if I add another 26Gb swap device (bringing me to 36Gb total), I am not able to mlock 38Gbs (just a mear 2Gbs more). I went crazy brought my physical swap to 48Gbs instead, and then I was able to mlock more then 38 (limit was not approx 43gbs)". See attachment for details.

I don't get it!

Yet another important piece of information dmesg gave us that we totally missed:

Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 50297364 Kbytes, lockable: 37945556 Kbytes, available: 43186360 Kbytes

It says right there! 36Gbs is the lockable memory (which was our initial limit before me increasing the swap to 48Gbs... even though increasing it an extra 26G didn't work).

I'm just looking to understand a bit more here. I'll probably end telling our DBAs that they can't lock more then 35Gbs. Bottom line! Explaining that it's not because we've got 48Gbs of ram that we need to max it out! (it's plain stupid!)

But this is more for our own knowledge... to also try to understand at what point would it start paging out (if it would) if we were to lock that 43Gbs...

Thanks, P.

Acclaimed Contributor
Dennis Handly
Posts: 25,089
Registered: ‎03-06-2006
Message 24 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

>When we mlock, it doesn't actually "lock" the memory.

The man page says it locks it in memory, so adding swap doesn't help:
The mlock() system call allows the calling process to lock a segment of the process virtual address space into memory. Any addressable segment of the process' address space may be locked. Locked segments are immune to all routine swapping.
Regular Advisor
PatRoy
Posts: 160
Registered: ‎08-15-2006
Message 25 of 26 (2,207 Views)

Re: HPUX 11.31 -> Disabling Pseudo-Swap??

>The mlock() system call allows the calling process to lock a segment of the process virtual address space...

Isn't talking about virtual memory here? If so, then isn't VM by definition: "The entire memory space available to the Unix Kernel. This consists of both the physical memory and all swap space."

If that's the case, then, in theory, it just seems normal that adding swap device would help. No?
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.