04-22-2007 07:37 PM
We got a machine with HP 11.23 upgraded with more memory and additional CPU. Although, the performance of all progs decreased over 15%. This is strange, as the server has more memory and CPU now. I was researching about any kernel parameters that must be updated, but I am not very experienced with kernel tuning. Can you imagine what could be wrong?
Solved! Go to Solution.
04-22-2007 08:00 PM
What model of HP server is this? How many CPUs did it have before and after, how much memory before an after?
What is the application, how are you determining that it is performing 15% slower?
How did you reach the conclusion that more memory/CPUs is what you needed to speed up your application?
04-22-2007 08:17 PM
This is an rx4640 with HP-UX 11.23
It has 4 CPU's and 32 GB memory. It is running primarely an SAP-BW instances. But I also noticed that perl-data collectors, that I wrote became slower. We needed more memory, because the SAP was eating it all (16GB) and the system became unstable (process deactivating and swaping too much). Now it works fine, but slower. For a db-load we needed 13hrs before upgrade and for the same amount of data we need more than 16 hrs now.
04-22-2007 08:37 PM
Shalom, remember I/O is the primary bottleneck on most systems these days. How fast disk writes are happening may have nothing to do with CPU and memory. It may have to do with the drive speed. Some Itanium systems have 10K SCSI instead of 15 K SCSI that was the standard in PA-RISC systems. This slows things down.
This doc will help you with tuning for performance.
Owner of ISN Corporation
04-22-2007 09:14 PM
If you've added memory and some of that is free you should belooking to increase the size of the Oracle SGA, and also keep a check on the size of your buffer cache. What size do you have dbc_min_pct and dbc_max_pct set to?
04-22-2007 09:46 PM
dbc_min_pct 5 Default Immed
dbc_max_pct 5 5 Immed
Why should more I/O appear after CPU and memory came, the applications are the same.
04-22-2007 09:55 PM
Please give the outputs of
# sysdef|grep dbc
It wiil give the output of
You might get more expert opinions based on that.
SEP, I am nto able to see the link. I am getting always a "System unavailable" message.
04-22-2007 10:05 PM
The output is in the attached txt file.
SEP: I also cannot see the link, it seems that the document was renamed or deleted.
04-22-2007 10:20 PM
It's not a great analogy, but think of your system as a water supply system.
Memory is the resevoir, CPU are the pumps, disk is the pipes out to consumers.
Now if the pipes were your bottleneck anyway, then adding more pumps and creating a bigger resevoir could actually make thinsg *worse*.
Like I said, not a good analogy! Do you have any disk IO metrics which might prove/disprove this theory? A simple sar -d 10 10 during the job thats taking lonegr might be a good starting point...
04-22-2007 11:53 PM
13:21:01 c2t0d0 1.60 0.50 3 26 0.00 8.76
c2t1d0 1.00 0.50 2 23 0.00 5.58
c4t0d0 87.20 0.62 620 32503 0.09 4.04
13:21:11 c2t0d0 1.90 0.50 3 34 0.00 11.69
c2t1d0 1.00 0.50 2 31 0.00 7.67
c4t0d0 88.51 0.62 551 29445 0.08 4.44
13:21:21 c2t0d0 0.90 0.50 2 23 0.00 8.57
c2t1d0 0.60 0.50 1 21 0.00 6.03
c4t0d0 91.00 0.58 526 28868 0.05 4.60
13:21:31 c2t0d0 2.20 0.50 4 36 0.00 7.82
c2t1d0 1.10 0.50 3 31 0.00 4.70
c4t0d0 88.89 0.59 505 28306 0.06 4.48
13:21:41 c2t0d0 1.30 0.50 5 68 0.00 3.54
c2t1d0 0.60 0.50 1 16 0.00 5.56
c4t0d0 88.61 0.69 423 24461 0.20 5.25
Average c2t0d0 3.14 2.73 7 305 7.52 12.78
Average c2t1d0 1.73 3.17 5 63 7.39 10.76
Average c4t0d0 88.86 41.38 558 28633 34.68 4.56
This is when the SAP-Data-Load-application is running. I see the bottleneck here.
I have old data collected, but only for memory and load, not for diskIO, but I will watch the system for some days to see what is happening.
Thank you for your answers.
04-23-2007 12:12 AM
You added CPU and memory, but did you also tune the database and storage to go with it?
You appear to have a bottleneck on one disk.
As a general point, a batch process such as a data load will consume all the resource allocated, sometimes to the detriment of another part of the system.
Increasing the processing power may have incidentally caused an overload of your database writers, causing head thrashing and huge queues to build up on the disk.
04-23-2007 01:14 AM
I'm assuming that c2t0d0 and c2t1d0 are youre mirrored boot disks. They don't seem to be so busy.
What is c4t0d0 ? is is a LUN on a SAN or is it a local disk? What do you have on this disk? presumably a filesystem (pvdisplay -v /dev/dsk/c4t0d0 should show us what logical volumes if any you have on this disk).
None of these figures look particularly alarming though, so maybe we're chasing the wrong thing here. How does the SAP load work? i.e. where does it load from and to in terms of disks/filesystems?)
04-23-2007 01:29 AM
dbc_max_pct 5 5 Immed
I found setting max to 7 works 11.11.
Bill Hassell has seen some evidence of buffer cache at higher levels helping Oracle performance on cooked filesystems and HP-UX 11.23
Owner of ISN Corporation
04-23-2007 01:38 AM
you are right, c2t0d0 and c2t1d0 is the mirrored boot device and c4t0d0 is a disk array with a bunch of harddrives connected through fiber-channel cards (SAN).
The output is in the atached file.
In this case the SAP-System is reading and writing data from the SAN to the SAN, there is stored the whole data, since the DB is there. Therefore the great disk usage.
04-23-2007 02:19 AM
1) Do you have 2 or more paths to this disk or just one? Does your server just have the one FC card init it?
2) what type of disk array is it? Do you know how many and what type of disks actually provide the LUN? The thing is that in sequential IOs which we usually see during loads, reading and writing to the same disk(s) can be an issue, even if there is a disk array with cache involved.
3) If you only have one LUN, you should probably look at a fairly high scsi queue depth (number of IOs that can be sent concurrently to the disk array). Check its current value using:
scsictl -m queue_depth /dev/rdsk/c4t0d0
if its set to the default of 8 you can probably increase it - the maximum value will depend on your disk array, so consult with your SAN admin or array vendor.
Of course all of this assumes that the problem is with IO, which we can't really be sure of yet. You said you looked at sar outputs both before and after the memory and CPU were added. What did you see?
04-23-2007 02:39 AM
Server has two cards.
Disk Array is MSA30 14x73Gb 15k rpm
queue depth is 8.
Unfortunately I don't have any I/O-data before the upgrade. I have only load and memory, but not disk. Both load and memory usage decreased after upgrade..
Thank you for your support.
04-23-2007 03:14 AM
04-23-2007 04:01 AM
so you have a SmartArray card in your system. We should be able to interrogate that to determine the configuration of your storage:
ioscan -kfnd ciss
Should find us the device file for the SmartArray controller.
replacing cissX with what you get back from the ioscan, should show us how the LUN is presented to the host.
04-23-2007 05:16 AM
Also as I pointed out earlier, all your reads and write are going to these 8 disks, which seems to me to probably be pretty inefficient.
Looking at the output there appears to be another drive configured (probably presented as c4t0d1) - what is that used for? I think you need to look at doing the SAP load so that it is reading from one set of disks and writing to another.
You could try upping the scsi queue depth, although I can't find a reccomended setting for the Smart Array card. It's probably safest to up it in small increments. If you see performance gains then keep going until you see no further perf gains, or start getting device queue full messages in the syslog.
I'd start with just 16 and see how that goes:
scsictl -m queue_depth=16 /dev/rdsk/c4t0d0
I'd be surprised if this makes much difference.