Re: SYBASE database crashing, IOPS suspected. (159 Views)
Reply
Occasional Contributor
PSeleznev
Posts: 8
Registered: ‎02-04-2014
Message 1 of 3 (206 Views)

SYBASE database crashing, IOPS suspected.

Hi,

I run NNMi 9.23 latest patch with Perf for Metrics latest patch and Perf for Traffic latest patch (updates few weeks ago) on REHL 6.4.

Network size is about 400 nodes and 12k interfaces. 

Problem is following. Every morning when I come to the office SYBASE database is down. 

I started to investigate and after all I suspect that hard drive speed is not enough. 

When it runs Perf for metrics it's ok but as soon as we start Perf for Traffic SYBASE DB starts to crash every other minute. 

I started iostat test with netflow off and got following result: 

 

avg-CPU: %user %nice %system %iowait %steal %idle
51.24 0.00 31.36 10.50 0.00 6.90

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
sda       0.00 34.10 1.20 345.70 51.20 3038.40 8.91 0.66 1.89 0.51 17.52
sdb       5.70 36467.40 553.20 914.90 50902.40 290316.20 232.42 234.76 152.91 0.68 99.94
sdc       0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.10 1.50 0.80 12.00 8.00 0.00 0.56 0.31 0.05
dm-2 0.00 0.00 1.00 0.40 48.80 3.20 37.14 0.03 24.43 12.64 1.77
dm-3 0.00 0.00 0.00 377.10 0.00 3016.80 8.00 0.70 1.85 0.43 16.23
dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-5 0.00 0.00 0.00 0.80 0.00 6.40 8.00 0.00 1.00 0.88 0.07

 

Basically it says that average queue size is 234, average execution time is 152 ms and at the moment it runs about 1200 IOPS at the moment. Is it normal load? I'm not responsible for the hardware but I guess we have RAID 5.

When I start netflow on one device for one interface, number of files in shared folder starts to grow consistently. Also IQ_SYSTEM_TEMP dataspace starts to grow quickly. 

 

I talked to out technicians and they've asked me what block size SYBASE operate? I can't find this parameter anywhere. Any suggestions?

Also I would like to know where is log file for SYBASE database. I can't find exact reason why it crashes. 

Any tips on troubleshooting such behavior would be appreciated. 

Please use plain text.
Trusted Contributor
msharma
Posts: 128
Registered: ‎07-18-2011
Message 2 of 3 (164 Views)

Re: SYBASE database crashing, IOPS suspected.

[ Edited ]

Hello,

While I don't have a sybase related answer, I can however imagine a scenario where this could take place.

 

>>"Is it normal load? I'm not responsible for the hardware but I guess we have RAID 5.

When I start netflow on one device for one interface, number of files in shared folder starts to grow consistently. Also IQ_SYSTEM_TEMP dataspace starts to grow quickly."

 

I am making a few assumptions here, and please correct me if I am wrong, but when you say you start iSPI Traffic, you mean to say that you start the Traffic Master and Traffic Leaf services, right?

 

The Traffic Leaf has a setting, for each collector, to store received flows in files.

If Traffic Leaf is on the same server as iSPI Metrics, the everytime Leaf receives a netflow packet, it will store the info (open any leaf collector, check the option that says "store flow in file", this must be set to false unless you have it set to true for some diagnostic purpose).

 

Second, iSPI Traffic Master has a setting which determines the rate at which Master writes data to the NNMi Shared Folder (%NNMDataDir%shared/perfSpi/datafiles/metric/final and a couple other sub dirs).

The setting is governed by time (2-3 minutes I think), and by a number (100000 by default).

Essentially, once Master has processed 100000 records, or 2 minutes have elapsed, (whichever happens earlier) Master will flush data (in the form of a CSV GZ file) to the NNMi shared folder.

The setting for master is configurable on the Master UI at Master Settings.

If the setting is too low (100 for example), then Master will frequently write data to NNMi Shared dir, which will increase the IO load on NNMi, and since ETL (Extract-Transform-Load) processes from iSPI Metrics have to pick up this data, they will also be stressed.

 

Just a couple of theories, in the mean time, let me see if I can give you some more info on SYBASE block size.

 

 

>>"Also I would like to know where is log file for SYBASE database. I can't find exact reason why it crashes. "

 

The prspi.log would be best best, if iSPI Metrics Sybase (ASE/IQ) is crashing (%NPSDataDir%NNMPerformanceSPI\logs\). Search for strings such as FATAL (show stoppers in most cases), or SEVERE, or WARN*(WARN/WARNING) etc.

 

A little light reading from this link may also help (now or in future):

http://support.openview.hp.com/selfsolve/document/KM1149979

Mohit Sharma,
HP Software Support

The views expressed in my contributions are my own and do not necessarily reflect the views and strategy of HP.

If you find this or any post resolves your issue, please be sure to mark it as an accepted solution.
Please use plain text.
Trusted Contributor
msharma
Posts: 128
Registered: ‎07-18-2011
Message 3 of 3 (159 Views)

Re: SYBASE database crashing, IOPS suspected.

Info on Sybase block size (not an HP link):

 

http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc00169.1510/html/iqperf/X3...

Mohit Sharma,
HP Software Support

The views expressed in my contributions are my own and do not necessarily reflect the views and strategy of HP.

If you find this or any post resolves your issue, please be sure to mark it as an accepted solution.
Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation