Rare heap corruption (254 Views)
Reply
Regular Advisor
blackwater
Posts: 129
Registered: ‎05-14-2009
Message 1 of 5 (254 Views)
Accepted Solution

Rare heap corruption

[ Edited ]

We have coredumps from time to time on one of our customer's servers. It seems to be caused by a heap corruption. On our servers we  have not managed to reproduce this problem so far. What approached (besides analyzing corefiles) can be used on a customer server to localize the problem?

 

I thoght that it could be librtc and tested the program with these settings:

>more rtcconfig
set heap-check off
set heap-check free on
set heap-check leaks off
set heap-check bounds on
abort_on_bounds=1
abort_on_bad_free=1

but using librtc results in significant performance decrease and I am not sure I can recomend to install librtc on a real customer server.

 

What is HP recomendation for localizing problems in this situations?

Please use plain text.
Acclaimed Contributor
Dennis Handly
Posts: 24,881
Registered: ‎03-06-2006
Message 2 of 5 (222 Views)

Re: Rare heap corruption

>It seems to be caused by a heap corruption.

 

I would hope you have figured out it is more than just "seems" and be more positive.  If it isn't heap corruption, then using librtc won't help.

 

>What approached (besides analyzing corefiles) can be used on a customer server to localize the problem?

 

Have you learned anything from the corefiles?

 

>I thought that it could be librtc and tested the program with these settings:

>more rtcconfig
>set heap-check off

 

Any reason you have off here?

Make sure "show heap-check" shows it is on.  Though this file may not be the same syntax as gdb's.


>using librtc results in significant performance decrease and I am not sure I can recommend to install librtc on a real customer server.  What is HP recommendation for localizing problems in this situations?

 

librtc is the only automated way of checking.  Really nothing else to recommend except to find patterns in the aborts you have.

Please use plain text.
Occasional Advisor
nitinjain
Posts: 5
Registered: ‎05-29-2009
Message 3 of 5 (220 Views)

Re: Seldom heap corruption

Definitely there will be performance impact.

The best you can do is run the application in Batch mode of librtc, you can use nudge feature to get the memory reports at any point of time by just sending a signal to the process.

You can obtain the reports at multiple times during the same application run.

 

It might be possible that the corruption happens but the crash is rare, such corruption cases could be found.

 

Even if the application crashes before you could obtain memory report, you can still obtain the corruption report from the core file as the process that dumped core had librtc preloaded.

 

You need not explicitly turn on/off individual rtc parameters you can just do "set heap-check on". It by default enables the required parameters.

 

But if the corruption itself happens rarely, they will be difficult to track down.

Please use plain text.
Regular Advisor
blackwater
Posts: 129
Registered: ‎05-14-2009
Message 4 of 5 (197 Views)

Re: Rare heap corruption

[ Edited ]

>>I thought that it could be librtc and tested the program with these settings:

>>more rtcconfig
>>set heap-check off

 

>Any reason you have off here?

 

Dennis, yes, there is a reason. I am interested in librtc reports about memory corruption and not interested about memory leaks and memory consumption. When "set heap-check off" I only get *.mem files and I don't get *.heap files.

 

 

Please use plain text.
Regular Advisor
blackwater
Posts: 129
Registered: ‎05-14-2009
Message 5 of 5 (196 Views)

Re: Seldom heap corruption

[ Edited ]

nitinjain,

 

> you can use nudge feature to get the memory reports at any point of time by just sending a signal to the process

Please explain how to get a memory report when a process run with LD_PRELOAD=librts.so by sending a signal?
I could not find it in the HP wd 6.1 documention.

Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation