PA Risk Executable on Itanium Server (13071 Views)
Reply
Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 1 of 28 (13,071 Views)
Accepted Solution

PA Risk Executable on Itanium Server

Hi
I have an executable compiled on HP-UX11i(11.11) that I ran on an HP-UX11i(11.23) Itanium server and it crashed. Gave following error:
[HP ARIES64]: Core file for 64 bit PA-RISC application
[HP ARIES64]: saved to core..
Memory fault(coredump)

BUT, When I ran the same exec on the Itanium server by using wdb (debuuger) it runs successfully! Any ideas please?

Thank you
Rao
Acclaimed Contributor
Dennis Handly
Posts: 25,058
Registered: ‎03-06-2006
Message 2 of 28 (13,071 Views)

Re: PA-RISC Executable on Integrity server

[ Edited ]

Well, if it aborts, you should just debug from the core file. Getting a stack trace would be the first step.

I've had a few cases where it runs in the debugger. The obvious cases are uninitialized variables.
I've seen other cases where the debugger's private mapping of shlibs causes readonly strings to be writable and then it won't abort. But that should fail on PA too.

Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 3 of 28 (13,071 Views)

Re: PA Risk Executable on Itanium Server

Dennis
Tks for the reply. I typed following (I am not familiar yet with debugging core files)

wdb -c core.usermaster usermaster

(usermaster is exec name)
It produced following output.

Detected 64-bit executable.
Invoking /usr/ccs/bin/gdb64.
HP gdb 3.4 for PA-RISC 2.0 (wide), HP-UX 11.00.
Copyright 1986 - 2001 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 3.4 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `usermaster'.
Program terminated with signal 11, Segmentation fault.
#0 fnc_mbStrLen (string=0x80000001000065d7 "") at multibyte.c:1029
1029 multibyte.c: No such file or directory.

Does this help you find anything? Also, is it possible for you to give me instructions to debug a core file, please? (to get stack trace)
Valued Contributor
Srimalik
Posts: 436
Registered: ‎11-28-2006
Message 4 of 28 (13,071 Views)

Re: PA Risk Executable on Itanium Server

you have to give "bt" command to get a stack trace.

fnc_mbStrLen is causing a segfault.
Please confirm that 0x80000001000065d7 is valid address. You can try printing the string if it is a invalid address the stack will change(print function will dump somewhere). Then you need to trace this variable.

You can use this link for more gdb commands

http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801?ciid=e8080f1bac...
abandon all hope, ye who enter here..
Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 5 of 28 (13,071 Views)

Re: PA Risk Executable on Itanium Server

Hi Srikrishnan
I did the following gdb core on the Itanium server

and typed bt to get stack trace.
========= gdb output==========
(gdb) bt
#0 fnc_mbStrLen (string=0x80000001000065d7 "") at multibyte.c:1029
#1 0xc0000000080693b0 in fnc_StrLen (string=0x9fffffffe00033a8 "\n")
at str.c:767
#2 0xc00000000805ef44 in fnc_LoadIniFile (
IniFilename=0x9fffffffdfef87b0 "app.ini") at param.c:362
#3 0xc00000000805f880 in fnc_GetParamStr (
Section=0xc00000000801ca40 "Memory",
Item=0xc00000000801ca48 "TakeOverSIGUSR1", Default=0x0) at param.c:694
#4 0xc00000000805fad8 in fnc_GetParamLong (
Section=0xc00000000801ca40 "Memory",
Item=0xc00000000801ca48 "TakeOverSIGUSR1", Default=0) at param.c:754
#5 0xc00000000804f63c in EachTime () at mem.c:692
#6 0xc00000000804dd20 in fnc_Malloc (size=1024) at mem.c:214
#7 0xc00000000805ee98 in fnc_LoadIniFile (
IniFilename=0x9fffffffdfef87b0 "app.ini") at param.c:342
#8 0xc00000000805f880 in fnc_GetParamStr (Section=0xc000000008018938 "Debug",
Item=0xc000000008018ad8 "Dgn", Default=0xc000000008018ae0 "Off")
at param.c:694
#9 0xc00000000804169c in fnc_InitDgn () at dgn.c:411
#10 0xc000000007fc3300 in cdm_Initialise () at cdmslab.c:7543
#11 0xc000000007e49fbc in link_Initialise () at appclk.c:257
#12 0x4000000000002cb4 in main () at usermaster.c:59

(gdb) p 0x80000001000065d7
$1 = 9223372041149769175
(gdb)
=====+++++ end gdb output =====
Does that mean it is a valid address?

Tks

Acclaimed Contributor
Dennis Handly
Posts: 25,058
Registered: ‎03-06-2006
Message 6 of 28 (13,071 Views)

Re: PA-RISC Executable on Integrity server

[ Edited ]

(gdb) p 0x80000001000065d7
$1 = 9223372041149769175
>Does that mean it is a valid address?

No, you just used calculator mode to convert hex to decimal.

You need to do:
(gdb) p string
(gdb) x /8gx 0x80000001000065d7-7

The first uses debug info. The second debugs at the machine code level.

For debugging core files there is more documentation here:
http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801/?ciid=09080f1ba...
See: Debugging core files using HP WDB

Exalted Contributor
Steven E. Protter
Posts: 33,806
Registered: ‎08-15-2002
Message 7 of 28 (13,071 Views)

Re: PA Risk Executable on Itanium Server

Shalom Rao,

For this and other reasons, you want to recompile.

The emulation that permits PA-RISC an executable to run on Intanium servers comes with a 70% performance penalty.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Acclaimed Contributor
Dennis Handly
Posts: 25,058
Registered: ‎03-06-2006
Message 8 of 28 (13,071 Views)

Re: PA-RISC Executable on Integrity server

[ Edited ]

>SEP: For this and other reasons, you want to recompile.

Because of the advanced compiler debugging options? ;-)
There is +check= (runtime checking) and +wlint.

>The emulation that permits PA-RISC an executable to run on Integrity servers comes with a 70% performance penalty.

It highly depends on what PA box you were using and whether the application was CPU bound. See:
http://www.hp.com/go/aries

Acclaimed Contributor
James R. Ferguson
Posts: 21,184
Registered: ‎07-06-2000
Message 9 of 28 (13,071 Views)

Re: PA Risk Executable on Itanium Server

Hi:

The degree of performance penality under Aries is dependent upon both options configured and work type. These are summarized in the ARIES manpages:

http://docs.hp.com/en/B3921-60631/ARIES.5.html

Regards!

...JRF...
Acclaimed Contributor
Dennis Handly
Posts: 25,058
Registered: ‎03-06-2006
Message 10 of 28 (13,071 Views)

Re: PA-RISC Executable on Integrity server

[ Edited ]

>Srikrishan: Please confirm that 0x80000001000065d7 is valid address.

This may have nothing to do with the abort. I.e. it's just a parm of the aborting function.

The real problem is on line 1029 of multibyte.c. What's on that line?
Since the executable has debug info, why not use it? ;-)

Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 11 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Thank you all for quick responses.

* I can't really re-compile on Itanium because I am using a third party library calls and they are not supported under Itanium. So, I really want to compile it under PA Risk and run it on Itanium. If it helps, I ran this same PA executable on a different Itanium server (located at a diff location in the company) and it works! That server is also has 11iv2. Couldn't really figured out the differences between our servers yet.

Dennis,
>The real problem is on line 1029 of multibyte.c. What's on that line?
>Since the executable has debug info, why not use it? ;-)

Can you please elobarate on how I can examine this issue?
Thank you
Acclaimed Contributor
Dennis Handly
Posts: 25,058
Registered: ‎03-06-2006
Message 12 of 28 (13,070 Views)

Re: PA-RISC Executable on Integrity server

[ Edited ]

>I ran this same PA executable on a different Integrity server and it works! That server is also has 11iv2. Couldn't really figured out the differences between our servers yet.

Check the patch levels of Aries, libc and linker.

>>The real problem is on line 1029 of multibyte.c. What's on that line?

>Can you please elaborate on how I can examine this issue?

I assume you have the source file multibyte.c?
Otherwise you could do "info locals".

Or you can debug at machine level:
(gdb) disas $pc-4*20 $pc+4*4
(gdb) info reg

Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 13 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Dennis
I am getting the patch info for linker,aries and libc.
I don't have source for multibyte.c ( there is a manpage for multibyte).
when I tried the steps you provided in gdb, it provided output that I can't comprehend.

TFYH
Rao
Acclaimed Contributor
Dennis Handly
Posts: 25,058
Registered: ‎03-06-2006
Message 14 of 28 (13,070 Views)

Re: PA-RISC Executable on Integrity server

[ Edited ]

>I don't have source for multibyte.c (there is a manpage for multibyte).

The man page for multibyte(3C) is HP's code. fnc_mbStrLen is your code.

>when I tried the steps you provided in gdb, it provided output that I can't comprehend.

;-)
Just attach them to the thread and I'll look at them.

Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 15 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Not to disagree slightly with Dennis as to the pertinence of the address -- but with that address being just over 4Gb, and maxdsiz_64bit defaulting to 4Gb... I can't help but wonder if you did something like realloc() of an array without checking the return for failure and assuming the reallocation succeeded... then walking off the end of the array into unallocated virtual memory.

Based on perusing documentation -- "maint info sections" should dump the executable+core memory ranges... it might be worth seeing what's there (and what isn't). If the data ends at 0x10000000 (or very close... you fail the allocation that pushes you over, it doesn't mean you always get exactly there).. that would fit.
Valued Contributor
Srimalik
Posts: 436
Registered: ‎11-28-2006
Message 16 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Rao,

The top two frames in the stack are:
#########################################
#0 fnc_mbStrLen (string=0x80000001000065d7 "") at multibyte.c:1029
#1 0xc0000000080693b0 in fnc_StrLen (string=0x9fffffffe00033a8 "\n")
at str.c:767
##########################################
In frame 1 (fnc_StrLen) it seems you are trying to calculate the length of string "\n"
which should be 1.

Then from this function you are calling fnc_mbStrLen with a empty string(length 0).
Maybe you should look into this.

Secondly the address passed to both these functions are in different areas(not near each other 0x8000000 and 0x9fffffff).

How does "\n" becomes ""...you need to trace the flow in your code.

Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
Note that the parameters are changed.

It may help.

-Sri
abandon all hope, ye who enter here..
Honored Contributor
Don Morris_1
Posts: 797
Registered: ‎05-08-2001
Message 17 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

0x9ffffff.... would be at the top of the private octant for data/heap/mmf/stack, so is very likely a stack variable address. 0x80000... is more likely a heap allocation address [or is meant to be but may rather be a bad pointer arithmetic result after a failed allocation, hence my query about a realloc() in this path]. That's because it is near the low end of the octant (where the heap/BSS starts and then grows towards the memory mappings, thread stacks and main process stack that live on the other end and come towards lower addresses).
Valued Contributor
Srimalik
Posts: 436
Registered: ‎11-28-2006
Message 18 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Hi, Don
I am not familiar with the octant concept.
As per your comments
0x80000... should heap area
0x9ffff... should be on stack

I see a different range of address in frame 8
i.e 0xc0000000080.. It lies near to the code seg, are these the environment variables?

It may be off topic for this thread, may not be related to the thread in any way.





abandon all hope, ye who enter here..
Acclaimed Contributor
Dennis Handly
Posts: 25,058
Registered: ‎03-06-2006
Message 19 of 28 (13,070 Views)

Re: PA-RISC Executable on Integrity server

[ Edited ]

>Don: Not to disagree slightly with Dennis as to the pertinence of the address -- but with that address being just over 4Gb, and maxdsiz_64bit defaulting to 4Gb.

That address is trivially close to the start of the PA64 data area, 26071 bytes.

I'm assuming two things:
1) If 64 bit, it needs more than 4 Gb. ;-)
2) The debugger would give an error when tracing the stack if the "string" had a bad address, rather than just "".
#0 fnc_mbStrLen (
string=0x80000001000065d7 )

This address is big for IPF64 but not for PA64.
Also, the debugger won't complain about bad alignment though.

>Don: Based on perusing documentation - "maint info sections" should dump the executable+core memory ranges.

A better option would just be to use gdb's "info file".

Or better yet, use the command:
$ elfdump -S -o core

>Sri: Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
>Note that the parameters are changed.

Yes, looking at the source would help.

>Don: 0x80000... is more likely a heap allocation address

Or a global/static variable. Try using:
info sym 0x80000001000065d7

>Sri: I am not familiar with the octant concept.

(PA has quadrants, IPF has octants. They are related to space or region registers.)

>0x80000... should heap area
>0x9ffff... should be on stack
>I see a different range of address in frame 8. i.e 0xc0000000080.. It lies near to the code seg, are these the environment variables?

No, these are shlib text, or shared memory.

Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 20 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Hello all,

My maxdsiz_64bit is 4GB.

I don't have source multibyte.c, may be it is part of the 3rd party libraries I am using.
I am attaching the gdb and elfdump output.

Thank you
Valued Contributor
Srimalik
Posts: 436
Registered: ‎11-28-2006
Message 21 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

>Sri: Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
>Note that the parameters are changed.


Did you try this thing ?
abandon all hope, ye who enter here..
Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 22 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Sri
>Sri: Try to find out: how you end up calling fnc_mbStrLen("") from fnc_StrLen("\n").
>Note that the parameters are changed.

I can't debug at that level (I don't know how, I am calling a 3rd party library and program crashes)
Valued Contributor
Srimalik
Posts: 436
Registered: ‎11-28-2006
Message 23 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

I thought its your code as the stack shows many fnc_**** names.

Just to confirm, is fnc_InitDgn your code or its also in some third party library? which third party library?

May be I am confused by the naming convention used.

From where did you get the third party libraries with debugging symbols?

PS: Every answer does not deserve 10 points

http://forums11.itrc.hp.com/service/forums/helptips.do?#28
abandon all hope, ye who enter here..
Advisor
Rao Uppuluri
Posts: 27
Registered: ‎04-30-1998
Message 24 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

No, fnc_InitDgn is not in my code. I am guessing it is coming from 3rd party(An accounting software function calls) or from HP library.

I compiled the code on PA with -g option and debugged it on Itanium using wdb and from that session I am getting all the info you see here.

In any case, looks like no one really surprised by the fact that the exec runs fine through debugger but crashes when run by itself! ( I have seen similar behaviour with some VB code but never understood why)

Tks
Rao
Valued Contributor
Srimalik
Posts: 436
Registered: ‎11-28-2006
Message 25 of 28 (13,070 Views)

Re: PA Risk Executable on Itanium Server

Hi, Rao
Its very strange that the 3rd party is giving you libs with full debugging symbols. Not many vendors do this(never seen this for any HP library). Are you intentionally using debug version of the third party libs?


anyway, If you know who own this lib and have a support contract; you can involve them to look into this. You can show them the stack and provide them the core file.

as a call to fnc_mbStrLen("") from fnc_StrLen("\n") seems fishy to me \, I am wondering why would somebody do that?

-Sri
abandon all hope, ye who enter here..
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.