Re: JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23 (476 Views)
Reply
Regular Advisor
Carlo Henrico_1
Posts: 188
Registered: ‎10-02-2000
Message 1 of 6 (476 Views)

JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23

Hi All

I have been running JBOSS on the above for a while now and the customer decided to upgrade to Oracle 10. System was reconfigured, SHLIB_PATH and LD_LIBRARY_PATH reset as well as ORACLE_HOME. However now when I start JBOSS I get the following:

Java HotSpot(TM) Server VM warning: Unexpected Signal 11 occured under user-defined signal handler 0x75abf0ea
Stack_Trace: error while unwinding stack
( 0) 0xcf41d804 report_and_die__7VMErrorFv + 0x154 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 1) 0xcf2f7850 JVM_handle_hpux_signal__Q2_2os4HpuxSFiP9__siginfoPvT1 + 0xaf8 [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 2) 0xcf2f332c signalHandler__Q2_2os4HpuxSFiP9__siginfoPv + 0x4c [/opt/java1.5/jre/lib/PA_RISC2.0/server/libjvm.sl]
( 3) 0xccc1a398 xehInterpretSavedSigaction + 0x2d8 [/opt/mqm/lib/libmqmcs_r.sl]
( 4) 0xccc1a678 xehExceptionHandler + 0x1c8 [/opt/mqm/lib/libmqmcs_r.sl]
( 5) 0xc0213280 _sigreturn [/usr/lib/libc.2]
( 6) 0xd36f57f8 BN_set_word + 0x10 [/u01/app/oracle/10g/lib32/libnnz10.sl]
( 7) 0xd3732a80 BN_mod_inverse + 0xd0 [/u01/app/oracle/10g/lib32/libnnz10.sl]
( 8) 0xd38fe3b8 BN_MONT_CTX_set_locked + 0x1d8 [/perago/lib/libpersecj.sl]
( 9) 0xd38b0464 RSA_eay_public_decrypt + 0x254 [/perago/lib/libpersecj.sl]
(10) 0xd3878fc4 RSA_public_decrypt + 0x24 [/perago/lib/libpersecj.sl]
(11) 0xd38b0f40 RSA_verify + 0x190 [/perago/lib/libpersecj.sl]
(12) 0xd390a7f0 EVP_VerifyFinal + 0x128 [/perago/lib/libpersecj.sl]
(13) 0xd38c255c ASN1_item_verify + 0x114 [/perago/lib/libpersecj.sl]
(14) 0xd38864ec X509_CRL_verify + 0x2c [/perago/lib/libpersecj.sl]
(15) 0xd3b34fd8 loadX509CRLBase64__11CertHandlerFP6MemBufP11Certificate + 0xf8 [/perago/lib/libpersec.sl]
(16) 0xd3b34a20 getCrl__11CertHandlerFPCcP11Certificate + 0x190 [/perago/lib/libpersec.sl]
(17) 0xd3b52280 updateCrl__5StoreFv + 0x108 [/perago/lib/libpersec.sl]
(18) 0xd3b51944 getPrivKey__5StoreF5DNameb + 0xe4 [/perago/lib/libpersec.sl]
(19) 0xd3b4cc7c signHash__4SignFP6MemBufR5DNameT1 + 0x6c [/perago/lib/libpersec.sl]
(20) 0xd3b27138 sign__7XmlSignFP6MemBufPcR5DNameT1 + 0x1b8 [/perago/lib/libpersec.sl]
(21) 0xd394df20 Java_com_perago_security_Crypto_signXml + 0x380 [/perago/lib/libpersecj.sl]
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (11) at pc=d36f57f8, pid=21566, tid=48
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0.03 jinteg:02.13.06-15:51 PA2.0 (aCC_AP) mixed mode)
# Problematic frame:
# C [libnnz10.sl+0x1f57f8] BN_set_word+0x10
#
# An error report file with more information is saved as hs_err_pid21566.log
#
# Please report this error to HP customer support.
#
run.sh[174]: 21566 Abort(coredump)

The first few lines of hs_err_pid21566.log are as follows:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (11) at pc=d36f57f8, pid=21566, tid=48
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0.03 jinteg:02.13.06-15:51 PA2.0 (aCC
_AP) mixed mode)
# Problematic frame:
# C [libnnz10.sl+0x1f57f8] BN_set_word+0x10
#

--------------- T H R E A D ---------------

Current thread (006e00c8): JavaThread "JMS SessionPool Worker-1" daemon [_threa
d_in_native, id=48, lwp_id=211441]

siginfo:si_signo=11, si_errno=0, si_code=0, si_addr=4cd9c29c

Registers:
r1: 09d7f800 rp: d3732a83 r3: 03ab7988 r4: 00000000
r5: 4cd9c26c r6: 4cd9c280 r7: 4cd9c294 r8: 03a8b700
r9: 4cd9c2d0 r10: 4cd9c2bc r11: 492e4944 r12: 03a8b6db
r17: 00000000 r18: 75c11898 r19: 75ae0d3c r20: 00000000
r21: 4f206580 r22: 00000000 r23: 00000000 r24: 00000000
r25: 03a8b6dd r26: 4cd9c294 dp: 00012c90 ret0: 0ea2db70
ret1: 0ea2db74 sp: 4f2065c0 r31: 4cd9c294
Stack pointer: (sp=4f2065c0)
4f2065a0: 75ae0d3c 00000100 03ab7988 00000000
4f2065b0: 4cd9c294 00000000 4cd9c26c 4cd9c280
Instructions: (pc=d36f57f8)
d36f57d8: e840d000 4fc33f01 e81f1ff7 00000000
d36f57e8: 6bc23fd9 37de0080 375f0000 6bd33fc1
d36f57f8: 4b590010 8f226018 f3170c00 37fc0000

Stack: [4f180000,4f200000), sp=4f2065c0, free space=537k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libnnz10.sl+0x1f57f8] BN_set_word+0x10


libnnz10.sl is an Oracle library.

Any ideas where I can start looking for the cause please? I have tried all options regarding ordering of the SHLIB_PATH and LD_LIBRARY_PATH variables. Even tried renaming libnnz10.sl ;-) - Oracle was not happy and crashed on me.

Thanks

Carlo
Live fast, die young - enjoy a good looking corpse!
Acclaimed Contributor
Dennis Handly
Posts: 25,274
Registered: ‎03-06-2006
Message 2 of 6 (476 Views)

Re: JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23

>libnnz10.sl is an Oracle library.

The problem could be there or bad input from the caller or all the way back to #8 in libpersecj.sl. Or back to your java program.

>Any ideas where I can start looking for the cause please? I have tried all options regarding ordering of the SHLIB_PATH and LD_LIBRARY_PATH variables.

Why would the *PATHs have anything to do with it?

One issue could be a thread stack overflow, since you are near the beginning of BN_set_word.

You could figure this out by disassembling the first few words of BN_set_word.
Either from this:
Instructions: (pc=d36f57f8)
d36f57e8: 6bc23fd9 37de0080 375f0000 6bd33fc1
d36f57f8: 4b590010 8f226018 f3170c00 37fc0000

Or by using gdb:
$ gdb /u01/app/oracle/10g/lib32/libnnz10.sl
(gdb) disas BN_set_word BN_set_word+4*8
(gdb) q

If the load/stores are relative to the input args, it is a bad address. If relative to $r30 or $sp, then it is a thread stack overflow.
Regular Advisor
Carlo Henrico_1
Posts: 188
Registered: ‎10-02-2000
Message 3 of 6 (476 Views)

Re: JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23

Hello Dennis

I am not sure what to make of the reply and gdb's response. gdb is not installed by the customer but mine at my office gives:

(gdb) disas BN_set_word BN_set_word+4*8
Dump of assembler code from 0x1f5288 to 0x1f52a8:
0x1f5288 : stw %rp,-0x14(%sp)
0x1f528c : ldo 0x40(%sp),%sp
0x1f5290 : copy %r26,%r31
0x1f5294 : stw %r19,-0x20(%sp)
0x1f5298 : ldw 8(%r26),%r25
0x1f529c : cmpib,> 1,%r25,0x1f52b0
0x1f52a0 : depd %r23,31,32,%r24
0x1f52a4 : copy %r31,%ret0
0x1f52a8 : cmpb,<>,n %ret0,%r0,0x1f52d8
End of assembler dump.

As far as the paths are concerned, on this site someone indicated that Oracle has a problem in this file and one should re-arrange the LD_LIB/SHLIB search sequence so that it might pick up the correct library first instead of Oracle's.
Live fast, die young - enjoy a good looking corpse!
Acclaimed Contributor
Dennis Handly
Posts: 25,274
Registered: ‎03-06-2006
Message 4 of 6 (476 Views)

Re: JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23

>gdb is not installed by the customer but mine at my office gives:
0x1f5298 : ldw 8(%r26),%r25

This indicates that the R26 (first arg) value is out of bounds, and matches the value in si_addr.

>someone indicated that Oracle has a problem in this file and one should re-arrange the LD_LIB/SHLIB search sequence so that it might pick up the correct library first

Do you know where the correct lib is?
Regular Advisor
Carlo Henrico_1
Posts: 188
Registered: ‎10-02-2000
Message 5 of 6 (476 Views)

Re: JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23

Unfortunately not. Have no idea what is causing this crash.

Carlo
Live fast, die young - enjoy a good looking corpse!
Acclaimed Contributor
Dennis Handly
Posts: 25,274
Registered: ‎03-06-2006
Message 6 of 6 (476 Views)

Re: JBOSS and ORACLE 10.2.0.3 on HP-UX 11.23

>Have no idea what is causing this crash.

On my 11.23 IPF system I found those functions in:
/opt/openssl/lib/hpux32/libcrypto.so

I also have a man page and header so I can be dangerous. :-)

It appears that the BIGNUM* passed to BN_set_word is out of bounds.
This could be due to any of the three BIGNUM*
passed to BN_mod_inverse from libpersecj.sl.

Analysis of the core file may indicate that.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.