Re: shutdown problem for rx3600 (821 Views)
Reply
Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 1 of 23 (1,385 Views)
Accepted Solution

shutdown problem for rx3600

i've got a rx3600 (11.31, of course) that always crashes on shutdown -r.  

 

when i was first made aware of the problem, the system would actually get into a crash-fail-to-dump loop that was only resolved by powercycling the system.  the powercycle always resolved the issue (albeit unelegantly) and the system would come up as expected.  we manually booted (hpux> boot /stand/current/vmunix) over the weekend thinking the system was confused about what kernel to boot from.  that only marginally helped since the system still crashes on shutdown but at least now it produces a crashdump (see attached).

 

to me, this crashdump looks like there's a hardware problem -- but nothing is raising a red flag.

 

i noticed the (dire :-) warning about USB driver in the crashinfo file.  could that be it?  the driver is old (C.01.05.08).  the HAA boot device is USB:

 

Primary bootpath : 0/6/0/0/0/0/1/0/0/0.0x0.0x4000000000000000 (/dev/rdisk/disk3)
HA Alternate bootpath : 0/0/2/1.0x0.0x10 (/dev/deviceFileSystem/Usb/MassStorage/dsk/disk@hp-1008+294=A60020000001)

 

the system has been recently patched with the latest support bundle, fwiw.

 

any ideas?

Honored Contributor
Bill Hassell
Posts: 14,217
Registered: ‎05-29-2000
Message 2 of 23 (1,374 Views)

Re: shutdown problem for rx3600

[ Edited ]

Defintely get that bad USB driver out of there. And unless you have a goiod reason for the USB path in HAA path, use setboot to set HAA to real boot path.

 

Also, latest patch bundle? That would be Sep 2012 for QPK, Feature and HWE. I don't see recent ARPA/TCP cumulative patch in the crashinfo list (PHNE_41436) but it may not be reported.

 

The double panic may indeed point to a hardware error. Connect to the (real) console and get a list of the last 10-20 entries in the SL (hint: SL, then E for error logs, and F for forward progress).  That may provide a better clue as to what is going on.

Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 3 of 23 (1,350 Views)

Re: shutdown problem for rx3600

the both system logs look clean.

 

i'm arranging for the latest USB driver to be installed and we'll see if that brings us any joy.

 

about setboot...i've notes that say do "setboot -h <primary boot disk>" but is this is for pa-risc boxes only?  does it work for itaniums as well?

 

about the patches....i spent an entire minute (ha!) looking at the list of patches in the latest bundle and it sure looks to me that the latest arpa patch (for example and since we're using it as a guinea pig) is NOT included.  i suspect i missed something...but....

Honored Contributor
Bill Hassell
Posts: 14,217
Registered: ‎05-29-2000
Message 4 of 23 (1,339 Views)

Re: shutdown problem for rx3600

setboot -h <legacy hardware path, lunpath hardware path or a persistent dsf>

No difference between PA and IA as far as setboot is concerned. It is just setting stable storage on the system board.

As far as PHNE_41436, oops -- that should have been PHNE_41617, 41714 or 42017 for 11.31 (41436 is for 11.23). The ARPA cumulative patch is full of fixes for panics and other networking issues. But you have the second most recent so you should be OK. There are warnings for 41714 as well as its successor. For 41714 (which you have), it is only a problem when you have HP-UX Transport SRP installed. Check with:

    swlist -l bundle HPUXTransportSRP

If nothing is found, then 41714 is OK.

Your best bet is to look at the SL logs.

Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 5 of 23 (1,245 Views)

Re: shutdown problem for rx3600

an update...

 

updating the usb driver has helped....somewhat.  at least now the box doesn't crash while shutting down.

 

it still won't reboot on it's own without a powercycle.  (don't ask for console log...it didn't get captured this time around...)

 

has anyone heard of /sbin/reboot actually failing?  i was wondering if we did:

1) shutdown (with no parms) and then

2) reboot

...would that be safe and helpful?

Acclaimed Contributor
Dennis Handly
Posts: 25,272
Registered: ‎03-06-2006
Message 6 of 23 (1,242 Views)

Re: shutdown problem for rx3600

>has anyone heard of /sbin/reboot actually failing?

 

What part is failing? The shutting down or the booting?

Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 7 of 23 (1,240 Views)

Re: shutdown problem for rx3600

shutdown (imo)

 

it reboots fine -- provided you powercycle the box.

 

if you do a typical/normal 'shutdown -r', it won't reboot on it's own.

 

you can see in previous console logs where it is (for example) closing lvm volumes, so i don't believe this is a problem with some rc script.  

Acclaimed Contributor
Dennis Handly
Posts: 25,272
Registered: ‎03-06-2006
Message 8 of 23 (1,237 Views)

Re: shutdown problem for rx3600

>if you do a typical/normal 'shutdown -r', it won't reboot on it's own.

 

Where does it stop?  Anything in the shutdown log?

 

>you can see in previous console logs

 

I don't see those?

Have you tried disconnecting that USB disk?

 

Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 9 of 23 (1,235 Views)

Re: shutdown problem for rx3600

the shutdownlog isn't helpful...just the normal one-liner.

 

anyhow, in this last go-round (where the usb driver was updated) we didn't get the shutdown dialog because the connection was lost.

 

the previous console/shutdown logs are a horror to look at and probably not helpful at this point.

 

afaik, the usb dvd drive is still connected.  but between updating the driver and modifying setboot, i don't think it's an issue any longer.  if you think otherwise, let me know.

 

i'm hoping for (yet another) reboot over the weekend....

Acclaimed Contributor
Dennis Handly
Posts: 25,272
Registered: ‎03-06-2006
Message 10 of 23 (1,228 Views)

Re: shutdown problem for rx3600

>the shutdownlog isn't helpful... just the normal one-liner.

 

That means it got from shutdown(1m) to reboot(1m), so your suggestion probably wouldn't work?

 

>if you think otherwise, let me know.

 

Only if you have nothing better to do.  ;-)

Honored Contributor
Matti_Kurkela
Posts: 6,271
Registered: ‎12-02-2001
Message 11 of 23 (1,202 Views)

Re: shutdown problem for rx3600

Look into /etc/rc.log.OLD: if something went wrong in the shutdown, this file is where the error messages would be recorded.

(Originally it would have been /etc/rc.log, but renamed to /etc/rc.log.OLD in the next system startup.)

MK
Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 12 of 23 (1,025 Views)

Re: shutdown problem for rx3600

ok...with the dvd drive pulled, 'shutdown -r' is still resulting in the box panicing upon shutdown completion.

 

the console listing is attached.  this still feels like something hardware related...

Acclaimed Contributor
Dennis Handly
Posts: 25,272
Registered: ‎03-06-2006
Message 13 of 23 (1,021 Views)

Re: shutdown problem for rx3600

[ Edited ]

>the console listing is attached.

 

These errors indicate you have a garbage file left in /etc/rc.config.d/:

/sbin/rc[82]: 1500: not found.

...

/sbin/rc1.d/K410Rpcd[30]: 1500: not found.

 

I don't think it is related to your crash (unless you incorrectly fiddled with a config file) but you should fix it.

 

>Closing open logical volumes... Done

>panic : Failed to halt all processors

...

>Closing open logical volumes... Done

>Bad News: pr == 0x144000406c61001d

 

It seems that this is where it goes wrong.  During the shutdown.

Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 14 of 23 (1,018 Views)

Re: shutdown problem for rx3600

yeah...i noticed the rc.config.d messages.  i, too, don't think they're related.

 

yeah #2 -- that's where it all goes wrong.  perhaps some kind hp-insider could plug some of these messages/number into their call tracker and say <this> is the problem?  :-)

Acclaimed Contributor
Dennis Handly
Posts: 25,272
Registered: ‎03-06-2006
Message 15 of 23 (1,012 Views)

Re: shutdown problem for rx3600

[ Edited ]

>I noticed the rc.config.d messages.

 

But have you fixed it yet?  :-)

 

>perhaps some kind hp-insider could plug some of these messages/number into their call tracker ...

 

That's what the HPSC is for.

Honored Contributor
Todd McDaniel_1
Posts: 1,387
Registered: ‎08-26-2002
Message 16 of 23 (972 Views)

Re: shutdown problem for rx3600

Donna, I don't see that you have reviewed the shutdown log. any data in there?

also, please bring the box down into single user mode first, then note any errors, and reboot from there. I will bet you a donut it will come up clean.
Unix, the other white meat.
Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 17 of 23 (970 Views)

Re: shutdown problem for rx3600

there's nothing (interesting) in the shutdown log.  i think we (collectively) have decided that the machine is actually shuting down ok.

 

it's when the machine goes to restart (we're at the hardware level now) that it has problems.

 

if you are correct, how are you going to collect your donut?  :-)

Honored Contributor
Todd McDaniel_1
Posts: 1,387
Registered: ‎08-26-2002
Message 18 of 23 (968 Views)

Re: shutdown problem for rx3600

[ Edited ]

virtual donut!

 

 

I'd suggest going down to single in steps and watch the console to see anything irregular.

I'm pretty sure something is hanging when it goes down, I'd almost bet its not umounting all the disks or other hardware device for some reason and actually hanging at the very bottom of the process, before it attempts to come up on reboot.

 

do you have a large amount of san attached disks? or nfs mounts on this server?

 

 

Regards,

 

Todd McDaniel

Unix, the other white meat.
Acclaimed Contributor
Dennis Handly
Posts: 25,272
Registered: ‎03-06-2006
Message 19 of 23 (958 Views)

Re: shutdown problem for rx3600

>I think we (collectively) have decided that the machine is actually shutting down ok.

 

I don't think so.  It is panicking when you are trying to shut it down:  ux06.console.listing.txt

panic : Failed to halt all processors

 

Seems like a hardware problem.

 

>I don't see that you have reviewed the shutdown log.

 

Did you mean the rc.log?  There were problems with it.  Possibly unrelated?

Honored Contributor
Todd McDaniel_1
Posts: 1,387
Registered: ‎08-26-2002
Message 20 of 23 (955 Views)

Re: shutdown problem for rx3600

yea rc.logs, iirc the shutdown log only shows when the box went down.

 

you might also inspect your /dev/ (devices) directory for any unusal entries. Make sure you can account for every device, hopefully not too many there!

 

grasping for straws here, but if possible, recreate your /dev/random and /dev/zero devices

 

Review swlist for any unnecessary products you can remove.

 

Random 3rd party software not managed by your depot...

Unix, the other white meat.
Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 21 of 23 (840 Views)

Re: shutdown problem for rx3600

just to *wrap* this issue up (ho ho ho)...

 

turned out the MP board was faulty.    it took a fair amount of digging on the CE's part to diagnose the issue...but all is well now.

Acclaimed Contributor
Torsten.
Posts: 23,438
Registered: ‎10-02-2001
Message 22 of 23 (832 Views)

Re: shutdown problem for rx3600

Since the "MP-board" also known as the UCIO
(Unified Core-I/O) board has not only the MP/iLO but also BMC and other components, a failure could affect more than the MP only.


From your initial post I noticed you still use the OLD USB driver (for the internal DVD and vMedia) - you should update this to get USB 2.0 support.

Hope this helps!
Regards
Torsten.

__________________________________________________

There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________

No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Trusted Contributor
donna hofmeister
Posts: 191
Registered: ‎08-29-2008
Message 23 of 23 (821 Views)

Re: shutdown problem for rx3600

the usb driver was was one of the first things we fixed.  

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.