(/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline. (346 Views)
Reply
Frequent Visitor
SanjeevRao
Posts: 4
Registered: ‎04-18-2013
Message 1 of 10 (346 Views)

(/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

[ Edited ]

Dear team

 

Recent days, one of the Rx2800 Server started performing really slow, CPU utilisation (sar -u) shows

16:09:31    %usr    %sys    %wio   %idle

16:09:33       4       0       0      96
16:09:35      44       0       0      56


(though this is slave app machine in active-active) network, other servers are really performing well,

 

pinging from one machine to this machine shows me 10% packet loss,  completely went through the network switch/firewall logs in terms of any network latency, but no error msg found so far,

 

ping result from neighbouring machine in the same VLAN

 

ping 1921.68.0.4
PING 1921.68.0.4: 64 byte packets
64 bytes from 1921.68.0.4: icmp_seq=0. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=1. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=2. time=1028. ms
64 bytes from 1921.68.0.4: icmp_seq=3. time=18. ms
64 bytes from 1921.68.0.4: icmp_seq=4. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=5. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=6. time=1226. ms
64 bytes from 1921.68.0.4: icmp_seq=7. time=216. ms
64 bytes from 1921.68.0.4: icmp_seq=8. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=23. time=1600. ms
64 bytes from 1921.68.0.4: icmp_seq=24. time=590. ms
64 bytes from 1921.68.0.4: icmp_seq=25. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=26. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=27. time=1534. ms
64 bytes from 1921.68.0.4: icmp_seq=28. time=524. ms
64 bytes from 1921.68.0.4: icmp_seq=29. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=30. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=31. time=1674. ms
64 bytes from 1921.68.0.4: icmp_seq=32. time=664. ms
64 bytes from 1921.68.0.4: icmp_seq=33. time=0. ms
----1921.68.0.4 PING Statistics----
packets transmitted,  packets received, 0% packet loss
round-trip (ms)  min/avg/max = 0/276/122

 

Now one of the HP friend says login to MP and check any I/O related issue in this machine? unfortunately, i dont have the MP ip with me

 

any clues/guidance will be greatly appreciated.

 

 

P.S. this thread has been moved from Servers > Integrity Servers to HP-UX > sysadmin - HP Forums Moderator

San
Please use plain text.
HP Pro
Hiren N Dave
Posts: 98
Registered: ‎04-10-2006
Message 2 of 10 (338 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Hi,

 

The %wio is the percentage of time the CPU spent on blocked I/O request.

 

Here, the %wio is very high which indicates a disk bottleneck.

 

You may want to use sar -d and iostat commands to find out if any disk problem is causing this issue.

 

Cheers.

I am an HP Employee

Was this post useful? - You may click the KUDOS! star to say thank you.
Please use plain text.
Frequent Visitor
SanjeevRao
Posts: 4
Registered: ‎04-18-2013
Message 3 of 10 (329 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

thanks Mr. Hiren

 

yes you are right, i was suspecting I/O as well, but didnt have details about MP connectivity,

 

below are sar -d result, when i tried sar -d 2 10, it doesnt show any result

 

19:06:23   device   %busy   avque   r+w/s  blks/s  avwait  avserv
19:06:27
        
19:06:29
19:06:31
19:06:33
19:06:36
19:06:37
19:06:40
19:06:41
19:06:43

so i tried  #sar -d 2 2

 

19:06:56   device   %busy   avque   r+w/s  blks/s  avwait  avserv
19:06:58
19:07:00   disk13    0.46    0.50      10     304    0.00    0.91

Average    disk13    0.24    0.50       5     159    0.00    0.91

 

is there any ways i can obtain the mp lan ip details from the servers itself? or the SAN (HP P6000)  will give any clues about MP Lan,

 

--

 

btw, i am searching for KUDO's button :(

San
Please use plain text.
Acclaimed Contributor
Dennis Handly
Posts: 24,700
Registered: ‎03-06-2006
Message 4 of 10 (316 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

>I am searching for KUDOs button

 

The Kudos buttons are those white stars to the left of each post.

Please use plain text.
HP Pro
Hiren N Dave
Posts: 98
Registered: ‎04-10-2006
Message 5 of 10 (296 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Hi,

 

To access MP details

# /opt/propplus/bin/cprop -summary -c "Management Processor"

 

Also check syslogs for any I/O errors.

 

Check hardware paths for all disk

# ioscan -fnC disk

# ioscan -m dsf

# ioscan -P health -C disk

 

Check for any I/O errors for any suspected disk

# dd if=dev/rdsk/cXtYdZ of=/dev/null bs=1024k count=10240

 

Check for any LVM errors

# vgdisplay -v | more

 

Check file system corrpution if no disk problem found

# umount /fs
# fsck -F vxfs -y -o full,nolog /dev/vg01/rlvol1
# mount /dev/vg01/rlvol1


Cheers!

I am an HP Employee

Was this post useful? - You may click the KUDOS! star to say thank you.
Please use plain text.
Honored Contributor
Steven Schweda
Posts: 9,055
Registered: ‎02-23-2005
Message 6 of 10 (295 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Please use plain text.
HP Pro
Hiren N Dave
Posts: 98
Registered: ‎04-10-2006
Message 7 of 10 (292 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Yes, the sar output was misinterpreted.

 

It certainly looks like network problem.

I am an HP Employee

Was this post useful? - You may click the KUDOS! star to say thank you.
Please use plain text.
Frequent Visitor
SanjeevRao
Posts: 4
Registered: ‎04-18-2013
Message 8 of 10 (288 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Hmm, past couple of days had a tough time with Network Operations team, Changed network cable, gathered the switch/fw logs, verified them to find any issues along with the Network team members, verified the cable/port connecting it to another standby linux (HP Proliant servers {tried with 2 servers}) got a good response from those two servers,  after that network team says firmly its an issue with HP server :-(,

With the above commands you referred, i tried most of them and attached here in .txt format,

 

 

San
Please use plain text.
HP Pro
Hiren N Dave
Posts: 98
Registered: ‎04-10-2006
Message 9 of 10 (282 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

 

Well, seems you have got your MP IP now.

 

Why this output shows SCAN instead of CLAIMED? Out of four paths, two paths might have problem here.

 

disk      6  0/0/0/9/0/0/0/4/0/0/0.1.22.0.0.0.1  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c8t0d1   /dev/rdsk/c8t0d1
disk     11  0/0/0/9/0/0/0/4/0/0/0.1.22.0.0.0.2  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c8t0d2   /dev/rdsk/c8t0d2
disk      7  0/0/0/9/0/0/0/4/0/0/0.1.23.0.0.0.1  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c10t0d1   /dev/rdsk/c10t0d1
disk     12  0/0/0/9/0/0/0/4/0/0/0.1.23.0.0.0.2  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c10t0d2   /dev/rdsk/c10t0d2


Anyway, it should not affect the network packet loss as the CPU usage is also normal so you might want to check network parameters like autonegotiation settings, duplicate IP, collision, any known patch level issue etc.

 

I recommend to log a case with HP solution center for detailed log verification.

 

I am an HP Employee

Was this post useful? - You may click the KUDOS! star to say thank you.
Please use plain text.
Frequent Visitor
SanjeevRao
Posts: 4
Registered: ‎04-18-2013
Message 10 of 10 (277 Views)

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Still that doesnt look like MP ip,

 

Not able to ping them, not able to access it

 

yes will log a ticket with HP Soln centre,

San
Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation