Re: EMC Clariion cx4/960 ioscan EXTREMELY slow (2215 Views)
Reply
Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 1 of 16 (2,392 Views)

EMC Clariion cx4/960 ioscan EXTREMELY slow

Hello gurus.

 

I have a pair of rx7640's hooked up to a cx4-960 Clariion.  Along with a multitude of other headaches, the ioscan -fnC disk is taking forever to return, (sometimes up to 5 - 7 minutes.)

 

The Clariion is setup with 6 luns through a Brocade 5100B switch.I can see no issues with the fiber cards on the system, (AH401A.)  Any ideas? 

 

Thank you!

 

Ron

Honored Contributor
Duncan Edmonstone
Posts: 5,684
Registered: ‎08-05-2000
Message 2 of 16 (2,390 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

What OS release are you running? If you are on 11.31 (11iv3) you can issue the following to at least determine what element is taking so long to scan:

 

ioscan -P ms_scan_time

 


HTH

Duncan
Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 3 of 16 (2,382 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Thanks Duncan!!

 

I ran that command, (I'm running 11.31.)  Everything comes back in milli seconds, except for anything coming off our fiber cards.  All the lunpaths, and EMC disks are into the 2 and 3 minute range. 

 

Any ideas where to go from here?

Outstanding Contributor
Pete Randall
Posts: 16,205
Registered: ‎11-03-1996
Message 4 of 16 (2,372 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

[ Edited ]

Unless you're looking for newly created LUNs, run ioscan with the -k switch to scan the kernel structures rather than the actual hardware:  ioscan -kfnC disk


Pete
Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 5 of 16 (2,362 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Thanks Pete!!

 

That worked as advertised!  However, upon futher frustration, I found that anything fiber related is very, VERY slow to respond.  That includes any powermt command, starting or stopping the Unisphere agent, or even just listing the fiber cards.  Booting the system is very slow too.  It seems to hang starting and configuring powerpath or unisphere.

 

I'm trying to rule out my system(s) that are hooked up to the cx4. 

Outstanding Contributor
Pete Randall
Posts: 16,205
Registered: ‎11-03-1996
Message 6 of 16 (2,353 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Glad that helped, Ron.  When you get an answer that you like, you can reward the responder by clicking on the kudos star.

 

Pete (I work for kudos) Randall


Pete
Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 7 of 16 (2,346 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Kudos awarded Pete.  Thanks again.  Any thoughts on my response?  We're getting to a critical phase here.  I am working overseas right now, and have to leave in 3 days for a medical semi-emergency.  I'd like to get at least connected before I go.  At least from there I can VPN in and work on from whatever morgue I'm laying in.

Outstanding Contributor
Pete Randall
Posts: 16,205
Registered: ‎11-03-1996
Message 8 of 16 (2,341 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Ron, I wish I could offer you some further assistance but I know absolutely nothing about EMC other than that it's spelled eee em cee.


Pete
Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 9 of 16 (2,339 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Thanks Pete.  I appreciate your help.  This is getting so frustrating, espcially with the language barrier, I'm beginning to spell EMC using different letters.

Honored Contributor
Matti_Kurkela
Posts: 6,271
Registered: ‎12-02-2001
Message 10 of 16 (2,336 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

I'm not a SAN administrator, but here are some thoughts:

 

Clariion is EMC's line of active/passive arrays. This means each disk/LUN should normally be accessed only through the SAN path that goes through the primary storage controller. If the non-primary controller is used to access the disks, it triggers a "trespass event" in the storage system, causing the primary controller to hand over the control of the disk/LUN to the other controller. The controller cannot serve disk requests while a trespass event is being handled, and the design intent is that it should only happen when there are problems with the primary controller or the SAN path(s) leading to it. Your storage administrator should be able to check if there is an excessive amount of trespass events.

 

HP-UX 11.31 can handle this type of storage, although according to EMC documentation, a March 2008 or later release of 11.31 is required. With an active/passive array and HP-UX 11.31, you should strongly prefer the new Agile DSFs (/dev/disk/diskNN or /dev/rdisk/diskNN) over of the old legacy DSFs (/dev/[r]dsk/cXtYdZ). You might even want to completely disable the old legacy DSFs (rmsf -L).

 

We once had a 11.31 system connected to an active/passive array of another manufacturer. It also behaved abysmally slow, until we found that our standard monitoring package (BMC Patrol with an old version of Hardware Sentry module) was using the legacy DSFs to monitor the state of all the disks it saw. It repeatedly polled each legacy DSF... and since each legacy DSF was associated with a specific SAN path, this kept triggering trespass events in the storage, and those frequent trespass events totally ruined the performance. The fix was to 1.) disable the legacy DSFs to prevent any other program from causing the same problem, and 2.) to update the Hardware Sentry module to a version that could actually use the new agile DSFs to monitor the local disks.

 

Either you or your SAN administrator (or whoever has the required EMC Powerlink account) should log in to the powerlink.emc.com website and find the latest version of a document titled "EMC Connectivity Guide for HP-UX". It is located in path:

Home > Support > Technical Documentation and Advisories > Host Connectivity/HBAs

> Installation/Configuration

 

It has a chapter titled "Configuration requirements for VNX series and CLARiion support with 11iv3" (starting at page 186 of the current version of the document). With your SAN administrator, you should carefully double-check each point mentioned: it has been my experience that active/passive arrays can require a higher level of attention to configuration details than the active/active ones.

 

Note that those requirements don't mention PowerPath at all, so I must assume you must comply with them, whether you're using PowerPath or not.

MK
Honored Contributor
Duncan Edmonstone
Posts: 5,684
Registered: ‎08-05-2000
Message 11 of 16 (2,328 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

MK,s advice is spot on - the problem you describe sounds very much like a Clariion where the host modes are not set correctly for 11iv3 (they have different settings from 11iv1 and 11iv2), resulting in the inactive paths being constantly trespassed by the MPIO subsystem round-robin'ing - which in turn causes the Clariion to contsnatly move the "owning controller" for any given LUN.

 

One quick and easy way to check if things are setup correctly - execute the following on one of your agile DSFs for the Clariion storage:

 

scsimgr lun_map -D /dev/rdisk/diskXX

 

You should see the same number of paths listed STANDBY as you see ACTIVE (standard characteristics for a ALUA disk array like the Clariion) If you don't then the Clariion host modes are not configured correctly for HP-UX11iv3


HTH

Duncan
Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 12 of 16 (2,308 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Thanks Duncan.

 

I ran the command, but it complained that I was still using the legacy DSF's.  If I run the rmsf command, will that immediately get rid of the old and bring in the new, or, (I imagine,) I have to run the insf command to see the new /dev/disk/disk##?

 

I'm only here a couple of more hours, then it's 2 days of travelling.  I'll have to look into this when I get home.  I appreciate any input while I'm in transit though.

 

Regards,

 

Ron

Acclaimed Contributor
James R. Ferguson
Posts: 21,184
Registered: ‎07-06-2000
Message 13 of 16 (2,301 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Hi:

 

There is an excellent whitepaper (and a script: '/usr/contrib/bin/vgdsf') to guide you in migrating from legacy to agile devices:

 

http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c01916036/c01916036.pdf

 

Regards!

 

...JRF...

Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 14 of 16 (2,260 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Hi!!

 

I took a look at that white paper briefly.  I even tried to run that script on another server, (BL870C,) that was not hooked up to the Clariion.  The first thing I noticed was that it did not change anything on the server, (perhaps a reboot is requried?)  Secondly, this script seems to only change the device file names if they are currently setup with a volume group.  The clariion has luns setup on it, however we have not been able to get as far as setting up luns on the system.  Is there anything else I may be missing?  I'm leaning towards the switch.  The system dmesg shows both fiber cards losing, then regaining connection.  If it were only one of the cards, I would replace it.  But both? 

Trusted Contributor
f. halili
Posts: 689
Registered: ‎06-28-2000
Message 15 of 16 (2,230 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

Are you using PowerPath? From my previous HPUX install with a Clarrion storage,  we were using 11.31 and it was better performance without the powerpath.

 

Cheers,

f. halili

Advisor
ronn48
Posts: 15
Registered: ‎08-09-2011
Message 16 of 16 (2,215 Views)

Re: EMC Clariion cx4/960 ioscan EXTREMELY slow

we ARE using PowerPath. 

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.