Fibre channel and switching (534 Views)
Reply
Valued Contributor
Brian Reiter
Posts: 303
Registered: ‎03-04-2004
Message 1 of 6 (534 Views)

Fibre channel and switching


Hi Folks,

This is a quick question regarding fibre channel devices and path switching when a path fails. I have a cluster consisting of two RX2800s running OpenVMS 8-4 Update V6.0. The shared storage is provided by a HP 2000 G3 FC controller, connected to the Itaniums (via a pair of San Switches) using the Qlogic FC (ISP253x) cards. The are also two 1 Gb network switches providing the cluster interconnect. The quorum disk resides on the disk controller.

Each Itanium has a connection to each SAN switch, as does the disk controller. In theory there are multiple paths to the disks. Doing a SHOW DEV/FULL on one of the disks served shows 3 paths, 2 FC and 1 MSCP. I can use the command SET DEVICE/SWITCH/PATH=xxxxxx to swithc paths manually however should one of the FC cables be disconnected the system fails back to MSCP, on recovery the system stalled for a second or two before continuing.

Should one of the SAN switches fail the system hangs until the switch is recovered.

The sysgen parameters involved with the fiber channel are at their defaults:

MPDEV_ENABLE                    1          1         0          1 Boolean    D
MPDEV_REMOTE                    1          1         0          1 Boolean    D
MPDEV_POLLER                    1          1         0          1 Boolean    D
MPDEV_LCRETRIES                 1          1         0        256 Pure-numbe D
MPDEV_D1                        0          0         0         -1            D
MPDEV_D2                        0          0         0         -1            D
MPDEV_D3                        0          0         0         -1            D
MPDEV_D4                        0          0         0         -1            D
MPDEV_AFB_INTVL               300        300         0         -1 Seconds    D


What I expected to see when a single SAN switch failed as a seamless transition to the other working FC paths, so are my expectations too high or ami I missing something obvious?

Hope you can help

Regards


Brian

Respected Contributor
Bob Blunt
Posts: 314
Registered: ‎05-01-2003
Message 2 of 6 (526 Views)

Re: Fibre channel and switching

Brian, I'm not sure that I've got the total picture of your config but it sounded like you've got all the giblets you should need.  Two separate HBAs on each system, two independant SAN switches and two physically independant SAN disk controllers each with a path to your disks?

 

One thing that every SAN setup training has emphasized for any SAN of ANY size has been making triple sure that all the parts and cables are labelled with great detail.  It *sounds* like you may have a cable or two crossed somewhere.  And when you fail one or the other switch you end up with dissimilar paths to the storage so the systems press MSCP into use?

 

Make sure that you can label the cables before you run them if at all possible.  Audit your setup so you're totally positive that each HBA from each system goes to the right switch and port...for instance make sure that both PGA0 HBAs go to switch 1 and that both PGB0 HBAs go to switch 2 and the same for your controller(s).  I didn't recognize the controllers that you mentioned but there ARE two of them that have independant fibre interfaces and a separate path each to your storage, correct?  Make sure that there isn't any direct connection between your SAN switches.  Does your configuration use any zoning?

 

The MSCP path should be there as a safety net fallback only for the circumstance where node X loses access to storage while node Y continues unaffected.  If all the connections from both nodes and both controllers are configured identically then a single failure (one cable from the HBA to the switch for one node, one switch, one path from the switch to one controller for example) should only result in the disks all shifting from PGA0 to PGB0 (again, for example).  The path switch might NOT be instantaneous because they wait for the next IO to occur before the change.

 

bob

Valued Contributor
Brian Reiter
Posts: 303
Registered: ‎03-04-2004
Message 3 of 6 (519 Views)

Re: Fibre channel and switching

Hi Bob,

 

A crossed cable somewhere was my thought too. However I've been through the wiring to try and work out if that was the case. It seems to be OK.

 

As for zoning. we're using whatever the default was duriing system configuration:

 

When I disconnect a single FC coinnection from the back of the Itanium and then do a SHOW DEV $1$<disk>,I see that both fibre paths are not responding. Which is a bit of a bind, and its the same on both Itaniums (so at least its consistent).

 

cheers

 

Brian

 

 

Honored Contributor
Bill Hall
Posts: 249
Registered: ‎07-21-2003
Message 4 of 6 (508 Views)

Re: Fibre channel and switching

Brian,

 

That sounds like you have one path from the rx2800 to a SAN switch and two paths from that same SAN switch to the HP 2000 array. 

 

Can you show us the output from a $show device/full $1$DGAn on one of your storage devices?

 

Bill

Bill Hall
Valued Contributor
Brian Reiter
Posts: 303
Registered: ‎03-04-2004
Message 5 of 6 (483 Views)

Re: Fibre channel and switching

Hi there,

 

Ah well back to getting more information:

 

From one of the nodes the SHO DEV/FULL shows the following paths:

 

  Path FGB0.2070-00C0-FF12-DA25  (CATA1), primary, current
    Error count                    0    Operations completed              14463
    Last switched to time:   28-MAY-2012 10:15:50.20   Count                  1
    Last switched from time: 28-MAY-2012 10:15:26.63

  Path FGB0.2470-00C0-FF12-DA25  (CATA1)
    Error count                    0    Operations completed                  2
    Last switched to time:   28-MAY-2012 10:15:26.63   Count                  1
    Last switched from time: 28-MAY-2012 10:15:50.20

  Path MSCP  (CATA2)
    Error count                    0    Operations completed                  0
    Last switched to time:   Never                     Count                  0
    Last switched from time: Never

 

At this point the system had just been restarted. The FG device show as follows:

 

Device FGA0:, device type QLogic ISP253x FC, is online, shareable, error logging
    is enabled.

    Error count                    0    Operations completed                 92
    Owner process                 ""    Owner UIC                      [SYSTEM]
    Owner process ID        00000000    Dev Prot              S:RWPL,O:RWPL,G,W
    Reference count                0    Default buffer size                   0
    Current preferred CPU Id      15    Fastpath                              1
    Current Interrupt CPU Id      15
    FC Port Name 5001-4380-1207-82B4    FC Node Name        5001-4380-1207-82B5

Device FGB0:, device type QLogic ISP253x FC, is online, shareable, error logging
    is enabled.

    Error count                    0    Operations completed                290
    Owner process                 ""    Owner UIC                      [SYSTEM]
    Owner process ID        00000000    Dev Prot              S:RWPL,O:RWPL,G,W
    Reference count                0    Default buffer size                   0
    Current preferred CPU Id       0    Fastpath                              1
    Current Interrupt CPU Id       0
    FC Port Name 5001-4380-1207-82B6    FC Node Name        5001-4380-1207-82B7

 

Would I have expected to to see both FGA and FGB devices in the path lists for the disk? When looking at the SAN Switch configuration, I can see that  each switch has two entries for the SAN and 1 entry per node. On both switches zoneshow displays:

 


CATASANB:admin> zoneshow
Defined configuration:
 cfg:   cfg     zone_0_0
 zone:  zone_0_0
                1,0; 1,1; 1,4; 1,5

Effective configuration:
 cfg:   cfg
 zone:  zone_0_0
                1,0
                1,1
                1,4
                1,5

Any clues would be helpful!

 

cheers

 

Brian

 

 

Valued Contributor
Brian Reiter
Posts: 303
Registered: ‎03-04-2004
Message 6 of 6 (435 Views)

Re: Fibre channel and switching

Hi there,

 

Well its all fixed. I went back through the SAN controller config and spotted that a couple of the hosts were showing as undefined rather than as OpenVMS. It was bad luck that this onlky exhibited itself when a SAN switch was rebooted, any other misconfigured hosts would/should have been spotted earlier.

 

Now my IA64 have 5 paths to the storage.

 

cheers

 

 

Brian

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.