RHEL 5.5 multipath problem (P700m + MSA2324sa) (8680 Views)
Reply
Frequent Advisor
Timo J
Posts: 76
Registered: ‎03-21-2000
Message 1 of 15 (8,680 Views)

RHEL 5.5 multipath problem (P700m + MSA2324sa)

Hello,

We are having problems configuring multipath with RHEL 5.5.

multipath -ll doesn't show anything and -v5 produces the following errors:

cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0x1f
cciss!c0d0: bus = 4
cciss!c0d0: dev_t = 104:0
cciss!c0d0: size = 234374976
cciss!c1d0: not found in pathvec
cciss!c1d0: mask = 0x1f
cciss!c1d0: bus = 4
cciss!c1d0: dev_t = 105:0

Collected below are the other background details of the installation.

--

HW configuration:
--------------------------------------------
BladeSystem c7000 Enclosure G2
ProLiant BL460c G7 (603588-B21)
Smart Array P700m (508226-B21)
3Gb SAS Switch (AJ865A)
MSA2324sa (AJ807A Rev B)

Base Install:
--------------------------------------
RHEL5 Update 5

Installed packages:
--------------------------------------
PSP 8.60
device-mapper-multipath-0.4.7-42.el5.i386.rpm
kpartx-0.4.7-42.el5.i386.rpm
HPDMmultipath-4.4.1.tar.gz

# rpm -qa --last
device-mapper-multipath-0.4.7-42.el5 kpartx-0.4.7-42.el5 mptlinux-4.00.13.07-5 hpvca-6.2.0-8 hponcfg-3.1.0-0 hpmouse-1.2.0-96 hpdiags-8.6.0-18 hpacucli-8.60-8.0 hp-smh-templates-8.6.0.14-16
hp-snmp-agents-8.6.0.18-18 hpsmh-6.2.0-12 hp-ilo-8.5.0-1.rhel5 hp-health-8.6.0.24-27 hp-OpenIPMI-8.5.0-1.rhel5 cpqacuxe-8.60-7.0
cpq_cciss-3.6.26-5

# /etc/multipath.conf
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/bin/true"
path_checker tur
rr_min_io 100
rr_weight uniform
failback immediate
no_path_retry 12
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][[0-9]*]"
}
devices {
device {
vendor "HP"
product "MSA2012sa|MSA2312sa|MSA2324sa"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -n -s /block/%n"
path_checker tur
path_selector "round-robin 0"
prio_callout "/sbin/mpath_prio_alua %d"
rr_weight uniform
failback immediate
hardware_handler "0"
no_path_retry 18
rr_min_io 100
}
}

# /var/log/dmesg (When only one cable connected)
SCSI subsystem initialized
HP CISS Driver (v 3.6.26-5)
cciss: Device 0x3230 has been found at bus 6 dev 0 func 0
ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 30 (level, low) -> IRQ 122
cciss0: Placing controller into performant mode
cciss/c0d0: p1 p2 p3 p4 < p5 >
cciss: Device 0x323a has been found at bus 12 dev 0 func 0
ACPI: PCI Interrupt 0000:0c:00.0[A] -> GSI 16 (level, low) -> IRQ 169
cciss1: Placing controller into performant mode
Fusion MPT SAS Host driver 4.00.13.07
Fusion MPT SPI Host driver 4.00.13.07
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com
usb 6-1: new full speed USB device using uhci_hcd and address 2
device-mapper: dm-raid45: initialized v0.2594l

# multipath -ll

# multipath -v5
cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0x1f
cciss!c0d0: bus = 4
cciss!c0d0: dev_t = 104:0
cciss!c0d0: size = 234374976
cciss!c1d0: not found in pathvec
cciss!c1d0: mask = 0x1f
cciss!c1d0: bus = 4
cciss!c1d0: dev_t = 105:0
dm-0: blacklisted
dm-1: blacklisted
dm-2: blacklisted
dm-3: blacklisted
dm-4: blacklisted
dm-5: blacklisted
dm-6: blacklisted
dm-7: blacklisted
dm-8: blacklisted
dm-9: blacklisted
loop0: blacklisted
loop1: blacklisted
loop2: blacklisted
loop3: blacklisted
loop4: blacklisted
loop5: blacklisted
loop6: blacklisted
loop7: blacklisted
md0: blacklisted
ram0: blacklisted
ram10: blacklisted
ram11: blacklisted
ram12: blacklisted
ram13: blacklisted
ram14: blacklisted
ram15: blacklisted
ram1: blacklisted
ram2: blacklisted
ram3: blacklisted
ram4: blacklisted
ram5: blacklisted
ram6: blacklisted
ram7: blacklisted
ram8: blacklisted
ram9: blacklisted
===== no paths =====
libdevmapper: ioctl/libdm-iface.c(1740): dm names NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-tmp NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-home NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-log NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-mcx NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-usr NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-var NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-archive NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-cdr NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-opt NF [16384]
libdevmapper: ioctl/libdm-iface.c(1740): dm table vg00-ready NF [16384]

# /sbin/hp_scsi_id -g -u -n -s /block/cciss\!c0d0
3600c0ff0001047c653f06b4d01000000

We need urgent help to get this problem solved - has anyone managed to get ths HW+SW combo working?
N/A
Honored Contributor
Alzhy
Posts: 4,373
Registered: ‎01-22-2002
Message 2 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Do you have the latest HP Device Mapper Multipath kit installed?

Can you pst your complete /etc/multipath.conf?


Hakuna Matata.
Frequent Advisor
Timo J
Posts: 76
Registered: ‎03-21-2000
Message 3 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Yes, we are using the latest HPDMmultipath-4.4.1 kit. Multipath config file is the stock one that HPDMultipath installer modifies during installation (see attachment).
N/A
Honored Contributor
Alzhy
Posts: 4,373
Registered: ‎01-22-2002
Message 4 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Try uncommenting cciss in the the blacklist section:


blacklist {
# wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z][[0-9]*]"
devnode "^cciss!c[0-9]d[0-9]*"
# device {
# vendor "HP"
# product "OPEN-*"
# }
}


Then multipath -F
Then do issue_lip and scan on each of your fc hosts.

Do you have lsscsi installed? Do the MAS luns even appear at all?
Hakuna Matata.
Frequent Advisor
Timo J
Posts: 76
Registered: ‎03-21-2000
Message 5 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

For additional info we are using SAS connection to the disk array - not fibre channel.

Hence I have to keed the cciss part commented or else all devices are blacklisted.
N/A
Honored Contributor
Alzhy
Posts: 4,373
Registered: ‎01-22-2002
Message 6 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Hmmm...
I never had any HP Array that's SAS connected... do you have documentation handy for your MSA array and possibly a HOWTO in a Linux environment?
Hakuna Matata.
Frequent Advisor
Timo J
Posts: 76
Registered: ‎03-21-2000
Message 7 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Managed to do so far...

1) Installed RHEL 5.5 again with following base install:

%packages
@admin-tools
@base
@core
@development-tools
@development-libs
@system-tools
@base-x
@editors

2) Installed PSP 8.60 with following packages:

cpq_cciss-3.6.26-5.rhel5.i686.rpm
mptlinux-4.00.13.07-5.rhel5.i686.rpm
hp-OpenIPMI-8.5.0-1.rhel5.i386.rpm
hp-ilo-8.5.0-1.rhel5.i386.rpm
hp-health-8.6.0.24-27.rhel5.i386.rpm
hp-snmp-agents-8.6.0.18-18.rhel5.i386.rpm
hpmouse-1.2.0-96.noarch.rpm
hpsmh-6.2.0-12.i386.rpm
hp-smh-templates-8.6.0.14-16.noarch.rpm
hpvca-6.2.0-8.linux.rpm
cpqacuxe-8.60-7.0.noarch.rpm
hpacucli-8.60-8.0.noarch.rpm
hpdiags-8.6.0-18.linux.i586.rpm
hponcfg-3.1.0-0.noarch.rpm

3) Installed HPDMmultipath-4.4.1

Situation after this:

# multipath -v3

...
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d0 104:0 50 [undef][ready]
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d1 104:16 10 [undef][ready]
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d2 104:32 10 [undef][ready]
3600c0ff0001021929bb9744d01000000 #:#:#:# cciss!c0d3 104:48 50 [undef][ready]
Found matching wwid [3600c0ff0001021929bb9744d01000000] in bindings file.
Setting alias to mpath0
cciss!c0d0: ownership set to mpath0
cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0xc
cciss!c0d0: state = 2
cciss!c0d0: prio = 50
cciss!c0d1: ownership set to mpath0
cciss!c0d1: not found in pathvec
cciss!c0d1: mask = 0xc
cciss!c0d1: state = 2
cciss!c0d1: prio = 10
cciss!c0d2: ownership set to mpath0
cciss!c0d2: not found in pathvec
cciss!c0d2: mask = 0xc
cciss!c0d2: state = 2
cciss!c0d2: prio = 10
cciss!c0d3: ownership set to mpath0
cciss!c0d3: not found in pathvec
cciss!c0d3: mask = 0xc
cciss!c0d3: state = 2
cciss!c0d3: prio = 50
mpath0: pgfailback = -2 (controller setting)
mpath0: pgpolicy = group_by_prio (controller setting)
mpath0: selector = round-robin 0 (controller setting)
mpath0: features = 0 (internal default)
mpath0: hwhandler = 0 (controller setting)
mpath0: rr_weight = 1 (internal default)
mpath0: minio = 100 (controller setting)
mpath0: no_path_retry = 18 (controller setting)
pg_timeout = NONE (internal default)
mpath0: set ACT_CREATE (map does not exist)
mpath0: failed to load map (a path might be in use)
mpath0: domap (0) failure for create/reload map
...

and

# multipath -d
create: mpath0 (3600c0ff0001021929bb9744d01000000) HP,MSA2324sa
[size=121G][features=0][hwhandler=0][n/a]
\_ round-robin 0 [prio=100][undef]
\_ #:#:#:# cciss!c0d0 104:0 [undef][ready]
\_ #:#:#:# cciss!c0d3 104:48 [undef][ready]
\_ round-robin 0 [prio=20][undef]
\_ #:#:#:# cciss!c0d1 104:16 [undef][ready]
\_ #:#:#:# cciss!c0d2 104:32 [undef][ready]

note!

# multipath
# multipath -ll >> doesn't return anything


Can anybody help me forward from here on???

-Timo
N/A
Occasional Visitor
markpag
Posts: 2
Registered: ‎04-12-2011
Message 8 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

I am having the same issue but with RedHat 5.6.....Let me know if you get any help.

Thanks
Occasional Visitor
Michael Moir_1
Posts: 3
Registered: ‎11-12-2001
Message 9 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Same trouble here. Looking for solution please.
Tell me and I will forget, Show me and I may remember, Let me do it and I will understand.
Occasional Visitor
markpag
Posts: 2
Registered: ‎04-12-2011
Message 10 of 15 (8,680 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Hi have opened a case with RedHat on this problem.

HP support has verified that the problem exists, at least with my RedHat5.6 version. I am now trying to resolve with RedHat. HP has no know resolution.

I will update this thread as I receive results.
Honored Contributor
Ralph Grothe
Posts: 2,631
Registered: ‎04-09-2000
Message 11 of 15 (8,673 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Don't know if this helps?

On a RHEL 5.6 box that I administer and that also uses the cciss driver to access the internal disks it helped to blacklist them in order to make multipath errors disappear.
I suspect the driver uses deprecated ioctl calls or is otherwise broken.
I'm not sure if this really matters, but I escaped the bang in the blacklist regex for it to work.

These are my vgroot/md1 internal disks:

# vgdisplay -v vgroot 2>/dev/null|awk '/PV Name/{system("mdadm -QD "$NF"|grep cciss")}'
0 104 2 0 active sync /dev/cciss/c0d0p2
1 104 18 1 active sync /dev/cciss/c0d1p2



# grep -A2 ^blacklist /etc/multipath.conf
blacklist {
devnode "^cciss\!c0d[01]"
}



Here I so far have only one SAN LUN attached.
And this is all multipath lists.


# multipath -ll
mpath0 (360060e801530d700000130d700000052) dm-8 HP,OPEN-V
[size=7.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 0:0:0:0 sda 8:0 [active][ready]
\_ 1:0:0:0 sdb 8:16 [active][ready]


Madness, thy name is system administration
Valued Contributor
DeafFrog
Posts: 294
Registered: ‎02-25-2009
Message 12 of 15 (8,673 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

[ Edited ]

Hi Timo ,

not sure , but check this thread , this might be of some help :

http://h30499.www3.hp.com/t5/HP-BladeSystem/SAS-Switch-Port-Mapping/m-p/4516011#M7777


Rehards,

FrogIsDeaf
Occasional Visitor
Michael Moir_1
Posts: 3
Registered: ‎11-12-2001
Message 13 of 15 (8,673 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Still unsuccessful mapping multiple paths to storage.
DL360 G6, MSA 2324sa
RHEL 5.6 -- 2.6.18-238.9.1.el5 x86_64
System is entirely up to date including:
device-mapper-multipath-0.4.7-42.el5_6.2, HPDMmultipath-4.4.1, PSP 8.70

# multipath -ll -v5
cciss!c1d0: not found in pathvec
cciss!c1d0: mask = 0x5
cciss!c1d0: bus = 4
cciss!c1d0: dev_t = 105:0
cciss!c1d0: size = 1432149760
cciss!c1d1: not found in pathvec
cciss!c1d1: mask = 0x5
cciss!c1d1: bus = 4
cciss!c1d1: dev_t = 105:16
cciss!c1d1: size = 1432149760
===== no paths =====

A) Wouldn't be appropriate to blacklist cciss from multipath.conf,
as that's driver for P800 SAS controller connected to MAS2324sa.
B) No SAS switches or Blade chassis involved here.
Server direct attached to storage.
C) Vendor and Model strings are contained in /sys/block/cciss!cXdX/device,
but not detected by multipath command?
Tell me and I will forget, Show me and I may remember, Let me do it and I will understand.
Occasional Visitor
Michael Moir_1
Posts: 3
Registered: ‎11-12-2001
Message 14 of 15 (8,673 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

Out of my support case with Red Hat (00456892), I was provided test patches for device-mapper-multipath and kpartx packages which resolved this issue for me.
Tell me and I will forget, Show me and I may remember, Let me do it and I will understand.
Occasional Collector
seinyan
Posts: 2
Registered: ‎10-01-2011
Message 15 of 15 (8,126 Views)

Re: RHEL 5.5 multipath problem (P700m + MSA2324sa)

There is bug in the device mapper multipah

https://bugzilla.redhat.com/show_bug.cgi?id=742906

 

Last that i recieved from RH is:

Ben Marzinski 2012-01-12 13:30:54 EST
"Personally, I favor leaving the bug open and fixing this in RHEL 5.9. "

 

I have same problems  with rhel 5 update 6, but with blade servers P712m and P2000 sa g3.

http://h30499.www3.hp.com/t5/Disk-Array/P2000-G3-SAS-P712m-blade-and-multipath-Redhat-Linux-5/td-p/5... 

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.