hp eva 4400 performance issue with CA replication and MPX110 (1066 Views)
Reply
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 1 of 21 (1,066 Views)

hp eva 4400 performance issue with CA replication and MPX110

when we create DR group by HP command view a critical performance issue appear on HP storage eva4400 (main site) many of vmware servers related to that vdisk lost path
and som times freez for seconds and cannot ping guest servers
we check event log for EVA4400 we found this message loged many times:

Excessive PING response time on the inter site link is preventing acceptable replication throughput: Reducing data exchange resources.

on vmware console we found this event:

Successfully restored access to volume 4a5dbb9e-eba9af52-c359-0022649da44b (server100) following connectivity issues.info

Lost connectivity to storage device naa.600508b4000a54db0000900000400000. Path vmhba0:C0:T0:L3 is down. Affected datastores: "VM_datastore1".error 1/4/2010 4:48:34 PM

Path redundancy to storage device naa.600508b4000a54db0000900000400000 degraded. Path vmhba0:C0:T0:L3 is down. 2 remaining active paths. Affected datastores: "VM_datastore1".warning 1/4/2010 4:48:07 PM
** we update firmware to the latest but the still same problem

more details:

Main Site EVA 4400 : version XCS 9522000
DR Site EVA 4000 : version 6.22
VMware : ESX4

we use HP distance Gateway MPX110 for replication between two site.
WAN speed 5mb distamce (60KM).

thanks in advance for any suggestion .
Honored Contributor
Patrick Terlisten
Posts: 1,784
Registered: ‎07-12-2005
Message 2 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Hello,

do you use synchronous oder asynchronous replication? What round-trip time do you have on the WAN link?

Best regards,
Patrick
Regards,
Patrick
Honored Contributor
Víctor Cespón
Posts: 2,397
Registered: ‎11-09-2007
Message 3 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Before creating vdisks for the production vdisks, you should check the status of the replication link. Look in the SAN switches counters, and EVAperf counters.

Check CA implementation guide too...
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01800459/c01800459.pdf

The most common error is bot to change the needed parameters on the SAN switches (In-onder delivery, port-based routing, increase the buffer credits on the ISL ports...)
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 4 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Hi Patrick
* we try to use synchronous and asynchronous replication when we creating Dr group the same problem appeared on both.
* round-trip time on the WAN link =45-55 ms

Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 5 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Hi vcpones
we already applied the port-based routing before:

FCSW1:admin> switchdisable
FCSW1:admin> aptpolicy 1
FCSW1:admin> switchenable

I have many questions:

1- are you recommend to run the follwoing command line on ISL ports.
portcfglongdistance 0 LD 1 69

2-
** also I read the following on brocade: to determine the optimal amount of credits:
Credit=(Round_trip_time+Receiving _port_processing_time)/Frame Transmission _time.
In other words, the optimal number of BB credits depends on three key parameters:
1) round trip time, i.e., the distance
2) frame processing time
3) frame transmission time*

how can we calculate that for our environment and what is the best configuration for that.

3- how to to setup In-onder delivery you mention.


Thanks

Regular Advisor
IvanForceville
Posts: 139
Registered: ‎06-18-2004
Message 6 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Apart from the port based routing make sure you set the following parameters

Needs to be set on the SAN switches
IOD (In Order Delivery) "Enabled"
DLS (Dynamic Load Sharing)"Disabled"

Can be done via the GUI (http://ip) of the switch => Switch Admin => Advanced

Or through CLI:
"iodsetâ en â dlsresetâ

Make sure you set these parameters on all switches in the fabric.

By the way... we have the exact same setup exept for the target EVA which is a 4400 but also the MPX110's and the 60km's DR :)
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 7 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Still problem not solved also after we set :
1- IOD (In Order Delivery) "Enabled"
2- DLS (Dynamic Load Sharing)"Disabled

the same performance issue appear when establish the connection

" Lost connectivity to storage device naa.600508b4000a54db0000700000920000. Path
vmhba1:C0:T0:L4 is down. Affected datastores: "VM_datastore1".
error
1/13/2010 1:30:33 PM


Path redundancy to storage device naa.600508b4000a54db0000700000920000 degraded. Path
vmhba1:C0:T1:L4 is down. 3 remaining active paths. Affected datastores: "VM_datastore1".
warning
1/13/2010 1:27:54 PM "

Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 8 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Does this performance issue related to the link speed between main site and DR site( 5M) or related to EVA storage or SAN switches.

thanks in ADV
Regular Advisor
IvanForceville
Posts: 139
Registered: ‎06-18-2004
Message 9 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Just to have an idea... how big are the Vdisks you replicate? What's the load on the Vdisks replicated.

We've seen some realy strange behaviour of CA with low bandwidhts.

Another thing: how are the MPX's configured
Do you have any compression enabled? What's the scaling factor? Do you use a QOS on the link between the sites? Is the 5MBps guaranteed or is this max bandwidth?
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 10 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110


thanks IVAn for your questions:

we try with Vdisks 90 gb and 130gb for replicate the same issue.

we try mpx110 without compression enabled and with the same problem.
we try the scaling factor disabled and with 0

about QOS it is disabled between two site because we configure the ports as ISL and

the 5MBps used also for another network and replication for oracle around 300mb data transaction>

again my question if the bandwidth not enough between two site make the source eva very busy????

Advisor
SDrake_1
Posts: 13
Registered: ‎04-20-2009
Message 11 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

@O.Odeh

Do you have other VMware hosts accessing the same LUNs? Do you get the same loss of connectivity?

What is your path policy in VMware? vCenter > Problem_Host > Configuration > Storage Adapters...

...Right-click the datastore with ID: naa.600508b4000a54db0000900000400000 and click Manage Paths

Simon
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 12 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

simon;
we have 8 VMware hosts see the same LUNs for VMotions.
and error message for vmoware appear on all vmware hosts.

the path policy in VMware round robin also we try fixed and mtu the same problem appeared

** ithink our issue with storage not with vmware because also when try to replicate windows LUN we found :
" Excessive PING response time on the inter site link is preventing acceptable replication throughput: Reducing data exchange resources "
Regular Advisor
IvanForceville
Posts: 139
Registered: ‎06-18-2004
Message 13 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Things I would do next:

- Log an HP call and send them supportshows of all the switches in the fabric. Let them look at the replication ports to check if there are any LLI (Link Line Interrupts) errors.

- The MPX110's are very itchy for fluctuating bandwidths. Use QOS, this is absolutely necessary.

- Check the writehistorylog size when replicating data. Do you see them getting filled up rather quickly?
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 14 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

simon;
we have 8 VMware hosts see the same LUNs for VMotions.
and error message for vmoware appear on all vmware hosts.

the path policy inthe VMware is round robin also we try fixed and mtu the same problem appeared

** I think our issue is with storage not with the vmware because when we try to replicate windows LUN ,we noticed the following:

" Excessive PING response time on the inter site link is preventing acceptable replication throughput: Reducing data exchange resources "
Advisor
SDrake_1
Posts: 13
Registered: ‎04-20-2009
Message 15 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Go with Ivan's response
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 16 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

THANx IVAN
- we Loged acall with hp support then they told us it is not support case it is installtion case then they close the case
we will try to open new case with them

about writehistorylog size when replicating data itnot filled quickly also
our problem appear from the begning after creating any dr group.
Occasional Advisor
Pesca
Posts: 7
Registered: ‎01-29-2010
Message 17 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Hello, same problem for Us, a year ago, with a 10Mb Wan, no solution - HP give us, Storage Mirroring it's ok but more admin

Real Pre requiste 2*40Mb without mpx or double disk volume, cache disk = vdisk because if only 1*40Mb and if the Wan HS > All Vm on San freeze during replication, even for a small vdisk


Sorry for language
Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 18 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

Hi Pesca,
you mention you face same problems on VMs with HP CA,
is the problem on CA replication or using the MPX110, or the WAN speed, please more detail and explanation for your issu you face

Thanks again for any suggestion to solve this issue.

we still running on same problem some tims VMs not responding when replicate any vdisk.
Occasional Advisor
Pesca
Posts: 7
Registered: ‎01-29-2010
Message 19 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

We had a replication problem over a wan 1*10Mb (fibre 1 kilometers for DR Site), Mpx, Ev4400 and CA.
Error message excessive ping, Vm on San freeze.
Latest 4400 Firmware test > Same problem

Now it's ok with HP Storage mirroring, but more admin, tunning ...

Another problem here
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1401330


Regular Advisor
Osama Odeh_1
Posts: 139
Registered: ‎06-18-2006
Message 20 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

this problem related to bandwidth between two sites.
Occasional Visitor
wittaya_misco
Posts: 1
Registered: ‎06-22-2010
Message 21 of 21 (1,066 Views)

Re: hp eva 4400 performance issue with CA replication and MPX110

try to change commit block size and rep again
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.