P2000 slow since RAID reallocation (1620 Views)
Reply
Occasional Advisor
Posts: 8
Registered: ‎07-28-2013
Message 1 of 8 (1,620 Views)
Accepted Solution

P2000 slow since RAID reallocation

Hi all,

 

I have an iSCSI P2000 loaded with 12 x 300gb DP SAS drives.  This is connected to 2 DL385 servers with 64gb RAM running XenServer. The SAN network is multipathed via 2 gigabit switches (4 x copper links from the P2000 to each switch, then each XenServer has a copper link to each switch)

 

Recently I re-evaluated my install-time decision to create 2 vdisks: a RAID-5 and a RAID-10. I decided the RAID-5 was pointless and destroyed it, then used those disks to extend the RAID-10 into one big array, with 2 hot-spare disks.

 

Since that was done (just over a week ago) disk performance has been quite horrible; looking at `top` on both the XenServer hosts show ~15% I/O wait. hdparm inside a VM shoes around 1.12mbps reads from the SAN.

 

There are 2 other Win2008R2 machines connected to the SAN also. I have previous performance tests showing they too are experiencing a significant difference so I don't beleive it is a problem on the XenServer hosts.

8k random reads previously ~24.75MB/s, now 1.67MB/s

64k random reads previously ~170MB/s, now 3.61MB/s

 

The RAID extension has completed, everything is "OK" in the SMU. The XenServers are quite "empty" (7 VM's on one of them, 4 VM's on the other) but it is a production environment.

 

I'd love some input on what could be wrong and where to investigate further.

Trusted Contributor
Posts: 131
Registered: ‎02-27-2013
Message 2 of 8 (1,583 Views)

Re: P2000 slow since RAID reallocation

When it comes to Read Performance, IMHO RAID5 provides equivalent Read performance compared to RAID10 in both random and sequential reads. Its the Write Performance where RAID10 gains edge over RAID5 since R5 has High Write Penalty.

 

Anyhow, from your detailed Problem Description I understand you must have gone through P2000 G3 Best Practices Guide: http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA3-2141ENW.pdf It would be very difficult to check why the Read Performance is so degraded on RAID Change.

 

 

I would also verify the multipathing and check for the dead paths, if its as it should be.

 

I would request you to open a Case with HP Technical Support for Performance Review with latest store.log and Performance Logs attached for recommendations.

 

 

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
Occasional Advisor
Posts: 8
Registered: ‎07-28-2013
Message 3 of 8 (1,574 Views)

Re: P2000 slow since RAID reallocation

Thanks for your reply AnkitM :)

 

All those performance figures were for the RAID-10; prior to and after the extension.

 

Yes, I have been through the Best Practices guide and followed the relevant sections as best we can.

 

I will look into creating a "store.log". Is there a specific team to deal with performance issues in the support dept?

Trusted Contributor
Posts: 131
Registered: ‎02-27-2013
Message 4 of 8 (1,566 Views)

Re: P2000 slow since RAID reallocation

Performance Cases are worked by L1 and L2 Technical Storage Engineer(s) as a collaboration. If required, they may ask for more details about the environment.

 

For Eg. Cabling Diagram, Switch logs (if its not DAS) etc.

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
Occasional Advisor
Posts: 8
Registered: ‎07-28-2013
Message 5 of 8 (1,560 Views)

Re: P2000 slow since RAID reallocation

I think I've found the problem; someone enabled flow control on both the switches. As soon as I disabled flow control, performance is back to ~35MB/s inside VM's, and I/O wait in top on the XenServer hosts is back down and load average is reducing already.

 

I will monitor for 24 hours and and confirm if this is indeed the issue.

Trusted Contributor
Posts: 131
Registered: ‎02-27-2013
Message 6 of 8 (1,555 Views)

Re: P2000 slow since RAID reallocation

Awesome! Thanks for sharing that with us! :)
Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
Occasional Advisor
Posts: 8
Registered: ‎07-28-2013
Message 7 of 8 (1,546 Views)

Re: P2000 slow since RAID reallocation

After 24 hours, this definitely seems like the issue. I/O wait is back to 0.0%, Load Average is back to normal, and disk I/O is also back to acceptable levels. Thanks for your input :)
Trusted Contributor
Posts: 131
Registered: ‎02-27-2013
Message 8 of 8 (1,536 Views)

Re: P2000 slow since RAID reallocation

No Problem. Happy to help! :)

 

If your issue is answered, please remember to mark your question Answered when you get the correct answer or helpful answer and award kudos to the person providing the answer. This helps others searching for a similar issue.

 

Ankit

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.