EVA 4400 disk performance (840 Views)
Reply
Occasional Advisor
bernikm
Posts: 9
Registered: ‎04-24-2012
Message 1 of 9 (840 Views)

EVA 4400 disk performance

Hi

 

We're running HP blade system on Hyper-V 2008 R2 cluster with EVA 4400 storage.

EVA has two disk groups

- diskgroup 0 (enclosuer 0) with 10 FC disks 400GB 10k

- diskgroup 1 (enclosure 1) with 8 FC disks 300GB 15k

 

We're dealing with a high impact on disk I/O performance on SQL servers running on Hyper-V hosts on VM (Win2008 + SQL 2008). 

R/W performance is between 20-30MB/s, but only on servers running SQL. On all other VM servers without SQL performance is between 50-60MB/s.

 

We have already consulted with Microsoft support, they didn't find any issues with cluster or VM, also there is no other HW impact on performance. 

The only conclusion by our appinion is that SQL servers are generating so many I/O requests, that EVA or HBA on hosts can't handle that.

Is this true, or how can we check that ? SQL profiler show normal activity, also there is no other issues with operating sistem, so we're certain that there must be some kidn of connection between VM - HOST - EVA.

Also running RW test directly on a host connected to EVA, Read results are between 200-300MB/s, but Write is between 60-100MB/s.

 

Any additional points how to proceed next are welcome.

 

Regards,

Mike

Please use plain text.
Honored Contributor
Jan Soska
Posts: 1,225
Registered: ‎01-22-2008
Message 2 of 9 (801 Views)

Re: EVA 4400 disk performance

Hello,

what raid level do your luns use? Generally you have really small number of disks there. But - if your luns perform directly ~300MB/s in read and 100MB/s in write and under virtualisation layer it it so different the problem is virtualization layer very probably...

Please use plain text.
Occasional Advisor
bernikm
Posts: 9
Registered: ‎04-24-2012
Message 3 of 9 (793 Views)

Re: EVA 4400 disk performance

LUN us RAID5, we did some testing also on RAID1 LUN but the performance is also low.

Yes, at first we thought that the problem is virtualization layer, but then we did some testing on DAS an iSCSI disks and we got higher RW as on SAN.

We also did testing on different virtualization platforms (Hyper-V 3.0, VMware 5) with the same results.

I think that there is a high IO from virtual machines to SAN, so that the EVA can't handle it, because moving the same VM on a DAS it gets higher RW ratio.

Is there any way to check how many IO requests come from VM throught Hypervisor to SAN ? And how many of them can EVA handle ?

Please use plain text.
Occasional Advisor
WissfeldA
Posts: 10
Registered: ‎10-06-2012
Message 4 of 9 (776 Views)

Re: EVA 4400 disk performance

On the VMware level you can use ESXTOP to analyze the IO your VMs are requesting:

 

VMware KB: Using esxtop to identify storage performance issues

 

On the EVA level you should use "Performance Data Collector" (formerly HP Command View EVAPerf )


EVAPERF should already be installed alongside your Command View installation.

Just start evaperf and take a look at the parameters - it's pretty self-explanatory.

 

evaperf vd -cont 5

 

for example shows the current statistics of all VDisks (vd) and refreshes every 5 seconds (-cont 5).

Please remember though that this is the sum of ALL IO on this vdisk, to narrow down VM-specific IO loads you need to ensure that only 1 VM is running on this VDisk/Datastore.

Other parameters that might be of interest for you are: vdrl vdwl (latencies)

Please use plain text.
Occasional Advisor
bernikm
Posts: 9
Registered: ‎04-24-2012
Message 5 of 9 (758 Views)

Re: EVA 4400 disk performance

[ Edited ]

Hi WissfeldA

 

Thank you for valuable informations.

We have 2 disk groups/datastores. Datastore0 has 11 VM, and Datastore1 has 4 VM. Eachn VM is running on its own LUN.

 

So if I got whis right, the best practice will be to shut down all VM and then start-up one by one and collect informations about storage performance with

- evaperf vd, vdrl, vdwl ?

 

I have already run evaperf with all the parameters, and here are results below. In time of collecting data, all VMs are running. I copied only VM that are causing 'high' R/W? Are these numbers below normal or critical ?

 

Regards

 

evaperf vd
ID  Read   Read    Read  Read   Read    Read Write  Write   Write  Flush Mirror Prefetch                Group DRM       Online Mirr Wr   Ctlr Redundancy NumberOf      Preferred     LUN                                     Node      
     Hit    Hit     Hit  Miss   Miss    Miss Req/s   MB/s Latency   MB/s   MB/s     MB/s                   ID Group     To          Mode                 Presentations Path                                                            
   Req/s   MB/s Latency Req/s   MB/s Latency                 (ms)                                                                                                                                                                      
                   (ms)                 (ms)                                                                                                                                                                                           
-- ----- ------ ------- ----- ------ ------- ----- ------ ------- ------ ------ -------- -------------------- --------- ------ ---- ---- ---- ---------- ------------- ------------- --------------------------------------- ----------
13    39   2.50     0.2   174   7.92     0.9     0   0.00     0.0   0.00   0.00     2.07 Default Disk Group 1 Nonmember A      Yes  Back R026 vraid5     3             no_preference Cluster_005_DG1                      EVA001
 0   114   7.44     0.5   171   4.85     5.1     2   0.01     0.1   0.06   0.06     8.64   Default Disk Group Nonmember A      Yes  Back R026 vraid5     3             no_preference Cluster_003                           EVA001


evaperf vdrl
ID     0   1.6   3.3   6.6    13    26    52   105   210   420 Ctlr LUN                                     Node      
       <     <     <     <     <     <     <     <     <    <=                                                        
     1.6   3.3   6.6    13    26    52   105   210   420   INF                                                        
-- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- --------------------------------------- ----------
14    40     2     8     3     0     0     0     0     0     0 R026 Cluster_009_DG1                       EVA001
 13   820    16    24    10     4     0     0     0     0     0 R026 Cluster_005_DG1                      EVA001
 0  1246    78   224   334    31     9     0     0     0     0 R026 Cluster_003                           EVA001

 

evaperf vdwl
ID     0   1.6   3.3   6.6    13    26    52   105   210   420 Ctlr LUN                                     Node      
       <     <     <     <     <     <     <     <     <    <=                                                        
     1.6   3.3   6.6    13    26    52   105   210   420   INF                                                        
-- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- --------------------------------------- ----------
14   200   185   508    30     0     0     0     0     0     0 C09V Cluster_009_DG1                       EVA001
14   236   352   314     0     0     0     0     0     0     0 R026 Cluster_009_DG1                       EVA001
 0    87     1     0     0     0     0     0     0     0     0 C09V Cluster_003                           EVA001
 0    41     0     0     0     0     1     0     0     0     0 R026 Cluster_003                           EVA001

 

 

 

 

Please use plain text.
Occasional Advisor
WissfeldA
Posts: 10
Registered: ‎10-06-2012
Message 6 of 9 (743 Views)

Re: EVA 4400 disk performance

Hi Mike,

 

just to clarify things ...

 

  1. You're running 2 EVA diskgroups called "Default Disk Group" and "Default Disk Group 1"
    Which one is the 8x15k, which one the other ?
  2. Those two EVA diskgroups hold different vDisks - one holds 11, the other one 4 ...
    Which Raid-Levels do you use there ? "Cluster_003" and "Cluster_005_DG1" are in vRaid 5, the other ones too ?
  3. Do you currently use vSphere or Hyper-V ?

According to your logs, i do not see any unusual entries.

Can you please post the following as well : EVAPERF PDA ; EVAPERF PDG; EVAPERF VDTS ?

Maybe we're able to pinpoint stuff further from here ... although i also suspect the low # of spindles being the root cause for low write throughput and/or write IO/sec.

 

And please also have a look at the following perfmon counters on your problematic servers as well :

  • Physical Disk: Avg. Disk Queue Length (Read Write as well)
  • Physical Disk: Disk Reads/sec - Writes/sec - Transfers/sec
  • Pyhsical Disk: Split IO/sec

Kind regards

Alex

Please use plain text.
Occasional Advisor
bernikm
Posts: 9
Registered: ‎04-24-2012
Message 7 of 9 (729 Views)

Re: EVA 4400 disk performance

Hi WissfeldA.

Thank you for you reply and informations. Here are some additional informations

 

1. Default Disk Group 10 FC disks 400GB 10k
    Default Disk Group1 8 FC disks  300GB 15k

 

2. Default Disk Group holds 10 vDisks
    Default Disk Group1 holds 4 vDisks
    All vDisks are RAID 5
    We did some testing with additional vDisk on RAID 1, but performance is not been increased

 

3. We use Hyper-V 2008 R2 (3 node cluster).
    We did some testing on vShpere 5, performance was quite the same as on Hyper-V


Reports from evaperf

 

PDA
Enc. Bay__1 Bay__2 Bay__3 Bay__4 Bay__5 Bay__6 Bay__7 Bay__8 Bay__9 Bay_10 Bay_11 Bay_12 Bay_13 Bay_14 Node      
       MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s   MB/s           
---- ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ------ ----------
   2   0.26   0.32   0.44   0.38   0.33   0.36   0.24   0.25      -      -      -      -      -      - EVA001
   1   1.99   2.26   2.27   2.46   2.60   2.66   2.25   3.06   2.33   2.78      -      -      -      - EVA001

 

 

PDG
Disk                 Average Average Average Average Average Average Average Average Number Ctlr Node      
Group                  Drive   Drive    Read    Read    Read   Write   Write   Write of                    
                       Queue Latency   Req/s    MB/s Latency   Req/s    MB/s Latency Disks                 
                       Depth    (ms)                    (ms)                    (ms)                       
-------------------- ------- ------- ------- ------- ------- ------- ------- ------- ------ ---- ----------
Default Disk Group         0       -       0    0.00     0.0       0    0.00     0.0 10     C09V EVA001
Default Disk Group         3       -      46    0.94     4.1       5    0.25    25.9 10     R026 EVA001
Default Disk Group 1       0       -       0    0.00     0.0       0    0.00     0.0 8      C09V EVA001
Default Disk Group 1       7       -      63    3.16    10.2      50    3.66    19.7 8      R026 EVA001

 


VDTS
ID     0    2K    4K    8K   16K   32K   64K  128K  256K  512K Ctlr LUN                                     Node      
       <     <     <     <     <     <     <     <     <    <=                                                        
      2K    4K    8K   16K   32K   64K  128K  256K  512K   INF                                                        
-- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ---- --------------------------------------- ----------
 3     0     0     1     0     0     0     0     0     0     0 C09V 6001-4380-024D-3B98-0000-7000-0005-0000 EVA001
 3     0     0     2     0     0     0     0     0     0     0 R026 6001-4380-024D-3B98-0000-7000-0005-0000 EVA001
11     0     0     0     0     0     0     0     0     0     0 C09V 6001-4380-024D-3B98-0000-7000-0015-0000 EVA001
11     0     0     0     0     0     0     0     0     0     0 R026 6001-4380-024D-3B98-0000-7000-0015-0000 EVA001
12     2     0    10     3     1     0     1     0     0     0 C09V Cluster_001                           EVA001
12     0     0     6    12     1     0     1     0     0     0 R026 Cluster_001                           EVA001
10     0     0     5     0     0     0     0     0     0     0 C09V Cluster_002                       EVA001
10     0     0     3     1     1     0     0     0     0     0 R026 Cluster_002                       EVA001
 5     0     0     0     0     0     0     0     0     0     0 C09V Cluster_004                           EVA001
 5     0     0     0     0     0     0     0     0     0     0 R026 Cluster_004                           EVA001
 7     0     0     2     0     0     0     0     0     0     0 C09V Cluster_006                            EVA001
 7     0     0     2     1     0     0     0     0     0     0 R026 Cluster_006                            EVA001
 2     0     0     0     0     4     1     0     0     0     0 C09V Cluster_007                           EVA001
 2     0     0     0     0     4     0     0     0     0     0 R026 Cluster_007                           EVA001
15     0     0     0     0     0     0     0     0     0     0 C09V Cluster_008                           EVA001
15     0     0     1     0     0     0     0     0     0     0 R026 Cluster_008                           EVA001
 6     0     0     0     0     0     0     0     0     0     0 C09V Cluster_010                         EVA001
 6     0     0     0     0     0     0     0     0     0     0 R026 Cluster_010                         EVA001
14     0     1     2    20     2    29   545     0     0     0 C09V Cluster_009_DG1                       EVA001
14     0     0     2    66     6    75  1754     3     0     0 R026 Cluster_009_DG1                       EVA001
 4    18     5     1     1     1     0     0     0     0     0 C09V Cluster_011                            EVA001
 4    17     3     3     8     0     0     0     0     0     0 R026 Cluster_011                            EVA001
13     0     0     1     1     0     0     0     0     0     0 C09V Cluster_005_DG1                      EVA001
13     0     0     1     0     0     0     0     0     0     0 R026 Cluster_005_DG1                      EVA001
 0     0     0     5     1     0     2     0     0     0     0 C09V Cluster_003                           EVA001
 0     8    34   647   148   135    27   124     0     0     0 R026 Cluster_003                           EVA001
 9     0     0     0     0     0     0     0     0     0     0 C09V Cluster_Witness                     EVA001
 9     0     0     0     0     0     0     0     0     0     0 R026 Cluster_Witness                     EVA001
 1     1     0     1     1     0     0     1     0     0     0 C09V Cluster_012_DG1                      EVA001
 1     0     0     3     0     0     0     1     0     0     0 R026 Cluster_012_DG1                      EVA001

 

 

 

Please use plain text.
Occasional Advisor
WissfeldA
Posts: 10
Registered: ‎10-06-2012
Message 8 of 9 (714 Views)

Re: EVA 4400 disk performance

Hi Mike,

 

as far as i see there are no obvious problems - but those EVAPERF captures only show a brief timeslice.

As a matter of fact your EVA is running in almost the lowest possible configuration with 8-10 pDisks per DG which tends to be limited in terms of IO/.

Although i do do not see the EVA being saturated with request in your samples, i highly assume the # of pDisks being the main problem.

You're running 10 spindles (in vRaid5) which leads to "theoretical SAFE IOPS of 600/sec". You're hosting 10 LUNS on this diskgroup which increases the overhead a little bit .. my guess is that your "Default Disk Group" is capable of ~500 IOPS in total for all VMs as a realistic estimate. (Default DiskGroup1 should even be a little bit lower, for an explanation google: HP_EVA_Calculator_v3.xlsx)

 

On the other hand i'm not quite sure what the actual problem is ... do you suffer from performance problems ?

If yes, then how do they show up ?

 

I can easily flood a large EVA array with 512b I/O leading to a "R/W bandwidth" of only a couple of MB/Sec.

So please understand that " [...] R/W performance is between 20-30MB/s, but only on servers running SQL [...]"  may not be enough. How did you measure this R/W performance (simple cut&paste or using IOMeter/<insert tool here>) ?

Also please have a look inside of the problematic VMs' PERFMON counters, especially the "physical drive Queue" and probably the IO counters mentioned in my last post.

 

Since i'm not very skilled on Hyper-V, i do not know how to gather those on the Hyper-V cluster, especially when those are in Core Installation mode, but those WMI counters should be present on the hypervisors as well.

 

Sorry that i could not help you quicker, but it's sometimes quite tedious without having "hands-on" ...

 

Regards

 

Alex

 

 

Please use plain text.
Occasional Advisor
bernikm
Posts: 9
Registered: ‎04-24-2012
Message 9 of 9 (703 Views)

Re: EVA 4400 disk performance

Hi Alex

 

Thanks again for all informations and points. You have give me some clues about diskgroups and disk IO throughput. Thanks !

 

We didn't use IOMeter, we use SQLIO tool with parameters below.

Can you reccommend us any other tool that we could use for testing ?

 

sqlio.exe -kR -t50 -s30 -f64 -b32 -i64 testfile.dat

Running 50 threads for 30 sec from 1GB testfile.dat using 32KB IOs over 2048KB stripes with 64 IOs per LUN

 

Doing the same test with 2KB IOs the results are 10x lower.

Since MS SQL is doing best on a 8KB, we did some testing also, but the result were below 1MB/s for write, and about 2-3MB for read.

 

We also presented a blank LUN to a host and did the same test on it, but the results were the same.

It looks like that all VM running on cluster are exceeding 500 IOPS that EVA can handle, but on the other hand EVAperf is not showing any difficulties with that.

 

During the weekend we wil try to shut down all VM on cluster and do some testing on EVA, and then try to run VM one-by-one and do testing if any of this VM is causing such an impact on disk performance.

I know it's a show in the dark, but we'll give it a try, what do you think ?

 

Thank again for all your help and knowledge.

 

Regards,

Miha

 

Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation