02-05-2013 05:17 AM
Recently, we purchased a HP MSL2024 Ultrium 6520 single drive LTO-6 Tape Library. We have connected it to a P800 controller inside of a HP DL360 G6. We also have a MSA60 connected to the P800.
All devices are latest firmware.
SAS Tape Library is hooked up to the second port on the P800 (I know that Port 1 is on a SAS expander as per a HP advisory doc). The MSA60 is hooked up to port 1 on the P800, and is working fine.
So when we went to set this up and use it for the first time, all went well until running our first backup. We are using Backup Exec 2012.
Backups have been failing with numerous errors, but are always associated with the device timing out (device disappears from the server).
When the server and tape library are start properly, it's available to the system. When we run a backup, one of these timeout errors occur, and I notice that the tape library actually disappeared out of the Device Manager. The event log on the server also indicates that the tape drive has been removed without properly being prepared for disconnection.
Essentially what is happening is that during a backup, the tape drive "Disappears" from the server. Restarting the library resolves this, and makes it visible once again.
At first I thought this was on high I/O load, until I did some testing. I had a backup running at around 4,800MB/min, and I was also copying HUGE dad to/from the MSA60 with read/write speeds of 100MB/sec. Did this for hours, and the tape library was still backing up with no issues. Later on that night with no load, the backup job errored out again. Before the backup failed, the backup had backed up over 1.4TB of data to a LTO-6 cartridge.
I created an HP case, we analysed logs. I already have the proper registry entries configured to disable AutoRun (to stop the polling issue), so that woudln't have an affect.
As of last discussion, HP said the library isn't compatible with the P800 controller because it's not in the quick spec documents, but after looking at all HP controllers quick specs, it looks like HP doesn't have any controllers that list the new HP Ultrium 6520 LTO-6 drives. I'm assuming it is compatible, but it's just that the documents haven't been updated. I doubt HP would sell a library that you can't use yet... Someone on the storage team told me I have to wait for HP to release a supported controller (I was like, are you friggin kidding me?), since one isn't released.
One more note, in the MSL2024 web interface, it reports that the drive is connected via SAS, until the backup fails in which case the web interface says the drive isn't detected. So both Device Manager on server, and web interface for MSL reports SAS connectivity is being lost somewhere. There's no specific errors.
Anyone have any recommendations? HP analysed the LTT logs, etc... and are finding no problems. By the way the server is in great condition and super clean (as far as drivers, issues, software, is concerned). Backup exec was previously configured for a different MSL2024 using SCSI which worked great forever, so I positive it's nothing to do with Backup Exec, especially since the tape drive is disappearing from the system.
Thanks in advance
02-05-2013 05:29 AM - edited 02-05-2013 05:31 AM
Few more notes...
Event log errors, when device disappears:
A tape drive connected to port 2E of array controller P800 located in server slot 2 has been removed.
The device 'Hewlett Packard LTO Ultrium-6 drive' (SCSI\Sequential&Ven_HP&Prod_Ultrium_6-SCSI\5&863e
The device 'Hewlett Packard MSL G3 Series library (x64 based)' (SCSI\Changer&Ven_HP&Prod_MSL_G3_Series\5&863ed4f&
I've also tried changing the cable with a new one, same thing happens.
Did a drive test with LTT and it reports everything is fine.
I've spoke with 3 HP spport guys (2 tape, and 1 storage):
First guy was awesome (Tape Team), and spent 6+ hours on the phone with me, and numerous e-mails, he said my config is supported, but not recommended with having the MSA60 on the same controller, he's replacing the tape drive. Unfortunately since this is new hardware just released, none are in Canada, and even with a 5 hour response care pack, it could be a while before we get the drive since they have to ship it in from the United States. I'm getting the feeling it'll be over a week before we get the replacement.
Second guy (storage), said that since the new Ultrium 6 LTO drive isn't inside the Quick specs for the P800, that it's not supported.
Third guy (Tape team), didn't really know what to say. He tried to help as much as he could, but was limited because of the documents. He said he couldn't say for sure if it was supported or not. Tried to help as he could, but I ended up ending the phone conversation just because we weren't getting anywhere. It was ultimately left that I should not use the MSL2024 library until HP releases a "Supported" SAS card that suppots the Ultrium 6520 Tape Drive/Library...
At first I thought it could be a P800 controller issue, but the MSA60 has been rock solid. Never disappears, and we've had no problems. Even with the Tape Library disappears, all is good with the MSA60. All we have to do is restart the library and it's re-detected by the system.
I'm pulling my hair out with this one. Ugh. One last thing, I've used both HP drivers, and the Backup Exec Tape drivers, and both have the same result. I've also disabled all the Insight Management Agents, with no change to the backups...
02-05-2013 08:55 AM
Looks like this is version 7.24(B).
MSL firmware should be 5.80.
Dou you already have these versions?
Hope this helps!
There are only 10 types of people in the world -
those who understand binary, and those who don't.
No support by private messages. Please ask the forum!
If you feel this was helpful please click the KUDOS! thumb below!
02-05-2013 05:37 PM
Here's one more thing to mention...
On a older MSL2024 LTO-4 connected over SCSI, we actually had way faster transfer rates, then with the new setup when it is backing up before it fails... I don't know if that helps with the diagnosis?
02-06-2013 02:21 AM
Ultrium Tape devices have a SCSI history buffer (SAS is Serial Attached SCSI). This buffer is dumped to a HP L&TT support ticket when it is created. HP Support should be able to read this and other IO related buffers to see if there are anomalies reported like Resets, Aborts, CRC errors etc.
I have no experince with CCISS driver event logging in WIndows but I could imagine that the driver supports a more verbose logging level which could yield hints on what is happening ?
02-06-2013 04:52 AM
We generated a support ticket and HP said that there was nothing inside of the support ticket that would show anything was wrong. Would this show the tape drive disconnect? Cuz that wasn't mentioned when the HP tape engineer went over the LTT support ticket.
It was noted that the tape drive had a poling warning, but we have all the proper registry entries created "AutoRun" set to 0, also we have used the Backup Exec drivers too which should take care of this for us.
HP said everything was fine with the LTT support ticket.
02-12-2013 11:36 AM
I've experienced a similar issue of the tape library disappearing in my old existing server (DL380-G5 with dual LTO-2 version of MSL2024) but it is U320-SCSI connected and I'm running a different backup software (NetVault). My spontaneous loss of the library under Windows 2003 in that system was eventually diagnosed as Windows' built-in Removable Storage Manager hijacking the library and once I disabled RSM from trying to use the library, my backup software no longer loses the connection to the library. Every once in a while, after a reboot, sometimes Windows 2003 re-enables the RSM and I have to go back in and disable it.
In my new server, I'm now seriously contemplating getting two separate Smart Array controllers... a p420 to handle the internal 25-disk cage and a p421 just to connect the external MSL2024. That's kind of a waste of an expensive SmartArray controller just to host a single tape library, but in the big picture, this will be an important server and I guess an extra $600 controller card is probably worth it.
02-13-2013 06:29 AM
Ya it is scary. We have 4 hour response but it's been a week and a half to get the new drive and we still don't have it... (They had to ship it in from the US apperently).
I'm hoping its the tape drive. In our case we don't have any software creating polling issues. It just chooses to disappear from the system after it chugs through 700GB of a backup. I can't find anything that shows what it could be.
And HP has no idea what it could be, thankfully the first guy said its a supported config and initiated the tape drive replacement, but all other HP people said that HP has no controllers right now that support the LTO6 libraries and that we have to "wait" for one to be released before we can use the new library (is this a joke? lol).
Hopefully the tape drive comes soon and I can rule out the tape drive being faulty. I still have no idea why a p800 wouldn't be able to have an MSA60 and a MSL2024 attached to it. An SAS device, is an SAS device...
And just for an update, I still have no updates, it's still not working. I wish someone could chime in with some experience, or with a similiar config :(
02-13-2013 12:20 PM - edited 02-13-2013 12:23 PM
I'm right now waiting on a callback from an SA for pre-sales tech questions on this topic. While speaking to my govt account rep about this, he said he remembered a similar case that was solved by going with a separate Smart Array controller dedicated to the tape library.
In my old DL380-G5 LTO2 system as reported in my previous post, I upgraded the library to a dual LTO5 SAS version of the MSL2024 back in December, and employed a Smart Array P411 controller card dedicated to the library only. Even though the G5 machines are not listed on the hardware compatability list as officially supported by the P411 card (only supported on G6 and later), I found a web document where HP had installed one in a DL380-G5 and attached an external disk array chassis to measure the SAS 6g speeds in a thruput test in a MS Exchange server environment, so I bought the P411 card anyway to connect the new library. So far it's been running perfectly fine every day since about the 2nd week of December 2012.
Even though your P800 card should theoretically support both your disk array and tape library simultaneously, maybe it has some hidden internal design issues when connecting SAS systems with vastly different traffic flow and command queing behaviours such as disk vs tape. It might be worth a test to see if a dedicated controller card would solve the problem. You might ask your HP support engineer assigned to your case if they could send you a test P411 card to see if it fixes the problem... that is, if you've got a PCIe slot open for another controller card.
02-14-2013 06:04 AM
Got the new tape drive yesterday... Tried it out, and still having issues.
The backup does 1.19TB of a 1.2TB backup job then fails.
Checking backup exec logs, it reports:
 02/13/13 22:26:41 Adamm Mover Error: DeviceIo: 03:07:00:00 - Device error 1117 on "\\.\Tape0", SCSI cmd 0a, 1 total errors
So, I'm now ready to try putting it on it's own controller card... The problem is, HP told me that no compatible controller cards are available yet to connect the MSL2024 LTO-6 tape library... Is this true?
I was hoping to avoid this since we have no free spots left in our DL360 G6. I'll have to remove a SCSI card that we have installed that we need for something else :(
02-20-2013 02:11 PM
Well the issue is finally fixed...
During the first initial phone call weeks ago. I was running firmware version 5.80. Even though 5.90 was available, we tried flashing it, but it was coming up with a certificate error and the library was not accepting it.
The HP tech told me not to worry as this is a minor update containing nothing critical, and that 5.80 can be treated as current...
I called back last night, since they closed the case just wondering what the heck was going on. I was just about to purchase a controller that I had no idea if it was going to work. Didn't know what would work since the second HP tech on the second call mentioned that currently no HBA or Smart Array controllers are compatible with the new LTO-6 tapes, and that I'd have to wait for HP to release one...
I mentioned that to the individual I spoke with last night, and he said he'd escalate it to higher engineers. E-Mailed me later that night saying:
Firmware needs to be at 5.90 in order to support LTO-6 tapes. And said some other stuff including listing a RAID controller that was compatible.
I jaw dropped. I couldn't beleive it. First, that the pre-configured MSL2024 with a half height LTO-6 drive would ship with a firmware that doesn't support it, and secondly that I've wasted over 50 hours of time trying to trouble shoot this issue.
Since we couldn't flash it in the first place with LTT. I logged in to the MSL web interface, and updated the firmware there.
It rebooted, I restarted Backup Exec services, confirmed the Symantec drive was loaded for the newly recognized Tape Drive/Library.
I ran a mini backup (130GB versus a full backup which is around 5TB), and BAM it completed. I was shocked, usually the backups failed right at the end.
I waited till later that night to initiate a full backup. Server 1, backed up, server 2 backed up. The whole backup job completely 100%.
So just to confirm:
1. A P800 controller that's connected to a MSA60, plus a MSL2024 LTO-6 SAS library works 100%
2. You need to be running firmware 5.90 or higher for it to support LTO-6 drives/libraries
3. I think backup exec needs to be 2012 SP1a or higher with the latest device drivers to support LTO-6.
Even when backing up the MSA60 unit with 12 drives RAID6, to the MSL2024 LTO-6 which is on the same controller, it still worked 100% and the MSA60 read was maxed out. No issues whatsoever!
Thought I'd post this in case any others are having this problem. This is how you fix it, and thanks to the others who chimed in and tried to help!
03-25-2013 12:30 PM
Hi. I also got an MSL2024 (LTO6) in combination with a HP 822 controller, connected to Linux server and got same issue.. Releasenotes for MSL 5.80 said there was support for LTO6 drives, but HP did pull back the firmware and marked it as not supported anymore. It's obvious why...