Schedule Job via DP runs into Timeout. But which timeout? (973 Views)
Reply
Occasional Advisor
BenFisch
Posts: 9
Registered: ‎03-27-2014
Message 1 of 9 (973 Views)
Accepted Solution

Schedule Job via DP runs into Timeout. But which timeout?

[ Edited ]

Hi, i want schedule a Database Job via HP-DP. I´ll get the following error message during the job.

I´ve posted every 10 minutes a command "...in progress" to prevent the ScriptoutputTimeout.

The default settings of this option in global conf are 15 minutes.

 

My question: Which timeout i have to change to prevent stopping this job without affecting my dayli backup jobs.

 

 

 

[Normal] From: VBDA@lnx-rbg01.xxxxxxxxxxx.com "Database Validation EADP"  Time: 24.03.2014 10:08:51

      STARTING Disk Agent for lnx-rbg01.xxxxxxxxx.com:/tmp "Database Validation EADP".

 

24.03.2014 10:09:19     create cloned volumes   succeeded (00:00:39)

24.03.2014 10:09:58     mount cloned volumes    succeeded (00:00:01)

24.03.2014 10:09:59     recover database  succeeded (00:00:35)

24.03.2014 10:10:34     validate database

24.03.2014 10:11:34     ... in progress  

24.03.2014 10:21:34     ... in progress  

24.03.2014 10:31:34     ... in progress  

24.03.2014 10:41:34     ... in progress  

24.03.2014 10:51:34     ... in progress  

24.03.2014 11:01:35     ... in progress  

24.03.2014 11:11:35     ... in progress  

24.03.2014 11:21:35     ... in progress  

24.03.2014 11:31:35     ... in progress  

24.03.2014 11:41:35     ... in progress   

24.03.2014 11:51:35     ... in progress  

24.03.2014 12:01:35     ... in progress  

24.03.2014 12:11:35     ... in progress  

[Major] From: BSM@ux-rbg11.xxxxxxxxx.com "EADP_VALIDATE"  Time: 24.03.2014 12:29:23

[61:1002]  The BMA named "ORA_BACKUP_2" on host ux-rbg11.xxxxxxxxx.com

      reached its inactivity timeout of 8400 seconds.

      The agent on host will be shutdown.

 

24.03.2014 12:21:35     ... in progress  

[Major] From: BSM@ux-rbg11.xxxxxxxxxxx.com "EADP_VALIDATE"  Time: 24.03.2014 12:36:48

[61:1002]  The VBDA named "Database Validation EADP" on host lnx-rbg01.xxxxxxxx.com

      reached its inactivity timeout of 300 seconds.

      The agent on host will be shutdown.

HP Expert
Bob_Clark
Posts: 1,499
Registered: ‎08-14-2013
Message 2 of 9 (929 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

You are talking about 3 different things here

 

  -   prevent the ScriptoutputTimeout.

The default settings of this option in global conf are 15 minutes

 

You correctly identified this as being in the 'global' file.  This timeout is used when running pre-exec or post-exec scripts, and the 15 minute default timeout is not enough to allow the script to finish.  I suggest that you run the script in stand-alone mode, aoutside of DP, and see how long it takes.  Then, in the 'global' file, find the line

 

# ScriptOutputTimeout=TimeoutInMinutes

 

make a copy of the line, edit one of the lines to remove the # sign and the 'space' that follows, and set it to the value you want, for example, to increase the timeout to 30 minutes

 

ScriptOutputTimeout=30

# ScriptOutputTimeout=TimeoutInMinutes

 

Save the file, stop and restart DP

 

=================

 

[61:1002]  The BMA named "ORA_BACKUP_2" on host ux-rbg11.xxxxxxxxx.com

      reached its inactivity timeout of 8400 seconds.

 

Tis can usually be reolved by implementing KeepAlive on the server giving the error, in your case 'ux-rbg11'.  I wrote several support tips on implementing KeepAlive on different platforms, you may be able to find one that is appropriate for you by searching the forum for 'KeepAlive'.  Let me know if you can't find it, and I will send you a copy

 

I would really suggest that you implement KeepAlive on servers pro-actively, but, at least, I woudl do it also on any disk agents involved with this backup

 

================

 

[61:1002]  The VBDA named "Database Validation EADP" on host lnx-rbg01.xxxxxxxx.com

      reached its inactivity timeout of 300 seconds.

 

If this is from the same session as the previous error, it may straighten itself out once you get KeepAlive implemented, and are no longer getting the 8400 second MA timeout.  However, if not, again, in the 'global' file, find this line

 

# SmPeerID=WaitForInMinutes

# default: 5 minutes

 

Edit as before, set the value to no more than 15 minutes, but, it will probably need a lot less than that

Occasional Advisor
BenFisch
Posts: 9
Registered: ‎03-27-2014
Message 3 of 9 (918 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

[ Edited ]

Thank you for your answer.

I´ve implemented the script as post script.

Means DP will backup nothing, just execut the script.

 

But one question. Some of our DB´s are bigger than 2 TB. The Validation Job runs 10 hours and more.

Does this mean, i have to set the ScriptOutputTimeout to 600 and more?

Does this timeout ignore the output of my script:"in progress" ?

I thought the timeout will check the output every 10 minutes and will reverse the timeout?

 

 

HP Expert
Bob_Clark
Posts: 1,499
Registered: ‎08-14-2013
Message 4 of 9 (903 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

The ScriptOutputTimeout is a one-time, finite value.  You can set this to be as high as you need it, but, if the script does not conclude in that time period, you will get the timeout error.  There is no periodic checking

 

If you are running this as a Post-exec script, the actual backup should have completed before it starts, but the sessin will show as 'In Progress' until the Post-exec script finishes

Occasional Advisor
BenFisch
Posts: 9
Registered: ‎03-27-2014
Message 5 of 9 (884 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

[ Edited ]

I´ve tested your solution, but still got the error.

 

09.04.2014 14:53:58 create cloned volumes succeeded (00:00:48)
09.04.2014 14:54:46 mount cloned volumes succeeded (00:00:01)
09.04.2014 14:54:47 recover database succeeded (00:00:47)
[Major] From: BSM@ux-rbg11.xxxxxxx.com "CIMP_VALIDATE" Time: 09.04.2014 16:56:05
[61:1002] The VBDA named "Database Validation CIMP" on host lnx-rbg01.xxxxxx.com
reached its inactivity timeout of 7200 seconds.
The agent on host will be shutdown.

 

These are my settings:

Cellmanager:

ux-rbg11:/.root#ndd -get /dev/tcp tcp_keepalive_interval
600000

 

Client:

# echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time
# echo 60 > /proc/sys/net/ipv4/tcp_keepalive_intvl
# echo 20 > /proc/sys/net/ipv4/tcp_keepalive_probes

 

[root@lnx-rbg01 /opt/omni]# grep OB2IP .omnirc
OB2IPCKEEPALIVE=1

 

ux-rbg11:/opt/omni#grep OB2IPC .omnirc
OB2IPCKEEPALIVE=1

 

After changing the value i´ve restarted the omnisv.

 

Do you have any ideas what went wrong?

 

 

 

Trusted Contributor
KasparsB
Posts: 48
Registered: ‎03-07-2013
Message 6 of 9 (875 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

To prevent media agent and disk agent timeouts:

1) 

[Major] From: BSM@ux-rbg11.xxxxxxxxx.com "EADP_VALIDATE"  Time: 24.03.2014 12:29:23

[61:1002]  The BMA named "ORA_BACKUP_2" on host ux-rbg11.xxxxxxxxx.com

      reached its inactivity timeout of 8400 seconds.

      The agent on host will be shutdown.

2)

[Major] From: BSM@ux-rbg11.xxxxxxx.com "CIMP_VALIDATE" Time: 09.04.2014 16:56:05
[61:1002] The VBDA named "Database Validation CIMP" on host lnx-rbg01.xxxxxx.com 
reached its inactivity timeout of 7200 seconds.
The agent on host will be shutdown.

 

you must edit the global options file.

Look for values:

1)

SmMaIdleTimeout=

# default:  140 minutes

2)

SmDaIdleTimeout=

# default: 120 minutes

 

BSM messages display these in seconds, but are configured in minutes in the global options file.

 

Why exactly are you trying to do validation in backup session?

If you do validataion in your backup session, the drive will be busy all time.

If you only wan to see validation job messages maybe you can just execute validation seperately and write it to log-file, then cat the log-file as a pre- or post- backup script.

 

Occasional Advisor
BenFisch
Posts: 9
Registered: ‎03-27-2014
Message 7 of 9 (872 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

I use the HP DP for this topic as "Scheduler" nothing else.

The validation script mounts a netapp snapshot and validate the clone of the db.

 

Does this changes impact my "normal" backup?

Filesystem, Linux/Windows ? 

 

For db backup i use Netapp Snapcreator scheduled by pre script in DP.

 

Trusted Contributor
KasparsB
Posts: 48
Registered: ‎03-07-2013
Message 8 of 9 (869 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

[ Edited ]

This does not impact any backups at all if everything is working.

These timeouts specify how long it should wait when there is no data input or backup activity.


In your case Media Agent did not receive any backup data from Disk Agent and timed-out.

This is implemented to prevent session not to finish in case of hung MA or DA.

Occasional Advisor
BenFisch
Posts: 9
Registered: ‎03-27-2014
Message 9 of 9 (843 Views)

Re: Schedule Job via DP runs into Timeout. But which timeout?

thx, now it works.

The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.