Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS (1481 Views)
Reply
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 1 of 15 (1,503 Views)

NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

[ Edited ]

Hello,

 

I have some problems with the configuration poll to some Cisco VSS clusters.

The first trigger that there is something wrong was that we got false alarms (node down) from that devices sometimes.

Figuring out the reason of that I saw that the configuration poll to a VSS cluster hang at the point where NNMi gets the FDB table from the switch. After 600 sec. it timed out and I get the false alarm.

 

That looks like the SNMPwalk will bring down the whole snmp service in the VSS so the state poller gets no answer and sends the node down.

 

And now the device is in state "Rediscovery in Progress" since ca. three hours. That can't be normal behavior?!?



Is there any known problem with such devices? I don't know if this is a problem on NNMi side or VSS side.

 

 

Thanks!

 

Sven

Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 2 of 15 (1,494 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

I did some more tests now with other devices. All configuration polls hang and can't finish.

For example on a cisco 2960 switch it breaks after the FdbAnalyzer has completed.

 

Looks like a bigger NNMi problem now?!?

 

 

Sven

Please use plain text.
Respected Contributor
Bharath M R
Posts: 410
Registered: ‎03-10-2010
Message 3 of 15 (1,491 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

are you able to do snmpwalk through command prompt ??

Thanks,
Bharath
Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 4 of 15 (1,489 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

Yes, that is possible on all devices which are in state "rediscovery in progress".

 

Thanks,

 

Sven

Please use plain text.
Respected Contributor
Bharath M R
Posts: 410
Registered: ‎03-10-2010
Message 5 of 15 (1,487 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

check out 

 

             Community configuration --> specific node configuration

 

check out each community string taken by NNMi node wise, some times NNMi takes wrong strings in specific configuration.

 

   check and retry config poll

Thanks,
Bharath
Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 6 of 15 (1,481 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

I checked the communities, they are all okay!

The state polling is working as well at this moment and the communication configuration of each devices is correct.

 

The configuration poll starts right and breaks at a specific place. So I don't think of a problem with the communities.

 

Is it possible that NNMi has an internal problem with the data from the polling?

 

In the system health report I saw a lot of late responses in the SNMP Health Agent from two of our VSS clusters.

 

Thanks!

 

Sven

Please use plain text.
Respected Contributor
Bharath M R
Posts: 410
Registered: ‎03-10-2010
Message 7 of 15 (1,479 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

oh k sorry.. i faced the the similar problem, with community string mismatch in specific node configuration, where me too was able to poll only through command line.

 

 

Thanks,
Bharath
Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 8 of 15 (1,473 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

No problem. Thanks for your reply, Bharath.

 

Today I have a Minor error on Disco Health Agent.

 

There are 199 devices which stuck in discovery process for 24h.

 

That means the configuration polling of the VSS cluster yesterday at 10:37 am causes the whole discovery agent to stuck.

Since that time I have such messages in the jbossServer.log:

 

2011-11-07 10:37:12,859 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id 41a020f:b3f1:4eb65149:19cb80f invoked while multiple threads active within it.
2011-11-07 10:37:12,863 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action 41a020f:b3f1:4eb65149:19cb80f aborting with 1 threads active!
2011-11-07 10:37:13,085 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id 41a020f:b3f1:4eb65149:19cbb6b invoked while multiple threads active within it.
2011-11-07 10:37:13,085 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action 41a020f:b3f1:4eb65149:19cbb6b aborting with 1 threads active!

 

In that thread  I read something about a bug with ID QCCR1B49172 in 9.01 Patch 2.

But I have Patch 4 installed where this failure should be fixed.

 

Any ideas?

 

 

Thanks!

 

 

Sven

Please use plain text.
HP Expert
Darren Hammond
Posts: 530
Registered: ‎09-20-2000
Message 9 of 15 (1,466 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

Hi Sven,

 

If your Cisco devices are replying normally to an snmpwalk, then maybe they are unhappy replying to SNMP getBulk requests.   By default,   NNMi will use getbulk, as in general, it's a more efficient way of retrieving information.

 

As a test,  can I suggest that you open up the "Communication Configuration" and select the "Specific Node Settings" tab?   You could add an entry for one of the problem devices,   fill in the settings that you wish for it but ensure that you uncheck the box "Enable SNMP GetBulk".  Save and close those windows and then select the device and check it's communication settings just to be sure that it has picked those up.

 

Then you can try a configuration poll a couple of more times to see if there is any difference.

 

You might also consider that some devices may work better using a specific SNMP version?   For example,  maybe SNMPV2 communication is an issue with certain devices,  in which case it might be worth setting the various versions as a test using the same method as above (in the specific node settings).

 

If the problem you see appears to lie only within the processing of bridge/FDB tables,  then it's possible that the issue only shows itself when processing per-vlan bridge tables.   NNMi will be sending directed requests to the device using the special community string derived from the vlan information that the devices are reporting. 

 

You could also try running snmpwalks to the device,  for the bridge tables,  but using the various special community strings (commstr@vlanid)  to see if any of them cause the hang situation that you describe. 

 

 Following on from this,  9.01 P3 (which you have already),  included a new file disco.NoVLANIndexing.   This enables you to switch off the per-vlan indexing of a device.   It does mean that you will not get connections derived from the bridge tables,  but that tends not to be an issue for most Cisco devices since they are using CDP or LLDP instead.     You can see the details of how to use this file by taking a look at the 9.01 deployment guide - it's in the section entitled  "Suppressing the Use of VLAN-indexing for Large Switches".   Let me know if you have problems getting your hands on the manual  (you can find them all at http://support.openview.hp.com).

 

So, a few things to try there,  I hope that this helps,

 

Best regards,

Darren

HP Support
If you find that this or any post resolves your issue, please be sure to mark it as an accepted solution.
Please use plain text.
Frequent Advisor
Rick Nichols_2
Posts: 55
Registered: ‎07-12-2002
Message 10 of 15 (1,272 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

We are seeing the same issues here with our our production and test NNMi 9.11, Patch-3, environments. We disabled the SNMP GetBulk as suggested in this thread and also tried the noVlanIndexing thing in test. So far nothing seems to be helping with these. Did you ever find a solution for the issue you were seeing? Thanks in advance for any help you can offer!

Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 11 of 15 (1,134 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

Hi,

 

no, I have no final solution for that, sorry!

 

I changed the polling cycles a little bit and changed the rediscovery cycle from 1 day to 5 days. That was working better till last week. But now I see the same issue again.

I will try the solution with the suppressing VLAN-indexing and give a short reply about the results.

 

But my feeling is that I need to upgrade to 9.1x or 9.2x for getting better discovery performance.

But that is not so easy at the moment for me to do that.

 

Sven

Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 12 of 15 (1,116 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

Hi again,

 

I tried to implement this file disco.NoVLANIndexing with two IP ranges of huge Cisco 6500 switches to exclude them from the FDB analysis.

 

I created the file under /var/opt/OV/shared/nnm/conf/disco/

 

and the put in this lines:

 

10\.100\.102\.0\-255
10\.100\.96\-99\.0\-255

I want to exclude two subnets 10.100.102.0 /24 and 10.100.96.0 /22.

 

Is that the right notation of the file? Which owner should that file have? bin bin as the other files under the folder "disco"?

 

After restarting the services I still get response for FDB from one of the devices in the subnets with a configuration polling. That seems that it is not working....

 

Thanks a lot for a short reply!

 

Sven

Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 13 of 15 (1,106 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

I found the reference pages in another thread here in the forum.

 

http://support.openview.hp.com/selfsolve/document/KM1097957/binary/nnmi_referencepgs_9.10.pdf

 

I didn't use the right notation for the IP addresses.

 

Will try it again!

 

 

Sven

Please use plain text.
Valued Contributor
bebu
Posts: 114
Registered: ‎06-14-2011
Message 14 of 15 (1,099 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

Hi,

 

nnmnoderediscover.ovpl -all or to the specific node is the only solution i got it for this issue, when ever it got struck up in the middle or pending for re-disco in progress. Just check it out!!!

 

 

Bebu

Please use plain text.
Valued Contributor
Sven Hruza
Posts: 229
Registered: ‎09-10-2007
Message 15 of 15 (1,074 Views)

Re: NNMi 9.01 P4 - Problems with snmpwalk to Cisco VSS

The FDB filter is working now.

 

Before the implementation I got more than 4500 entries one some switches and now I get only something about 120 entries.

That's okay from the first view. Now I have to wait if the dicovery and snmp polling is more stable than before.

 

I'll keep this thread updated.

 

 

Sven

Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation