Re: Performance for Metrics iSPI, ETL Services not starting (426 Views)
Reply
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 1 of 17 (643 Views)

Performance for Metrics iSPI, ETL Services not starting

We had a power outage and the NNMi/PerfSPI server was shutdown improperly, we booted the server up after power came online and then stopped all of the services and shutdown the server until power was more stable.  Now we have the server back up and running yet our Performance for Metrics SPI is not functioning properly now. 

 

It says ETL services are RUNNING but right below that things like :

 

Component_Heath is listed as NOT OK ... and this goes for everything listed there.  I can't post the logs here because of the environment.

 

I have tried everything to try and fix this issue, and nothing seems to work.  Any help or ideas would be greatly appreciated.  We are on version 9.20 of both NNMi and PerfSPI.

 

Just to clear this out of the way, we have done stopALL.ovpl and startALL.ovpl.  The checkconfig.ovpl came back normal and the dbsize.ovpl is normal as well.  Ran resetSPI.ovpl with option 4 and to no avail. 

 

I've used this product before in a different environment and have never run into the issues I'm seeing here.

 

Thanks!

Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 2 of 17 (638 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Also note that the csv.gz files are being placed into the Final directory for PerfSPI but are stuck there.
Please use plain text.
Valued Contributor
Vikramsingh_03
Posts: 43
Registered: ‎01-23-2013
Message 3 of 17 (601 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Hi Sobejhell,

 

  It clearly means that NNMi is doing its stuff and placed all those files into the final directory. But the problem is iSPI is not taking up the file.

 

Possible measure is to see whether the connection is running between NPS and NNMi.

 

Can i know if the iSPI is on different server or on the same server ??

 

Regards,

Vik

 

Appreciate my efforts by hitting Kudos.

Please use plain text.
Valued Contributor
Vikramsingh_03
Posts: 43
Registered: ‎01-23-2013
Message 4 of 17 (598 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

And, please issue the following command and particularly look over the Custom Poller and State Poller health errors, if any.

 

Issue the command  ->  nnmhealth.ovpl - print verbose.

 

And, can you send us the following log file :

 

1. perfMetric.#.#.csv files

2.statepoller.trace.#.#.log

3.nnm-trace.#.#.log

4.jbossServer.log

 

you can find these log files in $NNM_LOG folder.

 

Regards,

Vik

 

Appreciate my efforts by hitting Kudos.

Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 5 of 17 (583 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Vikramsingh_03-

iSPI is on the same server. Only monitoring about 50 devices.
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 6 of 17 (583 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

I will try and find these files but for security reasons I am not allowed to send log files. I will check them and inform you of any specific errors.
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 7 of 17 (578 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Vikramsingh_03

I checked all of these logs very thoroughly and did not find any errors that would relate to the Performance for Metrics iSPI, only small errors like unable to parse traps, etc.

This is what I see for the iSPI when I run a statusALL.ovpl :

C:\statusall.ovpl

HP NNM iSPI Performance BI Service is running.
HP NNM iSPI Performance Database and DB SQL Rewrite Proxy processes are running..
HP NNM iSPI Performance ETL Service is running.
NOT OK - PerfSPI_Diagnostics (Last Heartbeat was 2715 minutes ago)
NOT OK - Component_Health (Last Heartbeat was 2716 minutes ago)
NOT OK - Interface_Health (Last Heartbeat was 3042 minutes ago)
NOT OK - FrameRelayPvc_Health (Last Heartbeat was 2716 minutes ago)
NOT OK - AtmPvc_Health (Last Heartbeat was 2716 minutes ago).
HP NNM iSPI Performance Services all running OK.
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 8 of 17 (573 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Also wanted to note we ran a resetSPI.ovpl with option 4 today. Still not working. I'm not sure what our options are here outside of a reinstall and considering management here that's not a good solution. Right now they aren't very confident in the HP tools, which isn't good considering the amount of money spent on them.
Please use plain text.
Trusted Contributor
mostafa_hassan
Posts: 246
Registered: ‎12-11-2011
Message 9 of 17 (568 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

i faced this issue before that NPS Doesn't take the CSV files from the Final Shared Directory , i have opened a case "4646691888" with the support regarding that and they explained as following .

 

It seems that Sybase is running low on cache at some point some queries get stuck and lock the tables.

Looking at the prspi.cfg file you currently have 4 GB allocated to the Sybase cache.

 

Could you kindly edit the following two lines in the %nnmdatadir%/NNMPerformanceSPI/database/perfspi.cfg file:

-iqmc 2805

-iqtc 2805

To:

-iqmc 1350

-iqtc 7650

 

If it's like that i think u can follow the Above configuration changes ,

 

All the best .
Saying Thanks by hitting Kudos :)

Regards
Mostafa Hassan
HP AIS NNM-NA-OO
CCNA-CCNP-ITIL-VCA-Cloud-VCA DataCenter
Please use plain text.
Trusted Contributor
mostafa_hassan
Posts: 246
Registered: ‎12-11-2011
Message 10 of 17 (567 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

please consider to restart the service after changing that config

All the best .
Saying Thanks by hitting Kudos :)

Regards
Mostafa Hassan
HP AIS NNM-NA-OO
CCNA-CCNP-ITIL-VCA-Cloud-VCA DataCenter
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 11 of 17 (555 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Yesterday ran resetSPI.ovpl with option 5, ran a statusALL.ovpl and everything came up as OK! I was super happy, but the report menu wasn't coming up in NNMi so I thought I needed to restart the server, when it came back up, everything came back as NOT OK again when I ran statusALL.ovpl and the same steps I took to fix it the first time won't fix it again.

I will check my config file and see what it looks like in there. Thanks for the info, I'll inform you if anything changes.
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 12 of 17 (550 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

I changed the config file and the restarted services now all of my ETLs are OK.  Problem now is that I can't get the report menu to load and the performance tab is blank ...

Please use plain text.
Trusted Contributor
mostafa_hassan
Posts: 246
Registered: ‎12-11-2011
Message 13 of 17 (545 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

which browser are u using ?!

 

Did you applied the latest patch 3 for  i SPI ? 

 

 

All the best .
Saying Thanks by hitting Kudos :)

Regards
Mostafa Hassan
HP AIS NNM-NA-OO
CCNA-CCNP-ITIL-VCA-Cloud-VCA DataCenter
Please use plain text.
Trusted Contributor
mostafa_hassan
Posts: 246
Registered: ‎12-11-2011
Message 14 of 17 (544 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

you can also try to load the menu from the ISPI Program from Start menu !

All the best .
Saying Thanks by hitting Kudos :)

Regards
Mostafa Hassan
HP AIS NNM-NA-OO
CCNA-CCNP-ITIL-VCA-Cloud-VCA DataCenter
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 15 of 17 (530 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Thanks! I found out that for some reason, NNMi was set for HTTPS and PerfSPI was set for HTTP ... changed the port and now the menu comes up but I'm getting a DPR-ERR-2082 error. Something about the content manager and unabel to load any reports.
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 16 of 17 (523 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

I did a restart since I noticed after configuring the correct ports it said "will take effect when server is restarted" I previously read it as "service is restarted" ... it came back up and right away NNMi had problems starting services thus the ETL services were out of wack ... I did a stopall.ovpl then an ovstop -c ... ran ovstart -c and then startall.ovpl ... still a few ETL services were NOT OK, so I did a stopETL.ovpl and then a startETL.ovpl, everything came back OK and all of my reports are accessible ... no errors ... thanks everyone for your help! Also, if anyone runs across this post and needs information feel free to send me a personal message on here.
Please use plain text.
Frequent Advisor
sobejhall
Posts: 28
Registered: ‎12-12-2013
Message 17 of 17 (426 Views)

Re: Performance for Metrics iSPI, ETL Services not starting

Does anyone know if NNMi 9.2 Patch 4 corrects this issue?  I'm pretty tired of having to fix this every couple of days ...

Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation