Re: TRIM 7.31 - Incredibly slow IDOL indexing (5175 Views)
Reply
Advisor
Phil_The
Posts: 31
Registered: ‎09-03-2012
Message 1 of 10 (7,240 Views)
Accepted Solution

TRIM 7.31 - Incredibly slow IDOL indexing

Good morning,

 

We've recently upgraded from TRIM 6.24 to TRIM 7.31 (including an RMDB change to SQL). As part of this process during development and testing, I had a lot of trouble getting IDOL to work reliably. Browsing through forums, a lot of googling and some additional feedback from 3rd party vendors provided a lot of useful suggestions, but at the end of the day, our IDOL configuration is still problematic.

 

At the moment, IDOL processes a couple of hundred items in the status folder every 5 to 6 hours. Between these periods the status folder item count doesn't decrease, however I can see a lot of disk activity on dynterm\dbu.db and dynterm\dbuv.db. I would have assumed that the content engine was in the process of comitting to the IDOL database during this time, however ALL of the disk activity with the IDOL processes is READ only! 

 

I chose to do IDOL processing of documents in yearly batches based on dateRegistered.  It has taken IDOL, the better part of two weeks to empty 2/3rds of the status folder which represents roughly the last 8-9 months of content in TRIM. At this rate it will be months before we can declare that all existing content has been indexed and then turn on IDOL processing for new content.

 

During development, I experimented with multiple configuration settings on the various IDOL configuration files, boosted RAM to 16Gb+ etc. and I still cannot get decent performance on IDOL.

 

At the moment I'm running only 1 TRIM content service (as repeated testing with our configuration showed that this was more reliable than 2). Its configured on a virtualised Windows 2008 R2 server with 8GB of RAM and TIER 1 storage (SAS drives).

 

As we are still dealing with other teething problems since go-live that have a higher priority on actioning, I have not yet logged a job with HP; I will be undertaking this at the end of this week.

 

I was wondering if anyone else had experienced similar slow processing or can suggest to me if the behaviour in the italics paragraph is correct or if there is some other issue affecting IDOL processing.

 

Happy to forward additional information if requested and thanks in advance for your time.

 

Cheers, Phil.

Please use plain text.
Respected Contributor
Philip Harney
Posts: 284
Registered: ‎08-02-2009
Message 2 of 10 (7,217 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

I've had a little experience with it and run into the same type of issues initial and found that the CPU of the virtual was basically maxing out and then putting a wait on it.  Upped the CPU's in the virtual to stop this and managed to stop this occuring and did about a1.5 TB of documents in 12 hours.

 

But granted it was in Development and it was only  a quick test to make sure it would work.  How many CPU's are you running in your virtual ?

Please use plain text.
Advisor
Phil_The
Posts: 31
Registered: ‎09-03-2012
Message 3 of 10 (7,210 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

Currently running 2 CPUs with 8GB of RAM.

 

At one stage during development, I matched up the number of CPUs with the number of threads used when indexing (set both to 4). I was also running 2 content engines at that time. Performance was better from memory but when I rolled back to 1 content engine late in development, I changed the CPUs back to 2.

 

As IDOL isn't being used by clients at this point, I'm happy to alter the CPUs to 4 and use 4 threads on the next batch if you think it's worth the effort.

Please use plain text.
HP Expert
Neil Summers
Posts: 998
Registered: ‎10-08-2008
Message 4 of 10 (7,181 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

Regarding the read only issue, you really ought to check your IDOL configuration hasn't managed to set itself to read only, here's an extract from TRIM7.21_IDOLConfiguration.pdf (there isn't one for 7.3 yet so this one still applies):

 

TRIM IDOL database set to read only 
There may be circumstances where the IDOL services starts up without details of the current HP TRIM IDOL database configuration. In this case, the IDOL service would detect that a database existed that was not in the configuration file and would set the database to read-only (see option above). The read-only database setting is appended to the end of the Content Service configuration file 

To fix this issue, edit the HP TRIM Content Service configuration files and change the value to False
[TRIM_PM] 
DatabaseReadOnly=FALSE 
PM represents the database ID of the dataset.

 

 

Neil


Note: Any posts I make on this forum are my own personal opinion and do not constitute a formal commitment on behalf of HP.

(Please state the version of TRIM/HPRM you're using in all posts)

HP Software Support Online (SSO): http://support.openview.hp.com
Please use plain text.
HP Expert
Neil Summers
Posts: 998
Registered: ‎10-08-2008
Message 5 of 10 (7,179 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

Regarding the speed of the indexing, before logging a case with HP Support I recommend ensuring you've followed the guidelines in TRIM7.21_IDOLConfiguration.pdf for "Sizing and Requirements". In particular, pay attention to the requirements for storage, as this almost always tends to be the main bottleneck with indexing performance. Most importantly, if your storage platform isn't providing the IDOL indexing process with the sort of bandwidth that's recommended (120MB/s), you can expect it to run slowly. The requirements for IDOL reindexing performance are significantly higher than ISYS ever was.

Neil


Note: Any posts I make on this forum are my own personal opinion and do not constitute a formal commitment on behalf of HP.

(Please state the version of TRIM/HPRM you're using in all posts)

HP Software Support Online (SSO): http://support.openview.hp.com
Please use plain text.
Respected Contributor
Philip Harney
Posts: 284
Registered: ‎08-02-2009
Message 6 of 10 (7,166 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

Hello,

 

I guess you need to find out where your bottleneck is, do some reindexing and look at the disk throughput, cpu performance, RAM etc.  I also believe there is a IDOL Testing Tool you maybe able to get from Autonmy or HP if you are lucky that can tell you if you meet the specifications.

 

May not help but I found this

 

http://knowledgemanagement.ittoolbox.com/groups/technical-functional/autonomy-l/idol-indexing-perfor...

 

May also provide some more insight on things to check.

 

Phil

Please use plain text.
Advisor
Phil_The
Posts: 31
Registered: ‎09-03-2012
Message 7 of 10 (7,129 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

[ Edited ]

Well, I went through the IDOL 7.21 PDF again, considered my previous configurations and made yet another batch of changes:

 

In our TRIM 7.31 environment with 200 or so concurrent users:

 

Windows 2008 R2 server configured with 12GB of RAM

1 IDOL Content engine (rather than the default of 2 engines)

indexing on 3 threads (with roughly 4GB of RAM per thread) less O/S overhead.

 

For some inexplicable reason, this configuration now screams and has been consistently running for the last 2 weeks! One year's worth of content (210000 or so records) indexed in less than 2 days. They still appear to process in batches but apart from some slow performance on the dedicatd IDOL/events server (which is of minimal impact) everything's running fine.

 

I'll do some further checking on performance etc. but initial assessments indicate that it may have been the extra content engine, the 5 thread default and possibly not enough memory that conspired to crimp performance. VMWare virtualization was also another factor that may have contributed to this.

 

Thanks to all for your feedback regarding this.

 

Cheers, Phil

 

 

 

Please use plain text.
Advisor
Phil_The
Posts: 31
Registered: ‎09-03-2012
Message 8 of 10 (7,126 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

Hi Neil,

 

There were no configuration values within the file to set IDOL to read-only (based on your post.) Should these values be added anyway with a value set to false to ensure it remains that way, or is the absence of these values sufficient to ensure that it remains in Read/Write mode?

 

Cheers, Phil.

Please use plain text.
HP Expert
Neil Summers
Posts: 998
Registered: ‎10-08-2008
Message 9 of 10 (7,107 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

There should be no difference between having the DatabaseReadOnly parameter set to False, or not having the parameter in the file at all.

Neil


Note: Any posts I make on this forum are my own personal opinion and do not constitute a formal commitment on behalf of HP.

(Please state the version of TRIM/HPRM you're using in all posts)

HP Software Support Online (SSO): http://support.openview.hp.com
Please use plain text.
Advisor
Phil_The
Posts: 31
Registered: ‎09-03-2012
Message 10 of 10 (5,175 Views)

Re: TRIM 7.31 - Incredibly slow IDOL indexing

[ Edited ]

To wrap things up here:

Problem was basically a combination of two factors:

1. Attempting to provision IDOL in a virtual environment without sensibly allocating disk spindles to the various roles. Make sure you read: http://support.openview.hp.com/selfsolve/document/KM00387430?searchIdentifier=-35bcbcfa%3a13d7f305c2... as part of the process for determining required infrastructure.

2. Having the REPOSITORYSTORAGE parameter set to TRUE (which is the default value). This caused a LOT of additional and un-necessary disk read/writes during the indexing process.

As an anonymous IDOL expert quoted:

 

"Cache size: Cache size (and other features) are controlled via the configuration in IDOL. If IDOL reaches the maximum cache size then it will continue to work, but performance will be degraded if requesting something outside of the cache. You should receive a notice in the logs if the cache size is exceeded. One thing to note is that TRIM defaults to RepositoryStorage=true in the configuration. This is an inappropriate setting for most customers and will result in significant disc I/O. Basically that mode will ensure atomicity of the caches when a new document is indexed by duplicating the current index, adding the new document into that and then replacing the old cache with this one. If your cache is 50Gb in size, then an entirely new 50Gb file will have to be created and deleted! With repositoryStorage=false the new document will be added to the current cache in place. In theory if the box was to fail at the exact moment your cache could become corrupted. In reality, I’ve never seen RepositoryStorage=true turned on for any of my customers and the number of corrupt caches has been very minimal.”


There is a useful forum thread here: http://h30499.www3.hp.com/t5/HP-Records-Manager-and-HP-TRIM/Initial-TRIM-reindex-from-existing-Docum... that provides a lot of additional information on IDOL gleaned from a customer's investigation and experience; well worth a read.

Please use plain text.
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation