Re: Initial TRIM reindex from existing Document Store - Performance (3734 Views)
Reply
Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 1 of 22 (4,106 Views)

Initial TRIM reindex from existing Document Store - Performance

We are upgrading TRIM 6.24 to Trim 7.2.1  bit

 

Our environment is as follows:

 

-          All servers running Windows Server 2008 64 bit

-          2 workgroup servers, one has IDOL and document content services  The index server has 32 GB RAM and 4 vCPU(the blade is Dell M610 blade server running 72 GB and two quad core processors, xeon X5570 @ Ghz)

-          DB is SQL server 2008 64 bit on Server 2008 64 bit

-          All Virtual

 

Doc store we are running reindex against is 600GB

 

When we went to reindex the entire doc store it created 477 "batches" in the status folder.  These are processing 10 per day which means it would take 45 days for this to process.

 

HP suggests that the process is complete, yet everyday the status folder removes the batches and the dyntern folder increases in size and has current date and time stamps.  This would indicate that the process is still running.

 

Has anyone had these issues?  Any tuning that has been done?  Our initial guess was that this should only take about 48 hours to do.

 

The two document content services also chew up all the 32 GB on memory.  Is this normal?  Also, we alotted 200 GB for IDOL, 240 GB for each Index Service.  We can see some usage of space for the indexes but nothing for IDOL.  The incoming folders sizes don't change. 

 

Any help is appreciated.

 

Thanks.

 

Tim

Honored Contributor
EWillsey
Posts: 1,929
Registered: ‎04-20-2010
Message 2 of 22 (4,091 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Technically HP is correct.... the "reindex" is done!  TRIM has just sent off the records to IDOL and now it's processing them.  Have you looked at the log for the IDOL service and Content Services (1/2)?

 

What setting did you put your batches on?

Honored Contributor
Grundy
Posts: 2,852
Registered: ‎02-16-2009
Message 3 of 22 (4,069 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

What you are seeing is normal and expected with an out of the box IDOL setup.

If you dont customise the content services and throw resources at the reindex, it will run quite slowly.

 

Our recommendation is to click 'Cancel' on the reindex screen once it says 'waiting on IDOL'.

If you leave this screen open, it actually keeps sending status request commands to IDOL until it's done and this adds even more bulk to the 'Status' directory for IDOL to process.

Effectively slowing it down even more...

 

Make sure you have enough threads allocated to each content serivice and also ensure that they are each running on their own physical disk/LUN.

 

Disk performance is a huge factor for the IDOL reindex.



::::::::::::::::::::::
NOT A HP EMPLOYEE
::::::::::::::::::::::

Kapish.com.au
Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 4 of 22 (4,058 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

What do you mean by "setting on your batches?"

 

Thanks.

HP Expert
Neil Summers
Posts: 1,058
Registered: ‎10-08-2008
Message 5 of 22 (4,043 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

There are a number of tweaks you can make to your IDOL setup to increase performance of the initial reindex.

 

The HP TRIM 7.21 IDOL Configuration Guide is always a good place to start when looking for more advice on how to deploy and configure IDOL with TRIM.

 

For those wanting more detailed information than the above guide, you may wish to have a look at the IDOL Distributed Index Handler (DIH) Version 7.3 Administration Guide.

Neil


Note: Any posts I make on this forum are my own personal opinion and do not constitute a formal commitment on behalf of HP.

(Please state the version of TRIM/HPRM you're using in all posts)

HP Software Support Online (SSO): http://support.openview.hp.com
Honored Contributor
EWillsey
Posts: 1,929
Registered: ‎04-20-2010
Message 6 of 22 (4,032 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Anything newer available that talks dealing with initial loading of IDOL?
Honored Contributor
Grundy
Posts: 2,852
Registered: ‎02-16-2009
Message 7 of 22 (4,026 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

We're putting together some more detailed information about sizing and resources required for certain sized datasets/stores.

 

 



::::::::::::::::::::::
NOT A HP EMPLOYEE
::::::::::::::::::::::

Kapish.com.au
Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 8 of 22 (4,001 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

We have tried several different configurations, muliple index services and we haven't been able to get it to run as of yet.

 

I see in 7.3 they offer a new connector for doing indexing.  Any further notes on what this is and what it does?  Is this a replacement to the current index serives or does it just go after non electronic records?

 

IDOL CFS connector for TRIM

The IDOL CFS connector for TRIM is a connector that indexes all HP TRIM records into IDOL. It differs from the HP TRIM Document Content Index in that:

• It indexes non-electronic records

• It indexes metadata about the record

• It allows direct queries of the IDOL index in a way that obeys HP TRIM security rules

The CFS TRIM connector implements the following actions:

• Synchronize Group

• Synchronize Records

• Collect

• Insert Records

• Hold and Release Hold

• View

• Delete

TRIM stores notifications for the connector of metadata and/or content changes.

See the HP TRIM SDK Web site for details about implementing the HP TRIM CFS connector. It is available through the HP Software Support Online (SSO) portal http://support.openview.hp.com/. Click

Use self-solve knowledge search and then search for HP TRIM SDK.

 

Again, we are moving from 6.24 and trying to upgrade to 7.2.1.

 

Thanks.

 

Tim

Honored Contributor
Grundy
Posts: 2,852
Registered: ‎02-16-2009
Message 9 of 22 (3,995 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

There's nothing wrong with the connector in 7.2.X, we have several sites in production with it now.

 

The issues you are describing above are pretty generic. 

What do you mean you 'cant get it to run'?

 

I think you should open a support case with the TRIM Support team so we can go through this in detail with you and figure out where it has gone wrong. :)

 

 



::::::::::::::::::::::
NOT A HP EMPLOYEE
::::::::::::::::::::::

Kapish.com.au
Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 10 of 22 (3,980 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

We have been working daily with L 3 HP Trim Support for a couple of weekes and we have yet to get it working :)  We have tried several configurations.  Doesn't appear that it works at this time.

 

We have reached out to our channel manager to see if HP can provide us customer North American references that have successfully update and environment like ours....we are still waiting.

 

 

Two of the customer references we have advised us they abondoned their 7.2.1 upgrade for "various reasons".

 

Thanks.

 

 

Honored Contributor
Grundy
Posts: 2,852
Registered: ‎02-16-2009
Message 11 of 22 (3,970 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Yep, I've discussed the issues you are seeing with Ryan.

He is looking into the problems and we'll be looking into it too back here at HQ.



::::::::::::::::::::::
NOT A HP EMPLOYEE
::::::::::::::::::::::

Kapish.com.au
Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 12 of 22 (3,875 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

We have continued to try and get this working.  We ended up figuring a way to get this to run with 7.2.2 and are currently looking at 7.3.1.

 

Some key things that we found.

 

1.  We have less than 5 million documents so we only configured one Document Content Service.  Seems to be issues running two of them.  Also, we have yet to see anything write to or use the 200 GB requirement for IDOL(cache).

2.  Put the dyntern and status folders on separate LUNS.  CPU usage drops when you split these two off.

3.  We have about 1 million documents, when we did a full reindex, it would run but the math we did, would have us running the index for 10 + days.  We broke it down by years and seemed to process much better.  Does the ADDs first then all the Synchs (commits).  Our current doc store is 800 GB.  Took us just over 36 hours to do it in sections by years.

4.  There were also some tweaks we made to the .cfg for IDOL.  You also have to tweak the cfg for the Index Service, but this is just to change the paths for point 2.  HP has released some docs on some configs that can help.

5.  We are excluding .tfs and .jpegs from our index.  7.2.2 seems to successfully do .vmbxs.

 

I will post again if we try this with 7.3.1.

 

We worked with an excellent HP L3 resource based out of North America.

Honored Contributor
EWillsey
Posts: 1,929
Registered: ‎04-20-2010
Message 13 of 22 (3,872 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Thanks for sharing and letting us know!

Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 14 of 22 (3,810 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

We moved to 7.3.1.  7.2.2 appeared to be a version we could go with but decided if we were upgrading we might as well go current.

 

We are in the process of running through the index now.

 

Seems to be OK except for one minor issue that should be fixed with the next release.  If you are running the index and a record that can't be index is hit, the entire process stops.  Basically you need to rerun the reindex for the criteria you selected.  So every time you hit a records, it stops and you have to start from scratch.  The key is to do small chunks.  We are using date registered and doing aourn 25K worth of records.  Once you find all the "bad" records and remove them (we are just deleting them from the doc store, saving them off and will reimport them later), you should have no issues rerunning the process if need be.

 

We also found that this process uses a tremendous amout of disk.  We are seeing that the I/O block sizes are around 1 MB + and most of the other database apps are around 64 K.  So either throw more spindles at it or change the number of threads.

 

We are currently running with 3 threads from the 5 we had before.  Set in the idol.cfg, trim content service.cfg and via the reindex on the last tab that asks for the number of threads.  We are planning to give it more spindles.  The creation of the "batches" in the status folder is OK, its the commits that seem to be processing slow.  HP is looking into this for us.

 

Thanks.

 

 

Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 15 of 22 (3,743 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Eureeka!!!  We seem to have fianlly figured out the entire indexing process and have been able to tune our install to do 1M docs with a size of about 700 GB to run in about 15 hours.

 

I will try and break it down as we understand it.  We ran several test.  We ran by refrence, by batch, plain non-mirrored mode and ended up running in plain non-mirrored mode.

 

 

HP TRIM IDOL Content Service:  this is actually the idol engine that processes all the batches and populates the dyntern and nodetable folders.  This is all memory intensive.

HP TRIM IDOL Service:  The only purpose this serves is to control the dih.exe and dah.exe

Dih.exe:  this handler sends the batches to the HP TRIM IDOL Content Service

Dah.exe:  this is the handler for the client when the client uses document content searching

 

HP also has an event process that creates the batches and send them to the incoming folders.

 

How does the process work?

 

You will have your TRIM environment installed and configured.  There will be a database instance and a document store.

 

The index process is kicked off from the TRIM client.  The event processor creates the initial batches and sends them to the incoming folder for the DIH.  From there the DIH reads its config file and sends these batches based on the configuration to the HP TRIM Content Service.  The HP TRIM Content Service then processes the batches to create the index for the dyntern and nodetable folders.  This service also has its own configuration file that it reads to determine how it will process the batches.  Once this process is completed, the commit of the batches to the folders cleans out the status folder.

 

The key is that the creation of the batches is CPU and memory intensive, while the HP TRIM Content Service uses memory.

 

We will be running two sets of configs.  We will have one for mass index and one for everyday use.

 

 

We have two Workgroup Servers.  One is just a workgroup server with rendering, the other is the IDOL implemenation and workgroup server.

 

We kick off the mass index from Workgoup 1 and all the work is done on workgroup 2.  The creation of the batches is highly CPU intensive so it is too much to run for one server.

 

Each server for the mass index will have 4 CPU and 32 GB of RAM.  Based on how long you need it to run will determine the amount of RAM.  More the better.

 

Once the index is done we will drop Wrokgroup 1 to 1 CPU and 4 GB of RAM.  Workgoup 2 will gor to 2 CPU and 4 GB of RAM.

 

Our break down for the IDOL config for disk is like this:

 

We run 4 Content Engines.  2 on one disk (e:\) and 2 on another (f:\)  IDOL is on D:\

 

c:\ for logging and O/S  35 GB  (HP recommends moving them so this is where we picked)

d:\ 40 GB (we have tuned it with so much that we never see anything sit in teh incoming folder of DIH.

e:\ 120 GB

f:\ 120 GB  (these two will change if we have to add more data but both have about 30 GB free after the reindex process)

 

 

Where did we get our speed from?

 

Two setting changed everything for us.  The workgroup has 32 GB of RAM.  With 4 Content Engines we gave each on 6 GB leavin 8 for the OS and other needs.

 

For Mass Index we changed this. 

 

//---------------------------- Server Settings --------------------------------//

MaxSyncDelay=4500

[IndexCache]

IndexCacheMaxSize=6291456  (6GB per Cotnent Engine)

 

This way everything runs in memory then dumps.  Much more efficient for the commits. 

 

Beacuse we run 4 engines on two seperate disks we also added the following for pair of engines:

 

FlushLockFile=E:\HPTRIM\IDOL\FlushLockFile.lck  this basically will only let one engine work at a time.  The process is single threaded.  This way your MaxSyncDelay could be the same time for each engine.

 

Move your logging:

 

//---------------------------- Logging Settings -------------------------------//

[Logging]

LogDirectory=C:\Logs\Content Service 1

 

Create a new Language Type (did this with , its a blend of international and english)

 

//---------------------------- Language Types ---------------------------------//

Stoplist=international_large.dat  (for this one you need to create the new .dat file by combining the international.dat and English_large.dat files located here:  \\ServerXX\d$\Program Files\Hewlett-Packard\HP TRIM\IDOL\shared\langfiles

 

 

 

For everyday use we will change the following:

 

//---------------------------- Server Settings --------------------------------//

MaxSyncDelay=120

 

[IndexCache]

IndexCacheMaxSize=204800  Both these can be changed if needed.

 

To apply changes the IDOL services need to be stopped and started.  These configs aren’t controlled by Trim Enterprise Studio (TES)

 

 

 

 

To stop and start IDOL use the batch scripts that HP recommends.  There is a knowledge base article on this.

 

 

Some additional tuning.  Exlude extensions that don't index.  During the index process you can select the batch size, change this to 100 from the default 20.  Uncheck verbose logging on the last tab.

 

Here are our configs:

 

IDOL:

 

[Service]
ServicePort=9002
ServiceStatusClients=*.*.*.*
ServiceControlClients=*.*.*.*

//---------------------------- Server Settings --------------------------------//

[Server]
QueryClients=*.*.*.*
AdminClients=*.*.*.*
IndexClients=*.*.*.*

Port=9000
IndexPort=9001

Threads=5
MaxInputString=-1
MaxQueryTerms=-1
MaxFileUploadSize=-1
MaxResults=10000
QueryTimeoutInMilliseconds=60000

DateFormatCSVs=YYYYMMDDHHMMSS,AUTNDATE
KillDuplicates=*/DREREFERENCE
DocumentDelimiterCSVs=*/DOCUMENT
CombineIgnoreMissingValue=true


//------------------------- Distributed Architecture -------------------------//
[DistributionSettings]
MirrorMode=false
DistributionMethod=1
MaxResults=10000
IndexQueueSize=5000
BlockIfFull=true
//--DistributeByReference=true--//
Redistribute=true
#//--DistributeOnBatch=true--//
SimpleCombinatorMode=true

[DAHEngines]
Number=4

[DAHEngine0]
Host=TRIMWGTST02
Port=9100

[DAHEngine1]
Host=TRIMWGTST02
Port=9200

[DAHEngine2]
Host=TRIMWGTST02
Port=9300

[DAHEngine3]
Host=TRIMWGTST02
Port=9400

[DIHEngines]
Number=4

[DIHEngine0]
Host=TRIMWGTST02
Port=9100

[DIHEngine1]
Host=TRIMWGTST02
Port=9200

[DIHEngine2]
Host=TRIMWGTST02
Port=9300

[DIHEngine3]
Host=TRIMWGTST02
Port=9400

[Paths]
Incoming=D:\Program Files\Hewlett-Packard\HP TRIM\IDOL\TRIM IDOL Service\dih\incoming
Archive=D:\Program Files\Hewlett-Packard\HP TRIM\IDOL\TRIM IDOL Service\dih\archive
Failed=D:\Program Files\Hewlett-Packard\HP TRIM\IDOL\TRIM IDOL Service\dih\failed

//--------------------------- Document Security ----------------------//

[Security]
SecurityInfoKeys=123,144,564,231


//---------------------------Logging-------------------------------//
[Logging]
LogDirectory=C:\Logs\IDOL Service Logs
LogTime=TRUE
LogOutputLogLevel=TRUE
LogLevel=Normal
LogExpireAction=Compress
LogOldAction=delete
LogMaxOldFiles=10
MaxLogSizeKbs=20480

0=ApplicationLogStream
1=QueryLogStream
2=IndexLogStream

[ApplicationLogStream]
LogFile=application.log
LogTypeCSVs=application

[QueryLogStream]
LogFile=query.log
LogTypeCSVs=query

[IndexLogStream]
LogFile=index.log
LogTypeCSVs=index

//---------------------------- Language Types ---------------------------------//

[LanguageTypes]
LangDetectUTF8=TRUE
DefaultLanguageType=englishUTF8
DefaultEncoding=UTF8
LanguageDirectory=..\shared\langfiles
GenericStemming=true
GenericTransliteration=true

0=english

[english]
Encodings=UTF8:englishUTF8
Stoplist=international_large.dat
IndexNumbers=1
HyphenChars=NONE
AugmentSeparators=-–—@“”‘’_.…

 

Here is one of our Content Engines (we have 4 so make changes where appropriate):

 

//---------------------------- Service Settings -------------------------------//
[Service]
ServicePort=9102
ServiceStatusClients=*.*.*.*
ServiceControlClients=*.*.*.*

//---------------------------- Server Settings --------------------------------//
[Server]
Port=9100
IndexPort=9101
Threads=5
QueryClients=*.*.*.*
AdminClients=*.*.*.*
IndexClients=*.*.*.*
MaxInputString=-1
MaxQueryTerms=-1
MaxFileUploadSize=-1
MaxResults=10000
QueryTimeoutInMilliseconds=60000
DiskHash=20000000
RefHashes=4000000
TermSize=40
DelayedSync=true
MaxSyncDelay=120   ******everyday setting change for mass index
AdvancedPlus=TRUE
SplitNumbers=false
DateFormatCSVs=YYYYMMDDHHMMSS,AUTNDATE
KillDuplicates=*/DREREFERENCE
DocumentDelimiterCSVs=*/DOCUMENT
UnstemmedMinDocOccs=1
RepositoryStorage=TRUE
MinFreeSpaceMB=1024
CombineIgnoreMissingValue=true
FlushLockFile=E:\HPTRIM\IDOL\FlushLockFile.lck   *****we added this

[TermCache]
TermCacheMaxSize=0

[IndexCache]
IndexCacheMaxSize=204800  ******this is our everday setting, change this for Mass index

[SectionBreaking]
MinFieldLength=9000000
MaxSectionLength=900000000

//---------------------------- Path Settings ----------------------------------//
[Paths]
BitFieldPath=E:\HPTRIM\IDOL\TRIM Content Service 1\bitfield
DyntermPath=E:\HPTRIM\IDOL\TRIM Content Service 1\dynterm
IndexQueuePath=E:\HPTRIM\IDOL\TRIM Content Service 1\indexqueue
IndexTempPath=E:\HPTRIM\IDOL\TRIM Content Service 1\indextmp
MainPath=E:\HPTRIM\IDOL\TRIM Content Service 1\main
NodetablePath=E:\HPTRIM\IDOL\TRIM Content Service 1\nodetable
NumericPath=E:\HPTRIM\IDOL\TRIM Content Service 1\numeric
RefIndexPath=E:\HPTRIM\IDOL\TRIM Content Service 1\refindex
SecIndexPath=E:\HPTRIM\IDOL\TRIM Content Service 1\secindex
SortfieldPath=E:\HPTRIM\IDOL\TRIM Content Service 1\sortfield
StatusPath=E:\HPTRIM\IDOL\TRIM Content Service 1\status
StatePath=E:\HPTRIM\IDOL\TRIM Content Service 1\storedstate
TagPath=E:\HPTRIM\IDOL\TRIM Content Service 1\tagindex
LanguageDirectory=..\shared\langfiles

[Schedule]
PreCompactionBackup=False

//---------------------------- Logging Settings -------------------------------//
[Logging]
LogDirectory=C:\Logs\Content Service 1
LogTime=TRUE
LogOutputLogLevel=TRUE
LogLevel=Normal
LogExpireAction=Compress
LogOldAction=delete
LogMaxOldFiles=10
MaxLogSizeKbs=20480
0=ApplicationLogStream
1=QueryLogStream
2=IndexLogStream

[ApplicationLogStream]
LogFile=application.log
LogTypeCSVs=application

[QueryLogStream]
LogFile=query.log
LogTypeCSVs=query

[IndexLogStream]
LogFile=index.log
LogTypeCSVs=index

//---------------------------- Field Processing -------------------------------//
[FieldProcessing]
0=SetIndexFields
1=SetDateFields
2=SetDatabaseFields
3=SetTitleFields
4=SetHighlightFields
5=SetReferenceFields
6=SetRelatedDocsReferenceFields

[SetIndexFields]
Property=IndexFields
PropertyFieldCSVs=*/DRECONTENT,*/DRETITLE

[SetDateFields]
Property=DateFields
PropertyFieldCSVs=*/DREDATE

[SetDatabaseFields]
Property=DatabaseFields
PropertyFieldCSVs=*/DREDBNAME

[SetTitleFields]
Property=TitleFields
PropertyFieldCSVs=*/DRETITLE

[SetHighlightFields]
Property=HighlightFields
PropertyFieldCSVs=*/DRETITLE,*/DRECONTENT

[SetReferenceFields]
Property=ReferenceFields
PropertyFieldCSVs=*/DREREFERENCE

[SetRelatedDocsReferenceFields]
Property=RelatedDocsReferenceFields
PropertyFieldCSVs=*/DREROOTFAMILYREFERENCE

//---------------------------Properties-------------------------------//
[IndexFields]
Index=TRUE

[DateFields]
DateType=TRUE

[DatabaseFields]
DatabaseType=TRUE

[TitleFields]
TitleType=TRUE

[ReferenceFields]
ReferenceType=TRUE
TrimSpaces=TRUE

[HighlightFields]
HighlightType=TRUE

[RelatedDocsReferenceFields]
ReferenceType=TRUE
TrimSpaces=TRUE

//---------------------------- Language Types ---------------------------------//
[LanguageTypes]
LangDetectUTF8=TRUE
DefaultLanguageType=englishUTF8
DefaultEncoding=UTF8
LanguageDirectory=..\shared\langfiles
GenericStemming=true
GenericTransliteration=true
0=english

[english]
Encodings=UTF8:englishUTF8
Stoplist=international_large.dat
IndexNumbers=1
HyphenChars=NONE
AugmentSeparators=-–—@“”‘’_.…

//---------------------------- Databases --------------------------------------//
[Databases]
NumDBs=1
0=TRIM_DV

 

 

As I said this runs for 15 hours.   The process doesn't stop.  We leave it with 5 threads and let it run.  There are errors as to be expected.  KeyView has some issues with certain .vmbx and their attachments, unsupported formats, password protect docs, etc...

 

We are also runninng DRECOMPACT weekly.  We aren't using the schedule setting in the config files, instead we are using the following power shell which will be run by a weekly windows task on the server. 

 

#############################################################################################
# Description:                                                                              #
# This was written to Compact the IDOL Content Engines.                                     #
# It will also launch the command to compact the collections via the web browser (silently) #
#############################################################################################
# Declare IE as an object for each content engine that you need to backup
$ie1 = new-object -com internetexplorer.application
$ie2 = new-object -com internetexplorer.application
$ie3 = new-object -com internetexplorer.application
$ie4 = new-object -com internetexplorer.application
#
# Silently Launch IE to Backup the Directories
$ie1.navigate("http://localhost:9101/DRECOMPACT")
$ie1.visible = $false
$ie2.navigate("http://localhost:9201/DRECOMPACT")
$ie2.visible = $false
$ie3.navigate("http://localhost:9301/DRECOMPACT")
$ie3.visible = $false
$ie4.navigate("http://localhost:9401/DRECOMPACT")
$ie4.visible = $false
#
# All Done
Write-Host "DRECOMPACT Issued. Monitor the index log file of each content engine for completion status."

 

 

 

 

Hope that this helps.  Took us some time to get here.  This works for us.  Please do your homework for your environment.

 

Thanks.

 

 

 

 

 

Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 16 of 22 (3,741 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

This is all with HP TRIM 7.3.1.

 

Also wanted to say thanks to Ryan and the rest for helping us get here.

 

T

Honored Contributor
EWillsey
Posts: 1,929
Registered: ‎04-20-2010
Message 17 of 22 (3,740 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Frequent Advisor
tbanera
Posts: 28
Registered: ‎10-10-2012
Message 18 of 22 (3,735 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Sorry one more thing.  We are still try a few things but this is likely our final config.

 

The one we are playing with is this setting:  FlushLockFile=E:\HPTRIM\IDOL\FlushLockFile.lck 

 

Thanks.

 

Tim

HP Expert
Ryan Winston
Posts: 143
Registered: ‎02-04-2009
Message 19 of 22 (3,734 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Tim, your team did some amazing work digging into the details.  I've been in spectator mode the last few weeks ;)

The newest document, KM00387430, is designed to make sure your default instance is functional based on using the logs & test index batches.  Useful steps. 

What Tim has posted is more detailed in that it shows the critical settings & values needed to successfully reindex a large dataset with optimum efficiency.  And, despite IDOL recommendations against a Virtual Server, it's all done with Virtual Servers.

 

As Tim pointed out, the key is memory allocation during the reindex, having instances on multiple physical drives/LUNs, using the flushlock if you have multiple Content Services that must share a drive/LUN.  Also, noting the correct placement of custom settings is key, as well as to which config file you're applying them.

Super Advisor
JasonIMEX
Posts: 276
Registered: ‎06-17-2010
Message 20 of 22 (2,833 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Great Thread thanks guys..

 

One question though. Anyone having issues with the dih incoming and failed folders growing and not being flushed?

 

Seems they just sit there growing with no end in sight.

 

 

Super Advisor
JasonIMEX
Posts: 276
Registered: ‎06-17-2010
Message 21 of 22 (2,805 Views)

Re: Initial TRIM reindex from existing Document Store - Performance

Bumping for great justice!!

HP Expert
Neil Summers
Posts: 1,058
Registered: ‎10-08-2008
Message 22 of 22 (2,779 Views)

Re: Initial TRIM reindex from existing Document Store - Performance


JasonIMEX wrote:

Great Thread thanks guys..

 

One question though. Anyone having issues with the dih incoming and failed folders growing and not being flushed?

 

Seems they just sit there growing with no end in sight.


I'd say there's something wrong. Check the logs. DIH incoming should clear as the data files are successfully transferred to the content engines for processing.

Neil


Note: Any posts I make on this forum are my own personal opinion and do not constitute a formal commitment on behalf of HP.

(Please state the version of TRIM/HPRM you're using in all posts)

HP Software Support Online (SSO): http://support.openview.hp.com
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.