Service Manager 9.34: How to use the new SM Delta Data Migration Tool

As of version 9.34, SM provides a Delta Data Migration Tool to automate the transfer of “delta data” from one SM system (original) to another SM system (new). For more information, see the following topic in the SM online help system:  System Administration > Delta migration tool

 

Typical scenario

There are situations in which you need to migrate data from one SM server to another.  Take, for example, the following scenario:

  • The original system is running live in production
  • You make a copy of it
  • Within that copy, you proceed to make changes to the workflow
  • You then replace the original live system with the copy that has the new workflow

Of course, from the time you make a copy until the time the new system goes live, end users are still making changes on your original system. The following figure depicts this scenario.

 

Figure #1 – Time line diagram

Blog_1.jpg

Tip: You can also use this tool if you need to keep the data in a “test” system in sync with your “production” system, or the data in a “development” system in sync with a “test” system.

 

Data migration methods

The Delta Data Migration Tool enables you to keep the original system in sync with the copy you make in either of the following two ways (caution: for a table without the “sysmodtime” field, this tool cannot migrate “delta data” from the original system to a new system):

 

Method 1: Manual export/import of changed records

The following flow chart (Figure #2) illustrates the interactions of the operator with the Delta Data Migration Tool when using method 1 to migrate delta data.

 

Figure #2 – The Seven Stages of Method 1 – Manual Export/Import

Blog_2.jpg

 

To export the delta data by using method 1(depicted in stages 1-5 of Figure #2), follow these steps:

 

Stage 1

a. Log in to the original SM system as an administrator.

b. Open Tailoring > ScriptLibrary. Type DeltaMigration_Export in the Name field, press Enter. And then click Execute. A welcome page appears showing the instructions for migration.

c. Click Next on the welcome page and select Method 1: Manually export/import a single UNL file.

d. Click Next to check the steps when using method 1.

e. Click Next to specify the time range of the delta data. In stage 2, the Delta Data Migration Tool will generate an unload script to extract all the delta data within this range.

 

Stage 2

a. Click Next to have the Delta Data Migration Tool generate the unload script.

 

Stages 3-4

a. Open the generated unload script, and

b. Run it to have SM generate an UNL file.

 

Stage 5

a. Copy the UNL file to the new system.

 

To import the delta data by using method 1 (depicted in stages 6-7 of Figure #2), follow these steps:

 

Stage 6

a. Log in to the new SM system as an administrator.

b. Disable the triggers by adding $L.void=rtecall(“trigger”,$L.rc,0) in the format control file of login.DEFAULT file.

c. Log out and then log in to the new SM system again.

 

Stage 7

a. Open Database Manager, right click, and then select Import/Load.

b. Select the UNL file that you want to import.

c. Enable the triggers by removing $L.void=rtecall(“trigger”,$L.rc,0) in the format control file of login.DEFAULT.

d. Log out and then log in to the new SM system again.

 

Recommended usage of method 1

This method is simple and can be successfully applied when migrating delta data between SM systems with identical versions; for example, from SM 7.11 to SM 7.11 or from SM 9.34 to SM 9.34.

 

Sample migration scenario using method 1

Company A decides to transfer their live SM 7.11 system to a more powerful SM 7.11 system, with a go-live time scheduled for 02:00am on June 28th. They use method 1 to handle the migration of their delta data.

  • On June 25th, a new system with empty data is setup and tested.
  • At 4:00pm on June 27th, the live SM system is copied to the new system.
  • Since Company A’s next change window occurs from 00:00am to 02:00am on June 28th, the system data between 4:00pm on June 27th and 00:00am of June 28 (the begin time of the next change window) needs to be migrated to the new system.
  • At 00:00am on June 28th, the live system is brought down.
  • Method 1 of the DDelta Data Migration Tool is then used to extract all delta data from the original live SM system during the time range from 4:00pm on June 27th until 00:00am on June 28th.  Using method 1 of Delta Data Migration Tool, the captured delta data is imported into the new SM system.
  • At 01:00am on June 28th, the new SM system goes live, containing all expected business data.

Tips when using method 1

  • If you suspect that the table structure is different between the source and target systems, then you can use method 2 to check for any table structure differences.
  • If you then want to decide which table’s delta data to migrate, please use method 2.

Method 2: Scheduling an automated export/import

The following flow chart (Figure #3) illustrates the interactions of the operator with the Delta Data Migration Tool when using method 2 to migrate delta data.

 

Figure #3 - The Eight Stages of Method 2 – Automated Export/Import

Blog_3.jpg

To export the delta data by using method 2 (depicted in stages 1-5 of Figure #3), follow these steps:

 

Stage 1

a. Log in to the original SM system.

b. Open Tailoring > ScriptLibrary. Type DeltaMigration_Export in the Name field, press Enter, and then click Execute.

c. Click Next on the welcome page, and then select Method 2: Automatically export/import multiple UNLs.

d. Click Next to check the steps when using method 2.

e. Click Next to specify a folder for the exported delta data, and then specify the time range of the delta data.

f. Click Next to review, once the tool checks the tables, the migration suggestions provided. You can choose to ignore some tables even if they contain some delta data.

g. Click Next to verify the final confirmation for the migration. The resultant message displays the number of the tables that will be migrated and a time estimate for the export process.

 

Stages 2-3 (Stage 3 is not necessary because no additional operator interaction is required to launch the export process)

a. Click Next to start the scheduled background tasks to export the delta data.

 

Stage 4

a. Click Finish to exit the tool when the overall progress reaches 100%. 

Note: To check the export status, go to the selected folder and open “Overall Export Status Report.html”.

 

Stage 5

a. Copy the whole folder to the new SM system.

 

To import the delta data by using method 2 (depicted in stages 6-8 of Figure #3), follow these steps:

 

Stage 6

a. Log in to the new SM system as an administrator.

b. Open Tailoring > ScriptLibrary. Type DeltaMigration_Import in the Name field, press Enter, and then click Execute. A welcome page appears showing the instructions for migration.

c. Click Next on the welcome page.

d. Click Next, and then specify the folder that contains the exported delta data.

e. Click Next to select the tables to migrate. Note: We recommend that you do not import the tables when the structures are different.*

f. Click Next to check the final confirmation for the migration. The message displays the number of the tables that will be imported.

 

*  If tables are ignored (during the import operation) because the table structures are different, and if that structure mismatch introduces risks, then try to migrate the data in these tables by other methods. More information about how to do this will be provided in a follow-up Delta Data Migration Tool blog on Troubleshooting.

 

Stage 7

a. Click Next to start background schedule tasks to import the delta data.

 

Stage 8

a. Click Finish to exit the tool when the overall progress reaches 100%.

 Note: To check the import status, go to the selected folder and open “Overall Import Status Report.html”.

 

Note: If tables are ignored during the export operation because they contain no “last modification time” field, then check if a full-table migration is needed.

 

Recommended usage of method 2

Method 2 is more powerful than method 1, and can be applied to migrate delta data between different versions of SM, for example, from SM 7.11 to SM 9.34.

 

Sample migration scenario using method 2

Company A decides to upgrade their SM system from 7.11 to 9.34, with a go-live time scheduled for 02:00am on June 28th. They use method 2 to handle the migration of their delta data.

  • The upgrade process was started on June 1st, and finished at 4:00pm on June 26th. During this period of time, the original SM 7.11 system was still live.
  • At 9:00am on Jun 27th, method 2 is used to migrate delta data from Jun 1st to 9:00am on June 27th. It took 5 hours to migrate most of June’s data, but there are still new user changes occurring in the original SM 7.11 during that 5 hour period.
  • June 28th is a Saturday, and from 00:00am to 02:00am was Company A’s change window. The original SM 7.11 system is brought down at 00:00am, then method 2 is used again to migrate delta data, this time they are migrating delta data from 9:00am on June 27th to 00:00am on June 28th. It only takes 10 minutes to migrate the remaining delta data.
  • At 2:00am, the new SM 9.34 system goes live, containing all expected business data.

Tips when using method 2

Method 2 is mainly designed to handle table structure differences, which are very common when an older SM version is upgraded to a higher version.

  • If one table has no “sysmodtime” field, then manually migrate it later, for example, as a whole copy.
  • If one table exists in the original system, but not in the new system, then ignore it only after making sure the table is no longer needed in the new system. If the table is renamed or the data in it will be used by another table in the target environment, then manually migrate it later.
  • If one table has delta data to migrate, but that data is huge and useless, then you can choose not to export/import it.
  • If one table has delta data to migrate, and the table structure in the new system is the same as that in the original system, then simply export/import it.
  • If one table’s structure varies between the original system and the new system, then you should use the analyze function in the Delta Data Migration Tool.
    • If the analyze function doesn’t find a significant risk (for example, the lengths of some fields are only extended), then simply export/import it.
    • If the analyze function finds some risks (for example, the lengths of some fields are shortened, or some customized fields that exist in the original system don’t exist in the new system), then don’t export/import them, but manually migrate it later.

Prerequisites for using the Delta Data Migration Tool

Before you use the Delta Data Migration Tool to migrate your delta data, the following prerequisites must be met:

Note: The first requirement applies to both methods 1 and 2, while the last four requirements apply only to method 2.

 

1. Make sure that the new SM system is clean and does not contain any test records.

 Before the delta data migration is begun, the new system should include only the records or modifications coming from the original system. You must remove any test data that was created or modified on the new system during the delta data period. 

If there are some test records added to the new system during the delta data period, then the delta data from the original SM system cannot be migrated to the new system because of a data conflict. Even if you decide to overwrite the test records with the delta data from the original system, there is a risk of a data integrity violation.

For example, a new incident IM12345 (call it IM_A) is created in the original system during the delta data period, and another test record IM12345 (call it IM_B) is created in the new system, with several activity logs. If you then use IM_A in the original system to overwrite IM_B in the new system, then you will find that IM_A in the new system contains some incorrect activity logs, coming from IM_B.

 

2. Make sure that the date format of the original system does not have the y2k problem. To check for this problem, follow these steps:

a. Open System Navigator > Base System Configuration > Miscellaneous > System Information Record.

b. Go to Data Info.
c. If the date format contains only two digits for the year value, such as "mm/dd/yy", then change it to four digits (for example, "mm/dd/yyyy").

d. Save the modification and restart the SM service.

 

3. You must create a new administrator account in the original system. The date format and time zone settings of this account must be the same as those in the System Information Record.

Because scheduled background tasks will use the default time zone and date format in the System Information Record, creating a new administrator account with the same time zone setting and date format of System Information Record will avoid missing data.

 

4. Make sure that the five background schedulers (bg_load_unload 1… bg_load_unload 5) have been started in both of the SM systems. If the background schedulers have been started, then they should be visible in the System Status record. If they are not there, then click Start Scheduler and select dmt.unload.load.

 

5. Make sure that an empty folder is created on the original SM system for exporting the delta data.

 

Installing the Delta Data Migration Tool

The Delta Data Migration Tool is already built in for SM 9.34 and later versions. To install the tool in SM 9.33 and earlier versions, you need to deploy an UNL package. You can find the package in the upgrade folder, along with the other SM applications upgrade utility files. To install the Delta Data Migration Tool, follow these steps:

  • Log in to the SM system as administrator
  • Open Database Manager, right click, and then select Import/Load
  • Select DeltaMigrationTool.unl, and then click Load FG to deploy it

______________________________________

 

Content contributors for this article are HP SM R&D Development Engineers Fei Zhou (Pitt) and Yu-Min Chen

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the Community Guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
A 25+ year veteran of HP, Yvonne is currently a Senior Product Manager of HP ITSM software including HP Service Anywhere and HP Service Man...
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.