Data Synchronization Reference
If you are not familiar with the concept, start by the Overview
-
Want to see an example
-
Want to dig inside to troubleshoot common issues
Data Source Definition
The Data Sources are defined using the menu Admin Tools/Synchronization Data Sources.
Each data source defines:
-
The target class of the objects to be synchronized
-
The list of attributes/fields of the objects to synchronize
-
How to search/reconcile the objects with the objects already existing in iTop
-
The rules of synchronizing/updating and possibly deleting objects in iTop
-
How the synchronized objects behave for the iTop end-users (which fields are read-only, are users allowed to delete the objects…)
-
An optional hyperlink and icon to refer iTop users to the corresponding object in the external application
Description
This section of the Properties defines some general parameters for this Synchro Data Source
Property | Mandatory? | Meaning | Sample value |
---|---|---|---|
Name | Mandatory | The name which refers to this data source | Servers synchro from SCCS |
Description | Optional | An additional description, for information only | |
Status | Mandatory | implementation, production or obsolete: for filtering/classifying the data sources | production |
Target Class | Mandatory | The class of objects to synchronize | Server |
User | Mandatory | The user allowed to execute this synchro | admin |
Contact to notify | Optional | The contact to notify (by email) of the results when the synchronization is executed and contains at least one error | |
Icon's hyperlink | Optional | The hyperlink (URL) to an image representing this data source. This icon is shown instead of the default icon in the tooltip appearing on the “Lock” symbol next to the title of the synchronized object in iTop | |
Application's hyperlink | Optional | The hyperlink (URL) to the remote application
from which the synchronized data comes from. The hyperlink can
contain placeholders such as $this->attcode$ (which
will be replaced by the value of the attribute of the current
object) or $replica->column_name$ (which will be
replaced by the value of the named column from the replica) |
https://myapp.acme.com/display/server/$replica->primary_key$ |
Data table | Optional | The name of the MySQL table in which the replica information will be stored. If omitted, the system will generate a unique name for you. | synchro_data_servers_from_sccs |
Because it was required to be able through synchro_import.php to empty fields such as date, the format of date column was changed in release 2.3.4 from
date
to
varchar(10)
Search & reconciliation
This section of the properties defines
-
How the information contained in the replica table is matched against the content of the iTop database
-
What is the behavior of this synchro data source concerning the creation of new iTop objects.
Property | Meaning |
---|---|
Reconciliation policy | Use the attributes
(use_attributes): when a new replica is created, the
system will search for a matching object in iTop based on the list
of Synchro attributes with the flag
Reconciliation set to Yes .Use the primary key (use_primary_key): the
column primary_key of the replica is expected to
contain the identifier of the object in iTop. |
Action on zero | What to do when the reconciliation did not find
any object in iTop.Create instructs the system to create a new object of
the Target Class.Error will cause the system to report this case as an
error (i.e. this synchro data source is supposed to only update
existing objects). |
Action on one | What to do when the reconciliation found exactly
one matching object in iTop.Update instructs the system to update this object.Error will cause the system to report this case as an
error (i.e. this synchro data source is supposed to create objects
and never update existing objects). |
Action on many | What to do when the reconciliation found several
matching objects in iTop.Create instructs the system to create another
object.Error will cause the system to report this case as an
error.Take the first one instructs the system to pick the
first object in the list and to update it. |
Deletion rules
This section defines how the deletion of the synchronized objects are handled:
-
by the end-users in iTop,
-
by the synchronization system itself.
Property | Meaning |
---|---|
Users Allowed | Which end-users are allowed to delete synchronized
objects in iTop.Nobody when an object is synchronized, nobody is
allowed to delete it in iTop. only the synchronization system can
delete it.Administrators only only administrators are allowed to
delete a synchronized object.Everybody allowed to delete such objects the normal
access rules apply to such synchronized objects. |
Full load interval | The duration between two executions of the synchronization which are considered as having loaded (touched) all replicas. When a replica has not been touched for a duration greater than this value, it is considered as obsolete. |
Delete Policy | What to do when a replica becomes obsolete.Ignore do nothing, the associated object remains as is
in iTop.Delete Delete the associated object in iTop (and the
replica in the data table).Update Update the associated object as specified by
the Update rules (see below).Update then Delete Update the associated object as
specified by the Update rules (see below), then after
the Retention Duration , delete the object (and the
replica). |
Update rules | A list of values separated by semi-colons, of the
form attribute_code:value to specify which attribute
of the associated object to set and to which value. Example:
status:obsolete;description:no longer
synchronized . |
Retention duration | Only used when the Delete Policy is
set to Update then delete . In that case, an object
deleted from the Source is updated immediately and deleted from
iTop only after that Retention duration . |
If your set a Full load interval of 7 days, and a Retention duration of 14 days, an object no more provided by the source will be deleted after 3 weeks.
Attributes
The “Attributes” tab defines:
-
How to determine whether a new object must be created in iTop or an existing object must be associated with the replica (if the reconciliation rule is “Use Attributes” in the properties)
-
Which attribute(s) of the associated object must be updated, and how.
Attributes tab | Purpose |
---|---|
Attribute | It automatically lists each attributes of the DataSynchro “Target class”, with its label and code |
Reconciliation | Specify if the value in this column should be used to retrieve an existing iTop object. Used for new replica only |
Update | Specify if that iTop object attribute should be updated by the Datasynchro |
Update Policy | Specify if the attribute can be edited in iTop or not |
Reconciliation Key | Required for ExternalKey attributes only. Specify one attribute of the remote object used to retrieve it. It can be the id or a field which would be a unique identifier for the remote class |
The Reconciliation column
When a new record appears in the synchro_data_xxx
table, the synchronization mechanism will try to associate this
record with an existing iTop object. If one or more Attribute is
marked as “Yes” in the “Reconciliation ?” column, these
columns/attributes will be used to search for an iTop object that
matches all these values. If one object is found, it will become
associated with this record. If no object is found, a new iTop
object will be created.
The Update column
The Update column defines if the attribute must be updated or not.
-
Checked = Yes
: the attribute will be set/updated -
Unchecked = No
: the external source information for this attribute will be ignored, the existing value on the iTop object will be kept as is.
Null
and empty string
are not
equivalent-
Null ⇒ The existing iTop field value is kept as is,
-
Empty string ⇒ Reset iTop object field
You must use your own feeding mechanism for loading
Null
values in synchro_data_xxx
table, as
synchro_import.php can't
do it
The Update Policy column
The column Update Policy defines the behavior of the updated attribute when editing the iTop object. The possible values are:
-
Locked
: the attribute will be completely driven by the data synchronization and will appear as read-only (with a small lock icon next to it) in the iTop user interface, -
Unlocked
: the attribute will be updated by the synchronization (whenever the source data changes) but end-users can still modify the attribute in parallel inside iTop, -
Initialize if empty
: the synchronization will only set a value of this attribute if the value is empty in iTop. The attribute remains modifiable inside iTop.
The Reconcialiation key column
Finally, the Reconciliation key column defines
how to process attributes which are references to another iTop
object (i.e. Foreign keys). The supplied data can contain either
the actual value of the foreign key (id
) or another
attribute such as the name
of the object; which must
uniquely identify the object.
name
AND organization
).
If such a need arises, then you must use an ETL or a script to
perform the desired reconciliation before providing the
id
to iTop.Status
The Status tab displays information about the execution of the synchronization. If the synchronization has never been run, the tab displays the following message:
This synchro was never run. No log yet.
Otherwise the tab displays a schema similar to the one below:
Synchronization history
On the left of the screen, each execution of the synchro is displayed in an historical list, showing the overall status, the start date and time of the execution.
Click on an element of this list to show the corresponding status in detail, displayed in the two columns on the right.
The column at the left displays the status for the records in
the synchro_data_xxxx
table:
-
New
records are records which were added in the table since the previous execution of the synchro, -
Existing
records are records which were existing before and have been loaded again since the previous execution of the synchro, -
Disappeared
records are records which were not loaded since more than thefull_load_interval
duration. -
Ignored
records corresponds to objects no longer existing in iTop.
The column at the right displays the consequences of the synchro, in terms of modifications on the associated objects in iTop.
-
For each
New
record, the consequences can be either:-
The creation of the new object in iTop (
Created
), -
The update of an already existing object in iTop (
Updated
), -
The simple association with an object in iTop, which is already up-to-date (
Unchanged
), -
Or an
Error
when trying to either create or update an object in iTop (most probably because of inconsistent data).
-
-
For each
Existing
record, the consequences can be either:-
The update of the associated object (
Updated
), -
No change at all (
Unchanged
) -
Or an error when trying to update the associated object (
Errors
)
-
-
For each
Disappeared
record, the consequences (depending on the “Deletion rules”) can be either:-
The associated object in iTop has been deleted (
Deleted
), -
The associated object in iTop has been updated (
Updated
), -
The associated object in iTop was left unchanged (
No Action
), -
Or an error occurred when trying to update or delete the associated object in iTop (
Errors
).
-
Replicas
In iTop, the Replicas are used to store the status
information and the relation between the raw data loaded in the
synchro_data_xxx
table and the associated iTop
object.
The Replicas for a given data synchronization source can be displayed by clicking on the links at the top of the status tab. A Replica is displayed as shown below:
The content of the replicas is useful to understand the behavior of the data synchronization mechanism (and to get more information about the errors), but its data are maintained by the synchronization and should never be modified manually.
Fields of a Replica | Purpose |
Synchro Data Source | Synchro Data Source to which this Replica belongs |
Destination object (ID) | iTop object id corresponding to that replica. Can be empty when the replica is new and could not be synchronized due to an error |
Destination type | The Target class of the Synchro Data Source, which is also the class of the Destination object |
Status | Status of the Replica: Modified, New, Obsolete, Orphan, Synchronized |
Last seen | When that replica was seen for the last time in
the source data (same primary_key ) |
Object created? | Was the iTop object created by the Data Synchro? |
Last Error | Error |
Warnings | Warning message when some non blocking error occured |
Creation date | When the iTop object was created by the Synchro Data Data |
Last Modified Date | When that replica was modified by the source for the last time |
Replica's status | Meaning |
New | Newly received replica. Temporary status before syncho_exec.php occurs to create the iTop object |
Modified | Modified replica. Temporary status before syncho_exec.php occurs to update the iTop object |
Synchronized | Replica and iTop object are aligned |
Obsolete | Data is no more in the Source, iTop object was not deleted, otherwise the replica would be as well |
Orphan | This should not happen. In this situation the synchronized iTop object has disappeared without deleting the replica at the same time. Such replica can safely be manually deleted |
Using MySQL statements to populate the synchro_data_xxxx table
If you populate the table synchro_data_xxx
using
SQL commands (either with an ETL or a home made script), you
must take care of never creating duplicate
records. Remember that each record in the
synchro_data_xxx
table corresponds to exactly one
object in iTop.
In order to avoid such duplicates, you must
first query for an existing record before inserting a new record
into the table. To speedup this query you can either construct a
value that uniquely identifies the “source” object and store this
value in the “primary_key” column of the table (which has a unique
index), or use any combination of the record's column and alter the
definition of the synchro_data_xxxx
table to add your
own indexes.
INSERT … ON DUPLICATE KEY UPDATE
to insert records in the synchro_data_xxx
table. This
is not supported by the internal iTop mechanism which relies on the
“BEFORE INSERT” trigger.synchro_data_xxxx
table, do not use
the MySQL statement TRUNCATE TABLE
, since this
statement will actually drop and reconstruct the table… but not its
triggers! Instead use DELETE FROM
synchro_data_xxxx
.Then trigger the execution of the data synchronization between the iTop objects and the replicas of the temporary table by executing the page “synchro/synchro_exec.php” either in command line mode, or by invocating the web page (using wget for example).
Using synchro_import to populate the synchro_data_xxxx table
See: synchro_import.php
How to specify a list of related objects (link set)
Some classes have attributes called Link Sets. Example: UserLocal.profile_list
You can synchronize those lists by writing all the information in the corresponding column:
Here is an example specifying that the login will have one profile: 'Portal User'
INSERT login, profile_list INTO data_synchro_userlocal_1 VALUES ('johndoe', 'profileid->name:Portal User;reason:Customer')...;
You can also specify several links (several profiles).
More information in Import a LinkedSet