:: Version 2.6.0 ::

Import data via CLI

Purpose

iTop provides a generic web service to import data in itop based on a source csv file. This service can be used locally on the server via the command line interface or remotely via an HTTP post request.

Using this service, you can easily import data in iTop via a batch script that extract data from a third party tool and generate a csv formatted file. For instance a script data extract a list of contact from your Active Directory.

When it comes to import a real huge amount of data, the CLI mode will be a must because you will not reach the limitations of the web server (timeout and memory limits).

As opposed to the interactive CSV import, the CLI import lets you specify any attribute to reconcile the external keys. This is a workaround when some reconciliation keys are missing. Example: the class Software has only the name as reconciliation key (default setting in the datamode). You need to import DBServer objects with their respective software information (an external key pointing to the software object). The interactive CSV import will issue errors whenever the same software has two versions (same name, ambiguous!). In the CLI import you can specify both Software→Name and Software→Version and resolve the ambiguity issue.

Usage

Command line interface

All the parameters of the command line are listed in the section “Arguments” here after. The following line is an example of the mandatory argument. All arguments starts with --

php /var/www/itop/webservices/import.php --auth_user=login --auth_pwd=password --class=Organization --csvfile=file --reconciliationkeys="list of keys"

The reconciliation keys are very important to identify uniquely the element to be imported in iTop. The reconciliation keys must be present in the data file. As opposed to the interactive CSV import, you are not limited to the list of reconciliation keys defined by default for the class.

When the reconciliation can be performed on an external key, you can specify any of the target class attributes. Example: you are importing Person objects and need to reconcile on the location. You can specify Location→postal_code as one of the reconciliation keys (weird but…).

Moreover, the external key will be uniquely identified by all its attributes specified in the reconciliation scheme. In our example, if you consider that Name/Country uniquely identifies the locations, the use both Location→Name and Location→Country in your reconciliation scheme.

Arguments

Argument Description Defaut value
param_file Parameters file - see Parameters file -
auth_user User login - CLI mode only -
auth_pwd User password - CLI mode only -
class Class of the loaded objects Mandatory!
csvdata CSV contents (HTTP mode only, data may be posted through this page argument) Mandatory in HTTP mode
csvfile Path and name of the source file Mandatory in CLI mode
charset Character set encoding of the CSV data: UTF-8, ISO-8859-1, WINDOWS-1251, WINDOWS-1252, ISO-8859-15 use csv_file_default_charset from iTop configuration file, which default is 'ISO-8859-1'
date_format Input date format (used both for dates and datetimes) - Examples: Y-m-d, d/m/Y (France) <blank>
separator Column separator in CSV data , (coma)
qualifier Qualifier in CSV data “ (double quote)
output retcode to return the count of lines in error, “summary” to return a concise report, “details” to get a detailed report (each line listed) summary
reconciliationkeys label or code (depends on no_localize flag) of the columns (comma separated) used to identify existing objects and update them, or create a new one <blank>
simulate If set to 1, then the load will not be executed, but the expected report will be produced 0
with_archive If set to 1, then the import will retrieve archived objects as object to update and as valid external key 0
comment Comment to be added into the change log <blank>
no_localize If set to 0, then column headers, values and reconciliationkeys are supposed to be localized in the language of the logged in user. Set to 1 to use internal attribute codes and values (enums) 0 = label not case sensitive
Each reconciliation key must be present in the source file header and written the same way

Output

The commande line will return a summary report (see below) summarizing the argument used for the import and the result of the import:

  • Number of elements created
  • Number of elements updated
  • Number of elements in error

In case you have specified the option –result=details you will get one line for each element imported with the status of the import (see example below)

Result

#Output format: details
#Class: Organization
#Separator: ;
#Qualifier: \"
#Charset Encoding:UTF-8
#Data Size: 26
#Data Lines: 1
#Simulate: 0
#Columns: name, code
#Reconciliation Keys: name
#Change tracking comment: 
#Issues: 0
#Warnings: 0
#Created: 1
#Updated: 0
#Unchanged: 0
Line;Status;Object Class;Object Id;name (Name);code (Code)
0;created;Organization;3;\"WorldCompany\";\"WCY\"

Possible errors

Missing argument

ERROR: Missing argument 'class'

Such error is displayed when a mandatory argument is missing

Wrong argument

#Unknown class: 'toto'

This error occurs if the class specified does not exists in iTop. Be careful the class name is case sensitive.

#Unknown reconciliationkeys: 'attribute'

This error occurs if at least one column in the reconciliation keys list is not known.
The reason can be:

  • a code instead of a label was provided or vice-versa
  • the reconcialiation key is a External Field
  • the reconcialiation key is not present in the csv file
An External Field such as org_name cannot be used as a reconciliation key, use instead org_id→name
#Unknown column: 'Last Nme'

This error occurs if a column is unknown in the csv file

When using no_localize=0 which is the default, you must use the labels in the CSV file headers and in reconciliationkeys. For historical reason, in CLI mode, labels are not case sensitive so reconciliationkeys=“name,owner organization→name” works as well as =“Name,Owner organization→Name” which would be the standard.

Allowed users

Administrators will always be allowed to execute this page.

Non administrators will be allowed if they are allowed to modify objects of the given class.

Importing Images and Files

Some classes contain an image or a file document as one of their field. For example, the Person class, contains a Picture field which holds the picture of this person. When editing a Person interactively, the user can upload an image from her computer to provide the picture of the person. However when performing a CSV import, it is not possible to “upload” such a file or to put its content inside the CSV file itself, since the CSV format does not support this.

In such a case the CSV file must provide - in the appropriate CSV column - an URL to upload the file from. This URL must be accessible from the iTop server and return the expected image (or file document). If the upload fails or if the format of the uploaded document does not match the expected format for the field (for example if the URL does not return an image for the picture field), the import/update will fail for this field (and the whole line of the CSV will be rejected).

Limitations:
  • The URL must point directly to the image to upload (redirections are not supported)
  • The URL must be accessible from the iTop web server (beware some public websites may deny access to a PHP script - they check the HTTP User Agent string)
  • The URL must not require authentication, since the upload is done by the iTop server itself
  • URLs pointing to iTop itself (like the URLs provided by CSV/Excel export) are supported. But only if they point to the same iTop instance (in such a case the import will parse the provided URL and will read the document from the iTop database).
  • If you use an administrator account to perform the CSV import, you can specify a path to a file located on the iTop server itself (but this works only for administrators).

Use case: importing persons with their corresponding login into iTop

This use case describes how to import persons and their login in iTop. This is one way to load those information from an Active Directory. In that case you just have to write a script that create the two csv files person.csv and login.csv from your Active Directory.

Importing persons

We would like to create two contacts:

  • Claude Monet who belongs to organization Demo
  • Gustave Flaubert who belongs to organization IT Department

In this example the two organizations already exist in iTop

The class to import is UserLocal, and the data in csv file (login.csv) are:

person.csv
Last Name,Status,Organization->Name,Email,Phone,Notification,Function,First Name
"Monet","Active","Demo","monet@demo.com","","yes","","Claude"
"Flaubert","Active","IT Department","flaubert@it.com","","yes","","Gustave"

then run the following command line:

php webservices/import.php --auth_user=admin --auth_pwd=admin --csvfile="person.csv"
 --class="Person" --reconciliationkeys="Email"

Importing user accounts

We would like to create two user account:

  • support is configuration manager and change supervisor, related to the contact flaubert@it.com
  • portal is a Porta user related to the contact monet@demo.com

In this example the login are linked to a contact via the email and we import the Profiles that are linked to the contact. Look at the particular syntax for importing the profiles. For more information about importing link sets refer to Import a linkset

The class to import is UserLocal:

login.csv
Contact (person)->Email,Login,Language,Password,Profiles
"flaubert@it.com","support","EN US",1234,profileid->name:Configuration Manager|profileid->name:Change Supervisor
"monet@demo.com","portal","EN US",1234,profileid->name:Portal user

then run the following command line:

php webservices/import.php --auth_user=login--auth_pwd=password --csvfile="login.csv"
  --class="UserLocal" --reconcialiationkeys="Login"

Importing Locations

location.csv
"name","org_id->name","country","city"
"Bordeaux","Demo","France","Bordeaux"
"Grenoble","Demo","France","Eybens"
"Paris","Demo","France",""

then run the following command line:

php webservices/import.php --auth_user=login --auth_pwd=password --csvfile="location.csv"
   --charset="UTF-8" --no_localize=1 --class="Location" --reconcialiationkeys="name,org_id->name"

Limitations / Known issues

  • External Fields (such as such as virtualhost_id->org_name) cannot be used as reconciliation keys.
  • Workaround: org_name can be replaced by org_id->name
  • Limitation: the arrow operator (->) cannot be used mutiple times (i.e. virtualhost_id->org_id->name is not supported).

History

  • 0.9: first implementation of this feature
  • 2.0: added the flag no_localize
2_6_0/advancedtopics/importcli.txt · Last modified: 2019/01/09 16:40 (external edit)
Back to top
Contact us