Data collector for vSphere

This stand-alone application collects information about a whole data center from a vSphere server (using the vSphere Web Services) and synchronizes this information with iTop using several Synchronization Data Sources.

• Automated inventory of Servers (with their Brand, Model), Hypervisors (with their OS Family and OS Version), Farms, Virtual Machines (with their OS Family and OS Version).
• Optional inventory of IPv4 addresses and logical interfaces
• The collector can reside on any system with web access to both vSphere Web Services and iTop.
• Automatic creation and update of the Synchronization Data Sources in iTop.
• Starting with version 1.0.12 the collection mechanism is somehow extensible

This version of the collector does not collect the Data Stores (they do not exist in the iTop standard data model). Logical interfaces may be collected in the case where TeemIp (module or standalone) is used.

If IP collection is enabled, only IPv4 addresses are collected, not IPv6.

The current version of the collector is designed to collect information from a single vSphere server. To collect and reconcile information form several vSphere servers into one iTop, check the section Collecting from several vSphere servers below.

• PHP Version 5.3.0 (support of namespaces is required for the vSphere API library)
• An access to the vSphere web services API
• An access to the iTop web services (REST + synchro_import.php and synchro_exec.php)
• + Data collector Base requirements.

• Expand the content of the zip archive on a folder on the machine that will run the collector application. This machine must have a web access to both the vSphere server and the iTop server.
• Edit the content of the file conf/params.local.xml to suit your installation, supplying the appropriate credentials to connect to vSphere and iTop.

For example create the file params.local.xml in the conf directory with the following XML content:

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
  <itop_url>https://localhost/</itop_url>
  <itop_login>admin</itop_login>
  <itop_password>admin</itop_password>
  <vsphere_uri>192.168.10.12:443</vsphere_uri>
  <vsphere_login>admin</vsphere_login>
  <vsphere_password>admin</vsphere_password>
  <contact_to_notify>john.doe@demo.com</contact_to_notify>
  <synchro_user>admin</synchro_user>
  <default_org_id>Demo</default_org_id>
</parameters>

  <vsphere_connection_options>
        <ssl>
                <verify_peer>0</verify_peer>
                <verify_peer_name>0</verify_peer_name>
                <allow_self_signed>1</allow_self_signed>
        </ssl>
  </vsphere_connection_options>

The values collected for Brands, Models and OS Family may be very variable because they came from non homogeneous sources (BIOS, operating system…). The data collector contains a simple mechanism to filter / normalize the values via a manual tuning, before importing them into iTop: a simple mapping table.

Mapping tables are defined and maintained in the configuration file params.local.xml (you can copy an example definition from params.distrib.xml and ajust it at your will). brand_mapping is used for filtering the Brand values, model_mapping is used for Models, os_family_mapping is used for OS Families and os_version_mapping is used for OS Versions.

A mapping table is simply an ordered list of patterns and replacement strings. For each entry in the list, if the supplied value matches the pattern, it is replaced by the replacement string. The patterns are processed from the first one (top of the list) to the last one (bottom of the list). The processing stops after the first successful match.

The format of each entry in the mapping table is:

<pattern>/regular_expression/replacement_string</pattern>

Where:

• regular_expression is a PCRE regular expression to be matched against the input value,
• replacement_string is the resulting value if the input value matches the regular expression.
• Note: any character can be use as a delimiter around the regular expression, not only /. But the delimiter character can be present neither in the expression itself nor in the replacement string or the result will be undetermined.

The replacement string is the literal value to use, with the exception of placeholders in the form %1$s, %2$s… The placeholder %1$s corresponds to the part of the input string that matches the whole regular expression. Placeholders %2$s, %3$s corresponds to matching groups (i.e. parentheses) in the regular expression.

Example of a mapping table for the brands:

  <brand_mapping type="array">
    <!-- Syntax /pattern/replacement where:
      any delimiter can be used (not only /) 
      but the delimiter cannot be present in the "replacement" string
      pattern is a RegExpr pattern
      replacement is a sprintf string in which:
          %1$s will be replaced by the whole matched text,
          %2$s will be replaced by the first matched group, if any group is defined in the RegExpr
          %3$s will be replaced by the second matched group, etc...
    -->
    <pattern>/IBM/IBM</pattern>
    <pattern>/Hewlett Packard/Hewlett-Packard</pattern>
    <pattern>/Hewlett-Packard/Hewlett-Packard</pattern>
    <pattern>/Dell/Dell</pattern>
    <pattern>/.*/%1$s</pattern>
  </brand_mapping>

The result of such a mapping table will be:

• Any brand containing the letters IBM (case sensitive) will become exactly IBM
• Any brand containing Hewlett Packard (case sensitive) will become exactly Hewlett-Packard (notice the added dash between Hewlett and Packard)
• Any brand containing Hewlett-Packard (case sensitive) will become exactly Hewlett-Packard
• Any brand containing the letters Dell (case sensitive) will become exactly Dell

The collector automatically detects if TeemIp is present on the remote iTop application (as a module or even as a stand-alone application). Should that be the case, then the following parameters may trigger IP addresses and logical interfaces collection.

<?xml version="1.0" encoding="UTF-8"?>
  <parameters>
    ...
    <teemip_options type="hash">
      <collect_ips>yes</collect_ips>
      <default_ip_status>allocated</default_ip_status>
      <manage_ipv6>no</manage_ipv6>
      <manage_logical_interfaces>yes</manage_logical_interfaces>
      </teemip_options>
    ...
  </parameters>

Starting with version 1.0.12 of the application, Server and Hypervisor collectors are configurable by adding extra definitions in the XMl parameter file.

This configuration contains two parts:

1. The first part (under the <json> tag) allows to alter the definition of the Synchro Data Source (normally fixed by the .json file associated with the collector class). This is useful to synchronise a new field which was not synchronized by default, or to change a reconciliation rule on a field, etc… Any field under the attribute_list part of the JSON file can be modified by specifying a new value in the configuration.
2. The second part (the <source> tag) defines the way to collect the data for the attribute. This tag must contain a (partial) PHP expression relative to the Hypervisor object (which is actually a HostSystem in the vSphere model)

Both parts of the configuration can be used independently (one can alter only the JSON or only the collection, or both).

The example below configures the collection of the “Serial Number” on Servers (from the otherIdentifyingInfo[EnclosureSerialNumberTag]) and also uses the collected serial number as the reconciliation key between Hypervisors and physical Servers (the field server_id).

config.local.xml

<?xml version="1.0" encoding="UTF-8"?>
<parameters>
 
  ...
 
  <custom_synchro>
    <vSphereHypervisorCollector>
      <fields>
        <server_id>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
          <json>
            <reconciliation_attcode>serialnumber</reconciliation_attcode>
          </json>
        </server_id>
      </fields>
    </vSphereHypervisorCollector>
    <vSphereServerCollector>
      <fields>
        <serialnumber>
          <source>hardware->systemInfo->otherIdentifyingInfo[EnclosureSerialNumberTag]</source>
        </serialnumber>
      </fields>
    </vSphereServerCollector>
  </custom_synchro>
 
  ...
 
</parameters>

The execution of the command line will:

1. Connect to iTop to create the Synchronization Data Sources (or check their definition if they already exist, updating them if needed)
2. Connect to vSphere to collect the information about the Hypervisors, Farms, Virtual Machines
3. Upload the collected data into iTop
4. Synchronize the collected data with the existing iTop CIs.

If you have troubles connecting to the vSphere server, try the following troubleshooting steps:

1. Check that the connection from the system running PHP to the vSphere server is actually possible. Use a command line tool like wget to connect to the vSphere server. For example if your vsphere_uri is 192.168.10.12:443, try:

   wget --no-check-certificate -O - https://192.168.10.12:443

   If the connection does not succeed, there may be a firewall blocking your requests, or that vSphere is condifured to operate on a different port (for example 9443)… ask your IT department about it.

2. Once the connection seem to go fine, configure the proper vsphere_uri value in conf/params.local.xml and, from the command line (in the collectors subdirectory) launch:

   php check_soap.php

   The expected output (with vsphere_uri = 192.168.10.12:443) is:

   Connecting to https://192.168.10.12:443/sdk
   
   Ok, the response looks like a valid SOAP response.
   
   --------------------- DEBUG ----------------
   The request returned:
   <?xml version="1.0" encoding="UTF-8"?>
   <soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
    xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <soapenv:Body>
   <RetrieveServiceContentResponse xmlns="urn:vim25"><returnval><rootFolder type="Folder">group-d1</rootFolder><propertyCollector type="PropertyCollector">propertyCollector</propertyCollector><viewManager type="ViewManager">ViewManager</viewManager><about><name>VMware vCenter Server</name><fullName>VMware vCenter Server 4.1.0 build-345043</fullName><vendor>VMware, Inc.</vendor><version>4.1.0</version><build>345043</build><localeVersion>INTL</localeVersion><localeBuild>0</localeBuild><osType>win32-x86</osType><productLineId>vpx</productLineId><apiType>VirtualCenter</apiType><apiVersion>4.1</apiVersion></about><setting type="OptionManager">VpxSettings</setting><userDirectory type="UserDirectory">UserDirectory</userDirectory><sessionManager type="SessionManager">SessionManager</sessionManager><authorizationManager type="AuthorizationManager">AuthorizationManager</authorizationManager><perfManager type="PerformanceManager">PerfMgr</perfManager><scheduledTaskManager type="ScheduledTaskManager">ScheduledTaskManager</scheduledTaskManager><alarmManager type="AlarmManager">AlarmManager</alarmManager><eventManager type="EventManager">EventManager</eventManager><taskManager type="TaskManager">TaskManager</taskManager><extensionManager type="ExtensionManager">ExtensionManager</extensionManager><customizationSpecManager type="CustomizationSpecManager">CustomizationSpecManager</customizationSpecManager><customFieldsManager type="CustomFieldsManager">CustomFieldsManager</customFieldsManager><diagnosticManager type="DiagnosticManager">DiagMgr</diagnosticManager><licenseManager type="LicenseManager">LicenseManager</licenseManager><searchIndex type="SearchIndex">SearchIndex</searchIndex><fileManager type="FileManager">FileManager</fileManager><virtualDiskManager type="VirtualDiskManager">VirtualDiskManager</virtualDiskManager></returnval></RetrieveServiceContentResponse>
   </soapenv:Body>
   </soapenv:Envelope>
   ---------------------------------------------

   If the result is not the expected one, check that your vSphere server is properly configured to run as HTTPS (which should be the default).

In the vSphere web services SDK, Hypervisors and physical Servers are represented by the same object HostSystem.

The information from the vSphere HostSystem object is imported in iTop into the Server object using the following mapping:

The information from the vSphere HostSystem object is imported in iTop into the Hypervisor object using the following mapping:

In the vSphere web services SDK, a Farm is represented by the object ClusterComputeResource.

The information from the vSphere ClusterComputeResource object is imported in iTop into the Farm object using the following mapping:

The list of hypervisors belonging to a farm is collected via the property: ClusterComputeResource → host → name.

In the vSphere web services SDK, a Virtual Machine is represented by the object VirtualMachine. The information from the vSphere VirtualMachine object is imported in iTop into the VirtualMachine object using the following mapping:

If enabled (TeemIp required), the information from the vSphere IPv4 addresses is imported in iTop into the IPv4Address object using the following mapping:

If enabled (TeemIp required), the collector may register the logical interfaces attached to virtual machines. These are imported in iTop into the LogicalInterface objects using the following mapping: