iTop monitoring

This extensions is designed to help administrators monitor their iTop(s) via http(s) endpoint(s) and oversee incidents coming in production.

With it you can collect metrics, store your infrastructure history (time series) and compute alert rules to notify your SRE/Devops/Sysadmin teams.

You can define your own monitoring endpoints and secure them through iTop configuration.

For each endpoint you can configure which set of KPIs will be returned.

If you have specific requirement that are not covered by the existing metrics, you can also code your own metrics via PHP APIs.

You can pick below available KPIs to build/define each endpoint output.

Currently the extension supports only Prometheus format.
We believe it is not too complex to integrate other monitoring tools such as Nagios, Centreon or Icinga.

There are no aggregation (sum) metrics based on OQL searches except the count. Today you can do it only by coding your own PHP metric.

iTop 2.7

• Use the Standard installation process for this extension.

This documentation is structured in 3 parts:

iTop can expose further specific Http(s) endpoints dedicated to your own observability purpose.

You have to make sure Prometheus polls your iTop properly.

And then you can decide which monitoring metrics you want to expose via which endpoint among:

• iTop application metrics: brought by this extension with fully written metrics to check installation, backups, user activity,…
• iTop OQL metrics: you can add functional metrics based OQL query engine and on your specific datamodel (ticket/cmdb/others).
• iTop custom metrics: this is the extensible way of writing your own metrics in PHP by following the generic API (PHP snippets brought in your own extensions, delta XML).

With below iTop configuration section you can define:

• how you secure it as a monitoring agent

to which networks it will answer and for which token it will respond (HTTP 500 otherwise)

• a list of endpoints

each endpoint will be dedicated for a specific purpose or team. for example you can have a monitoring from your infra/sysadming team. and other ones from your support teams for example. each one will have its own metric computed and returned.

config-itop.php

  'combodo-monitoring' => array (
        'authorized_network' => ['10.0.0.0/16'],
        'access_token' => 'gabuzomeu123',
        'metrics' => array (
            'metric_group_name1' =>
                array (
                   //metrics...
                ),
        ),
   ),

monitoring page URL will be https://ITOPHOST/pages/exec.php?access_token=gabuzomeu123&collection=metric_group_name1&exec_env=production&exec_module=combodo-monitoring&exec_page=index.php

Once iTop monitoring endpoints are ready. you can tune Prometheus to poll iTop agents and collect data. here is a configuration example

prometheus.yml

  - job_name: 'itop'
    metrics_path: '/pages/exec.php'
    scheme: "https"
    params:
      exec_module: ['combodo-monitoring']
      exec_page: ['index.php']
      exec_env: ['production']
      collection: ['metric_group_name1']
    file_sd_configs:
      - files:
        - targets/itop/*.yml

client123.yml

- labels:
    role: 'saas-client'
    slug: 'client123'
    __param_access_token: 'gabuzomeu123'

  targets:
    - client123.itop-saas.com

this metric is useful to follow ongoing active sessions. it exposes a 3 types of metrics:

• a Counter of all active sessions
• a Sum of the elapsed time in seconds of all active sessions
• a Max of the elapsed time in seconds of among all active sessions

Each of this metric is computed with the login_mode and context labels. that way you can have inside information of how many sessions are authenticated and for which purpose (UI/Rest/Synchro) and how they were made (basic/token/LDAP/SSO/…).

To enable this KPI you have to enable tracking file generation:

config-itop.php

    'sessions_tracking.enabled' => true,

for more details: https://wiki.combodo.com/doku.php?id=latest:admin:itop_configuration_file#s-t

config-itop.php /combodo-monitoring

'itop_active_sessions' => 
array (
  'description' => 'itop version',
  'custom' => 
  array (
    'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ActiveSessionReader',
  ),
),

output combodo-monitoring

# Active session count
itop_active_session_count{login_mode="no_auth",context=""} 10
# Active session count
itop_active_session_count{login_mode="token",context="REST/JSON"} 8
# Active session count
itop_active_session_count{login_mode="token",context="Synchro"} 1
# Active session count
itop_active_session_count{login_mode="form",context=""} 1
# Sum of active session elapsed time in seconds
itop_active_session_elapsedinsecond_sum{login_mode="token",context="REST/JSON"} 0
# Sum of active session elapsed time in seconds
itop_active_session_elapsedinsecond_sum{login_mode="token",context="Synchro"} 0
# Sum of active session elapsed time in seconds
itop_active_session_elapsedinsecond_sum{login_mode="form",context=""} 0
# Max elapsed time in seconds amoung active sessions
itop_active_session_elapsedinsecond_max{login_mode="token",context="REST/JSON"} 0
# Max elapsed time in seconds amoung active sessions
itop_active_session_elapsedinsecond_max{login_mode="token",context="Synchro"} 0
# Max elapsed time in seconds amoung active sessions
itop_active_session_elapsedinsecond_max{login_mode="form",context=""} 0

login_mode=“no_auth” means sessions made without any authentication (like status.php).

this metric is usefull to check if your iTop was properly installed (setup metric).

itop_version will show which PHP core code version is installed. while itop_setup_version will expose which version currently installed/running.

if both metric values are different, this is a bad sign: your setup installation may have failed.

config-itop.php /combodo-monitoring

'itop_version' => 
array (
  'description' => 'itop version',
  'custom' => 
  array (
    'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopVersionReader',
  ),
),
'itop_setup_version' => 
array (
  'description' => 'itop setup version',
  'custom' => 
  array (
    'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopSetupVersionReader',
  ),
),

output combodo-monitoring

# iTop applicative version
itop_version{} 9997
 
# iTop after setup version (code + datamodel)
itop_setup_version{} 9997

this metric is also a setup installation check. if current metric value equals 1 then your iTop is stuck in readonly mode. and no user will be able to edit something… this is a bad sign: your setup installation may have failed.

config-itop.php /combodo-monitoring

'itop_readonly' => 
array (
  'description' => 'custom class (custom)',
  'custom' => 
  array (
    'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopReadOnlyCustomerReader',
  ),
),

output combodo-monitoring

# custom class (custom)
itop_readonly{} 0

this metric generator will expose 3 different metrics:

• the number of backup generated and stored on your itop
• the size of the latest one
• the age in hours of the latest one

if you count on backup generation automation this metric could help monitoring it is still running. indirectly it is a sign itop cron may faild somehow or may not be running.

config-itop.php /combodo-monitoring

'itop_backup' => 
array (
  'description' => 'itop backup',
  'custom' => 
  array (
    'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopBackupReader',
  ),
),    

output combodo-monitoring

# count iTop backup files.
itop_backup_count{} 1
 
# last iTop backup file size in bytes.
itop_backup_lastbackup_inbytes_size{} 72156
 
# last iTop backup file age in hours.
itop_backup_lastbackup_ageinhours_count{} 11

after a while and depending of your iTop activity (lot of users/data/automation) you may be facing performance issues. you may also have to do some cleanup.

current metric is usefull to detect tables that have grown a lot. they come in the radar though treshold configuration.

with below metric any object table that exceeds 250 MB (default) or 100000 objects will be exposed by current metric:

config-itop.php /combodo-monitoring

    'itop_big_table' => 
    array (
      'description' => 'list huge db tables',
      'custom' => 
      array (
        'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopDbBigTableReader',
      ),
    ),

output combodo-monitoring

itop_big_table_objectcount{table="priv_change"} 17206468
itop_big_table_diskspace_in_megabytes{table="priv_change"} 1541

you can change default thresholds (objects and/or disk space). with below metric any object table that exceeds 500 MB (default) or 500000 objects will be exposed by current metric:

config-itop.php /combodo-monitoring

    'itop_big_table' => 
    array (
      'description' => 'list huge db tables',
      'custom' => 
      array (
        'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopDbBigTableReader',
      ),
      'default_objectcount_threshold' => 500000,
      'default_diskspace_threshold' => 500,
    ),

and you can even have specific thresholds per tables (objet and/or disk). with below metric any object table that exceeds 500 MB (default) or 500000 objects will be exposed by current metric. but priv_change will appear in this metric if you have more tant 700K objects or 700 MB disk space used:

config-itop.php /combodo-monitoring

    'itop_big_table' => 
    array (
      'description' => 'list huge db tables',
      'custom' => 
      array (
        'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopDbBigTableReader',
      ),
      'default_objectcount_threshold' => 500000,
      'default_diskspace_threshold' => 500,
      'priv_change' => array(
          'objectcount_threshold' => 700000,
          'diskspace_threshold' => 700,
      )
                ]
    ),

this metric is usefull to check mail to ticket automation. it counts the number of failed configured mailbox. in other words if the count is greater than one you may have mailbox issues.

it means itop is not turing mail reception into tickets. and vice versa no ticket modification will trigger any mails sent to the outside.

config-itop.php /combodo-monitoring

'itop_mailbox_failed_connection_count' => 
array (
  'description' => 'Failures to connect to polled mailboxes',
  'custom' => 
  array (
    'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopMailboxReader',
  ),
),

output combodo-monitoring

# Failed connections to a mailbox
itop_mailbox_failed_connection_count{} 0

you can also make sure notification are properly sent. if not being alerted based on lelow OQL metric:

config-itop.php /combodo-monitoring

                    'itop_count_async_errors' =>
                    array (
                      'description' => 'Number of errors while processing Asynchronous Tasks for the last 7 days',
                      'oql_count' =>
                      array (
                        'select' => 'SELECT AsyncTask WHERE status="error" AND last_attempt > DATE_SUB(NOW(), INTERVAL 7 DAY)',
                      ),
                    ),

iTop is interacting with a database. deadlock may occur. in such case some logs are generating. this metric is counting the number of recent deadlock.

in fact deadlock is an example of how to use this metric generator. you could also add other metrics to check some error pattern in iTop local log files. and monitor if the number of ERRORS have increased after some intervention (infrastructure wise or on iTop during upgrades or datamodel changes).

config-itop.php /combodo-monitoring

'itop_deadlocks' => 
    array (
      'description' => 'custom class (custom)',
      'custom' => 
      array (
        'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\CountTextOccurrences',
        'file' => 'APPROOT/log/deadlock.log',
        'needle' => 'deadlock detected: user=',
      ),
    ),

output combodo-monitoring

# custom class (custom)
itop_deadlocks{} 0

this metric is combined with Brute force extension. it is usefull to oversee iTop activity. it counts the number of users logged in in the past minute per profiles (administrator/portal user/…).

config-itop.php /combodo-monitoring

'itop_eventlogin' => 
array (
  'description' => 'itop eventlogin with user profiles and type',
  'custom' => 
  array (
    'class' => '\\Combodo\\iTop\\Monitoring\\CustomReader\\ItopEventLoginReader',
  ),
),

output combodo-monitoring

# nb of connection in the last 1 hour.
itop_eventlogin{profiles="administrator",account_type="userlocal"} 1

in this section you can add as many metrics you want based on your datamodel. you only have to rely on OQL engine.

you can for example add CMDB (number of objects being created for example) or ticketing metrics (for example on your teams SLAs). (almost) anything possible from OQL can be configured here.

config-itop.php /combodo-monitoring

    'itop_count_tickets' => 
    array (
      'description' => 'Number of tickets',
      'oql_count' => 
      array (
        'select' => 'SELECT UserRequest',
      ),
      'static_labels' => 
      array (
        'env' => 'prod',
      ),
    ),

output combodo-monitoring

# Number of tickets
itop_count_tickets{env="prod"} 1

this metric is interesting. it is using OQL generic mechanism. but it is also an example on how to monitor collectors. you only have to configure your collectors to send event issue in case of collect/synchro incident.

this metric let you oversee any failed collector in the past minute. this is usefull to make sure you have up to date data toward external systems…

it counts objects using GROUPBY verb. labels are linked to found GROUPBY values.

config-itop.php /combodo-monitoring

'collector_issue_count' => 
    array (
      'description' => 'event issue count.',
      'oql_groupby' => 
      array (
        'select' => 'SELECT e FROM EventIssue AS e WHERE date > DATE_SUB(NOW(), INTERVAL 1 MINUTE)
AND userinfo = "Collector"',
        'groupby' => 
        array (
          'impact' => 'e.impact',
          'issue' => 'e.issue',
        ),
      ),
    ),

output combodo-monitoring

collector_issue_count{impact="object-storage-collector",issue="Collect-BackupArchivalCollector",userinfo="Collector"} 2

this is close to oql_group_by metrics. instead of counting objects, it exposes a specific object field value (below PACKAGE.version) per GROUP BY labels (below INSTANCE.slug field).

config-itop.php /combodo-monitoring

'manager_instance_applied_version' => 
    array (
      'description' => 'expected applied itop version',
      'oql_select' => 
      array (
        'select' => 'SELECT p,i FROM
Instance AS i
JOIN Package AS p
ON i.actual_package_id = p.id',
        'labels' => 
        array (
          'slug' => 'i.slug',
        ),
        'value' => 'p.version',
      ),
    ),

output combodo-monitoring

manager_instance_applied_version{slug="client1"} 9994
manager_instance_applied_version{slug="client2"} 9997
manager_instance_applied_version{slug="client3"} 9997
manager_instance_applied_version{slug="client4"} 9997
manager_instance_applied_version{slug="client5"} 9997
manager_instance_applied_version{slug="client6"} 9997

If current extension does not provider your metrics easily. you can code your own ones by creating a provider class. This class can generate further different kind of metrics. For example backup count/size/errors/…

namespace Combodo\iTop\YourExtension\Monitoring;


class MyMetricReader implements CustomReaderInterface
{
    private $aMetricConf;
    private $sMetricName;

    /**
    * $aMetricConf: iTop configuration subsection dedicated to your metric
    */
    public function __construct($sMetricName, $aMetricConf)
    {
        $this->aMetricConf = $aMetricConf;
        $this->sMetricName = 'itop_mymetric_';
    }

    /**
     * {@inheritDoc}
     */
    public function GetMetrics(): ?array
    {
        $aMetrics = [];
        foreach(['dev', 'staging', 'production'] as $sEnv){
            foreach (['kpi1' => 1, 'kpi2' => 2] as $sKey => $iCount) {
                $aMetrics[] = new MonitoringMetric($this->sMetricName."{$sKey}_count",
                    "my metric $sKey explanation.",
                    $iCount,
                    ['env' => $sEnv]
                );
            }
        }
        
        return $aMetrics;
    }

Then you can point at it in iTop monitoring configuration

config-itop.php /combodo-monitoring

'itop_mymetric' => 
    array (
      'description' => 'my metrics',
      'custom' => 
      array (
        'class' => '\\Combodo\\iTop\\YourExtension\\Monitoring\\MyMetricReader',
      ),
    ),

output combodo-monitoring

itop_mymetric_kpi1_count{env="dev"} 1
itop_mymetric_kpi2_count{env="dev"} 2
itop_mymetric_kpi1_count{env="staging"} 1
itop_mymetric_kpi2_count{env="staging"} 2
itop_mymetric_kpi1_count{env="production"} 1
itop_mymetric_kpi2_count{env="production"} 2