Sidebar

Combodo

iTop Extensions

Enhanced global search

name:
Enhanced global search
description:
Use a faster and more relevant global search with easy-to-filter results
version:
2.0.0
release:
2024-07-17
itop-version-min:
3.1.2
code:
combodo-fulltext-search
state:
stable
php-version-max:
PHP 8.3

This extension replace the standard global search with a faster and more accurate search.

For iTop versions older than 2.7.0 use 1.0.11
This extension requires Index FullText, so MySQL 5.6+ or Mariadb 10.0.5+

Features

  • Users can improve results accuracy by specifying required, optional and denied words.
  • Administrator can fine-tune the results relevance, by changing configuration parameters.
  • It uses MySQL fulltext feature to perform the global search.

Revision History

Release Date Version Comments
2024-07-17 2.0.0 * Updated untranslated tildes entries so it matches the en lang
* Add included_classes list in configuration
* index only String, Text, LongText, and case logs fields
* Uses Events to index object
* N°7579 - Enhanced global search : improve the indexing process
* Add ResourceGlobalSearchManagement class for access rights controls
* N°5821 JenkinsFile : introduce buildDiscarder
* N°5938 - Use iTop Theme for CSS for Enhanced global search
2022-11-10 1.2.8 * N°5400 - Reindex objects modified by SQL directly
* N°5492 - PHP 8 and PHP 8.1 : Add compatibility
2022-09-07 1.2.7 * N°5503 - Fix indexation (wrong type checking)
* N°5492 - PHP 8 : Add compatibility
2022-04-08 1.2.6 * N°5041 - Fix indexation for object creation
2022-04-06 1.2.5 (BROKEN) * N°5041 - Fix unnecessary re-index when modifying n-n links
2022-01-03 1.2.4 * Fix search needles signs (+, -, “”)
* Fix class names handling
* Fix CSS on field values (and organization name)
2021-12-21 1.2.3 * Modification of past search needles
* Display results in standard iTop table for a single class
* Classes display improvement
2021-06-02 1.2.1 * Compatibility iTop 2.7 and 3.0
* Fix CSS
2021-05-25 1.2.0 * Compatibility iTop 2.7 and 3.0
2021-05-25 1.1.4 * Prevent indexation concurrency
* Fix link for class drill down
* fix script refresh-search-index.cli.php to rebuild fully the index
* fix script populate-search-index.cli.php to populate the index
2021-05-04 1.1.3 - broken
2020-08-25 1.1.1 - Admin console
- populate script more robust (memory)
- Default configuration moved to XML
- Avoid reading blobs from database
- Exclude classes by configuration
2020-06-23 1.0.13 - for iTop 2.7.0 and above
- Fix searches with some reserved characters
- Search admin menu
2019-02-04 1.0.11 - Fix broken related link
2019-01-09 1.0.10 - Fix the friendly name limit
- Fix Scheduled task
- Fix class name in tests
- Add Combodo license
2019-01-09 1.0.9 - Fix memory limit
- Fix Case log indexation
- Fix email search
- Limit the friendlyname size to fit in the DB
2018-07-31 1.0.8 - Better indexation
- Better search
- Support only InnoDB and BOOLEAN mode
- Display related objects on search
- Display matching fields
- Admin page : for re-indexation
- Class drill-down
- Debug mode
2018-01-19 0.0.3 - Better indexation when updating data
- 'populate_search.php' uses the configuration for the type of table to create (MyISAM or InnoDB)
- No constraint on MySQL (MyISAM must be used in version before 5.6)
2018-01-09 0.0.2 - Better error management.
- No indexation at setup (performance issues).
- 'populate_search.php' have to be used to create an InnoDB fulltext index.
- MySQL >= 5.6
2017-12-14 0.0.1 - First version experimental.

Limitations

  • This extension relies on MySQL full-text feature.
  • It uses the main database configured for iTop.
  • It doesn't search in archived object (N°4135)

Requirements

  • MySQL 5.6 or above
  • Mariadb 10.0.5 or above
  • iTop 2.5.0 or above

Installation

Use the Standard installation process for this extension.

If the database is big, the index creation can run for a long time. It is possible to populate the full-text index by calling the script:

/env-production/combodo-fulltext-search/script/populate-search-index.cli.php

After a first install or an upgrade of the extension from an incompatible one (for example upgrading from 1.2.8 or before to 2.0.0 or after) the search index will be erased. You'll have to re-synchronize the entries in the global search management screen or populate the index using a script.

Configuration

Parameter Type Description Default value
object_weight_factor array Weight factor to apply for a given object name (format: 'classname' ⇒ 'value') the values can vary from 0 to 10 generally (0 means no results for this class). The default is 1 for every class not specified. The modification of this parameter needs a complete re-indexation of the database.
array(
  'Organization' = 20,
  'Contact' = 15,
  'Location' = 1.2,
  'Ticket' = 1.0,
  'FunctionalCI' = 0.8,
)
sentence_weight_factor float Weight factor for searches with multiple words matching exactly the list of words in that order. 1000.0
mandatory_weight_factor float Weight factor for searches with all the words of the search matching at least once. 50.0
start_with_weight_factor float Weight factor for searches with one word matching the beginning. 0.5
max_interactive_index_update_time_in_sec integer Time in seconds allowed for direct indexation update on change. 5
background_index_refresh_period_in_min integer Period in minutes for background indexation. 1
background_max_indexation_time_in_min integer Execution time limit in minutes for background indexation. 0.8
excluded_classes array List of classes excluded from search. It has to be the finalclass (i.e CustomerContract or ProviderContract but not Contract []
included_classes array List of classes to search when not empty []
configuration example 1
'combodo-fulltext-search' => array(
    'object_weight_factor' => array(
        'Organization' => 2.0,
        'Person' => 1.5,
        'Location' => 1.2,
        'SLT' => 0.8,
    ),
    'excluded_classes' => array(
       'Attachment',
    ),
),
configuration example 2
'combodo-fulltext-search' => array (
   'object_weight_factor' => array (
      'Organization' => 2.0,
      'Person' => 5.0,
      'Location' => 1.2,
      'SLT' => 0.8,
   ),
   'sentence_weight_factor' => 10.0,
   'mandatory_weight_factor' => 2.0,
   'start_with_weight_factor' => 0.5,
 
   // Max delay to re-index in realtime, 
   // above this it flags index entries as out of date,
   // they will be computed by the background task
   'max_interactive_index_update_time_in_sec' => 5,
   // Background task periodicity, should be more than iTop cron periodicity 
   'background_index_refresh_period_in_min' => 5,
   // Background task max time per run
   'background_max_indexation_time_in_min' => 1,  
 ),
  • All the strings from an object will be indexed excepted:
    • External keys
    • External fields
    • Words less than 3 characters
  • The friendly name of an object is indexed even if it depends on external fields
  • IP Addresses are indexed and can be searched completely or partially (starts with)
    • Example: 10.10.0.255 will be indexed and can be searched using 10.10.0.255 or 10.10.0.*
  • Email addresses are indexed:
    1. Fully (e.g. pablo.picasso@combodo.com)
    2. By name (e.g. pablo.picasso)
    3. By domain (e.g. combodo.com)
  • HTML fields will be converted to text before indexing (the HTML tags and attributes won't be indexed)

Background behavior

Create, update or delete object

When ever a user create, update or delete an object in iTop, the extension will recompute the fulltext search indexes. If it cannot do it within max_interactive_index_update_time_in_sec second, it flags out-of-date indexes and they will be processed by the itop-cron using a background task.

You may see that warning message when you change an organization name for example, as it is present in most iTop objects, so a lot of indexes will have to be updated, more than what can be processed in a user acceptable delay

Searching

Searching for one word When searching for a single word (example: demo) the following searches are done:

  1. The word: demo
  2. The words starting with the search: demo*

Searching for multiple words

When searching for multiple words, (example: user-level lock) the following searches are done:

  1. The exact sentence entered: “user_level lock”
  2. All the words are present: +user_level +lock
  3. At least one word is present: user_level lock

As - is a keyword to exclude a word, it is replaced by _ which is another keyword to represent any single character.

Computing weight

Each search is bringing points calculated by MySQL fulltext search, which are then weighted with the configuration parameters set.

Raw weight is provided by MySQL. Parameters are just a multiplier factor of the MySQL weight. You cannot influence the raw weight within iTop.

What values to set

To ensure that the search will be efficient within your iTop, there are additional actions for Administrator users:

  • Global search management action which allow to rebuild the indexes after a change of configuration parameter: object_weight_factor.
  • Debug action which display details on how the result ordering is generated, based on weight.
 Enhanced Global Search debug  Enhanced Global Search expand search
  • user_level lock has a MySQL provided weight
  • “user_level lock” has the above weight * sentence_weight_factor
  • Total Score is the sum of the various scores * object_weight_factor
Get from your users “sampling searches” on your real data, with the object expected to be in the top 5 results. Then change the various parameters, checking the result using the Debug action on those “sampling searches”.

Global Search Management

There is an administrator menu, which allow to

  • see on-going status of re-indexation processing
  • run re-indexation task
    • All the objects not indexed will be added to the search index
    • All the removed objects will be removed from the search index
    • All the existing entries will be re-indexed
    • The re-indexation process will be done in background task (launched by the CRON system)

 Enhanced Global Search refresh search index

During the indexation process, the current class being indexed is displayed on the screen.

Scripts

Another alternative to using this administrative page within iTop, is to run in CLI the below scripts located in:

<itop_path>/env-production/combodo-fulltext-search/script

populate-search-index.cli.php --verbose=0/1 --step-limit=<numer of second>

To be used once only, the day you install that extension
  1. It puts your iTop under maintenance
  2. Recreate the index table in database
  3. Reopen access to your iTop (also global search will not respond)
  4. Rebuild the fulltext index table
Ensure that MySQL/MariaDB variable wait_timeout is around 28 800 before running populate-search-index.cli.php !

To prevent situation where the script and the cron would both try to recompute indexes at the same time, we have used a MySQL lock feature available (not Galera). If your database does not have the Lock support, you'll have to stop the CRON system during the full indexation.

Some distribution, overwrite the default MySQL/MariaDB value for the wait_timeout parameter and put 600 (seconds). That delay is not enough to populate indexes for a big database, as a result, the script will return “MySQL has gone away” which is the answer to the unlock request done by the script when it finishes. In reality, the work was done, indexes have been recomputed, the error is misleading, it's just the lock which could not be released, because after wait_timeout seconds of inactivity on that session, MySQL/MariaDB has decided it was useless to wait more, and released the lock.
⇒ To get rid of that error, just restore default value for wait_timeout (28 800).

refresh-search-index.cli.php --verbose=0/1 --step-limit=<numer of second>

  • It does not put iTop under maintenance
  • It takes every existing index, and re-index them one by one
  • Missing index are created. So the script is adequate for an existing database on which you just installed the extension.
  • It is useful to remove entries corresponding to recently archived objects
If you are archiving objects through a system cron command line, then just after that run the refresh-search-index, to remove from Global Search results related to archived objects
Depending on the size of the objet to index populating or refreshing your full text index table may take time. It varies from 50 to 300 indexes per seconds

Usage

Just type the words in the global search field, then enter or click on the magnifier icon to get the result :

 Enhanced Global Search overview user

Add + at the beginning of required word
UserRequest:+PHP error 

will search for User Requests which contain mandatorily 'PHP' and maybe 'error'

Add - at the beginning of excluded word
UserRequest:PHP -class 

will search for User Requests which contain 'PHP' but not 'class'

Add * at the end of an incomplete word
Organization:Combo* Grenoble

will search for Organization which contain a word starting with 'Combo' and the exact word 'Grenoble'. This is useless in single word search, as it does it automatically.

Restricting the search to a class

By default the search searchs on all Classes with category searchable, including abstract classes, such as Ticket, Contact or FunctionalCI.

Restrict previous search to a single class Expend back to all classes
 Enhanced Global Search restrict search  Enhanced Global Search class search
 Enahnced Global Search expand search
Add <class-name>: at the beginning of a search pattern to limit the search to objects of this classe or one of its descendants
FunctionalCI:combodo

will search for any FunctionalCI which contains “combodo”, so the returned objects can be Server, PC, Enclosure,…

The class name can be the internal name (like FunctionalCI) of the display name in the current language used (like CI fonctionnel).

Displaying matching words

A menu allow to display some of the fields which have matched the requested pattern

 Enhanced Global Search matching fields

On each returned object, you can request related objects:  Enhanced Global Search related objects

Q&A

  1. How to give access to the Global Search Management screen ?
    • You can give access in Write mode to the resource : ResourceGlobalSearchManagement to give access to the menu and management screen
extensions/combodo-fulltext-search.txt · Last modified: 2024/09/02 15:42 by 127.0.0.1
Back to top
Contact us