Ticket Creation from eMails

This page shows the version 2.2 of the 'Ticket Creation from eMails' module. This version supports connecting to either a POP3 or an IMAP server. It uses the PHP imap extension for connecting to IMAP server and falls back to the PEAR POP3 implementation for connecting to POP3 servers.

This page describes an extension to iTop 1.2.1 or 2.0 to automate the creation or update of User Request tickets (or Incident tickets) from incoming eMails.

The modules listed in the page, turn incoming emails from a POP3 or IMAP mailbox into User Request tickets.

• The subject of the email becomes the title of the ticket.
• The body of the email becomes the description of the ticket (a plain text version of the body is produced if the body is in HTML)
• A contact with the same email address as the sender of the email MUST ALREADY exist in iTop.
• The organization of the ticket is the organization of the caller/email of the sender of the email.
• Unless otherwise specified all attachments and inline-images are uploaded as attachments in iTop.
• In case of an update, already existing attachments are not duplicated and attached again to the ticket.

The incoming emails are kept in the mailbox until the corresponding ticket in iTop is either closed or deleted. Then, the email is deleted from the mailbox.

If the incoming email is a reply to a notification about a given ticket or if the subject of the email matches a certain (configurable) pattern, then the ticket referenced by the email is updated instead of creating a new ticket. The update consists in adding the “new part” of the email message to the ticket's public log. New attachments are added to the ticket, if any.

A special trigger Trigger (when updated by mail) is defined for when a ticket is updated from an email, so that an administrator can configure notifications for this event.

• Only one mailbox is scanned
• Creates tickets of only one (configurable) type. Cannot create User Requests for some incoming emails and Incidents for others.

• iTop 1.2.1 or 2.0 (it should work with iTop 1.2 as well, but it's not guaranteed)
• PHP 5.2.1+ with the IMAP extension enabled if you want to connect to an IMAP server, or PEAR::NetSocket (iTop comes with its own copy of PEAR::POP3) if you want to connect to a POP3 server.
• A connection to a POP3 or IMAP server with a valid mailbox

1. Download the package: EmailSynchro-v2.2.zip and expand the two folders it contains “combodo-email-synchro” and “itop-simple-email-synchro” into the “extensions” directory of iTop.
2. For Connecting to an IMAP server, install the PHP IMAP extension or for a POP3 server install Pear::NetSocket as follows:
   1. On Windows: locate the directory where php is installed. Open command prompt AS AN ADMINISTRATOR and from the this directory launch 'go-pear.bat'. Accept the default answer to all questions.
   2. Then from this command prompt, type “pear install Net_Socket”
   3. On Linux Ubuntu, as root, launch the following command: apt-get install php-net-socket
   4. On Linux RedHat/Fedora/CentOS, as root, launch the following command: yum install php-pear-Net-Socket
3. If you have already installed iTop make sure that the configuration file config-itop in conf/production is NOT read-only.
4. Point your web browser to http(s)://<your_itop_root>/setup and follow the wizard. Make sure that you select the option to “Upgrade an existing iTop instance”:

Finally check the module “Ticket Creation from Emails” in the list of extensions at the end of the interactive wizard. Then complete the installation.

This page describes an extension to iTop, so it assumes that you have already downloaded iTop (and maybe already installed it too)

1. Download the package: EmailSynchro-v2.zip and expand the two folders it contains “combodo-email-synchro” and “itop-simple-email-synchro” into the “modules” directory of iTop.
2. For connecting to an IMAP server, install the PHP IMAP extension or for a POP3 server install Pear::NetSocket as follows:
   1. On Windows: locate the directory where php is installed. Open command prompt AS AN ADMINISTRATOR and from the this directory launch 'go-pear.bat'. Accept the default answer to all questions.
   2. Then from this command prompt, type “pear install Net_Socket”
   3. On Linux Ubuntu, as root, launch the following command: apt-get install php-net-socket
   4. On Linux RedHat/Fedora/CentOS, as root, launch the following command: yum install php-pear-Net-Socket
3. If you have already installed iTop, then make a backup of the iTop configuration file (“config-itop.php” at the root of the iTop installation) and make sure that the configuration file is NOT read-only.
4. Point your web browser to http(s)://<your_itop_root>/setup and follow the wizard. Make sure that you check the module “Ticket Creation from Emails”. then complete the installation.

Once the new module has been installed, edit the configuration file config-itop.php and look for the two new sections:

        'combodo-email-synchro' => array (
                'debug' => false,
                'notify_errors_to' => '',
                'notify_errors_from' => '',
                'introductory-patterns' => array (
                  0 => '/^le .+ a écrit :$/i',
                  1 => '/^on .+ wrote:$/i',
                  2 => '|^[0-9]{4}/[0-9]{1,2}/[0-9]{1,2} .+:$|',
                ),
                'periodicity' => 30,
        ),
 
 
        'itop-simple-email-synchro' => array (
                'mail_server' => 'imap.demo.com',
                'protocol' => 'imap',
                'port' => 143,
                'login' => 'mailbox-name',
                'pwd' => 'your-password',
                'mailbox' => '',
                'pop3_auth_option' => 'USER',
                'imap_options' => array (
                  0 => 'imap',
                ),
                'debug' => true,
                'exclude_attachment_types' => array (
                  0 => 'application/exe',
                ),
                'default_values' => array (
                ),
                'parts_order' => 'text/plain,text/html',
                'ticket_class' => 'UserRequest',
                'create_only' => false,
                'title_pattern' => '/R-([0-9]+)/',
        ),

You can re-arrange it to make it look a bit easier to understand:

        'combodo-email-synchro' => array (
                'debug' => false,
                'notify_errors_to' => '',
                'notify_errors_from' => '',
                'introductory-patterns' => array ('/^le .+ a écrit :$/i', '/^on .+ wrote:$/i', '|^[0-9]{4}/[0-9 {1,2}/[0-9]{1,2} .+:$|'),
        ),
 
        'itop-simple-email-synchro' => array (
                'protocol' => 'imap',
                'mail_server' => 'imap.demo.com',
                'port' => 143,
                'login' => 'mailbox-name',
                'pwd' => 'your-password',
                'mailbox' => '',
                'pop3_auth_option' => 'USER',
                'imap_options' => array ('imap'),
                'debug' => false,
                'exclude_attachment_types' => array ('application/exe'),
                'default_values' => array (),
                'parts_order' => 'text/plain,text/html',
                'ticket_class' => 'UserRequest',
                'create_only' => false,
                'title_pattern' => '/R-([0-9]+)/',
        ),

First of all, make sure the both entries for 'debug' are set to true, so that you can see what's happening.

Then provide some valid values at least for 'mail_server', 'login', pwd' and 'port'. If your server is an POP3 server, change the 'protocol' entry from 'imap' to 'pop3' and adjust the port number from 143 to 110 (or whatever makes sense for connecting to your mail server).

To test your settings:

1. Make sure there is at least one email in the mailbox you're going to poll
2. The source address from this email must be the email address of an existing contact in iTop
3. Open a command line prompt on the server and from the directory <itop_root>/webservices execute the command:

php cron.php --auth_user=<user> --auth_pwd=<pwd>

Windows users, run:

php.exe cron.php --auth_user=<user> --auth_pwd=<pwd>

Where <user> and <pwd> are valid credentials for an iTop account (for example an administrator account).

You should see something like this:

Processing Message Source: imap.demo.com GetMessagesCount returned: 1
Searching EmailReplicas: 'SELECT EmailReplica WHERE uidl IN (1)'
Dispatching new message: 1
iTop Simple Email Synchro: dispatch of the message 0 (1)
The message is new (unread).
iTop Simple Email Synchro: dispatch of the message 0 (1) returned 2 (PROCESS_MESSAGE)
Processing new message: 1
IMAPEmailSource: Fetching the message 0
iTop Simple Email Synchro: Processing message 0 (1)
Email body format: text/plain
ERROR:Object not following integrity rules: issues = Unexpected value for attribute 'service_id': Null not allowed,
Unexpected value for attribute 'servicesubcategory_id': Null not allowed, Unexpected value for attribute 'workgroup_id': Null not allowed, class = UserRequest, id = 
Exiting: 1343645038

At this point, the ticket is NOT created but this shows that iTop is able to connect to the given mailbox and read messages from it. You're almost done.

If you see an output like:

Fatal error: Uncaught exception 'ConfigException' with message 'Could not read configuration file (the file exists but cannot be read).
Do you have the rights to access this file?: file = /opt/test/email-synchro//config-itop.php' in /opt/test/email-synchro/core/config.class.inc.php on line 652

This means that the current user does not have the rights to read the iTop configuration file. Either adjust the rights on this file or launch the 'php' command from another account (or sudo it).

The ticket creation script populates the new ticket with the following information:

1. The 'caller' of the ticket is the contact with the same email address as the sender of the email.
2. The 'org_id' of the ticket is filled with the organization of the caller.
3. The 'title' of the ticket is filled with the 'Subject' of the email (or with 'No subject' if the subject is empty)
4. The 'description' of the ticket is filled with the 'text body' of the email (or with 'No description.' if the body is empty)
5. Email attachments are turned into attachments to the ticket.

In order to create a valid iTop Ticket, each other mandatory field must be filled with some valid default value. This is the purpose of the 'default_values' array in the configuration file.

For example, using the the standard iTop 1.2.1 data model (and the Demo data) the following configuration is enough to create tickets:

        'itop-simple-email-synchro' => array (
                'mail_server' => 'pop3.combodo.com',
                'port' => 110,
                'login' => 'test@combodo.com',
                'pwd' => 'your_pwd',
                'mailbox' => '',
                'imap_options' => array ('pop3'),
                'debug' => true,
                'exclude_attachment_types' => array ('application/exe'),
                'default_values' => array (
                        'service_id' => 'Computers and peripherals',
                        'servicesubcategory_id' => 'Repair',
                        'workgroup_id' => 'Hardware support',
                ),
                'parts_order' => 'text/plain,text/html',
                'ticket_class' => 'UserRequest',
                'create_only' => false,
        ),

From the command line, go to the directory 'webservices' and launch the following command:

php cron.php --auth_user=<admin_user> --auth_pwd=<admin_pwd> --verbose=1

The output should look like the following:

Starting: 1317719369
Planned duration = 600 seconds
Processing asynchronous task: ExecAsyncTask
Returned: processed 0 tasks
Processing asynchronous task: ProcessSLAResponseTicket
Returned: No ticket to process
Processing asynchronous task: EmailBackgroundProcess
Processing Message Source: test@combodo.com GetMessagesCount returned: 0
Returned: Message(s) read: 0, message(s) processed: 0, message(s) deleted: 0
Sleeping
...

By default the script will run to 10 minutes, but you can press CTRL-C to stop the script at any time.

If you see a message like

Failed to initialize the mailbox: <your_login>, the mailbox will not be polled. Reason (Cannot connect to <your_pop3_server> on port <your_port>)

Then check the credentials and connection to the mail server.

If you're trying to connect to the server through a SLL connection, add 'ssl' in the list of imap_options (and adjust the 'port' as well if needed). Then if you see the following error:

Warning: imap_open(): Couldn't open stream {imap.demo.com:993/imap/ssl} in /opt/test/email-synchro/modules/combodo-email-synchro/model.email-synchro.php on line 774

Call Stack:
    0.0007     376268   1. {main}() /opt/test/email-synchro/webservices/cron.php:0
    0.7834   33215716   2. CronExec() /opt/test/email-synchro/webservices/cron.php:206
    0.8142   33250056   3. EmailBackgroundProcess->Process() /opt/test/email-synchro/webservices/cron.php:86
    0.8142   33250772   4. iTopSimpleEmailProcessor->ListEmailSources() /opt/test/email-synchro/modules/combodo-email-synchro/model.email-synchro.php:1055
    0.8143   33251516   5. IMAPEmailSource->__construct() /opt/test/email-synchro/modules/itop-simple-email-synchro/model.itop-simple-email-synchro.php:71
    0.8143   33251976   6. imap_open() /opt/test/email-synchro/modules/combodo-email-synchro/model.email-synchro.php:774

Array
(
    [0] => Certificate failure for imap.demo.com: Server name does not match certificate: /OU=Domain Control Validated/CN=ssl0.mycompany.net
)

This means that the server is not sending back a certificate corresponding to its DNS name. You can bypass this verification by adding the 'novalidate-cert' option to the list of imap_options. An example of such a configuration is shown below:

        'itop-simple-email-synchro' => array (
                'mail_server' => 'pop3.combodo.com',
                'port' => 993,
                'login' => 'test@combodo.com',
                'pwd' => 'combodo',
                'mailbox' => '',
                'imap_options' => array ('imap', 'ssl', 'novalidate-cert'),
                'debug' => true,
                'exclude_attachment_types' => array ('application/exe'),
                'default_values' => array (
                        'service_id' => 'Computers and peripherals',
                        'servicesubcategory_id' => 'Repair',
                        'workgroup_id' => 'Hardware support',
                ),
                'parts_order' => 'text/plain,text/html',
        ),

In order to create a User Request ticket in iTop 1.2.1 the following fields are mandatory

• Organization (org_id)
• Caller (caller_id)
• Title (title)
• Description (description)
• Service (service_id)
• Service element (servicesubcategory_id)
• Workgroup (workgroup_id)

Organization, Caller, Title and Description are retrieved from the email and automatically filled with the corresponding value. However Service, Service element and Workgoup, cannot be retrieved from a standard email. This is why a default value must be provided for each of these fields. Note that for the ticket to be valid, the values for this 3 fields must be VALID VALUES for ORGANIZATION of the CALLER.

The 'default_values' settings can also be used to pre-fill any other field of the ticket.

The creation of ticket is integrated with iTop's generic background process 'cron.php'.

For information about how to schedule the background processing in iTop, refer to Background tasks

If your company uses some email templates for specific requests or processes, you may want to enhance the iTop to process such pre-formatted emails.

You may also want to enhance the mechanism in order to process emails from several inboxes at once.

All the processing that creates the tickets in iTop is located in the function ProcessMessage in the file model.itop-simple-email-synchro.php. Feel free to adapt the code inside this function to suit your own needs.

The module combodo-email-synchro provides the generic mechanism for processing an arbitrary number of email sources, listing the messages, decoding them… and should probably not be touched by your customization.

The configuration of the V2 extension is slightly different from the configuration of the V1 extension, so the new version is not a 'drop-in' replacement for the version 1.

In order to upgrade:

1. Replace the two directories modules/combodo-email-synchro and modules/itop-simple-email-synchro by their new version
2. Make a backup copy of of your configuration files and make it writeable by the web server
3. Point your browser to http(s)://<your_itop_server>/setup - Make sure that you check the module 'Ticket Creation from Emails' if it's not checked already
4. Once the installation completes edit the configuration file as explained above.

Note: the internal data used to maintain the linkage between the incoming emails and the iTop tickets remain the same, so the upgrade should be transparent to the iTop users.