Sync Data from Other Sources

This article contains the following topics:

The Sync Utility specifies how to synchronize data on a regular basis from various sources such as Lightweight Directory Access Protocol (LDAP), Representational State Transfer (REST) APIs such as ServiceNow, Cherwell, Freshservice or any other ITSM tools to Luma. The utility is used to import information such as users, groups, group membership and global user attributes. This utility does not import or update passwords in Luma. The synchronization can be scheduled at a recurring interval that is aligns to your organization's needs. When a user is added or updated from a domain, the Sync Utility automatically synchronizes the information with Luma. 

Initial Setup

This section explains the tasks that you must perform while getting started with the Sync Utility.

Verify System Requirements for LDAP Supported System

Ensure that your system meets the following requirements before running the Sync Utility:

• LDAP v2 and v3 Supported Servers: Microsoft Active Directory (AD)
• Directory OS: Windows, UNIX
• Latest version of Java (version 8 or later)
• Memory: 2 GB minimum
• Compatibility with HTTPS secure communications protocol
• Luma can import contacts and groups from all systems that support LDAP standards
• Luma can import data to Global User attribute 

Verify System Requirements for REST API Supported System

Ensure that your system meets the following requirements before running the Sync Utility:

• REST end points with appropriate credentials to pull the data
• OS: Windows, UNIX
• Latest version of Java (version 8 or later)
• Memory: 2 GB minimum
• Compatibility with HTTPS secure communications protocol
• Luma can import contacts and groups from all systems that support REST standards
• Luma can import data to Global User attribute

Note: For Importing data to Global User attribute, the Global User attribute must be created prior to running the sync tool

Acquire and Unzip the Utility

The Sync Utility is delivered as a ZIP file. This file contains the utility and supporting files necessary for the process. You can acquire the Sync Utility package by reaching out to the Serviceaide Support through email or the support portal. Specify the subject as require access to the Luma Sync Utility. Once the email is sent or a support ticket is created, the Support team will help you with the request.

Considerations to unzip the utility are as follows:

• Login as an administrator.
• The file can be unzipped to any environment with JRE configured.
• All JARs and files must remain together, exactly as they were unzipped. Do not move anything from the unzipped layout.
• The installation location must be able to connect to your LDAP server, REST endpoints and Luma.

Once you have unzipped the utility, you are ready to configure the utility.

Deploy

The Sync Utility is deployed as a stand-alone application on the client side. You need the Java Runtime Environment (JRE) to run this application. Set up the Sync Utility.

Set up a service account that has permissions to connect to the directory server and run queries.

Configure Files

Sync Utility includes the synctool.properties file in the Install directory \conf folder.

This file contains your tenant information for active directory contact sync using LDAP connection as shown in the following sample:

# comment
import-data-system.source-system-name = ADFS
# this can be only supported connection types (REST/LDAP) etc
import-data-system.connection-type = LDAP
import-data-system.ldapConfig.admin-user = Administrator@domain.com
import-data-system.ldapConfig.password = xxxxxxxxxx
import-data-system.ldapConfig.global-catalog-server = ldap://xx.xxx.xxx.xxx:xxxx
import-data-system.ldapConfig.domain-controller-server = ldap://xx.xxx.xxx.xxx:xxx
import-data-system.ldapConfig.search-base = OU=USERS,DC=ACME,DC=inc
import-data-system.ldapConfig.search-filter = (&(objectClass=user)(objectClass=person)(objectClass=organizationalPerson))
import-data-system.ldapConfig.response-type = String
# comment
import-export-mapping.import-data-model-key-mapping-with-export-data-model-key = [{\	
	"importDataKey": "",\	
	"exportDataKey": "",\	
	"staticColumn": "true",\	
	"staticColumnName": "Id",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "false",\	
	"javascriptImportDataKeyReference": "Id",\	
	"javascriptXformationFunciton": ""\
},\
{\	
	"importDataKey": "title",\	
	"exportDataKey": "Title",\	
	"staticColumn": "false",\	
	"staticColumnName": "",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "false",\	
	"javascriptImportDataKeyReference": "title",\	
	"javascriptXformationFunciton": ""\
},\
{\	
	"importDataKey": "givenName",\	
	"exportDataKey": "First Name",\	
	"staticColumn": "false",\	
	"staticColumnName": "",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "false",\	
	"javascriptImportDataKeyReference": "firstName",\	
	"javascriptXformationFunciton": ""\
},\
{\	"importDataKey": "givenName",\	
	"exportDataKey": "Middle Name",\	
	"staticColumn": "false",\	
	"staticColumnName": "",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "false",\	
	"javascriptImportDataKeyReference": "given Name",\	
	"javascriptXformationFunciton": ""\
},\
{\	
	"importDataKey": "sn",\	
	"exportDataKey": "Last Name",\	
	"staticColumn": "false",\	
	"staticColumnName": "",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "false",\	
	"javascriptImportDataKeyReference": "lastName",\	
	"javascriptXformationFunciton": ""\
},\
{\	
	"importDataKey": "givenName",\	
	"exportDataKey": "User Name",\	
	"staticColumn": "false",\	
	"staticColumnName": "",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "true",\	
	"javascriptImportDataKeyReference": "userName",\	
	"javascriptXformationFunciton": "function (entity) { return 'usr_' + parseInt((Math.random() * 20), 10) + entity.firstName.replace(/\s+/, '') } "\
},\
{\	
	"importDataKey": "mail",\	
	"exportDataKey": "Email Address",\	
	"staticColumn": "false",\	
	"staticColumnName": "",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "false",\	
	"javascriptImportDataKeyReference": "eMail",\	
	"javascriptXformationFunciton": ""\
},\
{\	
	"importDataKey": "mobile",\	
	"exportDataKey": "Phone",\	
	"staticColumn": "false",\	
	"staticColumnName": "",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "false",\	
	"javascriptImportDataKeyReference": "phone",\	
	"javascriptXformationFunciton": ""\
},\{\	
	"importDataKey": "dummy",\	
	"exportDataKey": "Role",\	
	"staticColumn": "false",\	
	"staticColumnName": "Role",\	
	"staticColumnValue": "",\	
	"isTransformationNeeded": "true",\	
	"javascriptImportDataKeyReference": "",\	
	"javascriptXformationFunciton": "function(entity) { if (entity.lastName.indexOf('admin') > -1) {return 'Administrator'; } else if (entity.lastName.indexOf('analyst') > -1) {return 'Analyst';} else {return 'Self-service user';}}"\
},\
{\	
	"importDataKey": "",\	
	"exportDataKey": "",\	
	"staticColumn": "true",\	
	"staticColumnName": "Associated Groups",\	
	"staticColumnValue": "G1",\	
	"isTransformationNeeded": "true",\	
	"javascriptImportDataKeyReference": "",\	
	"javascriptXformationFunciton": ""\
}]
export-data-system.target-system-name = LUMA
export-data-system.target-export-format = Excel
export-data-system.exportFileLocation = e:/exportrunner/conf/Template_for_Bulk_Upload.xlsx
export-data-system.exportSheetName = Contacts
export-data-system.connection-type = LUMA
	
export-data-system.luma.base-url = https://xxx
export-data-system.luma.auth-api = /public/tenant/login
export-data-system.luma.export-api = /tenant/@replace.tenantId/contact/upload
export-data-system.luma.userName = admin@example.com
export-data-system.luma.password = xxxx
export-data-system.luma.subDomain = xxxx

Following are the name-value pairs available for configuration:

Following are details of the above script:

• The supported connection types are REST and LDAP.
• Specify the source details such as
  • Admin user's email ID 
  • Password
  • Global catalog server URL 
  • Domain controller server URL 
  • Search base
  • Search filter
  • Response type
• The next step is to map the source details with that of Luma. For example, to map the first name of the source data with Luma, specify the following.
  • "importDataKey": "givenName",\
     Specify the source column name which is equivalent to the First Name in Luma.                                
  • "exportDataKey": "First Name",\
    Specify the Luma column name which matches the column name to be mapped from the source.
  • "staticColumn": "false",\
    If there is a mandatory field in Luma while mapping, however, that column is not available in the source file, then you can specify a static value for the column and specify the staticColumn as true. If the field is available in the source file, then specify it as false.
  • "staticColumnName": "",\
    If there is a mandatory field in Luma while mapping, however, that column is not available in the source file, then you can specify the static column name as the standard field name available in Luma.
  • "staticColumnValue": "",\
    Specify the static column value as per the staticColumn.
  • "isTransformationNeeded": "false",\
    If you need transformation of any function specified in the javascriptxformation function, then specify the value as true, else specify it as false.
  • "javascriptImportDataKeyReference": "firstName",\
    Specify an alias name to the importDataKey.
  • "javascriptXformationFunciton": ""\
     You can execute a Javascript function to define the value of row for requirements such as string concatenation, truncation or to perform simple logic such as when to choose a user type as Administrator or Analyst and so on. 

Following is the sample contact sync property file for Freshservice using REST connection:

# comment
import-data-system.source-system-name = freshservice
# this can be only supported connection types (REST/HTTP/LDAP/SQL) etc
import-data-system.connection-type = REST
# comment
import-data-system.http-is-rest = true
# comment
import-data-system.url = https://sample.freshservice.com/api/v2/requesters
# comment
import-data-system.http-method = GET
# comment
import-data-system.auth-type = basic
# comment
import-data-system.auth-token = xxx
# comment
import-data-system.http-headers = [{"headerName":"Content-Type","headerValue":"application/json"}]
# comment
import-data-system.http-url-parameters = [{"parameterName":"Content-Type","parameterValue":"application/json"}]
# comment
import-data-system.http-response-type = JSON# commentimport-data-system.http-response-root-json-path = $['requesters'][*]

Schedule

You can schedule the synchronization process to run as required. Ideally, run the synchronization process once every day. Schedule the process during your off-business hours. Windows or Linux scheduler can be used to schedule the sync utility as a background job.

Sync Utility will not update if the user accounts are inactive, expired or deleted in Luma.  

How the Sync Utility Works for LDAP

This topic describes how the synchronization works.

On Directory Server (Client Side)

• The Sync Utility uses a service account to connect to the global catalog server and domain controllers.
• A Search Base list is used to look for users from the list of Organization Units (OUs).
• The application cycles through each OU and looks for users in accordance with the search criteria. This criteria is defined as follows:
  (&(objectClass=user)(objectClass=person)(objectClass=organizationalPerson))
• The fields in the Directory Server are mapped with the fields in Luma.
• For each matching entry, the application looks for certain attributes. Examples of such attributes are first-name, last-name, and email. The complete list of attributes can be found in the synctool.properties file. 
• The application writes the matching entries to a .xls file in the Sync Utility after cycling through all the OUs. 
• The server response contains the errors and warnings that are encountered while processing the data. The response is generated in XML format and written in the INFO tag of the log file. The log file is at <:Install_Directory>/sync/logs/sync_data.log. You can set up file monitors on this log file. As part of your monitoring service, look for errors and warnings.

On Luma (Server Side) 

• On receiving the data from the LDAP, Luma starts immediate processing of the data.
• Based on the user ID in the incoming feed, the corresponding contact record is identified in Luma. 
• The incoming entry is compared with the data existing in Luma. An automatic update statement is created based on the difference between the two. This condition applies to changes in attributes such as phone, email, and title.

• Each new record is created as configured in the mapping directly to a specific role or use function as shown below (for example, if the function contains Analyst in the lastName of the user, it is mapped to the Analyst role and so on. If nothing is specified, then it considers the Self Service User role). These users have their primary group and license set to Self Service role.

  Java script function

  "javascriptXformationFunction": "function(entity) { if (entity.lastName.indexOf('admin') > -1) 
  {return 'Administrator'; } else if (entity.lastName.indexOf('analyst') > -1) 
  {return 'Analyst';} else {return 'Self-service user';}}"\

• The exceptions that are encountered during the synchronization are written to sync_data.log file. The log includes information about errors that are encountered while processing data as well as details about the contacts processed.