Syncing Artifacts with SharePoint

This article describes how you can sync the existing Knowledge Documents in your SharePoint site with Luma Knowledge.

In Luma Knowledge, we can create a Federated Service with SharePoint to make the Knowledge available to the end-user through the Luma Knowledge (Knowledge Search Widget) or any Luma-Supported Channel. The Data Sync Service creates the artifacts in Luma Knowledge based on the documents available in your SharePoint site. With recurring sync, the artifacts are automatically updated and created based on the updates on the SharePoint portal.

To configure the sync with SharePoint, you should follow the below steps:

Step 1: Create an Application to connect to your SharePoint site

The first step is to create an App principal with access to your SharePoint portal. The App enables Luma Knowledge to connect to SharePoint and download documents for artifact creation and updates.

To create the application, follow the below steps:

1. Navigate to your SharePoint site. For example, https://xxxxxxxxx.sharepoint.com/sites/Documents.

2. Open the App registration page. Append /_layouts/15/appregnew.aspx to the site address. The URL should look like:
   https://xxxxxxxxx.sharepoint.com/sites/Documents/_layouts/15/appregnew.aspx

3. The App registration page is now open.

4. Following details are required to create the App:

   • Client ID: This is the SharePoint App ID. This is autogenerated by the system. Click Generate.

   • Client Secret: This is the Password for the App to connect to SharePoint.  This is autogenerated by the system. Click Generate.

   • Title: Provide a user-friendly display name for the App.

   • App Domain: This is the Remote server host of the App. Use www.localhost.com if you don't have one.

   • Redirect URL: This is the Remote application endpoint. Use https://www.localhost.com if you don't have one.

     Open

     

5. Click Create to create the App.

Your app is now successfully created. Copy and retain the Client Id and Client Secret in a notepad or text editor. The details are required to set up Sharepoint Data Source.

Open

The next step is to provide permission to the app to your Sharepoint site. To do so, follow the below step:

1. To set permissions for the app, append _layouts/15/appinv.aspx to the site address. The URL should look like:
   https://xxxxxxxxx.sharepoint.com/sites/Documents/_layouts/15/appinv.aspx

2. This will open a new page.

3. Paste the Client Id (retained in the earlier step) in the App Id field and click Lookup.

4. This fetches and displays the details of the App Id created earlier.

5. Provide the Permission Request XML specifying what access the App has. The XML structure is below.

   <AppPermissionRequests AllowAppOnlyPolicy="true"> <AppPermissionRequest Scope="http://sharepoint/content/sitecollection" Right="FullControl" /> </AppPermissionRequests>

   You may replace the scope with the following values. These should be used as is, do not modify.

   • TENANT = http://sharepoint/content/tenant

   • SITE COLLECTION = http://sharepoint/content/sitecollection

   • SUB SITE = http://sharepoint/content/sitecollection/web

   • LIST/LIBRARY = http://sharepoint/content/sitecollection/web/list

   'Right' may also be populated with the following permissions. These should be used as is, do not modify.

   • Read = only read access

   • Write = add/edit/delete

   • FullControl = full permissions
     Note: Based on the Scope selected, the App may require approval from the Azure Global Administrator.

6. Click Create.

7. On the confirmation screen, click on Trust It.

9. You App is ready to be used.

10. Now Navigate to the Site Settings → Site App Permissions page to fetch the App identifier for your App. Append ‘appprincipals.aspx?Scope=Web’ to the site address. The URL should look like:
    https://xxxxxxxxxx.sharepoint.com/_layouts/15/appprincipals.aspx?Scope=Web

11. Copy the App Identifier and retain it in a notepad or text editor. The identifier is used in Data source configuration in the next step.

Step 2: Configure Source System- Data Source for Sharepoint

Now that the App is ready, the next step is configuring Sharepoint in Luma Knowledge. Here, we set up the connection type and connection details to the Sharepoint site you want to connect to.

To do so, follow the below steps:

1. On Luma Knowledge, navigate to Federated Service.

2. On the Manage Data Sources tab, click on Add Data Source.

    

3. Add new Data source details and click on Save.

4. Once the Data Source is created, click on the new data source to configure the Authorization Profile.

5. On Configure Authorization Profile page, add the following details:

   1. Select SHAREPOINT as Connection Type.

   2. On the SharePoint Details section, Add your SharePoint Domain.

   3. Click on Get Tenant Id From SharePoint App. This function automatically populates the Identified Sharepoint Tenant Id and Resolved Resource Client Id from SharePoint.

   4. Now configure your Sharepoint App details in Step 1.

      1. Add the App Identifier in Client ID.

      2. Add Client Secret.

   5. Once the details are added, click on Save.

6. The Sharepoint data source is now ready.

Step 3: Configure Destination system - Data Source for your Luma Knowledge Tenant

Once the Sharepoint Data Source is ready, the next step is to configure the Data source for your Luma Knowledge instance.

To do so, follow the below steps:

1. On the Manage Data Source tab, select LUMA KMS Current Host Data Source.

2. On Configure Authorization Profile page, click Edit.

3. Scroll down to the Separate HTTP Authentication Call section.

4. Update the URL to connect to your Luma Knowledge environment. e.g. https://lumaproedge.serviceaide.com/gateway/api/auth/login.

5. Set HTTP Method to POST.

6. In Request Headers tab, add the following Keys:
   Name : Content-Type Value : application/json Data Type: STRING
   Name : ClientType Value : Ajax Data Type: STRING
   Name : x-requested-with Value : XMLHttpRequest Data Type: STRING
   Name : accept Value : application/json Data Type: STRING

7. Provide your Tenant authentication details on the Body tab.

   1. Click on Add to add the new key value.

   2. Add the key NAME, e.g. JSON_BODY.

   3. In Value, provide the tenant and authentication details in the below JSON format.
      {"username":"<Login username>","password":"<password>","subdomain":"<tenant name>"}
      e.g. {"username":"Test","password":"XXXXXXX","subdomain":"Test"}

   4. Select JSON as Data Type.

8. Set Response Type to JSON.

9. Under Extract Response Using, set JSON path is $.token

10. Click on Save and update the configuration.

Note: Since Luma Knowledge (as Data source) is available out-of-the-box, a few of the above details are available by default. Please review the data and update as required.

Step 4: Configure the Object and Data Profile to Export data from Source System- Sharepoint

Once the Data Source is ready, the next step is to create Data Object. This represents the data or entities to be imported from SharePoint. This is the actual end-point from where the document should be downloaded.

To configure Data Object, follow the below steps:

1. On the Manage Data Objects tab, select the newly created Data Source in the Source Name field.

2. Click on Add Data Object

3. Add the following details:

   1. Provide Data Object Name.

   2. Select SHAREPOINT as Connection Type.

   3. Select the Sharepoint Data Source Name.

   4. Click Save.

4. Now select the new Data Object from the list and Configure Data Profile. Here we provide details on the data/documents to be imported from the source. Follow the below steps to configure the Data Profile.

   1. On the Configure Data Profile page, select the Connection Type as SHAREPOINT.

   2. Under the SharePoint Details section, add the following details:

      1. Add Data Object Name.

      2. Provide your Sharepoint Site name as Site Name.

      3. Next, add the Folders to be synced. Click on Add to configure one or more folders.

   3. Once the configuration is done, click on Save to complete the profile.

5. Once the Data Profile is ready, the next step is to configure Object Fields. These fields represent the information to be imported from the source SharePoint site.

6. Click on Create Data Objects to view the list of Data Objects for the Data Source. Based on the Connection Type, default Object Fields are automatically added.

7. You may add more Data Objects as required. To add more Data Object, follow the below steps:

   1. Click on Add Object Fields.

   2. Add Object Name and Data Type.

   3. Click on Save.
      The new Data Object is added to the list.

Step 5: Configure the Object and Data Profile to Import data into the Destination System - Luma Knowledge

The next step is to create the Data Object to import Documents into Luma Knowledge.

Follow the steps to complete the configuration:

1. On the Manage Data Objects tab, click on Add Data Object.

2. Add the Data Object Name and Connection details.

   The new Data Object is added to the list.

3. Select the new Data Object to configure the Data Profile.

4. On Configure Data Profile page, add the following details:

   1. Set Connection Type to REST.

   2. Add the Upload API URL to your Luma Knowledge instance.
      e.g. https://lumaproedge.serviceaide.com/gateway/api/v1/artifacts/upload

   3. Set HTTP Method to POST.

      1. Add the following to Request Headers.
         Name : Content-Type Value : multipart/form-data Data Type: STRING
         Name : ClientType Value : Ajax Data Type: STRING
         Name : x-requested-with Value : XMLHttpRequest Data Type: STRING

   4. Select JSON as the Response Type.

   5. Under Extract Response Using, add '$' to the Enter JSON Path. This enables the system to read the complete response received from the Source system (Sharepoint).

   6. Set Is Paged Response to True and Paging Strategy to POST_PAYLOAD.

   7. Click on Save to complete the configuration.

5. Now click on Create Data Object Fields to view the list of Data Objects for the Data Source. Based on the Connection Type, default Object Fields are automatically added. You may add more Data Object fields as required.

Step 6: Configure the Synchronization Job between SharePoint and Luma Knowledge

Once the Data Sources and Data Objects are configured, the next step is to configure the Sync between your Sharepoint site and Luma Knowledge. Here, we map the Data Object in Source - Destination systems and configure the frequency at which the data sync is executed.

To create a Synchronization pair, follow the below steps:

1. Navigate to the Manage Synchronization tab.

2. Click on the Add Sync Object button.

    

3. On the Configure Synchronization screen, add the following details:

   1. Add Sync Job Name.

   2. Add Description.

   3. Set ‘Enable’ to False. Once the sync job is configured and tested, we can enable the job.

   4. Set 'Log Level' to Debug. This ensures the execution logs available for each sync run. The logs can be used to troubleshoot any errors with the sync.

   5. In Push Data To (Data Destination), Select Luma Knowledge Data Source and Data Object information.

   6. In Fetch Data From (Data Source), Select the Sharepoint Data Source and Data Object information.

   7. Map Fields: Map the Data Object Fields in Sharepoint with Data Object Fields in Luma Knowledge. For each Destination Data Object Field, add the mappings as below:

   8. For Destination Data Object Artifact, click on Configure to add Custom Function. The function enables the system to read the documents from the Sharepoint site and create Artifacts in Luma Knowledge. You may use the following default function or create a new function as per your organization’s requirement and the mappings needed.

      function(model) { var jsonModel = { "name": model.Name, "summary": model.Name, "description": "", "category": "IT", //refers to the Domain in Knowledge Graph under which the Artifacts should be created. "sourceType": "WEB_UI", "artifactState": "DRAFT", "curationStatus": "IN_PROGRESS" }; return JSON.stringify(jsonModel); }

      The above function processes the documents using the FAQ generation process. Luma Knowledge degenerates the document in QnA pairs, and the artifact is created under the domain 'IT'.
      Add the script and update the following fields in the script as required:

      1. Update ‘Category’ to the Domain in Knowledge Graph where the Artifact should be created.

      2. Update ‘ArtifactState’ to the default Artifact state of the artifact.

      3. Click Save.

   9. Set Enable Validation Rules to Disable.

   10. If you want to enable validation rules, set Enable Validation Rules to Enable. You may add a Custom Subroutine to Validation Rules to validate if the data record should be imported.

   11. Once the details are added, Click the Save button.

The Sync Job is now ready.

Step 7: Test and Execute Synchronization

Once the data sync job is configured, you can now enable the sync and execute the job. The data sync can be executed manually or scheduled to run at a defined frequency using a CRON job.

Follow the below steps to execute the sync:

1. On Manage Synchronization Jobs, select the Sharepoint-Luma Knowledge sync job (created in Step 5).

2. Click Edit to view the updated details.

3. Set Enabled to True.

4. To manually execute the job,

   1. Set the Frequency to One Time and click Save.

   2. Now scroll down to the page and click Execute to trigger the job.

   3. Alternatively, you may also execute the job from the Manage Synchronization Jobs tab. Select the job and click on Execute Sync Pair.

   4. Upon successful completion of the job, you can view the details on Execution Logs.

       

   5. On the Execution Logs, you can view the status of each execution.

   6. Select the record to view the details on the documents imported and artifacts created. In the below example, 1 document was downloaded from SharePoint and uploaded into Luma Knowledge.

If you want to schedule the job execution at a specific interval, set Frequency to Recurring and add the CRON Job to schedule the automatic sync run. Once scheduled, the job is executed at a specific interval of time.