Login
    Contents x
    Exporting Data
    • 02 Feb 2024
    • 6 Minutes to read
    • Print
    • Dark
      Light
    Contents

    Exporting Data

    • Updated on 02 Feb 2024
    • 6 Minutes to read
    • Print
    • Dark
      Light
    Article Summary

    This article explains how to export data from Tegsoft systems to physical files.

    Prerequisites

    To perform the procedures in this article smoothly, Tegsoft recommends that you have basic knowledge of these topics:

    • Networking

    • SSH connection

    • Linux Command Line Interface

    The Export Process

    To execute the export process you need to prepare the export config file and the environment. Once these preparations are ready you can run the export process once or continuously like a service.

    Workspace and Environment

    All the data management processes are handled by the “Tegsoft Touch Data Processing” service. For export operations, this service can be executed from a command line interface as a service or a docker instance. The process requires the preparation of a config file.

    Export Config File

    The config file is in JSON format and has various sections. The file can be accessed via an http URL or a local file.

    A sample config file is shown below,

    {
  "activePeriods": [
    {
      "timeBegin": 200,
      "timeEnd": 500
    }
  ],
  "exportRules": [
    {
      "name": "Export rule",
      "description": "Export selected tables",
      "source": {
        "dbUser": "tobe",
        "dbPassword": "ab2037ef5bb349a1a46116581cb8fec5",
        "dbDriver": "com.ibm.db2.jcc.DB2Driver",
        "dbUrl": "jdbc:db2://127.0.0.1:50000/tobe",
        "PBXID": "ebd37af5-260a-436f-9bdf-44bcc9b1b946",
        "UNITUID": "4a55c1e3-edd5-46ef-b66f-d74634e8469a"
      },
      "destination": "/root/dataExport/",
      "exportType": "tableList",
      "tableList": [
        "TBLCCAGENTLOG",
        "TBLCCCDR",
        "TBLDMLLOG"
      ],
      "timeCondition": "last10Minutes"
    }
  ]
}

    Execution Periods

    Execution of the process can be limited to a specific period or periods, this is handled via “activePeriods” definition. The definition is an array and contains allowed periods. Each period is defined via blocks marked with beginning and ending marks. If a period has multiple markers like the “time between” and “date between” both period conditions need to match.

    If any period matches whether with a single condition or multiple conditions, execution can start if none matches then the process will not execute. If there is no period defined then the process will execute.

    The “activePeriods” element can be used globally in the config file or under each rule individually. If global conditions don’t match execution will not do anything, if conditions under the rule don’t match the related rule will not execute.

    Syntax,

    "activePeriods":[
      { // Period 1
         Condition 1, Condition 2, ... Condition N
      },
      {
         // Period 2
      },
....
      {
         // Period n
      }
]

    Both begin and end parameters are included.

    timeBegin: That field value represents the beginning time of the period in 24-hour format. Value needs to be in a decimal form. Examples,

    11pm or 23:00 → 2300

    8:30am or 08:30 → 830

    One past midnight 00:01 → 1

    timeEnd: That field value represents the ending time of the period in 24-hour format. Value needs to be in a decimal form. Examples,

    11pm or 23:00 → 2300

    8:30am or 08:30 → 830

    One past midnight 00:01 → 1

    time: That field value represents the exact time of the period in 24-hour format. Value needs to be in a decimal form. Examples,

    11pm or 23:00 → 2300

    8:30am or 08:30 → 830

    One past midnight 00:01 → 1

    The period is compared with the current time,

    If you have timeBegin: 1 and timeEnd: 2359

    That 22:10 will match or 08:30 will match but 00:00 will not match.

    dayOfMonthBegin: This is the starting point of the “day of the month” condition. This field takes integer values between 1 - 31

    dayOfMonthEnd: This is the ending point of the “day of the month” condition. This field takes integer values between 1 - 31

    dayOfMonth: This is the exact day of the month. This field takes integer values between 1 - 31

    dayOfWeekBegin: This is the starting point of the “day of the week” condition. This field takes integer values between 1 - 7. The week starts on Monday and ends on Sunday. Monday: 1 and Sunday: 7.

    dayOfWeekEnd: This is the ending point of the “day of the week” condition. This field takes integer values between 1 - 7. The week starts on Monday and ends on Sunday. Monday: 1 and Sunday: 7.

    dayOfWeek: This is the exact day of the week condition. This field takes integer values between 1 - 7. The week starts on Monday and ends on Sunday. Monday: 1 and Sunday: 7.

    dateBegin: This is the starting point of the date condition. This field takes string value in YYYYMMDD format like 20230727 is 27th of July 2023

    dateEnd: This is the ending point of the date condition. This field takes string value in YYYYMMDD format like 20230727 is 27th of July 2023

    date: This is the exact date condition. This field takes string value in YYYYMMDD format like 20230727 is 27th of July 2023

    Export Rules

    This section is for defining export actions and parameters. This article mainly covers topics related to this section.

    name: It is always good to have a rule name so that the process can be tracked, managed, and monitored. The name needs to be unique against all rules in the file.

    description: An optional description area for taking notes and sharing more information about the rule.

    source: Defines source database connection parameters including dbUser, dbPassword (encrypted), dbDriver, dbUrl, PBXID, and UNITUID.

    exportType: That field defines the process that will be executed. The field only supports the “tableList” value for now.

    tableList: is an array of table names that is going to be exported.

    timeCondition: Supports only the “last10Minutes” value for now. And will query to find data that has been changed/created for the last 10 minutes.

    destination: This is the export destination folder. This parameter is mandatory. destination can be a local file system, mounted NFS storage area, FTP, or SFTP-like destinations. Any kind of mountable destination can be used. This folder is the base folder for backup action. YYYYMMDD formated subfolder will be created during backup execution.

    Command Line Execution

    Before you start command line environment needs to be prepared. The environment must be prepared in a Tegsoft Database Instance.

    mkdir -p /root/dataProcessingWorkspace
cd /root/dataProcessingWorkspace

wget -O update.sh https://setup.tegsoftcloud.com/resources/dataProcessingWorkspace/update.sh
chmod +x /root/dataProcessingWorkspace/update.sh
/root/dataProcessingWorkspace/update.sh

    You can always run the command below for help.

    /root/dataProcessingWorkspace/help.sh

    Once you prepared your config file you can execute the process with the commands listed below (Please mind changing the config file name if needed).

    
export configUrl=/root/dataProcessingWorkspace/backupConfig.json
nohup /root/dataProcessingWorkspace/runProcess.sh > output.txt 2>&1 &

    Please mind that passing “-d” as an argument will run the process as a service and will not stop after the first execution.

    
export configUrl=/root/dataProcessingWorkspace/backupConfig.json
nohup /root/dataProcessingWorkspace/runProcess.sh -d > output.txt 2>&1 &

    During execution, you can exit the shell. The execution will continue and logs will be accessible via the output.txt file.

    Example Run and Output

    Here is an example config file,

    
{
  "exportRules": [
    {
      "name": "Export rule",
      "description": "Export selected tables",
      "source": {
        "dbUser": "tobe",
        "dbPassword": "ab2037ef5bb349a1a46116581cb8fec5",
        "dbDriver": "com.ibm.db2.jcc.DB2Driver",
        "dbUrl": "jdbc:db2://127.0.0.1:50000/tobe",
        "PBXID": "ebd37af5-260a-436f-9bdf-44bcc9b1b946",
        "UNITUID": "4a55c1e3-edd5-46ef-b66f-d74634e8469a"
      },
      "destination": "/root/dataExport/",
      "exportType": "tableList",
      "tableList": [
        "TBLCCAGENTLOG",
        "TBLCCCDR",
        "TBLDMLLOG"
      ],
      "timeCondition": "last10Minutes"
    }
  ]
}

    Here is an execution example,

    
export configUrl=/root/dataProcessingWorkspace/exportConfig.json 
nohup /root/dataProcessingWorkspace/runProcess.sh > exportOutput.txt 2>&1 &

    Example process log output,

    
2024-02-02 16:15:47,426 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] START export rule - Export rule
2024-02-02 16:15:47,443 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] jdbc:db2://127.0.0.1:50000/tobe
2024-02-02 16:15:47,504 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] EXPORT_ACTIVE - Starting export (tableList) of table TBLCCCDR into /root/dataExport//20240202
2024-02-02 16:15:47,539 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] EXPORT_ACTIVE - DONE export (tableList) of table TBLCCCDR into /root/dataExport//20240202
2024-02-02 16:15:47,539 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] EXPORT_ACTIVE - Starting export (tableList) of table TBLCCAGENTLOG into /root/dataExport//20240202
2024-02-02 16:15:47,562 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] EXPORT_ACTIVE - DONE export (tableList) of table TBLCCAGENTLOG into /root/dataExport//20240202
2024-02-02 16:15:47,565 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] EXPORT_ACTIVE - Starting export (tableList) of table TBLDMLLOG into /root/dataExport//20240202
2024-02-02 16:15:47,575 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] EXPORT_ACTIVE - DONE export (tableList) of table TBLDMLLOG into /root/dataExport//20240202
2024-02-02 16:15:47,582 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] DONE export rule - Export rule
2024-02-02 16:15:47,582 [INFO|com.tegsoft.touch.components.DataProcessing.DataProcessingService|DataProcessingService] LOOP_COMPLETE in 165 ms. SLEEP_5MIN

    Example JSON export,

    cat /root/dataExport/20240202/20240202161547-dataExport.json

    
{
   "executionEndDateTime":"02/02/2024 16:15",
   "success":true,
   "tableList":[
      "TBLCCAGENTLOG",
      "TBLCCCDR",
      "TBLDMLLOG"
   ],
   "datasets":[
      {
         "currentRowIndex":0,
         "currentRow":[
            
         ],
         "name":"TBLCCCDR",
         "rows":[
            {
               "INTRUNKID":"ea27068e-41c6-4ddd-8508-7cf303baaf69",
               "SRC":"05055302158",
               "SKILL":"7000",
               "PBXID":"91e5c35d-3849-434d-8ca7-9e349f01f6ff",
               "SKILLPARAM2":"1",
               "SKILLPARAM3":"6",
               "SKILLPARAM1":"1",
               "DISPOSITION":"ANSWERED",
               "AUDIOFILENAME":"20240202-1706879638.652",
               "LASTAPP":"Queue",
               "DCONTEXT":"fromea27068e-41c6-4ddd-8508-7cf303baaf69",
               "CLID":"Eray Gursoy <05055302158>",
               "LINENUMBER":4,
               "USERFIELD":"INBOUND",
               "CONTID":"8c6d3bb9-389b-4a13-ae05-43f05662ddf0",
               "DST":"902129125614",
               "AMAFLAGS":3,
               "CALLDATE":"2024-02-02 16:13:58.0",
               "dataRowState":"Unchanged",
               "LASTDATA":"7000;t;;;;;;;;",
               "UNIQUEID":"1706879638.652",
               "CHANNEL":"PJSIP/ea27068e-41c6-4ddd-8508-7cf303baaf69-0000005a",
               "DSTCHANNEL":"Local/6768@tegsoft-APP-AGENT-0000007c;1",
               "BILLSEC":0,
               "SKILLEVENT":"ABANDON",
               "CDRID":"eb42ad23-c2ae-4fae-9fce-aa60df85ebbc",
               "INROUTEID":"80a292af-df9f-415c-b329-da98b22ba119",
               "DURATION":16
            }
         ]
      },
      {
         "currentRowIndex":0,
         "currentRow":[
            
         ],
         "name":"TBLCCAGENTLOG",
         "rows":[
            {
               "SESSIONID":"7e911efe-7c07-46f6-ad8b-dab8b0d64c99",
               "SKILL":"7000",
               "LOGTYPE":"BUSY",
               "PHONE":"05055302158",
               "DIRECTION":"INBOUND",
               "LOGID":"60a0c118-f894-406f-80ce-dd66d2aaf44f",
               "dataRowState":"Unchanged",
               "ENDDATE":"2024-02-02 16:14:12.446",
               "UID":"7e911efe-7c07-46f6-ad8b-dab8b0d64c99",
               "STATUS":"BUSY",
               "INTERFACE":"Local/6768@tegsoft-APP-AGENT",
               "RINTERFACE":"Local/006768-91e5c35d-3849-434d-8ca7-9e349f01f6ff@tegsoft-CLUSTER-AGENT",
               "CALLID":"1706879638.652",
               "DESTCALLID":"1706879652.659",
               "BEGINDATE":"2024-02-02 16:14:12.359",
               "REASON":"POPUP",
               "SRCEVENT":"AgentCalled-1706879638.652",
               "CONTID":"8c6d3bb9-389b-4a13-ae05-43f05662ddf0"
            },
            {
               "SESSIONID":"7e911efe-7c07-46f6-ad8b-dab8b0d64c99",
               "SKILL":"7000",
               "LOGTYPE":"BUSY",
               "PHONE":"05055302158",
               "DIRECTION":"INBOUND",
               "LOGID":"66ccfccf-ea27-47ab-a20f-c65aaf60bdf0",
               "dataRowState":"Unchanged",
               "ENDDATE":"2024-02-02 16:14:10.338",
               "UID":"7e911efe-7c07-46f6-ad8b-dab8b0d64c99",
               "STATUS":"BUSY",
               "INTERFACE":"Local/6768@tegsoft-APP-AGENT",
               "RINTERFACE":"Local/006768-91e5c35d-3849-434d-8ca7-9e349f01f6ff@tegsoft-CLUSTER-AGENT",
               "CALLID":"1706879638.652",
               "DESTCALLID":"1706879648.653",
               "BEGINDATE":"2024-02-02 16:14:08.171",
               "REASON":"POPUP",
               "SRCEVENT":"AgentCalled-1706879638.652",
               "CONTID":"8c6d3bb9-389b-4a13-ae05-43f05662ddf0"
            },
            {
               "SESSIONID":"7e911efe-7c07-46f6-ad8b-dab8b0d64c99",
               "SKILL":"7000",
               "LOGTYPE":"BUSY",
               "PHONE":"05055302158",
               "DIRECTION":"INBOUND",
               "LOGID":"5d7cd80c-c093-45f2-a444-dabfe31c42c9",
               "dataRowState":"Unchanged",
               "ENDDATE":"2024-02-02 16:14:10.338",
               "UID":"7e911efe-7c07-46f6-ad8b-dab8b0d64c99",
               "STATUS":"BUSY",
               "INTERFACE":"Local/6768@tegsoft-APP-AGENT",
               "RINTERFACE":"Local/006768-91e5c35d-3849-434d-8ca7-9e349f01f6ff@tegsoft-CLUSTER-AGENT",
               "CALLID":"1706879638.652",
               "DESTCALLID":"1706879650.656",
               "BEGINDATE":"2024-02-02 16:14:10.261",
               "REASON":"POPUP",
               "SRCEVENT":"AgentCalled-1706879638.652",
               "CONTID":"8c6d3bb9-389b-4a13-ae05-43f05662ddf0"
            }
         ]
      }
   ],
   "executionDuration":0,
   "executionStartDateTime":"02/02/2024 16:15"
}


    Was this article helpful?
    What's Next
    Change password!
    Changing your password will log you out immediately. Use the new password to log back in.
    Change profile
    Success!
    First name must have atleast 2 characters. Numbers and special characters are not allowed.
    Last name must have atleast 1 characters. Numbers and special characters are not allowed.
    Enter a valid email
    Enter a valid password
    Your profile has been successfully updated.
    Logout