ESA On-premise Agent v2.0.0 Runbook

Owned by Saurabh Ranadive
Last updated: Dec 15, 2022 by Ricardo Andrade
6 min read

For Serviceaide Automation On-Premise Agent deployment, the following components should be installed on the on-premise server:

  • itas_mule_adaptor

  • itas_connectors

  • itas_service_bus

  • sa_tunneling_service

  • mule_run

Use 'lumauser' to execute all the docker container, deployment scripts, and all the docker commands instead of sudo user. 

Pre-Deployment Actions

Perform the following Pre-deployment actions:

  1. Set up artifacts

    1. Download and transfer itas-deployment-2.0.0-onprem.zip to /platform/deployment directory and unzip

      cd /platform/deployment unzip itas-deployment-2.0.0-onprem.zip -d itas-deployment-2.0.0-onprem

2. Set up "master.properties"

  1. Populate master.properties file located under /platform/deployment/itas-deployment-2.0.0-onprem/itas-deployment-2.0.0/deploy directory.

    ############ Common Configuration - BEGIN ############ DOCKER_REGISTRY=registry-1.docker.io STOP_DOCKER_RESTART_ON_APP_STARTUP_FAILURE=false ############ Common Configuration - END ############ ############ Application URLs - BEGIN ############ # Use Automation application URL to populate both of below properties ESA_APPLICATION_URL= SA_ONPREM_BRIDGE_URL= ############ Application URLs - END ############ ############ Versions & Servers - BEGIN ############ ITAS_SERVICE_BUS_VERSION=2.0.0 ITAS_MULE_ADAPTOR_VERSION=2.0.0 ITAS_DBUPGRADE_VERSION=2.0.0 ITAS_CONNECTORS_PACKAGE_VERSION=2.0.0 ITAS_MULE_BASE_VERSION=2.0.0 SA_TUNNELLING_SERVICE_VERSION=2.0.0 #ITAS Hosts #Provide ip of the host machine where docker container will run. If all the container are on the same machine then all ip will be same. ESA_ACTIVEMQ_MASTER_BROKER_SERVER_NAME= ESA_ACTIVEMQ_SLAVE_BROKER_SERVER_NAME= SA_TUNNELING_ACTIVEMQ_MASTER_BROKER_SERVER_NAME= ITAS_MULE_ADAPTOR_SERVER_NAME= ITAS_MULE_RUN_SERVER_NAME= ITAS_SERVICE_BUS_SERVER_NAME= SA_TUNNELING_SERVICE_SERVER_NAME= ############ Versions & Servers - END ############ ############ Database Configuration - BEGIN ############ #ITAS MySQL Database ESA_DB_HOST_NAME= ESA_DB_USERNAME= #Use the encryption utility to get the encrypted password and provide it here ESA_DB_PASSWORD= ESA_DB_PORT=3306 ESA_DB_TYPE=MYSQL ESA_DB_DRIVER=com.mysql.jdbc.Driver #ITAS Mule Database (MySQL) ESA_MULE_DB_HOST_NAME= ESA_MULE_DB_USERNAME= #Use the encryption utility to get the encrypted password and provide it here ESA_MULE_DB_PASSWORD= ESA_MULE_DB_PORT=3306 ESA_MULE_DB_DRIVER=com.mysql.jdbc.Driver ############ Database Configuration - END ############ ############ Base Machine Configuration - BEGIN ############ ITAS_BASE_MACHINE_SERVICE_BUS_DATASTORE_PATH=/platform/data/servicebus_store/itas ITAS_BASE_MACHINE_ATTACHMENTS_STORE_PATH=/platform/data/attachment_store/itas # Mule Run - Base Machine mount paths ITAS_MULE_DIR_PATH=/platform/data/mule_run MULE_RUN_INITIAL_MEMORY=512 MULE_RUN_MAX_MEMORY=512 ############ Base Machine Configuration- END ############ ############ Application Ports - START ############ #ITAS ActiveMQ Ports ESA_ACTIVEMQ_MASTER_BROKER_PORT=61516 ESA_ACTIVEMQ_SLAVE_BROKER_PORT=61515 #ITAS Jolokia Ports ITAS_MULE_ADAPTOR_JOLOKIA_PORT=7502 ITAS_SERVICE_BUS_MASTER_JOLOKIA_PORT=7505 SA_TUNNELING_SERVICE_JOLOKIA_PORT=7504 #ITAS JVM DEBUG Ports TAS_MULE_ADAPTOR_JVM_DEBUG_PORT=4502 SA_TUNNELING_SERVICE_JVM_DEBUG_PORT=4504 ITAS_SERVICE_BUS_MASTER_JVM_DEBUG_PORT=4505 #ITAS JMX Ports ITAS_MULE_ADAPTOR_JMX_PORT=2502 SA_TUNNELING_SERVICE_JMX_PORT=2504 ITAS_SERVICE_BUS_MASTER_JMX_PORT=2505 #Heap Memory allocations ITAS_MULE_ADAPTOR_MIN_HEAP=512 ITAS_MULE_ADAPTOR_MAX_HEAP=512 ITAS_SERVICE_BUS_MIN_HEAP=512 ITAS_SERVICE_BUS_MAX_HEAP=512 SA_TUNNELING_SERVICE_MIN_HEAP=512 SA_TUNNELING_SERVICE_MAX_HEAP=512 #Tunneling Service #ActiveMQ ports should be same as application activemq ports. e.g., for ITAS deployment ITAS_ACTIVEMQ_MASTER_BROKER_PORT and SA_TUNNELING_ACTIVEMQ_MASTER_BROKER_PORT should be same. SA_TUNNELING_ACTIVEMQ_MASTER_BROKER_PORT=61516 SA_TUNNELING_SERVICE_PORT=10095 ############ Application Ports - END ############ #ITAS Email Configurations #Below credentials used to send email notification for db/ootb content upgrade failure and application emails ITAS_DB_MIGRATION_ERRORS_ENABLE_EMAIL=true ITAS_EMAIL_SERVER_USERNAME= ITAS_EMAIL_SERVER_PASSWORD= ITAS_EMAIL_SERVER_NAME= ITAS_EMAIL_SERVER_PORT= ITAS_DB_MIGRATION_ERRORS_SEND_TO_EMAIL_ADDRESS= ############ Miscellaneous Common - BEGIN ############ MULE_ENCRYPTION_PASSWORD=serviceaide #proxy server configuration ENABLE_PROXY=false HTTP_PROXY_HOST= HTTP_PROXY_PORT= HTTP_PROXY_USER= HTTP_PROXY_PASS= HTTPS_PROXY_HOST= HTTPS_PROXY_PORT= HTTPS_PROXY_USER= HTTPS_PROXY_PASS= SOCK_PROXY_HOST= SOCK_PROXY_PORT= SOCK_PROXY_USER= SOCK_PROXY_PASS= #Below three properties are the list of hosts ip or domain name where proxy not needed. #example HTTP_PROXY_EXCLUDE_HOSTS=localhost|127.0.0.1|*.serviceaide.com|*.domain.local HTTP_PROXY_EXCLUDE_HOSTS= HTTPS_PROXY_EXCLUDE_HOSTS= SOCK_PROXY_EXCLUDE_HOSTS= ############ Miscellaneous Common - END ############ ############ On Premise Properties - BEGIN ############ SA_TUNNELING_TARGET_APP=ESA SA_TUNNELING_TENANT_EXTERNAL_ID= SA_TUNNELING_APPLICATION_IDENTIFIER= SA_TUNNELING_SECRET_KEY= ############ On Premise Properties - END ############

  2. Configure system ulimits in file /etc/security/limits.conf

    centos soft nproc 16384 centos hard nproc 16384 centos soft nofile 65536 centos hard nofile 65536 lumauser soft nproc 16384 lumauser hard nproc 16384 lumauser soft nofile 65536 lumauser hard nofile 65536 root soft nproc 16384 root hard nproc 16384 root soft nofile 65536 root hard nofile 65536

  3. Login to dockerhub using the below command:

    docker login https://registry-1.docker.io -u serviceaideindia # use the dockerhub password provided separately

  4. Setup directory structure

    sudo mkdir -p /platform/data/logs/itas/itas_mule_adaptor sudo mkdir -p /platform/data/logs/itas/itas_db_migration sudo mkdir -p /platform/data/logs/itas/itas_service_bus sudo chmod -R 777 /platform/data/logs/itas sudo mkdir -p /platform/data/attachment_store/itas sudo mdir -p /platform/data/servicebus_store/itas sudo chown -R 1000:1000 /platform/data/servicebus_store/itas sudo chown -R 1000:1000 /platform/data/attachment_store/itas

  5. Install mule_run (one-time activity, not required if you are upgrading / already have mule_run installed)

    cd /tmp sudo wget -q https://repository.mulesoft.org/nexus/content/repositories/public/org/mule/distributions/mule-standalone/3.9.0-hf2/mule-standalone-3.9.0-hf2.zip sudo unzip -qq mule-standalone-3.9.0-hf2.zip sudo mv mule-standalone-3.9.0-hf2 /platform/data/mule_run sudo chown -R 1000:1000 /platform/data/mule_run

In Master.properties file, the following properties are populated from your Automation tenant:

  • SA_TUNNELING_TENANT_EXTERNAL_ID - Populate your Automation Tenant External ID here.

  • SA_TUNNELING_APPLICATION_IDENTIFIER - Add the Execution Agent External ID here.

  • SA_TUNNELING_SECRET_KEY - Copy the On-premise Execution Agent Key here.

The keys are available on Automation → Execution Agent screen. Most of the properties on the file are self-explanatory. In case of any issue, please contact Serviceaide Support.

Start installation

Execute the below commands to start the installation:

cd /platform/deployment/itas-deployment-2.0.0-onprem/itas-deployment-2.0.0/deploy chmod +x onprem_stack.sh ./onprem_stack.sh deployall

Post Deployment Actions

Run the below command to clean up older unused volumes, images, etc after deployment.

#run command to delete all the unused docker images (This command will not delete docker images that are in use) docker rmi -f $(docker images | awk '{ print $3 }') > /dev/null 2>&1 #cleanup unsed volume, cache etc. docker system prune To check logs sudo tail -f `docker inspect --format='{{.LogPath}}' itas_muledb_migration`

Troubleshoot Errors

  1. If we get the below "permission denied error" : Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.37/containers/redis/stop: dial unix /var/run/docker.sock: connect: permission denied.

    [lumauser@lumatrialweb01 deploy]$ docker ps --> If permission denied comes then execute below command. [lumauser@lumatrialweb01 deploy]$ sudo setfacl -m user:lumauser:rw /var/run/docker.sock [lumauser@lumatrialweb01 deploy]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [lumauser@lumatrialweb01 deploy]$ docker login (Enter docker credentials)

     

  • Ensuring that if the deployment is completed or not :   
     Whenever we enter "Docker ps" command then we can see the status of the containers, If any container's status is less than 1 minute then focus on that and wait 1 or 2 minutes and issue again the same 'docker ps' command , if we see the same container's status is less than a minute then we can ensure that there will be some issue and later we can go ahead with "docker logs [container_name]"  

    [lumauser@ip-10-0-9-7 deploy]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5a1ee4f1d88d registry-1.docker.io/serviceaide/mule_run_base:2.0.0 "sh -c /opt/app/star…" 8 minutes ago Up 8 minutes mule_run 59b47a9d49b8 registry-1.docker.io/serviceaide/sa_tunneling_service:2.0.0 "sh -c /opt/app/star…" 9 minutes ago Up 9 minutes 0.0.0.0:2504->2504/tcp, 0.0.0.0:4504->4504/tcp, 0.0.0.0:10095->10095/tcp sa_tunneling_service cb070978f76c registry-1.docker.io/serviceaide/itas_mule_adaptor:2.0.0 "sh -c /opt/app/star…" 10 minutes ago Up 10 minutes 0.0.0.0:2502->2502/tcp, 0.0.0.0:4502->4502/tcp, 0.0.0.0:7502->7502/tcp itas_mule_adaptor 3f7b5cb0d0c1 registry-1.docker.io/serviceaide/itas_service_bus:2.0.0 "sh -c /opt/app/star…" 11 minutes ago Up 11 minutes 0.0.0.0:2505->2505/tcp, 0.0.0.0:4505->4505/tcp, 0.0.0.0:7505->7505/tcp, 0.0.0.0:61516->61516/tcp itas_service_bus [lumauser@ip-10-0-9-7 deploy]$

Â