How to run OPC UA PubSub on real-time Linux and TSN using open62541

Configure TSN parameters:

In this section, we will run the necessary scripts (setup.sh, application_dependencies.sh and build_open62541.sh) to configure the TSN parameters and build the open62541 PubSub application.

The setup.sh available in the package does the following,

• Updates the installed packages in the node
• Installs iproute2 package for kernel version 4.19
• Installs Linux PTP version 2.0
• Checks if 8021q module is loaded; if not, adds the same

If you are interested in knowing more about each command, read the comments included in the script for additional information. setup.sh

The application_dependencies.sh available in the package does the following,

• Configures traffic control parameters to transmit and receive
• Sets Egress and Ingress policies
• Tunes real-time behaviors
• Runs Linux PTP and PHC2SYS
• Kernel mechanism to allocate CPU to the processes

If you are interested in knowing more about each command, read the comments included in the script for additional information. application_dependencies.sh

The build_open62541.sh available in the package does the following,

• Clones the open62541 repository from GitHub
• Fetches the pull request which contains the recent version of TSN applications
• Builds the open62541 repository and compiles the TSN applications

If you are interested in knowing more about each command, read the comments included in the script for additional information. build_open62541.sh

Build OPC UA PubSub application

Run PTP and PHC2SYS

Run the application_dependencies.sh available in the demo_package folder. As shown in the architecture diagram, configure one of the nodes as PTP master and the other node as PTP slave.

The following messages appear while running application_dependencies.sh in the console. You can ignore the error messages while running this script.

If you have configured your node as a PTP master, your output log should be similar to the below image. The text “assuming the grand master role” ensures that you have configured the node as PTP master. Once you have seen the output, press Ctrl + C to return to the console.

Run OPC UA PubSub application in Performance Evaluation mode

After establishing the time synchronization successfully in both the nodes, we will run the OPC UA PubSub application to generate the performance measurement log file.

The -enableLatencyCsvLog flag computes the round trip time (RTT) of a counter variable sent from one node to another and looped back (more information on this in the next section below). By default, this application stops after collecting 1 million packets. You can terminate it in-between by pressing Ctrl + C. The log file (latencyT1toT8.csv) will be generated in the pubsub_TSN_publisher application build folder (only after the application is terminated), containing the RTT, missed counters, and repeated counters.

Now that we have seen the successful working of the PubSub applications, we will now setup munin for monitoring 24*7 real-time performance.

Configure munin:

In this section, we will setup the munin master and slave nodes. We recommend running the munin master in a third node. If you are using a high speed processor, you can use one of the two nodes (say, node1) as a munin master as shown below.

• Setup a Munin Master Webserver in one of the nodes (say, node 1)
  • • • Install Apache2 server and dependency packages

    apt-get install -y apache2 libcgi-fast-perl libapache2-mod-fcgid

  • • • Check the fcgid module status

    After installing the dependency packages, the fcgid module shall be enabled. Double-check using the command
    /usr/sbin/apachectl -M | grep -i cgi
    The above command shall return “fcgid_module (shared)”. If the output is blank, the fcgid module is not enabled. Enable the fcgid module using the command
    a2enmod fcgid

  • • • Install Munin Master Webserver

    apt-get install -y munin

• • • Configure the Munin Master Webserver by modifying /etc/munin/munin.conf file
      vi /etc/munin/munin.conf
      • • • Uncomment the following lines in /etc/munin/munin.conf and save it

        dbdir /var/lib/muninhtmldir /var/cache/munin/wwwlogdir /var/log/muninrundir /var/run/munintmpldir /etc/munin/templates

  • Open the apache2 configuration file in the path /etc/munin/ and perform the 2 changes mentioned below.
  • Note: Apache2 configuration filename shall vary based on the OS that is being used. It shall be either apache2.conf or apache24.conf.CHANGE 1: Ensure that the below lines are available in the apache configuration file. If not, add these changes to the file.<Directory /var/cache/munin/www>#Require local#Options NoneRequire all grantedOptions FollowSymLinks SymLinksIfOwnerMatch</Directory><Directory /usr/lib/munin/cgi>#Require localRequire all grantedOptions FollowSymLinks SymLinksIfOwnerMatch<IfModule mod_fcgid.c>SetHandler fcgid-script</IfModule><IfModule !mod_fcgid.c></IfModule></Directory>CHANGE 2: Check if the following lines are present in the apache configuration file
    <Location /munin-cgi/munin-cgi-graph>Require local
    If the above lines are present, replace with the below lines. If not present, ignore this change
    <Location /munin-cgi/munin-cgi-graph>Require all grantedOptions FollowSymLinks SymLinksIfOwnerMatch

  • Restart Apache2 and Munin services using the commands:
    /etc/init.d/apache2 restart/etc/init.d/munin restart/etc/init.d/munin-node restart
  • Verify Munin Master Webserver
    • To verify if the Munin Webservice has been hosted successfully, access the webservice from the URL: http://<munin_master_ipaddress>/munin/Example: If the IP Address of the system in which the Munin Master Webserver is installed is 172.17.8.11, access the Munin Master Webserver at http://172.17.8.11/munin/

• Setup Munin Client – (In both peer-to-peer nodes)
  • Setup both the nodes as Munin Client to monitor its RTT latency, PTP and PHC2SYS values continuously.
  • Follow the below steps in both the nodes to setup as a Munin client.
    • Install munin node
      apt-get install munin-node -y
    • Configure Munin Node
    • • • To make the Munin node communicate with Munin Master Webserver, modify the /etc/munin/munin-node.conf file.
      vi /etc/munin/munin-node.conf
      • • Add your Munin server IP address after the line mentioned below, in the /etc/munin/munin-node.conf file and save it.

      allow ^127\.0\.0\.1$
      Note that the IP address is in regex format, so assuming that the Munin server IP address is 172.17.8.11, the line should be
      allow ^172\.17\.8\.11$

      • • Restart the Munin Node

      /etc/init.d/munin-node restart

• • • • • Add Munin Nodes to Munin Master Webserver – (In node1)
          Munin server must be configured with the installed Munin nodes to its configuration to reflect in the browser. To add the Munin Nodes to Munin Master Webserver, follow the below steps.
          • Configure Munin Master Configuration
            • Open the file /etc/munin/munin.conf in Munin Master and insert the following lines at the end of the file and save the changes:
            vi /etc/munin/munin.conf[MuninNode1]address <munin_node1_ip_address>use_node_name yes[MuninNode2]address<munin_node2_ip_address><use_node_name yes

      Example:
      
      [MuninNodeIPC1]address 172.17.8.11use_node_name yes[MuninNodeIPC2]address 172.17.8.12use_node_name yes
      Note: The Node name specified within square brackets shall be the name of the Munin node that is displayed in the Munin Master Webserver.

    • • Restart Munin Master Webserver and Munin Node
  • • • Restart the Munin master to enable the configuration changes
        /etc/init.d/munin restart/etc/init.d/munin-node restart
        Note: Wait for at least 5 minutes to see the newly installed nodes in Munin Master Webserver. This wait is because the Munin Master is configured in such a way that it executes all its scripts and collects the values from the Munin nodes after every 5 minutes. So, any configuration change made to Munin nodes and Munin Master shall reflect only after 5 minutes.

• • • Traverse to the munin scripts directory,
      cd ~/open62541/examples/pubsub_realtime/pubsubTSN_muninScripts/
      Run the below script in both the nodes where the pubsub application is set to run (node1 & node2). To run the muninscript, follow the below command:
      ./munin_setup.sh -d <open62541_build_directory_from_home_folder> -i <IFACE>
      For example,
      ./munin_setup.sh -d ~/open62541/build -i enp2s0

To monitor real-time performance of PubSub applications

The -enableLongRunMeasurements flag computes the Round Trip Time (RTT) of a counter variable sent from one node to another and looped back. The generated CSV contains the current time, RTT, missed counters, and repeated counters.

For every 5 minutes the munin scripts are used to evalute the performance from the generated latency CSV and the data is updated in the munin graph.

To view the munin graph, type in the ip address of your munin server in the format <ip-of-munin-server>/munin where you will find both the nodes listed as munin slaves.

Recommended step – Use exit command to terminate the session from the nodes.

If you reboot the system, you have to run the steps under the section “Run PTP and PHC2SYS” and then run the OPC UA PubSub application in both the nodes.

Optional step 1: If you wish to make your TSN environment persistent after a reboot, follow the below steps:

• • • Step 1:

      To configure as PTP master:
      echo ./<FILE_PATH>/application_dependencies.sh -i <IFACE> -m >> /etc/rc.local

      Or to configure as PTP slave:
      echo ./<FILE_PATH>/application_dependencies.sh -i <IFACE> -s >> /etc/rc.local

      Step 2:

      Make the bootup script executable using the below command:
      chmod +x /etc/rc.local

      Step 3:

      Execute the bootup script using the below command:
      /etc/rc.local

Optional step 2: Use NTP in parallel

• • • 1. Run NTP on PTP master
    • 2. PHC2SYS should take the systemtime and update PTP hardware clock
    • 3. The above action will create time jumps. There will be filtering options needed so that the PTP time will not have any time jumps