How to run OPC UA PubSub on real-time Linux and TSN using open62541

This quick start guide serves as a starting point for a user to learn and evaluate OPC UA PubSub and TSN technologies for embedding into their products. This guide needs an x86 system with an Intel i210 Ethernet controller (as this controller supports features necessary for Time-Sensitive Networking) and leverages:

• Standard Debian 10 operating system and the real-time Linux kernel included in its package manager.
  Note: In most cases, this kernel would already provide you the best possible hard-real-time deterministic performance. To further improve, you may have to tune the configuration and recompile the kernel on your own.
• A later version of iproute2 package (with support for newer real-time socket options and an IEEE 802.1 Qbv like scheduler)
• A later version of LinuxPTP package (with support for IEEE 802.1 AS gPTP configuration)
• Open-source OPC UA stack open62541 – an open-source C (C99) implementation of OPC UA licensed under the Mozilla Public License v2.0 available in GitHub (with support for PubSub feature as specified in part 14 of OPC UA Specification)

Prerequisites

Hardware requirements:

We recommend at least two Intel x86-based systems with 4-cores and Intel i210 Ethernet Controllers. We used Intel Apollo Lake, Intel Whiskey Lake, and Intel core-i5 architectures during our tests.

Software requirements:

We selected Debian 10, as it is one of the more popular distributions. It also has a pre-compiled real-time kernel that can be installed using the Debian package manager. You can download the Debian GNU/Linux 10.9 distribution from here – Debian-10.9.0-amd64-xfce-CD-1. As it is not practically possible to provide a variant of the quick start guide for the many Linux distributions out there, we recommend that you use Debian for your initial tests and then switch to your chosen distribution. This is particularly important as our work focuses on high performance.

Architecture:

To set up the test architecture in this quick start guide, you would require two nodes with at least two Ethernet interfaces in each of the nodes, with one of them being an Intel i210 Ethernet controller. You need to connect one of the two Ethernet interfaces to your office network to download and install the required packages for this setup over the internet. The second interface (Intel i210) can be connected in a peer-to-peer fashion with the other node or via a TSN switch, as shown below. In case you are using a TSN switch between the two nodes, make sure you open all the gates in the TSN switch for your initial tests. It should be noted that we have used peer-to-peer networking for most of our tests.

Environment setup

In this section, we will set up the environment:

• Install real-time Linux kernel
• Configure network interface
• Configure TSN parameters

Install real-time Linux kernel:

The appearance of the “rt” keyword in the print message that appears as part of the command output shows that we have booted into the right kernel:

Configure network interface:

In this section, we will set up static IP address and VLAN configuration to the i210 interface. After reboot, log in as the root user and open the interfaces file using the below commands:

The above highlighted section depends on your local LAN environment. Contact your system administrator for this information.

Configure TSN parameters:

In this section, we will run the necessary scripts (setup.sh, application_dependencies.sh and build_open62541.sh) to configure the TSN parameters and build the open62541 PubSub application.

The setup.sh available in the package does the following,

• Updates the installed packages in the node
• Installs iproute2 package for kernel version 4.19
• Installs Linux PTP version 2.0
• Checks if 8021q module is loaded; if not, adds the same

If you are interested in knowing more about each command, read the comments included in the script for additional information. setup.sh

The application_dependencies.sh available in the package does the following,

• Configures traffic control parameters to transmit and receive
• Sets Egress and Ingress policies
• Tunes real-time behaviors
• Runs Linux PTP and PHC2SYS
• Kernel mechanism to allocate CPU to the processes

If you are interested in knowing more about each command, read the comments included in the script for additional information. application_dependencies.sh

The build_open62541.sh available in the package does the following,

• Clones the open62541 repository from GitHub
• Fetches the pull request which contains the recent version of TSN applications
• Builds the open62541 repository and compiles the TSN applications

If you are interested in knowing more about each command, read the comments included in the script for additional information. build_open62541.sh

Build OPC UA PubSub application

Run PTP and PHC2SYS

Run the application_dependencies.sh available in the demo_package folder. As shown in the architecture diagram, configure one of the nodes as PTP master and the other node as PTP slave.

If you have configured your node as a PTP master, your output log should be similar to the below image. The text “assuming the grand master role” ensures that you have configured the node as PTP master. Once you have seen the output, press Ctrl + C to return to the console.

If you have configured your node as a PTP slave, your output log should be similar to the below image. (Note: The master offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns).

Your PHC2SYS log should be similar to the below image. (Note: The sys offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns). Once you have seen the output, press Ctrl + C to return to the console.

Run OPC UA PubSub application in Performance Evaluation mode

After establishing the time synchronization successfully in both the nodes, we will run the OPC UA PubSub application to generate the performance measurement log file.

The -enableLatencyCsvLog flag computes the round trip time (RTT) of a counter variable sent from one node to another and looped back (more information on this in the next section below). By default, this application stops after collecting 1 million packets. You can terminate it in-between by pressing Ctrl + C. The log file (latencyT1toT8.csv) will be generated in the pubsub_TSN_publisher application build folder (only after the application is terminated), containing the RTT, missed counters, and repeated counters.

The first column in the latencyT1toT8.csv shows the RTT value, the second column shows the missed counters and the third column shows the repeated counters. The application runs successfully with RTT of 1ms, and 0 missed and repeated counters, as shown in the image above.

The application in node 1 (node where pubsub_TSN_publisher is running) is designed to encode and publish a packet containing a counter variable to node 2 every 250 microseconds. The counter variable is incremented at the start of each new 250 microseconds cycle. Node 2 (node where pubsub_TSN_loopback is running) is designed to decode the received packet and loop the counter variable back to node 1. The application at node 1 receives the counter variable and computes the round-trip time: T8 – T1 (time taken to publish and receive back the same counter variable). This information is captured in the latency CSV files. The CSV files also contain information on missed counters and duplicate counters – this can happen due to multi-threading issues or packets being lost during transmit/receive.

The below figure shows the flow of information and the timestamps at different stages (T1 to T8):

• T1 – Timestamp at which the counter variable was incremented (and handed over to the publisher)
• T2 – Ethernet packet outgoing timestamp at kernel space
• T3 – Ethernet packet incoming timestamp at kernel space
• T4 – Timestamp at which the counter variable is seen by the application on node 2
• T5 – Timestamp at which the counter variable was handed over to the publisher
• T6 – Ethernet loopback packet outgoing timestamp at kernel space
• T7 – Ethernet loopback packet incoming timestamp at kernel space
• T8 – Timestamp at which the counter variable is seen by the application on node 1 after the round trip

To setup munin for monitoring real-time performance

Now that we have seen the successful working of the PubSub applications, we will now setup munin for monitoring 24*7 real-time performance.

Configure munin:

In this section, we will setup the munin master and slave nodes. We recommend running the munin master in a third node. If you are using a high speed processor, you can use one of the two nodes (say, node1) as a munin master as shown below.

1. Setup a Munin Master Webserver in one of the nodes (say, node 1)
   1. Install Apache2 server and dependency packages
   apt-get install -y apache2 libcgi-fast-perl libapache2-mod-fcgid
   2. Check the fcgid module status
   After installing the dependency packages, the fcgid module shall be enabled. Double-check using the command
   /usr/sbin/apachectl -M | grep -i cgi
   The above command shall return “fcgid_module (shared)”. If the output is blank, the fcgid module is not enabled. Enable the fcgid module using the command
   a2enmod fcgid
   3. Install Munin Master Webserver
   apt-get install -y munin
   4. Configure the Munin Master Webserver by modifying /etc/munin/munin.conf file
      1. vi /etc/munin/munin.conf
      2. Uncomment the following lines in /etc/munin/munin.conf and save it
      dbdir /var/lib/munin
htmldir /var/cache/munin/www
logdir /var/log/munin
rundir /var/run/munin

tmpldir /etc/munin/templates
      3. Open the apache2 configuration file in the path /etc/munin/ and perform the 2 changes mentioned below.
      Note: Apache2 configuration filename shall vary based on the OS that is being used. It shall be either apache2.conf or apache24.conf.

      CHANGE 1: Ensure that the below lines are available in the apache configuration file. If not, add these changes to the file.

      <Directory /var/cache/munin/www>
#    Require local
#    Options None
    Require all granted
    Options FollowSymLinks SymLinksIfOwnerMatch
</Directory>

<Directory /usr/lib/munin/cgi>
#    Require local
    Require all granted
    Options FollowSymLinks SymLinksIfOwnerMatch
    <IfModule mod_fcgid.c>
        SetHandler fcgid-script
    </IfModule>
    <IfModule !mod_fcgid.c>
        SetHandler cgi-script
    </IfModule>
</Directory>
      

      CHANGE 2: Check if the following lines are present in the apache configuration file

      <Location /munin-cgi/munin-cgi-graph>
    Require local
      
      

      If the above lines are present, replace with the below lines. If not present, ignore this change

      <Location /munin-cgi/munin-cgi-graph>
    Require all granted
    Options FollowSymLinks SymLinksIfOwnerMatch
      
   5. Activate Munin master webserver
      
   Restart Apache2 and Munin services using the commands:
   /etc/init.d/apache2 restart
/etc/init.d/munin restart
/etc/init.d/munin-node restart
   
   6. Verify Munin Master Webserver
   To verify if the Munin Webservice has been hosted successfully, access the webservice from the URL: http://<munin_master_ipaddress>/munin/
   

   Example: If the IP Address of the system in which the Munin Master Webserver is installed is 172.17.8.11, access the Munin Master Webserver at http://172.17.8.11/munin/

2. Setup Munin Client – (In both peer-to-peer nodes)
   1. Setup both the nodes as Munin Client to monitor its RTT latency, PTP and PHC2SYS values continuously.
   2. Follow the below steps in both the nodes to setup as a Munin client.
      1. Install munin node
      apt-get install munin-node -y
      2. Configure Munin Node
         1. To make the Munin node communicate with Munin Master Webserver, modify the /etc/munin/munin-node.conf file.
         vi /etc/munin/munin-node.conf
         2. Add your Munin server IP address after the line mentioned below, in the /etc/munin/munin-node.conf file and save it.
         allow ^127\.0\.0\.1$

         Note that the IP address is in regex format, so assuming that the Munin server IP address is 172.17.8.11, the line should be

         allow ^172\.17\.8\.11$
      3. Restart the Munin Node
      /etc/init.d/munin-node restart
3. Add Munin Nodes to Munin Master Webserver – (In node1)
   Munin server must be configured with the installed Munin nodes to its configuration to reflect in the browser.
   To add the Munin Nodes to Munin Master Webserver, follow the below steps.
   1. Configure Munin Master Configuration Open the file /etc/munin/munin.conf in Munin Master and insert the following lines at the end of the file and save the changes:
      vi /etc/munin/munin.conf

[MuninNode1]
    address <munin_node1_ip_address>
    use_node_name yes
[MuninNode2]
    address <munin_node2_ip_address>
    use_node_name yes
      

      Example:

      [MuninNodeIPC1]
    address 172.17.8.11
    use_node_name yes
[MuninNodeIPC2]
    address 172.17.8.12
    use_node_name yes
      

      Note: The Node name specified within square brackets shall be the name of the Munin node that is displayed in the Munin Master Webserver.

   2. Restart Munin Master Webserver and Munin Node
   Restart the Munin master to enable the configuration changes
   /etc/init.d/munin restart
/etc/init.d/munin-node restart
   

   Note: Wait for at least 5 minutes to see the newly installed nodes in Munin Master Webserver. This wait is because the Munin Master is configured in such a way that it executes all its scripts and collects the values from the Munin nodes after every 5 minutes. So, any configuration change made to Munin nodes and Munin Master shall reflect only after 5 minutes.

To monitor real-time performance of PubSub applications

The -enableLongRunMeasurements flag computes the Round Trip Time (RTT) of a counter variable sent from one node to another and looped back. The generated CSV contains the current time, RTT, missed counters, and repeated counters.

For every 5 minutes the munin scripts are used to evalute the performance from the generated latency CSV and the data is updated in the munin graph.

To view the munin graph, type in the ip address of your munin server in the format <ip-of-munin-server>/munin where you will find both the nodes listed as munin slaves.

Recommended step – Use exit command to terminate the session from the nodes.

If you reboot the system, you have to run the steps under the section “Run PTP and PHC2SYS” and then run the OPC UA PubSub application in both the nodes.

Optional step 1: If you wish to make your TSN environment persistent after a reboot, follow the below steps: