How to run OPC UA PubSub on real-time Linux and TSN using open62541

This quick start guide serves as a starting point for a user to learn and evaluate OPC UA PubSub and TSN technologies for embedding into their products. This guide needs an x86 system with an Intel i210 Ethernet controller (as this controller supports features necessary for Time-Sensitive Networking) and leverages:

  • Standard Debian 10 operating system and the real-time Linux kernel included in its package manager.
    Note: In most cases, this kernel would already provide you the best possible hard-real-time deterministic performance. To further improve, you may have to tune the configuration and recompile the kernel on your own.
  • A later version of iproute2 package (with support for newer real-time socket options and an IEEE 802.1 Qbv like scheduler)
  • A later version of LinuxPTP package (with support for IEEE 802.1 AS gPTP configuration)
  • Open-source OPC UA stack open62541 – an open-source C (C99) implementation of OPC UA licensed under the Mozilla Public License v2.0 available in GitHub (with support for PubSub feature as specified in part 14 of OPC UA Specification)

Prerequisites

Hardware requirements:

We recommend at least two Intel x86-based systems with 4-cores and Intel i210 Ethernet Controllers. We used Intel Apollo Lake, Intel Whiskey Lake, and Intel core-i5 architectures during our tests.

Software requirements:

We selected Debian 10, as it is one of the more popular distributions. It also has a pre-compiled real-time kernel that can be installed using the Debian package manager. You can download the Debian GNU/Linux 10.7 distribution from here – Debian-10.7.0-amd64-xfce-CD-1. As it is not practically possible to provide a variant of the quick start guide for the many Linux distributions out there, we recommend that you use Debian for your initial tests and then switch to your chosen distribution. This is particularly important as our work focuses on high performance.

Architecture:

To set up the test architecture in this quick start guide, you would require two nodes with at least two Ethernet interfaces in each of the nodes, with one of them being an Intel i210 Ethernet controller. You need to connect one of the two Ethernet interfaces to your office network to download and install the required packages for this setup over the internet. The second interface (Intel i210) can be connected in a peer-to-peer fashion with the other node or via a TSN switch, as shown below. In case you are using a TSN switch between the two nodes, make sure you open all the gates in the TSN switch for your initial tests. It should be noted that we have used peer-to-peer networking for most of our tests.

OPC UA PubSub test architecture

Environment setup

In this section, we will set up the environment:
  • Install real-time Linux kernel
  • Configure network interface
  • Configure TSN parameters
Install real-time Linux kernel:
After installing the OS and setting up the network cables as shown in the architecture diagram, log in as the root user (We recommend executing all the commands given in this how-to guide as root user) and update the nodes using the below commands: apt-get update && apt-get upgrade
It might take a while for the update and upgrade to complete. You can then download and install the real-time kernel that is distributed by Debian and reboot the nodes using the below commands: apt-get install linux-image-rt-amd64 -y sudo reboot
After reboot, execute the below command and ensure that the newly installed real-time kernel is the one that is currently running. uname -r

The appearance of the “rt” keyword in the print message that appears as part of the command output shows that we have booted into the right kernel: RT kernel version 4.19.11

Configure network interface:

In this section, we will set up static IP address and VLAN configuration to the i210 interface. After reboot, log in as the root user and open the interfaces file using the below command:

vi /etc/network/interfaces
Copy and paste the following lines in the interfaces file in both the nodes and replace “X” as shown in the screenshot below. For e.g., In node 1 set the IP address as 192.168.100.1, and in node 2, set the IP address as 192.168.100.2. The PubSub TSN applications are configured to run with VLAN Id 8. For this purpose, VLAN configuration is done in the i210 interface as below: auto enp2s0
iface enp2s0 inet static
    address 192.168.100.X
    netmask 255.255.255.0
    broadcast 192.168.100.255
auto enp2s0.8
iface enp2s0.8 inet static
    address 192.168.8.X
    netmask 255.255.255.0

In our setup, the interface name is enp2s0; replace it with your nodes’ i210 interface name. Below is the screenshot of the interfaces files of the two nodes:TSN nodes interfaces file

Save the interfaces file and restart the networking service: /etc/init.d/networking restart
Now verify the static IP address and the connection between two nodes (from node 1 ping node 2 and vice versa) using the below commands: ip a ping 192.168.100.X
Configure TSN parameters:

In this section, we will run the necessary scripts (setup.sh and application_dependencies.sh) to configure the TSN parameters.

The setup.sh available in the package does the following,

  • Updates the installed packages in the node
  • Installs iproute2 package for kernel version 4.19
  • Installs Linux PTP version 2.0
  • Checks if 8021q module is loaded; if not, adds the same

If you are interested in knowing more about each command, read the comments included in the script for additional information. setup.sh

The application_dependencies.sh available in the package does the following,

  • Configures traffic control parameters to transmit and receive
  • Sets Egress and Ingress policies
  • Tunes real-time behaviors
  • Runs Linux PTP and PHC2SYS

If you are interested in knowing more about each command, read the comments included in the script for additional information. application_dependencies.sh

The above scripts are included in the demo_package.tar file. You can download them by using the below command: wget https://www.kalycito.com/wp-content/uploads/2021/02/demo_package.tar
Untar the demo_package.tar and enter into the demo_package folder: tar -xvf demo_package.tar && cd demo_package
Now, run the setup.sh script and reboot the nodes using the below commands: ./setup.sh sudo reboot

After successfully executing the setup.sh script, run the application_dependencies.sh available in the demo_package folder. As shown in the architecture diagram, configure one of the nodes as PTP master and the other node as PTP slave.

To configure the node as PTP master use the below command: ./application_dependencies.sh -i <IFACE> -m
To configure the node as PTP slave use the below command: ./application_dependencies.sh -i <IFACE> -s
The following messages appear while running application_dependencies.sh in the console. You can ignore these three “Invalid argument” messages shown below in the sample screenshot:
ntuple error
After configuring the node as PTP master or slave, the output log can be verified using the below command tail -f /var/log/ptp4l.log

If you have configured your node as a PTP master, your output log should be similar to the below image. The text “assuming the grand master role” ensures that you have configured the node as PTP master. Once you have seen the output, press Ctrl + C to return to the console.

ptp4l master log

If you have configured your node as a PTP slave, your output log should be similar to the below image. (Note: The master offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns). ptp4l slave log

The application_dependencies.sh would have run PHC2SYS utility to synchronize user clock and hardware clock. You can verify the output log of PHC2SYS in both the nodes using the below command: tail -f /var/log/phc2sys.log

Your PHC2SYS log should be similar to the below image. (Note: The sys offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns). phc2sys log

Build OPC UA PubSub application

After setting up the environment, we will build the open62541 PubSub application. To clone the open62541 stack from GitHub, use the below command: git clone https://github.com/open62541/open62541.git
Fetch the pull request and create a new branch using the below commands: (We are using this pull request as it contains the latest version of the TSN application required for our setup. Edit: This pull request is merged into the master and the SHA is 563b150def983ad0fba57fa73c446a65de9a0d58.) cd open62541/ git fetch origin pull/3996/head:local_branch && git checkout local_branch
Build the application in both the nodes using the following commands: mkdir build cd build/ cmake -DUA_BUILD_EXAMPLES=ON -DUA_ENABLE_PUBSUB=ON -DUA_ENABLE_PUBSUB_ETH_UADP=ON ..
You will receive the below message during CMake build in the console. You can ignore the below message.
cmake build
Now compile the application using the below command: make -j4

Run OPC UA PubSub application in Performance Evaluation mode

After successfully building the OPC UA PubSub application, we will run it by enabling the flag to generate the performance measurement log file.

Run the Loopback application in one of the nodes using the below command: (We recommend to run the pubsub_TSN_loopback application first to avoid missing the initial data published by pubsub_TSN_publisher). ./bin/examples/pubsub_TSN_loopback -interface <IFACE> -enableBlockingSocket
Run the Publisher application in the other node using the below command: ./bin/examples/pubsub_TSN_publisher -interface <IFACE> -enableBlockingSocket -enableLatencyCsvLog

The -enableLatencyCsvLog flag computes the round trip time (RTT) of a counter variable sent from one node to another and looped back (more information on this in the next section below). By default, this application stops after collecting 1 million packets. You can terminate it in-between by pressing Ctrl + C. The log file (latencyT1toT8.csv) will be generated in the pubsub_TSN_publisher application build folder (only after the application is terminated), containing the RTT, missed counters, and repeated counters.

Screenshot of a sample latencyT1toT8.csv log file:
sample opc ua pubsub packets

Performance Evaluation

The application in node 1 (node where pubsub_TSN_publisher is running) is designed to encode and publish a packet containing a counter variable to node 2 every 250 microseconds. The counter variable is incremented at the start of each new 250 microseconds cycle. Node 2 (node where pubsub_TSN_loopback is running) is designed to decode the received packet and loop the counter variable back to node 1. The application at node 1 receives the counter variable and computes the round-trip time: T8 – T1 (time taken to publish and receive back the same counter variable). This information is captured in the latencyT1toT8.csv file. The csv file also contains information on missed counters and duplicate counters – this can happen due to multi-threading issues or packets being lost during transmission/receive.

The below figure shows the flow of information and the timestamps at different stages (T1 to T8):
OPC UA PubSub application

  • T1 – Timestamp at which the counter variable was incremented (and handed over to the publisher)
  • T2 – Ethernet packet outgoing timestamp at kernel space
  • T3 – Ethernet packet incoming timestamp at kernel space
  • T4 – Timestamp at which the counter variable is seen by the application on node 2
  • T5 – Timestamp at which the counter variable was handed over to the publisher
  • T6 – Ethernet loopback packet outgoing timestamp at kernel space
  • T7 – Ethernet loopback packet incoming timestamp at kernel space
  • T8 – Timestamp at which the counter variable is seen by the application on node 1 after the round trip

The generated csv file (latencyT1toT8.csv) should be imported into a statistical computing program such as an R script to visualize the data and evaluate its performance. For this purpose, we used two R scripts, mkhisto.R and mkloghisto.R, which is available in the demo_package folder. The mkhisto.R script reads the latencyT1toT8.csv file and generates a histogram plot of PubSub RTT latency. The mkloghisto.R script reads the latencyT1toT8.csv file, and calculates the deviation of the obtained RTT latency from the expected 1 ms latency (which is 4*cycle-time), and generates a histogram plot for the same.

Execute the R scripts using the below commands in the node where pubsub_TSN_publisher application was executed: Rscript <FILE_PATH>/mkhisto.R latencyT1toT8.csv Rscript <FILE_PATH>/mkloghisto.R latencyT1toT8.csv

Both the histogram plots will be generated as PDF files (latencyT1toT8.pdf for mkhisto.R and latencyT1toT8-log.pdf for mkloghisto.R) in the current directory.

Below screenshot shows a sample graph of the obtained PubSub RTT latency generated by the mkhisto.R script:
OPC UA PubSub latency
Below screenshot shows a sample graph of the obtained PubSub RTT jitter generated by the mkloghisto.R script:
OPC UA PubSub Jitter

From the above plots, we see that all the packets/samples are in the range of desired 1 ms RTT (from latencyT1toT8.pdf) and the RTT jitter (from latencyT1toT8-log.pdf) is only a few micro-seconds. From this, we see that these nodes can run the PubSub application at a lower cycle-time of 250us and the behavior is deterministic within the boundaries seen in the plots. As part of our testing, we have further verified stability and deterministic behavior by running the setup for multiple days. If you wish to see the latency plots for long-term tests, look into the long-term plots in the OSADL QA farm.

Optional step 1: If you wish to make your TSN environment persistent after a reboot, follow the below steps:

Step 1:
To configure as PTP master: echo ./<FILE_PATH>/application_dependencies.sh -i <IFACE> -m >> /etc/rc.local
Step 2:
Make the bootup script executable using the below command: chmod +x /etc/rc.local
Step 3:
Execute the bootup script using the below command: /etc/rc.local

Optional step 2: To know more usage of the PubSub application,

./bin/examples/pubsub_TSN_loopback -help ./bin/examples/pubsub_TSN_publisher -help