How to run OPC UA PubSub on real-time Linux and TSN using open62541

This quick start guide serves as a starting point for a user to learn and evaluate OPC UA PubSub and TSN technologies for embedding into their products. This guide needs an x86 system with an Intel i210 Ethernet controller (as this controller supports features necessary for Time-Sensitive Networking) and leverages:

  • Standard Debian 10 operating system and the real-time Linux kernel included in its package manager.
    Note: In most cases, this kernel would already provide you the best possible hard-real-time deterministic performance. To further improve, you may have to tune the configuration and recompile the kernel on your own.
  • A later version of iproute2 package (with support for newer real-time socket options and an IEEE 802.1 Qbv like scheduler)
  • A later version of LinuxPTP package (with support for IEEE 802.1 AS gPTP configuration)
  • Open-source OPC UA stack open62541 – an open-source C (C99) implementation of OPC UA licensed under the Mozilla Public License v2.0 available in GitHub (with support for PubSub feature as specified in part 14 of OPC UA Specification)

Prerequisites

Hardware requirements:

We recommend at least two Intel x86-based systems with 4-cores and Intel i210 Ethernet Controllers. We used Intel Apollo Lake, Intel Whiskey Lake, and Intel core-i5 architectures during our tests.

Software requirements:

We selected Debian 10, as it is one of the more popular distributions. It also has a pre-compiled real-time kernel that can be installed using the Debian package manager. You can download the Debian GNU/Linux 10.9 distribution from here – Debian-10.9.0-amd64-xfce-CD-1. As it is not practically possible to provide a variant of the quick start guide for the many Linux distributions out there, we recommend that you use Debian for your initial tests and then switch to your chosen distribution. This is particularly important as our work focuses on high performance.

Architecture:

To set up the test architecture in this quick start guide, you would require two nodes with at least two Ethernet interfaces in each of the nodes, with one of them being an Intel i210 Ethernet controller. You need to connect one of the two Ethernet interfaces to your office network to download and install the required packages for this setup over the internet. The second interface (Intel i210) can be connected in a peer-to-peer fashion with the other node or via a TSN switch, as shown below. In case you are using a TSN switch between the two nodes, make sure you open all the gates in the TSN switch for your initial tests. It should be noted that we have used peer-to-peer networking for most of our tests.

OPC UA PubSub test architecture

Environment setup

In this section, we will set up the environment:
  • Install real-time Linux kernel
  • Configure network interface
  • Configure TSN parameters
Install real-time Linux kernel:
After installing the OS and setting up the network cables as shown in the architecture diagram, log in as the root user (We recommend executing all the commands given in this how-to guide as root user). To log in as the root user, use the below command: su - Now, update the nodes using the below command: apt-get update && apt-get upgrade
It might take a while for the update and upgrade to complete. You can then download and install the real-time kernel that is distributed by Debian and reboot the nodes using the below commands: apt-get install linux-image-rt-amd64 -y reboot
After reboot, execute the below command and ensure that the newly installed real-time kernel is the one that is currently running. uname -r

The appearance of the "rt" keyword in the print message that appears as part of the command output shows that we have booted into the right kernel: RT kernel version 4.19.11

Configure network interface:

In this section, we will set up static IP address and VLAN configuration to the i210 interface. After reboot, log in as the root user and open the interfaces file using the below commands:

su -
vi /etc/network/interfaces
Copy and paste the following lines at the end of the interfaces file in both the nodes and replace “X” as shown in the screenshot below and save the file. E.g., In node 1, set the IP address as 192.168.100.1, and in node 2, set the IP address as 192.168.100.2. The PubSub TSN applications are configured to run with VLAN Id 8. For this purpose, VLAN configuration is done in the i210 interface as below: auto enp2s0
iface enp2s0 inet static
    address 192.168.100.X
    netmask 255.255.255.0
    broadcast 192.168.100.255
auto enp2s0.8
iface enp2s0.8 inet static
    address 192.168.8.X
    netmask 255.255.255.0

Throughout this QSG we will be using enp2s0 for the interface name, replace it with your nodes’ i210 interface name.

Below is the screenshot of the interfaces files of the two nodes:TSN nodes interfaces file

The above highlighted section depends on your local LAN environment. Contact your system administrator for this information.

Save the interfaces file and restart the networking service: /etc/init.d/networking restart
Now verify the static IP address and the connection between two nodes (from node 1 ping node 2 and vice versa) using the below commands: ip a ping 192.168.100.X

Configure TSN parameters:

In this section, we will run the necessary scripts (setup.sh, application_dependencies.sh and build_open62541.sh) to configure the TSN parameters and build the open62541 PubSub application.

The setup.sh available in the package does the following,

  • Updates the installed packages in the node
  • Installs iproute2 package for kernel version 4.19
  • Installs Linux PTP version 2.0
  • Checks if 8021q module is loaded; if not, adds the same

If you are interested in knowing more about each command, read the comments included in the script for additional information. setup.sh

The application_dependencies.sh available in the package does the following,

  • Configures traffic control parameters to transmit and receive
  • Sets Egress and Ingress policies
  • Tunes real-time behaviors
  • Runs Linux PTP and PHC2SYS
  • Kernel mechanism to allocate CPU to the processes

If you are interested in knowing more about each command, read the comments included in the script for additional information. application_dependencies.sh

The build_open62541.sh available in the package does the following,

  • Clones the open62541 repository from GitHub
  • Fetches the pull request which contains the recent version of TSN applications
  • Builds the open62541 repository and compiles the TSN applications

If you are interested in knowing more about each command, read the comments included in the script for additional information. build_open62541.sh

The above scripts are included in the demo_package.tar file. Traverse to your home directory and you can download the .tar file by using the below commands: cd ~ wget https://www.kalycito.com/wp-content/uploads/2021/06/demo_package.tar
Untar the demo_package.tar and you will get the script files in the current folder tar -xvf demo_package.tar && cd demo_package
Now, run the setup.sh script using the below command: ./setup.sh It might take a while for the setup.sh script to complete.
Reboot the nodes after completing the initial setup reboot Note: We have modified the grub settings, so reboot is required.

Build OPC UA PubSub application

After reboot, log in as the root user and traverse to the demo_package folder using the below commands: su - cd ~/demo_package
Run the build_open62541.sh available in the demo_package folder to build the open62541 PubSub applications. ./build_open62541.sh

Run PTP and PHC2SYS

Run the application_dependencies.sh available in the demo_package folder. As shown in the architecture diagram, configure one of the nodes as PTP master and the other node as PTP slave.

To configure the node as PTP master use the below command: ./application_dependencies.sh -i enp2s0 -m
To configure the node as PTP slave use the below command: ./application_dependencies.sh -i enp2s0 -s
The following messages appear while running application_dependencies.sh in the console. You can ignore the error messages while running this script.
ntuple error
After configuring the node as PTP master or slave, the output log can be verified using the below command: tail -f /var/log/ptp4l.log

If you have configured your node as a PTP master, your output log should be similar to the below image. The text “assuming the grand master role” ensures that you have configured the node as PTP master. Once you have seen the output, press Ctrl + C to return to the console.

ptp4l master log

If you have configured your node as a PTP slave, your output log should be similar to the below image. (Note: The master offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns). ptp4l slave log

The application_dependencies.sh would have run PHC2SYS utility to synchronize user clock and hardware clock. You can verify the output log of PHC2SYS in both the nodes using the below command: tail -f /var/log/phc2sys.log

Your PHC2SYS log should be similar to the below image. (Note: The sys offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns). Once you have seen the output, press Ctrl + C to return to the console. phc2sys log


Run OPC UA PubSub application in Performance Evaluation mode

After establishing the time synchronization successfully in both the nodes, we will run the OPC UA PubSub application to generate the performance measurement log file.

Go to open62541 build directory to run the PubSub application using the below command: cd ~/open62541/build/
Run the Loopback application in one of the nodes using the below command: (We recommend to run the pubsub_TSN_loopback application first to avoid missing the initial data published by pubsub_TSN_publisher). ./bin/examples/pubsub_TSN_loopback -interface <IFACE> -enableBlockingSocket -pubMacAddress opc.eth://<MAC-of-node1>:8.3 -subMacAddress opc.eth://<MAC-of-node2>:8.3
Run the Publisher application in the other node using the below command: ./bin/examples/pubsub_TSN_publisher -interface <IFACE> -enableBlockingSocket -enableLatencyCsvLog -pubMacAddress opc.eth://<MAC-of-node2>:8.3 -subMacAddress opc.eth://<MAC-of-node1>:8.3
Note: Type the MAC address of the nodes in the format as mentioned in the below examples. Octets in the MAC address should be seperated by hyphen symbol (-).
In the node 2 terminal, type the below command: ./bin/examples/pubsub_TSN_loopback -interface enp2s0 -enableBlockingSocket -pubMacAddress opc.eth://00-d0-93-46-b2-ed:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-fc:8.3
In the node 1 terminal, type the below command: ./bin/examples/pubsub_TSN_publisher -interface enp2s0 -enableBlockingSocket -enableLatencyCsvLog -pubMacAddress opc.eth://00-d0-93-46-b2-fc:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-ed:8.3

The -enableLatencyCsvLog flag computes the round trip time (RTT) of a counter variable sent from one node to another and looped back (more information on this in the next section below). By default, this application stops after collecting 1 million packets. You can terminate it in-between by pressing Ctrl + C. The log file (latencyT1toT8.csv) will be generated in the pubsub_TSN_publisher application build folder (only after the application is terminated), containing the RTT, missed counters, and repeated counters.

Screenshot of a sample latencyT1toT8.csv log file:
sample opc ua pubsub packets

The first column in the latencyT1toT8.csv shows the RTT value, the second column shows the missed counters and the third column shows the repeated counters. The application runs successfully with RTT of 1ms, and 0 missed and repeated counters, as shown in the image above.

Performance Evaluation

The application in node 1 (node where pubsub_TSN_publisher is running) is designed to encode and publish a packet containing a counter variable to node 2 every 250 microseconds. The counter variable is incremented at the start of each new 250 microseconds cycle. Node 2 (node where pubsub_TSN_loopback is running) is designed to decode the received packet and loop the counter variable back to node 1. The application at node 1 receives the counter variable and computes the round-trip time: T8 – T1 (time taken to publish and receive back the same counter variable). This information is captured in the latency CSV files. The CSV files also contain information on missed counters and duplicate counters – this can happen due to multi-threading issues or packets being lost during transmit/receive.

The below figure shows the flow of information and the timestamps at different stages (T1 to T8):
OPC UA PubSub application

  • T1 – Timestamp at which the counter variable was incremented (and handed over to the publisher)
  • T2 – Ethernet packet outgoing timestamp at kernel space
  • T3 – Ethernet packet incoming timestamp at kernel space
  • T4 – Timestamp at which the counter variable is seen by the application on node 2
  • T5 – Timestamp at which the counter variable was handed over to the publisher
  • T6 – Ethernet loopback packet outgoing timestamp at kernel space
  • T7 – Ethernet loopback packet incoming timestamp at kernel space
  • T8 – Timestamp at which the counter variable is seen by the application on node 1 after the round trip

To setup munin for monitoring real-time performance

Now that we have seen the successful working of the PubSub applications, we will now setup munin for monitoring 24*7 real-time performance.

Configure munin:

In this section, we will setup the munin master and slave nodes. We recommend running the munin master in a third node. If you are using a high speed processor, you can use one of the two nodes (say, node1) as a munin master as shown below.

  1. Setup a Munin Master Webserver in one of the nodes (say, node 1)
    1. Install Apache2 server and dependency packages
      2. apt-get install -y apache2 libcgi-fast-perl libapache2-mod-fcgid
    2. Check the fcgid module status
      3. After installing the dependency packages, the fcgid module shall be enabled. Double-check using the command
      /usr/sbin/apachectl -M | grep -i cgi
      The above command shall return “fcgid_module (shared)”. If the output is blank, the fcgid module is not enabled. Enable the fcgid module using the command
      a2enmod fcgid
    3. Install Munin Master Webserver
      4. apt-get install -y munin
    4. Configure the Munin Master Webserver by modifying /etc/munin/munin.conf file
      1. vi /etc/munin/munin.conf
      2. Uncomment the following lines in /etc/munin/munin.conf and save it
        3. dbdir /var/lib/munin
        htmldir /var/cache/munin/www
        logdir /var/log/munin
        rundir /var/run/munin

        tmpldir /etc/munin/templates
      3. Open the apache2 configuration file in the path /etc/munin/ and perform the 2 changes mentioned below.
        4. Note: Apache2 configuration filename shall vary based on the OS that is being used. It shall be either apache2.conf or apache24.conf.

        CHANGE 1: Ensure that the below lines are available in the apache configuration file. If not, add these changes to the file.

        <Directory /var/cache/munin/www>
        #    Require local
        #    Options None
            Require all granted
            Options FollowSymLinks SymLinksIfOwnerMatch
        </Directory>

        <Directory /usr/lib/munin/cgi>
        #    Require local
            Require all granted
            Options FollowSymLinks SymLinksIfOwnerMatch
            <IfModule mod_fcgid.c>
                SetHandler fcgid-script
            </IfModule>
            <IfModule !mod_fcgid.c>
                SetHandler cgi-script
            </IfModule>
        </Directory>

        CHANGE 2: Check if the following lines are present in the apache configuration file

        <Location /munin-cgi/munin-cgi-graph>
            Require local

        If the above lines are present, replace with the below lines. If not present, ignore this change

        <Location /munin-cgi/munin-cgi-graph>
            Require all granted
            Options FollowSymLinks SymLinksIfOwnerMatch
    5. Activate Munin master webserver
      6. Restart Apache2 and Munin services using the commands:
      /etc/init.d/apache2 restart
      /etc/init.d/munin restart
      /etc/init.d/munin-node restart
    6. Verify Munin Master Webserver
      7. To verify if the Munin Webservice has been hosted successfully, access the webservice from the URL: http://<munin_master_ipaddress>/munin/

      Example: If the IP Address of the system in which the Munin Master Webserver is installed is 172.17.8.11, access the Munin Master Webserver at http://172.17.8.11/munin/

  2. Setup Munin Client – (In both peer-to-peer nodes)
    1. Setup both the nodes as Munin Client to monitor its RTT latency, PTP and PHC2SYS values continuously.
    2. Follow the below steps in both the nodes to setup as a Munin client.
      1. Install munin node
        2. apt-get install munin-node -y
      2. Configure Munin Node
        1. To make the Munin node communicate with Munin Master Webserver, modify the /etc/munin/munin-node.conf file.
          2. vi /etc/munin/munin-node.conf
        2. Add your Munin server IP address after the line mentioned below, in the /etc/munin/munin-node.conf file and save it.
          3. allow ^127\.0\.0\.1$

          Note that the IP address is in regex format, so assuming that the Munin server IP address is 172.17.8.11, the line should be

          allow ^172\.17\.8\.11$
      3. Restart the Munin Node
        4. /etc/init.d/munin-node restart
  3. Add Munin Nodes to Munin Master Webserver - (In node1)
    Munin server must be configured with the installed Munin nodes to its configuration to reflect in the browser.
    To add the Munin Nodes to Munin Master Webserver, follow the below steps.
    1. Configure Munin Master Configuration Open the file /etc/munin/munin.conf in Munin Master and insert the following lines at the end of the file and save the changes:
      vi /etc/munin/munin.conf

      [MuninNode1]
          address <munin_node1_ip_address>
          use_node_name yes
      [MuninNode2]
          address <munin_node2_ip_address>
          use_node_name yes

      Example:

[MuninNodeIPC1]
    address 172.17.8.11
    use_node_name yes
[MuninNodeIPC2]
    address 172.17.8.12
    use_node_name yes

Note: The Node name specified within square brackets shall be the name of the Munin node that is displayed in the Munin Master Webserver.

  • Restart Munin Master Webserver and Munin Node
    • Restart the Munin master to enable the configuration changes
    /etc/init.d/munin restart
    /etc/init.d/munin-node restart

    Note: Wait for at least 5 minutes to see the newly installed nodes in Munin Master Webserver. This wait is because the Munin Master is configured in such a way that it executes all its scripts and collects the values from the Munin nodes after every 5 minutes. So, any configuration change made to Munin nodes and Munin Master shall reflect only after 5 minutes.

    Traverse to the munin scripts directory, cd ~/open62541/examples/pubsub_realtime/pubsubTSN_muninScripts/
    Run the below script in both the nodes where the pubsub application is set to run (node1 & node2). To run the muninscript, follow the below command: ./munin_setup.sh -d <open62541_build_directory_from_home_folder> -i <IFACE>
    For example, ./munin_setup.sh -d ~/open62541/build -i enp2s0

    To monitor real-time performance of PubSub applications

    Go to open62541 build directory to run the PubSub application using the below command: cd ~/open62541/build/
    Run the Loopback application in one of the nodes (say node2) using the below command: (We recommend to run the pubsub_TSN_loopback application first to avoid missing the initial data published by pubsub_TSN_publisher). ./bin/examples/pubsub_TSN_loopback -interface <IFACE> -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://<MAC-of-node1>:8.3 -subMacAddress opc.eth://<MAC-of-node2>:8.3 > application_log.txt 2>&1 &
    Run the Publisher application in the other node (node1) using the below command: ./bin/examples/pubsub_TSN_publisher -interface <IFACE> -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://<MAC-of-node2>:8.3 -subMacAddress opc.eth://<MAC-of-node1>:8.3 > application_log.txt 2>&1 &
    Note: Type the MAC address of the nodes in the format as mentioned in the below examples. Octets in the MAC address should be seperated by hyphen symbol (-).
    In the node 2 terminal, type the below command: ./bin/examples/pubsub_TSN_loopback -interface enp2s0 -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://00-d0-93-46-b2-ed:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-fc:8.3 > application_log.txt 2>&1 &
    In the node 1 terminal, type the below command: ./bin/examples/pubsub_TSN_publisher -interface enp2s0 -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://00-d0-93-46-b2-fc:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-ed:8.3 > application_log.txt 2>&1 &

    The -enableLongRunMeasurements flag computes the Round Trip Time (RTT) of a counter variable sent from one node to another and looped back. The generated CSV contains the current time, RTT, missed counters, and repeated counters.

    For every 5 minutes the munin scripts are used to evalute the performance from the generated latency CSV and the data is updated in the munin graph.

    To view the munin graph, type in the ip address of your munin server in the format <ip-of-munin-server>/munin where you will find both the nodes listed as munin slaves.

    Recommended step - Use exit command to terminate the session from the nodes.

    If you reboot the system, you have to run the steps under the section “Run PTP and PHC2SYS” and then run the OPC UA PubSub application in both the nodes.

    Optional step 1: If you wish to make your TSN environment persistent after a reboot, follow the below steps:

    Step 1:
    To configure as PTP master: echo ./<FILE_PATH>/application_dependencies.sh -i <IFACE> -m >> /etc/rc.local
    Step 2:
    Make the bootup script executable using the below command: chmod +x /etc/rc.local
    Step 3:
    Execute the bootup script using the below command: /etc/rc.local

    Optional step 2: Use NTP in parallel

    You need to take care of the below items if NTP is enabled. If you are facing any difficulties on setting up the NTP, feel free to contact us.
    • 1. Run NTP on PTP master
    • 2. PHC2SYS should take the systemtime and update PTP hardware clock
    • 3. The above action will create time jumps. There will be filtering options needed so that the PTP time will not have any time jumps

    Optional step 3: Configure other VLAN Id for IFACE

    We have configured only VLAN Id 8 for the i210 interface in this QSG as we run our application with VLAN Id 8. If you want to add best effort network traffic in the system, you can configure other VLAN Id from 1 to 7 by following the below steps. Modify the /etc/network/interfaces file as a root user using the below commands: su - vi /etc/network/interfaces
    Copy and paste the following lines at the end of the interfaces file in both the nodes and replace “X” with the same value as configured for enp2s0. auto enp2s0.1
    iface enp2s0.1 inet static
        address 192.168.1.X
        netmask 255.255.255.0
    auto enp2s0.2
    iface enp2s0.2 inet static
        address 192.168.2.X
        netmask 255.255.255.0
    auto enp2s0.3
    iface enp2s0.3 inet static
        address 192.168.3.X
        netmask 255.255.255.0
    auto enp2s0.4
    iface enp2s0.4 inet static
        address 192.168.4.X
        netmask 255.255.255.0
    auto enp2s0.5
    iface enp2s0.5 inet static
        address 192.168.5.X
        netmask 255.255.255.0
    auto enp2s0.6
    iface enp2s0.6 inet static
        address 192.168.6.X
        netmask 255.255.255.0
    auto enp2s0.7
    iface enp2s0.7 inet static
        address 192.168.7.X
        netmask 255.255.255.0
    Modify the below commands instead of the existing egress and ingress policy configuration in application_dependencies.sh file
      for i in `seq 1 8`; do for j in `seq 0 7`;do sudo ip link set $IFACE.$i type vlan egress $j:$j ; done; done for i in `seq 1 8`; do for j in `seq 0 7`;do sudo ip link set $IFACE.$i type vlan ingress $j:$j ; done; done

    Download Whitepaper
    New to OPC UA?
    OPC UA online course
    Learn OPC UA with code walk through and open source examples
    Know More     Need help with TSN switches?
    OPCUA TSN
    Our engineers can guide you to setup and optimize your TSN network
    Know More     New to real-time Linux?
    Real time Linux
    Choose the right kernel, configure, tune, measure real-time performance and troubleshoot latencies
    Know More     New to NTP or PTP?
    OPCUA Time Synchronization
    Interested in strategies to distribute global time to your automation network?
    Contact Us     Get access to our OPC UA + TSN Performance Whitepaper
    OPCUA Performance Whitepaper
    Fill in the form below to download our OPC UA + TSN Performance whitepaper
    Download Whitepaper