How to run OPC UA PubSub on real-time Linux and TSN using open62541
This quick start guide serves as a starting point for a user to learn and evaluate OPC UA PubSub and TSN technologies for embedding into their products. This guide needs an x86 system with an Intel i210 Ethernet controller (as this controller supports features necessary for Time-Sensitive Networking) and leverages:
- Standard Debian 10 operating system and the real-time Linux kernel included in its package manager.
Note: In most cases, this kernel would already provide you the best possible hard-real-time deterministic performance. To further improve, you may have to tune the configuration and recompile the kernel on your own.
- A later version of iproute2 package (with support for newer real-time socket options and an IEEE 802.1 Qbv like scheduler)
- A later version of LinuxPTP package (with support for IEEE 802.1 AS gPTP configuration)
- Open-source OPC UA stack open62541 – an open-source C (C99) implementation of OPC UA licensed under the Mozilla Public License v2.0 available in GitHub (with support for PubSub feature as specified in part 14 of OPC UA Specification)
We recommend at least two Intel x86-based systems with 4-cores and Intel i210 Ethernet Controllers. We used Intel Apollo Lake, Intel Whiskey Lake, and Intel core-i5 architectures during our tests.
We selected Debian 10, as it is one of the more popular distributions. It also has a pre-compiled real-time kernel that can be installed using the Debian package manager. You can download the Debian GNU/Linux 10.9 distribution from here – Debian-10.9.0-amd64-xfce-CD-1. As it is not practically possible to provide a variant of the quick start guide for the many Linux distributions out there, we recommend that you use Debian for your initial tests and then switch to your chosen distribution. This is particularly important as our work focuses on high performance.
To set up the test architecture in this quick start guide, you would require two nodes with at least two Ethernet interfaces in each of the nodes, with one of them being an Intel i210 Ethernet controller. You need to connect one of the two Ethernet interfaces to your office network to download and install the required packages for this setup over the internet. The second interface (Intel i210) can be connected in a peer-to-peer fashion with the other node or via a TSN switch, as shown below. In case you are using a TSN switch between the two nodes, make sure you open all the gates in the TSN switch for your initial tests. It should be noted that we have used peer-to-peer networking for most of our tests.
Environment setupIn this section, we will set up the environment:
- Install real-time Linux kernel
- Configure network interface
- Configure TSN parameters
Install real-time Linux kernel:
su -Now, update the nodes using the below command:
apt-get update && apt-get upgrade
apt-get install linux-image-rt-amd64 -y
The appearance of the "rt" keyword in the print message that appears as part of the command output shows that we have booted into the right kernel:
Configure network interface:
In this section, we will set up static IP address and VLAN configuration to the i210 interface. After reboot, log in as the root user and open the interfaces file using the below commands:
iface enp2s0 inet static
iface enp2s0.8 inet static
Throughout this QSG we will be using enp2s0 for the interface name, replace it with your nodes’ i210 interface name.Below is the screenshot of the interfaces files of the two nodes:
The above highlighted section depends on your local LAN environment. Contact your system administrator for this information.
Configure TSN parameters:
In this section, we will run the necessary scripts (setup.sh, application_dependencies.sh and build_open62541.sh) to configure the TSN parameters and build the open62541 PubSub application.
The setup.sh available in the package does the following,
- Updates the installed packages in the node
- Installs iproute2 package for kernel version 4.19
- Installs Linux PTP version 2.0
- Checks if 8021q module is loaded; if not, adds the same
If you are interested in knowing more about each command, read the comments included in the script for additional information. setup.sh
The application_dependencies.sh available in the package does the following,
- Configures traffic control parameters to transmit and receive
- Sets Egress and Ingress policies
- Tunes real-time behaviors
- Runs Linux PTP and PHC2SYS
- Kernel mechanism to allocate CPU to the processes
If you are interested in knowing more about each command, read the comments included in the script for additional information. application_dependencies.sh
The build_open62541.sh available in the package does the following,
- Clones the open62541 repository from GitHub
- Fetches the pull request which contains the recent version of TSN applications
- Builds the open62541 repository and compiles the TSN applications
If you are interested in knowing more about each command, read the comments included in the script for additional information. build_open62541.sh
tar -xvf demo_package.tar && cd demo_package
./setup.shIt might take a while for the setup.sh script to complete.
rebootNote: We have modified the grub settings, so reboot is required.
Build OPC UA PubSub application
Run PTP and PHC2SYS
Run the application_dependencies.sh available in the demo_package folder. As shown in the architecture diagram, configure one of the nodes as PTP master and the other node as PTP slave.
./application_dependencies.sh -i enp2s0 -m
./application_dependencies.sh -i enp2s0 -s
tail -f /var/log/ptp4l.log
If you have configured your node as a PTP master, your output log should be similar to the below image. The text “assuming the grand master role” ensures that you have configured the node as PTP master. Once you have seen the output, press Ctrl + C to return to the console.
If you have configured your node as a PTP slave, your output log should be similar to the below image. (Note: The master offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns).
tail -f /var/log/phc2sys.log
Your PHC2SYS log should be similar to the below image. (Note: The sys offset values are represented in nanosecond (ns). This value should be within the range of -1000 ns to +1000 ns). Once you have seen the output, press Ctrl + C to return to the console.
Run OPC UA PubSub application in Performance Evaluation mode
After establishing the time synchronization successfully in both the nodes, we will run the OPC UA PubSub application to generate the performance measurement log file.
./bin/examples/pubsub_TSN_loopback -interface <IFACE> -enableBlockingSocket -pubMacAddress opc.eth://<MAC-of-node1>:8.3 -subMacAddress opc.eth://<MAC-of-node2>:8.3
./bin/examples/pubsub_TSN_publisher -interface <IFACE> -enableBlockingSocket -enableLatencyCsvLog -pubMacAddress opc.eth://<MAC-of-node2>:8.3 -subMacAddress opc.eth://<MAC-of-node1>:8.3
./bin/examples/pubsub_TSN_loopback -interface enp2s0 -enableBlockingSocket -pubMacAddress opc.eth://00-d0-93-46-b2-ed:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-fc:8.3
./bin/examples/pubsub_TSN_publisher -interface enp2s0 -enableBlockingSocket -enableLatencyCsvLog -pubMacAddress opc.eth://00-d0-93-46-b2-fc:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-ed:8.3
The -enableLatencyCsvLog flag computes the round trip time (RTT) of a counter variable sent from one node to another and looped back (more information on this in the next section below). By default, this application stops after collecting 1 million packets. You can terminate it in-between by pressing Ctrl + C. The log file (latencyT1toT8.csv) will be generated in the pubsub_TSN_publisher application build folder (only after the application is terminated), containing the RTT, missed counters, and repeated counters.
The first column in the latencyT1toT8.csv shows the RTT value, the second column shows the missed counters and the third column shows the repeated counters. The application runs successfully with RTT of 1ms, and 0 missed and repeated counters, as shown in the image above.
The application in node 1 (node where pubsub_TSN_publisher is running) is designed to encode and publish a packet containing a counter variable to node 2 every 250 microseconds. The counter variable is incremented at the start of each new 250 microseconds cycle. Node 2 (node where pubsub_TSN_loopback is running) is designed to decode the received packet and loop the counter variable back to node 1. The application at node 1 receives the counter variable and computes the round-trip time: T8 – T1 (time taken to publish and receive back the same counter variable). This information is captured in the latency CSV files. The CSV files also contain information on missed counters and duplicate counters – this can happen due to multi-threading issues or packets being lost during transmit/receive.
The below figure shows the flow of information and the timestamps at different stages (T1 to T8):
- T1 – Timestamp at which the counter variable was incremented (and handed over to the publisher)
- T2 – Ethernet packet outgoing timestamp at kernel space
- T3 – Ethernet packet incoming timestamp at kernel space
- T4 – Timestamp at which the counter variable is seen by the application on node 2
- T5 – Timestamp at which the counter variable was handed over to the publisher
- T6 – Ethernet loopback packet outgoing timestamp at kernel space
- T7 – Ethernet loopback packet incoming timestamp at kernel space
- T8 – Timestamp at which the counter variable is seen by the application on node 1 after the round trip
To setup munin for monitoring real-time performance
Now that we have seen the successful working of the PubSub applications, we will now setup munin for monitoring 24*7 real-time performance.
In this section, we will setup the munin master and slave nodes. We recommend running the munin master in a third node. If you are using a high speed processor, you can use one of the two nodes (say, node1) as a munin master as shown below.
- Setup a Munin Master Webserver in one of the nodes (say, node 1)
- Install Apache2 server and dependency packages
- Check the fcgid module status After installing the dependency packages, the fcgid module shall be enabled. Double-check using the command
- Install Munin Master Webserver
- Configure the Munin Master Webserver by modifying /etc/munin/munin.conf file
- Uncomment the following lines in /etc/munin/munin.conf and save it
- Open the apache2 configuration file in the path /etc/munin/ and perform the 2 changes mentioned below. Note: Apache2 configuration filename shall vary based on the OS that is being used. It shall be either apache2.conf or apache24.conf.
CHANGE 1: Ensure that the below lines are available in the apache configuration file. If not, add these changes to the file.
# Require local
# Options None
Require all granted
Options FollowSymLinks SymLinksIfOwnerMatch
# Require local
Require all granted
Options FollowSymLinks SymLinksIfOwnerMatch
CHANGE 2: Check if the following lines are present in the apache configuration file
If the above lines are present, replace with the below lines. If not present, ignore this change
Require all granted
Options FollowSymLinks SymLinksIfOwnerMatch
- Activate Munin master webserver
apt-get install -y apache2 libcgi-fast-perl libapache2-mod-fcgid
/usr/sbin/apachectl -M | grep -i cgi
The above command shall return “fcgid_module (shared)”. If the output is blank, the fcgid module is not enabled. Enable the fcgid module using the command
apt-get install -y munin
- Verify Munin Master Webserver
Example: If the IP Address of the system in which the Munin Master Webserver is installed is 172.17.8.11, access the Munin Master Webserver at http://172.17.8.11/munin/
- Setup both the nodes as Munin Client to monitor its RTT latency, PTP and PHC2SYS values continuously.
- Follow the below steps in both the nodes to setup as a Munin client.
- Install munin node
- Configure Munin Node
- To make the Munin node communicate with Munin Master Webserver, modify the /etc/munin/munin-node.conf file.
- Add your Munin server IP address after the line mentioned below, in the /etc/munin/munin-node.conf file and save it.
Note that the IP address is in regex format, so assuming that the Munin server IP address is 172.17.8.11, the line should be
- Restart the Munin Node
apt-get install munin-node -y
Munin server must be configured with the installed Munin nodes to its configuration to reflect in the browser.
To add the Munin Nodes to Munin Master Webserver, follow the below steps.
- Configure Munin Master Configuration
Open the file /etc/munin/munin.conf in Munin Master and insert the following lines at the end of the file and save the changes:
Note: The Node name specified within square brackets shall be the name of the Munin node that is displayed in the Munin Master Webserver.
Note: Wait for at least 5 minutes to see the newly installed nodes in Munin Master Webserver. This wait is because the Munin Master is configured in such a way that it executes all its scripts and collects the values from the Munin nodes after every 5 minutes. So, any configuration change made to Munin nodes and Munin Master shall reflect only after 5 minutes.
./munin_setup.sh -d <open62541_build_directory_from_home_folder> -i <IFACE>
./munin_setup.sh -d ~/open62541/build -i enp2s0
To monitor real-time performance of PubSub applications
./bin/examples/pubsub_TSN_loopback -interface <IFACE> -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://<MAC-of-node1>:8.3 -subMacAddress opc.eth://<MAC-of-node2>:8.3 > application_log.txt 2>&1 &
./bin/examples/pubsub_TSN_publisher -interface <IFACE> -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://<MAC-of-node2>:8.3 -subMacAddress opc.eth://<MAC-of-node1>:8.3 > application_log.txt 2>&1 &
./bin/examples/pubsub_TSN_loopback -interface enp2s0 -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://00-d0-93-46-b2-ed:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-fc:8.3 > application_log.txt 2>&1 &
./bin/examples/pubsub_TSN_publisher -interface enp2s0 -enableBlockingSocket -enableLongRunMeasurements -pubMacAddress opc.eth://00-d0-93-46-b2-fc:8.3 -subMacAddress opc.eth://00-d0-93-46-b2-ed:8.3 > application_log.txt 2>&1 &
The -enableLongRunMeasurements flag computes the Round Trip Time (RTT) of a counter variable sent from one node to another and looped back. The generated CSV contains the current time, RTT, missed counters, and repeated counters.
For every 5 minutes the munin scripts are used to evalute the performance from the generated latency CSV and the data is updated in the munin graph.
To view the munin graph, type in the ip address of your munin server in the format <ip-of-munin-server>/munin where you will find both the nodes listed as munin slaves.
Recommended step - Use exit command to terminate the session from the nodes.
If you reboot the system, you have to run the steps under the section “Run PTP and PHC2SYS” and then run the OPC UA PubSub application in both the nodes.
Optional step 1: If you wish to make your TSN environment persistent after a reboot, follow the below steps:
echo ./<FILE_PATH>/application_dependencies.sh -i <IFACE> -m >> /etc/rc.local
chmod +x /etc/rc.local
Optional step 2: Use NTP in parallel
- 1. Run NTP on PTP master
- 2. PHC2SYS should take the systemtime and update PTP hardware clock
- 3. The above action will create time jumps. There will be filtering options needed so that the PTP time will not have any time jumps
Optional step 3: Configure other VLAN Id for IFACE
iface enp2s0.1 inet static
iface enp2s0.2 inet static
iface enp2s0.3 inet static
iface enp2s0.4 inet static
iface enp2s0.5 inet static
iface enp2s0.6 inet static
iface enp2s0.7 inet static
for i in `seq 1 8`; do for j in `seq 0 7`;do sudo ip link set $IFACE.$i type vlan egress $j:$j ; done; done
for i in `seq 1 8`; do for j in `seq 0 7`;do sudo ip link set $IFACE.$i type vlan ingress $j:$j ; done; done