To develop a robust system on top of an RTOS, it is crucial to have a good understanding of how and when context switches happen in your system.
Visualizations can be invaluable to spot bugs or unintended behaviors.
In this article, I will outline one way of building a simple tracing solution from scratch.
In the process, we’ll learn some techniques which we can apply to debugging and analyzing embedded systems in general.
We have a simple 4-step plan:
Use FreeRTOS hooks to collect tracing data into a big buffer.
Use the debugger to get data from the embedded device to our PC.
Use a python script to convert the raw data into a .json trace.
Visualize the .json trace with spall.
I will give rough descriptions of how to implement each step. The goal isn’t to provide a readymade solution, but rather to give you the necessary resources to learn how to implement a system like this yourself.
There are well-established tools for tracing (e.g. Percepio or Segger SystemView), which are definitely worth considering.
Nonetheless, I believe a hand-written solution as presented here still is valuable.
For one, a hand-written solution can be tailored to fit exactly your needs and can be made to work in severely constrained environments (e.g. when you have limited available flash space for extra code).
However, to me the biggest advantage is educational. By making a simple tracer by hand you will be much better prepared to understand how to evaluate, integrate and use the more powerful off-the-shelf tools. Additionally, the techniques described here are not limited to specifically context switches on FreeRTOS. You can use the same techniques for many other problems where you want to gain a better understanding of your embedded system.
FreeRTOS allows you to insert hooks which get called in response to OS events, for example when tasks get context-switched in/out. It looks something like this:
// FreeRTOSConfig.h
#define traceTASK_SWITCHED_IN() writeTraceData(1, pxCurrentTCB->pcTaskName)
#define traceTASK_SWITCHED_OUT() writeTraceData(0, pxCurrentTCB->pcTaskName)
void writeTraceData(int switched_in, char *task_name);
For more details, see the FreeRTOS documentation: Trace Hook Macros – FreeRTOS™
From inside myFunction(), we can store the data we receive into a ring buffer. To give you an idea, I use a structure which looks like this:
// trace.c
struct TraceData {
struct {
char name[15]; // (We could also store task indices, and have a table mapping task indices to names)
uint8_t switched_in; // 1=switched_in, 0=switched_out.
uint32_t timestamp;
} events[2048];
int event_index; // Tracks where in ring buffer we are.
};
static struct TraceData TRACE_DATA;
void writeTraceData(int switched_in, char *task_name) {
int i = TRACE_DATA.event_index++ % ARRAYSIZE(TRACE_DATA.events);
strncpy(TRACE_DATA.events[i].name, task_name, 15);
TRACE_DATA.events[i].switched_in = switched_in;
TRACE_DATA.events[i].timestamp = read_timestamp();
}
The one missing piece is an implementation for read_timestamp(). On ARMv7 embedded systems, typically that means setting up some vendor specific timer peripheral to generate a high-resolution counter (You probably want to avoid xTaskGetTickCount, since this likely has low resolution compared to task switches, if task switches are happening in response to semaphores/mutex/etc being signaled).
Now for the real trick! How do we get TRACE_DATA from the embedded device to our laptop?
We could write some code to write it out to a UART or some other peripheral, and then try to capture it on our laptop. But maybe your UART already is being used for logging, so you need to make sure you don’t get log messages in the middle of your trace.
Instead, we can use our debugger to quickly extract the data, with the following gdb command:
(gdb) dump binary value trace.bin TRACE_DATA
This writes the raw content of the TRACE_DATA global variable to a file on our laptop.
If you are using gdb via some GUI, most GUIs have a way of directly sending gdb commands. For example, in VSCode there is a “Debug console” tab where you can enter this command.
We’ll be using Spall to visualize our data. Spall is a lightweight trace viewer GUI. It is intended for performance analysis, but can be used to visualize any data of the form “event A lasted from time T0 to T1”.
Spall supports both a binary format and a json format. The json format is documented here.
This is the same format that is also used by e.g. Perfetto.
I use a simple python script to convert the data in trace.bin to .json. Here, I use struct.unpack to convert binary data to python strings/integers, and then json.dump to build a json string.
// convert.py
import json
import struct
result = []
raw = open("trace.bin", "rb").read()
N = 2048
event_index, = struct.unpack_from("<I", raw, N*20) # (each event is 20 bytes)
event_write_head = event_index % N
event_count = min(event_index, N)
# Iterate ring buffer in the same order as it was written to.
for r in [range(event_write_head, event_count), range(0, event_write_head)]:
for index in r:
name, switched_in, timestamp_raw = struct.unpack_from("<15sBI", raw, index*20)
name = name.strip(b"\x00").decode("utf8")
timestamp = timestamp_raw # TODO convert from MCU timestamp to µs.
result.append({"cat":"function","name":name,"ph":('B' if switched_in==0 else 'E'),"pid":0,"tid":0,"ts":timestamp})
json.dump(result, open("trace.json", "w"))
Now we can open our .json file in the spall viewer:
From bare metal all the way up to Linux userspace, Morten has worked with many different tools and technologies. He has a keen interest in tools for program analysis, whether it be debuggers, tracers or network analyzers. Morten believes that one of the keys to deliver robust and efficient embedded software is a deep understanding of how systems work under the hood. As such, he is always looking for new ways of understanding the systems he works on.
