About your next automotive-friendly prototype
You have cool project idea and you want to make a prototype more compatible with the automotive world? Here are some design choices for you…
Of course, some old-school hard real-time C code running bare metal on a microcontroller would always rekindle the heart of automotive engineers, but for quick prototyping it is definitely not the most convenient target. On the other hand, using a Linux-based target is unfortunately not real-time, but enables to leverage many open-source libraries.
This article uses inastitch as sample project to explain how each library was used in a practical example.
Inastitch is a prototype for a digital rear-view mirror, by stitching 3 live video streams together into a single wider stream. Each camera is independent and connected to a network.
Logging with DLT
While LTTng is a famous choice for logging in the open-source world, DLT is more automotive-oriented. DLT was designed by Bosch as an Autosar standard, see this lengthy specs PDF.
One design goal of using DLT is to have a trace which contains everything required to debug a system made of multiple ECUs. When a explanation will be needed for a specific behavior, the idea is to provide an expert with a single DLT trace file, which should contain everything needed, instead of asking always more traces.
In inastitch, following the DLT terminology, each camera is an ECU and each binary is a different APP. Contexts (CTX) follow this pattern:
- APP:MAIN for general information such as binary version (i.e., git tag) and git hash to track the corresponding source code. Start-up parameters are also logged here.
- APP:FRAM for summarized information about video frames. Single frame should not be traced because it is too high frequency and would overload the trace budget.
- SYS:JOUR for syslog. The Linux system journal of the whole target is copied to DLT. It will help checking for side-effects coming from the Linux system.
Other interesting information to trace (which are not implemented in inastitch):
- Overall and per-process CPU load and memory allocation. It will help to debug infinite loop and memory leak.
DLT open-source implementation is: https://github.com/GENIVI/dlt-daemon
License: Mozilla Public License (MPL 2.0).
Check DLT packets with WireShark
A packet dissector for DLT was recently added to WireShark in version 3.2.0.
Note: dlt-daemon on each ECU will keep the trace in a local buffer until a dlt-client connects. There will not be any DLT communication to decode if no DLT viewer (for instance) is reading the trace. Likewise, the client can select different levels of tracing and only those traces will be sent over the network.
DLT and file transfer
Inastitch has a calibration feature where debug images are generated. Instead of writing them to /tmp and expecting them to be fetched over SSH, it is more practical to dump them in the DLT trace.
After enabling the FileTransfer plug-in in DLT viewer, files can be extracted from the trace itself.
DLT and coredumps
Again with the aim of having one single trace file for everything, it is convenient to embed compressed coredumps in the DLT as well.
See the DLT built parameter WITH_DLT_COREDUMPHANDLERand this wiki page.
TODO: enable DLT coredump handler in inastitch.
DLT injection
For more complex debugging, applications can be extended with debug modes with more tracing (over the trace budget) and more checks (i.e., runtime asserts). Those debug features would usually be enabled by restarting the application with special parameters like --verbose.
For a more automotive-friendly solution, I would rather suggest to enable those debug modes via a DLT injection. DLT communications do not work only one way (i.e., from dlt-daemon to dlt-client). Requests from the client (aka injections) can be forwarded to the application.
The advantage of this approach are:
- No SSH access to the target is required
- No need to restart the application to enable or disable the debug mode
- Everything is logged in the trace if someone analyses it later
TODO: add some DLT injections in inastitch.
Remote control with SOME/IP
SOME/IP is a RPC framework created at BMW. It is also an Autosar standard supported by Bosch. See this specs PDF.
SOME/IP can work statically by defining all methods and events on all involved nodes, but this architecture is difficult to maintain and update.
One advantage of this framework is Service Discovery (aka SD). It makes it easier to design an interface that can be provided by applications from different suppliers which may run in different configurations.
In inastitch, SOME/IP was used to control camera settings. All three cameras subscribe to a remote event published by the stitcher board. When the requested camera setting is changed by the stitcher application, SOME/IP takes care of notifying all the subscribers.
SOME/IP open-source implementation is: https://github.com/GENIVI/vsomeip
License: Mozilla Public License (MPL 2.0).
Trace SOME/IP with DLT
By enabling the tracing and the dlt options in vsomeip.json, it is possible to log every SOME/IP communication in DLT in the TC context. Unfortunately, the open-source version of DLT viewer does not come with a SOME/IP decoder plug-in.
Check SOME/IP packets with WireShark
A packet dissector for SOME/IP was recently added to WireShark in version 3.2.0.
Unlike DLT viewer, WireShark will decode SOME/IP requests and events.
Video stream with AVTP
AVB is a standard created for the professional audio/video world. It gains much traction in automotive now. AVTP is the transport protocol for AVB streams.
Because AVB networks are used for communications which are neither video nor audio, new features were added and is referred as TSN. It stands for Time-Sensitive Network.
AVB/TSN networks are extensions of the standard Ethernet network added to add guarantees on the delivery of stream packets from one or more Talkers to one or more Listeners.
In inastitch, video frames of all three cameras are sent in a synchronized manner so that they can be stitched without buffering (i.e., without increased latency).
Check AVTP packets with WireShark
Note: AVTP can be embedded onto Raw Ethernet. To save on implementation effort for inastitch, the streams are sent over UDP and IP.
This is an officially specified alternative.
Clock synchronization with gPTP
In the AVB world, clock synchronization is critical to deliver audio/video streams. gPTP is a special profile for PTP, the Precision Time Protocol.
In inastitch, PTP synchronizes the Linux system clocks among the camera boards. This clock is then used as reference for frame alignment.
Check PTP packets with WireShark
A packet dissector for PTP is available in WireShark since version 1.0.0.
Note: PTP is configured in inastitch to run over UDP, but it can also run in Raw Ethernet.
Other considerations
Update with packages
I know, sending zipped and statically built binaries by email is the best 😅
In inastitch, both camera and stitcher applications are delivered as Debian packages. Generating such package is made easy with CMake’s CPack extension. This solution integrates well with the Raspberry PI OS.
If the OS is built from scratch (e.g., yocto linux recipes), another more lightweight choice is to use OPKG as package manager.
Open-source licenses
Open-source software is great, but some licenses will force you to release the source, and sometimes the building tools of your ECU firmware.
In inastitch, library dependencies are reduced to simple libraries with permissive licenses.
For example:
libboostwith permissive Boost licenselibturbojpegwith BSD licenselibdltandlibvsomeipwith MPL license- OpenCV with BSD license
Better code analysis with clang
Many projects are built with gcc.
Even if the final binary is always built with gcc, keeping the option to build with clang as well will bring the possibility to use Google Sanitizers such as MemorySanitizer (aka MSAN).
Systemd and watchdog
Systemd has options to limit the memory usage or other resources of a service. It is also possible to have a watchdog managed by systemd.
Build static libraries
Inastitch is built with most of the its dependencies as static library. Hence the stitcher binary can run unmodified on both Raspberry Pi and Nvidia Jetson Nano.
