Overcoming the Challenges of Cross-Compiler Testing
SuperTest Strengthening Quality Control for Kyoto Microcomputer’s LLVM/Clang Based Compiler Products
From automotive to industrial control and consumer electronics, software developers rely on compilers, debuggers, and real-time operating systems to build robust applications. However, as embedded systems become more complex, so do the challenges of maintaining consistent compiler performance across different architectures. Compiler validation and rigorous testing are essential to delivering software that meets the stringent requirements of embedded applications.
Kyoto Microcomputer (KMC), a Japanese provider of embedded software development tools, has more than 20 years of experience in the design and supply of high-performance development tools for embedded microcontrollers in the automotive, industrial control, communications, computer and consumer electronics industries. The company is particularly renowned for the provision of innovative debugging tools and real-time operating systems. To facilitate the integration of these debugging tools and to increase cross-platform compatibility, KMC has also developed GCC-based products, but this time using the LLVM compiler infrastructure and toolchain with Clang as the C/C++ front end. This requires testing the LLVM/Clang compiler, the NetBSD-based C library (libc), and the C++ libraries that come with LLVM/Clang (libc++, libc++abi, libunwind). However, this development environment presents several challenges in testing the compiler and libraries. KMC selected Solid Sands’ SuperTest compiler test and validation suite to address these challenges.
The first challenge is that KMC’s cross-compiler setup prevents the use of LLVM/Clang’s standard test suite. Since the compiler is built on Ubuntu but generates executables that run only on Windows (a Canadian cross-build), the standard LLVM/Clang tests cannot be executed in this environment. The second challenge is that the NetBSD-based C library (libc) uses a test framework that depends on NetBSD, so tests cannot be run on the “bare metal” (AArch64, ARM, RISC-V32/64) target environments that KMC focuses on. In addition, the C++ libraries bundled with LLVM/Clang (libc++, libc++abi) and libunwind cannot be tested if the compiler host and target are different. For these reasons, KMC was in need of a large, reliable, standard C/C++ test suite for bare-metal target cross-compilers that could run on Windows in a compiler-independent manner. SuperTest provides just that.
SuperTest was up and running in less than two weeks, but the KMC development team spent over a year isolating error causes across the test environment, compiler, libc/libc++, test I/O stub code, and execution environment (simulator). As a result of this extensive process, around 10 improvements in C and 50 in C++ are still in progress. SuperTest has become an integral part of KMC’s quality control process. The development team runs it not only after every release build but also after each major change or when a series of smaller updates have accumulated, ensuring continuous validation throughout the development cycle. KMC’s LLVM/Clang based compiler solutions are already used by many of the company’s customers and SuperTest helps them achieve the continuous quality control they need.
“Quality control is critical in embedded software development because most customers require guaranteed support for several years to 10 years, and compiler binaries
typically cannot be changed during the life of the project. This is where SuperTest really comes into its own,” said Toshihiro Wakatsuki, Toolchain Development Leader at KMC.
With quality control in place, the next task was to leverage SuperTest’s capabilities to further strengthen the company’s quality assurance program. KMC’s integration of SuperTest into their test and validation toolchain was supported by Solid Sands’ distributor in Japan, Fujisetsubi Co., Ltd. and the Solid Sands customer support team in Amsterdam, which KMC found to be helpful and responsive.
The benefits of using SuperTest to complement or replace standard LLVM/Clang and GCC test suites (traceability between individual tests and the language specification, true compliance testing rather than regression testing, and reporting in an easy-to-read format) are widely recognized, and Solid Sands has given numerous presentations at
international conferences and webinars.
