.png){kind=small}
Intro
In the 2018th OSFC, we were presented AMD IOMMU enabling for PC Engines apuX (GX-412TC) platforms. You can watch the presentation video here Our hypervisor of choice was Xen and we used it to verify the PCI pass-through feature. Unfortunately, the booting process was not exactly stable and the platform from time to time hung on the same log:
|
|
Always the same place in code, it seems to start printing (XEN) Brought up 4 CPUs, so suspicious code is probably right after this log.
We started to write that post quite long ago, but because recent Xen 4.16.1 release we decide to get back to the problem and see what is the current state.
Debugging environment considerations
Because of that, I decided to debug Xen, but first I had to get through the compilation and deployment procedure. In general, I saw a couple of options for Xen compilation:
- Debian package modification - get sources through
apt source xenand continue with the Debian way of building packages - this can be done either on a host, in rootfs, or Docker container - directly from Xen source tree using
make debball- this can be done either on the host or in the container - directly from Xen source tree using
make build-xenand installation withsudo make install-xen
Internet is not straightforward about the best method (Xen documentation). The third option was the only option for me to build Xen. Here is the step-by-step instruction on how to build and install Xen from the source:
-
Install build essential:
1sudo apt-get install build-essential -
Enable source code in the
software-properties-gtk -
Install the python config package
1pip3 install python-config -
Clone Xen source
1git clone git@github.com:xen-project/xen.git -
Configure project
1cd xen && ./configure -
Build Xen
1make build-xen -
Install Xen
1sudo make install-xen
More to that all of the methods can be applied through frameworks. Debian rootfs can be built using isar and there is always a way to narrow everything to OpenEmbedded/Yocto meta layer which should build only what we need. The last option is good for production, but development may be hard in a limited environment that Yocto produces by default.
Xen dockerized building environment
I chose the second option for that purpose. I have prepared a Docker container with all required packages (3mdeb/xen-docker):
|
|
I cloned the Xen source and I run the docker container.
|
|
Then I built the Xen package:
|
|
Note:
--enable-systemdrequires alibsystemd-devpackage to be installed in the container.
Build result will be placed in $XEN_SRC_DIR/dist as
xen-upstream-<version>.deb. For Debian-based systems, it is easy to install
it with dpkg or apt. The package contains all necessary components for
the host OS along with the Xen kernel.
Note that the host OS still will require Dom0 Kernel for Xen
To update the pxe-server with new Xen image, install the
xen-upstream-<version>.deb in rootfs which hosts the VMs and copy the Xen
kernel from /boot/xen-<version>.gz to tftpboot/httpboot directory (gunzip the
kernel first).
After whole this effort, I was able to boot my freshly built Xen on apu2c4 with Debian host and Debian guest OS. Now that I have prepared the developing procedure I can start narrowing down all the issues.
Possible use cases
The ability to build Xen with custom changes opens the door to creating custom solutions and products. In addition, it provides an opportunity to workaround well-known problems and helps fix them in the mainline in the long term. A adequate example of this approach is adding support for the Trenchboot in the Xen. As described in the blog post, our engineers added workaround support in Xen for Secure Startup via SKINIT. With the help of Andrew Cooper (Xen maintainer), after a few code changes, the patch was merged into the main.
As a long-time maintainer of PC Engines platforms, 3mdeb sees the opportunity to create hardened router products that help maintain the security of the corporate network. In addition, custom Xen gives ability to verify various firmware features e.g. IOMMU configuration and many others which are verified by Xen kernel while booting.
Furthermore, once a custom Xen design has been created, the stability of the solution must be ensured. The easiest way to achieve this is to create a CI/CD that builds and tests the solution. Our test automation team has historically provided Xen tests for our PC Engines platform in every release. This is now outdated, as the tests were run using Xen 4.8 (the current version of Xen at the date of this blog post is 4.17). We plan to improve the transparent validation infrastructure for Dasharo-compatible products soon.
Summary
If you think we can help in improving the security of your firmware or you
looking for someone who can boost your product by leveraging advanced features
of the used hardware platform, feel free to book a call with us
or drop us an email to contact<at>3mdeb<dot>com. If you are interested in
similar content feel free to sign up for our newsletter.
