How to boot Xen over PXE and NFS on PC Engines apu2
From time to time we face requests to correctly enable support for various Xen
features on PC Engines apu2 platform. Doing that requires firmware
modification, which 3mdeb is responsible for.
Xen have very interesting requirements from firmware development perspective.
Modern x86 have a bunch of features that support virtualization in hardware.
Those features were described in Xen FAQ.
It happens that most requesting were IOMMU and SR-IOV. First, give the ability
to dedicate PCI device to given VM and second enables so-called Virtual
Functions, what means on a physical device (e.g. Ethernet NIC) can be
represented by many PCI devices. Connecting IOMMU with SR-IOV give the ability
for hardware-assisted sharing of one device between many VMs.
All those features are very nice and there is work spread on various forums,
which didn’t get its way to mainline yet. Starting with this blog post we want
to change that.
To start any work in that area we need a reliable setup. I had a plan to build
something pretty simple using our automated testing infrastructure.
Unfortunately, this has to wait a little bit since when I started this work I had
to play with a different configuration.
If you don’t have PXE, DHCP (if needed) and NFS set up I recommend to read
above blog post or just use pxe-server
and dhcp-server.
Xen installation in Debian stable
I assume you have PXE+NFS boot of our Debian stable. To netboot simply enter
iPXE:
1
2
3
iPXE> dhcp net0
Configuring (net0 00:0d:b9:43:3f:bc).................. ok
iPXE> chain http://192.168.42.1:8000/menu.ipxe
After boot, you can log in with presented credentials [root:debian]:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Debian GNU/Linux 9 apu2 ttyS0 [root:debian]
apu2 login: root
Password:
Last login: Mon Jul 16 23:45:56 UTC 2018 on ttyS0
Linux apu2 4.14.50 #13 SMP Mon Jun 18 00:36:23 CEST 2018 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@apu2:~#
Xen installation in Debian is quite easy and there is community website
describing the process, but to quickly dive in:
I took a break from Xen debugging and found that after upgrading kernel and
rootfs I’m getting below error message:
1
2
3
4
root@apu2:~# xl dmesg
xencall: error: Could not obtain handle on privileged command interface: No such file or directory
libxl: error: libxl.c:108:libxl_ctx_alloc: cannot open libxc handle: No such file or directory
cannot init xl context
I’m not a Xen developer and it looked pretty cryptic to me. It happens that
xen.service also fails to run:
1
2
3
4
5
6
7
8
9
10
11
12
● xen.service - LSB: Xen daemons
Loaded: loaded (/etc/init.d/xen; generated; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2018-05-02 11:20:00 UTC; 33s ago
Docs: man:systemd-sysv-generator(8)
Process: 392 ExecStart=/etc/init.d/xen start (code=exited, status=1/FAILURE)
May 02 11:20:00 apu2 systemd[1]: Starting LSB: Xen daemons...
May 02 11:20:00 apu2 xen[392]: Starting Xen daemons: xenfs failed!
May 02 11:20:00 apu2 systemd[1]: xen.service: Control process exited, code=exite
May 02 11:20:00 apu2 systemd[1]: Failed to start LSB: Xen daemons.
May 02 11:20:00 apu2 systemd[1]: xen.service: Unit entered failed state.
May 02 11:20:00 apu2 systemd[1]: xen.service: Failed with result 'exit-code'.
It happen that during upgrading of my rootfs I forget to install all required
packages to Xen rootfs directory. So, now you should not face this problem when
using pxe-server, but if you see something similar please make sure you have
all modules correctly loaded or compiled in. You can check my working kernel config
Xen boot log
Below boot log analysis was performed on v4.6.9 release candidate.
The thing that we are concerned about and want to fix is
1
(XEN) AMD-Vi: IOMMU not found!
There are some patches pending to enable IOMMU. Of course, enabling this
features open new universe with various advanced virtualization features which
we hope to discuss in further blog posts.
Trying Xen boot params
I tried to use iommu=on amd_iommu=on which doesn’t change anything with
firmware not-IOMMU capable.
Summary
In further posts, I would like to get through IOMMU enabling by leveraging great
community work from Kyosti and Timothy. Also, I would like to exercise and prove
various virtualization features of PC Engines apu2. If you are interested in
commercial enablement of advanced SoC features feel free to let us know at
contact@3mdeb.com. Also feel free to contribute to pxe-server mini-project
as well as comment below.
Founder and Embedded Systems Consultant at 3mdeb as well as freelance CTO of Vitro Technology and CEO of LPN Plant. Passionate about building firmware that enables advanced hardware features in modern products. Dedicated to customers that treat embedded software security and upgradeability as forethought. Open source firmware evangelist interested in platform security and trusted computing. In favor of fixed price projects with a clear definition of success.
.png){kind=small}
