.png){kind=small}
Continuing blog post series around Xen and IOMMU enabling in coreboot we are reaching a point in which some features seem to work correctly on top of recent patch series in firmware.
What we can do at this point is PCI passthrough to guest VMs. Previously trying that on Xen caused problems:
- random hangs
- firmware cause Linux kernel booting issues (hang during boot)
- IOMMU disabled - unable to use PCI passthrough
Now we can see something like that in dom0:
|
|
Of course, after above operation, we can’t access enp2s0 in dom0. Having
the ability to set pass through we can think about creating pfSense HVM and
having isolation between various roles on our PC Engines apu2 router.
What are the pros of that solution:
- price - this is DIY solution where you just pay price of apu2 and spent some time with setup, of course, you can also pay for that to companies like 3mdeb, what should be still cheaper than other commercial solutions - this makes it attractive to SOHO
- scalability - you can decide how much resources of your router you want to give to the firewall; the remaining pool can be used for other purposes this saves you a couple of cents on the energy bill
- security - even if attacker get access to pfSense (very unlikely), escaping VM and gaining full control and persistence on hardware is not possible without serious Xen bug, on the other hand, bugs in on the other VMs (e.g. network storage, web application, 3rd party software) cannot be leveraged to gain control over the router
- virtual machine - VMs by itself have a bunch of advantages, somewhere mentioned above, but other are easier migration, lower cost to introduce in existing network
Required components
- PC Engines apu2c4
pxe-server- or other means of booting Debian based Dom0 with Xen 4.8 and Linux 4.14.59 (or any other modern kernel which has correct support enabled as in this kernel config)- 2 connected Ethernet ports
- some storage (min 10GB)
Prepare Xen
I’m using apic=verbose,debug iommu=verbose,debug for better visibility of Xen
configuration. More to that we need some preparation in Dom0:
Storage
|
|
PCI passthrough
|
|
After above commands 02:00.0 should be listed in xl pci-assignable-list
output:
|
|
xl allows assigning devices even if IOMMU is not present, but it will issue an
error during VM creation.
Xen pfsense.cfg
First let’s create
|
|
Then you can create VM:
|
|
Install pfSense
After running VM you can attach to console:
|
|
You should see pfSense installer boot log:
|
|
Unfortunately, pfSense had problem getting DHCP offer and didn’t configure IP address - we tried to figure out what is wrong but my BSD-fu is low. We also checked static IP configuration, but there is no result either. This leads us to ask on the forum.
Xen debian.cfg
|
|
Of course, you have to replace apu2_ip_addr with correct IP. After xl create debian.cfg you can run VNC (tightvnc worked for me) and proceed with the
installation.
PCI passthrough in Debian
Below screenshot show device 02:00.0, which is apu2 middle NIC,
passed-through to VM.
PCI passthrough on Debian worked without any issue DHCP offer was received correctly and I could proceed with performance checks.
Speedtest
Simplest possible test is comparison of throughput between eth0 and eth1. The first is connected directly to our company switch and the second connects pfSense HVM using PCI passthrough.
I used speedtest-cli v2.0.2.
Results for apu2 Dom0:
|
|
Results for Debian HVM with NIC PCI passthrough:
iperf
Below results are for very simple LAN connection apu3 -> switch -> apu2:
|
|
Unfortunately, our switch is probably not well suited for testing 1GbE. Those tests should be repeated with directly connected ports/devices.
Results for Debian HVM with NIC PCI passthrough:
As you can see there is no difference between results, based on that we can conclude that PCI passthrough works and there is no overhead when using IOMMU.
Below log show results from Debian PV and prove how virtualized drivers lead to performance overhead.
|
|
Possible problems
xen-pciback not loaded
|
|
Solution:
|
|
PCI device not assignable
|
|
Assign PCI device using xl pci-assignable-add.
No IOMMU
|
|
This error means you don’t have IOMMU correctly enabled. For AMD platforms xl dmesg contain:
|
|
Lack of block backend
xen-blkback should be loaded or compiled in otherwise blow error pop-up.
|
|
Crash after couple tries
After a couple tries of creating pfSense VM I faced below error:
|
|
Solution: recompile kernel with BLK_DEV_LOOP
Read-only not supported
|
|
Solution: change pfsense.cfg by adding rw to img file.
References
- Install pfSense 2.1 RC0 amd64 on Xen 4.3 as PV HVM
- Virtual firewall
- xl.cfg manual for Xen 4.8
- Unanswered question about DMA attacks
- Google blog post about fuzzying PCIe
Summary
I hope this post was useful for you. Please feel free to share your opinion and if you think there is value, then share with friends.
We plan to present above results during OSFC 2018 feel free to catch us there and ask questions.
We believe there are still many devices with VT-d or AMD-Vi advertised in
specs, but not enabled because of buggy or not-fully-featured firmware. We are
always open to support vendors who want to boot hardware by extending and
improving their firmware. If you are user or vendor struggling with hardware
which cannot be fully utilized because of firmware, feel free to contact us
contact<at>3mdeb<dot>com.
Founder and Embedded Systems Consultant at 3mdeb as well as freelance CTO of Vitro Technology and CEO of LPN Plant. Passionate about building firmware that enables advanced hardware features in modern products. Dedicated to customers that treat embedded software security and upgradeability as forethought. Open source firmware evangelist interested in platform security and trusted computing. In favor of fixed price projects with a clear definition of success.