.png){kind=small}
Continuing blog post series around Xen and IOMMU enabling in coreboot we are reaching a point in which some features seem to work correctly on top of recent patch series in firmware.
What we can do at this point is PCI passthrough to guest VMs. Previously trying that on Xen caused problems:
- random hangs
- firmware cause Linux kernel booting issues (hang during boot)
- IOMMU disabled - unable to use PCI passthrough
Now we can see something like that in dom0:
1 2 3 4 5 6 7 |
modprobe xen-pciback root@apu2:~# xl pci-assignable-add 02:00.0 [ 136.778839] igb 0000:02:00.0: removed PHC on enp2s0 [ 136.887658] pciback 0000:02:00.0: seizing device [ 136.888115] Already setup the GSI :32 root@apu2:~# xl pci-assignable-list 0000:02:00.0 |
Of course, after above operation, we can’t access enp2s0 in dom0. Having
the ability to set pass through we can think about creating pfSense HVM and
having isolation between various roles on our PC Engines apu2 router.
What are the pros of that solution:
- price - this is DIY solution where you just pay price of apu2 and spent some time with setup, of course, you can also pay for that to companies like 3mdeb, what should be still cheaper than other commercial solutions - this makes it attractive to SOHO
- scalability - you can decide how much resources of your router you want to give to the firewall; the remaining pool can be used for other purposes this saves you a couple of cents on the energy bill
- security - even if attacker get access to pfSense (very unlikely), escaping VM and gaining full control and persistence on hardware is not possible without serious Xen bug, on the other hand, bugs in on the other VMs (e.g. network storage, web application, 3rd party software) cannot be leveraged to gain control over the router
- virtual machine - VMs by itself have a bunch of advantages, somewhere mentioned above, but other are easier migration, lower cost to introduce in existing network
Required components
- PC Engines apu2c4
pxe-server- or other means of booting Debian based Dom0 with Xen 4.8 and Linux 4.14.59 (or any other modern kernel which has correct support enabled as in this kernel config)- 2 connected Ethernet ports
- some storage (min 10GB)
Prepare Xen
I’m using apic=verbose,debug iommu=verbose,debug for better visibility of Xen
configuration. More to that we need some preparation in Dom0:
Storage
1 2 3 |
pvcreate /dev/sda1 vgcreate vg0 /dev/sda1 lvcreate -npfsense -L10G vg0 |
PCI passthrough
1 2 |
modprobe xen-pciback xl pci-assignable-add 02:00.0 |
After above commands 02:00.0 should be listed in xl pci-assignable-list
output:
1 2 |
root@apu2:~# xl pci-assignable-list 0000:02:00.0 |
xl allows assigning devices even if IOMMU is not present, but it will issue an
error during VM creation.
Xen pfsense.cfg
First let’s create
1 2 3 4 5 6 7 8 |
me = "pfSense-2.4.3" builder = "hvm" vcpus = 2 memory = 2048 pci = [ '02:00.0' ] nographics = 1 serial = "pty" disk=[ '/root/pfSense-CE-memstick-serial-2.4.3-RELEASE-amd64.img,,hda,rw', '/dev/vg0/pfsense,,hdb,rw' ] |
Then you can create VM:
1 2 3 4 5 6 |
root@apu2:~# xl create pfsense.cfg Parsing config from pfsense.cfg root@apu2:~# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 512 4 r----- 448.3 pfSense-2.4.3 8 2048 2 r----- 29.5 |
Install pfSense
After running VM you can attach to console:
1
|
xl console 8 |
You should see pfSense installer boot log:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 |
Booting...
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p7 #10 r313908+986837ba7e9(RELENG_2_4): Mon Mar 26 18:08:25 CDT 2018
root@buildbot2.netgate.com:/builder/ce-243/tmp/obj/builder/ce-243/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 5.0.1 (tags/RELEASE_501/final 320880) (based on LLVM 5.0.1)
VT(vga): text 80x25
XEN: Hypervisor version 4.8 detected.
CPU: AMD GX-412TC SOC (998.20-MHz K8-class CPU)
Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Features2=0xbef82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,HV>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x40005f1<LAHF,CR8,ABM,SSE4A,MAS,Prefetch,IBS,DBE>
Structured Extended Features=0x8<BMI1>
XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "XenVMMXenVMM"
real memory = 2139095040 (2040 MB)
avail memory = 2016161792 (1922 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s)
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
SMP: AP CPU #1 Launched!
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80682e80, 0) error 1
random: entropy device external interface
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8065c1c0, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8065c270, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8065c320, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80682d20, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80682dd0, 0) error 1
kbd1 at kbdmux0
netmap: loaded module
module_register_init: MOD_LOAD (vesa, 0xffffffff81162bc0, 0) error 19
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc120-0xc12f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 24 at device 2.0 on pci0
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3034000-0xf3034fff at device 3.0 on pci0
vgapci0: Boot video device
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xc100-0xc11f mem 0xf3000000-0xf301ffff,0xf3030000-0xf3033fff irq 32 at device 4.0 on pci0
igb0: Using MSIX interrupts with 3 vectors
igb0: Ethernet address: 00:0d:b9:43:3f:bd
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
xenpv0: <Xen PV bus> on motherboard
granttable0: <Xen Grant-table Device> on xenpv0
xen_et0: <Xen PV Clock> on xenpv0
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
xenstore0: <XenStore> on xenpv0
evtchn0: <Xen event channel user-space device> on xenpv0
privcmd0: <Xen privileged interface user-space device> on xenpv0
debug0: <Xen debug handler> on xenpv0
orm0: <ISA Option ROM> at iomem 0xec800-0xeffff on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 10.000 msec
nvme cam probe device init
xenballoon0: <Xen Balloon Device> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xs_dev0: <Xenstore user-space device> on xenstore0
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
xenbusb_back0: <Xen Backend Devices> on xenstore0
xbd0: 620MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ada0
xbd0: features: flush, write_barrier
xbd0: synchronize cache commands enabled.
xbd1: 10240MB <Virtual Block Device> at device/vbd/832 on xenbusb_front0
xbd1: attaching as ada1
xbd1: features: flush, write_barrier
xbd1: synchronize cache commands enabled.
Trying to mount root from ufs:/dev/ufs/FreeBSD_Install [ro,noatime]...
Setting hostuuid: 81dda54a-6bfd-4458-b8f2-5950cddb471a.
Setting hostid: 0x0bb6f4ee.
Starting file system checks:
/dev/ufs/FreeBSD_Install: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ufs/FreeBSD_Install: clean, 40755 free (43 frags, 5089 blocks, 0.0% fragmentation)
eval: cannot create /etc/hostid: Read-only file system
/etc/rc: WARNING: could not store hostuuid in /etc/hostid.
Mounting local filesystems:.
random: unblocking device.
mtree: /etc/mtree/BSD.sendmail.dist: No such file or directory
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path:
/etc/rc: WARNING: $hostname is not set -- see rc.conf(5).
Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
Feeding entropy: dd: /entropy: Read-only file system
dd: /boot/entropy: Read-only file system
.
Starting Network: lo0 igb0 enc0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
groups: lo
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0d:b9:43:3f:bd
hwaddr 00:0d:b9:43:3f:bd
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0 media: Ethernet: link state changed to UP
autoselect (1000baseT <full-duplex>)
status: active
enc0: flags=0<> metric 0 mtu 1536
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: enc
Starting devd.
Starting Network: igb0.
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0d:b9:43:3f:bd
hwaddr 00:0d:b9:43:3f:bd
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
Starting Network: enc0.
enc0: flags=0<> metric 0 mtu 1536
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
groups: enc
add host 127.0.0.1: gateway lo0 fib 0: route already in table
add host ::1: gateway lo0 fib 0: route already in table
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Generating host.conf.
eval: cannot create /etc/host.conf: Read-only file system
eval: cannot create /etc/host.conf: Read-only file system
eval: cannot create /etc/host.conf: Read-only file system
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Starting local daemons:/dev/md3: 8.0MB (16384 sectors) block size 32768, fragment size 4096
using 4 cylinder groups of 2.03MB, 65 blks, 384 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 4352, 8512, 12672
Welcome to pfSense!
Please choose the appropriate terminal type for your system.
Common console types are:
ansi Standard ANSI terminal
vt100 VT100 or compatible terminal
xterm xterm terminal emulator (or compatible)
cons25w cons25w terminal
Console type [vt100]: |
Unfortunately, pfSense had problem getting DHCP offer and didn’t configure IP address - we tried to figure out what is wrong but my BSD-fu is low. We also checked static IP configuration, but there is no result either. This leads us to ask on the forum.
Xen debian.cfg
1 2 3 4 5 6 7 8 9 |
name = "debian-9.5.0" builder = "hvm" vcpus = 2 memory = 2048 pci = [ '02:00.0' ] disk=[ '/root/debian-9.5.0-amd64-netinst.iso,,hdc,cdrom', '/dev/vg0/debian,,hdb,rw' ] vnc=1 vnclisten='apu2_ip_addr' boot='d' |
Of course, you have to replace apu2_ip_addr with correct IP. After xl create
debian.cfg you can run VNC (tightvnc worked for me) and proceed with the
installation.
PCI passthrough in Debian
Below screenshot show device 02:00.0, which is apu2 middle NIC,
passed-through to VM.
PCI passthrough on Debian worked without any issue DHCP offer was received correctly and I could proceed with performance checks.
Speedtest
Simplest possible test is comparison of throughput between eth0 and eth1. The first is connected directly to our company switch and the second connects pfSense HVM using PCI passthrough.
I used speedtest-cli v2.0.2.
Results for apu2 Dom0:
1 2 3 4 5 6 7 8 9 10 |
(speedtest-venv) root@apu2:~# speedtest-cli Retrieving speedtest.net configuration... Testing from Vectra Broadband (109.241.231.46)... Retrieving speedtest.net server list... Selecting best server based on ping... Hosted by Volta Communications Sp. z o.o (Gdansk) [2.28 km]: 36.105 ms Testing download speed................................................................................ Download: 81.67 Mbit/s Testing upload speed................................................................................................ Upload: 15.38 Mbit/s |
Results for Debian HVM with NIC PCI passthrough:
iperf
Below results are for very simple LAN connection apu3 -> switch -> apu2:
1 2 3 4 5 6 7 8 9 |
(speedtest-venv) root@apu2:~# iperf -s -B 192.168.3.101 ------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address 192.168.3.101 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.3.101 port 5001 connected with 192.168.3.102 port 34004 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.10 GBytes 941 Mbits/sec |
Unfortunately, our switch is probably not well suited for testing 1GbE. Those tests should be repeated with directly connected ports/devices.
Results for Debian HVM with NIC PCI passthrough:
As you can see there is no difference between results, based on that we can conclude that PCI passthrough works and there is no overhead when using IOMMU.
Below log show results from Debian PV and prove how virtualized drivers lead to performance overhead.
1 2 3 4 5 6 7 8 |
root@debian-pv:~# iperf -c 192.168.3.128 ------------------------------------------------------------ Client connecting to 192.168.3.128, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3 ] local 192.168.3.105 port 56204 connected with 192.168.3.128 port 5001 [ ID ] Interval Transfer Bandwidth [ 3 ] 0.0-10.0 sec 746 MBytes 625 Mbits/sec |
Possible problems
xen-pciback not loaded
1 2 3 4 5 6 7 8 9 |
root@apu2:~# xl create pfsense.cfg Parsing config from pfsense.cfg libxl: error: libxl_pci.c:409:libxl_device_pci_assignable_list: Looks like pciback driver not loaded libxl: error: libxl_pci.c:1225:libxl__device_pci_add: PCI device 0:2:0.0 is not assignable libxl: error: libxl_pci.c:1304:libxl__add_pcidevs: libxl_device_pci_add failed: -3 libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add pci devices libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1 libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1 libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed |
Solution:
1
|
modprobe xen-pciback |
PCI device not assignable
1 2 3 4 5 6 |
libxl: error: libxl_pci.c:1225:libxl__device_pci_add: PCI device 0:2:0.0 is not assignable libxl: error: libxl_pci.c:1304:libxl__add_pcidevs: libxl_device_pci_add failed: -3 libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add pci devices libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 2 libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 2 libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 2 failed |
Assign PCI device using xl pci-assignable-add.
No IOMMU
1 2 3 4 5 6 7 8 |
root@apu2:~# xl create pfsense.cfg Parsing config from pfsense.cfg libxl: error: libxl_pci.c:1209:libxl__device_pci_add: PCI device 0000:02:00.0 cannot be assigned - no IOMMU? libxl: error: libxl_pci.c:1304:libxl__add_pcidevs: libxl_device_pci_add failed: -1 libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add pci devices libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 9 libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 9 libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 9 failed |
This error means you don’t have IOMMU correctly enabled. For AMD platforms xl
dmesg contain:
1 2 |
root@apu2:~# xl dmesg|grep -i iommu (XEN) AMD-Vi: IOMMU not found! |
Lack of block backend
xen-blkback should be loaded or compiled in otherwise blow error pop-up.
1 2 3 4 5 6 7 8 9 |
root@apu2:~# xl create pfsense.cfg Parsing config from pfsense.cfg libxl: error: libxl_device.c:1086:device_backend_callback: unable to add device with path /local/domain/0/backend/vbd/1/51712 libxl: error: libxl_create.c:1255:domcreate_launch_dm: unable to add disk devices libxl: error: libxl_device.c:1086:device_backend_callback: unable to remove device with path /local/domain/0/backend/vbd/1/51712 libxl: error: libxl.c:1647:devices_destroy_cb: libxl__devices_destroy failed for 1 libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1 libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1 libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed |
Crash after couple tries
After a couple tries of creating pfSense VM I faced below error:
1 2 3 4 5 6 7 8 9 10 |
root@apu2:~# xl create pfsense.cfg Parsing config from pfsense.cfg libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block add [490] exited with error status 1 libxl: error: libxl_device.c:1237:device_hotplug_child_death_cb: script: Failed to find an unused loop device libxl: error: libxl_create.c:1255:domcreate_launch_dm: unable to add disk devices libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/block remove [604] exited with error status 1 libxl: error: libxl_device.c:1237:device_hotplug_child_death_cb: script: /etc/xen/scripts/block failed; error detected. libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1 libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1 libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed |
Solution: recompile kernel with BLK_DEV_LOOP
Read-only not supported
1 2 3 4 5 6 7 8 |
root@apu2:~# xl create pfsense.cfg Parsing config from pfsense.cfg libxl: error: libxl_dm.c:1433:libxl__build_device_model_args_new: qemu-xen doesn't support read-only IDE disk drivers libxl: error: libxl_dm.c:2182:device_model_spawn_outcome: (null): spawn failed (rc=-6) libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model did not start: -6 libxl: error: libxl.c:1575:libxl__destroy_domid: non-existant domain 1 libxl: error: libxl.c:1534:domain_destroy_callback: unable to destroy guest with domid 1 libxl: error: libxl.c:1463:domain_destroy_cb: destruction of domain 1 failed |
Solution: change pfsense.cfg by adding rw to img file.
References
- Install pfSense 2.1 RC0 amd64 on Xen 4.3 as PV HVM
- Virtual firewall
- xl.cfg manual for Xen 4.8
- Unanswered question about DMA attacks
- Google blog post about fuzzying PCIe
Summary
I hope this post was useful for you. Please feel free to share your opinion and if you think there is value, then share with friends.
We plan to present above results during OSFC 2018 feel free to catch us there and ask questions.
We believe there are still many devices with VT-d or AMD-Vi advertised in
specs, but not enabled because of buggy or not-fully-featured firmware. We are
always open to support vendors who want to boot hardware by extending and
improving their firmware. If you are user or vendor struggling with hardware
which cannot be fully utilized because of firmware, feel free to contact us
contact<at>3mdeb<dot>com.
Founder and Embedded Systems Consultant at 3mdeb as well as CTO of vitroTV. Passionate about building firmware that enables advanced hardware features in modern products. Dedicated to customers that treat embedded software security and upgradability as forethought. In favor of fixed price projects with the clear definition of success. Available as freelance CTO for startups and small business.