OS Dev





TrenchBoot Anti Evil Maid for Qubes OS

Published at January 31, 2023 · Michał Żygowski ·  14 min read

Qubes OS Anti Evil Maid (AEM) software heavily depends on the availability of the DRTM technologies to prevent the Evil Maid attacks. However, the project has not evolved much since the beginning of 2018 and froze on the support of TPM 1.2 with Intel TXT in legacy boot mode (BIOS). In the post we show how existing solution can be replaced with TrenchBoot and how one can install it on the Qubes OS. Also the post will also briefly explain how TrenchBoot opens the door for future TPM 2.0 and UEFI support for AEM....

Categories: bootloader firmware hypervisor os-dev security


Qubes OS summit 2022 - Summary

Published at October 5, 2022 · Norbert Kamiński ·  10 min read

Three weeks ago 3mdeb with Qubes OS team had organized next edition of the Qubes OS summit. This year summit was a face-to-face event hosted in Berlin, which took place from the 9th to the 11th of September....

Categories: miscellaneous os-dev


Infrastructure for Xen development and debugging

Published at July 4, 2022 · Piotr Król · Norbert Kamiński ·  5 min read

Back in 2018 at OSFC, we've presented AMD IOMMU enabling for PC Engines apuX (GX-412TC) platforms. Our hypervisor of choice was Xen and we used it to verify the PCI pass-through feature. Unfortunately, the booting process was not exactly stable. In this article, you can check how to prepare infrastructure for Xen development and debugging...

Categories: firmware os-dev security



First impression on Nezha RISC-V SBC

Published at November 19, 2021 · Cezary Sobczak ·  9 min read

Nezha is a AIoT development board customized by AWOL based on Allwinner's D1 chip. It is the world's first mass-produced development board that supports 64bit RISC-V instruction set and Linux system....

Categories: firmware iot os-dev




GRUB mini–summit 2020

Published at November 2, 2020 · Kamila Banecka ·  5 min read

GRUB mini–summit 2020. This year we cannot miss this opportunity to meet again and face the new challenges of GRUB/GRUB2. So,dear reader, feel invited to look at GRUB with a magnifying glass....

Categories: firmware os-dev security







Trying to fix ESXi 6.7.0 boot issue, part one

Published at March 4, 2020 · Krystian Hebel ·  14 min read

First mentions that updated versions of VMware's ESXi 6.7.0 installer doesn't start on PC Engines platforms come from the beginning of 2019. Older versions of ESXi worked fine. 'Shutting down firmware services...' is the last line printed before hang or reboot....

Categories: firmware os-dev


GRUB2 and 3mdeb minisummit 2019

Published at February 19, 2020 · Piotr Król ·  7 min read

In December 2019 we had pleasure to meet Daniel Kiper #GRUB2 maintanaer in 3mdeb office in Gdańsk. We discussed various #GRUB2, #Xen, #firmware, #coreboot, #security and #TPM related topics. Results of that "minisummit" was presented in following blog post in form of presentations and videos....

Categories: firmware os-dev security



How L4 Genode hypervisor stands against proprietary RTOS solution

Published at January 10, 2020 · Krystian Hebel ·  5 min read

A microkernel is a minimal computer operating system kernel which provides no operating system services at all, only the mechanisms needed to implement such services. A concept is tolerated inside the μ-kernel only if moving it outside the kernel would prevent the implementation of the system’s required functionality. In this article we will show our considerations on using L4 microkernels on VM....

Categories: os-dev




Qubes OS and 3mdeb 'minisummit' 2019

Published at August 7, 2019 · Piotr Król ·  8 min read

In May we had pleasure to meet Marek Marczykowski-Górecki #QubesOS Project Lead in 3mdeb office in Gdańsk. We discussed various #QubesOS, #Xen, #firmware, #coreboot, #security and #TPM related topics. Results of that "minisummit" was presented in following blog post....

Categories: firmware os-dev security



Yocto meta-rte is now open for everyone

Published at March 13, 2019 · Marta Szelecka ·  3 min read

Yocto meta-rte is now open for everyone We are happy to announce that our 3mdeb’s Yocto meta-rte is now available on our GitHub. But let’s say briefly what the Yocto Project is and why we decided to work with it. First of all, like everything that we love the most, Yocto Project is open sourced. The project is hosted by the Linux Foundation and gives you templates, methods, and set of interoperable tools for creating OS images for embedded Linux systems....

Categories: os-dev


Quick start guide to kas - best tool for setting up the Yocto projects

Published at February 7, 2019 · Maciej Pijanowski ·  8 min read

Introduction If you are using the Yocto Project, you certainly have encountered the hassle of managing multiple layers and tracking their revisions. I’ve been using the Yocto Project for nearly 3 years by now and have mostly been using the tool for this purpose. While I’m not a huge fan of it, it is relatively simple to use and gets the job of fetching layers and controlling their revisions done properly....

Categories: os-dev


debos image for HummingBoard Edge

Published at October 10, 2018 · Maciej Pijanowski ·  8 min read

Intro In my previous posts I have shared my first experience with debos and how to run debos in a container. In today’s post, I’d like to present how can we use all of that to generate base Debian image for an ARM board. My board of choice for this particular example will be the HummingBoard Edge. The post is inspired by the feedback from the new users (such as this one) that there are no end-to-end examples how to quickly start using this tool....

Categories: os-dev


RTE for automated kernel deployment and everyday use

Published at October 3, 2018 · Piotr Król ·  9 min read

We continue our effort to enable IOMMU and as side effect I have to play with various technologies to exercise reliable development environment which base on RTE. In this blog post I would like to present semi-automated technique to debug firmware, Xen and Linux kernel. The goal is to have set of tools that help in enabling various features in Debian-based dom0. We would like: update Linux kernel which is exposed over HTTP server update rootfs provided through NFS I will use following components:...

Categories: firmware os-dev


debos in docker - the second attempt

Published at August 23, 2018 · Maciej Pijanowski ·  5 min read

Intro In the previous post, I have shared my first experience with the Debian images builder - debos. I have posted my current results on the issue but since there was no response, I’ve decided to try to move forward by myself. Just to remind - I was stuck at the following error (when building for arm64): 1 2 3 4 5 6 7 8 9 10 11 12 2018/07/26 18:36:39 Debootstrap (stage 2) | chroot: failed to run command '/debootstrap/debootstrap': Exec format error 2018/07/26 18:36:39 debootstrap....

Categories: os-dev


Xen HVM guests on PC Engines apu2

Published at August 16, 2018 · Piotr Król ·  15 min read

Continuing blog post series around Xen and IOMMU enabling in coreboot we are reaching a point in which some features seem to work correctly on top of recent patch series in firmware. What we can do at this point is PCI passthrough to guest VMs. Previously trying that on Xen caused problems: random hangs firmware cause Linux kernel booting issues (hang during boot) IOMMU disabled - unable to use PCI passthrough Now we can see something like that in dom0:...

Categories: firmware os-dev


Our first look at debos - new Debian images generator

Published at July 27, 2018 · Maciej Pijanowski ·  8 min read

What is debos debos is quite a new tool allowing easier Debian images generation. It seems to be following current trends - it is written in Go, using YAML as an input format. The idea of taking away debootstrap shell scripts and replacing it with a single, simple YAML file looks tempting enough to give it a try. Full feature description can be found in this introductory post on Collabora’s blog....

Categories: os-dev


How to boot Xen over PXE and NFS on PC Engines apu2

Published at July 18, 2018 · Piotr Król ·  9 min read

From time to time we face requests to correctly enable support for various Xen features on PC Engines apu2 platform. Doing that requires firmware modification, which 3mdeb is responsible for. Xen have very interesting requirements from firmware development perspective. Modern x86 have a bunch of features that support virtualization in hardware. Those features were described in Xen FAQ. It happens that most requesting were IOMMU and SR-IOV. First, give the ability to dedicate PCI device to given VM and second enables so-called Virtual Functions, what means on a physical device (e....

Categories: firmware os-dev


ssh reverse tunnel for PXE, NFS and DHCP setup on Qubes OS

Published at December 5, 2017 · Piotr Król ·  6 min read

At some point I stuck in the forest with WiFi connection and no physical access to router to create nice networking for my coreboot development needs. Recently I switched my laptop to Qubes OS what give interesting flexibility, but also additional problems. My key requirement is to boot system over PXE, so I can easily do kernel development and play with Xen. Because only available connection for my apu2 platform was directly to my laptop I had to provide configured DHCP server and PXE server on it....

Categories: os-dev


Installing OpenWRT on APU3 platform

Published at May 12, 2017 · Kamil Wcisło ·  13 min read

This guide should be considered as a simple walk-through for using APU3 platform in some generic use-cases. I’m trying to explain how to work with the device and use it in a generic manner. There is a part about the coreboot firmware, which could be used as a reference of how to start customizing it for own purposes. Configuring the hardware At first, let’s figure out some basic requirements for our new device:...

Categories: firmware os-dev


Nerves project triage on BeagleBone Black Black

Published at March 10, 2017 · Piotr Król ·  6 min read

Recently one of my customers brought to my attention Nerves. It aims to simplify use of Elixir (functional language leveraging Erlang VM) in embedded systems. This system has couple interesting features that are worth of research and blog post. First is booting directly to application which is running in BEAM (Erlang VM). Nerves project replace systemd process with programming language virtual machine running application code. Concept is very interesting and I wonder if someone tried to use that with other VMs ie....

Categories: firmware os-dev


PC Engines APU2 netboot Debian installation

Published at March 26, 2016 · Piotr Król ·  3 min read

In previous post I described how to setup PXE server and boot Debian installer using it. I mentioned that provided setup is limited and some extensive configuration is needed to make it useful for real world example. Since that time I learned that there is chain command in iPXE, which give ability to use arbitrary TFTP server as boot file source. Using RPi PXE server For example by changing my test network topology from previous post to something like that:...

Categories: os-dev


Emulate Rapberry Pi 2 in QEMU

Published at December 30, 2015 · Piotr Król ·  4 min read

In the process of planning system testing for one of my clients I found that someone from Microsoft published patches with BCM2836 support to QEMU mailing list. I thought it is very interesting, because if it is possible to setup emulated Raspberry Pi many use cases can be tested faster and in more automatic way. For example checking how application behave when running on more then one device at once, testing massive deployment process, stress testing and finally speed up debug-fix-test process....

Categories: app-dev os-dev


Linux, RPi and USB over IP updated

Published at October 27, 2015 · Piotr Król ·  4 min read

Because of increasing interesting in USB over IP topic I decided to refresh my old post. I will focus on doing the same thing with more recent version of Raspabian. If you need more information please read my previous post. Setup SD card First get recent version of Raspbian, then unzip and dd it to SD card: 1 sudo dd bs=4M if=2015-09-24-raspbian-jessie.img of=/dev/sdc If you are impatient and want to know what happen in background you can use this method of tracking dd progress:...

Categories: os-dev


Building Android 4.2 LiveSuit image for Cubietruck (Allwinner A20)

Published at September 16, 2015 · Piotr Król ·  7 min read

Treating A20 boards like outdated piece of HW by vendors makes building Android for Cubietruck not trivial task. Finding documentation, mailing list or blog post that clearly describe steps is almost impossible. Most of links to SDK are broken and instructions outdated. Because of that I decided to leave couple notes for me and all of you lost in this madness. Hopefully below steps can build foundation for future development and improvements....

Categories: os-dev


Setup for Linux kernel development on Cubietruck

Published at September 1, 2015 · Piotr Król ·  8 min read

During last couple of months I see quite big interest in building products on A20 SoC. This chip can be bought for 6USD in quantity. Most important features are: Dual-Core ARM Cortex-A7 (ARMv7) Mali-400 MP2 HDMI, VGA and LCD MMC and NAND OTG and 2 Host ports Tracking media related to low-end mobile market IMHO the hottest SoCs are Allwinner A20 and Rockchip RK3288. A20 ship with dozen development boards like Cubieboard or pcDuino series, Banana Pi, MarsBoard or Hummingbird....

Categories: os-dev


Raspberry Pi kernel repository aggregating patches for Linux mainline

Published at October 15, 2014 · Piotr Król ·  3 min read

Since several months I’m trying to find my way to embedded Linux programming. My hardware set was very limited I had only one board that can be called “embedded” and it was Raspberry Pi. Because I am more interested in firmware/OS level then hardware I tried to figure out what is going on with RPi kernel. After taking brief review of raspberrypi/linux GitHub repository I realized that close to my heart is upstreaming effort....

Categories: os-dev


How to fix backlight issue on IdeaPad y510p

Published at August 23, 2014 · Piotr Król ·  1 min read

Today I decide to switch to latest kernel (3.17-rc1) on my IdeaPad y510p. I hit only one annoying problem until now - after booting my main screen was dimmed. I tried all instructions from top google hits for all possible configurations of keywords linux, y510p, backlight issue, etc. Especially I tried all methods from Arch Wiki. Finally I found solution, by greping modinfo for my Intel graphics card: 1 2 3 4 5 [23:55:24] pietrushnic:~ $ sudo modinfo i915|grep backlight parm: invert_brightness:Invert backlight brightness (-1 force normal, \ 0 machine defaults, 1 force inversion), please report PCI device ID, subsystem \ vendor and subsystem device ID to dri-devel@lists....

Categories: os-dev


Linux (Debian Wheezy) on Lenovo y510p

Published at May 16, 2014 · Piotr Król ·  4 min read

After long analysis I decide to buy new laptop. I had about $1000 (or 3000PLN) and most important things to me were: i7 CPU - because of performance (of course at least 4700 series) SSD - again performance 17.3” - working space no OS/FreeDos/Linux - I will not pay additional fee to M$ for system that I won’t use Full HD resolution at least 8GB RAM non-glare display First I realize that my budget is to small for such a hardware....

Categories: os-dev


0x6: Root file system for embedded system

Published at June 7, 2013 · Piotr Król ·  10 min read

Introduction To make our embedded linux work as virtual development platform we need some environment after booting. There is many approaches to get working root file system but I will use the easiest one as an exercise. I don’t want to create full embedded distribution (this is good plan for future works). Right now I will be happy with simple initramfs based on BusyBox. For all interested in creating own root filesystem there are few places where you can find information:...

Categories: os-dev


0x5: Qemu network configuration and tftp for Virtual Development Board

Published at June 7, 2013 · Piotr Król ·  7 min read

Introduction This was not trivial task to me. As usual google is your friend and RTFM works. First we will set tftp which we use to download modified kernel for U-Boot. Second I will show how to setup bridged network for QEMU needs and finally we will perform some basic test of our setup. Let’s go. Setup tftpd First install: 1 sudo apt-get install tftpd tftp Make sure that /srv/tftp is writable for your user....

Categories: app-dev os-dev


0x4: Linux kernel for embedded system

Published at June 7, 2013 · Piotr Król ·  3 min read

A little history Thinking about embedded linux probably leads to first try of porting linux to different architecture. I did google research (I know I should probably read mailing list archive) and found that there were few attempt to port linux to different platform. There is no clear information about which port of linux was first. This is probably because many hackers didn’t report their effort. Arguably earliest out-of-tree version was probably for Acron A5000 (arm), Motorola 68000 (m68k) around Spring/Summer of 1994....

Categories: os-dev


0x2: Toolchain for Virtual Development Board

Published at June 7, 2013 · Piotr Król ·  4 min read

Introduction This is probably the most complicated topic of all related to embedded development but we need to deal with it at the beginning. I read a lot about toolchains but still don’t know enough to explain details. I think that best answers are in crosstool-ng documentation. What is toolchain ? Toolchain as the name said is a set of tools chained together, so output of one tool is the input for different tool....

Categories: app-dev os-dev


0x1: Qemu as an environment for embedded board emulation

Published at June 7, 2013 · Piotr Król ·  2 min read

Table of contents Introduction Compilation Kudos Introduction QEMU is a CPU emulator using dynamic binary translation to convert guest CPU instructions into host CPU instructions[1]. It supports many architectures from x86, through ARM and MIPS, to MicroBlaze. According to compilation configuration target list QEMU targets 26 different softmmu types. Only for ARM it supports 33 machines (like ARM Versatile/PB (ARM926EJ-S) or Samsung NURI board (Exynos4210)) and 28 CPUs (with cortex-a9 and pxa270)....

Categories: app-dev os-dev


Yet another quick build of arm-unknown-linux-gnueabi

Published at April 3, 2013 · Piotr Król ·  3 min read

So I decide to check what is going on with crosstool-ng and refresh my old post about building arm-unknown-linux-gnueabi toolchain. Last post was pretty popular, so definitely this is direction I should follow :). I will not repeat myself, so if you encounter any problems please check last post, section with known problems in crosstool-ng doc/ directory or RTFM. Let’s begin: Get the latest crosstool-ng As usual I’m trying to use latest version possible....

Categories: app-dev os-dev


Building ARM toolchain? part 2: gcc and eglibc

Published at April 12, 2012 · Piotr Król ·  4 min read

Unfortunately after few tries of cross compiling eglibc using different source for instructions I always end with hard to solve issues. Luckily, in the sources of eglibc I noticed instructions for cross-compiling written long time ago by Jim Blandy(I know i should start here). Lot of thanks to him for it. Below I describe my experience which I gained during eglibc cross compilation for arm-unknown-linux-gnueabi and procedure that I used. Commands below contain some constants that I used in previous works....

Categories: app-dev os-dev


Building ARM toolchain - part 1: libs and binutils

Published at March 20, 2012 · Piotr Król ·  3 min read

Searching the Internet for information on how to build arm toolchain from scratch I realize that it is very hard to find information about this matter (and recent one even harder). I will try to fill this lack of information and try to build toolchain. My main goal is to use a component based on the GNU public license, and using them in as the newest version as it is possible....

Categories: app-dev os-dev


Quick build of arm-unknown-linux-gnueabi with crosstool-ng

Published at March 14, 2012 · Piotr Król ·  2 min read

You might be surprised at how much you have to make to correctly build arm-unknown-linux-gnueabi config based toolchain with crosstool-ng. As you can see examples of many open source projects, the man’s work is a rare resource. The result of this economic fact is that the attempt to build configuration arm-unknown-linux-gnueabi is not a simple task and during an operation you can come across many problems. Although I am not afraid of problems and effectively try to fight them and of course sharing the results of my work....

Categories: app-dev os-dev