.png){kind=small}
Hello there! This post will introduce some of the very popular attacks that target electronic devices - the RAM attacks, but the main topic will be the verification of ram-wipe software solution protection from the attacks.
The Kicksecure project engaged 3mdeb to evaluate the effectiveness of ram-wipe while it was still in testing. This research was supported by Power Up Privacy, a privacy advocacy group dedicated to empowering privacy-focused projects with the resources they need to fulfill their mission of making the world a better place.
Introduction to RAM attacks
Here, the focus is on two types of attacks: cold-boot and warm-boot attacks.
Warm-boot attacks focus on reading data left in RAM by the victim’s operating
system or application using the attacker’s operating system or application
without disconnecting power from the device. For example, rebooting the victim’s
operating system via systemctl reboot (without breaking the victim’s system
execution flow) and booting some malware to dump or search RAM. That is, the
platform is not being moved into the G3 state according to ACPI specifications
and the operating system code execution flow is not being broken.
On the other hand, the cold boot attack requires the platform to be cold booted, which appears when a platform loses power, that is, the G3 state according to ACPI documentation. Then, the device is booted again.
This attack can utilize the ability to break software execution flow by sudden power loss. In such a case, the device execution flow breaks, and the tasks that should be executed during a normal shutdown sequence are not being executed, and everything present in RAM before the shutdown stays there for a specific for the hardware period of time (that could be extended) and could be read.
During such a cold boot, an attacker boots malware or a specific operating system equipped with tools needed to read RAM and read it, gaining access to all data and secrets present in RAM before the shutdown. By doing so, the attacker can get access, among others, to keys that are used to encrypt certain secrets and file systems.
Below is a graphical representation of the attacks:
Legend:
- Initial state: the platform has no power.
- Black arrows: standard boot/power off process.
- Black dashed arrow: getting the platform to power off state with breaking software execution flow.
- Red dashed arrows: malware boot process.
- 1: The platform executing firmware or bootloader code.
- 2: Firmware or bootloader boots operating system or application that manipulates secrets in RAM.
- 3: Malware in the form of an operating system or application that dumps secrets from RAM.
The only difference between the attacks is how the malware is being booted (check the red dashed arrows): from platform power off state with the previous system not properly shut down or without powering off the platform.
ram-wipe
ram-wipe is a full software
solution designed by Kicksecure and published under AGPL-3+ license. The goal is
to protect from RAM attacks by utilizing GNU/Linux tools. Currently, the only
solution available on the market with the same goal is Tails Memory
Erasure. The ram-wipe is
available for testing and evaluation, more information can be found in its
installation and
design documentation.
Using the tree tool inside the ram-wipe repository shows that ram-wipe
uses dracut and systemd:
|
|
And file tool shows that all executables are shell scripts:
|
|
Though shell scripts are easy to analyze, the most complicated thing here is to
put all variables dependencies from the scripts and services that are being
executed in different environments: in Linux during shutdown (systemd,
service), inside RAM disk during shutdown, and inside RAM disk after booting
second Linux kernel using kexec. Because of the complexity, it is better to
demonstrate its structure via diagrams.
The first diagram demonstrates the system boot process:
The only thing to note here is the way ram-wipe decides whether to run or not
during reboot/poweroff/halt: it checks Linux kernel command line parameter
wiperam and then creates a file that tells dracut services to fall back to
RAM disk during reboot/poweroff/halt, where the first ram-wipe stage will
run.
Then, the system mounts rootfs and switches to userspace:
Here, ram-wipe relies on the ram-wipe-kexec-prepare.service for setting up
the second stage and on the dracut-shutdown.service for getting back to the
RAM disk and launching the first ram-wipe stage.
After that, the system switches back to the RAM disc:
Now the ram-wipe first wipe stage is being launched. The wiperam and
wiperamexit are Linux kernel command line boot parameters used to skip this
stage (the former one) or prevent the ram-wipe loop (the latter one).
Then, in case the second stage is enabled, kexec is being run during execution
of the first stage that results in booting to the second kernel:
Note: The first wipe stage before loading Linux kernel via
kexecchecks whether secure boot is enabled bymokutiltool, so it is possible to run it on platforms with Secure Boot protection enabled.
This phase wipes the RAM again, but this time during booting the second kernel
when the RAM disk is being mounted first time. After wiping is done - the system
is forced to reboot/powerof/halt (depending on the wiperamaction Linux kernel
command line parameter) without complete boot.
Now, when the workflow is clear, it is important to note two things:
- Both
ram-wipestages rely on thesdmemtool, which is being called twice with the same arguments-l -l -v, resulting in RAM being overwritten with zeroes (-l -lmeans only onesdmemwalk filling up RAM with zeroes,-vis for verbose). This fact will be used later for testing. ram-wipefully relies on Linux tools andsystemdservices execution workflow. This fact will be very important during testing and writing down conclusions.
Testing methodology and tools
As ram-wipe is based on setting RAM to zeroes during Linux shutdown or reboot
workflows, to check whether it succeeded or not, the RAM should be checked after
Linux has finished executing. The main problem here is the way the RAM will be
checked. There are two ways: checking the RAM externally, that is, by software
that will not use the wiped RAM for execution (executing software in another RAM
or chip), or by making sure that the software load address and size will be
known, so to be able to conclude what has been overwritten and why some
addresses are not zeroed.
This post will use the second way by executing an EFI application during the early device boot stage. The problem with some EFI structures (e.g., logs) being randomly loaded into RAM was solved by checking where those structures are being written before every test. During checking, the application records the addresses in memory modified by firmware and excludes it from dumping.
The application can be found here. It is a simple application that runs automatically during boot and dumps specified RAM range values into files. The application was written as minimalistic as possible to prevent it from modifying RAM.
The final testing methodology using the application is as follows:
- Boot to the application and choose option 1 to write the pattern.
- Reboot to the application and choose option 2 to exclude RAM modified by firmware.
- Reboot to OS and run
ram-wipe. - Boot to the application and dump memory.
It is important to note that the booting to the OS was set as automatic
by changing the Dasharo firmware boot order. But the
application was booted by choosing the needed drive in the One Time Boot menu.
When booting the application by manually choosing the hard drive, it is
important to take exactly the same steps every time. Otherwise, the firmware
will generate another set of data in RAM and make the second step from above
useless.
After RAM has been dumped, it is being analyzed by a Python
script
that checks what addresses were overwritten by zeroes and what addresses still
contain any values. Though this script shows what addresses contain zeroes and
what do not, the dumped RAM will still be analyzed manually in depth to check
what addresses were not wiped and why. To analyze results, further the addresses
used by kernels and RAM disks during both stages could be checked via dumping
/proc/iomem and analyzing dmesg logs. This will give extra information why
some addresses were not zeroed because ram-wipe as a software solution cannot
overwrite kernel address space as well as some other processes address space.
The tests will be done for one warm boot attack via rebooting Linux and for one
cold boot attack utilizing QEMU Monitor system_reset functionality.
Testing environment
The hardware requirements are rather small: the ability to run EFI applications
and as little RAM as possible to reduce the time needed to dump RAM. Because the
ram-wipe is a software solution and, therefore, does not depend on hardware
(except for the fact that more RAM will take more time to wipe) - the QEMU with
Dasharo
Firmware is being used
as a test platform.
While there is no difference between the real platform and QEMU for warm boot attacks, the real cold boot attack cannot be done on QEMU because, QEMU is software, and it cannot not be reset via power loss. Hence, the QEMU RAM cells will not lose their charge as on the real platform. In this case, the cold boot attack could be decomposed into two parts: the software execution flow brake and platform power loss.
The consequence of the software execution flow brake is that everything that was
going to be executed will not execute. For example, if somebody is watching a
film - the film will immediately stop, if an application wants to wipe its
passwords from RAM - the password will not be wiped, etc. This is important
because, as was described above, ram-wipe is a full software solution and
relies on Linux kernel and systemd services execution during
reboot/poweroff/halt. As was described previously it is the key difference
between warm boot and cold boot attacks.
The consequence of the power loss is that all data that was stored in volatile memory is going to be lost because of the fact, that volatile memory relies on storing its data in cells built up from transistors and capacitors. The loss means that after platform loses its power and then booted up again, the volatile memory holds random data instead of what was written to it before the power loss. It is important to note that it takes some time for the capacitors to lose their charge, so the data could be acquired from the volatile memory during a short period of time after power loss.
The fact that data loss after platform power loss actually prevents the attacker from getting all the data from RAM. Sometimes, the regions that were used to store secrets will lose their charge faster than other regions, sometimes not. But security should not rely on random events, should it? If you want to get more information on why the memory loses its data after power loss and how long it takes, you should check the series of our blog posts:
- Research of RAM data remanence times
- Research of RAM data remanence times, part 2
- Conclusions from RAM data remanence tests
Because the data loss after platform power loss makes it harder to get data
during a cold boot attack, it will be easier to test protection from cold boot
attacks on systems where software execution flow could be broken without losing
power, so to eliminate the randomness of data loss and dump all data from RAM,
that will prove, whether the RAM was wiped by ram-wipe or not. Hence, QEMU
might come in handy here because it provides a system_reset
feature that brakes
software execution without reallocating memory for QEMU.
As an operating system, the Debian Trixie testing release with encrypted
rootfs was used. To install and reproduce the results, follow the following
steps:
-
Install Debian Trixie following instructions from the Debian Testing article with
rootfsandswapprotected by LUKS. -
Migrate to
dracut(thedracutversion in this set-up is106-6, thesystemdversion is257.5-2):1 2sudo apt install dracut sudo dracut -fNote: If you want to do tests with LUKS on Debian Trixie, it is important to do the fifth step from Kicksecire’s dracut migration Wiki page and then run
sudo dracut -fagain. Otherwise, you might end up with LUKS unlock prompt not showing up during boot. -
Verify that
dracutis being used by checking system logs, e.g.:1 2 3 4 5 6 7 8 9$ sudo dmesg | grep dracut (...) [ OK ] Finished dracut-initqueue.service - dracut initqueue hook. (...) Starting dracut-pre-pivot.service - dracut pre-pivot and cleanup hook... [ OK ] Finished dracut-pre-pivot.service - dracut pre-pivot and cleanup hook. (...) [ OK ] Stopped dracut-pre-pivot.service - dracut pre-pivot and cleanup hook. (...) -
Compile and install
ram-wipepackage and its dependency -helper-scripts:1 2 3 4 5 6 7 8 9 10 11 12 13 14 15$ sudo apt install build-essential debhelper debhelper-compat dh-python\ dh-apparmor config-package-dev $ git clone https://github.com/Kicksecure/helper-scripts.git $ cd helper-scripts $ git checkout 37a9ef2e84e352e37b7e84bd8ed36e5d31218778 $ dpkg-buildpackage -b $ cd - $ sudo apt install ./helper-scripts_31.3-1_all.deb $ git clone https://github.com/Kicksecure/ram-wipe.git $ cd ram-wipe $ git checkout ed9c2978b2c69cebd37b3e1412b648d8f00584e4 $ dpkg-buildpackage -b $ cd - $ sudo apt install ./ram-wipe_3.3-1_all.deb $ sudo dracut -fNote: Do not mind the
failed to signissues. These will appear if the GPG key is not configured properly and is not important here. -
Verify that
ram-wipeis being launched during reboot or shutdown by checking system logs. For the first stage, check for the following logs:1 2 3 4 5 6 7(...) [ 66.353653] wipe-ram-shutdo (4396): drop_caches: 3 [ 66.359901] dracut INFO: wipe-ram.sh: First RAM wipe pass completed, OK. (1/2) [ 66.368166] dracut INFO: wipe-ram.sh: Checking if there are still mounted encrypted disks... [ 66.379157] dracut INFO: wipe-ram.sh: Success, there are no more mounted encrypted disks, OK. [ 66.388631] dracut INFO: wipe-ram.sh: Now running 'kexec --exec'.. (...)The second stage has issues regarding printing, so its logs might not be visible on serial or graphical output. But the OOM Killer logs caused by
ram-wipeandsdmemshould be visible.
The test setup using the following keys for LUKS:
|
|
Important note: this security information leak was done on purpose, do not share your secrets with anyone!
ram-wipe tests
Running first stage only
To make ram-wipe run only the first stage on reboot, the Linux kernel command
line parameter wiperamexit was set to no. This resulted in kexec failure
because of the ram-wipe-kexec-prepare.service not starting. Hence, the second
Linux kernel was not booted, and the second ram-wipe stage was not running:
|
|
The bootloader (GRUB in this case) and Linux kernel got the following memory addresses from firmware (inf. from the EFI application):
|
|
With the following pages and addresses excluded as used by firmware (inf. from the EFI application):
|
|
The memory provided by firmware to bootloader and Linux was the same for both attacks.
Warm boot attack on first stage
The /proc/iomem showed the Linux kernel loaded in the following addresses:
|
|
Hence, these ranges should be excluded from the analysis because sdmem, which
is used in ram-wipe, is an application and cannot overwrite kernel space
addresses. Analysis of other addresses:
0x1000-0x10000: not zeroed, buthexdumpshowed that the only data in this region is the pattern written by the EFI application. Hence, this region was not used by Linux or bootloader.0x100000-0x800000: 524288 bytes zeroed and 6815744 non-zeroed. All non-zeroed bytes contain the pattern written by the EFI application. Hence, these bytes were not used by Linux or bootloader.0x1600000-0x75ff0000: 2026242048 zeroed and 28049408 non-zeroed. Most of the non-zeroed addresses contain some data. Other addresses contain the pattern.0x7bd48000-0x7d922000: 26517504 bytes zeroed and 2686976 non-zeroed. Non-zeroed bytes contain the GRUB executable and its modules.0x7f5ee000-0x7f5f4000: zeroed.0x7fc00000-0x7fc70000: zeroed.0x7fd20000-0x7fd2d000: zeroed.0x100000000-0x180000000: this region is very fragmented in terms of zeroed and non-zeroed addresses. 1607467008 bytes was zeroed, and 540016640 was not zeroed. With 36451452 bytes taken by the kernel and 503565188 used by processes.
After raw analysis, it is high time for some real attack. The attack will be
done by searching the dumped memory for LUKS keys using aeskeyfind, and the
results are:
|
|
So indeed, the first ram-wipe stage saved the keys.
Cold boot attack on first stage
The /proc/iomem showed the first Linux kernel loaded in the following
addresses:
|
|
Hence, these ranges should be excluded from analysis because sdmem used in
ram-wipe is an application and cannot overwrite kernel space addresses.
Analysis of other addresses:
0x1000-0x10000: not zeroed, buthexdumpshowed that the only data in this region is the pattern written by the EFI application. Hence, this region was not used by Linux or bootloader.0x100000-0x800000: 524288 bytes zeroed and 6815744 not zeroed. All non-zeroed bytes contain the pattern written by the EFI application. Hence, these bytes were not used by Linux or bootloader.0x1600000-0x75ff0000: 71892992 bytes zeroed and 1982398464 bytes non-zeroed. Most of the non-zeroed addresses contain mostly GRUB data. Other addresses contain the pattern.0x7bd48000-0x7d922000: not zeroed, contains process data.0x7f5ee000-0x7f5f4000: not zeroed, contains process data.0x7fc00000-0x7fc70000: not zeroed, contains process data.0x7fd20000-0x7fd2d000: not zeroed, contains process data.0x100000000-0x180000000: this region is very fragmented in terms of zeroed and not zeroed addresses. 328466432 bytes was zeroed and 1819017216 was not zeroed.
Now, let’s do the attack on LUKS keys:
|
|
Both keys were successfully stolen after cold boot! Hence, running only the
first ram-wipe stage does not protect from cold boot attack.
Running second stage only
To work around running only the second ram-wipe stage on reboot, the following
modification was used:
|
|
Dasharo firmware proved to be pretty stable in its memory allocation during boot by providing the same memory map:
|
|
The memory provided by firmware to bootloader and Linux was the same for both attacks.
Warm boot attack on second stage
This time /proc/iomem shows the Linux kernel is hidden under the following
addresses:
|
|
And the kernel loaded by kexec before the second ram-wipe stage execution:
|
|
Analysis of other addresses:
0x1000-0x10000: not zeroed, buthexdumpshowed that the only data in this region is the pattern written by the EFI application. Hence, this region was not used by Linux or bootloader.0x100000-0x800000: 524288 bytes zeroed and 6815744 not zeroed. All non-zeroed bytes contain the pattern written by the EFI application. Hence, these bytes were not used by Linux or bootloader.0x1600000-0x7bd20000: 2047803392 zeroed and 6488064 non-zeroed. Some of not zeroed addresses contain some data, others contain the pattern.0x7bd48000-0x7d922000: zeroed.0x7f5ee000-0x7f5f4000: zeroed.0x7fc00000-0x7fc70000: zeroed.0x7fd20000-0x7fd2d000: zeroed.0x100000000-0x180000000: this region is very fragmented in terms of zeroed and non-zeroed addresses. 1875509248 bytes was zeroed and 271974400 was not zeroed. The situation is the same ass before: a part of the non-zeroed addresses was used by the kernel and another part by processes.
It is about time for the same attack as before:
|
|
And again, no keys dumped.
Cold boot attack on second stage
This time /proc/iomem shows the Linux kernel is hidden under the following
addresses:
|
|
Analysis of other addresses:
0x1000-0x10000: not zeroed, buthexdumpshowed that the only data in this region is the pattern written by the EFI application. Hence, this region was not used by Linux or bootloader.0x100000-0x800000: 524288 bytes zeroed and 6815744 not zeroed. All non-zeroed bytes contain the pattern written by the EFI application. Hence, these bytes were not used by Linux or bootloader.0x1600000-0x7bd20000: 84869120 zeroed and 1969422336 not zeroed. Some of non-zeroed addresses contain some data, others contain the pattern.0x7bd48000-0x7d922000: not zeroed, contains data.0x7f5ee000-0x7f5f4000: not zeroed, contains data.0x7fc00000-0x7fc70000: not zeroed, contains data.0x7fd20000-0x7fd2d000: not zeroed, contains data.0x100000000-0x180000000: this region is very fragmented in terms of zeroed and non-zeroed addresses. 350945280 bytes was zeroed and 1796538368 was not zeroed. Still a lot of processes data not zeroed.
The attack on LUKS is still successful:
|
|
Running both stages
Running both ram-wipe stages on reboot did not require any changes because it
is the default behavior.
And again, Dasharo firmware, because of its stability and predictability, saves this blog post from another copy of logs from the EFI application because the memory map and excluded pages used by the firmware are the same as the ones shown in previous sections and are the same for two attacks described in the next two sections.
Warm boot attack on both stages
This time /proc/iomem shows the Linux kernel is loaded under the following
addresses:
|
|
And the kernel loaded by kexec before second ram-wipe stage execution:
|
|
Analysis of other addresses:
0x1000-0x10000: not zeroed, buthexdumpshowed that the only data in this region is the pattern written by the EFI application. Hence, this region was not used by Linux or bootloader.0x100000-0x800000: 524288 bytes zeroed and 6815744 not zeroed. All non-zeroed bytes contain the pattern written by the EFI application. Hence, these bytes were not used by Linux or bootloader.0x1600000-0x7bd20000: 2054160384 zeroed and 131072 not zeroed. Some of not zeroed addresses contain some data. Other addresses contain the pattern.0x7bd48000-0x7d8d7000: zeroed.0x7f5ee000-0x7f5f4000: zeroed.0x7fc00000-0x7fc70000: zeroed.0x7fd20000-0x7fd2d000: zeroed.0x100000000-0x180000000: this region is very fragmented in terms of zeroed and not zeroed addresses. 1892876288 bytes was zeroed and 254607360 was not zeroed.
And the attack shows negative results:
|
|
Cold boot attack both stages
This time /proc/iomem shows the Linux kernel is placed under the following
addresses:
|
|
Analysis of other addresses:
0x1000-0x10000: not zeroed, buthexdumpshowed that the only data in this region is the pattern written by the EFI application. Hence, this region was not used by Linux or bootloader.0x100000-0x800000: 524288 bytes zeroed and 6815744 not zeroed. All non-zeroed bytes contain the pattern written by the EFI application. Hence, these bytes were not used by Linux or bootloader.0x1600000-0x7bd20000: 84606976 zeroed and 1969684480 not zeroed. Most of non-zeroed addresses contain some data.0x7bd48000-0x7d922000: not zeroed, contains data.0x7f5ee000-0x7f5f4000: not zeroed, contains data.0x7fc00000-0x7fc70000: not zeroed, contains data.0x7fd20000-0x7fd2d000: not zeroed, contains data.0x100000000-0x180000000: this region is very fragmented in terms of zeroed and not zeroed addresses. 319815680 bytes was zeroed and 1827667968 was not zeroed. Still a lot of processes data not zeroed.
Yet another pack of stolen LUKS keys:
|
|
Summary
The ram-wipe software solution could protect from warm boot attacks, but:
ram-wipecannot wipe RAM used by kernel, because it is an application that is not allowed to modify Linux kernel address space. Hence, if your secrets are stored inside kernel address space, theram-wipewill not help protecting them from a warm boot attack.ram-wipewipes memory previously used by some processes, but it cannot wipe RAM of currently used processes (you can even notice OOM Killer logs whenram-wipetries to wipe RAM unassigned to it). Hence, you should make sure thatram-wipecan and will wipe RAM, where your process left your secrets. This is why it’s important to show the difference between zeroed and non-zeroed addresses in RAM during tests -ram-wipedoes not just fail to wipe the kernel address space, but also leaves parts of other processes' memory unwiped!
What about cold boot attacks? The tests showed that ram-wipe does not wipe
anything during cold boot attacks (the zeroes at some addresses in RAM are
caused by some processes zeroing out memory). The reason for this is hidden in
the design: as discussed above, ram-wipe relies on proper system
reboot/shutdown/halt sequences and if these sequences are not executed (what is
true for cold boot attack) - ram-wipe wiping stages will not be launched.
Hence, ram-wipe will not protect from cold boot attacks!
Thoughts on how to protect your secrets from cold boot attacks:
- Make
ram-wipewipe RAM on boot too. This will make sure, that Linux userspace processes will not get access to secrets leftovers in RAM. This will need building a chain of trust up to Linux kernel and RAM disk though, because there will be no point in wiping during Linux boot if an attacker will boot malware instead of the needed kernel. - Wipe RAM earlier in the boot sequence. For example, make firmware wipe RAM for you. This will make sure RAM is being wiped as soon as possible during boot and that the program that is responsible for this is fully trusted and has a very small TCB.
Other improvements include fixing the printing issues, making sure that secrets are being wiped and rewriting shell scripts, making sure that they are solid and trustworthy (for example, by following already classic lectures on how to write secure and stable shell scripts).
During the review process of this blog post Kicksecure announced improvements based on this blog post, feel free to check it and suggest any further changes that will bring all of us closer to more secure components and solutions.
Thank you for reading this blog post, and see you again soon!
Unlock the full potential of your hardware and secure your firmware with the
experts at 3mdeb! If you’re looking to boost your product’s performance and
protect it from potential security threats, our team is here to help.
Schedule a call with us
or drop us an email at contact@3mdeb.com to start unlocking the hidden
benefits of your hardware. And if you want to stay up-to-date on all things
firmware security and optimization, be sure to
sign up for our newsletter.
Don’t let your hardware hold you back, work with 3mdeb to achieve more!
