sebastians site
Adventures in Open Source Hard and Software
Hey nixos, where's my I2C?
The HAM radio research group at the University Kaiserslautern runs the aerospectator, a public dump1090 instance. It has been first installed by a student as part of their seminar paper and has been kept running by the group ever since. Somewhere in 2019 I inherited the maintenance of the setup, because part of my professional skill set is prodding and poking Raspberry Pis to make them run reliably (I work on an embedded linux distro for CM3 and CM4 based hardware platforms). Also, everyone else was just getting frustrated by having to climb on the roof every few months to replace the SD-card.
One of my first changes was to create an ansible playbook to simply redeployment. I also added a piwatcher as hardware watchdog, to reset the old slightly underpowered Pi1 if it got stuck.
Fast-forward a couple of years: At some point in 2023 we decided to switch as much of the groups systems over to nixos and make them deployable with colmena. That setup proved less fragile than a wild mix debian versions with ansible. However, it also meant that we needed to get nix running on a Pi1 (with 23.05 that was not too complicated) and we needed to cross-compile everything all the time for that one system. However since we rarely changed the config for that system, it was workable. The piwatcher was a bit of challenge, but after learning how to wrap its userspace-tool and the necessary system configs into a flake it just worked.
And then nixos 24.05 attacked
While updating to 24.04 I was unfortunately unable to keep the cross compilation for the Pi1 working. After a lengthy, yet ultimately unsuccessfully, adventure in the engine room of nixos, it was decided to upgrade it, since Pi3 hardware is so cheap, that replacing the tired old Pi1 is probably the better option. It also enabled us to use aarch64 nixos caches, which meant a lot less cross-compiling.
I whipped up a quick config to generate a custom SD-card image with all our "usual stuff", and especially the tools to feed our watchdog, preinstalled.
1 {
2 description = "Build raspi 3 bootstrap images";
3 inputs = {
4 nixpkgs.url = "github:nixos/nixpkgs/nixos-24.05";
5 piwatcher.url = "git+ssh://forgejo@ourserver:/piwatcher.git";
6 };
7 outputs = { self, nixpkgs }:
8 let
9 pkgs = nixpkgs.legacyPackages.x86_64-linux;
10 pkgs_aarch64 = nixpkgs.legacyPackages.aarch64-linux;
11 in
12 {
13 formatter.x86_64-linux = pkgs.nixpkgs-fmt;
14 nixosConfigurations.rpi3 = nixpkgs.lib.nixosSystem {
15 modules = [
16 "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
17 {
18 nixpkgs.config.allowUnsupportedSystem = true;
19 nixpkgs.hostPlatform.system = "aarch64-linux";
20
21 imports = [
22 piwatcher.nixosModules.default
23 ];
24
25 # Stolen from nixos-hardware/blob/master/raspberry-pi/3/default.nix
26 boot.kernelPackages = pkgs_aarch64.linuxPackages_rpi3;
27 nixpkgs.overlays = [
28 (final: super: {
29 makeModulesClosure = x:
30 super.makeModulesClosure (x // { allowMissing = true; });
31 })
32 ];
33
34 services.openssh.enable = true;
35 users.users.root.openssh.authorizedKeys.keys = [
36 # Add keys here...
37 ];
38
39 # Talk to watchdog, to disable it
40 services.piwatcher = {
41 enable = true;
42 default-timeout = 0;
43 wakeup-timer = 0;
44 timeout = 0;
45 package = piwatcher.packages.x86_64-linux.cross-aarch64-linux;
46 };
47
48 environment.systemPackages = with pkgs_aarch64; [
49 i2c-tools
50 dtc
51 ];
52
53 system.stateVersion = "24.05";
54 }
55 ];
56 };
57 images.bootstrap-rpi3 =
58 self.nixosConfigurations.rpi3.config.system.build.sdImage;
59 };
60 }
And that's when I hit a really annoying roadblock.
The piwatcher flake wasn't working any more.
The Pi kept getting power cycled by the watchdog.
However after some investigation I noticed the piwatcher tool was working fine.
It just couldn't talk to the hardware.
There was no I2C device under /dev.
dmesg and lsmod both show i2c-dev is loaded and doing something.
So it must be a device tree issue.
On a regular Pi I'd set dtparam=i2c_arm=on in /boot/config.txt and the bus should magically appear after a reboot.
So ... what's the nixos way of doing that? There's got be an option for that!
Turns out there is boot.loader.raspberryPi.firmwareConfig.
It's description states These options are deprecated, unsupported, and may not work like expected.,
which is probably true as it does not change my config.txt at all.
The config.txt is generated by nixos is actually pretty generic and bare-bones:
1 [pi3]
2 kernel=u-boot-rpi3.bin
3
4 [pi02]
5 kernel=u-boot-rpi3.bin
6
7 [pi4]
8 kernel=u-boot-rpi4.bin
9 enable_gic=1
10 armstub=armstub8-gic.bin
11
12 # Otherwise the resolution will be weird in most cases, compared to
13 # what the pi3 firmware does by default.
14 disable_overscan=1
15
16 # Supported in newer board revisions
17 arm_boost=1
18
19 [cm4]
20 # Enable host mode on the 2711 built-in XHCI USB controller.
21 # This line should be removed,
22 # if the legacy DWC2 controller is required
23 # (e.g. for USB device mode) or if USB support is not required.
24 otg_mode=1
25
26 [all]
27 # Boot in 64-bit mode.
28 arm_64bit=1
29
30 # U-Boot needs this to work,
31 # regardless of whether UART is actually used or not.
32 # Look in arch/arm/mach-bcm283x/Kconfig
33 # in the U-Boot tree to see if this is still
34 # a requirement in the future.
35 enable_uart=1
36
37 # Prevent the firmware from smashing the framebuffer
38 # setup done by the mainline kernel
39 # when attempting to show low-voltage or
40 # overtemperature warnings.
41 avoid_warnings=1
Just editing config.txt to bypass nix also didn't help.
I added dtparam=i2c_arm=on and it didn't make a difference.
But since we are already looking at FIRMWARE partition, do you notice something?
There are a lot of dbt files, but they all have pi4 and cm4 in their name.
Where's the devicetree file for the Pi3?
$ ls
armstub8-gic.bin bootcode.bin fixup4db.dat start.elf start_cd.elf
bcm2711-rpi-4-b.dtb config.txt fixup4x.dat start4.elf start_db.elf
bcm2711-rpi-400.dtb fixup.dat fixup_cd.dat start4cd.elf start_x.elf
bcm2711-rpi-cm4.dtb fixup4.dat fixup_db.dat start4db.elf u-boot-rpi3.bin
bcm2711-rpi-cm4s.dtb fixup4cd.dat fixup_x.dat start4x.elf u-boot-rpi4.bin
Sidebar: Devicetrees
This, dear reader, is the point where my professionally experience in messing around with raspis comes into play. Seems like you've just won a little sidebar rant about devicetrees and how to get them into the linux kernel.
Traditionally on x86 and similar platforms the linux kernel would use BIOS, or these days UEFI interfaces to discover important hardware such as storage devices, network cards, videos cards, buses like USB. Your typical ARM SoC don't have a BIOS or UEFI (unless you are lucky enough to get to enjoy UEFI on aarch64). Instead, a devicetree is provided to the kernel as binary blob, somewhere in the RAM. The tree basically is a binary serialized representation of all the buses and attached peripherals, as well as some necessary setup parameters for your SoC. Usually, the devicetree is loaded by the bootloader along with the kernel and the initial ramdisk.
However, on the Raspberry Pi things can get a bit complicated, especially if there's a second stage bootloader like u-boot involved.
Normally without a second stage bootloader, the first stage bootloader embedded in the Pis ROM looks for the
first FAT32 formatted partition on the SD-Card, or in case of a compute module the onboard eMMC,
and load the file bootcode.bin from there.
There are also (presumably) newer versions of the bootloader that can boot from FAT16 as well,
but you never know which version you end up with on a Pi3.
The bootcode.bin is then execute by the "VideoCore GPU".
Yes that's right the GPU bootstraps the CPU. Somebody at Broadcom thought this was a sensible way of doing things.
This has some funny lesser known side effects, e.g. the overclocking the GPU,
but not the CPU can speed up boot times,
in a headless setup.
The Pi4 and Pi5 appear to load this code from an EEPROM instead.
After that continues to load start.elf, which brings up the actual ARM core,
reads config.txt, prepares a device tree and finally hands the execution over to the kernel.
See the official documentation for more info.
If u-boot is used a second stage bootloader, then u-boot.bin pretends to be the linux kernel, so that it can be executed by start.elf.
U-boot much like the linux kernel, needs to know to access the SoC peripheral,
e.g. mass storage to load the actual OS, a UART to display the display its console, or a network interface to perform a network boot.
For that it also needs a device tree.
On Pi3 and older u-boot has two possible sources for that:
It can use the device tree conveniently provided by the Raspberry Pi bootloader.
Alternatively, it can have its own devicetree, which is embedded into it during compilation.
After that it can forward one of those two trees to the kernel, or it can load a third tree from storage forward that one to the kernel.
On Pi4 and Pi5 things are little more complicated. Parameters like memory parameters and video timings are read from a ROM by the Raspberry Pi bootloader and added to the initial devicetree. This means all stage following the first stage bootloader have to reuse that devicetree, otherwise the linux kernel in the last stage will fail boot properly in some cases. Fortunately, you can just instruct u-boot to use the tree provided in RAM by the Raspberry Pi bootloader and then have pass that tree on to the linux kernel.
And that's almost all that you need to know about devicetrees.
There is one more thing though: device tree overlays.
Overlays are fragments of a device trees that can be added on top of a full tree,
overwriting some of its properties.
They can be applied either by a bootloader, e.g. the Raspberry Pi bootloader using dtoverlay in your config.txt or by u-boot using the ftd apply command.
Alternatively, if you are feeling extra adventurous you can also apply an overlay to the device tree currently in use,
after the kernel has booted.
If for some reason you can not use overlays in your setup, you can also just merge a devicetree blob with some overlays
at compile time using a command line utility.
Back to the problem at hand
As I found out, there is no device tree file for the Pi3 in the FIRMWARE partition.
However, the booted Pi3 knows about its network card, USB works and there even a console on the HDMI port,
so evidently it gets a device tree from somewhere.
Using dtc we can even decompile it back into a text and look at it:
$ dtc -I fs /sys/firmware/devicetree/base
[ ... a bunch of warnings that I removed ]
/dts-v1/;
/ {
#address-cells = <0x01>;
model = "Raspberry Pi 3 Model B+";
serial-number = "000000001d7c907a";
#size-cells = <0x01>;
interrupt-parent = <0x01>;
compatible = "raspberrypi,3-model-b-plus\0brcm,bcm2837";
fixedregulator_3v3 {
regulator-max-microvolt = <0x325aa0>;
regulator-always-on;
regulator-min-microvolt = <0x325aa0>;
regulator-name = "3v3";
compatible = "regulator-fixed";
phandle = <0x97>;
};
fixedregulator_5v0 {
regulator-max-microvolt = <0x4c4b40>;
regulator-always-on;
regulator-min-microvolt = <0x4c4b40>;
regulator-name = "5v0";
compatible = "regulator-fixed";
phandle = <0x98>;
};
memory@0 {
device_type = "memory";
reg = <0x00 0x3b400000>;
};
arm-pmu {
interrupts = <0x09 0x04>;
interrupt-parent = <0x18>;
compatible = "arm,cortex-a53-pmu\0arm,cortex-a7-pmu";
};
thermal-zones {
cpu-thermal {
polling-delay = <0x3e8>;
polling-delay-passive = <0x00>;
thermal-sensors = <0x02>;
phandle = <0x3c>;
coefficients = <0xfffffde6 0x64960>;
trips {
phandle = <0x3d>;
cpu-crit {
temperature = <0x1adb0>;
hysteresis = <0x00>;
type = "critical";
};
};
cooling-maps {
phandle = <0x3e>;
};
};
};
soc {
dma-ranges = <0xc0000000 0x00 0x3f000000 0x7e000000 0x3f000000 0x1000000>;
#address-cells = <0x01>;
#size-cells = <0x01>;
compatible = "simple-bus";
ranges = <0x7e000000 0x3f000000 0x1000000 0x40000000 0x40000000 0x1000>;
phandle = <0x3f>;
[...]
i2c@7e804000 {
pinctrl-names = "default";
#address-cells = <0x01>;
pinctrl-0 = <0x15>;
interrupts = <0x02 0x15>;
clocks = <0x08 0x14>;
#size-cells = <0x00>;
clock-frequency = <0x186a0>;
compatible = "brcm,bcm2835-i2c";
status = "disabled";
reg = <0x7e804000 0x1000>;
phandle = <0x2b>;
};
[...]
};
};
There is a i2c@7e804000 in it, but messing around with config.txt didn't change anything about it,
therefore it can't be the tree loaded by the Raspberry Pi bootloader.
Similarly, if it were loaded by u-boot I'd expect a dtb file somewhere on the FIRMWARE partition.
At that point I was totally stuck and did something that I should have done in the beginning:
I looked at the u-boot article in the nixos wiki.
As it turns out u-boot has a feature called
Generic Distro Configuration Concept,
which replace the imperative u-boot bootscripts with a simpler config file.
The way this is set up in nixos, u-boot looks for a file called extlinux/extlinux.conf on
all partitions on all storage devices it can find.
The file is located in /boot/extlinux/extlinux.conf when looking at the bootet OS on the pi.
$ cat /boot/extlinux/extlinux.conf
# Generated file, all changes will be lost on nixos-rebuild!
# Change this to e.g. nixos-42 to temporarily boot to an older configuration.
DEFAULT nixos-default
MENU TITLE ------------------------------------------------------------
TIMEOUT 50
LABEL nixos-default
MENU LABEL NixOS - Default
LINUX ../nixos/1w09bbijmmcw0hw2m9w0vycnzpkswrmi-linux-6.1.63-stable_20231123-Image
INITRD ../nixos/bcfkyw1mgp4ykpajw1cz6pbh1vag1ygv-initrd-linux-6.1.63-stable_20231123-initrd
APPEND init=/nix/store/r1cbbp0kdzi3k8apcfnfsx7rqxgn923y-nixos-system-aerospectator-24.05pre-git/init loglevel=4
FDTDIR ../nixos/ldxaavrzzc60izvc9q2mfqdfph76b8dw-device-tree-overlays
This tells u-boot where to find the kernel, the initrd and most importantly the devicetree.
And that's where I finally found the elusive bcm2837-rpi-3-b-plus.dtb that gets loaded for my Pi3.
$ ls -alh /boot/nixos/ldxaavrzzc*-device-tree-overlays/broadcom/
total 772K
dr-xr-xr-x 2 root root 4.0K Aug 10 11:47 .
dr-xr-xr-x 4 root root 4.0K Aug 10 11:47 ..
-r--r--r-- 1 root root 31K Aug 10 11:47 bcm2710-rpi-2-b.dtb
-r--r--r-- 1 root root 34K Aug 10 11:47 bcm2710-rpi-3-b-plus.dtb
-r--r--r-- 1 root root 33K Aug 10 11:47 bcm2710-rpi-3-b.dtb
-r--r--r-- 1 root root 31K Aug 10 11:47 bcm2710-rpi-cm3.dtb
-r--r--r-- 1 root root 32K Aug 10 11:47 bcm2710-rpi-zero-2-w.dtb
-r--r--r-- 1 root root 32K Aug 10 11:47 bcm2710-rpi-zero-2.dtb
-r--r--r-- 1 root root 54K Aug 10 11:47 bcm2711-rpi-4-b.dtb
-r--r--r-- 1 root root 54K Aug 10 11:47 bcm2711-rpi-400.dtb
-r--r--r-- 1 root root 38K Aug 10 11:47 bcm2711-rpi-cm4-io.dtb
-r--r--r-- 1 root root 55K Aug 10 11:47 bcm2711-rpi-cm4.dtb
-r--r--r-- 1 root root 51K Aug 10 11:47 bcm2711-rpi-cm4s.dtb
-r--r--r-- 1 root root 74K Aug 10 11:47 bcm2712-rpi-5-b.dtb
-r--r--r-- 1 root root 34K Aug 10 11:47 bcm2837-rpi-3-a-plus.dtb
-r--r--r-- 1 root root 34K Aug 10 11:47 bcm2837-rpi-3-b-plus.dtb
-r--r--r-- 1 root root 33K Aug 10 11:47 bcm2837-rpi-3-b.dtb
-r--r--r-- 1 root root 31K Aug 10 11:47 bcm2837-rpi-cm3.dtb
-r--r--r-- 1 root root 32K Aug 10 11:47 bcm2837-rpi-zero-2.dtb
-r--r--r-- 1 root root 54K Aug 10 11:47 bcm2838-rpi-4-b.dtb
Now I know that the devicetree is generated by nix somewhere and that it lives in the nix store,
along with the kernel.
That's pretty cool, because the FIRMWARE partition is usually not mounted and it's contents exist
out of the immutable world of the nix store.
It also means that there is no automatic way to update the FIRMWARE partition using nixos-rebuild
or in my case using colmena apply (at least none that I found, please do correct me on this).
However, with this setup only the bootloader lives in the firmware partition,
so I can update the kernel and the devicetree by just changing our configuration and reapplying it.
Moreover, it also means that the hardware.devicetree should just allow us to setup a overlay.
1 { config, pkgs, ... }: {
2 hardware.deviceTree.overlays = [
3 {
4 name = "enable-i2c";
5 dtsText = ''
6 /dts-v1/;
7 /plugin/;
8 / {
9 compatible = "brcm,bcm2837";
10 fragment@0 {
11 target = <&i2c1>;
12 __overlay__ {
13 status = "okay";
14 };
15 };
16 };
17 '';
18 }
19 ];
20 }
I don't want to go too much into the details of how overlays work,
but this snippet targets i2c1 which is just an alias for i2c@7e804000 from the devicetree snippet above
and flips the status property from disabled to okay.
That's all that dtparam=i2c_arm=on would do on a regular pi.
After applying the new config and rebooting, I suddenly have the missing device file. The piwatcher starts to work, and I could just do what any sane person would do, that is stop poking at it and enjoy that it finally works. Maybe touch some grass instead of computers for a change. Then again, a sane person would have not ended up in this situation in the first place. Let's see just how deep this rabbit hole actually is.
The missing bits
I've successfully set up an overlay. So the changes I made must be somewhere™ on that pi.
$ ls -R /boot/nixos/iclw8i0iwj7i0cdq34f48c29s1ianp4i-device-tree-overlays
/boot/nixos/iclw8i0iwj7i0cdq34f48c29s1ianp4i-device-tree-overlays:
broadcom overlays
/boot/nixos/iclw8i0iwj7i0cdq34f48c29s1ianp4i-device-tree-overlays/broadcom:
bcm2710-rpi-2-b.dtb bcm2710-rpi-zero-2-w.dtb bcm2711-rpi-cm4-io.dtb
bcm2837-rpi-3-a-plus.dtb bcm2837-rpi-zero-2.dtb bcm2710-rpi-3-b-plus.dtb
bcm2710-rpi-zero-2.dtb bcm2711-rpi-cm4.dtb bcm2837-rpi-3-b-plus.dtb
bcm2838-rpi-4-b.dtb bcm2710-rpi-3-b.dtb bcm2711-rpi-4-b.dtb
bcm2711-rpi-cm4s.dtb bcm2837-rpi-3-b.dtb bcm2710-rpi-cm3.dtb
bcm2711-rpi-400.dtb bcm2712-rpi-5-b.dtb bcm2837-rpi-cm3.dtb
/boot/nixos/iclw8i0iwj7i0cdq34f48c29s1ianp4i-device-tree-overlays/overlays:
hat_map.dtb overlay_map.dtb
That's ... odd. There is no enable-i2c.dto. There are no overlay files at all.
I could also trawl trough /nix using something like find /nix -iname *.dto ... or I could just take advantage of how nix works.
Looking the path it's clear there has to be a derivation called device-tree-overlays
somewhere in the nixpkgs repo.
I can just grep the entire repo for that string.
And bingo: pkgs/os-specific/linux/device-tree/default.nix#L28
1 { lib, stdenv, stdenvNoCC, dtc }: {
2 applyOverlays = (base: overlays': stdenvNoCC.mkDerivation {
3 name = "device-tree-overlays";
4 nativeBuildInputs = [ dtc ];
5 buildCommand = let
6 overlays = lib.toList overlays';
7 in ''
8 mkdir -p $out
9 cd "${base}"
10 find -L . -type f -name '*.dtb' -print0 \
11 | xargs -0 cp -v --no-preserve=mode --target-directory "$out" --parents
12
13 for dtb in $(find "$out" -type f -name '*.dtb'); do
14 dtbCompat=$(fdtget -t s "$dtb" / compatible 2>/dev/null || true)
15 # skip files without `compatible` string
16 test -z "$dtbCompat" && continue
17
18 ${lib.flip (lib.concatMapStringsSep "\n") overlays (o: ''
19 overlayCompat="$(fdtget -t s "${o.dtboFile}" / compatible)"
20
21 # skip incompatible and non-matching overlays
22 if [[ ! "$dtbCompat" =~ "$overlayCompat" ]]; then
23 echo "Skipping overlay ${o.name}: incompatible with $(basename "$dtb")"
24 elif ${if (o.filter == null) then "false" else ''
25 [[ "''${dtb//${o.filter}/}" == "$dtb" ]]
26 ''}
27 then
28 echo "Skipping overlay ${o.name}: filter does not match $(basename "$dtb")"
29 else
30 echo -n "Applying overlay ${o.name} to $(basename "$dtb")... "
31 mv "$dtb"{,.in}
32 fdtoverlay -o "$dtb" -i "$dtb.in" "${o.dtboFile}"
33 echo "ok"
34 rm "$dtb.in"
35 fi
36 '')}
37
38 done
39 '';
40 });
41 }
And there is it is on line 32.
The shell script that builds this derivation uses fdtoverlay
to apply all the overlays to the devicetree files at build time.
This means all overlays specified in my config will just be merged into bcm2837-rpi-3-b-plus.dtb while the derivation is build.
TL;DR
Finally, here's the short version:
- NixOS on the Pi3 (and older) uses u-boot to load the devicetree and discards the tree provided by the Raspberry Pi bootloader.
So nothing you do to the devicetree using
config.txtwill have any effect. - The devicetree and the kernel are loaded from the nix store, so they can just be updated along with everything else,
without ever touching the
FIRMWAREpartition. - Device tree overlays are applied at build time. So there is only one
bcm2837-rpi-3-b-plus.dtband no*.dtbofiles at all. - All the regular options to interact with the devicetree should just work™.
Also check out aerospectator.amateurfunk.uni-kl.de if you want to know what is flying in and out of the biggest American air force base on foreign soil. (Of course you'll only see planes with an active ADS-B transponder, so it's mostly just the boring stuff).
For comments you can use your fediverse account to reply to this toot.
Published: 15.09.2024 16:17 Updated: 15.09.2024 17:17