Yocto guide
TODO: change test build qemu solution to file copy, start wiki publish
Introduction
Hey today I'm going to trying a new format: Writing notes! I'm going to try and accomplish the following tasks today:
- Create a rootfs - Create an initrd - Create a qemu virt kernel - Package this is in to a distro and qemu virt BSP
I understand Yocto's poky distribution already does this, but this is more about learning the process using OpenEmbedded.
Chapter 1: Download
The first thing we want to do is make a directory for our project:
mkdir oe-test cd oe-test
I named mine 'oe-test' but you can name it what you want.
Next we download the code. We do this using Git so we can easily switch to a version on update.
git clone https://git.openembedded.org/bitbake git clone https://git.openembedded.org/openembedded-core
Now we want to switch to specific tags:
git -C bitbake switch --detach 2.8.0 git -C openembedded-core switch --detach 2024-04-scarthgap
These are both the latest releases as of writing. I found these specific URLs and settings by digging through their online GitWeb: https://git.openembedded.org
This combination only gives us the bare minimum required for building a virtual system. We will add our own layers for anything more.
As we are building we will also source the build environment script:
source openembedded-core/oe-init-build-env build
This puts us in our build directory named 'build'.
Chapter 2: Inspection
Let's have a look at the files we've downloaded and get an idea for how this project fits together and works:
First, a top level map:
bitbake - The build system openembedded-core - Bitbake recipes and documentation build - Build output
The core of OpenEmbedded is the BitBake build system, so it's worth taking some time to understand it.
BitBake is a bit like 'make' where you tell it to build a specific file, or in this case, recipe. It looks at the recipe, and finds recipes that recipe needs and builds those first, and so on. It can do these in parallel, and cache results. Standard building stuff.
What makes BitBake special is that you can change recipes using other recipes or configuration files without modifying the original. This is done using configurations or append files. A set of recipes and modifications can be stored in a separate directory as a 'layer', which is the intended way of organizing files using BitBake.
Let's look at a real world example: openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb
This package has the ability to enable or disable PAM using the 'DISTRO_FEATURES' variable. We can set this value:
- Per-build in our build configuration - Per-machine in our machine configuration - Per-distro in our distro configuration - Per-package in a .bbappend for dropbear
I mentioned above the concept of 'machine' and 'distro'. These are not BitBake concepts but concepts created by openembedded-core. Let's look at some interesting parts of openembedded-core. We see two layers:
meta - The core recipes used for creating a Linux distro meta-skeleton - An example layer with configs and packages
By default we only use the first layer. Inside that layer is:
classes* - BitBake code re-used in recipe types recipes* - BitBake recipes to build conf/ - Global configuration settings conf/distro/ - Distro configuration settings conf/machine/ - Machine configuration settings
The concept of distributions and different machines are just configuration files that set package variables. It's useful to separate these out as it means you can modify packages, distros and machines separately during development without needing to change multiple configurations at once like in a system like Buildroot.
Selecting which distro and machine to use are done using variables, much like any other aspect of OpenEmbedded. oe-init-build-env sets up the build directory with a conf/local.conf file that has some defaults.
This information should give you a basic enough understanding to follow along, but I highly recommend reading the full Yocto and BitBake manuals:
https://docs.yoctoproject.org/overview-manual/index.html https://docs.yoctoproject.org/ref-manual/index.html https://docs.yoctoproject.org/bitbake/
The development manuals are helpful too:
https://docs.yoctoproject.org/bsp-guide/index.html https://docs.yoctoproject.org/dev-manual/index.html
This system is much more complicated than something like Buildroot which only requires specifying manual entries to build, but it solves a lot of problems Buildroot introduces such as keeping configurations in sync and building being able to build multiple roots.
Chapter 3: Test build
Starting off, let's build the minimal image:
bitbake core-image-minimal
This will take a long time. Long enough that I should've done this while writing the previous chapter...
It looks like on my machine building QEMU failed! This stops building everything else including the kernel, but I don't want that. So instead I should run and inspect build failures after the build:
bitbake --continue core-image-minimal
The full build log is available in the build directory:
build/tmp-glibc/log/cooker/qemux86-64/console-latest.log
In this case it says:
ERROR: qemu-system-native-8.2.1-r0 do_compile: oe_runmake failed
We can get a devshell and run the task ourselves like this:
bitbake qemu-system-native -c devshell ../temp/run.do_compile # Run this in the devshell
In my case it gives me this error:
/usr/lib/libgdk_pixbuf-2.0.so.0: undefined reference to `g_once_init_leave_pointer'
This is suspicious: OpenEmbedded is trying to mix its own built libraries with my host libgdk_pixbuf and getting confused as mine is a newer version that uses a new symbol. Builds should never link with files in the host operating system. This type of issue is known as a leak, and are usually tricky to troubleshoot. Let's try anyway.
Looking in ../build I found that pixbuf is mentioned in meson-logs/meson-log.txt, it is added from the output of this command:
/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/bin/pkg-config --cflags gvnc-1.0
Running that command in the devshell gives an error, so the devshell is not having information leaked in to it from the host. So something must be happening when Meson is building to introduce a leak.
Looking at the manual page, pkg-config uses environment variables to help find packages. Looking in the meson log I found this:
env[PKG_CONFIG_PATH]: /home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/lib/pkgconfig:/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig
While most of the PKG_CONFIG_PATH is correct, the end has /usr/lib/pkgconfig and /usr/share/pkgconfig. This will cause a leak!
I searched QEMU's source code for 'PKG_CONFIG_PATH' but didn't find anything, so this leak is most likely from OpenEmbedded somewhere. There's quite a lot of files to look for in OpenEmbedded, so it would be a lot of work trying to find where the leak is.
Luckily, we can ask bitbake for the recipe's environment:
bitbake -e qemu-system-native > env
Looking in the file we quickly see this:
# line: 158, file: /home/jookia/yocto-test/openembedded-core/meta/recipes-devtools/qemu/qemu.inc do_configure() { # Append build host pkg-config paths for native target since the host may provide sdl BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH fi
In the qemu.inc file we can see the matching code:
do_configure:prepend:class-native() { # Append build host pkg-config paths for native target since the host may provide sdl BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH fi }
Fixing this can't be done with a configuration, so we will need to create a layer and create a bbappend file. First let's create a layer from the build directory:
bitbake-layers create-layer ../meta-bugfixes bitbake-layers add-layer ../meta-bugfixes
Then in the meta-bugfixes directory we can add a bbappend:
mkdir -p recipes-devtools/qemu cat <<EOF >recipes-devtools/qemu/qemu-system-native_%.bbappend do_configure:remove() { # Append build host pkg-config paths for native target since the host may provide sdl BHOST_PKGCONFIG_PATH=\$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") if [ ! -z "\$BHOST_PKGCONFIG_PATH" ]; then export PKG_CONFIG_PATH=\$PKG_CONFIG_PATH:\$BHOST_PKGCONFIG_PATH fi } EOF
Note three things:
- The filename has '%' instead of the version number - do_configure_remove() is used instead of
do_configure:prepend:class-native()
- This removal only removes matching shell script lines - The extra slashes introduced are just so making the file using cat
works
This solution here is not ideal, it would be better to report a bug and have this fixed properly. But it's a good example of the tools available for solving issues like this in a pinch.
In our build directory we can now run:
bitbake -e qemu-system-native > env
After confirming the change was made and applied we can finish the build:
bitbake qemu-system-native
On my machine this compiles without error. We can now test the image using the included runqemu program:
runqemu nographic slirp core-image-minimal
This will launch the image using an emulator in our terminal. The username is 'root'. Hitting ctrl-a ctrl-x will stop the emulator.
We can look at the image contents in build/tmp-glibc/deploy/images/qemux86-64. It contains files such as:
bzImage-qemux86-64.bin - The kernel image core-image-minimal-qemux86-64.rootfs.ext4 core-image-minimal-qemux86-64.rootfs.manifest core-image-minimal-qemux86-64.rootfs.qemuboot.conf core-image-minimal-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-qemux86-64.rootfs.tar.bz2 core-image-minimal-qemux86-64.rootfs.testdata.json modules-qemux86-64.tgz
We can also see that all the build Linux software is packaged in build/tmp-glibc/deploy/ipk.
To build another image we can run:
bitbake core-image-minimal-dev runqemu nographic slirp core-image-minimal-dev
This creates an identical image with debug symbols. We find a set of files for a new rootfs next to the other ones:
core-image-minimal-dev-qemux86-64.rootfs.ext4 core-image-minimal-dev-qemux86-64.rootfs.manifest core-image-minimal-dev-qemux86-64.rootfs.qemuboot.conf core-image-minimal-dev-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-dev-qemux86-64.rootfs.tar.bz2 core-image-minimal-dev-qemux86-64.rootfs.testdata.json
Chapter 4: Multiple builds
While you can build multiple root filesystems, that's about as far multiple outputs go. If you need to build for a different machine or a different distro you will need to use another configuration file.
Bitbake does support a way to use multiple configuration files, but I'm not exactly sure why you would want to use it instead of multiple build directories, especially if you have to do multi-architecture builds.
The first thing we want to do is re-locate our build cache and downloads:
cd build mv sstate-cache downloads ..
They will now be in our oe-test directory. Next, open up build/conf/local.conf and add these lines to the top:
DL_DIR ?= "${TOPDIR}/../downloads" SSTATE_DIR ?= "${TOPDIR}/../sstate-cache"
This will save a lot of time for the next step where we create a new build directory. In the oe-test directory run:
source openembedded-core/oe-init-build-env build2 cd build2 bitbake-layers add-layer ../meta-bugfixes
Then perform the same edits to build/conf/local.conf to set DL_DIR and SSTATE_DIR. But also add this line to enable systemd:
INIT_MANAGER = "systemd"
Because of the shared state directory, this should re-use a lot of already built components. Let's build:
bitbake core-image-minimal
Indeed it did, but I found it was spending time rebuilding gcc-cross-x86_64! That would mean rebuilding basically everything else too. Why? I canceled the build immediately to look.
In each build directory I did this:
bitbake -e gcc-cross-x86_64 | grep -v /home > cross.env
Then in the main directory I ran:
diff build/cross.env build2/cross.env
This answers my question of why the compiler is being rebuilt pretty fast:
11338c11337 < #define STANDARD_STARTFILE_PREFIX_1 "/usr/lib/" --- > #define STANDARD_STARTFILE_PREFIX_1 "/lib/" 11340c11339 < #define SYSTEMLIBS_DIR "/usr/lib/" --- > #define SYSTEMLIBS_DIR "/lib/"
In retrospect it's obvious: systemd requires a merged /usr, so the compiler will have to put its libraries in /usr/lib. This requires a rebuild of the compiler and probably everything else! Oh well.
After building again:
bitbake core-image-minimal
I can now run the image in QEMU and verify it works:
runqemu nographic slirp core-image-minimal
The image boots to systemd managed system. Success!
Chapter 5: Recipes
one configuration of a recipe at once tmp-glibc/deploy/images/qemux86-64/core-image-minimal-dev-qemux86-64.rootfs.manifest oe-pkg something
bitbake -g core-image-minimal
bitbake -s bitbake-layers show-recipes bitbake-layers show-recipes -r
bitbake-layers show-recipes -r | grep linux
linux-yocto-tiny (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-tiny to enable it) linux-yocto-tiny (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-tiny to enable it)
PACKAGECONFIG
MACHINE_FEATURES DISTRO_FEATURES EXTRA_IMAGE_FEATURES
pn-buildlist task-depends.dot
- where is this recipe created
- what variables can we tweak?
- what variables can we tweak?
enabling a feature appending a feature
Chapter X: Distros and Machine
distro/machine layers
Chapter X: Packaging
oe-pkgdata-util
- SDK
- TODO: multi configs?
- todo nfsroot, unfsd?
https://stackoverflow.com/questions/47429670/manually-building-a-kernel-source-from-yocto-build SRC_URI = "git:///path-to-linux-source/.git/;branch=${KBRANCH};protocol=file"
INHERIT += "externalsrc" EXTERNALSRC:pn-myrecipe = "/path/to/my/source/tree"
wic genimage
packaging a project:
mkdir -p ../meta-bugfixes/recipes-general/evtone/ recipetool create -B main -S v1.4 https://git.lumina-sensum.com/git/Jookia/evtone.git -o ../meta-bugfixes/recipes-general/evtone/evtone_1.4.bb bitbake evtone
- add 'inherit meson', remove configure stuff
PV v1.4 tmp-glibc/work/core2-64-oe-linux/evtone/1.4/image tmp-glibc/deploy/ipk/core2-64/evtone changing version removes the old ipk in deploy/
Chapter X: Root filesystems
- what components are in this image
- multiple kernels? - build only rootfs - build only kernel
- adding evtone to the image
- TODO: introspect and ask:
- how do we modify it for a different machine? - what is an image? - can we build just the rootfs?
- initrd
Chapter X: Mainline kernel
Chapter X: Bootloader
Chapter X: Disk image
Chapter X: Board layer
Chapter X: BSP layer
Chapter X: Development
- using a custom srcdir for a package
- sdks kind of aren't needed for us