Yocto guide
TODO: clean up formatting, maybe use inline code formatting?
In this guide I want to accomplish the following embedded Linux tasks using Yocto:
- Create a rootfs
- Create an initramfs
- Create a qemu virt kernel
- Package this in to proper layers
Yocto offers a starting distribution named Poky but for learning purposes I'll be using OpenEmbedded and BitBake directly.
Downloading code[edit | edit source]
The first thing we want to do is make a directory for our project:
$ mkdir oe-test $ cd oe-test
I named mine 'oe-test' but you can name it what you want.
Next we download the code. We do this using Git so we can easily switch to a version on update.
$ git clone https://git.openembedded.org/bitbake $ git clone https://git.openembedded.org/openembedded-core
Now we want to switch to specific tags:
$ git -C bitbake switch --detach 2.8.0 $ git -C openembedded-core switch --detach 2024-04-scarthgap
These are both the latest releases as of writing. I found these specific URLs and settings by digging through their online GitWeb: https://git.openembedded.org
This combination only gives us the bare minimum required for building a virtual system. We will add our own layers for anything more.
As we are building we will also source the build environment script:
$ source openembedded-core/oe-init-build-env build
This puts us in our build directory named 'build'.
Inspection[edit | edit source]
Let's have a look at the files we've downloaded and get an idea for how this project fits together and works:
First, a top level map:
- bitbake - The build system
- openembedded-core - BitBake recipes and documentation
- build - Build output
The core of OpenEmbedded is the BitBake build system, so it's worth taking some time to understand it.
BitBake is a bit like 'make' where you tell it to build a specific file, or in this case, recipe. It looks at the recipe, and finds recipes that recipe needs and builds those first, and so on. It can do these in parallel, and cache results. Standard building stuff.
What makes BitBake special is that you can change recipes using other recipes or configuration files without modifying the original. This is done using configurations or append files. A set of recipes and modifications can be stored in a separate directory as a 'layer', which is the intended way of organizing files using BitBake.
Let's look at a real world example: openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb
This package has the ability to enable or disable PAM using the 'DISTRO_FEATURES' variable. We can set this value:
- Per-build in our build configuration
- Per-machine in our machine configuration
- Per-distro in our distro configuration
- Per-package in a .bbappend for dropbear
I mentioned above the concept of 'machine' and 'distro'. These are not BitBake concepts but concepts created by openembedded-core.
Let's look at some interesting parts of openembedded-core. We see two layers:
- meta - The core recipes used for creating a Linux distro
- meta-skeleton - An example layer with configs and packages
By default we only use the first layer. Inside that layer is:
- classes* - BitBake code re-used in recipe types
- recipes* - BitBake recipes to build
- conf/ - Global configuration settings
- conf/distro/ - Distro configuration settings
- conf/machine/ - Machine configuration settings
The concept of distributions and different machines are just configuration files that set package variables. It's useful to separate these out as it means you can modify packages, distros and machines separately during development without needing to change multiple configurations at once like in a system like Buildroot.
Selecting which distro and machine to use are done using variables, much like any other aspect of OpenEmbedded. oe-init-build-env sets up the build directory with a conf/local.conf file that has some defaults.
This information should give you a basic enough understanding to follow along, but I highly recommend reading the full Yocto and BitBake manuals:
The development manuals are helpful too:
This system is much more complicated than something like Buildroot which only requires specifying manual entries to build, but it solves a lot of problems Buildroot introduces such as keeping configurations in sync and building being able to build multiple roots.
First build[edit | edit source]
Starting off, let's build the minimal image:
$ bitbake core-image-minimal
This will take a long time. Long enough that I should've done this while writing the previous chapter...
It looks like on my machine building QEMU failed! This stops building everything else including the kernel, but I don't want that. So instead I should run and inspect build failures after the build:
$ bitbake --continue core-image-minimal
The full build log is available in the build directory: build/tmp-glibc/log/cooker/qemux86-64/console-latest.log
In this case it says:
ERROR: qemu-system-native-8.2.1-r0 do_compile: oe_runmake failed
We can get a devshell and run the task ourselves like this:
$ bitbake qemu-system-native -c devshell $ ../temp/run.do_compile # Run this in the devshell
In my case it gives me this error:
/usr/lib/libgdk_pixbuf-2.0.so.0: undefined reference to `g_once_init_leave_pointer'
This is suspicious: OpenEmbedded is trying to mix its own built libraries with my host libgdk_pixbuf and getting confused as mine is a newer version that uses a new symbol.
Builds should never link with files in the host operating system. This type of issue is known as a leak, and are usually tricky to troubleshoot. Let's try anyway.
In the devshell looking in ../build I found that pixbuf is mentioned in meson-logs/meson-log.txt, it is added from the output of this command:
/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/bin/pkg-config --cflags gvnc-1.0
Running that command in the devshell gives an error, so the devshell is not having information leaked in to it from the host. So something must be happening when Meson is building to introduce a leak.
Looking at the manual page, pkg-config uses environment variables to help find packages. Checking the meson log I found this:
env[PKG_CONFIG_PATH]: /home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/lib/pkgconfig:/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig
While most of the PKG_CONFIG_PATH is correct, the end has /usr/lib/pkgconfig and /usr/share/pkgconfig. This will cause a leak!
I searched QEMU's source code for PKG_CONFIG_PATH but didn't find anything, so this leak is most likely from OpenEmbedded somewhere. There's quite a lot of files to look for in OpenEmbedded, so it would be a lot of work trying to find where the leak is.
Luckily, we can ask bitbake for the recipe's environment:
bitbake -e qemu-system-native > env
Looking in the file we quickly see this:
# line: 158, file: /home/jookia/oe-test/openembedded-core/meta/recipes-devtools/qemu/qemu.inc
do_configure() {
# Append build host pkg-config paths for native target since the host may provide sdl
BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "")
if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
fi
In the qemu.inc file we can see the matching code:
do_configure:prepend:class-native() {
# Append build host pkg-config paths for native target since the host may provide sdl
BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "")
if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
fi
}
For now we're just going to remove the block of code from the qemu.inc file. Directly modifying openembedded-core is easy but often a bad idea, so we'll go through some better options later in this guide.
After removing the code we can check the environment again:
$ bitbake -e qemu-system-native > env
After confirming the change was made by looking for PKG_CONFIG_PATH and applied we can finish the build:
$ bitbake qemu-system-native
On my machine this compiles without error. We can now test the image using the included runqemu program:
$ runqemu nographic slirp core-image-minimal
This will launch the image using an emulator in our terminal. The username is 'root'. Hitting ctrl-a ctrl-x will stop the emulator.
We can look at the image contents in build/tmp-glibc/deploy/images/qemux86-64. It contains files such as:
bzImage-qemux86-64.bin - The kernel image core-image-minimal-qemux86-64.rootfs.ext4 core-image-minimal-qemux86-64.rootfs.manifest core-image-minimal-qemux86-64.rootfs.qemuboot.conf core-image-minimal-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-qemux86-64.rootfs.tar.bz2 core-image-minimal-qemux86-64.rootfs.testdata.json modules-qemux86-64.tgz
We can also see that all the build Linux software is packaged in build/tmp-glibc/deploy/ipk.
To build another image we can run:
$ bitbake core-image-minimal-dev $ runqemu nographic slirp core-image-minimal-dev
This creates an identical image with debug symbols. We find a set of files for a new rootfs next to the other ones:
core-image-minimal-dev-qemux86-64.rootfs.ext4 core-image-minimal-dev-qemux86-64.rootfs.manifest core-image-minimal-dev-qemux86-64.rootfs.qemuboot.conf core-image-minimal-dev-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-dev-qemux86-64.rootfs.tar.bz2 core-image-minimal-dev-qemux86-64.rootfs.testdata.json
So we have managed to build a single kernel and two root filesystems.
Multiple builds[edit | edit source]
While you can build multiple root filesystems, that's about as far multiple outputs go. If you need to build for a different machine or a different distro you will need to use another configuration file.
BitBake does support a way to use multiple configuration files, but I'm not exactly sure why you would want to use it instead of multiple build directories, especially if you have to do multi-architecture builds.
The first thing we want to do is re-locate our build cache and downloads:
$ cd build $ mv sstate-cache downloads ..
They will now be in our oe-test directory. Next, open up build/conf/local.conf and add these lines to the top:
DL_DIR ?= "${TOPDIR}/../downloads"
SSTATE_DIR ?= "${TOPDIR}/../sstate-cache"
This will save a lot of time for the next step where we create a new build directory. In the oe-test directory run:
$ source openembedded-core/oe-init-build-env build2
Then perform the same edits to build/conf/local.conf to set DL_DIR and SSTATE_DIR. But also add this line to enable systemd:
INIT_MANAGER = "systemd"
Because of the shared state directory, this should re-use a lot of already built components. Let's build:
$ bitbake core-image-minimal
Indeed it did, but I found it was spending time rebuilding gcc-cross-x86_64! That would mean rebuilding basically everything else too. Why? I cancelled the build immediately to look.
In each build directory I did this:
$ bitbake -e gcc-cross-x86_64 | grep -v /home > cross.env
Then in the main directory I ran:
$ diff build/cross.env build2/cross.env
This answers my question of why the compiler is being rebuilt pretty fast:
11338c11337 < #define STANDARD_STARTFILE_PREFIX_1 "/usr/lib/" --- > #define STANDARD_STARTFILE_PREFIX_1 "/lib/" 11340c11339 < #define SYSTEMLIBS_DIR "/usr/lib/" --- > #define SYSTEMLIBS_DIR "/lib/"
In retrospect it's obvious: systemd requires a merged /usr, so the compiler will have to put its libraries in /usr/lib. This requires a rebuild of the compiler and probably everything else! Oh well.
After building again:
$ bitbake core-image-minimal
I can now run the image in QEMU and verify it works:
$ runqemu nographic slirp core-image-minimal
The image boots to systemd managed system. Success!
Recipes[edit | edit source]
We can use oe-pkgdata-util to list and inspect built packages like this:
$ oe-pkg-data-util list-pkgs | grep zlib
zlib
zlib-dbg
zlib-dev
zlib-doc
zlib-src
zlib-staticdev
$ oe-pkg-data-util pkg-info zlib-dev
zlib-dev 1.3.1-r0 zlib 1.3.1-r0 113538
$ oe-pkg-data-util list-pkg-files zlib-dev
zlib-dev:
/usr/include/zconf.h
/usr/include/zlib.h
/usr/lib/libz.so
/usr/lib/pkgconfig/zlib.pc
We can also find the list of packages for a particular root filesystem in its manfiest file, for example:
$ grep 'openssl' tmp-glibc/deploy/images/qemux86-64/core-image-minimal-dev-qemux86-64.rootfs.manifest openssl-conf core2-64 3.2.1-r0 openssl-dev core2-64 3.2.1-r0 openssl-ossl-module-legacy core2-64 3.2.1-r0
While this is useful for dealing with built packages and images, most work is done with BitBake recipes. We can list buildable recipes like so:
$ bitbake -s | grep ^linux-yocto linux-yocto :6.6.23+git-r0
We can list all recipes available like this:
$ bitbake-layers show-recipes -r | grep ^linux-yocto linux-yocto linux-yocto-dev (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-dev to enable it) linux-yocto-rt (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-rt to enable it) linux-yocto-tiny (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-tiny to enable it)
In this case we can only build linux-yocto but there are other packages available.
Listing dependencies of a recipe is a bit trickier. You can dump a recipe's dependency graph and view it like so:
$ bitbake -g core-image-minimal # pn-buildlist now lists all recipes required to build core-image-minimal # task-depends.dot lists all tasks required to build core-image-minimal
But this lists every dependency, not just immediate. To list immediate dependencies run:
$ bitbake -e vim | grep -P '^R?DEPENDS.*='
DEPENDS="pkgconfig-native autoconf-native automake-native libtool-native libtool-cross virtual/x86_64-oe-linux-gcc virtual/x86_64-oe-linux-compilerlibs virtual/libc ncurses gettext-native desktop-file-utils acl gtk+3 xt virtual/update-alternatives"
RDEPENDS:${KERNEL_PACKAGE_NAME}-base=""
RDEPENDS:vim="ncurses-terminfo-base vim-xxd"
RDEPENDS:vim-staticdev="vim-dev (= 9.1.0114-r0)"
Two types of dependencies are shown: DEPENDS and RDEPENDS. DEPENDS is for build time dependencies, RDEPENDS is for runtime dependencies. For our purposes we only care about the RDEPENDS with the recipe name: "RDEPENDS:vim".
It can also be useful to look at reverse dependencies. For example, to find out where busybox is being packaged or added to a build sysroot:
$ bitbake -g core-image-minimal $ grep -P ' -> "busybox.do_(packagedata|populate_sysroot)' task-depends.dot "core-image-minimal.do_rootfs" -> "sqlite3.do_packagedata" "python3.do_package" -> "sqlite3.do_packagedata" "python3.do_package_write_ipk" -> "sqlite3.do_packagedata" "python3.do_prepare_recipe_sysroot" -> "sqlite3.do_populate_sysroot" "sqlite3.do_create_spdx" -> "sqlite3.do_packagedata" "sqlite3.do_package_qa" -> "sqlite3.do_packagedata" "sqlite3.do_package_write_ipk" -> "sqlite3.do_packagedata"
Note that these BitBake commands will only tell us information about the package based on the overall configuration of the system. They won't tell you about dependencies that aren't enabled. Generally these are enabled or disabled using PACKAGECONFIG, so by looking at that variable we can find possible dependencies.
Here's an example of finding available features and dependencies for dropbear:
$ bitbake -e dropbear | sed '1,/^# $PACKAGECONFIG/d;/PACKAGECONFIG=/,$d'
# :append[pn-qemu-system-native] /home/jookia/oe-test/build/conf/local.conf:215
# " sdl"
# set /home/jookia/oe-test/openembedded-core/meta/conf/documentation.conf:321
# [doc] "This variable provides a means of enabling or disabling features of a recipe on a per-recipe basis."
# set? /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:51
# "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}"
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:52
# [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}"
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:53
# [system-libtom] "--disable-bundled-libtom,--enable-bundled-libtom,libtommath libtomcrypt"
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:54
# [disable-weak-ciphers] ""
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:55
# [enable-x11-forwarding] ""
# pre-expansion value:
# "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}"
The fancy sed command is just to find the PACKAGECONFIG block in the bitbake -e dump. You can look for it manually.
Let's focus on the pam feature in the output:
# [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}"
The bracketed text is the feature and the quoted text is the PACKAGECONFIG variable with elements separated by commas in this style:
PACKAGECONFIG[foo] = "--enable-foo,--disable-foo,foo_depends,foo_runtime_depends,foo_runtime_recommends,foo_conflict_packageconfig"
It seems like the pam argument DEPENDS on libpam (the third column) and uses the PAM_PLUGINS variable for its RDEPENDS. This makes sense as PAM uses pluggable modules decided by configuration rather than this specific package.
You may also see the variables DISTRO_FEATURES or MACHINE_FEATURES used to set PACKAGECONFIG. These variables are set in configurations and used to make broad changes to many recipes at once rather than setting individual package features.
Layers[edit | edit source]
BitBake supports the concept of layers: Sets of BitBake recipes that can add to or override existing recipes. This allows developers and packagers to create sets of re-usable layers independent of each other that can be combined to make a final project.
By convention the layers start with the prefix 'meta-' and fall in to the following categories:
- Package layers that provide software to use
- Distro layers that configure the way the target system works
- Machine layers that add support for specific hardware
For this guide's project we will most likely need:
- The core OpenEmbedded layer
- A package layer for any new packages
- A distro layer that uses systemd and the packages
- A machine layer for the qemu virt device
As a side note, layers are not intended for modifying the code of existing recipes. You should still use regular development branches and patches for that. For instance, with the qemu recipe bug I found earlier I made a new Git branch for the small change. These patches are then ideally sent upstream to the parent project.
Packaging[edit | edit source]
packaging a project:
mkdir -p ../meta-bugfixes/recipes-general/evtone/ recipetool create -B main -S v1.4 https://git.lumina-sensum.com/git/Jookia/evtone.git -o ../meta-bugfixes/recipes-general/evtone/evtone_1.4.bb bitbake evtone
- add 'inherit meson', remove configure stuff
PV v1.4 tmp-glibc/work/core2-64-oe-linux/evtone/1.4/image tmp-glibc/deploy/ipk/core2-64/evtone changing version removes the old ipk in deploy/
Chapter X: Root filesystems
- what components are in this image
- multiple kernels? - build only rootfs - build only kernel
- adding evtone to the image
- TODO: introspect and ask:
- how do we modify it for a different machine? - what is an image? - can we build just the rootfs?
- initrd
Chapter X: Mainline kernel
Chapter X: Bootloader
Chapter X: Disk image
Chapter X: Board layer
Chapter X: BSP layer
Chapter X: Development
- using a custom srcdir for a package
- sdks kind of aren't needed for us
- SDK
- TODO: multi configs?
- todo nfsroot, unfsd?
https://stackoverflow.com/questions/47429670/manually-building-a-kernel-source-from-yocto-build SRC_URI = "git:///path-to-linux-source/.git/;branch=${KBRANCH};protocol=file"
INHERIT += "externalsrc" EXTERNALSRC:pn-myrecipe = "/path/to/my/source/tree"
wic genimage