Yocto guide: Difference between revisions
(Initial WIP import) |
(→Recipes: Fill section) |
||
(6 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
TODO: | TODO: clean up formatting, maybe use inline <code>code</code> formatting? | ||
In this guide I want to accomplish the following embedded Linux tasks using Yocto: | |||
* Create a rootfs | |||
* Create an initramfs | |||
* Create a qemu virt kernel | |||
* Package this in to proper layers | |||
Yocto offers a starting distribution named Poky but for learning purposes I'll be using OpenEmbedded and BitBake directly. | |||
== Downloading code == | |||
The first thing we want to do is make a directory for our project: | The first thing we want to do is make a directory for our project: | ||
$ mkdir oe-test | |||
$ cd oe-test | |||
I named mine 'oe-test' but you can name it what you want. | I named mine 'oe-test' but you can name it what you want. | ||
Line 28: | Line 21: | ||
to a version on update. | to a version on update. | ||
$ git clone https://git.openembedded.org/bitbake | |||
$ git clone https://git.openembedded.org/openembedded-core | |||
Now we want to switch to specific tags: | Now we want to switch to specific tags: | ||
$ git -C bitbake switch --detach 2.8.0 | |||
$ git -C openembedded-core switch --detach 2024-04-scarthgap | |||
These are both the latest releases as of writing. I found these specific | These are both the latest releases as of writing. I found these specific URLs and settings by digging through their online GitWeb: https://git.openembedded.org | ||
URLs and settings by digging through their online GitWeb: | |||
https://git.openembedded.org | |||
This combination only gives us the bare minimum required for building | This combination only gives us the bare minimum required for building a virtual system. We will add our own layers for anything more. | ||
a virtual system. We will add our own layers for anything more. | |||
As we are building we will also source the build environment script: | As we are building we will also source the build environment script: | ||
$ source openembedded-core/oe-init-build-env build | |||
This puts us in our build directory named 'build'. | This puts us in our build directory named 'build'. | ||
== Inspection == | |||
Let's have a look at the files we've downloaded and get an idea for how this project fits together and works: | |||
First, a top level map: | |||
* bitbake - The build system | |||
* openembedded-core - BitBake recipes and documentation | |||
* build - Build output | |||
The core of OpenEmbedded is the BitBake build system, so it's worth taking some time to understand it. | |||
BitBake is a bit like 'make' where you tell it to build a specific file, or in this case, recipe. It looks at the recipe, and finds recipes that recipe needs and builds those first, and so on. It can do these in parallel, and cache results. Standard building stuff. | |||
BitBake is | What makes BitBake special is that you can change recipes using other recipes or configuration files without modifying the original. This is done using configurations or append files. A set of recipes and modifications can be stored in a separate directory as a 'layer', which is the intended way of organizing files using BitBake. | ||
or | |||
Let's look at a real world example: openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb | |||
This package has the ability to enable or disable PAM using the 'DISTRO_FEATURES' variable. We can set this value: | |||
* Per-build in our build configuration | |||
* Per-machine in our machine configuration | |||
* Per-distro in our distro configuration | |||
* Per-package in a .bbappend for dropbear | |||
I mentioned above the concept of 'machine' and 'distro'. These are not BitBake concepts but concepts created by openembedded-core. | |||
Let's look at some interesting parts of openembedded-core. We see two layers: | |||
at some interesting parts of openembedded-core. We see two layers: | |||
* meta - The core recipes used for creating a Linux distro | |||
* meta-skeleton - An example layer with configs and packages | |||
By default we only use the first layer. Inside that layer is: | By default we only use the first layer. Inside that layer is: | ||
* classes* - BitBake code re-used in recipe types | |||
* recipes* - BitBake recipes to build | |||
* conf/ - Global configuration settings | |||
* conf/distro/ - Distro configuration settings | |||
* conf/machine/ - Machine configuration settings | |||
The concept of distributions and different machines are just | The concept of distributions and different machines are just configuration files that set package variables. It's useful to separate these out as it means you can modify packages, distros and machines separately during development without needing to change multiple configurations at once like in a system like Buildroot. | ||
configuration files that set package variables. It's useful to separate | |||
these out as it means you can modify packages, distros and machines | |||
separately during development without needing to change multiple | |||
configurations at once like in a system like Buildroot. | |||
Selecting which distro and machine to use are done using variables, much | Selecting which distro and machine to use are done using variables, much like any other aspect of OpenEmbedded. oe-init-build-env sets up the build directory with a conf/local.conf file that has some defaults. | ||
like any other aspect of OpenEmbedded. oe-init-build-env sets up the | |||
build directory with a conf/local.conf file that has some defaults. | |||
This information should give you a basic enough understanding to follow | This information should give you a basic enough understanding to follow along, but I highly recommend reading the full Yocto and BitBake manuals: | ||
along, but I highly recommend reading the full Yocto and BitBake manuals: | |||
https://docs.yoctoproject.org/overview-manual/index.html | * [https://docs.yoctoproject.org/overview-manual/index.html Yocto Project Overview and Concepts Manual] | ||
https://docs.yoctoproject.org/ref-manual/index.html | * [https://docs.yoctoproject.org/ref-manual/index.html Yocto Project Reference Manual] | ||
https://docs.yoctoproject.org/bitbake/ | * [https://docs.yoctoproject.org/bitbake/ BitBake User Manual] | ||
The development manuals are helpful too: | The development manuals are helpful too: | ||
https://docs.yoctoproject.org/bsp-guide/index.html | * [https://docs.yoctoproject.org/bsp-guide/index.html Yocto Project Board Support Package Developer’s Guide] | ||
https://docs.yoctoproject.org/dev-manual/index.html | * [https://docs.yoctoproject.org/dev-manual/index.html Yocto Project Development Tasks Manual] | ||
This system is much more complicated than something like Buildroot which | This system is much more complicated than something like Buildroot which only requires specifying manual entries to build, but it solves a lot of problems Buildroot introduces such as keeping configurations in sync and building being able to build multiple roots. | ||
only requires specifying manual entries to build, but it solves a lot of | |||
problems Buildroot introduces such as keeping configurations in sync and | |||
building being able to build multiple roots. | |||
== First build == | |||
Starting off, let's build the minimal image: | Starting off, let's build the minimal image: | ||
$ bitbake core-image-minimal | |||
This will take a long time. Long enough that I should've done this while | This will take a long time. Long enough that I should've done this while writing the previous chapter... | ||
writing the previous chapter... | |||
It looks like on my machine building QEMU failed! This stops building | It looks like on my machine building QEMU failed! This stops building everything else including the kernel, but I don't want that. So instead I should run and inspect build failures after the build: | ||
everything else including the kernel, but I don't want that. | |||
So instead I should run and inspect build failures after the build: | |||
$ bitbake --continue core-image-minimal | |||
The full build log is available in the build directory: | The full build log is available in the build directory: build/tmp-glibc/log/cooker/qemux86-64/console-latest.log | ||
In this case it says: | In this case it says: | ||
ERROR: qemu-system-native-8.2.1-r0 do_compile: oe_runmake failed | |||
We can get a devshell and run the task ourselves like this: | We can get a devshell and run the task ourselves like this: | ||
$ bitbake qemu-system-native -c devshell | |||
$ ../temp/run.do_compile # Run this in the devshell | |||
In my case it gives me this error: | In my case it gives me this error: | ||
/usr/lib/libgdk_pixbuf-2.0.so.0: undefined reference to `g_once_init_leave_pointer' | |||
This is suspicious: OpenEmbedded is trying to mix its own built libraries with my host libgdk_pixbuf and getting confused as mine is a newer version that uses a new symbol. | |||
Builds should never link with files in the host operating system. This type of issue is known as a leak, and are usually tricky to troubleshoot. Let's try anyway. | |||
files in the host operating system. This type of issue is known as a | |||
leak, and are usually tricky to troubleshoot. Let's try anyway. | |||
In the devshell looking in ../build I found that pixbuf is mentioned in meson-logs/meson-log.txt, it is added from the output of this command: | |||
meson-logs/meson-log.txt, it is added from the output of this command: | |||
/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/bin/pkg-config --cflags gvnc-1.0 | |||
Running that command in the devshell gives an error, so the devshell | Running that command in the devshell gives an error, so the devshell is not having information leaked in to it from the host. So something must be happening when Meson is building to introduce a leak. | ||
is not having information leaked in to it from the host. So something | |||
must be happening when Meson is building to introduce a leak. | |||
Looking at the manual page, pkg-config uses environment variables to | Looking at the manual page, pkg-config uses environment variables to help find packages. Checking the meson log I found this: | ||
help find packages. | |||
env[PKG_CONFIG_PATH]: /home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/lib/pkgconfig:/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig | |||
While most of the PKG_CONFIG_PATH is correct, the end has | While most of the PKG_CONFIG_PATH is correct, the end has /usr/lib/pkgconfig and /usr/share/pkgconfig. This will cause a leak! | ||
/usr/lib/pkgconfig and /usr/share/pkgconfig. This will cause a leak! | |||
I searched QEMU's source code for | I searched QEMU's source code for PKG_CONFIG_PATH but didn't find anything, so this leak is most likely from OpenEmbedded somewhere. There's quite a lot of files to look for in OpenEmbedded, so it would be a lot of work trying to find where the leak is. | ||
anything, so this leak is most likely from OpenEmbedded somewhere. | |||
There's quite a lot of files to look for in OpenEmbedded, so it would be | |||
a lot of work trying to find where the leak is. | |||
Luckily, we can ask bitbake for the recipe's environment: | Luckily, we can ask bitbake for the recipe's environment: | ||
bitbake -e qemu-system-native > env | |||
Looking in the file we quickly see this: | Looking in the file we quickly see this: | ||
# line: 158, file: /home/jookia/oe-test/openembedded-core/meta/recipes-devtools/qemu/qemu.inc | |||
do_configure() { | |||
# Append build host pkg-config paths for native target since the host may provide sdl | |||
BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") | |||
if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then | |||
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH | |||
fi | |||
In the qemu.inc file we can see the matching code: | In the qemu.inc file we can see the matching code: | ||
do_configure:prepend:class-native() { | |||
# Append build host pkg-config paths for native target since the host may provide sdl | |||
BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") | |||
if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then | |||
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH | |||
fi | |||
} | |||
For now we're just going to remove the block of code from the qemu.inc file. Directly modifying openembedded-core is easy but often a bad idea, so we'll go through some better options later in this guide. | |||
After removing the code we can check the environment again: | |||
$ bitbake -e qemu-system-native > env | |||
After confirming the change was made by looking for PKG_CONFIG_PATH and applied we can finish the build: | |||
$ bitbake qemu-system-native | |||
On my machine this compiles without error. We can now test the image using the included runqemu program: | |||
$ runqemu nographic slirp core-image-minimal | |||
This will launch the image using an emulator in our terminal. The username is 'root'. Hitting ctrl-a ctrl-x will stop the emulator. | |||
We can look at the image contents in build/tmp-glibc/deploy/images/qemux86-64. It contains files such as: | |||
bzImage-qemux86-64.bin - The kernel image | |||
core-image-minimal-qemux86-64.rootfs.ext4 | |||
core-image-minimal-qemux86-64.rootfs.manifest | |||
core-image-minimal-qemux86-64.rootfs.qemuboot.conf | |||
core-image-minimal-qemux86-64.rootfs.spdx.tar.zst | |||
core-image-minimal-qemux86-64.rootfs.tar.bz2 | |||
core-image-minimal-qemux86-64.rootfs.testdata.json | |||
modules-qemux86-64.tgz | |||
We can also see that all the build Linux software is packaged in build/tmp-glibc/deploy/ipk. | |||
To build another image we can run: | |||
$ bitbake core-image-minimal-dev | |||
$ runqemu nographic slirp core-image-minimal-dev | |||
This | This creates an identical image with debug symbols. We find a set of files for a new rootfs next to the other ones: | ||
core-image-minimal-dev-qemux86-64.rootfs.ext4 | |||
core-image-minimal-dev-qemux86-64.rootfs.manifest | |||
core-image-minimal-dev-qemux86-64.rootfs.qemuboot.conf | |||
core-image-minimal-dev-qemux86-64.rootfs.spdx.tar.zst | |||
core-image-minimal-dev-qemux86-64.rootfs.tar.bz2 | |||
core-image-minimal-dev-qemux86-64.rootfs.testdata.json | |||
So we have managed to build a single kernel and two root filesystems. | |||
== Multiple builds == | |||
While you can build multiple root filesystems, that's about as far multiple outputs go. If you need to build for a different machine or a different distro you will need to use another configuration file. | |||
BitBake does support a way to use multiple configuration files, but I'm not exactly sure why you would want to use it instead of multiple build directories, especially if you have to do multi-architecture builds. | |||
The first thing we want to do is re-locate our build cache and downloads: | |||
$ cd build | |||
$ mv sstate-cache downloads .. | |||
They will now be in our oe-test directory. Next, open up build/conf/local.conf and add these lines to the top: | |||
DL_DIR ?= "${TOPDIR}/../downloads" | |||
SSTATE_DIR ?= "${TOPDIR}/../sstate-cache" | |||
This will save a lot of time for the next step where we create a new build directory. In the oe-test directory run: | |||
$ source openembedded-core/oe-init-build-env build2 | |||
Then perform the same edits to build/conf/local.conf to set DL_DIR and SSTATE_DIR. But also add this line to enable systemd: | |||
INIT_MANAGER = "systemd" | |||
Because of the shared state directory, this should re-use a lot of | Because of the shared state directory, this should re-use a lot of | ||
already built components. Let's build: | already built components. Let's build: | ||
$ bitbake core-image-minimal | |||
Indeed it did, but I found it was spending time rebuilding | Indeed it did, but I found it was spending time rebuilding gcc-cross-x86_64! That would mean rebuilding basically everything else too. Why? I cancelled the build immediately to look. | ||
gcc-cross-x86_64! That would mean rebuilding basically everything else | |||
too. Why? I | |||
In each build directory I did this: | In each build directory I did this: | ||
$ bitbake -e gcc-cross-x86_64 | grep -v /home > cross.env | |||
Then in the main directory I ran: | Then in the main directory I ran: | ||
$ diff build/cross.env build2/cross.env | |||
This answers my question of why the compiler is being rebuilt pretty | This answers my question of why the compiler is being rebuilt pretty fast: | ||
fast: | |||
11338c11337 | |||
< #define STANDARD_STARTFILE_PREFIX_1 "/usr/lib/" | |||
--- | |||
> #define STANDARD_STARTFILE_PREFIX_1 "/lib/" | |||
11340c11339 | |||
< #define SYSTEMLIBS_DIR "/usr/lib/" | |||
--- | |||
> #define SYSTEMLIBS_DIR "/lib/" | |||
In retrospect it's obvious: systemd requires a merged /usr, so the compiler | In retrospect it's obvious: systemd requires a merged /usr, so the compiler will have to put its libraries in /usr/lib. This requires a rebuild of the compiler and probably everything else! Oh well. | ||
will have to put its libraries in /usr/lib. This requires a rebuild of the | |||
compiler and probably everything else! Oh well. | |||
After building again: | After building again: | ||
$ bitbake core-image-minimal | |||
I can now run the image in QEMU and verify it works: | I can now run the image in QEMU and verify it works: | ||
$ runqemu nographic slirp core-image-minimal | |||
The image boots to systemd managed system. Success! | The image boots to systemd managed system. Success! | ||
== Recipes == | |||
------------------ | |||
We can use oe-pkgdata-util to list and inspect built packages like this: | |||
$ oe-pkg-data-util list-pkgs | grep zlib | |||
zlib | |||
zlib-dbg | |||
zlib-dev | |||
zlib-doc | |||
zlib-src | |||
zlib-staticdev | |||
$ oe-pkg-data-util pkg-info zlib-dev | |||
zlib-dev 1.3.1-r0 zlib 1.3.1-r0 113538 | |||
$ oe-pkg-data-util list-pkg-files zlib-dev | |||
zlib-dev: | |||
/usr/include/zconf.h | |||
/usr/include/zlib.h | |||
/usr/lib/libz.so | |||
/usr/lib/pkgconfig/zlib.pc | |||
We can also find the list of packages for a particular root filesystem in its manfiest file, for example: | |||
$ grep 'openssl' tmp-glibc/deploy/images/qemux86-64/core-image-minimal-dev-qemux86-64.rootfs.manifest | |||
openssl-conf core2-64 3.2.1-r0 | |||
openssl-dev core2-64 3.2.1-r0 | |||
openssl-ossl-module-legacy core2-64 3.2.1-r0 | |||
While this is useful for dealing with built packages and images, most work is done with BitBake recipes. We can list buildable recipes like so: | |||
bitbake- | $ bitbake -s | grep ^linux-yocto | ||
linux-yocto :6.6.23+git-r0 | |||
We can list all recipes available like this: | |||
$ bitbake-layers show-recipes -r | grep ^linux-yocto | |||
linux-yocto | |||
linux-yocto-dev (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-dev to enable it) | |||
linux-yocto-rt (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-rt to enable it) | |||
linux-yocto-tiny (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-tiny to enable it) | |||
In this case we can only build linux-yocto but there are other packages available. | |||
Listing dependencies of a recipe is a bit trickier. You can dump a recipe's dependency graph and view it like so: | |||
$ bitbake -g core-image-minimal | |||
# pn-buildlist now lists all recipes required to build core-image-minimal | |||
# task-depends.dot lists all tasks required to build core-image-minimal | |||
But this lists every dependency, not just immediate. To list immediate dependencies run: | |||
$ bitbake -e vim | grep -P '^R?DEPENDS.*=' | |||
DEPENDS="pkgconfig-native autoconf-native automake-native libtool-native libtool-cross virtual/x86_64-oe-linux-gcc virtual/x86_64-oe-linux-compilerlibs virtual/libc ncurses gettext-native desktop-file-utils acl gtk+3 xt virtual/update-alternatives" | |||
RDEPENDS:${KERNEL_PACKAGE_NAME}-base="" | |||
RDEPENDS:vim="ncurses-terminfo-base vim-xxd" | |||
RDEPENDS:vim-staticdev="vim-dev (= 9.1.0114-r0)" | |||
Two types of dependencies are shown: DEPENDS and RDEPENDS. DEPENDS is for build time dependencies, RDEPENDS is for runtime dependencies. For our purposes we only care about the RDEPENDS with the recipe name: "RDEPENDS:vim". | |||
Note that these BitBake commands will only tell us information about the package based on the overall configuration of the system. They won't tell you about dependencies that aren't enabled. Generally these are enabled or disabled using PACKAGECONFIG, so by looking at that variable we can find possible dependencies. | |||
distro/machine layers | Here's an example of finding available features and dependencies for dropbear: | ||
$ bitbake -e dropbear | sed '1,/^# $PACKAGECONFIG/d;/PACKAGECONFIG=/,$d' | |||
# :append[pn-qemu-system-native] /home/jookia/oe-test/build/conf/local.conf:215 | |||
# " sdl" | |||
# set /home/jookia/oe-test/openembedded-core/meta/conf/documentation.conf:321 | |||
# [doc] "This variable provides a means of enabling or disabling features of a recipe on a per-recipe basis." | |||
# set? /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:51 | |||
# "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}" | |||
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:52 | |||
# [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}" | |||
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:53 | |||
# [system-libtom] "--disable-bundled-libtom,--enable-bundled-libtom,libtommath libtomcrypt" | |||
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:54 | |||
# [disable-weak-ciphers] "" | |||
# set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:55 | |||
# [enable-x11-forwarding] "" | |||
# pre-expansion value: | |||
# "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}" | |||
The fancy sed command is just to find the PACKAGECONFIG block in the bitbake -e dump. You can look for it manually. | |||
Let's focus on the pam feature in the output: | |||
# [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}" | |||
The bracketed text is the feature and the quoted text is the PACKAGECONFIG variable with elements separated by commas in this style: | |||
PACKAGECONFIG[foo] = "--enable-foo,--disable-foo,foo_depends,foo_runtime_depends,foo_runtime_recommends,foo_conflict_packageconfig" | |||
It seems like the pam argument DEPENDS on libpam (the third column) and uses the PAM_PLUGINS variable for its RDEPENDS. This makes sense as PAM uses pluggable modules decided by configuration rather than this specific package. | |||
You may also see the variables DISTRO_FEATURES or MACHINE_FEATURES used to set PACKAGECONFIG. These variables are set in configurations and used to make broad changes to many recipes at once rather than setting individual package features. | |||
== Layers == | |||
layer example: | |||
configuring systemd | |||
package layers | |||
distro layers | |||
board/machine layers | |||
distro layer configuration | |||
set systemd, add vim | |||
add vim | |||
TODO: insert section about extending/modifying openembedded-core | |||
Line 464: | Line 431: | ||
Chapter X: Mainline kernel | Chapter X: Mainline kernel | ||
== Fixing bugs == | |||
So how do we fix this? We have two options: | |||
* Copy the openembedded-core/meta/recipes-devtools/qemu/ directory to a new layer and change qemu.inc to remove this block of code | |||
* Change qemu.inc directly and create a patch or Git branch for tracking changes to the openembedded-core code | |||
I've heard that some organizations have a policy preventing modification of the openembedded-core repository so they would have to copy the entire directory and update the recipe. They would then have to manually pull in updates from the repository whenever there's a new update. | |||
For me I would rather use a Git branch as then I can just use Git to merge in any updates from the repository. But it does makes me ask: Why even use layers if you can use Git branches? | |||
-------------------------- | -------------------------- | ||
Revision as of 20:34, 8 May 2024
TODO: clean up formatting, maybe use inline code
formatting?
In this guide I want to accomplish the following embedded Linux tasks using Yocto:
- Create a rootfs
- Create an initramfs
- Create a qemu virt kernel
- Package this in to proper layers
Yocto offers a starting distribution named Poky but for learning purposes I'll be using OpenEmbedded and BitBake directly.
Downloading code
The first thing we want to do is make a directory for our project:
$ mkdir oe-test $ cd oe-test
I named mine 'oe-test' but you can name it what you want.
Next we download the code. We do this using Git so we can easily switch to a version on update.
$ git clone https://git.openembedded.org/bitbake $ git clone https://git.openembedded.org/openembedded-core
Now we want to switch to specific tags:
$ git -C bitbake switch --detach 2.8.0 $ git -C openembedded-core switch --detach 2024-04-scarthgap
These are both the latest releases as of writing. I found these specific URLs and settings by digging through their online GitWeb: https://git.openembedded.org
This combination only gives us the bare minimum required for building a virtual system. We will add our own layers for anything more.
As we are building we will also source the build environment script:
$ source openembedded-core/oe-init-build-env build
This puts us in our build directory named 'build'.
Inspection
Let's have a look at the files we've downloaded and get an idea for how this project fits together and works:
First, a top level map:
- bitbake - The build system
- openembedded-core - BitBake recipes and documentation
- build - Build output
The core of OpenEmbedded is the BitBake build system, so it's worth taking some time to understand it.
BitBake is a bit like 'make' where you tell it to build a specific file, or in this case, recipe. It looks at the recipe, and finds recipes that recipe needs and builds those first, and so on. It can do these in parallel, and cache results. Standard building stuff.
What makes BitBake special is that you can change recipes using other recipes or configuration files without modifying the original. This is done using configurations or append files. A set of recipes and modifications can be stored in a separate directory as a 'layer', which is the intended way of organizing files using BitBake.
Let's look at a real world example: openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb
This package has the ability to enable or disable PAM using the 'DISTRO_FEATURES' variable. We can set this value:
- Per-build in our build configuration
- Per-machine in our machine configuration
- Per-distro in our distro configuration
- Per-package in a .bbappend for dropbear
I mentioned above the concept of 'machine' and 'distro'. These are not BitBake concepts but concepts created by openembedded-core.
Let's look at some interesting parts of openembedded-core. We see two layers:
- meta - The core recipes used for creating a Linux distro
- meta-skeleton - An example layer with configs and packages
By default we only use the first layer. Inside that layer is:
- classes* - BitBake code re-used in recipe types
- recipes* - BitBake recipes to build
- conf/ - Global configuration settings
- conf/distro/ - Distro configuration settings
- conf/machine/ - Machine configuration settings
The concept of distributions and different machines are just configuration files that set package variables. It's useful to separate these out as it means you can modify packages, distros and machines separately during development without needing to change multiple configurations at once like in a system like Buildroot.
Selecting which distro and machine to use are done using variables, much like any other aspect of OpenEmbedded. oe-init-build-env sets up the build directory with a conf/local.conf file that has some defaults.
This information should give you a basic enough understanding to follow along, but I highly recommend reading the full Yocto and BitBake manuals:
The development manuals are helpful too:
This system is much more complicated than something like Buildroot which only requires specifying manual entries to build, but it solves a lot of problems Buildroot introduces such as keeping configurations in sync and building being able to build multiple roots.
First build
Starting off, let's build the minimal image:
$ bitbake core-image-minimal
This will take a long time. Long enough that I should've done this while writing the previous chapter...
It looks like on my machine building QEMU failed! This stops building everything else including the kernel, but I don't want that. So instead I should run and inspect build failures after the build:
$ bitbake --continue core-image-minimal
The full build log is available in the build directory: build/tmp-glibc/log/cooker/qemux86-64/console-latest.log
In this case it says:
ERROR: qemu-system-native-8.2.1-r0 do_compile: oe_runmake failed
We can get a devshell and run the task ourselves like this:
$ bitbake qemu-system-native -c devshell $ ../temp/run.do_compile # Run this in the devshell
In my case it gives me this error:
/usr/lib/libgdk_pixbuf-2.0.so.0: undefined reference to `g_once_init_leave_pointer'
This is suspicious: OpenEmbedded is trying to mix its own built libraries with my host libgdk_pixbuf and getting confused as mine is a newer version that uses a new symbol.
Builds should never link with files in the host operating system. This type of issue is known as a leak, and are usually tricky to troubleshoot. Let's try anyway.
In the devshell looking in ../build I found that pixbuf is mentioned in meson-logs/meson-log.txt, it is added from the output of this command:
/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/bin/pkg-config --cflags gvnc-1.0
Running that command in the devshell gives an error, so the devshell is not having information leaked in to it from the host. So something must be happening when Meson is building to introduce a leak.
Looking at the manual page, pkg-config uses environment variables to help find packages. Checking the meson log I found this:
env[PKG_CONFIG_PATH]: /home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/lib/pkgconfig:/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig
While most of the PKG_CONFIG_PATH is correct, the end has /usr/lib/pkgconfig and /usr/share/pkgconfig. This will cause a leak!
I searched QEMU's source code for PKG_CONFIG_PATH but didn't find anything, so this leak is most likely from OpenEmbedded somewhere. There's quite a lot of files to look for in OpenEmbedded, so it would be a lot of work trying to find where the leak is.
Luckily, we can ask bitbake for the recipe's environment:
bitbake -e qemu-system-native > env
Looking in the file we quickly see this:
# line: 158, file: /home/jookia/oe-test/openembedded-core/meta/recipes-devtools/qemu/qemu.inc do_configure() { # Append build host pkg-config paths for native target since the host may provide sdl BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH fi
In the qemu.inc file we can see the matching code:
do_configure:prepend:class-native() { # Append build host pkg-config paths for native target since the host may provide sdl BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH fi }
For now we're just going to remove the block of code from the qemu.inc file. Directly modifying openembedded-core is easy but often a bad idea, so we'll go through some better options later in this guide.
After removing the code we can check the environment again:
$ bitbake -e qemu-system-native > env
After confirming the change was made by looking for PKG_CONFIG_PATH and applied we can finish the build:
$ bitbake qemu-system-native
On my machine this compiles without error. We can now test the image using the included runqemu program:
$ runqemu nographic slirp core-image-minimal
This will launch the image using an emulator in our terminal. The username is 'root'. Hitting ctrl-a ctrl-x will stop the emulator.
We can look at the image contents in build/tmp-glibc/deploy/images/qemux86-64. It contains files such as:
bzImage-qemux86-64.bin - The kernel image core-image-minimal-qemux86-64.rootfs.ext4 core-image-minimal-qemux86-64.rootfs.manifest core-image-minimal-qemux86-64.rootfs.qemuboot.conf core-image-minimal-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-qemux86-64.rootfs.tar.bz2 core-image-minimal-qemux86-64.rootfs.testdata.json modules-qemux86-64.tgz
We can also see that all the build Linux software is packaged in build/tmp-glibc/deploy/ipk.
To build another image we can run:
$ bitbake core-image-minimal-dev $ runqemu nographic slirp core-image-minimal-dev
This creates an identical image with debug symbols. We find a set of files for a new rootfs next to the other ones:
core-image-minimal-dev-qemux86-64.rootfs.ext4 core-image-minimal-dev-qemux86-64.rootfs.manifest core-image-minimal-dev-qemux86-64.rootfs.qemuboot.conf core-image-minimal-dev-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-dev-qemux86-64.rootfs.tar.bz2 core-image-minimal-dev-qemux86-64.rootfs.testdata.json
So we have managed to build a single kernel and two root filesystems.
Multiple builds
While you can build multiple root filesystems, that's about as far multiple outputs go. If you need to build for a different machine or a different distro you will need to use another configuration file.
BitBake does support a way to use multiple configuration files, but I'm not exactly sure why you would want to use it instead of multiple build directories, especially if you have to do multi-architecture builds.
The first thing we want to do is re-locate our build cache and downloads:
$ cd build $ mv sstate-cache downloads ..
They will now be in our oe-test directory. Next, open up build/conf/local.conf and add these lines to the top:
DL_DIR ?= "${TOPDIR}/../downloads" SSTATE_DIR ?= "${TOPDIR}/../sstate-cache"
This will save a lot of time for the next step where we create a new build directory. In the oe-test directory run:
$ source openembedded-core/oe-init-build-env build2
Then perform the same edits to build/conf/local.conf to set DL_DIR and SSTATE_DIR. But also add this line to enable systemd:
INIT_MANAGER = "systemd"
Because of the shared state directory, this should re-use a lot of already built components. Let's build:
$ bitbake core-image-minimal
Indeed it did, but I found it was spending time rebuilding gcc-cross-x86_64! That would mean rebuilding basically everything else too. Why? I cancelled the build immediately to look.
In each build directory I did this:
$ bitbake -e gcc-cross-x86_64 | grep -v /home > cross.env
Then in the main directory I ran:
$ diff build/cross.env build2/cross.env
This answers my question of why the compiler is being rebuilt pretty fast:
11338c11337 < #define STANDARD_STARTFILE_PREFIX_1 "/usr/lib/" --- > #define STANDARD_STARTFILE_PREFIX_1 "/lib/" 11340c11339 < #define SYSTEMLIBS_DIR "/usr/lib/" --- > #define SYSTEMLIBS_DIR "/lib/"
In retrospect it's obvious: systemd requires a merged /usr, so the compiler will have to put its libraries in /usr/lib. This requires a rebuild of the compiler and probably everything else! Oh well.
After building again:
$ bitbake core-image-minimal
I can now run the image in QEMU and verify it works:
$ runqemu nographic slirp core-image-minimal
The image boots to systemd managed system. Success!
Recipes
We can use oe-pkgdata-util to list and inspect built packages like this:
$ oe-pkg-data-util list-pkgs | grep zlib zlib zlib-dbg zlib-dev zlib-doc zlib-src zlib-staticdev $ oe-pkg-data-util pkg-info zlib-dev zlib-dev 1.3.1-r0 zlib 1.3.1-r0 113538 $ oe-pkg-data-util list-pkg-files zlib-dev zlib-dev: /usr/include/zconf.h /usr/include/zlib.h /usr/lib/libz.so /usr/lib/pkgconfig/zlib.pc
We can also find the list of packages for a particular root filesystem in its manfiest file, for example:
$ grep 'openssl' tmp-glibc/deploy/images/qemux86-64/core-image-minimal-dev-qemux86-64.rootfs.manifest openssl-conf core2-64 3.2.1-r0 openssl-dev core2-64 3.2.1-r0 openssl-ossl-module-legacy core2-64 3.2.1-r0
While this is useful for dealing with built packages and images, most work is done with BitBake recipes. We can list buildable recipes like so:
$ bitbake -s | grep ^linux-yocto linux-yocto :6.6.23+git-r0
We can list all recipes available like this:
$ bitbake-layers show-recipes -r | grep ^linux-yocto linux-yocto linux-yocto-dev (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-dev to enable it) linux-yocto-rt (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-rt to enable it) linux-yocto-tiny (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-tiny to enable it)
In this case we can only build linux-yocto but there are other packages available.
Listing dependencies of a recipe is a bit trickier. You can dump a recipe's dependency graph and view it like so:
$ bitbake -g core-image-minimal # pn-buildlist now lists all recipes required to build core-image-minimal # task-depends.dot lists all tasks required to build core-image-minimal
But this lists every dependency, not just immediate. To list immediate dependencies run:
$ bitbake -e vim | grep -P '^R?DEPENDS.*=' DEPENDS="pkgconfig-native autoconf-native automake-native libtool-native libtool-cross virtual/x86_64-oe-linux-gcc virtual/x86_64-oe-linux-compilerlibs virtual/libc ncurses gettext-native desktop-file-utils acl gtk+3 xt virtual/update-alternatives" RDEPENDS:${KERNEL_PACKAGE_NAME}-base="" RDEPENDS:vim="ncurses-terminfo-base vim-xxd" RDEPENDS:vim-staticdev="vim-dev (= 9.1.0114-r0)"
Two types of dependencies are shown: DEPENDS and RDEPENDS. DEPENDS is for build time dependencies, RDEPENDS is for runtime dependencies. For our purposes we only care about the RDEPENDS with the recipe name: "RDEPENDS:vim".
Note that these BitBake commands will only tell us information about the package based on the overall configuration of the system. They won't tell you about dependencies that aren't enabled. Generally these are enabled or disabled using PACKAGECONFIG, so by looking at that variable we can find possible dependencies.
Here's an example of finding available features and dependencies for dropbear:
$ bitbake -e dropbear | sed '1,/^# $PACKAGECONFIG/d;/PACKAGECONFIG=/,$d' # :append[pn-qemu-system-native] /home/jookia/oe-test/build/conf/local.conf:215 # " sdl" # set /home/jookia/oe-test/openembedded-core/meta/conf/documentation.conf:321 # [doc] "This variable provides a means of enabling or disabling features of a recipe on a per-recipe basis." # set? /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:51 # "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:52 # [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:53 # [system-libtom] "--disable-bundled-libtom,--enable-bundled-libtom,libtommath libtomcrypt" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:54 # [disable-weak-ciphers] "" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:55 # [enable-x11-forwarding] "" # pre-expansion value: # "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}"
The fancy sed command is just to find the PACKAGECONFIG block in the bitbake -e dump. You can look for it manually.
Let's focus on the pam feature in the output:
# [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}"
The bracketed text is the feature and the quoted text is the PACKAGECONFIG variable with elements separated by commas in this style:
PACKAGECONFIG[foo] = "--enable-foo,--disable-foo,foo_depends,foo_runtime_depends,foo_runtime_recommends,foo_conflict_packageconfig"
It seems like the pam argument DEPENDS on libpam (the third column) and uses the PAM_PLUGINS variable for its RDEPENDS. This makes sense as PAM uses pluggable modules decided by configuration rather than this specific package.
You may also see the variables DISTRO_FEATURES or MACHINE_FEATURES used to set PACKAGECONFIG. These variables are set in configurations and used to make broad changes to many recipes at once rather than setting individual package features.
Layers
layer example: configuring systemd
package layers distro layers board/machine layers
distro layer configuration set systemd, add vim add vim
TODO: insert section about extending/modifying openembedded-core
Chapter X: Packaging
oe-pkgdata-util
- SDK
- TODO: multi configs?
- todo nfsroot, unfsd?
https://stackoverflow.com/questions/47429670/manually-building-a-kernel-source-from-yocto-build SRC_URI = "git:///path-to-linux-source/.git/;branch=${KBRANCH};protocol=file"
INHERIT += "externalsrc" EXTERNALSRC:pn-myrecipe = "/path/to/my/source/tree"
wic genimage
packaging a project:
mkdir -p ../meta-bugfixes/recipes-general/evtone/ recipetool create -B main -S v1.4 https://git.lumina-sensum.com/git/Jookia/evtone.git -o ../meta-bugfixes/recipes-general/evtone/evtone_1.4.bb bitbake evtone
- add 'inherit meson', remove configure stuff
PV v1.4 tmp-glibc/work/core2-64-oe-linux/evtone/1.4/image tmp-glibc/deploy/ipk/core2-64/evtone changing version removes the old ipk in deploy/
Chapter X: Root filesystems
- what components are in this image
- multiple kernels? - build only rootfs - build only kernel
- adding evtone to the image
- TODO: introspect and ask:
- how do we modify it for a different machine? - what is an image? - can we build just the rootfs?
- initrd
Chapter X: Mainline kernel
Fixing bugs
So how do we fix this? We have two options:
- Copy the openembedded-core/meta/recipes-devtools/qemu/ directory to a new layer and change qemu.inc to remove this block of code
- Change qemu.inc directly and create a patch or Git branch for tracking changes to the openembedded-core code
I've heard that some organizations have a policy preventing modification of the openembedded-core repository so they would have to copy the entire directory and update the recipe. They would then have to manually pull in updates from the repository whenever there's a new update.
For me I would rather use a Git branch as then I can just use Git to merge in any updates from the repository. But it does makes me ask: Why even use layers if you can use Git branches?
Chapter X: Bootloader
Chapter X: Disk image
Chapter X: Board layer
Chapter X: BSP layer
Chapter X: Development
- using a custom srcdir for a package
- sdks kind of aren't needed for us