Jump to content
Toggle sidebar
JookWiki
Search
Create account
Log in
Personal tools
Create account
Log in
Pages for logged out editors
learn more
Contributions
Talk
Navigation
Main page
Recent changes
Random page
All pages
Help about MediaWiki
Tools
What links here
Related changes
Special pages
Page information
Editing
Yocto guide
Page
Discussion
English
Read
Edit
Edit source
View history
More
Read
Edit
Edit source
View history
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
TODO: clean up formatting, maybe use inline <code>code</code> formatting? In this guide I want to accomplish the following embedded Linux tasks using Yocto: * Create a rootfs * Create an initramfs * Create a qemu virt kernel * Package this in to proper layers Yocto offers a starting distribution named Poky but for learning purposes I'll be using OpenEmbedded and BitBake directly. == Downloading code == The first thing we want to do is make a directory for our project: $ mkdir oe-test $ cd oe-test I named mine 'oe-test' but you can name it what you want. Next we download the code. We do this using Git so we can easily switch to a version on update. $ git clone https://git.openembedded.org/bitbake $ git clone https://git.openembedded.org/openembedded-core Now we want to switch to specific tags: $ git -C bitbake switch --detach 2.8.0 $ git -C openembedded-core switch --detach 2024-04-scarthgap These are both the latest releases as of writing. I found these specific URLs and settings by digging through their online GitWeb: https://git.openembedded.org This combination only gives us the bare minimum required for building a virtual system. We will add our own layers for anything more. As we are building we will also source the build environment script: $ source openembedded-core/oe-init-build-env build This puts us in our build directory named 'build'. == Inspection == Let's have a look at the files we've downloaded and get an idea for how this project fits together and works: First, a top level map: * bitbake - The build system * openembedded-core - BitBake recipes and documentation * build - Build output The core of OpenEmbedded is the BitBake build system, so it's worth taking some time to understand it. BitBake is a bit like 'make' where you tell it to build a specific file, or in this case, recipe. It looks at the recipe, and finds recipes that recipe needs and builds those first, and so on. It can do these in parallel, and cache results. Standard building stuff. What makes BitBake special is that you can change recipes using other recipes or configuration files without modifying the original. This is done using configurations or append files. A set of recipes and modifications can be stored in a separate directory as a 'layer', which is the intended way of organizing files using BitBake. Let's look at a real world example: openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb This package has the ability to enable or disable PAM using the 'DISTRO_FEATURES' variable. We can set this value: * Per-build in our build configuration * Per-machine in our machine configuration * Per-distro in our distro configuration * Per-package in a .bbappend for dropbear I mentioned above the concept of 'machine' and 'distro'. These are not BitBake concepts but concepts created by openembedded-core. Let's look at some interesting parts of openembedded-core. We see two layers: * meta - The core recipes used for creating a Linux distro * meta-skeleton - An example layer with configs and packages By default we only use the first layer. Inside that layer is: * classes* - BitBake code re-used in recipe types * recipes* - BitBake recipes to build * conf/ - Global configuration settings * conf/distro/ - Distro configuration settings * conf/machine/ - Machine configuration settings The concept of distributions and different machines are just configuration files that set package variables. It's useful to separate these out as it means you can modify packages, distros and machines separately during development without needing to change multiple configurations at once like in a system like Buildroot. Selecting which distro and machine to use are done using variables, much like any other aspect of OpenEmbedded. oe-init-build-env sets up the build directory with a conf/local.conf file that has some defaults. This information should give you a basic enough understanding to follow along, but I highly recommend reading the full Yocto and BitBake manuals: * [https://docs.yoctoproject.org/overview-manual/index.html Yocto Project Overview and Concepts Manual] * [https://docs.yoctoproject.org/ref-manual/index.html Yocto Project Reference Manual] * [https://docs.yoctoproject.org/bitbake/ BitBake User Manual] The development manuals are helpful too: * [https://docs.yoctoproject.org/bsp-guide/index.html Yocto Project Board Support Package Developer’s Guide] * [https://docs.yoctoproject.org/dev-manual/index.html Yocto Project Development Tasks Manual] This system is much more complicated than something like Buildroot which only requires specifying manual entries to build, but it solves a lot of problems Buildroot introduces such as keeping configurations in sync and building being able to build multiple roots. == First build == Starting off, let's build the minimal image: $ bitbake core-image-minimal This will take a long time. Long enough that I should've done this while writing the previous chapter... It looks like on my machine building QEMU failed! This stops building everything else including the kernel, but I don't want that. So instead I should run and inspect build failures after the build: $ bitbake --continue core-image-minimal The full build log is available in the build directory: build/tmp-glibc/log/cooker/qemux86-64/console-latest.log In this case it says: ERROR: qemu-system-native-8.2.1-r0 do_compile: oe_runmake failed We can get a devshell and run the task ourselves like this: $ bitbake qemu-system-native -c devshell $ ../temp/run.do_compile # Run this in the devshell In my case it gives me this error: /usr/lib/libgdk_pixbuf-2.0.so.0: undefined reference to `g_once_init_leave_pointer' This is suspicious: OpenEmbedded is trying to mix its own built libraries with my host libgdk_pixbuf and getting confused as mine is a newer version that uses a new symbol. Builds should never link with files in the host operating system. This type of issue is known as a leak, and are usually tricky to troubleshoot. Let's try anyway. In the devshell looking in ../build I found that pixbuf is mentioned in meson-logs/meson-log.txt, it is added from the output of this command: /home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/bin/pkg-config --cflags gvnc-1.0 Running that command in the devshell gives an error, so the devshell is not having information leaked in to it from the host. So something must be happening when Meson is building to introduce a leak. Looking at the manual page, pkg-config uses environment variables to help find packages. Checking the meson log I found this: env[PKG_CONFIG_PATH]: /home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/lib/pkgconfig:/home/jookia/oe-test/build/tmp-glibc/work/x86_64-linux/qemu-system-native/8.2.1/recipe-sysroot-native/usr/share/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig While most of the PKG_CONFIG_PATH is correct, the end has /usr/lib/pkgconfig and /usr/share/pkgconfig. This will cause a leak! I searched QEMU's source code for PKG_CONFIG_PATH but didn't find anything, so this leak is most likely from OpenEmbedded somewhere. There's quite a lot of files to look for in OpenEmbedded, so it would be a lot of work trying to find where the leak is. Luckily, we can ask bitbake for the recipe's environment: bitbake -e qemu-system-native > env Looking in the file we quickly see this: # line: 158, file: /home/jookia/oe-test/openembedded-core/meta/recipes-devtools/qemu/qemu.inc do_configure() { # Append build host pkg-config paths for native target since the host may provide sdl BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH fi In the qemu.inc file we can see the matching code: do_configure:prepend:class-native() { # Append build host pkg-config paths for native target since the host may provide sdl BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "") if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH fi } For now we're just going to remove the block of code from the qemu.inc file. Directly modifying openembedded-core is easy but often a bad idea, so we'll go through some better options later in this guide. After removing the code we can check the environment again: $ bitbake -e qemu-system-native > env After confirming the change was made by looking for PKG_CONFIG_PATH and applied we can finish the build: $ bitbake qemu-system-native On my machine this compiles without error. We can now test the image using the included runqemu program: $ runqemu nographic slirp core-image-minimal This will launch the image using an emulator in our terminal. The username is 'root'. Hitting ctrl-a ctrl-x will stop the emulator. We can look at the image contents in build/tmp-glibc/deploy/images/qemux86-64. It contains files such as: bzImage-qemux86-64.bin - The kernel image core-image-minimal-qemux86-64.rootfs.ext4 core-image-minimal-qemux86-64.rootfs.manifest core-image-minimal-qemux86-64.rootfs.qemuboot.conf core-image-minimal-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-qemux86-64.rootfs.tar.bz2 core-image-minimal-qemux86-64.rootfs.testdata.json modules-qemux86-64.tgz We can also see that all the build Linux software is packaged in build/tmp-glibc/deploy/ipk. To build another image we can run: $ bitbake core-image-minimal-dev $ runqemu nographic slirp core-image-minimal-dev This creates an identical image with debug symbols. We find a set of files for a new rootfs next to the other ones: core-image-minimal-dev-qemux86-64.rootfs.ext4 core-image-minimal-dev-qemux86-64.rootfs.manifest core-image-minimal-dev-qemux86-64.rootfs.qemuboot.conf core-image-minimal-dev-qemux86-64.rootfs.spdx.tar.zst core-image-minimal-dev-qemux86-64.rootfs.tar.bz2 core-image-minimal-dev-qemux86-64.rootfs.testdata.json So we have managed to build a single kernel and two root filesystems. == Multiple builds == While you can build multiple root filesystems, that's about as far multiple outputs go. If you need to build for a different machine or a different distro you will need to use another configuration file. BitBake does support a way to use multiple configuration files, but I'm not exactly sure why you would want to use it instead of multiple build directories, especially if you have to do multi-architecture builds. The first thing we want to do is re-locate our build cache and downloads: $ cd build $ mv sstate-cache downloads .. They will now be in our oe-test directory. Next, open up build/conf/local.conf and add these lines to the top: DL_DIR ?= "${TOPDIR}/../downloads" SSTATE_DIR ?= "${TOPDIR}/../sstate-cache" This will save a lot of time for the next step where we create a new build directory. In the oe-test directory run: $ source openembedded-core/oe-init-build-env build2 Then perform the same edits to build/conf/local.conf to set DL_DIR and SSTATE_DIR. But also add this line to enable systemd: INIT_MANAGER = "systemd" Because of the shared state directory, this should re-use a lot of already built components. Let's build: $ bitbake core-image-minimal Indeed it did, but I found it was spending time rebuilding gcc-cross-x86_64! That would mean rebuilding basically everything else too. Why? I cancelled the build immediately to look. In each build directory I did this: $ bitbake -e gcc-cross-x86_64 | grep -v /home > cross.env Then in the main directory I ran: $ diff build/cross.env build2/cross.env This answers my question of why the compiler is being rebuilt pretty fast: 11338c11337 < #define STANDARD_STARTFILE_PREFIX_1 "/usr/lib/" --- > #define STANDARD_STARTFILE_PREFIX_1 "/lib/" 11340c11339 < #define SYSTEMLIBS_DIR "/usr/lib/" --- > #define SYSTEMLIBS_DIR "/lib/" In retrospect it's obvious: systemd requires a merged /usr, so the compiler will have to put its libraries in /usr/lib. This requires a rebuild of the compiler and probably everything else! Oh well. After building again: $ bitbake core-image-minimal I can now run the image in QEMU and verify it works: $ runqemu nographic slirp core-image-minimal The image boots to systemd managed system. Success! == Recipes == We can use oe-pkgdata-util to list and inspect built packages like this: $ oe-pkg-data-util list-pkgs | grep zlib zlib zlib-dbg zlib-dev zlib-doc zlib-src zlib-staticdev $ oe-pkg-data-util pkg-info zlib-dev zlib-dev 1.3.1-r0 zlib 1.3.1-r0 113538 $ oe-pkg-data-util list-pkg-files zlib-dev zlib-dev: /usr/include/zconf.h /usr/include/zlib.h /usr/lib/libz.so /usr/lib/pkgconfig/zlib.pc We can also find the list of packages for a particular root filesystem in its manfiest file, for example: $ grep 'openssl' tmp-glibc/deploy/images/qemux86-64/core-image-minimal-dev-qemux86-64.rootfs.manifest openssl-conf core2-64 3.2.1-r0 openssl-dev core2-64 3.2.1-r0 openssl-ossl-module-legacy core2-64 3.2.1-r0 While this is useful for dealing with built packages and images, most work is done with BitBake recipes. We can list buildable recipes like so: $ bitbake -s | grep ^linux-yocto linux-yocto :6.6.23+git-r0 We can list all recipes available like this: $ bitbake-layers show-recipes -r | grep ^linux-yocto linux-yocto linux-yocto-dev (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-dev to enable it) linux-yocto-rt (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-rt to enable it) linux-yocto-tiny (skipped: Set PREFERRED_PROVIDER_virtual/kernel to linux-yocto-tiny to enable it) In this case we can only build linux-yocto but there are other packages available. Listing dependencies of a recipe is a bit trickier. You can dump a recipe's dependency graph and view it like so: $ bitbake -g core-image-minimal # pn-buildlist now lists all recipes required to build core-image-minimal # task-depends.dot lists all tasks required to build core-image-minimal But this lists every dependency, not just immediate. To list immediate dependencies run: $ bitbake -e vim | grep -P '^R?DEPENDS.*=' DEPENDS="pkgconfig-native autoconf-native automake-native libtool-native libtool-cross virtual/x86_64-oe-linux-gcc virtual/x86_64-oe-linux-compilerlibs virtual/libc ncurses gettext-native desktop-file-utils acl gtk+3 xt virtual/update-alternatives" RDEPENDS:${KERNEL_PACKAGE_NAME}-base="" RDEPENDS:vim="ncurses-terminfo-base vim-xxd" RDEPENDS:vim-staticdev="vim-dev (= 9.1.0114-r0)" Two types of dependencies are shown: DEPENDS and RDEPENDS. DEPENDS is for build time dependencies, RDEPENDS is for runtime dependencies. For our purposes we only care about the RDEPENDS with the recipe name: "RDEPENDS:vim". It can also be useful to look at reverse dependencies. For example, to find out where busybox is being packaged or added to a build sysroot: $ bitbake -g core-image-minimal $ grep -P ' -> "busybox.do_(packagedata|populate_sysroot)' task-depends.dot "core-image-minimal.do_rootfs" -> "sqlite3.do_packagedata" "python3.do_package" -> "sqlite3.do_packagedata" "python3.do_package_write_ipk" -> "sqlite3.do_packagedata" "python3.do_prepare_recipe_sysroot" -> "sqlite3.do_populate_sysroot" "sqlite3.do_create_spdx" -> "sqlite3.do_packagedata" "sqlite3.do_package_qa" -> "sqlite3.do_packagedata" "sqlite3.do_package_write_ipk" -> "sqlite3.do_packagedata" Note that these BitBake commands will only tell us information about the package based on the overall configuration of the system. They won't tell you about dependencies that aren't enabled. Generally these are enabled or disabled using PACKAGECONFIG, so by looking at that variable we can find possible dependencies. Here's an example of finding available features and dependencies for dropbear: $ bitbake -e dropbear | sed '1,/^# $PACKAGECONFIG/d;/PACKAGECONFIG=/,$d' # :append[pn-qemu-system-native] /home/jookia/oe-test/build/conf/local.conf:215 # " sdl" # set /home/jookia/oe-test/openembedded-core/meta/conf/documentation.conf:321 # [doc] "This variable provides a means of enabling or disabling features of a recipe on a per-recipe basis." # set? /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:51 # "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:52 # [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:53 # [system-libtom] "--disable-bundled-libtom,--enable-bundled-libtom,libtommath libtomcrypt" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:54 # [disable-weak-ciphers] "" # set /home/jookia/oe-test/openembedded-core/meta/recipes-core/dropbear/dropbear_2022.83.bb:55 # [enable-x11-forwarding] "" # pre-expansion value: # "disable-weak-ciphers ${@bb.utils.filter('DISTRO_FEATURES', 'pam', d)}" The fancy sed command is just to find the PACKAGECONFIG block in the bitbake -e dump. You can look for it manually. Let's focus on the pam feature in the output: # [pam] "--enable-pam,--disable-pam,libpam,${PAM_PLUGINS}" The bracketed text is the feature and the quoted text is the PACKAGECONFIG variable with elements separated by commas in this style: PACKAGECONFIG[foo] = "--enable-foo,--disable-foo,foo_depends,foo_runtime_depends,foo_runtime_recommends,foo_conflict_packageconfig" It seems like the pam argument DEPENDS on libpam (the third column) and uses the PAM_PLUGINS variable for its RDEPENDS. This makes sense as PAM uses pluggable modules decided by configuration rather than this specific package. You may also see the variables DISTRO_FEATURES or MACHINE_FEATURES used to set PACKAGECONFIG. These variables are set in configurations and used to make broad changes to many recipes at once rather than setting individual package features. == Layers == BitBake supports the concept of layers: Sets of BitBake recipes that can add to or override existing recipes. This allows developers and packagers to create sets of re-usable layers independent of each other that can be combined to make a final project. By convention the layers start with the prefix 'meta-' and fall in to the following categories: * Package layers that provide software to use * Distro layers that configure the way the target system works * Machine layers that add support for specific hardware For this guide's project we will most likely need: * The core OpenEmbedded layer * A package layer for any new packages * A distro layer that uses systemd and the packages * A machine layer for the qemu virt device As a side note, layers are not intended for modifying the code of existing recipes. You should still use regular development branches and patches for that. For instance, with the qemu recipe bug I found earlier I made a new Git branch for the small change. These patches are then ideally sent upstream to the parent project. == Packaging == packaging a project: mkdir -p ../meta-bugfixes/recipes-general/evtone/ recipetool create -B main -S v1.4 https://git.lumina-sensum.com/git/Jookia/evtone.git -o ../meta-bugfixes/recipes-general/evtone/evtone_1.4.bb bitbake evtone # add 'inherit meson', remove configure stuff PV v1.4 tmp-glibc/work/core2-64-oe-linux/evtone/1.4/image tmp-glibc/deploy/ipk/core2-64/evtone changing version removes the old ipk in deploy/ Chapter X: Root filesystems --------------------------- - what components are in this image - multiple kernels? - build only rootfs - build only kernel # adding evtone to the image # TODO: introspect and ask: - how do we modify it for a different machine? - what is an image? - can we build just the rootfs? - initrd Chapter X: Mainline kernel -------------------------- Chapter X: Bootloader --------------------- Chapter X: Disk image --------------------- Chapter X: Board layer ---------------------- Chapter X: BSP layer -------------------- Chapter X: Development ---------------------- - using a custom srcdir for a package - sdks kind of aren't needed for us # SDK # TODO: multi configs? # todo nfsroot, unfsd? https://stackoverflow.com/questions/47429670/manually-building-a-kernel-source-from-yocto-build SRC_URI = "git:///path-to-linux-source/.git/;branch=${KBRANCH};protocol=file" INHERIT += "externalsrc" EXTERNALSRC:pn-myrecipe = "/path/to/my/source/tree" wic genimage
Summary:
Please note that all contributions to JookWiki are considered to be released under the Creative Commons Zero (Public Domain) (see
JookWiki:Copyrights
for details). If you do not want your writing to be edited mercilessly and redistributed at will, then do not submit it here.
You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource.
Do not submit copyrighted work without permission!
To edit this page, please answer the question that appears below (
more info
):
Who owns this wiki?
Cancel
Editing help
(opens in new window)