Session 05: Contributing to Unikraft

The focus of this session will be on porting new libraries to Unikraft and preparing them for upstreaming to the main organization’s GitHub.

Being a library operating system, the unikernels created using Unikraft are mainly a collection of internal and external libraries, alongside the ported application. As a consequence, a large library pool is mandatory in order to make this project compatible with as many applications as possible.

Reminders

From earlier sessions we saw that we can add an external library as a dependency for an application by appending it to the $LIBS variable of the application’s Makefile:

LIBS := $(UK_LIBS)/my_lib

Having done that, we can then select it in the menuconfig interface in order to be included in the build process.

Running an unikernel built for kvm can be done using the qemu command as follows:

$ qemu-system-x86_64 -kernel unikraft_unikernel -nographic

The -nographic argument redirects the output generated by the unikernel to the console.

In Session 02: Behind the Scenes we saw that there are two types of libraries:

internal, which are generally part of the kernel / core (schedulers, file systems, etc.);
external: which generally provide user space-level functionalities

The external libraries should be placed in the $UK_LIBS folder, which is by default $UK_WORKDIR/libs, and the applications should be placed in the $UK_APPS folder, which is by default $UK_WORKDIR/apps.

Overview

Support Files

Session support files are available in the repository. If you already cloned the repository, update it and enter the session directory:

$ cd path/to/repository/clone

$ git pull --rebase

$ cd content/en/docs/sessions/05-contributing-to-unikraft/

$ ls -F
index.md  work/

If you haven’t cloned the repository yet, clone it and enter the session directory:

$ git clone https://github.com/unikraft/summer-of-code-2021

$ cd summer-of-code-2021/content/en/docs/sessions/05-contributing-to-unikraft/

$ ls -F
index.md  work/

Git Structure

The organization’s GitHub contains the main Unikraft repository and separate repositories for external libraries, as well as already ported apps. In the previous sessions, we saw that the Unikraft repository consists of internal libraries, platform code and architecture code. It doesn’t have any external dependencies, in contrast to the external libraries or applications, which can have external dependencies.

External libraries can have more specific purposes. So, we can port a library even just for a single application. The process of adding new internal libraries is almost the same as for external ones, so further we will focus on porting an external library.

Also, the main repository has open issues to which you can contribute. In general, this process is done by solving the issue on a fork of the project, and after that making a pull request (PR) with your solution.

Example of External Library

Let’s focus for now on an already ported library: lib-libhogweed. Let’s examine its core components. Go to the work/01-tut-porting/libs/libhogweed/ directory and follow the bookmarks marked with USOC_X, where X is the index of the item in the list, from the files specified in the sections below.

Glue Code

In some cases, not all the dependencies of an external library are already present in the Unikraft project, so the solution is to add them manually, as glue code, to the library’s sources.

Another situation when we need glue code is when the ported library comes with test modules, used for testing the library’s functionalities. The goal, in this case, is to wrap all the test modules into one single function. In this way, we can check the library integrity if we want so by just a single function call. Moreover, we can create a test framework which can periodically check all of the ported libraries, useful especially for detecting if a new library will interfere with an already ported one.

Moving back to libhogweed, a practical example of the second case is the run_all_libhogweed_tests(int v) function from libhogweed/testutils_glue.c, line #674, which calls every selected (we will see later how we can make selectable config variables) test module and exits with EXIT_SUCCESS only if it passes over all the tests. For exposing this API, we should also make a header file with all of the test modules, as well as our wrapper function.

Note: Check libhogweed/include/testutils_glue.h.

Config.uk

The Config.uk file stores all the config variables, which will be visible in make menuconfig. These variables can be accessed from Makefile.uk or even from C sources, by including "uk/config.h", using the prefix CONFIG_.

Moving to the source code, libhogweed/Config.uk, we have:

The main variable of the library which acts as an identifier for it:

config LIBHOGWEED
	bool "libhogweed - Public-key algorithms"
	default n

We can also set another library’s main variable, in this case newlib, which involves including it in the build process:
```
select LIBNEWLIBC
```

Creating an auxiliary menu, containing all the test cases:

menuconfig TESTSUITE
     bool "testsuite - tests for libhogweed"
     default n
     if TESTSUITE
         config TEST_X
             bool "test x functionality"
             default y
     endif

Each test case has its own variable in order to allow testing just some tests from the whole suite.

Makefile.uk

The libhogweed/Makefile.uk file is used to:

Register the library to Unikraft’s build system:
```
$(eval $(call addlib_s,libhogweed,$(CONFIG_LIBHOGWEED)))
```
As you can see, we are registering the library to Unikraft’s build system only if the main library’s config variable, LIBHOGWEED, is set.

Set the URL from where the library will be automatically downloaded at build time:

LIBHOGWEED_VERSION=3.6
LIBHOGWEED_URL=https://ftp.gnu.org/gnu/nettle/nettle-$(LIBHOGWEED_VERSION).tar.gz

Declare helper variables for the most used paths:
```
LIBHOGWEED_EXTRACTED = $(LIBHOGWEED_ORIGIN)/nettle-$(LIBHOGWEED_VERSION)
```
There are some useful default variables, for example:
- $LIBNAME_ORIGIN: represents the path where the original library is downloaded and extracted during the build process;
- $LIBNAME_BASE: represents the path of the ported library sources(the path appended to the $LIBS variable).
You can check all reserved variables in the main documentation.
Set the locations where the headers are searched:
```
// including the path of the glue header added by us
LIBHOGWEED_COMMON_INCLUDES-y += -I$(LIBHOGWEED_BASE)/include
```
You should include the directories with the default library’s headers as well as the directories with the glue headers created by you, if it’s the case.

Add compile flags, used in general for suppressing some compile warnings and making the build process neater:

LIBHOGWEED_SUPPRESS_FLAGS += -Wno-unused-parameter \
        -Wno-unused-variable -Wno-unused-value -Wno-unused-function \
        -Wno-missing-field-initializers -Wno-implicit-fallthrough \
        -Wno-sign-compare

LIBHOGWEED_CFLAGS-y   += $(LIBHOGWEED_SUPPRESS_FLAGS) \
        -Wno-pointer-to-int-cast -Wno-int-to-pointer-cast
LIBHOGWEED_CXXFLAGS-y += $(LIBHOGWEED_SUPPRESS_FLAGS)

LIBHOGWEED_SRCS-y += $(LIBHOGWEED_EXTRACTED)/bignum.c

Register the library’s tests:
```
ifeq ($(CONFIG_RSA_COMPUTE_ROOT_TEST),y)
LIBHOGWEED_SRCS-y += $(LIBHOGWEED_EXTRACTED)/testsuite/rsa-compute-root-test.c
LIBHOGWEED_RSA-COMPUTE-ROOT-TEST_FLAGS-y += -Dtest_main=rsa_compute_root_test
endif
```
There are situations when the test cases have each a main() function. In order to wrap all the tests into one single main function, we have to modify their main function name by using preprocessing symbols.
You can read more about compile flags in the main documentation.
Note: A good practice is to include a test only if the config variable corresponding to that test is set.

This step is very customizable, being like a script executed before starting to compile the unikernel.

In most cases, and in this case too, the libraries build their own config file through a provided executable, usually named configure:

$(LIBHOGWEED_EXTRACTED)/config.h: $(LIBHOGWEED_BUILD)/.origin
	$(call verbose_cmd,CONFIG,libhogweed: $(notdir $@), \
        cd $(LIBHOGWEED_EXTRACTED) && ./configure --enable-mini-gmp \
    )
LIBHOGWEED_PREPARED_DEPS = $(LIBHOGWEED_EXTRACTED)/config.h

$(LIBHOGWEED_BUILD)/.prepared: $(LIBHOGWEED_PREPARED_DEPS)

UK_PREPARE += $(LIBHOGWEED_BUILD)/.prepared

We can also do things like generating headers using the original building system, modify sources, etc.

Warm-Up

Let’s check the integrity of this library using its test suite through the exposed wrapper function.

For this task, you have to move, or clone, the library in the $UK_LIBS folder and the work/01-tut-porting/apps/app-libhogweed application in the $UK_APPS folder. Fill the TODO lines from the application code: add the libhogweed library as a dependency in its Makefile and call from main.c the function exposed by the library for running the test suite.

Disable some tests, rebuild, and run again the checker application.

Note: The libhogweed library depends on newlib.

Note: Remember to select the test suite from menuconfig. You can also check the library’s README.md for additional information.

Summary

We need a large library pool in order to make the Unikraft project compatible with as many applications as possible.

There are also many ways in which you can contribute to the Unikraft project, and you can find them in the issues section of the main repository.

Practical Work

Moving to a more hands-on experience, let’s port a new library.

Support Files

Session support files are available in the repository. If you already cloned the repository, update it and enter the session directory:

$ cd path/to/repository/clone

$ git pull --rebase

$ cd content/en/docs/sessions/05-contributing-to-unikraft/

$ ls -F
index.md  work/

If you haven’t cloned the repository yet, clone it and enter the session directory:

$ git clone https://github.com/unikraft/summer-of-code-2021

$ cd summer-of-code-2021/content/en/docs/sessions/05-contributing-to-unikraft/

$ ls -F
index.md  work/

00. Prepare

Let’s suppose that we need kd tree support and that we found a C library, kdtree, that does what we need. After downloading and inspecting this library, we can see that it also has a set of examples, which can be used by us to test if we ported this library properly. Move the skeleton of this library, work/02-task-porting/src/libs/kdtree/, in the $UK_LIBS directory and complete the porting process by following the TODO lines.

01. Declare Library Identifier

Let’s start by declaring a new config variable in the Config.uk file. As stated before, this variable will represent the library’s identifier.

02. Register it to the Build System

For the next steps, the working file will be Makefile.uk from the library’s skeleton. Let’s use the previously declared variable: register the library to the build system only if the variable is set.

03. Set its URL

Having the library registered, set the URL from where it will be downloaded at build time, and explicitly fetch it.

04 Helper Variables

Make a variable with the path of the default directory obtained by extracting the original library’s archive.

05. Headers Location

Add the directory which contains the library’s header.

Hint: Inspect $LIBKDTREE_EXTRACTED.

06. Add Sources

Add the library’s C sources.

Hint: Inspect $LIBKDTREE_EXTRACTED.

07. Additional Requirements

Check the original library’s README to see if it needs to be configured first, and add the proper rule if so.

08. Intermediary Check

Until now we have registered the library and its sources, and we should be able to compile an unikernel with it if it doesn’t have any more unresolved dependencies. Move the work/02-task-porting/src/apps/app-kdtree application in the $UK_APPS directory, fill its Makefile, and use it to build an unikernel with our ported library as a dependency!

If needed, provide additional flags in order to suppress the compile warnings generated by this library.

Note: You can leave the application’s main() function empty, the resulted unikernel will just print the Unikraft` banner.

Hint: You can readme, but the solution isn’t here.

09. Add Test Config Variables

Now let’s make a wrapper for the test cases provided as examples. Uncomment lines #7-#15 from the library’s Config.uk and complete TODO_9 by adding new config variables for each test case.

10. Register Test Sources

Moving back to the library’s Makefile.uk, register the tests sources to the build system.

Note: Inspect the functions from the tests.

Note: Don’t forget to uncomment the lines.

11. Wrapper Glue

Integrate all the test functions into a glue main. Also, update the library’s include/test_suite_glue.h header accordingly.

Note: You can use test_suite_glue.c from the library’s skeleton.

12. Register Glue Code

Note: Don’t forget to uncomment the lines.

13. Final Verification

Test the resulted library by calling the test function from the app-kdtree application.

14. Give Us Feedback

We want to know how to make the next sessions better. For this we need your feedback. Thank you!

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified September 4, 2021: Fix ukstore link and add more resources (18c8d7c)

Session 05: Contributing to Unikraft

Reminders

Overview

Support Files

Git Structure

Example of External Library

Glue Code

Config.uk

Makefile.uk

Warm-Up

Summary

Practical Work

Support Files

00. Prepare

01. Declare Library Identifier

02. Register it to the Build System

03. Set its URL

04 Helper Variables

05. Headers Location

06. Add Sources

07. Additional Requirements

08. Intermediary Check

09. Add Test Config Variables

10. Register Test Sources

11. Wrapper Glue

12. Register Glue Code

13. Final Verification

14. Give Us Feedback

Further Reading

Feedback