Simple Multitasker


Even though Codezero hypervisor implements it’s own round-robin, preemptive scheduler, it is a common requirement for containers to implement their own scheduling. For example, Linux kernel implements its own guest scheduler, on top of the base Codezero scheduler. Due to it’s complexity we do not present this case here. Instead, we present a simple multitasker that implements a basic guest scheduler, as part of a baremetal container.

To get started, load the appropriate meta-config first:

./ vx-cswitch-test

If you’re doing this for a first time, it will copy a necessary files into codezero-meta/cswitch-test directory.

The Guest-Side Scheduling demonstration uses serial port to print output messages. Use:


script to connect to it.

In another terminal start the debug session using:


And use continue command to start the system.

Basic concepts

IRQ handling

Codezero provides guests with an ability to handle asynchronous IRQs. Implementing IRQ handling is a first necessary part for using Guest-Side Scheduling as it allows a form of timer-based preemption.

Each thread in a IRQ handling container must have an IRQ handler registered. This means that before starting a thread, an l4_exchange_register call must be issued on behalf of a newly spawned thread. In cswitch-test it’s done in the following part of the code:

exregs_set_handler_pc(&exregs, l4_irq_handler, (unsigned long)timer_irq_handler);
exregs_set_handler_sp(&exregs, l4_irq_handler, (unsigned long)init_irq_stack);

if ((err = l4_exchange_registers(&exregs, ids.tid)) < 0) {
        printf("l4_exchange_registers error\n");

The timer_irq_handler is a pointer IRQ handler function that will fired asynchronously upon IRQ arrival and the init_irq_stack is a pointer to a IRQ stack memory that will be used as a stack.

For the specific IRQ to get delivered to the guest it must be first registered via l4_irq_control like this:

/* Register and enable irqs */
if ((err = l4_irq_control(IRQ_CONTROL_REGISTER, IRQ_USER_HANDLER, TIMER_IRQ)) < 0) {
        printf("%s: FATAL: Timer irq could not be registered. "
               "err=%d\n", __FUNCTION__, err);

Also the container has an ability to disable/enable IRQs delivery globally (guest-wise), so it needs to enable them:

/* Enable container IRQ delivery */
irq_global_state = PARAVIRT_IRQS_ENABLED;

And the underlying hardware (a timer, in this case) must be initialized:

/* Start the periodic timer */
timer_init_periodic(timer_base, TIMER_VALUE);

The handler function is doing several important steps to properly handle the IRQ.

The first is to globally disable the IRQS for the crucial part of handling current one:

irq_global_state = PARAVIRT_IRQS_DISABLED;

Acknowledge the IRQ to hardware device:


Then the IRQ must be acknowledge to the hypervisor:

if ((err = l4_irq_control(IRQ_CONTROL_ACK_MASK, 0, irq)) < 0) {
       printf("Irq ack/mask failed. err=%d, irq=%u\n", err, irq);
       return err;

and re-enabled:

if ((err = l4_irq_control(IRQ_CONTROL_ENABLE, 0, TIMER_IRQ)) < 0) {
        printf("Irq enable failed. err=%d, irq=%u\n", err, TIMER_IRQ);

Afterwards, the IRQ delivery can be globally re-enabled:

irq_global_state = PARAVIRT_IRQS_ENABLED;

To end the handler and return to the interrupted context, the l4_context_switch is used with CSWITCH_PREV_CONTEXT flag.

l4_context_switch(0, CSWITCH_PREV_CONTEXT, 0);

Context switching

The other part of the Guest-Side Scheduling is context switching. This part is implemented in cswitch-test example in the schedule function.

While the selection of destination thread to switch to is arbitrary and depends on the guest scheduling logic, the switching itself must be done using l4_context_switch API call with a CSWITCH_STOP flag, like this:

    if ((l4_context_switch(threads[to].tid, CSWITCH_STOP, 0) < 0))

This call will pause the currently running thread, no matter if in normal context or in asynchronous IRQ handling procedure and resume the destination thread which tid (thread id) is passed as a first argument. This in effect works exactly like a normal context switch in native operating systems, switching from one thread to the other.

Further reading

The interesting file is codezero-meta/cswitch-test/main.c. It’s well documented so use the source code to understand how does Guest-Side Scheduling works. You may also want to read the Hypervisor API pages for the involved syscalls.

Comments are closed.