Toro kernel

A dedicated kernel for multi-threading applications.

Friday, December 08, 2017

Reducing CPU usage on VMs that run Toro

Last days I worked on reducing the CPU usage of Toro. I observed that VMs that run Toro consume 100% of CPU which makes any solution based on Toro impossible for production. I identified four situations in which an idle loop is the issue:

  1. During a spin locks
  2. When there is no a thread in ready state 
  3. When there is no thread 
  4. When threads only poll a variable
Cases 1, 2 and 3 are in the kernel code. However, case 4 is when a user thread does idle work by polling a variable. So in this case, the solution would be harder since the scheduler has to figure out that the thread is only polling. Intel proposes different mechanisms to reduce the impact of idle loops [1, 2]. In particular, I was interested on the use of mwait/monitor instructions. However, I found this is not very well supported on all Hypervisors. So I have to base on the instructions hlt (halt) and pause.

First, I tackled case 1 by introducing the pause instruction inside the loop. This relaxes the CPU when a thread is getting exclusive access to a resource. Cases 2 and 3 were improved by using hlt which just halts the CPU until next interruption. To tackle case 4, I proposed two APIs to tell the scheduler when a thread is polling a variable. When scheduler figures out that all threads in a core are polling a variable, it just turns the core off.

I tested this in my baremetal host (4 cores, 8 GB, 2GHz) in Scaleway with KVM and a VM running Toro. I also installed Monitorix to monitor the state of the host. To stress the system, I generate http traffic and monitor the CPU usage. A few seconds after the stress stopped, the CPU usage of the qemu process is only about 1%. This usage goes up and down depending on the stress. By topping, I get a patter like this:   

CPU%
6.6    0.6     0:46.92 qemu-system-x86
63.8  0.6     0:48.84 qemu-system-x86 (stress)
99.7  0.6     0:51.84 qemu-system-x86 (stress)
45.5  0.6     0:53.21 qemu-system-x86 (stress)
2.3    0.6     0:53.28 qemu-system-x86
4.0    0.6     0:53.40 qemu-system-x86
6.6    0.6     0:53.60 qemu-system-x86

This is not always the case and sometimes Toro takes longer to turn off. This may happen when a socket is not correctly closed and it ends up by timeout. I need to experiment more to measure how much reaction the system losses when the core is halted. This recent work, however, seems very promising!

[1] https://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/IdleTalk.pdf
[2] Intel Volume 3, section 8.10.2 and 8.10.4

Thursday, November 30, 2017

Running Toro on top of VirtualBox

I am glad to announce that Toro is supported by VirtualBox. After bug fixing in the bootloader, Toro perfectly boots on VirtualBox:



This way Toro can run on Hyper-V, VMWare, QEMU, KVM and now VirtualBox. 

Wednesday, November 08, 2017

Running Toro on top of Hyper-V

Hello folks! I am working on the bootloader of Toro to be able to boot on HyperV. First experiments seem very interesting! I could run the example ToroHello on top of HyperV.



I found that the problem was the use of mm0 register in the bootloader. While this works well in KVM or QEMU, in Hyper-V the system hangs/crashes. By using a general purpose register like EBX, the problem is fixed and the kernel is loaded. I could not figure out why this is happening. Also, it was a bit tricky to wake up the other cores but, finally, I got them up. In the the near future, Hyper-V will be installed on every Windows in this context Toro can be an option of containers in a Windows environment. 

Saturday, July 15, 2017

The four steps to deploy a Toro application in the Cloud


Hi folks! Toro makes very simple the deployment of standalone-applications on Clouds. I just committed the script SetupCloudGuest.sh which is meant to install the needed tools to run a Toro application in a VM and then makes it public on internet by relying on port forwarding. These are the four steps:
1) Get a machine in Scaleway that will be used to run KVM and the Toro's guests.
2) Clone Toro repo. This contains the needed scripts to setup the host.
3) Run torokernel/tests/SetupCloudGuest.sh. This installs KVM and other needed tools.
3) Download the image that run the guest. For example, you can download TorowithFileSystem.img from here. The image must be copied to torokernel/tests. This workflow supposes that the development is done in other machine and the Toro image is just copied from your local machine to the host.
4) Finally, run Cloudit.sh TorowithFileSystem onlykvm
 
That's all folks! You can get the guest's console by using a VNC Viewer to port 5900, or connect to port 80 of guest by connecting to port 80 of host.