The other day I was trying out Linux Containers (LXC) on my Debian Jessie server and found that Debian's support for controlling it with libvirt is somewhat inconclusive. I already had a basic libvirt setup for controlling a handful of virtual machines so I really wanted it to work also with LXC if possible.

Background

The basic steps to create a container and control it with libvirt are the following (ref):

  1. Create a container using the LXC userspace tools.
  2. Create a libvirt XML domain definition.
  3. "Define" the libvirt container domain using the libvirt userspace tools.
  4. Start the libvirt container domain using the libvirt userspace tools.

See for example Scott Lowe's blog post for information on how to create the actual container. Once that is done you need to write a basic libvirt domain definition.

Domain definition

The following is a minimal domain definition.

<domain type='lxc'>
  <name>debian</name>
  <memory unit="GiB">8</memory>
  <os>
    <type>exe</type>
    <init>/sbin/init</init>
  </os>
  <vcpu>4</vcpu>
  <devices>
    <emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
    <filesystem type='mount'>
      <source dir='/var/lib/lxc/debian/rootfs'/>
      <target dir='/'/>
    </filesystem>
    <interface type='bridge'>
      <source bridge='br0'/>
    </interface>
    <console type='pty'/>
  </devices>
</domain>

View raw file

Most elements are self-explanatory but I want to point out a few of them:

  • <memory> is mandatory even if you don't want to set a memory limit. This is problematic; see the next section.
  • <vcpu> defaults to 1 if not set, but you probably want to allow the container access to more than one CPU core.
  • I am using an existing network bridge (br0). libvirt supports many types of virtualized networks (NAT, host-only, PCI device passthrough etc.) and each has a slightly different syntax for the <interface> element. See the official documentation for further information.
  • You need some sort of console for the container to start. I am not sure if this is a restriction imposed by libvirt or if the Linux kernel or init system simply refuse to boot without one. Easiest seems to be a pseudo terminal (PTY) which you can then later attach to (for example using the console subcommand of virsh).

The cgroups memory controller

As mentioned, it is mandatory to set a memory limit (even if it is a really high limit) for libvirt to accept the domain definition. The limit is enforced by the cgroups memory controller. The Linux kernel in Debian Jessie is compiled with support for the memory controller (kernel config parameter CONFIG_MEMCG=y). However, it is operationally disabled by default (kernel config paramenter CONFIG_MEMCG_DISABLED=y). Raymond P. Burkholder researched this and found that this is for performance reasons and links to relevant Debian bug reports, because apparently enforcing the memory limit has a certain amount of overhead.

I'll deal with that performance problem when it makes itself apparent and for now I want to enable the cgroups memory controller. This is easily done by setting the cgroup_enable=memory kernel boot parameter. Append it to GRUB_CMDLINE_LINUX in /etc/default/grub, update-grub and reboot.

Double consoles

Now the container starts properly in libvirt. However, attaching to its console and attempting to login yields output similar to the following:

~~~ [ OK ] Started Permit User Sessions. Starting Getty on tty3... [ OK ] Started Getty on tty3. Starting Getty on tty2... [ OK ] Started Getty on tty2. Starting Getty on tty4... [ OK ] Started Getty on tty4. Starting Getty on tty1... [ OK ] Started Getty on tty1. Starting Console Getty... [ OK ] Started Console Getty. [ OK ] Reached target Login Prompts. [ OK ] Reached target Multi-User System. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes.

Debian GNU/Linux 8 debian console

debian login: Debian GNU/Linux 8 debian tty1

debian login: Debian GNU/Linux 8 debian console

debian login: Debian GNU/Linux 8 debian tty1

debian login: root Password: p

Login incorrect ~~~

Two getty instances are started, one on the "console" and one on tty1 and both are competing for your input. Parts of the login credentials are read by each getty instance, causing the login to fail and the password to be partially printed to the terminal. This has been fixed by LXC upstream in the Debian container template recently, so that getty is now only started on one of the consoles.

Debian being Debian, we can probably look forward to this fix downstream in 2017 or 2018 or so. Until then, it can be worked around by logging into the container (using LXC tools) and disabling getty on tty1-6 so that it only runs on console. Console is a "pseudo TTY" which by default is not declared to be a "secure TTY" (allowing root to login using it). To allow root login it must also be added to /etc/securetty. The following shell script performs those two actions:

#!/bin/bash
# Start the container with lxc-start and run this in the console to prepare the
# OS for libvirt.

set -e

# Disable getty on TTYs
systemctl mask getty-static
rm /etc/systemd/system/getty.target.wants/*

# Allow root login from pseudo TTY 0 (console)
echo >> /etc/securetty
echo pts/0 >> /etc/securetty

# We are done. Power off and then start the container in libvirt.
poweroff

View raw file

Now the container can be started and controlled in libvirt.