Is it possible to start LXC container inside LXC container?

I'm going to dispel a few myths here.

This is just a bad idea. I'm sorry. – Jacob Mar 5 at 20:30

I don't see how this is a bad idea. It's really just a chroot inside a chroot. On one hand, it could possibly decrease performance in some negligible manner (nothing compared to running a VM inside a VM). On the other hand, it's likely to be more secure (e.g. more isolated from the root host system and it's constituents).

Do you actually have a real reason to do this? Please remember that questions here should be about actual problems that you face. – Zoredache Mar 5 at 21:52

I agree 100% with the poster's following comment. Furthermore, I think it's safe to assume that everybody who posts a question on here likely thinks that they have a real reason to do [ it ]..

I think, that lxc should be able to simplify VM migration(and backup+recovery too). But I'm not sure about cases, when there is no access to host OS(cheap vps for example). – Mikhail Mar 6 at 11:17

I actually came across this question back in June when I was first diving into LXC for PaaS/IaaS projects, and I was particularly interested in the ability to allow users to emulate cloud environments for development purposes.

LXCeption. We're too deep. – Tom O'Connor Mar 6 at 22:46

I laughed a little bit when I read this one, but that's not, at all, the case :)

Anyway, I eventually set up a VirtualBox environment with a stock install of Ubuntu 12.04 LTS Server Edition after reading all this, thinking that this was 100% possible. After installing LXC, I created a new container, and installed LXC inside the container with apt-get. Most of the installation progressed well, but resulted in error eventually due to a problem with the cgroup-lite package, whose upstart job failed to start after the package had been installed.

After a bit of searching, I came across this fine article at stgraber.org (the goodies are hiding under the "Container Nesting" section):

sudo apt-get install lxc
sudo lxc-create -t ubuntu -n my-host-container -t ubuntu
sudo wget https://www.stgraber.org/download/lxc-with-nesting -O /etc/apparmor.d/lxc/lxc-with-nesting
sudo /etc/init.d/apparmor reload
sudo sed -i "s/#lxc.aa_profile = unconfined/lxc.aa_profile = lxc-container-with-nesting/" /var/lib/lxc/my-host-container/config
sudo lxc-start -n my-host-container
(in my-host-container) sudo apt-get install lxc
(in my-host-container) sudo stop lxc
(in my-host-container) sudo sed -i "s/10.0.3/10.0.4/g" /etc/default/lxc
(in my-host-container) sudo start lxc
(in my-host-container) sudo lxc-create -n my-sub-container -t ubuntu
(in my-host-container) sudo lxc-start -n my-sub-container

Installing that AppArmor policy and restarting the daemon did the trick (don't forget to change the network ranges, though!). In fact, I thought that particular snippet was so important that I mirrored it @ http://pastebin.com/JDFp6cTB just in case the article ever goes offline.

After that, sudo /etc/init.d/cgroup-lite start succeeded and it was smooth sailing.

So, yes, it is possible to start an LXC container inside of another LXC container :)

With Ubuntu 14.04 (trusty) you can simply add the following in the parent container config:

lxc.mount.auto = cgroup
lxc.aa_profile = lxc-container-default-with-nesting

reference: https://ubuntu.com/server/docs/containers-lxc (search for "nesting)

Make sure that you have pre-configured network before booting to avoid an long pause before the login screen appears!

HTH

Yes you can do nested LXC containers and despite the 1st comment there are times and use-cases where Nested containers are certainly useful. See Stephane Graber's 10 part LXC blog but in particular the section Container Nesting -

Stephane Graber's 10 part series on LXC

use-cases: Suppose you want a mult-tenant LXC environment. Create 1 Master container for each person or organization making sure to enable Nesting by adding the 2 cmds to the LXC container config file. Next in each Master container create your nested sub-containers where you install the apps, desktops etc that each group requires. NOTE that whereas the default network for the Master containers will be 10.0.3.x the nested containers will be 10.0.4.x by default (you can change either if you need to).

What is the biggest advantage I 've had using Nested LXC? If you lxc-stop the Master container and lxc-clone it.. you clone not only the Master but all the sub-containers... this is handy for quick backups. This approach is also handy if you ever want to do LXC live migration w/CRIU. When you migrate one of the Master containers to another Machine... you are actually migrating it and all nested containers as well.

Lastly, for a cool example of LXC nesting use Stephane Graber & others built a simulator for "The Internet" using LXC, BPG & OSPF all in 1 LXC container. Inside that 1 LXC "master or parent" LXC container there are 512 nested LXC containers each running Quagga for BGP/OSPF routing. Together those 512 Internet "nodes" simulate the Internet. This implementation was used at the 2014 NSEC security conference for all the attendees to experiment with security in the Internet.

The source for this is on Githug at: 2014 NSEC LXC simulator for The Internet github code

Also... did you know you can now install ALL of Openstack into a single LXC container. Each of the Openstack "services" (nova, swift etc) are then all installed into "nested" lxc containers inside the "master/parent" container.

Takes a while to install everything but when done you have a nice testing OpenStack environment on your laptop or desktop to experiment with.

If you want to stop OpenStack you just lxc-stop the master/parent container same to restart Openstack.

see: Openstack Single Installer instructions