No network connectivity in the LXC container set up in the "routed" mode

There are several things are required to make your scheme work:

1. Check the forwarding on the host interfaces: ip -4 netconf show dev vetha8400046 and ip -4 netconf show dev eth0. You should see the enabled forwarding in the output. Otherwise enable it with the sysctl -w net.ipv4.conf.<iface>.forwarding=1 command. Add the corresponded strings into the /etc/sysctl.conf file to make it persistent.

2. Check the routing on the host: ip -4 route get 8.8.8.8 from 192.168.33.11 iif vetha8400046. You should see the valid route through your default gateway (... via 10.0.2.2 dev .... Check also the reverse route with the ip -4 route get 192.168.33.11 from 8.8.8.8 iif eth0.

3. Ping the container from host and vice versa.

4. Setup the NAT on the host: iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE. Ping some external host from the container. The counters of this rule in the iptables-save -c should be non-zero.

5. Ask the question if you stuck.

This is the correct setup for having Debian 10 containers accessible in our local network by their own static IPs, and them having access to the internet.

Our Vagrantfile:

Vagrant.configure("2") do |config|

  config.vm.define "main" do |main|
    main.vm.box = "bento/ubuntu-20.04"
    main.vm.box_version = "202010.24.0"
    main.vm.hostname = "lxc-host"

    main.vm.network "public_network", auto_config: false

    main.vm.provision "shell",
      run: "always",
      inline: "ip address add **192.168.1.200**/24 dev eth1"
    main.vm.provision "shell",
      run: "always",
      inline: "ip link set eth1 up"

    main.vm.provider :virtualbox do |vb|
        vb.memory = 1024
    end
  end

end

Notice that we are setting up "public" network, it means that when starting up Vagrant, you will be asked to choose interface to use, choose the one that you are using to connect to your local network.

Notice also that we are setting up only a single IP for now, this (192.168.1.200) will be our host's IP. We are not setting up container's IP here.

After starting up our host, enable IP forwarding in its configuration, by uncommenting: net.ipv4.ip_forward=1 in /etc/sysctl.conf and restarting sysctl by executing systemctl restart systemd-sysctl .

Now, assuming that you installed and set up LXD correctly on the host, you can init lxd with the following configuration:

config:
  core.https_address: '[::]:8443'
  core.trust_password: true
networks: []
storage_pools:
- config:
    source: [path-to-storage-pools-directory]
  description: ""
  name: default
  driver: dir
profiles:
- name: default
  config:
  description: ""
  devices:
    root:
      path: /
      pool: default
      type: disk
- name: test-container
  config:
    user.user-data: |
      #cloud-config
      bootcmd:
        - echo 'nameserver 8.8.8.8' > /etc/resolvconf/resolv.conf.d/tail
        - systemctl restart resolvconf
  description: Mail Server LXD profile
  devices:
    eth0:
      ipv4.address: 192.168.1.201
      nictype: routed
      parent: eth1
      type: nic
cluster: null

Change [path-to-storage-pools-directory] into actual path to the directory where you are going to store your pools.

Note that we are not using cloud-init to set up networking, as it would collide with Debian's networking service making it fail to start.

Note also that we are using cloud-init to set up nameserver to use by our guest.

192.168.1.201 will be our guest's IP. This setup assumes that the local network is 192.168.1.0, if your network is different, change IPs accordingly. Also make sure that the IPs used here are not used by anything else in the network already.

We are using two profiles here but it's up to you how you organize your configuration.

Now let's run our container:

lxc launch images:debian/10/cloud our-actual-test-container --profile default --profile test-container

It should be running, and be available in the local network, but it won't have access to the internet, and that's the tricky part. This is caused by routing set up by the lxd based on the assumption that default gateway is the one to the vagrant/virtualbox network. We need to fix that.

When executing ip r on the host, you should see the list of routes - something like that:

default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100 
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100 
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.200 
192.168.1.201 dev vethc5cebe03 scope link

The issue is with the default one, which is routing traffic into the vagrant/virtualbox network instead of our actual gateway. Let's remove it:

ip route del default via 10.0.2.2

And let's add the rule that will forward packets to the actual gateway of our local network:

ip route add default via 192.168.1.1

(assuming that 192.168.1.1 is IP of our local gateway)

Now the traffic from the container will be going to the correct gateway, and we are done!