cdauth's questions

As it is common, lxd provides two systemd units: lxd.socket and lxd.service. When lxd.socket is started, it starts lxd.service as soon as any application (such as lxc) tries to access the lxd daemon.

I would like lxd to start on boot, without having to run any command. However, when I try to enable lxd.service, it enables lxd.socket instead:

# systemctl enable lxd.service
Created symlink /etc/systemd/system/sockets.target.wants/lxd.socket → /usr/lib/systemd/system/lxd.socket.

What is going on here? How can I enable lxd.service?

The system is running Fedora 39. These are the unit files:

/usr/lib/systemd/system/lxd.service:

[Unit]
Description=LXD - main daemon
After=network-online.target openvswitch-switch.service lxcfs.service lxd.socket
Requires=network-online.target lxcfs.service lxd.socket
Documentation=man:lxd(1)

[Service]
Environment=LXD_DOCUMENTATION=/usr/share/doc/lxd-doc/html
Environment=LXD_OVMF_PATH=/usr/share/edk2/ovmf
Environment=LXD_UI=/usr/share/lxd-ui/ui
ExecStart=/usr/bin/lxd --group lxd
ExecStartPost=/usr/bin/lxd waitready --timeout=600
KillMode=process
TimeoutStartSec=600s
TimeoutStopSec=30s
Restart=on-failure
LimitNOFILE=1048576
LimitNPROC=infinity
TasksMax=infinity

[Install]
Also=lxd-containers.service lxd.socket

/usr/lib/systemd/system/lxd.socket:

[Unit]
Description=LXD - unix socket
Documentation=man:lxd(1)

[Socket]
ListenStream=/var/lib/lxd/unix.socket
SocketGroup=lxd
SocketMode=0660
Service=lxd.service

[Install]
WantedBy=sockets.target

I have a server lanserver running in my private LAN that is connected to a public server publicserver using Wireguard. publicserver forwards TCP connections to certain ports to lanserver through the Wireguard connection using an iptables DNAT rule.

On lanserver, Wireguard is set up as a NetworkManager connection. It is routing all internet traffic through Wireguard using AllowedIPs = 0.0.0.0/0, ::/0. This causes the following ip rules to be set up on lanserver:

[root@lanserver ~]# ip rule show
0:      from all lookup local
31100:  from all lookup main suppress_prefixlength 0
31101:  not from all fwmark 0xcb2e lookup 52014
32766:  from all lookup main
32767:  from all lookup default
[root@lanserver ~]# ip route show table 52014
default dev wg0 proto static scope link metric 50

This setup works properly. An incoming TCP connection to publicserver gets forwarded to lanserver using the iptables rule. The response is sent back through publicserver because the ip rule 31101 matches it.

There is one exception: When I try to open a TCP connection to publicserver using IPv6 from my personal computer pc, which is also inside my private LAN, it does not work. The problem seems to be that both pc and lanserver have an IPv6 address in the same public subnet. publicserver forwards the connection to lanserver successfully, but the response is not routed back through the Wireguard connection but directly to pc due to ip rule 31100.

How can I make sure that all responses for connections coming in through the Wireguard interface on lanserver are also sent back through the Wireguard interface, regardless of whether their source IP is in a local subnet?

I can think of solutions in the following directions:

• Disable IPv6 on lanserver, causing it to not be on the same subnet as pc. Not a very good solution.
• Use SNAT on publicserver for the port forward. Not an acceptable solution, since some of the services behind the forwarded ports need to know the true source IP.
• Explicitly mention the local IPv6 subnet in the AllowedIPs of the Wireguard peer. This does not work because the IPv6 subnet changes every 24 hours.
• Add a custom IP rule that somehow matches all connections coming in through wg0 and uses the routing table 52014 for them. I'm not sure how exactly to specify such a rule. Also, the problem is that the number of the routing table changes every time the Wireguard connection is restarted. The right place to create such a rule would probably be the PostUp script, but NetworkManager does not seem to allow specifying one.

I would like to restrict some users on my server to only being able to execute certain commands. To do this, the most common approach that I could find is to use rbash.

While I can find many websites talking about rbash, I am having trouble to find any information how to use it right. The most common approach that I could find was to create a symlink from /bin/rbash to /bin/bash, set the restricted users’ login shell to /bin/rbash and then set a custom PATH in ~/.bash_profile in the user home directory.

However, I was quite shocked to find out that with this setup users can still copy files to the server using scp and they can even open an unrestricted shell by using ssh user@host -t bash! What seems to be happening is that the SSH server is passing the command to the login shell on the server using -c, so ssh user@host -t bash causes the server to run /bin/rbash -c bash, which works because .bash_profile is not executed yet to restrict the path. scp similarly causes the server to run /bin/rbash -c scp.

Now I have come across the ForceCommand directive of sshd. This directive basically always causes the configured command to be passed as -c to the login shell, ignoring any command that the client has specified. So if ForceCommand is set to rbash, that will always execute the command /bin/rbash -c rbash on the server, regardless whether the client was called with -t bash or as scp or whatever. Unfortunately, /bin/rbash -c rbash causes the .bash_profile not to be executed, so we end up with a restricted shell but a normal PATH, so we can just call bash there to escape it.

What I would like to achieve:

• There should be no way to avoid the restricted shell for users connecting via SSH
• Ideally, it would still be possible to execute commands that are permitted in the restricted shell by using ssh user@server permitted_command
• The configuration should not be SSH-only, so users logging in for example on the TTY should also be restricted.

On my server I have nginx running using the default docker nginx image (version 1.13.8) as a reverse proxy for Lychee (using wonderfall/lychee), a photo gallery software written in PHP. As far as I can tell, the Lychee image itself also runs nginx with php7.1-fpm.

The gallery works fine, only when trying to download an image or an album, I get a strange error in the browser. Chromium says “Failed - Network error”, and other browsers give similar error messages. This error comes up when the download has finished, and when using a browser that keeps failed downloads, it can be seen that the downloaded file is actually complete.

The error only comes up when accessing Lychee through nginx and when using HTTP/2. The nginx log reports:

2018/01/15 15:37:34 [error] 8#8: *2960 upstream prematurely closed connection while sending to client, client: 192.168.255.1, server: example.com, request: "GET /2018/php/index.php?function=Photo::getArchive&photoID=123&password= HTTP/2.0", upstream: "http://10.93.1.23:8888/php/index.php?function=Photo::getArchive&photoID=123&password=", host: "example.com"

The Lychee log reports nothing.

When trying to download a file using curl -v, this is the output:

*   Trying 1.2.3.4...
* TCP_NODELAY set
* Connected to example.com (1.2.3.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [226 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2754 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [556 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=example.com
*  start date: Dec  4 10:54:54 2017 GMT
*  expire date: Mar  4 10:54:54 2018 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55e8df551160)
} [5 bytes data]
> GET /2018/php/index.php?function=Photo::getArchive&photoID=123&password= HTTP/2
> Host: example.com
> User-Agent: curl/7.57.0
> Accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
< HTTP/2 200 
< server: nginx/1.13.8
< date: Mon, 15 Jan 2018 15:37:34 GMT
< content-type: application/octet-stream
< content-length: 530149
< x-powered-by: PHP/7.1.2
< set-cookie: PHPSESSID=123; path=/2018/
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< cache-control: no-store, no-cache, must-revalidate
< pragma: no-cache
< content-disposition: attachment; filename="IMG_4872.jpg"
< strict-transport-security: max-age=31536000
< 
{ [3691 bytes data]
* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Closing connection 0
} [5 bytes data]
* TLSv1.2 (OUT), TLS alert, Client hello (1):
} [2 bytes data]

When downloading with HTTP/1.1, everything works fine:

*   Trying 1.2.3.4...
* TCP_NODELAY set
* Connected to example.com (1.2.3.4) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [223 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [112 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2754 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [556 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=example.com
*  start date: Dec  4 10:54:54 2017 GMT
*  expire date: Mar  4 10:54:54 2018 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
} [5 bytes data]
> GET /2018/php/index.php?function=Photo::getArchive&photoID=1234&password= HTTP/1.1
> Host: example.com
> User-Agent: curl/7.57.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 200 OK
< Server: nginx/1.13.8
< Date: Mon, 15 Jan 2018 15:43:36 GMT
< Content-Type: application/octet-stream
< Content-Length: 530149
< Connection: keep-alive
< X-Powered-By: PHP/7.1.2
< Set-Cookie: PHPSESSID=1234; path=/2018/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-Disposition: attachment; filename="IMG_4872.jpg"
< Strict-Transport-Security: max-age=31536000
< 
{ [3691 bytes data]
* Connection #0 to host example.com left intact

Finally, here is the output when accessing Lychee directly:

*   Trying 10.93.1.23...
* TCP_NODELAY set
* Connected to 10.93.1.23 (10.93.1.23) port 8888 (#0)
> GET /php/index.php?function=Photo::getArchive&photoID=1234&password= HTTP/1.1
> Host: 10.93.1.23:8888
> User-Agent: curl/7.56.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Mon, 15 Jan 2018 15:46:06 GMT
< Content-Type: application/octet-stream
< Content-Length: 530149
< Connection: keep-alive
< X-Powered-By: PHP/7.1.2
< Set-Cookie: PHPSESSID=1234; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-Disposition: attachment; filename="IMG_4872.jpg"
< 
{ [14070 bytes data]
* Connection #0 to host 10.93.1.23 left intact

I don't understand why nginx says that Lychee prematurely closed the connection, when actually the whole file has been transfered (the Content-Length matches also), and running curl by hand also doesn't show any errors. Does anyone have an idea?

I have a server in my local network that connects to an external VPN server and gets assigned the public IP address 2a02:c200:1:10:3:0:4315:2. Currently, when other computers in my local network access the server using its public IP address, the connection is naturally routed over the internet via the VPN server. I would like to set up my network to route connections to the public IP address of the server locally.

Basically, I need to set up a route 2a02:c200:1:10:3:0:4315:2/128 via fe80::baae:edff:fe79:122 (which is the local address of the server).

The router in my local network does not let me specify static IPv6 routes. So I am trying to set up a router advertisement on the server that advertises the route automatically. Unfortunately, I cannot find any documentation about advertising routes with radvd.

Like in this question, I have configured radvd as follows:

interface enp0s25 {
    AdvSendAdvert on;
    AdvDefaultLifetime 0;

    route 2a02:c200:1:10:3:0:4315:2/128 {
    };
};

This is the advertisement that arrives on my computer (shown using radvdump):

#
# radvd configuration generated by radvdump 2.11
# based on Router Advertisement from fe80::baae:edff:fe79:122
# received by interface wlp2s0
#

interface wlp2s0
{
    AdvSendAdvert on;
    # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
    AdvManagedFlag off;
    AdvOtherConfigFlag off;
    AdvReachableTime 0;
    AdvRetransTimer 0;
    AdvCurHopLimit 64;
    AdvDefaultLifetime 0;
    AdvHomeAgentFlag off;
    AdvDefaultPreference medium;
    AdvSourceLLAddress on;

    route 2a02:c200:1:10::/128
    {
        AdvRouteLifetime 30;
    }; # End of route definition

}; # End of interface definition

As you can see, the route prefix got truncated for some reason. Unrelated to that, my computer (Linux) does not set up any route, despite setting net.ipv6.conf.all.accept_ra_rt_info_max_plen = 128.

Before I added AdvDefaultLifetime 0;, a route default via fe80::baae:edff:fe79:122 was set up, which is not supposed to be, so I followed the advice from this mail to add this parameter.

Does anyone have an idea what I could be doing wrong, or a link to some documentation about advertising routes with radvd?

I have a host1 that has a WAN interface eth0 (1.2.3.4) and a LAN interface eth1 (10.41.82.1). Inside the LAN is a host2 with the IP address 10.41.82.2.

Incoming connections to 1.2.3.4:22 are forwarded to host2 using the following iptables rules on host1:

root@host1:~# iptables -t nat -A OUTPUT     -d 1.2.3.4 -p tcp --dport 22 -j DNAT --to 10.41.82.2:22
root@host1:~# iptables -t nat -A PREROUTING -d 1.2.3.4 -p tcp --dport 22 -j DNAT --to 10.41.82.2:22

This works fine, except when host2 is trying to access 1.2.3.4:22:

root@host2:~# telnet 1.2.3.4 22
Trying 1.2.3.4...

root@host2:~# ping 1.2.3.4
PING 1.2.3.4 56(84) bytes of data.
64 bytes from 1.2.3.4: icmp_seq=1 ttl=64 time=0.125 ms
64 bytes from 1.2.3.4: icmp_seq=2 ttl=64 time=0.279 ms

root@host2:~# tcptraceroute 1.2.3.4 22
traceroute to 1.2.3.4, 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
(and so on)

It seems like the packets are being forwarded in a circle. As you can see, pinging works, so the routes and IP forwarding are set up properly.

What did I do wrong and how can I fix the problem?

UPDATE:

As mentioned in BillThor’s response and as is described in the frozentux tutorial on DNAT, I need to add an SNAT rule. I have done this now (note that I have added the LAN address of host2 for -d, as this rule is applied in POSTROUTING where the destination of the packet has already been changed):

root@host1:~# iptables -t nat -A POSTROUTING -d 10.41.82.2 -p tcp --dport 22 -j SNAT --to 10.41.82.1

Unfortunately, this does not solve the problem, and everything is still the same as described above. Interestingly, the SNAT rule never seems to be matched, according to the counters shown when running iptables -t nat -nvL.

UPDATE 2:

I discovered that when I run tcpdump -i eth1, even when I run it with parameters irrelevant to this case, such as tcpdump -i eth1 port 34238, it suddenly works. But only while tcpdump is running.

What I didn’t mention so far because I thought it was irrelevant is that eth1 is actually a bridge, and host2 is a Xen domain. I am starting to suspect that my problem might be related to this.

I have a server running on my local network that acts as a router for the computers in my network. I want to achieve now that outgoing TCP requests to certain IP addresses are tunnelled through an SSH connection, without giving the people from my network the possibility to use that SSH tunnel to connect to arbitrary hosts.

The approach I had in mind until now was to have an instance of redsocks listening on localhost and to redirect all outgoing requests to the IP addresses I want to divert to that redsocks instance. I added the following iptables rule:

iptables -t nat -A PREROUTING -p tcp -d 1.2.3.4 -j DNAT --to-destination 127.0.0.1:12345

Apparently, the Linux kernel considers packets coming from a non-127.0.0.0/8 address to an 127.0.0.0/8 address as “Martian packets” and drops them. What worked, though, was to have redsocks listen on eth0 instead of lo and then have iptables DNAT the packets to the eth0 address instead (or using a REDIRECT rule). The problem about this is that then every computer on my network can use the redsocks instance to connect to every host on the internet, but I want to limit its usage to a certain set of IP addresses only.

Is there any way to make iptables DNAT packets to 127.0.0.1? Otherwise, does anyone have an idea how I could achieve my goal without opening up the tunnel to everyone?

Update: I have also tried to change the source of the packets, without any success:

iptables -t nat -A POSTROUTING -p tcp -s 192.168.1.0/24 -d 1.2.3.4 -j SNAT --to-source 127.0.0.1
iptables -t nat -A POSTROUTING -p tcp -s 192.168.1.0/24 -d 127.0.0.1 -j SNAT --to-source 127.0.0.1

When I try to download a file via HTTP, the downloads sometimes stop after around 30 MB. The download rates goes down to 0 B/s and no data keeps coming. When I stop the download and resume again, the download still hangs. But when I redownload it from byte 0 again, everything works fine up to 30 MB when it stops again. Sometimes, after some hours, it just works again without problems. The position in the file when the download stops is variable, but most of the time it is around 30–35 MB.

As a download manager I use wget. The same behaviour happens though using curl and other download managers. The error occurs independently of the server I download from. I have also observed this error on other Linux computers in my network. All computers on my network run Gentoo Linux on x86.

All internet connections on my network go through a server on my network which runs a transparent Squid proxy on port 80. That server is connected to a router, which is a Speedport W 700V by Deutsche Telekom AG. That router is connected to the internet using ADSL, with 448 kbit/s down speed and 96 kbit/s up speed.

I am almost sure that my transparent proxy is not the problem. I turned that off without resolving the issue. I also connected to the router directly via WLAN without resolving the issue. I also tried to download over another port via HTTP. Furthermore, I tried to download the file using IPv6 with a gateway6 tunnel from my computer, which resulted in exactly the same problem.

Now the strange thing is that everything works fine using FTP and HTTPS (also with wget on the same computer). Even more strange: when I resume the download that hanged over HTTP using FTP or HTTPS, download a few bytes that way, stop wget and then resume again using HTTP, it loads data again! But after a few MB, it may stop again. Unfortunately, files downloaded that way are always broken (the MD5 sum is not correct), so at some point, there must have been bogus data. I tried searching for HTML error messages in the downloaded file, but grep -i html does not find anything. (I cannot think of a way to search for GZIP-compressed HTML error messages in the file, so I did not try that.)

I tried using strace on wget when it failed to resume a download, you can find the entire output on pastebin. The important lines are repeated every second:

clock_gettime(CLOCK_MONOTONIC, {326102, 62176435}) = 0
)                    = 1
write(2, "78% [++++++++++++++++++++++++++++"..., 19578% [+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++                                  ] 110,683,685 --.-K/s              ) = 195
select(4, [3], NULL, NULL, {0, 949999}) = 0 (Timeout) 

I have absolutely no idea what could be the reason of this problem. It seems like whatever causes the issue speaks HTTP. It seems to speak HTTP that intelligently that it even regognises it in an IPv6-over-IPv4 tunnel. But what could that be and why does it only happen sometimes? The other possibility would be that there is a problem on my computer that is the same on other Gentoo Linux computers as well.

Has anyone ever had such a problem? What could be the reason and where do I have to continue investigating to find out more about the issue?

Update:

I have just run into the problem again and tried to resume the download over the router’s WLAN, and this time it worked. Maybe I did something wrong during my last tests with the WLAN. Now maybe my transparent proxy server is in fact the problem. It is a very basic Squid proxy server that does not cache anything. Maybe the fact is interesting that a second Squid proxy runs on the same computer on another port.

Update:

A download hung again and this time I turned off all firewall settings and stopped all proxy servers. I failed to resume the download from my network server, which is directly connected to the router. So my proxy server definitely is not the cause the problem. I will try to upgrade the firmware of my router now, although I do not have admin access to it. I will see what I can do.

Update:

Upgrading to the newest firmware of my router didn’t help. I can see no other possibility than this being my ISP’s fault. My “solution” is now to tunnel all traffic through an SSH server somewhere else.