nagylzs's questions

I have three servers, fully connected via wireguard. They run Ubuntu Server 22.04 and postgresql repmr cluster with streaming replication.

All computers have a public address, but PostgreSQL instances, and database clients are using the internal addresses (on wireguard VPN).

On one of the computers, I see this in the logs:

2024-07-26 07:23:14.463 UTC [147915] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 07:25:56.242 UTC [148509] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 07:28:17.567 UTC [148818] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 07:33:13.234 UTC [149090] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 07:48:42.721 UTC [149723] FATAL:  terminating walreceiver due to timeout
2024-07-26 07:52:17.298 UTC [151521] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 08:01:25.141 UTC [151889] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 08:02:16.337 UTC [152868] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 08:06:13.169 UTC [152951] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly
2024-07-26 08:22:04.180 UTC [153377] FATAL:  could not receive data from WAL stream: server closed the connection unexpectedly

Also, when I try to connect to the primary db from a go or a python program, then sometimes I see "connection timeout", or "connection reset by peer", "connection was closed in the middle of operation" and similar messages. It is important to note, that these only happen on one computer, and not on the others.

On the server side (primary postgresql) I see this in the logs:

2024-07-26 12:31:36.667 UTC [3778655] telegraf@telegraf LOG:  could not receive data from client: Connection reset by peer
2024-07-26 12:31:36.897 UTC [3777638] telegraf@telegraf LOG:  could not receive data from client: Connection reset by peer
2024-07-26 12:31:39.462 UTC [3775606] telegraf@telegraf LOG:  could not receive data from client: Connection reset by peer
2024-07-26 12:31:39.480 UTC [3780628] telegraf@telegraf LOG:  could not receive data from client: Connection reset by peer

These errors happen just a few times per hour. It is enough to make my applications unreliable, but they are intermittent. I ran this ping test between the public addresses:

ping -c 3600 primary.public.com
# waited an hour...
--- primary.public.com ping statistics ---
3600 packets transmitted, 3600 received, 0% packet loss, time 3603052ms
rtt min/avg/max/mdev = 72.849/73.214/101.325/0.881 ms

I also ran a ping test on the private IP address:

ping -c 1008 primary.private.com
# waited...
--- primary.private.com ping statistics ---
1008 packets transmitted, 783 received, 22.3214% packet loss, time 1013304ms
rtt min/avg/max/mdev = 80.742/91.383/256.720/16.133 ms

In other words, 22% of the ping packets are lost over wireguard.

The MTU value for all wireguard devices are the default 1420.

3: dev0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.241.64.3/32 scope global dev0
       valid_lft forever preferred_lft forever

Also tried to test MTU using this script:

size=1272
while ping -s $size -c1 -M do primary.internaladdress.com >&/dev/null; do 
  ((size+=4))
done
echo "Max MTU size: $((size-4+28))

And it also printed 1420.

And please note that the problem only exists between two computer of the three. E.g. it is bad between A and B, but it is good between B-C.

It must be noted, that the problematic computer is far away (on a different continent). But that should not cause this.

As far as I understand, wireguard encapsulates IP packets into encrypted UDP packets, and the TCP protocol takes care of resending the packets that are lost.

It is very strange that IP packets between public addresses have 0% drop date, but wireguard/UDP packets have more than 20%. Is it possible, that UDP packets are dropped by some router or switch? Maybe QoS is happening?

These servers are rented, and they are very far away from each other. Obviously, I can't do anything to eliminate packet drops. I understand that UDP will always be unreliable. But I wonder if I can fix TCP connections somehow. Even if they slow down sometimes (even if they cannot communicate for one or two seconds), they should not reset the connection. What are my options?

The official documentation ( https://docs.saltproject.io/en/latest/ref/cache/all/salt.cache.etcd_cache.html#module-salt.cache.etcd_cache ) shows this example config:

etcd.host: 127.0.0.1
etcd.port: 2379
etcd.protocol: http
etcd.allow_reconnect: True
etcd.allow_redirect: False
etcd.srv_domain: None
etcd.read_timeout: 60
etcd.username: None
etcd.password: None
etcd.cert: None
etcd.ca_cert: None

In my particluar case, I have an etcd cluster with PKI (TLS cert based) authentication. The salt module documentation does not say any specific about that use case. It only says this:

Related docs could be found in the python-etcd documentation.

The "related documentation" is in fact very vague here https://python-etcd.readthedocs.io/en/latest/#create-a-client-object - it is not a real documentation, only a list of trivial examples:

import etcd

client = etcd.Client() # this will create a client against etcd server running on localhost on port 4001
client = etcd.Client(port=4002)
client = etcd.Client(host='127.0.0.1', port=4003)
client = etcd.Client(host='127.0.0.1', port=4003, allow_redirect=False) # wont let you run sensitive commands on non-leader machines, default is true
client = etcd.Client(
             host='127.0.0.1',
             port=4003,
             allow_reconnect=True,
             protocol='https',)

Of course none of this will work with a proper etcd cluster. However, the source code reveals that it is possible to give a list of hosts:

https://github.com/jplana/python-etcd/blob/master/src/etcd/client.py#L74

        Initialize the client.
        Args:
            host (mixed):
                           If a string, IP to connect to.
                           If a tuple ((host, port), (host, port), ...)

But I don't have any idea how to put a tuple of tuples into the salt master config file. This is what I have tried:

cache: etcd
etcd.host: [('salt1.lacinet', 2739), ('salt2.lacinet', 2739)]
etcd.protocol: https
etcd.allow_reconnect: True
etcd.allow_redirect: False
etcd.srv_domain: None
etcd.read_timeout: 60
etcd.username: root
etcd.password: None
etcd.cert: ('/etc/salt/etcd/root.key.pem', '/etc/salt/etcd/root.cert.pem')
etcd.ca_cert: '/etc/salt/etcd/ca.cert.pem'

But it results in an error:

[ERROR   ] Could not discover the etcd hosts from None: None of DNS query names exist: _etcd._tcp.None., _etcd._tcp.None.lacinet.
[ERROR   ] Failed to get list of machines from https://['salt1.lacinet', 'salt2.lacinet']:2379/v2: LocationParseError("Failed to parse: https://['salt1.lacinet', 'salt2.lacinet']:2379/v2/machines")
[ERROR   ] Error in function _pillar:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/etcd/client.py", line 296, in machines
    response = self.http.request(
  File "/usr/lib/python3/dist-packages/urllib3/request.py", line 75, in request
    return self.request_encode_url(
  File "/usr/lib/python3/dist-packages/urllib3/request.py", line 97, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 318, in urlopen
    u = parse_url(url)
  File "/usr/lib/python3/dist-packages/urllib3/util/url.py", line 392, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://["('salt1.lacinet'", '2739)', "('salt1.lacinet'", '2739)']:2379/v2/machines

The error itself is very suspicious:

Failed to parse: https://["('salt1.lacinet'", '2739)', "('salt1.lacinet'", '2739)']:2379/v2/machines

I suspect that the host list is not recognized as a list or tuple, but as a string.

Given this error, it might even be possible that salt-master minion cache does not support multiple etcd hosts? But that would be very "interesting", since the main reason of installing etcd is to provide a HA key-value storage for multiple salt masters. What is the point if it can only use a single etcd server?

I'll likely have problems with the TLS cert based authentication as well, because it also uses a tuple (or it should, as given in https://github.com/jplana/python-etcd/blob/master/src/etcd/client.py#L90 ):

            cert (mixed):   If a string, the whole ssl client certificate;
                            if a tuple, the cert and key file names.
            ca_cert (str): The ca certificate. If pressent it will enable
                           validation.

It is not clear how to specify a tuple of filenames in the config. Also not clear if the ca_cert configuration is the path of a cert file, or it is the PEM cert string itself. None of this is documented anywhere (or just I could not find it).

Can anybody please tell me howto use a proper TLS-auth etcd cluster with salt-master as the minion cache? Is it possible at all?

I have multiple computers (Win10 pro) placed at different remote locations (my partners) that I need to manage. I need to access them from a central location. So I have a central VPN server, and I want multiple Win10 pro instances to connect to it. I'm using L2TP, but I would like to migrate to wireguard. With L2TP, it is possible to start rasdial.exe in the background, from the task scheduler. The main goal is to start the tunnel as soon as the computer starts up, and automatically reconnect if the tunnel goes down. I need this BEFORE any user logs into the computer (e.g. in the background). I wonder if the same can be done with wireguard?

I noticed that there is a program called C:\Program Files\Wireguard\wg.exe and its command line options are very similar to the wg (linux) program:

C:\Program Files\WireGuard>wg.exe --help
Usage: wg.exe <cmd> [<args>]

Available subcommands:
  show: Shows the current configuration and device information
  showconf: Shows the current configuration of a given WireGuard interface, for use with `setconf'
  set: Change the current configuration, add peers, remove peers, or change peers
  setconf: Applies a configuration file to a WireGuard interface
  addconf: Appends a configuration file to a WireGuard interface
  syncconf: Synchronizes a configuration file to a WireGuard interface
  genkey: Generates a new private key and writes it to stdout
  genpsk: Generates a new preshared key and writes it to stdout
  pubkey: Reads a private key from stdin and writes a public key to stdout
You may pass `--help' to any of these subcommands to view usage.

But I think it is for configuration only. I think it cannot be used to activate a tunnel. The original documentation ( https://www.wireguard.com/quickstart/#command-line-interface ) explains that "the interface can ... be activated with ifconfig(8) or ip-link(8)" but of course that works for unix only. In that documentation, there is another note "Non-Linux users will instead write wireguard-go wg0" but I'm not sure what they mean here. There is no program name "wireguard-go" anywhere. The network adapter for the tunnel only shows up in the network adatpter list when the tunnel is already active. E.g. there is no network adapter that I could "enable" or "activate" when the tunnel is down. Finally, there is wireguard.exe. When started without parameters, it is a GUI application. It cannot be run without a logged in user and a desktop, and it does not activate any tunnel automatically. It has some interesting command line options. "wireguard.exe /managerservice" is used to start it as a windows service. "wireguard.exe /tunnelservice CONFIG_PATH" looks promising, but I was not able to start that. I get this error, that "the process could not connect to the service manager" or something similar. (I also get this error when I start it as administrator.)

I'm out of ideas. How should I do this?

I need to create a postgresql 11 docker image that has support for sha3-512 hash function. By default, the official postgres docker images do not support this. I already have some systems that use the official docker images. I need a new image that works exactly like the official, except that it has support for these hash functions. My idea was to create a copy of the official Dockerfile and modify it.

According to the configuration notes section here: https://www.postgresql.org/docs/11/pgcrypto.html#id-1.11.7.34.10 the SHA224/256/384/512 hash functions are supported only if postgresql was configured and built with --with-openssl

I have copied the Dockerfile and the entrypoint script from here:

https://github.com/docker-library/postgres/tree/master/11/stretch

and then, I'm stuck. I have no idea how or where to add "--with-openssl". I suspect that the configuration and compilation happens at this section:

# build .deb files from upstream's source packages (which are verified by apt-get)
            apt-get update; \
            apt-get build-dep -y \
                postgresql-common pgdg-keyring \
                "postgresql-$PG_MAJOR=$PG_VERSION" \
            ; \
            DEB_BUILD_OPTIONS="nocheck parallel=$(nproc)" \
                apt-get source --compile \
                    postgresql-common pgdg-keyring \
                    "postgresql-$PG_MAJOR=$PG_VERSION" \
            ; \

but at this point, I don't know what to do, or what to change.

The error itself is like this

select digest('test','sha3-512')
SQL Error [22023]: ERROR: Cannot use "sha3-512": No such hash algorithm

Seemingly, there are no additional hash algorithms added:

postgres=# \df digest
                       List of functions
 Schema | Name | Result data type | Argument data types | Type 
--------+------+------------------+---------------------+------
(0 rows)

postgres=# select * from pg_available_extensions where name='pgcrypto';
name    |default_version|installed_version|comment                |
--------+---------------+-----------------+-----------------------+
pgcrypto|1.3            |1.3              |cryptographic functions|

I have a website with nginx that returns index.html (a single page react app) for almost all requests. Any request for almost any path (e.g. /some/arcticle) will also return index.html, and the react app (react router) will translate the path into API calls. This is how it works, and it cannot be changed (the website is huge, it would be too much work, and I'm in no position to change this anyway).

There should be two exceptions for these requests:

1. All requests with starting with /api/* path are passed to an upstream (application server). So a different backend will handle all actual API requests.
2. The other exception should be facebook external hits. There is a different endpoint under /api/open_graph on the application server for that. E.g. /api/open_graph should be preprended to the original path. That endpoint returns actual content (and not a common single page react app that has no real content). The format is also different - normal API calls usually return JSON data, but the open_graph endpoint returns simple HTML.

Example nginx config:

upstream www {
    # ...
}

server {
    # ...

    # Use /api/open_graph on the upstream for facebook external hits
    if ($http_user_agent ~* "^facebookexternalhit.*$") {
        rewrite ^/(.*)$ /api/open_graph/$1 permanent;
    }

    # API requests will go to upstream
    location ~ ^/api/ {
        proxy_pass          http://www;
        proxy_read_timeout  90;
        proxy_redirect      http://www https://example.com;
    }

    # All other requests use a react app,
    # react router handles all further requests
    location / {
        index index.html;
        error_page 404 =200 /index.html;
    }

}

The above config works like this:

# curl -A "facebookexternalhit" -s -D - -o /dev/null https://example.com/article1
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Wed, 20 Oct 2021 08:14:13 GMT
Content-Type: text/html
Content-Length: 162
Location: https://example.com/api/open_graph/article1
Connection: keep-alive
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

This is almost good, except that facebook graph api does not handle redirects. (Who knows why?). So instead of redirecting to /api/open_graph/* the server should directly connect the upstream and forward the request.

But I don't know how? The naive solution would be:

    # Use /api/open_graph on the upstream for facebook external hits
    if ($http_user_agent ~* "^facebookexternalhit.*$") {
        rewrite ^/(.*)$ /api/open_graph/$1 permanent;
        proxy_pass          http://www;
        proxy_read_timeout  90;
        proxy_redirect      http://www https://example.com;
    }

But it does not work, because proxy_pass can only be used inside a location. It cannot be used inside an if. If I try that config then I get:

nginx: [emerg] "proxy_pass" directive is not allowed here in /etc/nginx/sites-enabled/example-www:62
nginx: configuration file /etc/nginx/nginx.conf test failed

I might be possible to change the program code in the upstream (e.g. add a filter for the user agent there), but it would be quite difficult.

Is there a workaround for this within nginx?

This question is related to: nginx - proxy_pass on user_agent but that wasn't working for me either (threw an error)

Using Ubuntu 20.04 LTS, I have this in /etc/fail2ban/jail.local:

[DEFAULT]
bantime   = 3600
banaction = iptables
blocktype = drop


[sshd]
enabled   = true
protocol  = tcp
port      = ssh
filter    = sshd
logpath   = /var/log/auth.log
maxretry  = 3

But this is what I see when I list iptables rules:

╰─# iptables -L f2b-sshd -n -v
Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination
   13  1356 REJECT     all  --  *      *       222.187.232.205      0.0.0.0/0            reject-with icmp-port-unreachable
   18  1516 REJECT     all  --  *      *       221.181.185.153      0.0.0.0/0            reject-with icmp-port-unreachable
   17  1064 REJECT     all  --  *      *       222.186.180.130      0.0.0.0/0                  777 55854 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

The problem is that it uses REJECT (with ICMP) instead of DROP.

The action.d/iptables.conf contains this:

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>

It is the default iptables action file, shipped with the official fail2ban apt package for this OS version.

Also tried to add "blocktype=drop" under [sshd] but it has no effect.

I'm not sure how to debug this, because the fail2ban service does not log the actual iptables commands.

What am I missing?

I need to create an ext4 fs over a zpool. I wanted to use all (or almost all) available space, as shown below.

# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
nat2012  1,81T   124K  1,81T        -         -     0%     0%  1.00x    ONLINE  -
# zfs list
NAME      USED  AVAIL     REFER  MOUNTPOINT
nat2012   105K  1,76T       24K  none
# zfs create -V 1.75T nat2012/backup
bad volume size '1.75T': invalid numeric suffix '.75T'
# zfs get avail nat2012
NAME     PROPERTY   VALUE  SOURCE
nat2012  available  1,76T  -
# zfs get -H avail nat2012
nat2012 available   1,76T   -

My problem is that "1.75T" is not accepted as a volume size. (But it is displayed as a size.) Its value is rounded to 0.01T which is 10G. Next thing I tried is to get the available size in bytes, Mbytes or GBytes or blocks, or practically anything that tells me the exact size, and can be used with "zfs create -V". I was suprised when I saw that there is no "zfs get" option for that. According to the docs, the "-H" option should "display output in a form more easily parsed by scripts". It omitted the headers and used tabs to separate the fields, but the formatting is bad. Size is still displayed in TBytes and rounded with +/- 10G precision. Smaller issue but it is formatted using the current locale (in my case, the decimal point "." is replaced with ","). It would be an integer number in KBytes.

Please note that the "df" command cannot be used in this particular case, because nothing is mounted. I could do this to display free blocks in Kbytes:

# mkdir -p /nat2012
# zfs set mountpoint=/nat2012 nat2012
# df -k | grep nat2012
nat2012        1885339520        128 1885339392   1% /nat2012

But this would give a false result. As you can see, it is now mounted as zfs filesystem, and the dataset already uses 1% of the available space. This does not tell me the exact size that is available for creating a new ext4 volume. (The -V option is mandatory - if you don't specify it, then ext4.mkfs will not be able to format it.)

Of course, I can always do a trial, something like this:

# zfs create -V 1760G nat2012/backup
cannot create 'nat2012/backup': out of space
# zfs create -V 1755G nat2012/backup
cannot create 'nat2012/backup': out of space
# zfs create -V 1750G nat2012/backup
cannot create 'nat2012/backup': out of space
# zfs create -V 1740G nat2012/backup
# zfs destroy nat2012/backup
# zfs create -V 1745G nat2012/backup
cannot create 'nat2012/backup': out of space
# zfs create -V 1744G nat2012/backup
cannot create 'nat2012/backup': out of space
# zfs create -V 1743G nat2012/backup

At this point, I can be sure that there is at most 1G space left unused. This is very clumsy. I can live with this, but I have to do the same things for a number of drives in the future.

Is there a better way to "use all available space" for creating a fixed size volume?

I have a disk with these layers: sata disk, luks, zpool, ext4

The ext4 fs was created with these commands:

cryptsetup -v luksFormat /dev/sdb
cryptsetup luksOpen /dev/sda store02
zpool create zstore02 /dev/mapper/store02
zfs create -V 1600G zstore02/dsk02
mkfs.ext4 -L dsk02 /dev/zstore02/dsk02

System is Ubuntu 20.04.1 LTS. This pool was later imported on another 20.04.1 system.

I want to create a read/write snapshot and mount it. But it tells me "out of space":

root@computer:~# zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zstore02  1,81T  1,50T   320G        -         -     7%    82%  1.00x    ONLINE  -
root@computer:~# zfs list
NAME             USED  AVAIL     REFER  MOUNTPOINT
zstore02        1,61T   148G       24K  /zstore02
zstore02/dsk02  1,61T   262G     1,50T  -
root@computer:~# zfs snapshot zstore02/dsk02@test
cannot create snapshot 'zstore02/dsk02@test': out of space
root@computer:~# zfs get reservation zstore02/dsk02
NAME            PROPERTY     VALUE   SOURCE
zstore02/dsk02  reservation  none    local
root@computer:~# zfs set reservation=10G zstore02
root@computer:~# zfs set reservation=10G zstore02/dsk02
root@computer:~# zfs snapshot zstore02/dsk02@test
cannot create snapshot 'zstore02/dsk02@test': out of space
root@computer:~#

It does not work with reserved=none nor reserved=10G.

Apparently, "zpool list" reports 300G free space, and "zfs list" reports 262G free space. But somehow I cannot create a snapshot. Why?

UPDATE

If I try to create a new volume, then it works (sorry for the Hungarian locale):

root@computer:~# zfs create -V 1G zstore02/test
root@computer:~# mkfs.ext4 -L test /dev/zstore02/test
mke2fs 1.45.5 (07-Jan-2020)
Eszközblokkok eldobása: kész
Fájlrendszer létrehozása 262144 4 blokkal és 65536 inode-dal
Fájlrendszer UUID: 14e07b33-5d25-465a-aeb8-7fbfe2499dfd
Tartalék szuperblokkok tárolva a blokkokon:
        32768, 98304, 163840, 229376

Csoporttáblák foglalása: kész
Inode táblák írásakor: kész
Napló létrehozása (8192 blokk): kész
Szuperblokkok és fájlrendszer-könyvelési információk írása: kész

root@computer:~# zfs snapshot zstore02/test@snap1

root@computer:~# zfs list -t snapshot
NAME                  USED  AVAIL     REFER  MOUNTPOINT
zstore02/test@snap1     0B      -     32,7M  -
root@computer:~#

It works for any volume that I create, except zstore02/dsk02.

But why?

The basic situation:

• There are multiple schools connected to a country wide network. (Some thousand schools.)
• We need to deliver e-learning content to them using a network that was created specifically for these schools. There are POPs in different counties, and all of these schools are accessing the internet through these points.
• Big part of the overall network traffic will be educational videos that are uploaded to Vimeo. Smaller part will be images, and finally static HTML/CSS files and REST API calls.
• These videos are embedded into web pages. (Pages representing lessons in online books.) We already have a business Vimeo account that allows us to embed videos into web pages, without displaying advertisements.
• Most schools will teach the same lesson at a given time, so they will need (about the) same videos at the same time.

The total network traffic could be reduced effectively by caching these videos at the POPs. This is especially important, because the total bandwidth is not enough to stream videos to all clients simultaneously, without caching. But I'm not sure how to do this. Using squid would be a very good solution if we could install them separately for every school. Unfortunately, it is not possible. These schools do not have the infrastructure to operate a squid server 24/7, they have very different networks, with different devices and local admins have very different skills. It would probably be better to cache the content at the POPs.

Here are some questions:

• What protocol is used for video streaming nowdays? What I see from the logs is that the player is downloading m4s files (mp4 video segments?) using HTTPS protocol.
• Is it possible to cache these videos transparently? If it uses https then it seems impossible to do (but I might be wrong).
• Can you suggest something that can reduce the required network bandwidth? It might be possible to write a special web page that plays these videos from a cache.

I need to write scripts that create services on docker swarm clusters. An example script would be simiar to this:

docker service create \
    --name postgres \
    --mode global \
    --constraint "node.labels.postgres==master" \
    --network my-network \
    --env POSTGRES_USER="postgres" \
    --env POSTGRES_PASSWORD="****" \
    postgres:10

The POSTGRES_USER and POSTGRES_PASSWORD environment variables should come from docker secrets. For example:

echo "example_password" | docker secret create postgres-password -

I understand that the secret can be accessed as a file from inside a running container. But in the a above example, it must be passed to the "service create" command as an environment variable. It is used by the entry point of the container, so it must be presented BEFORE the container is created. So how can I pass a secret to the --env switch of the "docker service create" command?

I have a zabbix server and an agent, on two different computers. The agent runs in active mode, e.g. I have this in the config file:

StartAgents=0
ServerActive=my.zabbix.server.com
Hostname=my.zabbix.agent.com

The zabbix server can be reached from the machine with the agent, e.g.:

telnet my.zabbix.server.com 10051
Trying 111.111.111.111...
Connected to my.zabbix.server.com.
Escape character is '^]'.
Connection closed by foreign host.

Moreover, the host auto registration was turned on on the server, and the agent has successfuly registered the host when I first started it. So the connection must be alive. This is what I see in the agent's log when I start it:

83074:20171128:082440.324 Starting Zabbix Agent [my.zabbix.agent.com]. Zabbix 3.4.1 (revision 71734).
83074:20171128:082440.324 **** Enabled features ****
83074:20171128:082440.324 IPv6 support:          YES
83074:20171128:082440.324 TLS support:           YES
83074:20171128:082440.324 **************************
83074:20171128:082440.324 using configuration file: /usr/local/etc/zabbix34/zabbix_agentd.conf
83074:20171128:082440.324 agent #0 started [main process]
83076:20171128:082440.325 agent #1 started [collector]
83077:20171128:082440.326 agent #2 started [active checks #1]

In other words, the agent could connect to the server, it even recognized its version. Nothing else happens in the agent log.

On the server, it still says that the host is unreachable!

What could be the problem?

UPDATE: on the front end, I see this message:

I'm not sure why it wants to connect to 10050? It is used for passive agents. My agent should be active.

UPDATE2: If I delete the host from the zabbix server, and restart the agent, then the following happens:

The host is auto-registered on the server again. The agent log:

14551:20171128:193954.483 Starting Zabbix Agent [my.zabbix.server.com]. Zabbix 3.4.1 (revision 71734).
14551:20171128:193954.484 **** Enabled features ****
14551:20171128:193954.484 IPv6 support:          YES
14551:20171128:193954.484 TLS support:           YES
14551:20171128:193954.484 **************************
14551:20171128:193954.484 using configuration file: /usr/local/etc/zabbix34/zabbix_agentd.conf
14551:20171128:193954.484 agent #0 started [main process]
14553:20171128:193954.485 agent #1 started [collector]
14554:20171128:193954.485 agent #2 started [active checks #1]
14554:20171128:193954.614 no active checks on server [my.zabbix.server.com:10051]: host [my.zabbix.agent.com] not found

where:

• my.zabbix.server.com is the server's FQDN
• my.zabbix.agent.com is the agent's FQDN, and also the HostName parameter in the agent's config.

So it seems, that the agent registers the host successfully, but for some reason, the server tries to get the information from the agent in passive mode. Despite the fact, that the agent was configured in active mode.

UPDATE 3: Although the agents are sending data, the host list still shows a problem:

Availability/ZBX has a red flag, and a message saying that "Get value from agent failed: cannot connect to [[ip_address_here]:1050]: [4] interrupted system call". I have checked every single item and every discovery for these hosts, and all of them have type="Zabbix Agent Active". So I don't understand why the server is trying to connect to them in passive mode??? This does not cause a real "problem" (e.g. something that generates an action and sends out notifications from the zabbix server), but it is very disturbing to see red flags on the screen.

Until this problem is fully solved, I won't even accept my own answer.

UPDATE 4: after changing all item types, discovery types, and the types of the item prototypes of the low level discovery rules of all templates that are connected to my hosts, and all of the templates linked form there, the ZBX red flags finally disappeared. I believe that I'm an experienced software user, but it was quite difficult to understand what is going on, and change all of the parameters to make it work.

I have this rnd.key file on my system:

key "rndc-key1" {
    algorithm hmac-md5;
    secret "xxxxxxxxxxxxxxx==";
};
key "rndc-key2" {
        algorithm hmac-md5;
        secret "yyyyyyyyyyy==";
};

Then I use them for different zones:

zone "somedomain1.com" {
    type master;
    file "/etc/bind/master/db.somedomain1.com";
    allow-update {
    key rndc-key1;
    };
};


zone "somedomain2.com" {
    type master;
    file "/etc/bind/master/db.somedomain2.com";
    allow-update {
    key rndc-key2;
    };
};

When I try to run "rndc freeze" then I get this error:

rndc: error: /etc/bind/rndc.key:5: 'key' redefined near 'key'
rndc: could not load rndc configuration

What does it mean? What is the problem here? Is it not possible to use different keys for different zones?

Given a raidz1 vdev on FreeBSD 10 with this config:

root@gw:~ # zpool status
  pool: data
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
  scan: resilvered 26.1G in 0h43m with 0 errors on Sat Dec 21 07:07:41 2013
config:

        NAME        STATE     READ WRITE CKSUM
        data        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            ada1    ONLINE       0     0     0  block size: 512B configured, 4096B native
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0  block size: 512B configured, 4096B native

errors: No known data errors

I would like to correct for the blocksize. Is it possible to change the blocksize without offlining the pool? I just wanted to know if these are the right commands to execute. I could try without asking, but the devil never sleeps...

zpool detach data ada1
zpool labelclear ada1
zpool attach data ada1

Is it the right way to do it?

UPDATE - things I have tried:

• "zpool remove" could be used to remove ada1 before clearing and adding it again. But "zpool remove" does not support raidz vdevs.

• "zpool replace --f ada1" could be used to forcedly rebuild ada1 like if it was a new devide, but zpool does not allow this (even not forcedly) for a device that is already part of an active

I believe that I could achieve this the following way:

• shut down the system
• remove ada1 and wipe data on it completely
• put back ada1
• start up the system
• then issue "zpool replace data ada1"

Could this work? But only after shutting down the computer completely. :-(

I have written a buggy program that has accidentally created about 30M files under /tmp. (The bug was introduced some weeks ago, and it was creating a couple of subdirectories per second.) I could rename /tmp to /tmp2, and now I need to delete the files. The system is FreeBSD 10, the root filesystem is zfs.

Meanwhile one of the drives in the mirror went wrong, and I have replaced it. The drive has two 120GB SSD disks.

Here is the question: replacing the hard drive and resilvering the whole array took less than an hour. Deleting files /tmp2 is another story. I have written another program to remove the files, and it can only delete 30-70 subdirectories per second. It will take 2-4 days to delete all files.

How is it possible that resilvering the whole array takes an hour, but deleting from the disk takes 4 days? Why do I have so bad performance? 70 deletions/second seems very very bad performance.

I could delete the inode for /tmp2 manually, but that will not free up the space, right?

Could this be a problem with zfs, or the hard drives or what?

#yum install python26 python26-devel python26-setuptools gcc
#easy_install psutil
[root@vps021 ~]# easy_install psutil
Searching for psutil
Reading http://pypi.python.org/simple/psutil/
Best match: psutil 3.2.2
Downloading https://pypi.python.org/packages/source/p/psutil/psutil-3.2.2.tar.gz#md5=56df1727c9e26076a80da4c3f8f67976
Processing psutil-3.2.2.tar.gz
Running psutil-3.2.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-JLmzFu/psutil-3.2.2/egg-dist-tmp-1dna_f
warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: manifest_maker: MANIFEST.in, line 18: 'recursive-include' expects <dir> <pattern1> <pattern2> ...
psutil/_psutil_linux.c:12:20: error: Python.h: No such file or directory

But python26-devel is installed. How is this possible? I'm not sure how to install python psutil on this system? (I'm also very unsatisfied by having only python version 2.6 but unfortunately, I'm tied to this system.)

This is a FreeBSD 9.1 amd64 computer. It has 5 disks installed. The ada0 and ada1 disks are used with a hw raid to provide the root filesystem:

root@gw:/home/gandalf # ls /dev | grep ada
ada0
ada1
ada2
ada3
ada4
root@gw:/home/gandalf # zpool status
  pool: zroot
 state: ONLINE
  scan: none requested
config:

    NAME          STATE     READ WRITE CKSUM
    zroot         ONLINE       0     0     0
      raid/r0s1a  ONLINE       0     0     0

errors: No known data errors

I want to create a raidz pool for the remaining disks:

root@gw:/home/gandalf # zpool create -f data raidz1 ada2 ada3 ada4
cannot create 'data': one or more devices is currently unavailable
root@gw:/home/gandalf # dmesg | grep ada2
ada2 at ata4 bus 0 scbus6 target 0 lun 0
ada2: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad16
root@gw:/home/gandalf # dmesg | grep ada3
ada3 at ata5 bus 0 scbus7 target 0 lun 0
ada3: <SAMSUNG HD103UJ 1AA01118> ATA-7 SATA 2.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 953868MB (1953523055 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad18
GEOM_RAID: Intel-fb8732fa: Disk ada3 state changed from NONE to ACTIVE.
GEOM_RAID: Intel-fb8732fa: Subdisk Volume0:0-ada3 state changed from NONE to ACTIVE.
root@gw:/home/gandalf # dmesg | grep ada4
ada4 at ata6 bus 0 scbus8 target 0 lun 0
ada4: <TOSHIBA DT01ACA100 MS2OA750> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada4: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad20
root@gw:/home/gandalf # dmesg | grep GEOM_RAID

Aha, so ada3 is already part of another raid volume? Let's see:

root@gw:/home/gandalf # dmesg | grep GEOM_RAID
GEOM_RAID: SiI-130628113902: Array SiI-130628113902 created.
GEOM_RAID: SiI-130628113902: Disk ada0 state changed from NONE to ACTIVE.
GEOM_RAID: SiI-130628113902: Subdisk SiI Raid1 Set:1-ada0 state changed from NONE to STALE.
GEOM_RAID: SiI-130628113902: Disk ada1 state changed from NONE to ACTIVE.
GEOM_RAID: SiI-130628113902: Subdisk SiI Raid1 Set:0-ada1 state changed from NONE to STALE.
GEOM_RAID: SiI-130628113902: Array started.
GEOM_RAID: SiI-130628113902: Subdisk SiI Raid1 Set:0-ada1 state changed from STALE to ACTIVE.
GEOM_RAID: SiI-130628113902: Subdisk SiI Raid1 Set:1-ada0 state changed from STALE to RESYNC.
GEOM_RAID: SiI-130628113902: Subdisk SiI Raid1 Set:1-ada0 rebuild start at 0.
GEOM_RAID: SiI-130628113902: Volume SiI Raid1 Set state changed from STARTING to SUBOPTIMAL.
GEOM_RAID: SiI-130628113902: Provider raid/r0 for volume SiI Raid1 Set created.
GEOM_RAID: Intel-fb8732fa: Array Intel-fb8732fa created.
GEOM_RAID: Intel-fb8732fa: Force array start due to timeout.
GEOM_RAID: Intel-fb8732fa: Disk ada3 state changed from NONE to ACTIVE.
GEOM_RAID: Intel-fb8732fa: Subdisk Volume0:0-ada3 state changed from NONE to ACTIVE.
GEOM_RAID: Intel-fb8732fa: Array started.
GEOM_RAID: Intel-fb8732fa: Volume Volume0 state changed from STARTING to DEGRADED.
GEOM_RAID: Intel-fb8732fa: Provider raid/r1 for volume Volume0 created.
root@gw:/home/gandalf # 

Yes, indeed. I want to get rid of raid/r1 completely. However, the controller was already set to "IDE" mode in the BIOS. So why it is creating a raid volume??? I have also tried overwritting the first 16k data of ada3 and reboot the computer, but it did not help.

How can I delete /dev/raid/r1 ?

root@gw:/home/gandalf # graid status
   Name      Status  Components
raid/r0  SUBOPTIMAL  ada0 (ACTIVE (RESYNC 4%))
                     ada1 (ACTIVE (ACTIVE))
raid/r1    DEGRADED  ada3 (ACTIVE (ACTIVE))
root@gw:/home/gandalf # graid delete raid/r1
graid: Array 'raid/r1' not found.
root@gw:/home/gandalf # graid delete /dev/raid/r1
graid: Array '/dev/raid/r1' not found.
root@gw:/home/gandalf #

Thanks

Today I have found this on a server (FreeBSD 8.2 STABLE):

NAME STATE READ WRITE CKSUM
data DEGRADED 1.38K 0 0
  raidz1-0 DEGRADED 1.38K 0 0
    ad10 ONLINE 1.38K 0 0
    ad12 ONLINE 0 0 0
    ad14 ONLINE 0 0 0
    ad16 REMOVED 0 0 0 

I quickly pulled down the wrong HDD and put in a new one. After that, I typed in this unlucky command:

zpool add data ad16

The result was that a new ad16 device appeared in the pool:

NAME        STATE     READ WRITE CKSUM
data        DEGRADED     0     0     0
  raidz1-0  DEGRADED     0     0     0
    ad10    ONLINE       0     0     0
    ad12    ONLINE       0     0     0
    ad14    ONLINE       0     0     0
    ad16    FAULTED      0     0     0  corrupted data
  ad16      ONLINE       0     0     0

The first ad16 device is FAULTED and part of data/raidz1-0 volume. The second ad16 device is ONLINE, and not part of any volume. The problem is that they have the same name so the replace command does not work:

gw# zpool replace -f data ad16 ad16
invalid vdev specification
the following errors must be manually repaired:
/dev/ad16 is part of active pool 'data'

I think that I should remove the ONLINE ad16 disk before I could replace the FAULTED ad16 disk. But this doesn't work because I'm not able to put it offline, nor remove it:

gw# zpool offline data ad16
gw# zpool status
  pool: data
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
 scan: scrub in progress since Thu Apr 18 03:23:06 2013
    26.1G scanned out of 3.13T at 50.7M/s, 17h52m to go
    0 repaired, 0.81% done
config:

    NAME        STATE     READ WRITE CKSUM
    data        DEGRADED     0     0     0
      raidz1-0  DEGRADED     0     0     0
        ad10    ONLINE       0     0     0
        ad12    ONLINE       0     0     0
        ad14    ONLINE       0     0     0
        ad16    OFFLINE      0     0     0
      ad16      ONLINE       0     0     0

errors: No known data errors
gw# zpool remove data ad16
cannot remove ad16: only inactive hot spares, cache, top-level, or log devices can be removed

I guess that the "offline ad16" command is targeting the FAULTED device. But I would like to offline the other one. I have also tried to boot the system into single user mode, with this new disk removed, but it resulted in both ad16 devices being UNAVAIL, and the whole pool unusable (which is strange, because there are enough disks to make it work...)