Isabell Cowan's questions

I've added a device to my existing Btrfs file system recently, but it doesn't report the new device as having even allocated any space for the data. Take a look:

Before I started:

# btrfs fi show /
Label: 'LINUX'  uuid: f0b4d012-91d8-43bd-90b5-e398a012ccff
    Total devices 1 FS bytes used 126.72GiB
    devid    1 size 272.40GiB used 128.03GiB path /dev/sda3
# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 273.4G  0 disk 
├─sda1   8:1    0     1M  0 part 
├─sda2   8:2    0     1G  0 part /boot
└─sda3   8:3    0 272.4G  0 part /pool
sdb      8:16   0  68.3G  0 disk 

So now I want to add /dev/sdb:

# btrfs device add /dev/sdb /
# btrfs balance start --full-balance /

< an hour or so later ... >

# btrfs fi usage /
Overall:
    Device size:         340.73GiB
    Device allocated:    128.03GiB
    Device unallocated:  212.70GiB
    Device missing:          0.00B
    Used:                126.68GiB
    Free (estimated):    213.10GiB  (min: 213.10GiB)
    Data ratio:               1.00
    Metadata ratio:           1.00
    Global reserve:      338.14MiB  (used: 0.00B)

Data,single: Size:126.00GiB, Used:125.60GiB
   /dev/sda3     126.00GiB

Metadata,single: Size:2.00GiB, Used:1.08GiB
   /dev/sda3       2.00GiB

System,single: Size:32.00MiB, Used:16.00KiB
   /dev/sda3      32.00MiB

Unallocated:
   /dev/sda3     144.37GiB
   /dev/sdb       68.33GiB

Huh, I thought that was supposed to do it, lets try this:

# btrfs balance start -dconvert=single -mconvert=raid1 /

< an hour or so later ... >

# btrfs fi show /
Label: 'LINUX'  uuid: f0b4d012-91d8-43bd-90b5-e398a012ccff
    Total devices 2 FS bytes used 126.72GiB
    devid    1 size 272.40GiB used 128.03GiB path /dev/sda3
    devid    2 size 68.33GiB used 2.03GiB path /dev/sdb
# btrfs fi usage /
Overall:
    Device size:         340.73GiB
    Device allocated:    130.06GiB
    Device unallocated:  210.67GiB
    Device missing:          0.00B
    Used:                127.80GiB
    Free (estimated):    211.03GiB  (min: 105.69GiB)
    Data ratio:               1.00
    Metadata ratio:           2.00
    Global reserve:      338.47MiB  (used: 0.00B)

Data,single: Size:126.00GiB, Used:125.64GiB
   /dev/sda3    126.00GiB

Metadata,RAID1: Size:2.00GiB, Used:1.08GiB
   /dev/sda3      2.00GiB
   /dev/sdb       2.00GiB

System,RAID1: Size:32.00MiB, Used:16.00KiB
   /dev/sda3     32.00MiB
   /dev/sdb      32.00MiB

Unallocated:
   /dev/sda3    144.37GiB
   /dev/sdb      66.30GiB

What the heck?

Kernel said anything during balance? Just a whole lot of this:

Feb 13 21:55:06 localhost kernel: BTRFS info (device sda3): found 7541 extents
Feb 13 21:55:06 localhost kernel: BTRFS info (device sda3): relocating block group 621927202816 f
Feb 13 21:55:18 localhost kernel: BTRFS info (device sda3): found 7020 extents
Feb 13 21:55:28 localhost kernel: BTRFS info (device sda3): found 7020 extents
Feb 13 21:55:28 localhost kernel: BTRFS info (device sda3): relocating block group 620853460992 f
Feb 13 21:55:39 localhost kernel: BTRFS info (device sda3): found 2540 extents
Feb 13 21:55:48 localhost kernel: BTRFS info (device sda3): found 2540 extents
Feb 13 21:55:48 localhost kernel: BTRFS info (device sda3): relocating block group 619779719168 f
Feb 13 21:55:58 localhost kernel: BTRFS info (device sda3): found 3971 extents

I'm using OpenDKIM to sign mail on my Postfix server. It works as intended when sending with SMTPS from it's origin domain (let's say example.com). However, mail set out from SMTP clients on the LAN are not being signed. Said clients do not have from the same origin domain (for example client1.lan), but are translated with smtp_generic_maps to the same origin (specifically [email protected]). Ideally, I want to sign that outgoing mail.

In /etc/postfix/main.cf I have:

smtpd_milters = unix:/var/run/opendkim/opendkim.sock
non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock

In /etc/opendkim.conf I have:

ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts

In /etc/opendkim/TrustedHosts I have:

::1
127.0.0.1
localhost
ip6-localhost
ip6-loopback
mail
10.0.0.0/24
*.lan

I've been trying all do to set up MCollective on my puppet cluster. No matter what I do, I can't seem to get the MCollective server to connect. The MCollective server(s) are running mostly on Ubnutu Xenial. The ActiveMQ broker (5.14.3) is running on Debian Stretch. I'm running puppet 4.x on all nodes. I've used every transport connector I can thing if, and they all fail to connect. Let me dump some log files at you.

In mcollective.log I'm getting Connection reset by peer:

I, [2017-01-27T15:43:59.869501 #18729]  INFO -- : activemq.rb:139:in `on_ssl_connecting' Establishing SSL session with stomp+ssl://[email protected]:61614
E, [2017-01-27T15:44:00.070995 #18729] ERROR -- : activemq.rb:149:in `on_ssl_connectfail' SSL session creation with stomp+ssl://[email protected]:61614 failed: Connection reset by peer - SSL_connect
I, [2017-01-27T15:44:00.071371 #18729]  INFO -- : activemq.rb:129:in `on_connectfail' TCP Connection to stomp+ssl://[email protected]:61614 failed on attempt 24

Oddly enough, in the ActiveMQ log, I also seem to be getting Connection reset by peer:

ERROR | Could not accept connection from null : {}
java.io.IOException: java.io.IOException: Connection reset by peer
    at org.apache.activemq.transport.nio.NIOSSLTransport.initializeStreams(NIOSSLTransport.java:188)[activemq-client.jar:]
    at org.apache.activemq.transport.stomp.StompNIOSSLTransport.initializeStreams(StompNIOSSLTransport.java:57)[activemq-stomp.jar:]
    at org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:543)[activemq-client.jar:]
    at org.apache.activemq.transport.nio.NIOTransport.doStart(NIOTransport.java:174)[activemq-client.jar:]
    at org.apache.activemq.transport.nio.NIOSSLTransport.doStart(NIOSSLTransport.java:462)[activemq-client.jar:]
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[activemq-client.jar:]
    at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)[activemq-client.jar:]
    at org.apache.activemq.transport.stomp.StompTransportFilter.start(StompTransportFilter.java:65)[activemq-stomp.jar:]
    at org.apache.activemq.transport.AbstractInactivityMonitor.start(AbstractInactivityMonitor.java:169)[activemq-client.jar:]
    at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)[activemq-client.jar:]
    at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072)[activemq-broker.jar:]
    at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218)[activemq-broker.jar:]
    at java.lang.Thread.run(Thread.java:745)[:1.8.0_111]

So they're both resetting the connection. Huh. Before you ask, no: there are no iptables rules, and yes: there is a route between the two nodes. Let's take a peak at lsof -i just to be sure and then I'll throw some config files at you.

java    20833 activemq   84u  IPv6  53552      0t0  TCP *:61614 (LISTEN)

activemq.xml:

<!DOCTYPE activemq [
  <!ENTITY keyStores SYSTEM "keyStores.xml">
]>
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>

    <broker xmlns="http://activemq.apache.org/schema/core"
            useJmx="false"
            brokerName="broker"
            dataDirectory="${activemq.base}/data">

      <persistenceAdapter>
        <kahaDB directory="${activemq.base}/data/kahadb"/>
      </persistenceAdapter>

      <sslContext>
        &keyStores;
      </sslContext>

      <transportConnectors>
        <transportConnector
          name="stomp+nio"
          uri="stomp+nio+ssl://0.0.0.0:61614?needClientAuth=true&amp;transport.enabledProtocols=TLSv1,TLSv1.1,TLSv1.2"/>
      </transportConnectors>

      <plugins>
        <simpleAuthenticationPlugin>
          <users>
            <authenticationUser username="mcollective" password="password" groups="mcollective,everyone"/>
            <authenticationUser username="admin" password="password" groups="mcollective,admins,everyone"/>
          </users>
        </simpleAuthenticationPlugin>
        <authorizationPlugin>
          <map>
            <authorizationMap>
              <authorizationEntries>
                <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />
                <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />
                <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
                <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />
                <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
              </authorizationEntries>
            </authorizationMap>
          </map>
        </authorizationPlugin>
      </plugins>
    </broker>

</beans>

keyStores.xml:

<sslContext
    keyStore="/etc/activemq/keystore.jks"
    keyStorePassword="password"
    trustStore="/etc/activemq/truststore.jks"
    trustStorePassword="password" />

mcollective/server.cfg:

# /etc/mcollective/server.cfg

# ActiveMQ connector settings:
connector = activemq
direct_addressing = 1
plugin.activemq.pool.size = 1
plugin.activemq.pool.1.host = broker.example.com
plugin.activemq.pool.1.port = 61614
plugin.activemq.pool.1.user = mcollective
plugin.activemq.pool.1.password = password
plugin.activemq.pool.1.ssl = 1
plugin.activemq.pool.1.ssl.ca = /etc/puppetlabs/puppet/ssl/certs/ca.pem
plugin.activemq.pool.1.ssl.cert = /etc/puppetlabs/puppet/ssl/certs/mail.example.com.pem
plugin.activemq.pool.1.ssl.key = /etc/puppetlabs/puppet/ssl/private_keys/mail.example.com.pem
plugin.activemq.pool.1.ssl.fallback = 0

# SSL security plugin settings:
securityprovider = ssl
plugin.ssl_client_cert_dir = /etc/puppetlabs/mcollective/clients
plugin.ssl_server_private = /etc/puppetlabs/mcollective/server_private.pem
plugin.ssl_server_public = /etc/puppetlabs/mcollective/server_public.pem

# Facts, identity, and classes:
identity = mail.example.com
factsource = yaml
plugin.yaml = /etc/puppetlabs/mcollective/facts.yaml
classesfile = /var/lib/puppet/state/classes.txt

# No additional subcollectives:
collectives = mcollective
main_collective = mcollective

# Registration:
# We don't configure a listener, and only send these messages to keep the
# Stomp connection alive. This will use the default "agentlist" registration
# plugin.
registerinterval = 600

# Auditing (optional):
# If you turn this on, you must arrange to rotate the log file it creates.
rpcaudit = 1
rpcauditprovider = logfile
plugin.rpcaudit.logfile = /var/log/mcollective-audit.log

# Authorization:
# If you turn this on now, you won't be able to issue most MCollective
# commands, although `mco ping` will work. You should deploy the
# ActionPolicy plugin before uncommenting this; see "Deploy Plugins" below.

# rpcauthorization = 1
# rpcauthprovider = action_policy
# plugin.actionpolicy.allow_unconfigured = 1

# Logging:
logger_type = file
loglevel = debug
logfile = /var/log/mcollective.log
keeplogs = 5
max_log_size = 2097152
logfacility = user

# Platform defaults:
# These settings differ based on platform; the default config file created by
# the package should include correct values. If you are managing settings as
# resources, you can ignore them, but with a template you'll have to account
# for the differences.
libdir = /usr/share/mcollective/plugins
daemonize = 1

The keys/certs in keystore.jks and truststore.jks are correct and so is the password. The shared keys and certs are also available. So lets try connecting with openssl:

root@mail:/etc/puppetlabs/puppet/ssl# openssl s_client -connect broker.example.com:61614 -CAfile certs/ca.pem -cert certs/mail.example.com.pem -key private_keys/mail.example.com.pem
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1485554633
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

I'm not exactly sure how to interpret this, maybe someone else knows. Any new good guesses would be helpful, I'm stuck.

Is it safe to use fallocate --dig-holes on a file while it is being modified/written? For example on a QCOW2 image opened by a KVM guest?