itsafire's questions

I'm doing internet servers for like 30 years now. The last three years I went back into operations and working with all the new infrastructure wonders. Feels really great. From time to time I am doing work on our mailserver that is used as a relay for customer machines. The mail queue is always a place to have a look at for things going wrong with mails.

When checking on some of the destination domains that are currently not reachable, I see a lot of domains that have no MX records set at all. My knowledge is: if you want to be reliably reachable by mail add at least two MX records of mail servers that will be able to receive mail for that domain. One master and one fall back machine. Having no MX record at all the mail server will fall back to retrieving the IP Adress of the destination mail server via the A record of its domain.

The domains in question are from businesses that are real no spam hosts or such.

These sites are misconfigured. Roughly 15 years ago such configuration would be a sure pointer to an IT department running on windows without a clue about the internet.

Did I miss anything important changes in mail configuration or is the mail misconfiguration just on the rise?

I have a oVirt 4.2 datacenter. I like to upgrade to 4.3. What steps are necessary and where do I get documentation. The oVirt documentation is sparse on this matter to say the least.

oVirt 4.2 comes with librbd1 und librados2 from the Ceph Hammer release which is 0.94.5.

I need to update both libraries to the luminous version which is 12.x. because my ceph server is not able to talk to clients with the old 0.94.5 version.

How to do that ?

I am missing some tools on my ovirt nodes, which has been installed via the ovirt node ng installer.

My idea is to yum install epel-release. But I don't want to mix up versions of ovirt packages with the epel-release repo...

Is it safe to add ?

My setup comprises of ceph mimic (centos 7, setup with ceph-ansible), cinder/keystone combo on pike release, and ovirt 4.2.5.1.

The external cinder provider is setup and I can create disks.

When creating a vm and starting it, the VM shows up in the ovirt dashboard as "wait for launch"

on the ovirt node, that is supposed to run the VM I checked libvirt:

# virsh --readonly list

 Id    Name                           State
----------------------------------------------------
14    testceph                       paused

checking the domain configuration also seems ok... Most important the ceph mons are given in the disk configuration.

# virsh --readonly dumpxml

<domain type='kvm' id='15'>
  <name>testceph</name>
  <uuid>036a2385-2b4f-48f9-bcf9-8f2882ecde36</uuid>
  <metadata xmlns:ns0="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0">
    <ns0:qos/>
    <ovirt-vm:vm xmlns:ovirt-vm="http://ovirt.org/vm/1.0">
    <ovirt-vm:clusterVersion>4.2</ovirt-vm:clusterVersion>
    <ovirt-vm:destroy_on_reboot type="bool">False</ovirt-vm:destroy_on_reboot>
    <ovirt-vm:launchPaused>false</ovirt-vm:launchPaused>
    <ovirt-vm:memGuaranteedSize type="int">2730</ovirt-vm:memGuaranteedSize>
    <ovirt-vm:minGuaranteedMemoryMb type="int">2730</ovirt-vm:minGuaranteedMemoryMb>
    <ovirt-vm:resumeBehavior>auto_resume</ovirt-vm:resumeBehavior>
    <ovirt-vm:startTime type="float">1535016868.02</ovirt-vm:startTime>
    <ovirt-vm:device mac_address="00:1a:4a:16:01:78">
        <ovirt-vm:specParams/>
        <ovirt-vm:vm_custom/>
    </ovirt-vm:device>
</ovirt-vm:vm>
  </metadata>
  <maxMemory slots='16' unit='KiB'>16777216</maxMemory>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static' current='1'>16</vcpu>
  <iothreads>1</iothreads>
  <resource>
    <partition>/machine</partition>
  </resource>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>oVirt</entry>
      <entry name='product'>oVirt Node</entry>
      <entry name='version'>7-5.1804.el7.centos.2</entry>
      <entry name='serial'>49434D53-0200-9031-2500-31902500FB7F</entry>
      <entry name='uuid'>036a2385-2b4f-48f9-bcf9-8f2882ecde36</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>Nehalem</model>
    <topology sockets='16' cores='1' threads='1'/>
    <numa>
      <cell id='0' cpus='0' memory='4194304' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='variable' adjustment='0' basis='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' error_policy='report'/>
      <source file='/rhev/data-center/mnt/192.168.10.6:_media_ovirt-cd-images/c394242c-81ae-4d6a-a193-65157cc84702/images/11111111-1111-1111-1111-111111111111/ubuntu-server-18.04.iso' startupPolicy='optional'/>
      <backingStore/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ua-fcca0dff-d833-4f28-b782-78ce0b016afe'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='network' device='disk' snapshot='no'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='c6020051-6cd3-4ddf-982e-3d94c080de9c'/>
      </auth>
      <source protocol='rbd' name='volumes/volume-9b95b28c-9eec-4110-9973-88c161d3503f'>
        <host name='192.168.20.21' port='6789'/>
        <host name='192.168.20.22' port='6789'/>
        <host name='192.168.20.23' port='6789'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <serial>9b95b28c-9eec-4110-9973-88c161d3503f</serial>
      <boot order='1'/>
      <alias name='ua-9b95b28c-9eec-4110-9973-88c161d3503f'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='scsi' index='0'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='00:1a:4a:16:01:78'/>
      <source bridge='ovirtmgmt'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>
      <link state='up'/>
      <mtu size='1500'/>
      <alias name='ua-446090cf-6758-4b3d-bd87-eb8b61442a46'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channels/036a2385-2b4f-48f9-bcf9-8f2882ecde36.ovirt-guest-agent.0'/>
      <target type='virtio' name='ovirt-guest-agent.0'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channels/036a2385-2b4f-48f9-bcf9-8f2882ecde36.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <alias name='channel2'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='spice' port='5900' tlsPort='5901' autoport='yes' listen='192.168.10.11' passwdValidTo='1970-01-01T00:00:01'>
      <listen type='network' address='192.168.10.11' network='vdsm-ovirtmgmt'/>
      <channel name='main' mode='secure'/>
      <channel name='display' mode='secure'/>
      <channel name='inputs' mode='secure'/>
      <channel name='cursor' mode='secure'/>
      <channel name='playback' mode='secure'/>
      <channel name='record' mode='secure'/>
      <channel name='smartcard' mode='secure'/>
      <channel name='usbredir' mode='secure'/>
    </graphics>
    <graphics type='vnc' port='5902' autoport='yes' listen='192.168.10.11' keymap='en-us' passwdValidTo='1970-01-01T00:00:01'>
      <listen type='network' address='192.168.10.11' network='vdsm-ovirtmgmt'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='32768' vgamem='16384' heads='1' primary='yes'/>
      <alias name='ua-31537f3a-f1cf-4269-839c-bc82721ff7f3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='5'/>
      <alias name='ua-beeefd9c-c2bd-4836-83f0-a28657219b3e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <alias name='ua-7d25e1b5-4d03-4bc3-80f6-83a80d69b391'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c565,c625</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c565,c625</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

And yes. I can see the ovirt node and the ceph node talk to each other. But the ovirt node is only talking to the ceph monitor. There is no OSD involved. Both nodes can talk to each other, jumbo frames are allowed and working. In this output the ceph monitor is 192.168.20.21 and the ovirt node is 192.168.20.11

[root@ceph1 ~]# tcpflow -c -i enp3s0 src or dst host 192.168.20.11
tcpflow: listening on enp3s0
192.168.020.021.06789-192.168.020.011.44734: ceph v027
192.168.020.011.44734-192.168.020.021.06789: ceph v027
192.168.020.011.44734-192.168.020.021.06789: *D
192.168.020.011.44734-192.168.020.021.06789:
192.168.020.021.06789-192.168.020.011.44734:
192.168.020.021.06789-192.168.020.011.44736: ceph v027
192.168.020.011.44736-192.168.020.021.06789: ceph v027
192.168.020.011.44736-192.168.020.021.06789: *D
192.168.020.021.06789-192.168.020.011.44736:
192.168.020.021.06789-192.168.020.011.44738: ceph v027
192.168.020.011.44738-192.168.020.021.06789: ceph v027
192.168.020.011.44738-192.168.020.021.06789: *D!
192.168.020.021.06789-192.168.020.011.44738:
192.168.020.021.06789-192.168.020.011.44740: ceph v027
192.168.020.011.44740-192.168.020.021.06789: ceph v027
192.168.020.011.44740-192.168.020.021.06789: *D"
192.168.020.021.06789-192.168.020.011.44740:
192.168.020.021.06789-192.168.020.011.44742: ceph v027
192.168.020.011.44742-192.168.020.021.06789: ceph v027
192.168.020.011.44742-192.168.020.021.06789: *D#
192.168.020.021.06789-192.168.020.011.44742:
192.168.020.021.06789-192.168.020.011.44754: ceph v027
192.168.020.011.44754-192.168.020.021.06789: ceph v027
192.168.020.011.44754-192.168.020.021.06789: *D)
192.168.020.021.06789-192.168.020.011.44754:

And this goes on and on until the qemu log on the ovirt nodes shows:

2018-08-23T09:34:31.233493Z qemu-kvm: -drive file=rbd:volumes/volume-9b95b28c-9eec-4110-9973-88c161d3503f:id=cinder:auth_supported=cephx\;none:mon_host=192.168.20.21\:6789\;192.168.20.22\:6789\;192.168.20.23\:6789,file.password-secret=ua-9b95b28c-9eec-4110-9973-88c161d3503f-secret0,format=raw,if=none,id=drive-ua-9b95b28c-9eec-4110-9973-88c161d3503f,serial=9b95b28c-9eec-4110-9973-88c161d3503f,cache=none,werror=stop,rerror=stop,aio=threads: 'serial' is deprecated, please use the corresponding option of '-device' instead
2018-08-23T09:39:31.281126Z qemu-kvm: -drive file=rbd:volumes/volume-9b95b28c-9eec-4110-9973-88c161d3503f:id=cinder:auth_supported=cephx\;none:mon_host=192.168.20.21\:6789\;192.168.20.22\:6789\;192.168.20.23\:6789,file.password-secret=ua-9b95b28c-9eec-4110-9973-88c161d3503f-secret0,format=raw,if=none,id=drive-ua-9b95b28c-9eec-4110-9973-88c161d3503f,serial=9b95b28c-9eec-4110-9973-88c161d3503f,cache=none,werror=stop,rerror=stop,aio=threads: 
error connecting: Connection timed out
2018-08-23 09:39:31.291+0000: shutting down, reason=failed

So what keeps ovirt and ceph keep from talking ? The image introduction via cinder is done, but somehow ovirt is not contacting the ceph osds...

Due to upgrading the ovirt self-hosted engine I had the cluster in global maintenance mode. I was not aware, that issuing shutdown -r now within the engine VM will in fact not reboot the engine, but will only make it shutdown. After reading the engine troubleshooting guide once again I now know better.

For reasons unknown the root password for the node the engine is running on, is unknown. Thus I cannot get into the node via shell or web interface.

So... If I'd deactivate the global maintenance mode on a different node of the cluster that is capable of running the engine by issuing a hosted-engine --set-maintenance --mode=none, would this trigger a restart of the hosted engine ?

The cluster is running ovirt 4.1.9.

A node in our ovirt cluster shows an available upgrade. The node has the hosted engine deployed on it, but it is currently served on another node. Do I need to un-deploy the hosted engine first before making a minor upgrade ?

Currently for every upgrade I am moving the node to maintenance and then I will un-deploy the hosted engine. After that the node is active again and I have to put it into maintenance again before starting the upgrade. After the upgrade the node is active again, so it has to be put into maintenance again and then deploy the hosted engine once again.

That's a lot of steps to take for an upgrade. Can I skip the un-deployment of the hosted engine ?

I have an existing ovirt 4.1 cluster to which I would like to add a new node. Do I need to install the exact Node version the other nodes are running on? Or is it ok to use the latest minor version for the node?

I keep images on a private registry. When docker pulls an image and it is not found on the private registry it is instead fetched from the default docker registry.

How can one disable the default docker registry lookup ? I don't want to fetch images from the default docker registry at all.

I'm running on a debian hosts.

When I was checking a machine set up by our sysadmin team, I encountered an completely empty fstab. In fact it was the default fstab installed by a debootstrap.

# UNCONFIGURED FSTAB FOR BASE SYSTEM

I was going WTF first, but since the system is running for quite a while, I feel intrigued by this configuration. The system mounts seem to be fine:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=4123203,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3300328k,mode=755)
/dev/disk/by-uuid/477aa706-1737-4a08-8493-e617a58b8284 on / type ext4 (rw,relatime,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6600640k)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
cgroup on /sys/fs/cgroup type tmpfs (rw,relatime,mode=755)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio)

So my question is: Is this common practice and if not what side effects could arise from this configuration.