Being in the process to migrate the clients to a new cell (also changing the port from 5555 to 5565) I added a File Jukebox to the new cell, configure the repository files, and then scanned and imported those. That took a while, but succeeded.
When migrating StoreOnce, I basically did the same, used the existing Store, but in the new cell I see no media repository or media slots, etc.
So I wonder (how) can I import the media in the Store to the new cell?
Following the manual to install a new Cell Manager (CM) for Micro Focus Data Protector 11.02, I got stuck at the point when configuration should start using the Windows GUI:
host:port definitely does not work)
The issue was related to certificates, and even if the client had stored a different certificate for the new CM, it seemed to present the certificate of the other CM, causing a connection failure.
After significant work I was able to connect to the new CM, but lost connection to the old CM (but that's a different story)...* for the "group", and second I could not log in (In the old installation I was never asked about user or password; I assume the Windows Active Directory credentials were accepted somehow).
After spending half a day with reading docs, trying different things (like creating and testing multiple variants of user entries), and eventually browsing all logs on the CM, I entered the user as displayed as "Web Username" on the output of /opt/omni/bin/omniusers -list.
And that worked!So I wonder:
Is it expected that the user has to enter the username like "user_id|group|client"? If not, what might be the problem in the user created on the CM (I created the user using Windows user's loginname as "name", Active Directories name as "group", and the fully-qualified hostname as "client")?
I want to import a Windows Data Protector client that uses (legacy) port 5555 into a Cell Server running 11.02.
When I try it, I'm offered the certificate to accept, but the I get an [12:1625] Import host failed. error.
I suspect that it's due to the fact that recent installations default to using port 5565 instead.
So my question is: Is there a way to specify the port to use when importing a client? (I tried to add the port as ":portnr" to the host name, but that was not accepted.)
Time is OS like Linux is typically derived from a clock chip (RTC), or maintained by software using either periodic interrupts or some hardware registers (e.g. CPU's TSC cycle counter) for implementation.
Obviously in a virtual machine there is no direct hardware access (e.g. to RTC), so keeping the correct time may be tricky.
Specifically I'm wondering about the two POSIX clock implementations: CLOCK_REALTIME and CLOCK_MONOTONIC (there are more).
There are two major "disturbances" I'm considering:
Processes running in an operating system on bare hardware are interrupted only by the operation system (that has control then). So the operating system can keep the time easily.
An operating system running in a VM does not continuously have control over the CPU. For example if the OS "does not have the CPU", it cannot process timer interrupts. In turn that could cause the timer interrupts to be lost completely, be delayed by some seemingly random amount (jitter), or maybe even be processed in rapid sequence (processing "delayed" interrupts now). Likewise the clock would not progress as linearly as expected.
CLOCK_REALTIME: If the OS is missing CPU, the real-time clock could either be slowed down (lack behind), or jump forward occasionally to keep upCLOCK_MONOTONIC: If the OS is missing CPU, the real-time clock could either be slowed down (in relation to other VMs or wall-time), or jump forward occasionally to keep upCLOCK_REALTIME: Obviously if the real-time clock is slow, it cannot be used as an absolute timing measure, but it would look consistent within the VM.
If the clock keeps up by jumping forward variable amounts of time, it could be used as an absolute measure, but it would be bad for measuring any performance (duration) within the VM.CLOCK_MONOTONIC: Advancing the monotonic clock only if the VM "has the CPU" will provide a consistent view of elapsed time within the VM.
Making the clock jump forward variable amounts of time would prevent use for performance (duration) measurements within the VM.When live migration requires copying of gigabytes of RAM from one node to another, there will be some "freezing time" when the VM cannot run, lets say 3 seconds.
Now should the real-time jump forward by 3 seconds also, or should it loose the three seconds until being corrected manually or automatically at some later time? Likewise when the monotonic clock is being used to measure "uptime", should it take those three seconds into account by adding those, or should it account for the time when the VM actually had the CPU?
Like above, but there are more frequent short delays instead of occasional larger ones.
What approach does Xen use?
How does VMware handle that? Are there configurable options? (I know that in Xen the VMs can be synced from the hypervisor, or run independently (e.g. synced from external by using NTP))
Are there any "best practices"?
I wonder (after another failing live migration): Which VM attributes (properties) are copied from source to destination when a live migration of a PVM is being performed?
In our case the libvirt framework is being used in a pacemaker cluster.
Specifically I'm wondering about:
A recent failure I saw was this:
The swap device of a VM was changed from LVM LV to separate disk, so the new disk had been added via block-attach while the obsolete LV was removed in the VM.
The VM configs that the pacemaker cluster uses were updates on each node (but libvirt seems to have its own copies in RAM).
The VM ran fine until it was live-migrated: There were some error messages, but the VM continued for another 40 minutes until it stopped writing to systemd journal. On the console I saw repeating messages like these:
[94124.120477] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 232s!
[94154.815980] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 263s!
[94185.599474] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 293s!
[94216.278977] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 324s!
[94247.062530] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 355s!
[94277.682031] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 386s!
[94308.401531] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 416s!
[94339.157047] BUG: workqueue lockup - pool cpus=0-1 flags=0x5 nice=0 stuck for 447s!
Actually the new VM lacked the separate swap disk. But instead of panic-ing (and rebooting) the VM seemed to wait for something that would not happen.
After reboot I found these messages in jurnal:
Mar 23 20:02:19 v04 kernel: Freezing user space processes ... (elapsed 0.008 seconds) done.
Mar 23 20:02:19 v04 kernel: OOM killer disabled.
Mar 23 20:02:19 v04 kernel: Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
Mar 23 20:02:19 v04 kernel: PM: freeze of devices complete after 0.562 msecs
Mar 23 20:02:19 v04 kernel: suspending xenstore...
Mar 23 20:02:19 v04 kernel: PM: late freeze of devices complete after 0.104 msecs
Mar 23 20:02:19 v04 kernel: PM: noirq freeze of devices complete after 13.428 msecs
Mar 23 20:02:19 v04 kernel: xen:grant_table: Grant tables using version 1 layout
Mar 23 20:02:19 v04 kernel: Suspended for 1.170 seconds
Mar 23 20:02:19 v04 kernel: PM: noirq restore of devices complete after 0.166 msecs
Mar 23 20:02:19 v04 kernel: PM: early restore of devices complete after 0.085 msecs
Mar 23 20:02:19 v04 kernel: vbd vbd-51744: 2 reading other end details from device/vbd/51744
Mar 23 20:02:19 v04 kernel: xenbus: resume (talk_to_otherend) vbd-51744 failed: -2
Mar 23 20:02:19 v04 kernel: dpm_run_callback(): xenbus_dev_resume+0x0/0x130 returns -2
Mar 23 20:02:19 v04 kernel: PM: Device vbd-51744 failed to restore: error -2
Mar 23 20:02:19 v04 kernel: PM: restore of devices complete after 9.374 msecs
Mar 23 20:02:19 v04 kernel: OOM killer enabled.
Mar 23 20:02:19 v04 kernel: Restarting tasks ... done.
...
Mar 23 20:40:26 v04 systemd-logind[1034]: Failed to start session scope session-117.scope: Connection timed out
-- Reboot --
So likewise assume I've extended the VMs RAM while it was running. Will the RAM vanish after being live-migrated, or will the RAM setting be "copied over"?
Since upgrading a Xen host that also is a node in a pacemaker cluster from SLES15 SP2 to SLES15 SP3, I see these messages roughly every 30 seconds, and I wonder:
The messages look like this:
kernel: vif vif-6-2 vif6.2: Guest Rx stalled
kernel: vif vif-6-2 vif6.2: Guest Rx ready
As it seems these message repeat about every 30 seconds, and they seem to be related to VMs (PVM) that have the corresponding interface in "link down" state.
After realizing that my leap-seconds file is expired, I updated it.
However while ntpq indicates expire=202112280000, it did not re-read the current file.
I suspect that the file is being checked once per day only.
Thus the question is there a ntpq configuration command to force a re-read of the leap-seconds file immediately?
I'm using ntp 4.2.8p15 on Linux.
As a matter of fact, the new leap-seconds file was read one hour after the last expiration log message (I have expire=202206280000 now), but still I'd like to see an answer.
So it seems the file is checked every hour.
I have a few VM images based on a minimal QCOW2 image (say release 15.2).
Each VM's configuration is stored in a persistent image.
Now, when there is a newer QCOW2 image (say release 15.3), how would I "upgrade" the OS to be based on the new image?
That is I want to image have everything that the new (15.3) image has, except where I did configuration changes (like stuff in /etc).
I tried a qemu-img rebase, but the OS in the persistent image is still the old one (e.g.: /etc/os-release).
So I wonder what the correct procedure should look like.
Also I want to get rid of the older "15.2" images eventually.
I had a support case for Microfocus Data Protector 10.91 that turned out to be a file permission problem eventually.
User hpdp had group users instead of the intended omni, so I changed the groups of all affected files and directories after fixing the local user.
Actually the user should have been taken from LDAP, but was created locally with a different group, causing the problem. So I removed the local user and imported the user from LDAP. I had stopped Data Protector, rebooted the machine, and started Data Protector successfully. So everything seemed fine.
Backups work, the reporting Server is refreshed periodically, but suddenly some reports popped up an error saying that user should be removed and recreated.
Looking at the Users context, I see that no users and groups are displayed any more.
But the ClassSpec and UserList files (as well as WebAccess) are still there and filled.
Even when I try to recreate the admin group I get an unspecific error "Error while adding group" with identifier 159:9995.
I have no idea what might be wrong, but I checked that the AppServer Java process runs with the correct UID and GID, and the files in question can be read via the GID at least.
In the logs I see something that looks as an SQL statement to look up users.
/opt/omni/bin/omniusers -list also outputs nothing.
In DPserver.log I found "Get GroupName: User does not belong to a valid group.".
I have a Perl program with tradition (i.e. I use it over 10 years), but today I realized that it cannot find Data::UUID any more.
I also tried to install a corresponding package from the SLES software repository, but I could not find one.
AFAIR it still worked in January this year.
I'm using perl-5.18.2-12.23.1.x86_64 from SLES 12 SP5. Could it be a packaging error? It also seems that several "perldelta" manual pages (describing the changes between versions) are missing, so I cannot look up whether the standard distribution had changed.
I also see the same problem in openSUSE Leap 15.3.
When a new LUNs are presented to Fibre Channel SAN from a Pure Storage FlashArray//X70 R3, Linux (SLES) does not find those (as opposed to other FC storage systems like the HP EVA or 3PAR StorServ).
Even after a rescan-scsi-bus.sh the new LUNs are not detected by the Linux kernel.
The only two methods that are known to work are:
Reboot the machine after the LUN mappings have changed.
Use option -i for rescan-scsi-bus.sh, re-initializing the loop via LIP primitive.
The second method causes I/O errors for existing LUNs on the loop when they are being accessed while LIP is executing.
Naturally for an enterprise server or even HA cluster neither frequent reboots nor collateral I/O errors are acceptable as a solution. It doesn't help that I wouldn't have bought that storage system. It's there now.
Micro Focus Data Protector 10.91 should work with Linux distributions like SLES using systemd, but it seems client instalation does not work: When I try to install the client, I get an error message ("[110:101] Client import failed!") while the installation steps before (Core Component, Core Technology Stack, Disk Agent) looked successful.
When examining the system, I see there is a omni.socket service running, but the referenced omni.service does not exist (there is no corresponding unit file).
On closer inspection I found a [email protected] in /usr/lib/systemd/system/, but that seems invalid.
What can I do to fix this?
Here is how the service files look like:
# /usr/lib/systemd/system/omni.socket
[Unit]
Description=DATA-PROTECTOR-INET
PartOf=omni.service
[Socket]
ListenStream=5555
Accept=yes
MaxConnections=1000000
MaxConnectionsPerSource=100000
[Install]
WantedBy=sockets.target
# /usr/lib/systemd/system/[email protected]
[Unit]
Description=DATA-PROTECTOR-INET
Requires=omni.socket
[Service]
StandardInput=socket
PIDFile=/var/run/omni.pid
ExecStart=/opt/omni/lbin/inet -log /var/opt/omni/log/inet.log
Type=simple
KillMode=process
[Install]
WantedBy=default.target
To me those service files seem highly incomplete: They are missing dependencies like network and local filesystems.
To clarify: I'm not looking for installation instructions like those the product provides already (they don't work). Instead I'm looking for a way to make the buggy version work while the vendor is fixing their bugs.
Using indirect locks of virtlockd (which is used by libvirtd) requires to use a cluster-wide shared filesystem like OCFS2.
In turn this means that virtlockd must be started after the shared filesystem is mounted (otherwise the locks created would be local at best).
Naturally libvirtd must be started after virtlockd, and any VM after virtlockd.
So I want for start: pacemaker, DLM, OCFS mount, virtlockd, libvirtd, VMs...
And for stop I want the opposite direction.
I have configured all those primitives (specifically systemd:libvirtd.service and systemd:virtlockd), clones and constraints correctly (I hope), but still I'm having issue with virtlockd.
In a system like SLES15 systemd is controlling those services, and it seems systemd has its own life controlling (starting) services even if they are all disabled.
So the question: Did anybody manage to succeed with such a setup?
I found this "Drop-In" in the status output for virtlockd.service:
/run/systemd/system/virtlockd.service.d/50-pacemaker.conf
It contains:
[Unit]
Description=Cluster Controlled virtlockd
Before=pacemaker.service pacemaker_remote.service
[Service]
Restart=no
A corresponding file /run/systemd/system/libvirtd.service.service.d/50-pacemaker.conf exists:
[Unit]
Description=Cluster Controlled libvirtd.service
Before=pacemaker.service pacemaker_remote.service
[Service]
Restart=no
Could these cause the problems I'm seeing (systemd starting libvirtd-ro.socket, libvirts-admin.socket and libvirtd.service, then starting virtlockd)?
It seems the resources are started in the correct order when the node boots (e.g. after being fenced), but when pacemaker is restarted (e.g. via crm cluster restart), systemd interferes and starts the virtlockd before pacemaker wants to start it.
Maybe the difference is the /run directory.
Another issue I found is that even though /etc/libvirt/libvirtd.conf contains listen_tls = 1, starting libvirtd through pacemaker as indicated results in a libvirtd not having opened the TLS socket, preventing VM live migration.
I wrote a program (monitoring plugin) that reads the cpu usage numbers from /proc/stat.
Unfortunately it seems the numbers do not match the manual page proc(5), especially I get different proportions for real machines and Xen paravirtualized machines:
The amount of idle time is different between these machines:
(An explanation on the "code blocks" following:
The first parts show the fields of the "cpu" line (e.g.: "cpu#1" is the first field of the CPU-line), followed by a boot count ("epoch"), the UNIX time, and the actual value, each being separated with a colon (:).
Next line starting with "stat: OK" is the output of my monitoring plugin; here it outputs the differences for debugging purposes, but usually it would output difference rates.
It also adds user-readable labels to the numbers.
"time" is the time difference since last call in seconds.
Finally I added the CPU-related lines from /proc/stat (with some time elapsed since plugin output).)
First a two-CPU, 6 core, two threads each physical machine:
Idle time seems to be about 900 times the sum of the other CPU states, corresponding to 99.89% idle and 0.06% user CPU.
Also note that the relation of idle time to elapsed time is about 2398.5; when divided by USER_HZ (100) you get the number of CPUs roughly.
It looks odd to me:
# physical two 6-core cpus, 2 threads each
cpu#1=0:1596547833:2667804
cpu#2=0:1596547833:90388
cpu#3=0:1596547833:1257514
cpu#4=0:1596547833:2735255340
cpu#5=0:1596547833:142707
cpu#6=0:1596547833:0
cpu#7=0:1596547833:107191
cpu#8=0:1596547833:0
cpu#9=0:1596547833:0
cpu#10=0:1596547833:0
stat OK: epoch=0, time=354, cpu.usr=581, cpu.ni=24, cpu.sys=288, cpu.idl=849070, cpu.iow=29, cpu.hirq=0, cpu.sirq=13, cpu.st=0, cpu.vgst=0, cpu.usr0=0
# cat /proc/stat
cpu 2668778 90430 1257998 2736664140 142741 0 107213 0 0 0
cpu0 116314 1436 53622 113861868 3864 0 81296 0 0 0
cpu1 142008 4782 32464 114026161 9767 0 4396 0 0 0
cpu2 167052 3058 63902 113932120 12818 0 1966 0 0 0
cpu3 120029 4260 28712 114058016 3337 0 1478 0 0 0
cpu4 145332 2972 61798 113983716 16115 0 1037 0 0 0
cpu5 114346 6809 27875 114060364 4110 0 1124 0 0 0
cpu6 126193 3720 54701 113999094 12348 0 968 0 0 0
cpu7 108188 4859 27436 114067537 6028 0 976 0 0 0
cpu8 121890 2820 51548 114020211 13474 0 940 0 0 0
cpu9 102942 4235 26150 114076765 3423 0 977 0 0 0
cpu10 125984 2724 48521 114014015 13950 0 845 0 0 0
cpu11 89154 4047 26674 114085160 7735 0 885 0 0 0
cpu12 116730 3894 397743 113663892 2352 0 884 0 0 0
cpu13 84306 4424 26164 114096015 2767 0 871 0 0 0
cpu14 127293 3539 44438 114033462 1294 0 922 0 0 0
cpu15 77740 3958 26201 114105245 358 0 854 0 0 0
cpu16 133217 3043 41476 114034324 737 0 958 0 0 0
cpu17 88893 4497 25736 114094645 662 0 838 0 0 0
cpu18 125887 2812 39150 114024555 1309 0 806 0 0 0
cpu19 65198 3560 25976 114092343 21838 0 802 0 0 0
cpu20 109361 3292 37270 114059144 1381 0 764 0 0 0
cpu21 71055 4094 26435 114111750 759 0 859 0 0 0
cpu22 118589 3643 37525 114052728 1567 0 883 0 0 0
cpu23 71069 3943 26468 114110998 737 0 875 0 0 0
Then a Xen paravirtualized machine with two virtual CPUs. The idle time is about 74 times the sum of the other CPU states, corresponding to 98.66% idle and 8% user cpu. Again if you take the proportion of idle time to elapsed time, you get 197.4, roughly corresponding to 2 CPUs. Here's one problem: User CPU and Idle exceed 100%.
## virtual 2 cpus (Xen PV)
cpu#1=0:1596547988:1162034
cpu#2=0:1596547988:227660
cpu#3=0:1596547988:3036855
cpu#4=0:1596547988:701649884
cpu#5=0:1596547988:1037577
cpu#6=0:1596547988:0
cpu#7=0:1596547988:31478
cpu#8=0:1596547988:355862
cpu#9=0:1596547988:0
cpu#10=0:1596547988:0
stat OK: epoch=0, time=36, cpu.usr=16, cpu.ni=7, cpu.sys=28, cpu.idl=7108, cpu.iow=4, cpu.hirq=0, cpu.sirq=0, cpu.st=5, cpu.vgst=0, cpu.usr0=0
> cat /proc/stat
cpu 1162136 227690 3037149 701727879 1037664 0 31481 355901 0 0
cpu0 531438 112727 1469157 350497090 791387 0 31011 192100 0 0
cpu1 630698 114962 1567991 351230788 246276 0 470 163801 0 0
I know that these numbers in /proc/stat are USER_HZ, but that shouldn't matter as a common factor, right?
I feel the idle proportion does not match the rest of the CPU states (too high?), but I fail to recognize what's wrong.
(I also realized that for multiple cores you can never read read those numbers from /proc/stat consistently, but the differences would be small enough to ignore)
When a package wants a runtime directory, there exists systemd's tmpfiles.d that may contain the temporary files a package needs. So when installing an RPM package, it is the correct thing to install a package.conf file to /usr/lib/tmpfiles.d/, and then do a systemctl daemon-reload?
Obviously not: The newly required directories are not created. It seems one must explicitly run systemctl start systemd-tmpfiles-setup.service. I'm confused: In pre-systemd area I simply added the runtime directory to the package, but now /var/run is empty after each boot.
What's the recommended clean solution for RPM and systemd?