Questions[failed](server)

In General

I've been scouring the web looking for actually helpful documentation about systemd and this error, but the majority of it has been troubleshooting niche cases or goes into such extreme detail that the information is useless. I'm trying to set up 1 simple, custom service. It should be a simple matter of registering then enabling the service, but apparently systemd makes that more complicated than it needs to be.

This has been a thorn in my side for years, and I'm finally at a point where I'm ready to abandon any operating system hosting systemd in favor of something that isn't completely insane in its implementation and will run as expected for a simple init script.

Out of general

Before I completely give up on systemd, I want to give it one more shot. What I want to know in detail (but explained in a scope suitable for a junior systems developer) is

1. (forest view, ie toplevel explanation desired without technical detail) What types of services does systemd manage, and how (chronologically) did systemd expand to manage them?
2. (trees view, ie include technical detail in description) Where specifically do each of these systems live on a typical Debian system?
3. (forest) How do sym(or hard)links factor into tying all these systems together, and what happens if symlinks aren't managed properly?
4. (branch view, as much nitty gritty as possible, don't hold back) How does systemd manage running services (ie: what kind of workflow does it follow)?
5. (trees) Similar to the previous question, are these different types of services started in tiers? (I know there's dependencies and runlevels .. but how are these assigned to different system levels .. ie /opt software vs kernel level)
6. (branch) How does one go about determining which level service handler to use?
7. (branch) How do you determine which runlevel it should be using, as well as additional options related to the service's successful execution
8. (branch) Where can individual (working) examples be found or used as a template? I would think there would be some sort of skeleton folder for different kinds of services, but I've yet to find it.
9. (branch) If a new service isn't create correctly, what steps can be taken to remove the faulty service and start clean? Many of my attempts to start clean after a failed service creation clearly have been stopped by remnant symlinks and other vestigial files left behind by the initial failed attempt, and despite using "find" and "ls" (and yes, executing sync and echo 3 > cache) to categorically search and destroy any file even remotely related to the service, the service still fails.
10. (trees) To make life easier, a quick way to find any symlinks that might be gunking up system.d's process (which shows absolute path)

Bought a little more general

My specific error

1. systemctl enable lidarr.service

   Failed to enable unit: Unit /run/systemd/generator.late/lidarr.service is transient or generated.

2. cat of /etc/init.d/lidarr.service .. meow

3. ls of /etc/init.d/

4. ls of /etc/systemd/system, part 1

5. ls of /etc/systemd/system, part 2

6. ls of /etc/systemd/system, part 3

Things I've tried to do to fix it

• Daemon reload
• Delete all traces of the failed service
• Flush the cache
• Remove any symlinks (that I could find)
• Chmod +x the service
• Manually create a symlink where others were created, pointing to the service (this has worked sporadically in the past .. again, systemd has too many handlers floating around that behavior isn't predictable)
• Cry, drink somewhat heavily
• Plead, bargain
• A full system reinstall .. yes, the nuclear option

Pretty please, with sugar on top .. make this not be so confusing

Edit1: Unlike what I originally thought when creating this question, it seems that the problem was NOT directly caused by the unplanned power-cycle, but by a mistake I might have done with beadm in the past, and which became only effective due to the first reboot since then.

This was (and still is) the actual core of the question:

I'd think that most of the changes since that snapshot OUGHT to be still somewhere on the disk. Is there any remote chance (short of writing my own zfs-reader for the raw device) to access specific files that were changed since that last snapshot?

Further Info as requested in comments:

beadm list

BE Name          Flags Mountpoint Space   Policy Created          
---------------- ----- ---------- ------- ------ ---------------- 
11.4.23.69.3     NR    /          73.93G  static 2020-07-27 12:08 
solaris          -     -          95.53G  static 2020-05-23 19:35 
solaris-backup-1 -     -          374.41M static 2020-07-22 14:37 

zfs list:

NAME                               USED  AVAIL  REFER  MOUNTPOINT
rpool                              382G   713G  73.5K  /rpool
rpool/ROOT                         170G   713G    31K  none
rpool/ROOT/11.4.23.69.3           75.5G   713G  63.3G  /
rpool/ROOT/11.4.23.69.3/var       3.12G   713G  1.14G  /var
rpool/ROOT/solaris                94.2G   713G   143G  /
rpool/ROOT/solaris-backup-1       98.9M   713G  48.5G  /
rpool/ROOT/solaris-backup-1/var      1K   713G  1.13G  /var
rpool/ROOT/solaris/var             503M   713G  1.29G  /var
rpool/VARSHARE                     102M   713G  24.7M  /var/share
rpool/VARSHARE/kvol               27.7M   713G    31K  /var/share/kvol
rpool/VARSHARE/kvol/dump_summary  1.22M   713G  1.02M  -
rpool/VARSHARE/kvol/ereports      10.2M   713G  10.0M  -
rpool/VARSHARE/kvol/kernel_log    16.2M   713G  16.0M  -
rpool/VARSHARE/pkg                  63K   713G    32K  /var/share/pkg
rpool/VARSHARE/pkg/repositories     31K   713G    31K  /var/share/pkg/repositories
rpool/VARSHARE/sstore             30.0M   713G  30.0M  /var/share/sstore/repo
rpool/VARSHARE/tmp                20.0M   713G  20.0M  /var/tmp
rpool/VARSHARE/zones                31K   713G    31K  /system/zones
rpool/dump                        63.1G   713G  63.1G  -
rpool/export                      20.5G   713G    32K  /export
rpool/export/home                 20.5G   713G  7.26G  /export/home
rpool/export/home/avl             9.30G   713G  9.30G  /export/home/avl

(most of that - except my homedir - is what the machine came with)

Only the root-filesystem seems to have been rolled back, my homedir still has all the recent files.

According to output of df -kl the root filesystem is currently this one: rpool/ROOT/11.4.23.69.3, which already corresponds to the newest BE available.

I also hope to learn from answers what really might have caused the rollback. No, I don't exactly remember my beadm invocations. I only remember that I changed the BE for next boot, but then changed it back to current BE and didn't reboot - until that electric power failure.

Maybe answers here might also save someone else later.

I've got an HP DL 380 G6 that has been having strange intermittent issues. It all started around a year ago when it was in production. I got the 10 PM call that something was wrong, stumbled to the datacenter, and found this server in a boot loop flashing on and off. I finally got the thing to boot through a combination of breaking it down to minimum configuration, cursing, and dumb luck. I say the latter because it hasn't been consistent since.

Since that night it's been part of my test environment. It was running just fine for a number of months before I turned it off simply because I didn't need it. Now I'd like to get it on again and the problem is back. I've tried the following:

• Removing power and waiting a day.
• Minimizing the memory configuration.
• Cursing
• Following the classic HP flowchart
• Pulling the procs and reseating them both in the primary socket to determine if one is perhaps bad.
• Breaking the server down to minimum configuration.
• More creative cursing.

iLO appears to be non-functional in this state, and when I was able to get into back when it worked there were no faults recorded. The UID light doesn't respond when pressed, except when it's first plugged in. Almost as if the fault strikes several seconds after power is applied. In the past when it has turned on it did so suddenly after thorough prodding, but without touching the power button.

Anyone seen anything like this before? Is this machine doomed to sit with it's one amber eye staring into oblivion? Should I just accept that trying to resurrect a 6 year old server that's now going for sub-$200 prices on ebay maybe isn't worth it? Have I shamed it into a non-functional state with my offensive and often out-dated slurs? Will I ever run out of one-liner questions? Time will tell.

Errors: Windows event Log - Backup started at '5/5/2011 12:34:07 AM' failed as Volume Shadow copy operation failed for backup volumes with following error code '2155348129'. Please rerun backup once issue is resolved.

The description for Event ID 24583 from source SQLWRITER cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

ICommandText::Execute 0x80040e14 SQLSTATE: 42000, Native Error: 3013 Error state: 1, Severity: 16 Source: Microsoft SQL Native Client Error message: BACKUP DATABASE is terminating abnormally.

SBS backup Error - A Volume Shadow Copy Service operation failed. Unknown error (0x800423f4).

System: SBS 2008 SP2 Running Full Backup to external HDD using Windows Server Backup.

What I've tried: - Investigated applying certain updates and hotfixes but research showed that it would not resolve the isse. http://support.microsoft.com/kb/959962.

• Checked to see if the drive was set to auto-mount and that possibly being the reason why it wasn't detected - but my command prompt instructions informed me that it was already enabled.

• Went through disk management to addure that there was no OEM disk partition that was active as it is also known to cause issues (no OEM active)

• Removed the lock on the External HDD from sbs backup allowing me to run individual back-up tests on the external HDD.

• Configured the set-up to scan once (including C and E) but the same error occurred.

• Used Windows Backup (as opposed to SBS backup) to create a once off backup but also failed with the same error NOTE: aware that SBS Backup uses Windows server backup.

===> Ran Backup with SQL VSS writer serice stopped and full backup was successful - however if this service is stopped the backup does not contain SQL db backup. Currently running SQL 2005 at SP3 (which is updated to the latest version) ruling out possible updates to resolve the issue. May need to create Batch files to disable culprit before backup and reenable after backup but this is a work around not the solution.

• Stopped the service SQL server VSS writer then ran Regedit. Changed the registry key to use MSDEWriter instead of SQLServerWriter.The MSDEVersionChecking value is located in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Settings (Value changed from 1 (uses VSSwriter) to 0 (uses MSDEwriter)). Enabled the VSS service and ran a backup on the premises that if that backup is successful whilst VSS service is running the change has worked and it is now using msde to backup the SQL DB (did not work when scheduled in, oddly enough the edited registry doesnt appear to change the primary writer from VSS to MSDE (or perhaps the presence of the service is enough to end the backup before initialization)

==================================================================================

In cmd vssadmin list writers confirms:

Writer name: 'SqlServerWriter' Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a} Writer Instance Id: {02638a86-5f10-480f-a994-81b7a47132f3} State: [8] Failed Last error: Non-retryable error

Dead out of ideas here to resolve this issue. Any assistance or possible solutions would be greatly appreciated.

iperf -c 192.168.3.2 -i 2
connect failed: Connection refused

3.2 is another machine. I think the port might be blocked.

How can I make this work?