S. Nixon's questions

I have Ubuntu 18.04 Server installed on a VMware platform and I have two established VGs: rootvg is 500GB and hosts the O/S on /dev/sda. The datavg was 2TB and contains our PostgreSQL database environment, stored on /dev/sdb.

Recently, our team decided they need more disk space and so through the virtual machine they added another 1TB to the server for use by the datavg. They did this by changing the size of the virtual disk that they have currently in use. The problem with this is that I am not able to see the new disk space.

If I look at the datavg using vgdisplay, I see the old 2TB size:

vgdisplay datavg

  --- Volume group ---
  VG Name               datavg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  35
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                24
  Open LV               24
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <2.00 TiB
  PE Size               4.00 MiB
  Total PE              524287
  Alloc PE / Size       521216 / <1.99 TiB
  Free  PE / Size       3071 / <12.00 GiB
  VG UUID               vMxLPX-M3RG-BIVR-MsiF-dWtF-xLOl-4VCTGW

But if I check the disk in fdisk, it reports the new 3TB size.

fdisk /dev/sdb
    
Command (m for help): F
Unpartitioned space /dev/sdb: 3 TiB, 3298533834752 bytes, 6442448896 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

Start        End    Sectors Size
 2048 6442450943 6442448896   3T

I am not sure exactly what needs to be done to make the 2TB disk show up now as 3TB and give me access to the extra 1TB that was added. Originally, I thought I would need to add a new partition in fdisk, then use pvcreate and extendvg to add that as a new disk to the volume group. But I am now confused, because I expected datavg to already have at least one partition defined that is being used for the existing LVs on datavg. I cannot find any existing partitions on /dev/sdb.

Command (m for help): p
Disk /dev/sdb: 3 TiB, 3298534883328 bytes, 6442450944 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb32bb8b8

Can anyone point me in the right direction here? Is there an easy way to alter the size of a VG after it was created and is in use, or do we have to back up VG contents, destroy VG, recreate VG larger, reload the backup? Is there an equivalent to resize2fs but for VGs?

Thanks,

Steve N.

I have been searching for a backup tool that would allow me to create a bootable copy of the Ubuntu 18.04 Server (or higher) operating system image that is installed on any of my machines.

I come from an IBM AIX background. On that platform, there are a couple of tools that can be used (mksysb, mkdvd) that allow you to generate an image of the O/S and the way it is currently configured. In the event of a DR situation, you are able to slip the DVD into the drive, boot the system, and it will go ahead and re-install exactly as it was at the time the backup was made. These are HOT backups. You do not need to have the users logged off.

In Ubuntu, what I found so far was that there is no easy solution to generate an ISO of how the system is installed at this moment. Most open-source packages I found require that the server be rebooted into some sort of maintenance mode to be able to create the media needed. On a production server, that means getting all other users off and/or shutting down over a weekend, provided the system is not used 24/7.

Does anyone have a routine or a product they use to back up just the Ubuntu operating system, in a manner that can be restored back to the way it was? I mean not having to recreate users and passwords, keep any printers that have been defined, save the netplan, hosts, services files, etc.

Our servers are all installed with the O/S on one RAID-1 set of disks (rootvg), and any databases installed in a separate RAID-5 array on a separate volume group (datavg), if that helps. The idea being that we can re-install or upgrade the Ubuntu on rootvg without affecting the data in the other volume group (s).

I have two PCs running Ubuntu 18.04 LTS. They were created from the same linux base image. Recently I discovered that on one of the two boxes, I am unable to use crontab to view or edit the individual user cron tables.

ServerB is giving this error as any user, including root:

$ crontab -l
crontab: usage error: only one operation permitted
usage:  crontab [-u user] file
        crontab [ -u user ] [ -i ] { -e | -l | -r }
                (default operation is replace, per 1003.2)
        -e      (edit user's crontab)
        -l      (list user's crontab)
        -r      (delete user's crontab)
        -i      (prompt before deleting user's crontab)

I thought at first that a running process may have the cron locked, but the problem persisted after a full reboot of the server.

Researching this online only indicates that some admins have chosen to recompile crontab to eliminate the "remove" flag. That was NOT the case here. This crontab command came with the O/S and is identical to the one on the working server.

There are also no messages in the /var/log/syslog or in /var/log/cron.log to show why it may be failing. crontab -e or crontab -l both return the error above.

This is the /usr/bin/crontab from both the working and problem servers. Identical dates and sizes.

-rwxr-sr-x 1 root crontab 39352 Nov 16  2017 /usr/bin/crontab
-rwxr-sr-x 1 root crontab 39352 Nov 16  2017 /usr/bin/crontab

$ diff -s crontab.serverA crontab.serverB
Files crontab.serverA and crontab.serverB are identical

Finally, permissions and ownerships are identical on both servers for the /var/spool/cron path and files.

I could just add any entries to the /etc/crontab file for the selected user, but just trying to understand why this error is appearing in the first place. Is there some sort of lock file left behind somewhere that could cause this?

I currently have about forty Ubuntu Servers (mostly 18.04 and 20.04) that I am installing ClamAV and maldetect on so we have at least some monitoring for malware on the systems. What we will likely do is set up an email address ([email protected]) to collect and view all the reports periodically to check for any viruses or malware.

One question I had was whether these products have the ability to be monitored from a remote central server. If I have a Windows PC or Ubuntu Desktop, can I have that system go and poll for the latest scans, or have the monitored servers send me XML or JSON data in some sort of email?

The idea would be that I would like to have one system that I go to and can examine all the servers from a central console instead of having to log into each one, or open 40+ emails each day.

I have two servers that (at least at one point) were identical and working fine. Ubuntu 18.04 Server pl2poland is a backup server for pl3poland and hosts a replication PostgreSQL database. We use the Admin user "invadm" to log on to both servers using SSH.

For some reason, the pl2poland server suddenly has started giving us broken pipe messages when you try to log on as the invadm user.

This message shows up when I try to SSH to pl2poland from other servers:

$ ssh invadm@pl2poland
invadm@pl2poland's password:
Write failed: Broken pipe

I do have other logins that work on pl2poland, but if I try to ssh from any user (even root) on pl2poland to the invadm user, I get this slightly different error:

root@pl2poland: ssh invadm@pl2poland
invadm@pl2poland's password:
packet_write_wait: Connection to 192.168.23.5 port 22: Broken pipe

I believe that there must be an issue with that specific user, but I've tried various things and cannot track down a fix for it. The other questions I've seen here all suggest a timeout may be happening and to adjust settings in /etc/ssh/ssh_config. But this is happening immediately, not after a few minutes of idle time.

Here is a what happens within ssh -vv messages at the point where I enter the password. It is a valid password (confirmed) and the ssh even says "Authentication Succeeded", but then kicks us out.

debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /.ssh/id_dsa
debug1: Trying private key: /.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug1: Next authentication method: password
invadm@pl2poland's password:
debug2: we sent a password packet, wait for reply
debug1: Authentication succeeded (password).
Authenticated to pl2poland ([192.168.23.5]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting [email protected]
debug1: Entering interactive session.
Write failed: Broken pipe

I tried changing the password, checking permissions on /home/invadm and /home/invadm/.ssh, tried a suggestion to use "ssh -o IPQoS=throughput". Nothing seems to make a difference.

Can someone point me to a file to check or an article to read that maybe I did not spot yet?

Thanks,

Steve N.

Just noticed something odd happened to two of my Linux servers. We have two dozen Ubuntu 18.04 LTS servers, and all of them ask us for a prompt in the same way:

# ssh admin@pl3poland
admin@pl3poland's password:

However, we recently started enforcing stronger password requirements on two servers. We changed the settings in /etc/pam.d/common-password for libpwquality.

Before:

password        requisite                       pam_pwquality.so retry=3 

After:

password        requisite                       pam_pwquality.so retry=3 minlen=12 difok=3 minclass=4 maxrepeat=2 dictcheck=1 usercheck=1

It seems that after these values were changed, now the password prompt shows up differently. This is what comes up now:

# spawn ssh admin@spain
Password:

This is a relatively minor difference, but it has affected a few of our automated routines where we use expect scripts to connect to systems. I've changed the script to look for "sword:" instead of the full "password:", but I'm just curious why that would change, and if there is any way to change it back?

Looking through documentation, I found a command called "password-prompt" but it does not look like that is something I can use to permanently set the password prefix.

Anyone able to direct me to a .conf file or some documentation that explains how to change this? I come from an AIX background, and there was a file called /etc/security/login.cfg that we could change to adjust the "herald" for each login. But I don't see something similar in Ubuntu.

Thx

Steve N.

I have two Ubuntu 18.04 servers at my office that are for development work, and we have the GNU Debugger installed. All of these packages are always installed using the "sudo apt install" command. Somehow, one of the servers has a newer version of gdb than on the other.

#DVLP1

# sudo apt list --installed |grep ^gdb/
gdb/bionic-updates,now 8.1.1-0ubuntu1 amd64 [installed]

# sudo apt-cache policy gdb
gdb:
  Installed: 8.1.1-0ubuntu1
  Candidate: 8.1.1-0ubuntu1
  Version table:
 *** 8.1.1-0ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     8.1-0ubuntu3 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

#DVLP2

# sudo apt list --installed |grep ^gdb/
gdb/bionic,now 8.2-0ubuntu1~18.04 amd64 [installed]

# sudo apt-cache policy gdb
gdb:
  Installed: 8.2-0ubuntu1~18.04
  Candidate: 8.2-0ubuntu1~18.04
  Version table:
 *** 8.2-0ubuntu1~18.04 500
        500 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status
     8.1.1-0ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     8.1-0ubuntu3 500
        500 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

Some of our developers on DVLP1 are complaining that they are receiving the following warning, which they are not getting with gdb on DVLP2:

==28205==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)

The only thing that I can see which is different is the gdb, so I was going to try and upgrade it to the same version on both. While gdb version 8.2 is listed in the repositories on the first system, it still only shows version 8.1.1 as the most recent copy on the second system.

I'm guessing this has something to do with the selected repositories we are downloading from. I noticed one box uses "archive.ubuntu.com" and the other "us.archive.ubuntu.com" for the bionic main. However, even after I changed the sources.list to have the same deb entries (and did the apt update), it was not showing a version higher than gdb 8.1.1

At this point, is there an easy way for me to install a package that is listed in DVLP2's apt-cache, but not in DVLP1? It looks like the 8.2 version originated from ppa.launchpad.net, but I don't remember having this as a deb entry. How would I add it to sources.list if I want to try that copy (and can I trust its contents)?

Thx Steve

I have a situation where I need to allow an outside consultant to access one of our Ubuntu 18.04 systems. We would like to lock that user's account in a way so that when it logs in, they can only access three or four directories, and only use certain LINUX commands like cd, /bin/cp, more, less, cat, for example.

I know that using sudo, I can configure a list of commands that this user is allowed to use from their $HOME directory. But is there also a way (using sudo) to allow the user to access Directory X but not allow them into Directory Y? For example, the user $HOME is /home/outsider, but I have three folders I want them to also be able to copy files into and read files from:

/workdir
/software-repository
/logs

I know I can control access to these folders through the group access rights (chmod). But is there a way to NOT allow a specific user access to some directory or file in sudo despite having read access to it on the system?

For example, /etc/hosts file has the following privileges:

-rw-r--r-- 1 root root 277 Sep 11 18:09 /etc/hosts

So only root can edit this file, but anyone can read it. Is the solution for this example just to chmod 0600 the /etc/hosts? I'm afraid that would cause problems for other apps that may be trying to read the hosts file.

And I can't just chroot the user's account, because they will need to be able to place files in a directory that is outside their $HOME structure.

Right now the only thought is to create a menu that comes up when they log into that account that traps them into a limited number of choices, such as:

1) View log file
2) Copy XML template
3) Test print report
0) Log off

If anyone has done this before, is there a better way?

Thx,

Steve

I have this very simple source code used as a test program:

#include <stdio.h>
#include <stdlib.h>

int main()
{
        FILE *fp;

        fp = fopen("file.txt", "w");

        fclose(fp);
        fclose(fp);

        printf("Completed successfully\n\n");
        return(0);
}

I am running this code on two Ubuntu servers, both running Ubuntu 18.04 LTS. It is dying on MachA at the fopen, while on MachB it runs fine and gets to "Completed successfully".

The exact error is:

./testpgm
free(): double free detected in tcache 2
Abort(coredump)

Using gdb, it appears the code is using the same search path. The same shared libraries are reported in both systems:

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007ffff7dd3f10  0x00007ffff7df4570  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff7a032d0  0x00007ffff7b7bb7c  Yes         /lib/x86_64-linux-gnu/libc.so.6

The only thing I can think of that might affect this is on the problem machine, I have g++-9 and I am not sure how I got that installed because it does not appear to be listed in the apt-get as an installable package anymore.

Problem server:

g++-7/bionic-updates,bionic-security,now 7.5.0-3ubuntu1~18.04 amd64 [installed]
g++-8/bionic-updates,bionic-security,now 8.4.0-1ubuntu1~18.04 amd64 [installed]
g++-9/bionic,now 9.3.0-11ubuntu0~18.04.1 amd64 [installed]

Working server:

g++-8/bionic-updates,bionic-security,now 8.4.0-1ubuntu1~18.04 amd64 [installed]

Is it possible that the g++-9 was available in apt for download and then was removed due to problems? I'm debating trying to remove that package to see if it solves the issue, but wanted to know if anyone else has encountered this issue, or has any suggestions for other things to look for.

UPDATE:

As suggested by waltinator, I tried the strace command, but it did not offer much insight as to why this may be happening. The strace shows pretty much the same data between the working and problem system. Only difference at the beginning was an attempt to access ld.so.preload on the copy that is working.

Where it actually fails, I still don't understand why.

WORKING SYSTEM

openat(AT_FDCWD, "file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
close(3)                                = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
write(1, "Completed successfully\n", 23Completed successfully
) = 23
write(1, "\n", 1
)                       = 1

FAILING SYSTEM

brk(NULL)                               = 0x5559121ec000
brk(0x55591220d000)                     = 0x55591220d000
openat(AT_FDCWD, "file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
close(3)                                = 0
writev(2, [{iov_base="free(): double free detected in "..., iov_len=40}, {iov_base="\n", iov_len=1}], 2free(): double free detected in tcache 2
) = 41
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7faddf948000
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0

The ulimit values only differ because the amount of memory is 64GB on failing system and 128GB on working system. thought maybe it was a memory shortage, but I can run other programs, such as starting Postgres and access a database.

I have some automated tools that need to be able to run as a specific user. The username is "ig_root" and it is only used by a monitoring tool (checks disk space, database, CPU workload, etc) that sends us warnings when something may need human intervention.

What we need to be able to do is to issue a sudo command to run things as the ig_root user. I have my "invadm" account that is a member of sudo group, and it can "sudo su - root", but it prompts for a password. What I need to set up is to be able to do something similar for a non-root user but that does NOT prompt for the password.

In other words, as "invadm" user we want to do this:

# sudo -u ig_root check_dbspace

...without seeing this:

[sudo] password for invadm:

But I also want conecting to root to still require the password entry for everyone:

# sudo su - root
[sudo] password for invadm:

Is that possible?

My current visudo entries look like this on my test server:

# User privilege specification
root    ALL=(ALL:ALL) ALL
invadm  ALL=(ig_root) NOPASSWD: ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

Any ideas as to what I'm missing? I thought maybe I have to issue the systemctl and stop/restart something, but I learned that there is no sudo daemon to restart. I also tried logging out and back in to pick up any new environment... no luck.

Right now, the only account that sudo does not ask for a password is when I'm root user.

# sudo -u ig_root whoami
ig_root

My boss is concerned about unauthorized logins and tracking what servers the staff is connecting to. He has come up with this method that he wants me to try and implement. I don't want someone to do this for me, but I want to share the proposed method and see what may be available out there to do this.

The concept is simple... We would set up a "login server". All users would be connecting to an Ubuntu LINUX server, let's call it "central". The server has had oathtool installed to allow for two factor authentication, and we have the feature working. I enter login name "sample" and it prompts for the six digit code from Google Authenticator. Next it prompts for the personal password. If both are successful, user is allowed into the system.

Now here is where it gets a bit strange. From this central authentication PC, the company wants our users to then have to SSH to various other servers on our network. Example being you may have to "ssh dvlp_server" to do stuff in development, or jump on a customer server using "ssh acme-corp". But what they do NOT want is that any users be allowed to go straight to that destination server without first authorizing through the central login server. In other words, I can open a putty session and ssh to central as "steve@central" and then ssh to dvlp-server from there, but I should never be allowed to use putty and go straight to "steve@dvlp-server".

Is that even possible to prevent? Is there some sort of /etc/hosts.allow file that forces the SSH sessions to ONLY connect if they originated from a particular server or IP address?

The only way I am aware of at this time would be to add logic to the /etc/profile so that after you log in, it validates your hostname or IP (using who -m field) and kicks you back out if the name is not "central" or the IP is not "xxx.xxx.yyy.zzz". But that means I am only catching the login AFTER they've already logged on and then forcing them out. The intention was to not accept logins from that non-central server in the first place!

If anyone can point me in the right direction in terms of commands, links to better authentication methods, etc. I could really use that. I'm not sure that what my boss wants me to implement is really the best approach to this. At this time, we are not considering Active Directory or LDAP servers but maybe we should? We already have been using RSA SecurID 2FA on some of our servers. But I think they want an open-source solution.

Thanks in advance for any ideas on better approaches.

Steve

O/S: Ubuntu 18.04 LTS

I have a Dell PowerEdge R440 with a RAID controller that has been configured with a bunch of SSD drives on it. The RAID-5 array was set up at hardware level before Ubuntu was installed. I've been trying to figure out a way to examine the RAID array and have it show me the status of the array using LINUX commands, without having to boot to the BIOS. But I can't seem to find a simple screen that would tell me Disk X has failed and Disk Y is a hot spare, etc.

I tried using smartctl from the smartmontools package. It gives me details, but I have not figured out a way to get a simple summary showing all the disks and the current status for each one. Is there a combination of flags or options that would give me such a summary?

These commands worked so far for me:

# smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/bus/0 -d megaraid,0 # /dev/bus/0 [megaraid_disk_00], SCSI device
/dev/bus/0 -d megaraid,1 # /dev/bus/0 [megaraid_disk_01], SCSI device
/dev/bus/0 -d megaraid,2 # /dev/bus/0 [megaraid_disk_02], SCSI device
/dev/bus/0 -d megaraid,3 # /dev/bus/0 [megaraid_disk_03], SCSI device
/dev/bus/0 -d megaraid,4 # /dev/bus/0 [megaraid_disk_04], SCSI device
/dev/bus/0 -d megaraid,5 # /dev/bus/0 [megaraid_disk_05], SCSI device
/dev/bus/0 -d megaraid,6 # /dev/bus/0 [megaraid_disk_06], SCSI device
/dev/bus/0 -d megaraid,7 # /dev/bus/0 [megaraid_disk_07], SCSI device

These smartctl commands also work for me but I'm swamped with data that I'm not sure how to read:

# smartctl --all -d megaraid,1 /dev/sdb
# smartctl --xall -d megaraid,1 /dev/sdb

My background is with IBM AIX systems, and they have a handy diagnostics utility that can show you the status of a RAID array in a simple screen like this one:

------------------------------------------------------------------------ 
Name      Resource  State       Description              Size
------------------------------------------------------------------------ 
hdisk3    FC0100FF  Optimal     4K RAID 5 Array         1552GB  pdisk6
          000C08FF  Active      4K SSD Array Member        N/A  pdisk7  
          000C09FF  Active      4K SSD Array Member        N/A  pdisk8  
          000C0AFF  Active      4K SSD Array Member        N/A  pdisk9  
          000C0BFF  Active      4K SSD Array Member        N/A  pdisk10 
          000C0CFF  Active      4K SSD Array Member        N/A   
hdisk4    FC0200FF  Optimal     4K RAID 10 Array        1143GB  pdisk0  
          000C06FF  Active      4K Array Member            N/A  pdisk1  
          000C07FF  Active      4K Array Member            N/A

Is there some utility or tool that is available in Ubuntu that would be able to give a similar simple status report? The smartctl does not seem to give that (possibly because the devices don't support it?), when just asking for a health check:

# smartctl --health -d megaraid,1 /dev/sdb
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-39-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: ATA return descriptor not supported by controller fi
rmware
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

Or is the "PASSED" message shown above ultimately what I'm looking for?

Thanks for any advice.

Steve N.

I use lvcreate and mkfs to create new filesystems on my Ubuntu Server 18.04 LTS systems. One thing I find confusing is that the system does not automatically add any new filesystems that I create to the /etc/fstab file. The only ones added are the ones that were created when the system was initially created during install.

Is there any way (mkfs flag, or separate apt package) that new filesystems could be inserted automatically in the /etc/fstab file rather than requiring the administrator to manually edit the file?

I have a software package that (I am told) has to read the TZ environment variable, and if that variable is not set it can cause dates/times to be incorrect. Not sure if it is doing its own offset calculations or what. It used to work on an IBM AIX platform, and now it's going to run on Ubuntu Linux 18.04 LTS.

From what I understand, Ubuntu does not have the TZ variable preset to any value. It instead uses the time zone that it finds in either /etc/localtime or /etc/timezone (not sure which, or both?).

My question is this... If this package indeed needs TZ to not be null or blank, what should be the best approach to setting that value?

One person suggested that we put the TZ in /etc/profile, so it is set for every login. Another suggested that we should instead be reading it from some other file, such as just in the .profile for the users accessing that software. My suggestion was to put the export TZ=??? into the environment file that exists for the software, which gets executed every time the software is started.

Anyone see any issues with which option gets selected? One situation I can think of is that if we place it in the /etc/profile, it is like permanently overriding the default time zone setting for every account that logs on. This may not be a big deal as I think this system will be exclusively used with the software package that needs the TZ variable set. Just trying to see if there are any other caveats to consider. Things like how it might affect Java connections, that sort of thing.