Cerin's questions

I think something generated millions of files or folders in my /tmp on Ubuntu 18.04, as I'm unable to run ls /tmp or even rm -rf /tmp/*. So I tried rebooting, but now my machine's stuck at the step:

A start job is running for Create Volatile Files and Directories

I let it run for 10 hours, and it's still going, so it's likely swapping to disk and will never complete. What do I do? How do I clear my /tmp folder if I can't even boot the system?

Edit: I appended "systemd.debug-shell=1 systemd.show_status=false" so I could get to a shell. Then I killed the systemd-tmpfiles process and then ran mkdir /empty; time rsync -a --delete /empty/ /tmp/ to do a fast delete as suggested here. However, after 30 minutes, it's still running.

I'm trying to create an AWS auto-scaling group, and I want my instances to have a tag with Key=Name and Value=processor%i, where the "%i" is dynamically replaced with a 1, 2, 3, etc as the instances are added.

Is this possible? I can't find any documentation suggesting how to do this.

For some development servers, I want to make all the Apache log files accessible via the web so developers can more easily debug. I've figured out how to modify the Apache site config to make the default /var/log/apache2 directory accessible, but unless I manually chmod the directory to be accessible to Apache's www-data user, the files return the "Forbidden" error.

It looks like, by default on Ubuntu, Apache writes its logs with user root and group adm. How do I change this to user group www-data so web users can read them?

Googling this, some have suggested editing the value for APACHE_RUN_GROUP in /etc/apache2/envvars, but this is already set to www-data.

How do you run an SSH daemon on a different port other than the standard 22?

I screwed up something with my SSH configuration on a cloud VM, so that now SSH always prompts me for a password. My early attempts to fix this locked me out, forcing me to umount the disk, mount it inside a working VM, and fix the files to a known working state. However, it's still forcing me to enter a password.

I'd like to update my /etc/ssh/sshd_config and restart the main ssh daemon, which I can test against, while still having another one running, using the old configuration, that I can use to revert the configuration in case the first one locks me out. How would I do this, specifically on Ubuntu?

What's the best practice when creating A records that point to localhost? Is it considered taboo or are there some valid cases where it's permissible?

I have a dozen domains where I setup "local." subdomains that point to 127.0.0.1 to help when locally developing web applications linked to those domains. Myself and other developers on my team often have to work with multiple sites on our localhosts, and referring to IPs in the URL or keeping a local map of domains in your hosts file tends to cause confusion and gets difficult to maintain over time.

So I thought registering these as subdomains would be a quick and easy way to provide shortcuts for everyone. And it did. It worked for a few weeks, then it suddenly stopped working. So I logged into my provider's DNS controls, and sure enough, all the "local." A records had been deleted.

My host is Godaddy.

Before I go nuclear on Godaddy for modifying my DNS records without my permission, is there a legitimate reason why they would have deleted them? Might they have thought it was some sort of security issue or violation? Godaddy's just a terrible company in general, but switching DNS providers isn't a chore that's high on my list. Is this something that I should be legitimately angry about?

I have a script that dynamically creates Amazon EC2 instances and adds their ssh key to my ~/.ssh/known_hosts. However, whenever I need to refresh the instance by terminating and recreating it, I get obnoxious warning messages like:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
<fingerprint>.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in ~/.ssh/known_hosts:94
  remove with: ssh-keygen -f "~/.ssh/known_hosts" -R <hostname>
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

The warning is understandable, but even if I run that command to remove the old key, logging in still gives me the warning:

Warning: Permanently added '<hostname>' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for '<hostname>' differs from the key for the IP address '<ip>'
Offending key for IP in ~/.ssh/known_hosts:96

The solution is to manually remove that line in my known_hosts, but is there a way to automate this by running a single command beforehand?

I didn't select a VPC when creating a MySQL RDS database, so it created it inside an auto-generated "default" VPC. I'm now unable to add it to any of my pre-defined security groups, or access it from instances in my existing VPC. The obvious solution would be to relocate it to the proper VPC, but there doesn't seem to be any option to edit the VPC on the "Modify db instance" screen. Is there any way to select a different VPC, or is my only option to delete the db and re-create it inside the correct VPC?

How do you free up PostgreSQL connections that aren't properly being closed by client applications?

I have a data-mining app that spins up multi processes, all connection to a local PostgreSQL 9.1 database to retrieve data. It runs fine for a few hours, but then dies with the error:

FATAL:  remaining connection slots are reserved for non-replication superuser connections

Researching this shows that this is most likely caused by the app not properly closing it's connections. However, even with the app killed, these connections are never freed. Isn't there some sort of timeout where PostgreSQL will automatically close a connection?

I also tried increasing Postgres's max_connections from 100 to 200, but restarting gave me the error:

2014-02-23 10:51:15 EST FATAL:  could not create shared memory segment: Invalid argument
2014-02-23 10:51:15 EST DETAIL:  Failed system call was shmget(key=5432001, size=36954112, 03600).
2014-02-23 10:51:15 EST HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMMAX.  To reduce the request size (currently 36954112 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
    If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
    The PostgreSQL documentation contains more information about shared memory configuration.

My system is Ubuntu 12.04 and has 8GB of memory, and all other PG settings are defaults, so I'm not sure why it thinks the system doesn't have enough memory.

I then tried to use pgbouncer to pool and reuse connections. This seemed to work a little better, but even this eventually ran out of connections, giving me the error:

ERROR:  no more connections allowed

How do I further diagnose and fix this problem?

When buying an SSL certificate, what factors determine how many months/years you get it issued for? Is it arbitrary, or is one term length more appropriate in some circumstances?

I'm encountering a weird problem where a Fedora Linux VPS server reports 100% CPU, and effectively becomes unusable, but I don't know why because the high load prevents me from SSHing into it to see what's wrong.

How do I prioritize or configure SSH so that I'm still able to connect even if some process is consuming all other CPU?

How do you decrease or shrink the size of a KVM virtual machine disk?

I allocated a virtual disk of 500GB (stored at /var/lib/libvirt/images/vm1.img), and I'm finding that overkill, so now I'd like to free up some of that space for use with other virtual machines. There seems to be a lot answers on how to increase image storage, but not decrease it. I found the virt-resize tool, but it only seems to work with raw disk partitions, not disk images.

Edit: I'm using an LVM with an Ext4 formatted partition.

Edit: GParted screenshot showing my LVM parition layout. I need to do a lot more then just resize the filesystem. I know of no safe way to resize an LVM. And please don't tell me to use pvresize. Despite its name, it does not support resizing LVMs.

I did try sudo pvresize /dev/vda5, but it just says physical volume "/dev/vda5" changed but doesn't actually reduce the size.

I tried start parted to manually set the partition size (very dangerous), but doing print all just gives me the error "/dev/sr0 unrecognised disk label".

Edit: By following these instructions, I was able to successfully shrink both my logical and physical volumes (although I had to remember to activate and deactivate lvm before and after certain commands, which the instructions omit.

Now GParted is showing 250G of unallocated free space. How do I remove this from the KVM disk image and give it back to the underlying hypervisor?

I have a virtual machine that recently had its disk image increased from 20GB to 50GB, and fdisk -l verifies that the VM can see this new size. Now I need to resize my root LVM partition to fill the extra 30GB.

I've found several articles about resizing LVM, but the few that cover resizing the root partition all claim you need to boot from a LiveCD. Is there any way to do this without taking down the server? The server is critical, so I'd like to minimize downtime.

Edit: Output of fdisk -l:

[root@fedora-host ~]# sudo fdisk -l

Disk /dev/sda: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders, total 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00097c90

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048    83886079    41430016   8e  Linux LVM

Disk /dev/mapper/VolGroup-lv_root: 36.1 GB, 36104568832 bytes
255 heads, 63 sectors/track, 4389 cylinders, total 70516736 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup-lv_root doesn't contain a valid partition table

Disk /dev/mapper/VolGroup-lv_swap: 6308 MB, 6308233216 bytes
255 heads, 63 sectors/track, 766 cylinders, total 12320768 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup-lv_swap doesn't contain a valid partition table

Edit: How do I resize the physical partition? fdisk can see the free space, but I don't know how to resize primary LVM partition to use it. I tried booting into a LiveCD and using parted'd resize command, but all it gives me is the error "Unable to detect file system". I found this guide, which says I need to delete the partition and create a new one with the correct size, but that sounds very dangerous.

Final Edit: Parted's resize command is oddly unable to resize LVM partitions. Go figure. Instead, I simply deleted the old partition and created a new one with the new range, as outlined in the link above, and that correctly resized the LVM partition. I then followed the advice below to resize the volumes and filesystems inside the LVM partition.

How do I resize (specifically increase) the disk size of a Fedora guest VM running under VMWare ESXi 4.1?

I have a Fedora 16 VM with an lvm2 formatted disk, and I've increased its disk size using the vSphere client from 50GB to about 250GB. I rebooted the guest, and it correctly shows this size using fdisk -l /dev/sda. However, df -H still shows the old size.

I've found a few KB articles explaining how to resize partitions for some flavors of Linux, but nothing for Fedora with ext4. That article seems to imply I have to create a completely new partition, and that I can't simply expand the existing partition. Using Gparted, it also prevents me from simply resizing the existing partition. Is this impossible to do under Linux?

Edit: It turns out the drive is formatted as LVM2, which apparently most Linux tools, such as GParted, don't support.

Edit: The partition in logical volume I need to resize is the root partition. Is there any way to do this without a LiveCD?

How do you allow a specific user permission to write to an NFS partition?

I've mounted an NFS share on my localhost (a Fedora install), and I can read and write as root, but I'm unable to write as the apache user, even though all the files and directories in the share on my localhost and remote host are owned by apache.

For example, I've mounted it via this line in my /etc/fstab:

remotehost:/data/media     /data/media             nfs     _netdev,soft,intr,rw,bg        0 0

And both locations are owned by apache:

[root@remotehost ~]# ls -la /data
total 24
drwxr-xr-x.  6 root    root    4096 Jan  6  2011 .
dr-xr-xr-x. 28 root    root    4096 Oct 31  2011 ..
drwxr-xr-x   4 apache  apache  4096 Jan 14  2011 media

[root@localhost ~]# ls -la /data
total 16
drwxr-xr-x   4 apache apache 4096 Dec  7  2011 .
dr-xr-xr-x. 27 root   root   4096 Jun 11 15:51 ..
drwxrwxrwx   5 apache apache 4096 Jan 31  2011 media

However, when I try and write as the apache user, I get a "Permission denied" error.

[root@localhost ~]# sudo -u apache touch /data/media/test.txt'
touch: cannot touch `/data/media/test.txt': Permission denied

But of course it works fine as root. What am I doing wrong?

I have a small internal network of physical machines running hypervisors, which in turn run several KVM Ubuntu virtual machines. How would I setup an internal dynamic DNS server so that when I run a script to create a new virtual machine, that VM could automatically register itself in the DNS server?

Bind seems to be the standard DNS server for Linux, but it seems designed for a much more "static" DNS model. Dynamically updating this would require a complicated script that would have to SSH into the DNS server, edit configuration files, and then restart the server. This doesn't seem like a very elegant solution. Are there better options?

I saw a similar question, although they're asking for a solution for a public setting on Amazon. My servers are entirely private, and I don't want to rely on an external VM host or Dynamic DNS provider.

Is anyone aware of a tool for automating the creation and deployment of virtual machines across different hypervisor platforms? I work on web applications running on EC2, Xen, KVM, and VMWare, and I'm currently writing separate deployment scripts for each platform. Since there's a lot of redundancy, I'd like to find a single tool that can deploy to each platform.

I'm aware of tools that only work with a single platform, like Vagrant. The only one I've found that claims to work with nearly all VM platforms is VM Builder. How does it compare to each platform's native tools? Are there any other options like it?

How do I find out why Gmail is seemingly refusing connections from my specific IP range?

I'm getting a "Connection timeout" error when attempting to send email through smtp.gmail.com using the email facilities in Django running on a Linux machine on Amazon's EC2 service.

Oddly, I can send mail using sendmail, but it initially gets the same timeout error and then falls back to an alternative server, which accepts a connection, e.g.

~$ echo -e "Subject:test subject\ntest body\n" | sendmail -v myuser@mydomain
Loading lookup modules from /usr/lib/exim/4.72-1.fc13/lookups
Loaded 0 lookup modules
LOG: MAIN
 <= app@domain U=app P=local S=443
Loading lookup modules from
/usr/lib/exim/4.72-1.fc13/lookups
Loaded 0 lookup modules
delivering 1ReChV-0001Hx-Ss
Connecting to aspmx.l.google.com [74.125.93.26]:25 ... failed:
Connection timed out (timeout=5m)
LOG: MAIN
 aspmx.l.google.com [74.125.93.26] Connection timed out
Connecting to ALT1.aspmx.l.google.com [173.194.66.26]:25 ... connected

This is only occurring from Amazon's cloud. I can connect to smtp.gmail.com from my localhost using the same credentials, and send email just fine. And I can ping smtp.gmail.com just fine from everywhere, so it doesn't seem to be a routing or firewall issue.

Google lists no problems on their status dashboard.

The server in question sends very low volumes of email (10-20 a day), so I'd be surprised if it was flagged for spamming.

I'm trying to use pg_dump on my localhost to dump a 4GB PostgreSQL database on a remote machine. My localhost reports 15GB of space available. I'm piping the output to gzip. However, after about 15 minutes of processing, pg_dump aborts, stating "pg_dump: [tar archiver] could not write to output file: No space left on device". I continually monitor the amount of free disk space on my machine, and it always remains in the ~10GB range. Why is pg_dump failing prematurely due to low disk space, even though there's still plenty of space?

My command looks like:

pg_dump -c --host=${HOST} --username=${DATABASEUSER} --blobs --format=t ${DATABASE} | gzip -c > /tmp/db-backup.tar.gz

How do you delete all partitions on a device from the command line on Linux (specifically Ubuntu)? I tried looking at fdisk, but it presents an interactive prompt. I'm looking for a single command, which I can give a device path (e.g. /dev/sda) and it'll delete the ext4, linux-swap, and whatever other partitions it finds. Essentially, this would be the same thing as if I were to open GParted, and manually select and delete all partitions. This seems fairly simple, but unfortunately, I haven't been able to find anything through Google.

On a Linux server, how would I find a list of all current SSH connections, or disconnect a specific SSH connection?