Amos Shapira's questions

I want to execute a script before starting a service every time the service is started.

I know that I can use ExecStartPre inside the service definition file, but I wonder if it's possible to achieve without editing a file which comes from the service's package.

The script is a simple command which exits when it's done, it creates files which are used by the main service (it fetches TLS certificates and keys from secure storage).

We hit an issue with one of our Kafka brokers (running over AWS EC2) suffering from many TCP retransmits.

We'd like to know which of the TCP connections it has open suffers from this.

Is there a way to break down the total count of TCP retransmits by socket, port or host (short of tcpdump)?

So far, I haven't found anything in the Linux docs or under /proc, or any external tools which provide this info.

The route53domains console can show list of pending domain transfers/purchases but I didn't find a way to get this information from the AWS CLI "route53domains" subcommand. Is this possible?

We use StackExchange blackbox as well as Hiera-eyaml-gpg to encrypt files in our Git repo using GPG keys.

I configured Git to decrypt files using Textconv configuration, so it automatically decrypts and compares both blackbox and .eyaml files when I execute commands like git show ..., git diff ... or git log -p ....

I also use GPG agent on my OSX, so it remembers the passphrase for a while.

The problem I hit is specifically with eyaml - if I execute git show HEAD..FETCH_HEAD and the agent needs to refresh the passphrase then it will automatically pop-up a nice TUI asking to re-type it. This works great if it was triggered via Blackbox but not by gpg-hiera-eyaml.

I'd like to find a way to tell gpg "now please ask for the passphrase if needed" directly from the command line, without having to trigger some blackbox command.

Is this possible?

Thanks.

EDIT: This is with GPG 1.4.20 on OSX.

EDIT: I tried preset'ing the passphrase with gpg-preset-passphrase -v -c $(gpg --list-secret-keys --with-fingerprint --with-colons | awk -F: '$1 == "fpr" { print $10 }') but it doesn't make a difference - git diff with eyaml files still tries to read the passphrase and fails.

I have less already displaying colored output (e.g. from Vagrant or Packer) by having the LESS environment variable set to -R.

What I'd like to be able to do now is to search for specific colors, e.g. ESC[1;31m for red.

I tried various combinations (e.g. \[1;31m) but it seems that less ignores them in its search when in -R mode. When I toggle -R off it finds them, though. This could make sense when you think of it because the entire sequence is considered as having "zero width" so there is nothing for less to show when it finds it.

Is there another trick I can use?

To recap - the aim is to keep viewing colored output but also be able to search the color escape sequence at the same time.

I'm looking for a way to see all our AWS account limits (e.g. maximum number of buckets in each region, maximum number of ELB's etc) in one location.

I found the page at http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html, but this lists only the default limits. It won't tell me our limits after some of them were raised upon our request.

I also found aws ec2 describe-account-attributes, for instance, but this gives only EC2-related limits.

We are running a couple of Puppet Master version 3.7 (Not PE) on AWS EC2 instances behind an Elastic Load Balancer (mainly for high availability).

Currently the load balancer's check is just TCP connect on port 8140. But this didn't detect that one of the instances was hung (it was still accepting the initial TCP connection but not doing anything with it).

I'm looking for way to actually send a meaningful "no-op" request to the puppet master and have it send back a result.

More specific requirements:

1. It shouldn't trigger heavy processing (the checks happen every few seconds)
2. It should succeed if the puppet master is able to actually handle "real request" and fail otherwise.
3. It should be possible to execute from ELB (i.e. basically it should return "200 OK" on successful https requests on a static path.

Is there a good static path I can give the health check to use? So far I haven't found any.

We are struggling with a common problem of trying to provide an Oracle 11g instance for our developers to test with. Our standard build agents are based on Ubuntu 12.04 but Oracle supports only RedHat based OS's.

There are some hacks about installing Oracle on Ubuntu floating around the net but they seem to be very fragile and a matter of hit-and-miss.

We want to avoid having to support an entire build server Puppet configuration on top of RedHat(/CentOS) just in order to support Oracle so I was beginning to wonder whether we can employ Docker to provide a RedHat-like environment required just for Oracle on top of our standard Ubuntu systems. The idea is that Oracle will run in its own container but the rest of the build agent will still run on standard Ubuntu, able to talk to Oracle over TCP.

I know what LXC's are and the difference between them and full-fledged virtualization like KVM/VirtualBox etc, but I was wondering whether it could still be possible.

Thanks.

We use graphite to track history of disk utilisation over time. Our alerting system looks at the data from graphite to alert us when the free space falls below a certain number of blocks.

I'd like to get smarter alerts - what I really care about is "how long do I have before I have to do something about the free space?", e.g. if the trend shows that in 7 days I'll run out of disk space then raise a Warning, if it's less than 2 days then raise an Error.

Graphite's standard dashboard interface can be pretty smart with derivatives and Holt Winters Confidence bands but so far I haven't found a way to convert this to actionable metrics. I'm also fine with crunching the numbers in other ways (just extract the raw numbers from graphite and run a script to do that).

One complication is that the graph is not smooth - files get added and removed but the general trend over time is for disk space usage to increase, so perhaps there is a need to look at local minimum's (if looking at the "disk free" metric) and draw a trend between the troughs.

Has anyone done this?

I have two identically configured CentOS 5 servers (possibly more in the future). I need to run a cron job on any one of them and that it'll run only on one of them.

I know about RedHat Cluster Suite (we use it on other servers), but it's a too big a gun to use for this task, plus it doesn't really behave well for less than three nodes.

Is there anything light-weight I can use for that?

The servers can communicate with each other directly. I suppose I can develope something over ssh or nrpe (two server which are already installed on these servers), but I was wondering whether there is something already available.

I've setup a couple of "at" jobs to execute at specific times on CentOS 5. This morning I don't see these commands in the queue ("atq"), as expected, but I also don't see were can I check whether the daemon responsible to execute them did this on time or had any issues. Unfortunately I didn't log enough info from the scripts themselves. Any ideas?

I'm shredding files on large ext3 file systems (e.g. see this article for how I do this). In order to achieve this I need to mount the ext3 file systems without journaling. To most of them I get by with just unmounting them and re-mounting them as ext2. But some file systems can't be unmounted (e.g. /) and attempts to use "-o remount,data=writeback" fails with errors.

Is it possible to turn off journaling of a mounted ext3 file system without having to unmount it?

We are creating a large backup ext3 partition on 16TB RAID array and the mke2fs took 2.5 hours to complete.

Are there any known tricks we can use to shorten this time?

The aim is to lower the time to get us back on our feet if a disaster strikes and we have to re-create that file system. I see options like lazy_bg and uninit_groups but they seem to be for ext4 file systems, which is not relevant for us yet.

I still see people recommend use of sync; sync; sync; sleep 30; halt incantations when talking about shutting down or rebooting Linux.

I've been running Linux since its inception and although this was the recommended procedure in the BSD 4.2/4.3 and SunOS 4 days, I can't recall that I had to do that for at least the last ten years, during which I probably went through shutdown/reboot of Linux maybe thousands of times.

I suspect that this is an anachronism since the days that the kernel couldn't unmount and sync the root filesystem and other critical filesystems required even during single-user mode (e.g. /tmp), and therefore it was necessary to tell it explicitly to flush as much data as it can to disk.

These days, without finding the relevant code in the kernel source yet (digging through http://lxr.linux.no and google), I suspect that the kernel is smart enough to cleanly unmount even the root filesystem and the filesystem is smart enough to effectively do a sync(2) before unmounting itself during a normal shutdown/reboot/poweorff.

The "sync; sync; sync" is only necessary in extreme cases where the filesystem won't unmount cleanly (e.g. physical disk failure) or the system is in a state that only forcing a direct reboot(8) will get it out of its freeze (e.g. the load is too high to let it schedule the shutdown command).

I also never do the sync procedure before unmounting removable devices, and never hit a problem.

Another example - Xen allows the DomU to be sent a shutdown command from the Dom0, this is considered a "clean shutdown" without anyone having to login and type the magical sync; sync; sync first.

Am I right or was I lucky for a few thousands of system shutdowns?

I currently use a server SSL certificate without a pass-phrase in order to allow Apache to start up unattended.

There are signs from customers to require us to protect the SSL certificate more securely. I'm not sure yet what they are aiming for, but for now I guess that they don't want an unprotected SSL certificate on the disk. I suppose I can't avoid having it in the clear inside Apache's memory but let's assume that's acceptable.

I came up with an elaborate system to keep the pass-phrase in the memory of a process on an internal server (i.e. not on the front-line web server) and hand it over to the front-line server using an Apache SSLPassPhraseDialog (http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslpassphrasedialog). The internal server will have to have the pass-phrase typed in to it when it starts, and we'll have multiple such servers load-balanced for high availability.

My question is:

1. How do the "big boys" protect their SSL certificate? Do they just force their stuff to type the pass phrase on server restart or do they keep it unencrypted like the rest of us?
2. My experience with open source is that there is a very good chance that someone has already solved whatever problem I face - is such a system available already?
3. Would it be reasonable, from business level perspective, to just say that we keep the certificate unencrypted and just have rapid procedures to revoke it if stolen?