Michael Martinez's questions

Does kubernetes capture - in the event log or elsewhere - when Secrets are deleted? I'm not seeing this showing up in the event log: kubectl get events --field-selector involvedObject.kind=Secret --watch and I'm not finding it in /var/log/pods/kube_system-kube-apiserver/either.

We're using something called external-secrets-operator to manage secrets, and we would like to be informed when a secret is deleted, as this isn't supposed to happen with ESO, so it's an edge case that we'd like to monitor for.

The man page for rngd on Linux says -o (/dev/random) is used for random number output while -r (/dev/hwrng) for random number input. What does this mean? How do these device files differ and why are they used differently? My understanding is the the kernel generates its own random sequences from hardware events and sticks them into /dev/random. So why would rngd need an "input" device?

By default, the way Grafana displays unaltered values (those that don't have any functions applied to them) is to display the arithmetic average of the values it retrieves from the data source. For example, when zoomed out to a five-minute granularity, Grafana averages the one minute values together to display a single value for that 5-minute period. Can this behavior be changed so that something like a maximum or a sum is displayed instead of the average?

Everyone knows that databases tend to do lots of small random I/O while big data things like Kafka tend to do large sequential I/O, but if I'm approaching this as a sysadmin without making assumptions, how do I determine what a running application is doing on my system, or in general whether my system is seeing? Without knowing how the application is written, how do I determine whether it is doing mostly sequential or random I/O, thereby making it easier for me to choose the right kind of disks, etc, to deploy?

I can use iostat to get the average request size (avgrq-rz) of each IOP and a count of the number of IOPS (r/s + w/s). How do I determine whether these are mostly sequential or random?

I'm seeing what seems to me to be contradictory results when looking at disk performance with dd versus iostat on two hosts (EC2 instances with an EBS drive). The hosts are identical except that one uses EXT4-formatted ebs and the other XFS-formatted ebs.

If I look at iostat, the EXT4 host seems to outperform the XFS host. TBoth are doing roughly the same write throughput (about 25MB/s) at about 100% utilization, but the EXT4 host on average has less await (lower disk latency). It's this smaller await that makes me say EXT4 is outperforming XFS:

EXT4:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00    11.00    0.00 6331.00     0.00    26.96     8.72    71.00   11.32   0.16  99.60

XFS:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00     2.00    0.00 6211.00     0.00    27.38     9.03   144.95   23.24   0.16 100.40

But, if using dd to measure performance, XFS is the clear winner, as it takes much less time to complete a fully synchronous write. The command is dd bs=1M count=256 if=/dev/zero of=./test conv=fdatasync:

EXT4:

2.8 MB/s

XFS:

24.0 MB/s

What would be the reason for EXT4 looking much better with iostat but looking much worse with dd?

UPDATE 5/25/2018: After running the hosts for a couple days, dd and sync now show equivalent response times for both ext4 and xfs. I suspect this has to do with a difference (if any?) in the way they handle something like sparse files. The first day the hosts were up they were both busy laying down a bunch of new files on the filesystem (this is a graphite carbon-cache application). This has settled down to where small updates are being written to these files, but new files are no longer being created and the total amount of used disk space is no longer increasing.

So, there must be something fundamentally different in the way XFS allocates new disk blocks versus EXT4. Any insight into what this could be, would be welcome.

i'm learning about queueing discipline as part of tuning the network stack on our carbon-c-relay cluster. My understanding is that the queueing discipline layer exists solely to lower latency and provide fairness to different types of network packets. Without this layer the driver queue would be used as a simple FIFO which would be fine for high network loads where latency is not a priority.

I'm running a carbon-c-relay cluster, which means I don't care about latency - I care about making sure all of the packets carrying metrics data are processed and that none are dropped. So I should be able to disable the queueing discipline layer and tune the driver queue instead (increase both the driver queue and the socket buffers to accomodate the network load.)

Is that correct? Is there a way to disable the queueing discipline layer?

We run an Apache Cassandra cluster where each host has a few hundred thousand files open at any given time.

We'd like to be able to get a count of open files at periodic intervals and feed this number into graphite, but when we run lsof under collectd, it ends up taking a few minutes to complete and chewing up an inordinate amount of CPU in the meantime.

I'm wondering if there's an alternate and more friendly means of getting the same data that lsof provides, or even a way of running lsof that won't eat into CPU as noticeably? (Although I assume this latter method would likely take much longer to complete than it currently does... not ideal).

Perhaps the kernel maintains some variable somewhere that contains the number of open files? Wishful thinking?

Update:

In reponse to one of the answers, we're already using the -b and -n flags. Here's the full command as I have it running under collectd:

sudo lsof -b -n -w | stdbuf -i0 -o0 -e0 wc -l

running aws ec2 describe-instances will return a json text similar to the following:

{
    "Reservations": [
        {
            "Instances": [
                  "PublicDnsName": "ec2..."
                    "VpcId": "vpc-...",
                        ...
            "Instances": [

I know for each "Instance" I can extract the contents of a single field, for example PublicDnsName, using jq as follows: jq '.Reservations[].Instances[].PublicDnsName' which will list the dns names for my instances But how do I extract two or more fields and separate them by a space or comma or something? I want PublicDnsName and VpcId to be listed side-by-side for each Instance.

Specifically what I'm looking for is a list of instances where VpcId is null, undefined, or non-existent. In other words I'd like a list of my Classic instances and I need this through api so I can process the results.

We all know that Linux/Unix will automatically block attempts to write to a file that currently open for writing by another process. But is it possible to block/lock read access to a file if another process has it open for writing? I've got two different scripts, both start at random times and run various times during the day: one script overwrites a particular file; the other one reads from that file. I want the second one to block(wait) or fail if the first one has the file open.

When you issue a salt call from salt master (eg. salt '*' test.ping), the output includes the minions that it is unable to talk to, with the hostname and then on the next line a message "Minion did not return."

Is there a way to to restrict the output only to those minions for which it is able to connect? I thought there was a way to do this, but I'm not seeing it in the man page or documentation.

According to the man page, the string to the right of =~ is treated as an extended regular expression. I've tried the following things to match an "ok" surrounded by whitespace on either side, but nothing is working. Any idea what I'm doing wrong? Maybe there's a shell option that needs to be enabled?

if ! [[ $RESULT =~ \s"ok"\s ]]; then

if ! [[ $RESULT =~ [:space:]"ok"[:space:] ]]; then

if ! [[ $RESULT =~ [ \t\r\n\v\f]"ok"[ \t\r\n\v\f] ]]; then

NOTE: self answered question below.

Is there a unix/linux utility, perhaps a shell builtin or an external command, that will limit the number of bytes placed into a bash variable? For example, I want to make sure STRING gets at most 1000 bytes (or some arbitrary number I choose) from the output of the curl (or other) command:

STRING=$(curl -s http://localhost:8010/status?)

Let me edit this to make clear that I'm just using curl as an example, but I'd like this to work with any command.

Most people configure PAM and NSS ldap modules to point to ldap servers that handles a single search base as defined in pam_ldap.conf or libnss-ldap.conf. The man pages do not describe any option to specify multiple bases.

Is it possible to configure a Linux system to use more than one base that are handled by different ldap servers, so that if a user is not found in the first domain, the system searches for the user in the next one, and keeps trying each different base until it finds the user or until it has exhausted all of them?

The case scenario that I'm working on is that I have a local LDAP server I set up serving base dc=yyyy and I've got a separate ldap directory I set up in jumpcloud with base ou=Users,o=<orgid>,dc=jumpcloud,dc=com. This directory has different set of users from the first. When a user logs in, I'd like the system to search first the local ldap directory; if it doesn't find the user there I'd like it to authenticate against the jumpcloud directory.

Would like to ask, particularly those who run large environments with lots of users, if there are any standardized- or industry-recommended ways to securely distribute the verification codes to remote users and/or force the user to rerun google-authenticator the very first time they log in via ssh to a Linux host that is configured with the libpam module requiring a user to enter the unix password plus the verification code every time they log in over ssh? Assuming the system administrator sets up the account, is there a way to either;

(a) not require google-authenticator the first time they log in, and force them to run it so that they can securely retrieve their verification code -or- (b) the system administrator runs it for them as part of creating the account, and securely (and easily, given there may be a lot of users) distributes the code plus the unix password to the user?

Note that a system admin can use the "chage" command to force a user to change their unix password the first time they log in.

Am interested in all practical solutions to this, particularly those that are implemented successfully in large environments. How do you get the two-factor codes to new users? Send it via txt message to their cell phones? Write it on pieces of paper and hand it to them? ....?

Is there some way to get the date that an lvm snapshot was created?

lvs --version LVM version: 2.02.66(2) (2010-05-20)

Neither lvdisplay nor lvs provides this information. Looking through the man pages on these two commands there are not any options to give it. For that matter, is there a way to get the creation date of any logical volume?

Here is what I see when I run lvdisplay:

~# id; lvdisplay vol0/m0000700.20150108 uid=0(root) gid=0(root) groups=0(root) --- Logical volume --- LV Name /dev/vol0/m0000700.20150108 VG Name vol0 LV UUID ShKkjV-tbL4-SjKv-dfA4-7fft-2Pl1-UKThSs LV Write Access read/write LV snapshot status active destination for /dev/vol0/m0000700 LV Status available # open 0 LV Size 109.05 GiB Current LE 27918 COW-table size 110.00 GiB COW-table LE 28160 Allocated to snapshot 26.12% Snapshot chunk size 4.00 KiB Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:28

I've got a Redhat box whose serial console (dev/ttyS0) is acting strangely. Firstly, it is echoing all input, including passwords typed at the login prompt. Secondly, running stty sane or other stty commands results in the following error:

stty: standard input: unable to perform all requested operations

When I press "Backspace" it gives me a ^H instead, and when I try to set this to erase via stty erase ^H it says: stty: missing argument to erase

so something is goofy and I'd like to be able to "reset" the serial console without rebooting. Any thoughts?

I would like to do the following in a Windows environment:

monitor a folder for the appearance of new file(s), when such files appear, move them into a different folder. Does not necessarily need to occur in real-time, but would like to keep it to within, say, 15 minutes of the appearance of the file.

In Linux, there is a kernel hook to monitor in realtime the appearance of new files in a directory, I don't know if something similar exists in Windows, or if that would even be a desirable solution. I'd like some suggestions on how to achieve this. I use rsync regularly in Linux; is it as reliable under Windows? More importantly, is rsync capable of handling a situation where a file has appeared but has not yet been completely written to - in this case can rsync (or other tool) wait until the file has been completely written before trying to move it?

I'm sure other sysadmins have had to deal with this type of issue, so rather than reinventing the wheel myself, I'd like to hear what others have done.

Linux bash (or other shell.) Have you ever fat fingered "echo $?" into something like "echo $4" thereby permanently preventing you from knowing the return code for the previous command? I do this often (and it's so annoying), so I figured I'll ask: does bash and/or linux have a "return code stack" or array or something that holds the return codes from previous commands, not just the last one that you ran? I'm talking about simple commands not involving pipes.

We have several applications which are generating their own plain-text log files, which I would like to forward to a remote syslog server for centralized logging. I do not have access to root on these machines, nor can I reconfigure syslog to redirect output to a remote machine.

I've found some solutions online, but they're mostly people's homemade bash scripts, and I'm looking for something more robust that is suitable for implementation in a potentially high-volume production environment.

Preferably something designed with an eye for a small footprint, background daemon that keeps running, that can keep up with a lot of lines, etc. - What solutions are currently available?

interesting little dilemma. This is Centos. Last night I logged into the physical console on a tty. I launched a command and let it run overnight. The tty is still logged in, but I am currently logged in remotely via ssh (I am not at the console.)

I'd like to know the exact command parameters I used (I don't remember), so I'd like to see the command I ran on that tty. Since the shell is still running it has not written anything to bash_history - the history for that instance is still in memory.

So, my question is if there is a way to remotely retrieve what I want, perhaps:

(a) sending a signal to the running shell, causing it to dump its bash history (b) examining the environment of the running shell to obtain its history information (c) examining the most recent x # of lines from the tty session so that I can see what I typed

or some other means ....