JDS's questions

I am building a Kinesis Firehose Delivery Stream that will stream into Redshift.

This process has an S3 bucket as an intermediary.

The Cloudformation docs for AWS::KinesisFirehose::DeliveryStream state that two required directives are User and Password for a user with INSERT privs into the Redshift cluster.

From here: https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-redshift

User name An Amazon Redshift user with permissions to access the Amazon Redshift cluster. This user must have the Amazon Redshift INSERT permission for copying data from the S3 bucket to the Amazon Redshift cluster

Should I create this user via Cloudformation? For example, an IAM user? How would an IAM user be granted INSERT permissions here? Also, when spinning up a Firehose stream in the console, the implication from the UI is that this is just a SQL user created with CREATE USER syntax.

So do I need to specify the username and password in the CF stack template, and then use "CREATE USER WITH PASSWORD " SQL directly on the RS cluster after it is up and running?

Doing it this way strikes me as a dependency problem with the stack. (Although there are plenty of things that CF can't do naturally, such as create Route53 records or upload a file into S3, so maybe there's precedent for this)

I can't find any clarification on this and I can't find any CF stack examples that actually use Redshift as a Kinesis Firehose destination.

Environment

• EC2/T2
• Ubuntu 18.04

I have several instances in this environment; all are built off the same Ubuntu AMI.

Sometimes, some instances get stuck on the boot console with a message "Press enter for maintenance (Or press CTRL-D to Continue)"

I tried shutting down, detaching the root vol, and attaching it to a running recovery workstation, to fsck the volume. fsck comes up clean, no matter how much I try to force recovery.

The syslog is old -- no new log entries are added as of the most current boot attempt -- perhaps because the box never actually boots.

I'm at a loss of what else to try.

I know the "EC2 Best Practice" is to throw away and redeploy the instance, and that's fine, but I want to figure out what's actually causing this because it has happened several times, sporadically, and I'd like to prevent it if at all possible.

Change AMIs? Some other fsck-type fix to the root vol? Something else?

Preface: I am a Linux admin. I don't really "get" (or like) Windows.

I am in the process of remediating Windows 2016 servers with the CIS recommendations. It is mostly just creating a GPO set according to CIS specs, and applying it to the server(s) in question. I'm using Tenable's Nessus Audit Scanner to check the validity of the settings.

Here is where you can get the CIS spec I am using: https://www.cisecurity.org/benchmark/microsoft_windows_server/

(There isn't a direct download for it but it is free to download.)

Many of the exact details don't matter, so for this question I will focus on one specific example that I should be able to extrapolate to resolve the other problems. Broadly, the problem seems to be that I am trying to apply Registry edits via GPO and I guess I don't understand how to do that. However, the CIS guideline is very specific on the steps to remediate.

So, for example, I'm trying to apply CIS guideline 19.1.3.1, "Ensure 'Enable screen saver' is set to 'Enabled'"

The steps to do this are listed as:

To establish the recommended configuration via GP, set the following UI path to Enabled: User Configuration\Policies\Administrative Templates\Control Panel\Personalization\Enable screen saver

Ok, so, I did that. I know the GPO itself is applied because all the other GPO settings now show up on the server. Also, the Nessus Audit Scan now shows "OK" for most of the items I just applied.

The only items that don't seem to work are items that are Registry settings.

When I examine the Registry, I see the key I'm trying to set to a value isn't even there. For this example, that key is:

HKEY_USERS[USER SID]\SOFTWARE\Policies\Microsoft\Windows\Control Panel\Desktop:ScreenSaveActive

So, how do I get a Registry setting to show up via GPO?

Specifically, how do I make "User Configuration" Registry items show up?

I'll preface this with: I am a Linux admin. Windows to me is like me driving a UK car -- mostly operates the same but the steering wheel, buttons, and levers are in the wrong place and labels are spelled funny.

I have a server that is a domain member. There are GPO applied from the domain. Normal enough.

When I run auditpol on this server, I see policies that are set that are not set in secpol.msc and not set in the domain GPO. I also compared the list of applied GPO from running gpresult and found that there are only three GPO being applied. (This list of 3 GPO was the list I expected to see, so that was good).

Example:

Run on the member server:

PS C:\Windows\system32> .\auditpol.exe /get /category:\*
System audit policy
Category/Subcategory                      Setting
System
  Security System Extension               No Auditing
  System Integrity                        Success and Failure
  IPsec Driver                            No Auditing
  Other System Events                     Success and Failure
  Security State Change                   Success
Logon/Logoff
  Logon                                   Success and Failure
...(truncated)...

And

PS C:\Windows\system32> .\gpresult.exe /v /r /scope computer
...(truncated)...
RSOP data for CORP\fflintstone on MGMTWIN01A : Logging Mode
-----------------------------------------------------------

OS Configuration:            Member Server
OS Version:                  10.0.14393
Site Name:                   XYZ
Roaming Profile:             N/A
Local Profile:               C:\Users\fflintstone
Connected over a slow link?: No


COMPUTER SETTINGS
------------------
    CN=MGMTWIN01A,OU=Windows,OU=Servers,DC=corp,DC=example,DC=com
    Last time Group Policy was applied: 11/2/2018 at 2:13:01 PM
    Group Policy was applied from:      corpdc01a.corp.example.com
    Group Policy slow link threshold:   500 kbps
    Domain Name:                        CORP
    Domain Type:                        Windows 2008 or later
...(truncated)...
    Applied Group Policy Objects
    -----------------------------
        Default Domain Policy (CORP)
        Windows Allow RDP Access
        Windows Startup Script

    The following GPOs were not applied because they were filtered out
    -------------------------------------------------------------------
        Local Group Policy
            Filtering:  Not Applied (Empty)
...(truncated)...

The three GPO listed by gpreseult as "applied" all do not contain any of the settings listed as "Success" or "Success and Failure" in my example auditpol snippet.

Where are they being set? How can I track this down?

Preface: I am not a Windows admin. I am a Linux admin.

I have a Windows 2016 server with AD DNS that handles internal DNS forward and reverse lookups.

Somewhere, somehow, some process is automatically adding PTR reverse lookup records for CNAMEs. This is breaking our Kerberos authentication for SSH between servers, as the canonical lookup of a host that has CNAMEs will return too many FQDNs and Kerberos balks.

The Kerberos SSH problem was resolved when I removed the CNAME reverse lookups.

Then, the next day, presto! all the PTR -> CNAME entries were back in AD DNS

Example, from the top (I've changed the names to be generic, but the overall setup is the same):

A pair of network operations boxes that run SMTP and Squid proxy have A records

netops01.example.com -> 10.1.2.3
netops02.example.com -> 10.4.5.6

And this same box has CNAMEs

netops
proxy
mailserver

For some reason, there are reverse lookup entries in AD DNS like so:

3.2.1.10.in-addr.arpa. 3600 IN  PTR netops.example.com
3.2.1.10.in-addr.arpa. 3600 IN  PTR proxy.example.com
3.2.1.10.in-addr.arpa. 3600 IN  PTR mailserver.example.com
3.2.1.10.in-addr.arpa. 3600 IN  PTR netops01.example.com

The last entry is the A record for this host. All the others are CNAMEs to that A record + another A record for the second host.

Neither I or the other admin that handles ops created these. Nor did we set up any scheduled task to recreate reverse lookups. Nor can I imagine a good reason to have a reverse lookup point to CNAMEs or to multiple results. (Maybe there is a good reason? I've never done this before, though)

So, I deleted all the reverse lookups except the A record one. Kerberos SSH works!

Next day, the records all returned.

I'm not even sure how on Windows to troubleshoot (thus my preface at the top)

• How can I find in the Windows logs what performed this action?
• Is there something about Windows DNS that would automatically do this? (Create PTRs for CNAMEs)
• Is there a way to turn that off?
• Anything else I'm missing?

I have the following:

1. OpenSSL-generated, Self-signed Internal CA cert
2. OpenSSL-generated Internal-CA signed, wildcard cert

This cert protects our internal websites. e.g. "myservice.corp.example.com"

In this example, the wildcard cert has the following fields of note:

CN = ".corp.example.com" DNS = ".abc.corp.example.com" DNS = "*.xyz.corp.example.com"

(i.e. those last two are SANs)

To make the CA chain acceptable to the OS, I install the CA certs on workstations.

This works fine on macOS. "Fine" == "Green site-is-secure icon in browsers". I install the CA cert in the site-wide keychain, and set the Trust to "Always Trust"

However, on Windows 10, I install the CA cert via the Certificates snap-in (or by right-clicking on the cert -> Install). No matter what I do, I get the following error notices in the cert:

1. Certificate -> General -> "The integrity of this certificate cannot be guaranteed. The certificate may be corrupted or may have been altered."
2. Certificate -> Certification Path -> Certificate Status -> "This certificate has an invalid digital signature"

Finally, the RSA key is 2048 bits, and the signature algorithm on both the CA cert and the self-signed cert are sha256

I've scoured the web but can't find any resolution that helps me yet, but it appears it may be one of the following:

1. Issuer and Subject can't match. Or must match? not sure. (They do match in my CA cert)
2. Minimum key length not satisfied. (we are using 2048 bits, though)
3. something else?

My problem is I don't even know how to troubleshoot this on Windows. I'm a Linux admin. So, requesting troubleshooting steps and possible resolutions.

I've searched everywhere but can't find a definitive answer: Is there an official Ubuntu 14.04 AMI for HVM that runs on T2.medium? (Or T2.any, actually)

It looks like the answer is "no".

So the corollary question is: Why not? Why does Canonical limit Ubuntu from running on T2.medium?

Final corollary question: I found a community AMI that runs on T2.medium. Is this my only option? How do the community AMIs differ from the official?

Edit

I still get the experience as I've described, but it is inconsistent and buggy. I'm adding the details here in case it helps someone else. I've taken some screenshots.

1. Launch via the EC2 web UI, T2.MEDIUM and many other instance sizes are grayed out. I get here via EC2->Instances->Launch Instance (button)->Search for "ubuntu"->Choose "Ubuntu Server 14.04 LTS (HVM)"

2. Launch from Ubuntu Cloud Images directory (https://cloud-images.ubuntu.com/locator/), I can choose the same AMI ID by browsing the directory and clicking the AMI ID itself which is linked to the AWS EC2 Launch Instance Wizard.

Note: The AMI ID is the same in both URLs!

This should be super easy, I think I'm missing something obvious.

The OpenVPN docs state that you can run the management interface on a Unix domain socket. OK, no problem, I tried that.

openvpn --dev tun --management /dev/openvpn unix

This seems to work; device is created, and OpenVPN starts.

How does one connect to the management interface though? It isn't TCP so Netcat won't work. I tried echoing commands directly to the socket and got an error:

$ echo "help"| /dev/openvpn
bash: /dev/openvpn: No such device or address

I know I'm missing something basic, but I could find zero examples on the internet of anyone actually connecting to the management interface on a Unix domain socket.

Server: OpenVPN 2.3.8 on Ubuntu 14.04

How do I force a session to disconnect after a set time?

I want VPN sessions to last no longer than 24 hours. Currently, clients appear to be able to stay connected indefinitely.

I've looked at the --inactive param and that is pretty close to what I want, but I also want to force disconnect after 24 hours. i.e. VPN sessions should never last longer than 24 hours.

Edit OpenVPN Kill Script

I created a script to do this; it runs in an hourly cron job. I've posted it on Github here: https://github.com/poolpog/kill-openvpn-clients

This is for Ubuntu 14.04 and Centos 7.

I need to limit the number of users actively running as root. i.e. Logged in as root on the CLI.

Basically, I want only one user at a time to be able to run commands as root. The purpose here is auditing.

I looked into setting limits in /etc/security/limits.conf but the pam_limits.so module only seems to affect logins. Or login shells. Not sure. But whatever the specifics, it does prevent a user from SSHing to a box more than once, but does not prevent more than one user becoming root via "sudo su". Thus setting limits.conf can still allow more than one user being logged in as root at a time.

Here is the limits.conf line I tried that limits this:

root            hard    maxlogins            1

Next I tried limiting users in the @admins group. I figured that those users are the only ones allowed to sudo su to root (based on custom sudo rules we have in place).

@admins            hard    maxlogins            1

This seems to do what I want but seems clunky/wrong. Call it a gut feeling -- I don't quite have a handle on what I see as wrong about this one.

Finally, "Why?". Why do I have this requirement?

We are trying to implement controls to meet PCI-DSS 3.1 requirement 8.5 "Do not use group, shared, or generic IDs, passwords, or other authentication methods" -- emphasis on the "shared". In a Windows environment, you just grant users authorization to do whatever, and no one shares the primary Administrator account. Linux environments are designed such that for some things, you really want to be logged in as root. There must be a PCI-compliant way to solve this problem in a Linux environment.

Got this email from AWS, "Amazon EC2 Instance scheduled for retirement". If you use AWS heavily, you've probably seen this email. Here's the meat of it:

We have important news about your account. EC2 has detected degradation of the underlying hardware hosting one or more of your Amazon EC2 instances in the us-east-1 region. Due to this degradation, your instance(s) could already be unreachable. Running instances will be stopped or terminated after 12:00AM UTC on YYYY-MM-DD. The affected instances are listed below:

i-12345678

<...other stuff, not relevant to my question...>

What do you need to do?

If you can still access the instance, we recommend the following:

<...other stuff, not relevant to my question...>

• If your instance's root device is an EBS volume, you can replace the instance by creating an AMI of your instance, and launching a new instance from the AMI. For more information please see Amazon Machine Images... other stuff, not relevant to my question

The AWS docs, and the email itself, indicate I must create an AMI from the current instance to resolve this. I've heard, though, that simply stopping the instance and restarting it will migrate it to new hardware.

Is this true?

If simply stopping/starting won't solve this, is there a simple solution for this that isn't as clunky as creating an AMI and launching from that?

Using the aws-cli client (https://github.com/aws/aws-cli), is there a way to find instances using a date range filter? Or using an "earlier than X date" or "last X days" filter?

It seems that the only date-related filter is to specify an exact date, or a partial date with string wildcards. For example, I've found that specifying a date as follows works:

aws ec2 describe-instances --filters "Name=launch-time,Values=2015-03\*"

For example, that gets all instances launched in March, 2015.

What I want is equivalent to this POSIX "find" command, "find all things from last 30 days":

find . -mtime -30

All other things being equal, how would a storage array's IOPS performance change if one used larger disks.

For example, take an array with 10 X 100GB disks.

Measure IOPS for sequential 256kb block writes (or any IOPS metric)

Let's assume the resulting measured IOPS is 1000 IOPS.

Change the array for one with 10 X 200GB disks. Format with same RAID configuration, same block size, etc.

Would one expect the IOPS to remain the same, increase, or decrease? Would the change be roughly linear? i.e. increase by 2X or decrease by 2X (as I've increased the disk capacity by 2X)

Repeat these questions with 10 X 50GB disks.

Edit: More Context

This question resulted as a conversation among my Sysadmin team that is not well versed in all things storage. (Comfortable with many aspects of storage, but not the details of managing a SAN or whatever). We are receiving a big pile of new Netapp trays that have higher disk capacity per-disk -- double capacity -- than our existing trays. The comment came up that the IOPS of the new trays would be lower just because the disks were larger. Then a car analogy came up to explain this. Neither comment sat well with me so I wanted to run it out to The Team, i.e. Stack-Exchange-land.

The car analogy was something about two cars, with different acceleration, the same top speed, and running a quarter mile. Then change the distance to a half mile. Actually, I can't remember the exact analogy, but since I found another one on the interwebz that was similar I figured it was probably a common IOPS analogy.

In some ways, the actual answer to the question doesn't matter that much to me, as we are not using this information to evaluate a purchase. But we do need to evaluate the best way to attach the trays to an existing head, and best way to carve out aggregates and volumes.

VMWare tools not installing on Ubuntu 12.04. I'm using Chef to manage the installation, but the Apt commands fail if run manually.

I'm using the VMWare tool Debian repo. Example:

$ cat /etc/apt/sources.list.d/vmware-tools-source.list
deb http://packages.vmware.com/tools/esx/5.0u2/ubuntu precise main

When trying to install, most packages seem to go ok, but one, "vmware-tools-foundation", does not.

Example:

$ apt-get -q -y install vmware-tools-esx-nox=8.6.10-1.precise 
Reading package lists...
Building dependency tree...
Reading state information...
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
 vmware-tools-esx-kmods-3.2.0-23-generic : Depends: vmware-tools-foundation (>= 8.6.10) but it is not going to be installed
 vmware-tools-esx-nox : Depends: ...snip list of deps...
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
$ apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  vmware-tools-foundation
The following NEW packages will be installed:
  vmware-tools-foundation
0 upgraded, 1 newly installed, 0 to remove and 118 not upgraded.
7 not fully installed or removed.
Need to get 0 B/5,886 B of archives.
After this operation, 86.0 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
(Reading database ... 103499 files and directories currently installed.)
Unpacking vmware-tools-foundation (from .../vmware-tools-foundation_8.6.10-1.precise_all.deb) ...

VMware Tools cannot install because it appears that another installation of
VMware Tools is already present. Please remove the previous installation and
then attempt to install this copy of VMware Tools again.

dpkg: error processing /var/cache/apt/archives/vmware-tools-foundation_8.6.10-1.precise_all.deb (--unpack):
 subprocess new pre-installation script returned error exit status 1
Errors were encountered while processing:
 /var/cache/apt/archives/vmware-tools-foundation_8.6.10-1.precise_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

The key seems to be this error: "VMware Tools cannot install because it appears that another installation of VMware Tools is already present. Please remove the previous installation and then attempt to install this copy of VMware Tools again."

However, I've tryed removing and purging and can't seem to "trick" VMWare tools into thinking the packages are gone. Apt thinks they are gone.

Is there some service/file/cache/lock left that VMWare tools sees that makes it think that VMWare tools are still installed?

I've googled and googled but there is no answer to this question with my particular circumstances on the interwebs. VMWare's documentation of this error is minimal.

The Ohai docs are incomplete. Here's what I've been able to do so far:

• I've created a custom plugin that adds one piece of node data called "my_custom_data"
• it works when I load it manually in IRB
• I've used the Ohai cookbook to get it loaded on the servers that need it

However, Ohai doesn't load it, neither during Chef runs nor if I run Ohai manually.

The docs, here, are of little use in answering this question. http://docs.opscode.com/ohai.html

I've done some research into this and have not found a definitive answer.

We have a web app that is using the PHP+Memcache session handler.

I have several questions, all interrelated, but ultimately my issue is, "Why are PHP sessions apparently not expiring when we think they should be?" i.e. The end user should be logged out of the app after a set time, but is not.

Here are the dots, please help me connect them, and tell me where I am mistaken:

• My understanding is that Memcache expires keys based on the set time, in seconds (or unix timestamp for larger values).
• The expiration is lazy -- i.e. nothing is deleted in advance
• The PHP memecache session handler uses the sessions.gc_max_lifetime to set the memcache key expiration. idk, maybe it doesn't?
• Memcache should, on serving a requested key and seeing that it is expired, not serve it (and then maybe also delete it?). But at least not serve it.
• This act of not serving it should, to PHP, equate to a deleted session and the user being logged out.

Users are not being logged out.

How can I even debug this? Memcache isn't exactly transparent.

The example case that has not been working is a site with a session timeout set to two hours. An example user would last use the site at night, and then, 8 - 10 hours later, come back to the site and still be logged on.

PHP has several session settings that control the probability that garbage collection (i.e. deletion of expired sessions) will occur.

My question is: does this probability apply to all running Apache threads on all virtualhosts on any given Apache server?

For example:

• If the probability is set to 1/100, does that mean that any Apache thread that runs a PHP process goes into the pool of processes that may have gc run?
• or, does that apply only to the threads runnning for any given VirtualHost? Or other sort of partitioning?
• Finally, if a PHP process that triggers gc, does it perform gc on all PHP sessions? or just the sessions for that VirtualHost? Or other sort of partioning?

I'm trying to gauge wether I need to tune the gc_probability setting for a given VirtualHost, for the whole server, or what.

The suggestion found at these links works for me:

• How to setup ssh's umask for all type of connections
• http://ubuntuforums.org/showthread.php?t=1107974#5

Summary: Use PAM to inject the umask, using the following line in /etc/pam.d/sshd

session    optional     pam_umask.so umask=0027

However this only works for taking away permissiveness on the files/directories in question. i.e. I found it to work, but only to further restrict the umask.

For example, setting the umask to 0077 works.

However, increasing permissiveness, such as allowing default group write access, does not work.

There seems to be some underlying default umask that I cannot override.

I have tried changing the umask in the following places, as well:

• /etc/init.d/ssh => Doesn't work unless I upgrade to OpenSSH 5.4, which is not going to happen (there is an additional directive to set umask for the internal-sftp option in the newest OpenSSH)
• /etc/init/ssh.conf => didn't work
• /etc/login.defs => didn't work
• /etc/pam.d/sshd => didn't work
• /etc/profile=> didn't work. Profile is not hit by SFTP since it isn't an interactive shell
• /etc/ssh/sshd_config => didn't work

None worked. How can I allow more permissive masking for OpenSSH Chrooted SFTP?

Requirements:

• Works with configuration only (i.e. no patching, no distro upgrades)
• Works for Ubuntu 10.04LTS
• Works for OpenSSH_5.3p1 Debian-3ubuntu4, OpenSSL 0.9.8k 25 Mar 2009

According to the man page, xargs will quit if one of the execution lines exits with an error of 255:

If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens.

How can I get xargs to not do this?

I have a 1500 or so line batch job that I want to run, 50 lines at a time. I was finding that it was always dying at a certain line, and not completing the job. Not good!

An even better question, the question describing what I am trying to do, is:

How can I run a 1500 line batch script, 50 lines at a time, so that it does not quit the job in the middle, and so that the output is captured to a log file of some kind?

I've been using the term "Single Source" for authentication schemes that use a single authentication source (e.g. a single LDAP service) but are not Single Sign-on. i.e. You have to log on more than once, but you are using the same credentials.

Is there a commonly used technical term or acronym for "Single Source" authentication? I couldn't find one.