Jim Dennis's questions

TL;DR Will OpenBSD policy based routing help with a multi-homed server/gateway situation? If so, how do I configure it?

Long Form

I'm managing an OpenBSD with two ISP links and VPN tunnels to remote routing nodes.

Initially we used multiple default routes with varying metrics -- the preferred route through a static IP address a NAT router which, in turn has dynamically allocated addresses (it's basically a cable modem).

In practice this was not ideal but it works well enough. New connections established from the gateway (hereinafter referred to simply as 'gw') would select the higher speed, lower latency route if it was up; and go out through the cable modem if the link was down. Inbound connection could only come through the better route since the other IP addresses were behind NAT (not routable from the outside.

Now we need to route traffic through an additional proxy/VPN router nodes out "in the cloud" to mitigate risks DDoS on our static IP addresses.

Those are connecting to the gateway via tunnels.

first. Then we found that our admin access would sporadically drop.

To complicate matters further this gateway has additional active interfaces to specific VLANs. They're unrelated to this problem but can't be disturbed.

Possible solution

It's my impression that we should be using policy based routing, rdomains. I guess that means I create routing tables for each of my three involved interfaces and any connection on any of those (including the tun0 tunnel interface) should be routed through the table for that domain (and thus each can have its own default route).

Am I on the right track?

Here's a diagram and a sanitized list if interface settings:

 ________
| tunnel |                       _______
 ~~~+~~~~                       | GW    |======++
    |                            ~+~+~+~       ||                   
    |      _________              | | |        ||                                        
    +-----| prefISP |-------------+ | |      __||____       .........               
           ~~~~~~~w~                | +-----| Switch |-----( Cluster )                           
                                    |        ~~~~~~~~       ^^^^^^^^^           
           _________           .....|......    ||                              
          | fallISP |---------( LAN / WiFi )===++
           ~~~~~~~~~           ^^^^^^^^^^^^

    Diagram: I want to avoid asymmetric routing when accessing GW through the tunnel, through                                           the preferred ISP, and when accessing GW or the cluster (through the GW or from the LAN).


 Sanitized interface info:

    em3:  inet 123.45.67.118 netmask 0xfffffff8 broadcast 123.45.67.119        description: prefISP
    em0:  inet 10.1.1.100    netmask 0xffffff00 broadcast 10.1.1.255           description: fallISP
    tun0: inet 192.168.2.2 --> 192.168.2.1 netmask 0xffffff00                  description: tunnel
    em1:  VLAN_TRUNK
          vlan1000: inet 172.29.1.1 netmask 0xffffff00 broadcast

As noted: em3 is our link to the preferred (faster) ISP; tun0 goes through it; em0 is on the same segment as the office LAN/Wifi and serves as our fallback ISP; and GW has additional links to the cluster and the switch.

So the latest configuration of our Nova compute nodes are using raw /dev/sdX devices (no labels nor partitions) as components for an md0 (raid0) array on which they're hosting an XFS filesystem. When one of the underlying hard disks fail, then the raid remains blissfully unaware of this.

This is confirmed by other cases like mdadm did not notice a failed disk in raid0

The question then arises. After we replace a failed hard disk, how do we reassemble this array without being forced to perform a new mkfs? Or would it be sufficient to fsck the filesystem and have it rediscover the (no-longer "bad" blocks)? Is that even a thing? (If the OS tries to use the blocks on the failed device I presume that the drivers have to simply return "bad blocks" for that entire range. Traditionally in Unix filesystems backblocks are forever ... you never try to reclaim them. Is there a switch to xfs_repair to force it to re-evaluate bad blocks?

Am I misunderstanding the underlying mechanics here?

The GNU screen utility has a neat feature for binding keys to screen commands or macros. The stuff command is particularly handy to expand some abbreviation to some other string as your typing.

For example if I'm in screen and I use my meta key ([Ctrl-A] by default) followed by :bindkey -t #@@ stuff "set -o vi; bind C-l:clear-screen C-i:complete" ... then ##@ becomes a quick way to set change settings of a bash shell to my preferences even when I using a shared account (for example when I've used sudo to get to the root shell on a server while trying to troubleshoot or fix it).

That's great and you can add those sorts of bindings into your ~/.screenrc file.

But if you want to make a macro for something more sensitive ... a password for example ... then you might want to add a key binding to your running screen session without storing the contents in any file, anywhere.

How do you do that?

You'd think it would be easy using the screen -X (capital X) switch which takes a command. But the obvious attempts to do that fail:

screen -X 'bindkey -t #@p stuff mysecretpasswordhere'

Then the screen status bar with briefly light up with an error like -X: unknown command ...

Why is that error message so unhelpful?

We have a number of script (for Openstack operations) which require privileged execution in our environment. For some of the scripts we'd like to refer back to the USERNAME or UID of the person who initiated the sudo session.

sudo already presents us with SUDO_UID, SUDO_USER environment settings. This also works fine when using sudo -s to start a shell session. However, if I use sudo -i or sudo su - then these settings are scrubbed from the environment.

Is there anything I can put into the /etc/sudoers or /etc/sudo.conf (perhaps even a "PLUGIN") which can allow me to start an interactive shell via sudo with root's environment initialized but ALSO with these specific SUDO_* settings preserved?

I have a domain hosted on my own linode under bind9 I also have a VPC in AWS and I want to maintain a DNS subdomain under Route53. I tried following the instructions at: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/MigratingSubdomain.html

Made the following changes to my /etc/bind9/named.conf:

  zone "aws.starshine.org" {
      type slave;
      file "/var/lib/bind/aws.starshine.org";
      masters { 205.251.197.214;
                205.251.195.5;
                205.251.198.215;
                205.251.192.111;
        };
  };

The IP addresses there were gathered from this:

 for i in "ns-1494.awsdns-58.org" "ns-773.awsdns-32.net" "ns-1751.awsdns-26.co.uk" "ns-111.awsdns-13.com"; do
     echo -en "$i\t"; dig +short "$i";
     done

... and those names were pasted from the output from this command:

aws route53 get-hosted-zone --id /hostedzone/Z24Z8xxxxxxxIN

If I run commands like: dig aws.starshine.org. @ns-111.awsdns-13.com I see the SOA record. If I add ns I see the Amazon NS records. But if I query through normal NDS or through my own authoritative DNS server for starshine.org I don't see the delegation.

Here's what I get from a couple of those dig commands:

dig aws.starshine.org @ns.starshine.org.

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> aws.starshine.org @ns.starshine.org.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 49466
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;aws.starshine.org.             IN      A

apogee:/var/lib/bind# dig aws.starshine.org

;; ...
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 41291
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;aws.starshine.org.             IN      A

;; AUTHORITY SECTION:
starshine.org.          200     IN      SOA     ns1.starshine.org. hostmaster.starshine.org. 2014091602 2000 1000 691200 600

I don't understand why I'm getting NXDOMAIN and SERVFAIL in these cases. I've completely restarted my BIND server processes (/etc/init.d/bind9 restart).

I see the following in my logs:

Nov 23 05:26:26 apogee named[1438]: zone aws.starshine.org/IN/internal-in: Transfer started.
Nov 23 05:26:27 apogee last message repeated 2 times

So, what am I doing wrong in my delegation? Do I need to enable something on the AWS Route53 side? It's showing me an SOA and NS records (and one A record that I've added and can query just find.

(Setting my resolv.conf (on my nodes in the VPC for example) to point at the AWS DNS name servers does allow me to see the subdomain as one would expect. (However that breaks all other DNS with messages about: Status: REFUSED and WARNING: recursion requested but not available.

I forgot to mention it in my earlier post, but I did also had IN NS "glue" records to my starshine.org zone file like so:

;; GLUE for aws.starshine.org hosted in AWS:
aws.starshine.org.      IN  NS  ns-1494.awsdns-58.org.
                        IN  NS  ns-773.awsdns-32.net.
                        IN  NS  ns-1751.awsdns-26.co.uk.
                        IN  NS  ns-111.awsdns-13.com.

ns-1494.awsdns-58.org.      IN A   205.251.197.214
ns-773.awsdns-32.net.       IN A   205.251.195.5
ns-1751.awsdns-26.co.uk.    IN A   205.251.198.215
ns-111.awsdns-13.com.       IN A   205.251.192.111

I also tried adding a list of forwarders to my named.conf:

zone "aws.starshine.org" {
    type forward;
    forwarders { 205.251.197.214;
                 205.251.195.5;
                 205.251.198.215;
                 205.251.192.111;
        };
    };

I'm probably not going to use this for the long term, but I still want to know why this isn't working.

I've set up a simple ssh tunnel from my Linode instance to one AWS EC2 instance within my VPC. I have three running instances, one has a public IP (not an elastic IP, just a public IP). My ifup command brings up the tunnel using something like:

manual tun0
iface tun0 inet static
    pre-up /usr/bin/ssh -i /root/.ssh/awsvpn -S /var/run/ssh-aws-vpn-control -NMfw 0:0 jump.aws.mydomain.org
    pre-up sleep 5
    address     192.168.11.2
    pointopoint 192.168.11.1
    broadcast   255.255.255.0
    up  route add -net 10.11.0.0 netmask 255.255.0.0 gw 192.168.11.2
    post-down /usr/bin/ssh -i /root/.ssh/awsvpn -S /var/run/ssh-aws-vpn-control -O exit jump.aws.mydomain.org

... run from the Linode to the awsgw (jump.aws.mydomain.org). The IP addresses are being resolved via /etc/hosts.

After this I can see my tunnel from both sides. I can ping to the far side of the tunnel (192.168.11.) and to the eth0 on the far side (10.11.0.xx on the AWS side, 123.45.67. on the Linode side). (Note: all address have been sanitized here). I can also ssh through the tunnel from either side.

The part that is NOT working is any attempt to reach the other two nodes within my VPC through that tunnel. I can ping and ssh from jump.aws.* to either of them (anode and bnode).

On the jump system I have done:

net.ipv4.ip_forward = 1

... and

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  10.0.0.0/8           0.0.0.0/0

... and on the other two nodes I've set this as my default router/gateway:

## From anode:
root@anode:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.11.0.35      0.0.0.0         UG    0      0        0 eth0
10.11.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.11.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0

On jump the routing table looks like this:

## From jump:
root@ip-10-11-0-35:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.11.0.1       0.0.0.0         UG    0      0        0 eth0
10.11.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
192.168.11.2    0.0.0.0         255.255.255.255 UH    0      0        0 tun0

... and, finally, on the Linode it looks like this:

## From linode
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         123.45.67.89    0.0.0.0         UG    0      0        0 eth0
10.11.0.0       0.0.0.0         255.255.0.0     U     0      0        0 tun0
192.168.11.1    0.0.0.0         255.255.255.255 UH    0      0        0 tun0
123.45.67.89    0.0.0.0         255.255.255.0   U     0      0        0 eth0

There are no other iptables rules currently active on any of these nodes. The only active rule(s) are on the jump box in my VPC. The AWS SG is allowing all traffic among the VPC nodes (and I've already established that I can reach anode and bnode from jump and the linode from there.

If I run tcpdump -n -v dst 10.11.0.39 (the anode) on jump (the NAT router I'm trying to configure) and try to ssh from the Linode to anode through it I do see the traffic hitting the jump/router. I see successful ARP traffic as well. But I don't see any traffic going out to the anode nor do I see any being received from a tcpdump on anode itself (other than the ssh between jump and anode).

For that matter the NAT/Masquerading isn't working at all. Regardless of the tunnel it's as if my default route on anode (and bnode) are being ignored.

Is this some artifact of Amazon's AWS VPC SDN (software defined networking)? How is it that I can configure a default route to point at one of my nodes ... a node I can ping and ssh to ... and my traffic isn't being routed to that node?

(All of the AWS EC2 nodes in this configuration are Debian Wheezy 7.1 AMI with "shellshock" patches ... and the VPC is in us-west-2 (Oregon)).

What am I missing?

I created an AWS VPC and three instances therein (head, nodeA, and nodeB). I also associated them all with a security group that allows fully open networking among the private (RFC-1918) addresses within my VPC block. In other words any of these three source address can touch any TCP or UDP port or use any ICMP protocol to any of the other addresses within the VPC netblock (10.x.0.0/16).

Networking among the three instances is working. Names are propated into /etc/hosts on each instance and those are working.

Now I want to use another address (10.x.0.100) as a custom VIP within that block, assigning it as an IP alias (eth0:0) to nodeA, running a service on that, then model breaking the service in various way and having nodeB take over that VIP (using tools like heartbeat and pacemaker.

I written a one-line takevip shell script which simply does ifconfig eth0:0 10.X.0.100 ... and executed it on nodeA (and ONLY one nodeA).

That's where I'm stuck. How do I register this address as in use within my virtual network segment?

I've tried ping -I10.x.0.100 10.x.0.1` (pinging the virtual router with my VIP as the source). That gets 100% packet loss. (Pinging .0.1 from all other instances works fine, as expected).

I've double and triple checked that my interface configuration (ifconfig -a) on nodeA is correct and that there's no conflict on nodeB (yet).

My boss recently asserted, in conversation, that Redis supports some configuration options for controlling maximum key or key/value sizes ... so we could set some option to prevent our applications from creating keys or key/value pairs larger than, say, 50KB.

My impression is that no such option exists and that we'd have to patch the sources and build our own to add such feature. (For this question forcing the applications programmers to mediate all access through Lua scripts or through something like twemproxy would NOT be an option).

Did I miss something in the Redis: Documentation somewhere?

Also what are the best practices for failover these days? Is Redis Sentinel ready for prime time? Is the Linux-HA OCF Heartbeat/Pacemaker/Cluster Glue trio still the best for this?

Is there a way to get network/bandwidth metrics from a given Redis process? Something I missed in the output of the INFO command? Some patch or compile time configuration option?

Or would I need to add some iptables packet counting commands and write something to poll those instead?

I'm working on creating a suite of self-training exercises and guides (perhaps eventually a collection of videos or Slideshare presentations) covering a range of systems administration and "ops" related topics.

My plan is to prototype and present them primarily assuming that the students will create AWS accounts and work through the examples using EC2 instances and a limited set of related resources. I'll recommend that students create one base instance using some less expensive VPS provider, such as Linode, as their "home base" for most of their work, then spin up instances (mostly t1.micro, where that's possible) during each exercise, and shut them down after each work session. (The goal is to make these feasible for students financing their own professional education ... so I'll be expending extra effort into practices and exercise session designs which minimize service costs).

The plan will mostly be using free AWS Marketplace AMIs for CentOS, Debian, Ubuntu, and FreeBSD as needed. (I'll also explore alternative clouds eventually, unless Amazon Inc. comes along to sponsor my project) --- though I might create some custom AMIs based on those and post them to the Marketplace as part of my offerings.

Early lessons will emphasize the use of Python Boto (it's what I'm comfortable with so far) and some tools I'm building will help manage this infrastructure that each student creates ... and I'll be having the students use Ansible throughout most of the exercises (for session set-up and tear down of collections of instances). One suite of lessons will be on Puppet, another will cover Chef. Most of the later lessons (Hadoop, Cassandra, MongoDB, Zookeeper) will (require/assume?) that the student is building out their clusters using one of these.

This, finally, brings me to my question. (Apologies for the rambling preamble, but it feels necessary to set the context).

Last night I set up the simplest Puppet (version 3) configuration that couple possibly be useful for a student of systems administration. A pair of CentOS 6.4 instances, one as the puppetmaster, the other as a client. I wired them together using the ugly ec*.internal names (in the hosts files and configs). Then made changes (creating user, group, and ssh key entries in my site.pp manifest) and confirmed that they were working (running the puppet agent --test command on the client). Naturally the biggest obstacles were: make changes to the security group configuration, disable the default CentOS IPTables configuration, disable the SELinux enforcement mode, and fiddle with the dns_alt_names = directive in the puppetmaster's /etc/puppet/puppet.conf. (Later I'll figure out how to properly grant the necessary SELinux permissions to the Ruby/Apache/Passenger components to re-enable SELinux).

Tonight, as I was about to pick up where I left off, it dawned on me (rather belatedly) that I really need stable IP and reverse DNS for this model. Having to reconfigure the puppet master and regenerates its PEM certificates, and those for all clients, at he beginning of each work session would be more tedious than educational (and scripting all that would have very little applicability to any normal future work environment.

So, duh, need to either allocate an Elastic IP (at least one) and request a reverse DNS entry for it ... or I need to re-work my assumption and base these exercises on a pre-requisite set of VPC set-up lessons.

For now I've just setup the EIP and entered a request for the PTR over-ride (which, by the way, seems to be conflate in Amazon's workflow with some STMP enablement request form).

I'm looking for suggestions on whether I should do this around VPC (and predicate the lessons on setting up a DNS server within that virtual network, perhaps hosted on the puppetmaster itself, at least initially) or whether I should build the lessons around using one or two generically named Elastic IPs (and possibly later segue into migrating the puppetmaster services behind HAProxy or some similar front end, as a different exercise).

Suggestions? Criticisms?

I'm starting a new project and considering using Ansible or Salt for deployment automation and, perhaps, more sophisticated orchestration (server management and federation).

With Salt I'm wondering if there's any integration between it and Graphite or Zenoss or Ganglia ... using the Salt 0mq connections to relay the data from the Salt "minions" to the monitoring/graphing database/collectors.

Has anyone else looked at this?

I get the impression that one can, potentially, have multiple JobTracker nodes configured to share the same set of MR (TaskTracker) nodes. I know that, conventionally, all the nodes in a Hadoop cluster should have the same set of configuration files (conventionally under /etc/hadoop/conf/ --- at least for the Cloudera Distribution of Hadoop (CDH). Can we define multiple Job Trackers in mapred-site.xml? Something like:

<configuration>
   <property>
     <name>mapred.job.tracker</name>
     <value>jt01.mydomain.not:8021</value>
   </property>
   <property>
     <name>mapred.job.tracker</name>
     <value>jt02.mydomain.not:8021</value>
   </property>
...
</configuration>

Or is there some other allowed syntax for this?

What are the implications of doing this. Does each JobTracker get information about the load on each TaskTracker node. In other words can the two JobTracker co-ordinated their scheduling across the TT nodes only based on the gossip information from the TTs or would they need to talk to one another?

Is this documented anywhere?