hookenz's questions

Nginx has a nice feature that enables you to restrict access to resources by source IP address.

However, I've noticed that may not work in situation where nginx is behind a proxy or load balancer. In my case the Amazon ELB.

I want to provide a status page for datadog agent and the Amazon ELB health checks. So I want to allow local connections and those from the ELB but deny everything else.

The following does not work as expected as it allows traffic originating outside the ELB to access the status page as well. The reason being because the IP will appear to originate from the ELB and the allow rules don't look at the headers to determine the real source of the request. Is there a way to tell nginx what source IP to use to determine allow/deny rules?

location /status {
    stub_status;
    access_log off;
    allow 127.0.0.1;
    allow 10.0.0.0/16;
    deny all;
}

Lets say I start with a certificate.

Using openssl I can print it out like this:

openssl x509 -in cert.pem -text -noout

And I'll get some output such as Validity, Issuer and Subject along with Authority Key Identifier and Subject Key Identifier.

How do I use these fields to work out the next certificate in the chain?

And then once I obtain the next certificate, work out what that next certificate should be etc.

Basically I'm wanting to work out the full chain and get things in the right order for the EC2 load balancer. Since Network Solutions don't seem to just give you a bundle that works. They give you individual certs and I've tried and tried lots of different orderings for EC2 and still haven't gotten it to work. My last bet is to try openssl and work this out manually rather than guessing.

I'm wanting to passthrough infiniband to a docker container so that I can run some high performance apps over ipoib and use rdma.

Currently, I'm doing this with xen virtual machines. Now I'm looking into using CoreOS and docker as a much lighter weight and easier to manage alternative.

I have an IPoIB device ib0 with a static IP assigned to it of 10.10.10.10. I've managed to get that exposed inside a docker container with the following:

docker run --net=host --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm -t -i ubuntu:14.04 /bin/bash

Great, that works. ib0 is available inside the docker container.

Now lets suppose I have a dual port HCA. On the host these appear as ib0 & ib1 and have two ip's assigned. 10.10.10.10/ib0 and 10.10.10.11/ib1

And now I want to pass ib0 to the first container and ib1 to the second. Using the method above both will appear in both containers because of the --net=host option. However, not specifying it means the devices do not appear at all.

Another scenario is I have a lot of machines that using SR-IOV to passthrough infiniband devices to xen virtual machines. How could I pass instead a virtual function infiniband device to a docker container and have that appear?

Note: pipework doesn't work in this situation but if I understand it better it might be able to be hacked to do what I want. I just don't quite understand what it's doing... yet.

I often hear this term used "We have a T1"... used on SF and other sites. I googled it and it seems like an ancient technology possibly related to frame relay but I'm not sure.

Maybe things have changed and the term means different things now.

What speed is a T1, do users get all 24 channels at 1.5Mbit? How does it relate to Frame Relay? something we used in my company over 15 years ago before ADSL became competitive.

T1 is not something offered in my part of the world, that's why I'm asking really.

I've spent several days on this now and I've managed to get SR-IOV working with the Mellanox Infiniband card using the latest firmware.

The Virtual Functions appear in Dom0 as

06:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] 06:00.2 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] 06:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function] 06:00.4 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

I've then detached 06:00.1 from Dom0 and assigned it to xen-pciback.

I've passed this into a Xen test domain.

lspci inside the test DomU shows:

00:01.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

I have the following modules loaded in DomU

mlx4_ib
rdma_ucm
ib_umad
ib_uverbs
ib_ipoib

dmesg output for mlx4 drivers shows:

[   11.956787] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
[   11.956789] mlx4_core: Initializing 0000:00:01.1
[   11.956859] mlx4_core 0000:00:01.1: enabling device (0000 -> 0002)
[   11.957242] mlx4_core 0000:00:01.1: Xen PCI mapped GSI0 to IRQ30
[   11.957581] mlx4_core 0000:00:01.1: Detected virtual function - running in slave mode
[   11.957606] mlx4_core 0000:00:01.1: Sending reset
[   11.957699] mlx4_core 0000:00:01.1: Sending vhcr0
[   11.976090] mlx4_core 0000:00:01.1: HCA minimum page size:512
[   11.976672] mlx4_core 0000:00:01.1: Timestamping is not supported in slave mode.
[   12.068079] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
[   12.184072] mlx4_core 0000:00:01.1: mlx4_ib: multi-function enabled
[   12.184075] mlx4_core 0000:00:01.1: mlx4_ib: operating in qp1 tunnel mode

I've even got the ib0 device appearing.

ib0       Link encap:UNSPEC  HWaddr 80-00-05-49-FE-80-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.10.10.10  Bcast:10.10.10.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:2044  Metric:1
          RX packets:117303 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256 
          RX bytes:6576132 (6.5 MB)  TX bytes:0 (0.0 B)

I can even locally ping 10.10.10.10.

However, those pings aren't sent out on the infiniband fabric.

It appears to be because the link is down. ibstat shows:

CA 'mlx4_0'
    CA type: MT4100
    Number of ports: 1
    Firmware version: 2.30.3000
    Hardware version: 0
    Node GUID: 0x001405005ef41f25
    System image GUID: 0x002590ffff175727
    Port 1:
        State: Down
        Physical state: LinkUp
        Rate: 10
        Base lid: 9
        LMC: 0
        SM lid: 1
        Capability mask: 0x02514868
        Port GUID: 0x0000000000000000

How do I get it UP? the domU link is UP but not the VF one?

And the answer is actually found here: According to this link: http://www.spinics.net/lists/linux-rdma/msg13307.html

What do I need for the slave VF's port to become active? I'm running opensm 3.3.13 on a different box, is that new enough? (does SR-IOV require any SM support?)

Yes, as Hal noted, at minimum you need opensm 3.3.14 (http://marc.info/?l=linux-rdma&m=133819320432335&w=2) as it is the 1st version to support alias-guid et al stuff needed for SRIOV, 3.3.15 is also out now, so you want the 2nd version that supports this... basically you need IB link for the PPF and the slave to get an alias guid registered for it @ the SM. We (IL team) were off Tues/Wed as of a holiday, will try to get you further details tonight and if not, by tomorrow, sure.

I've now upgraded OpenSM and will report back soon.

EDIT: OK, it's now working. However I'm getting a log blowout for opensm. The OpenSM process is writing hundreds of entries per second of the form:

Sep 30 20:36:26 707784 [7DC1700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 707810 [7DC1700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID
Sep 30 20:36:26 708096 [8DC3700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 708119 [8DC3700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID
Sep 30 20:36:26 708391 [FF5B0700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 708421 [FF5B0700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID
Sep 30 20:36:26 708696 [3DB9700] 0x01 -> validate_requested_mgid: ERR 1B01: Wrong MGID Prefix 0x8000 must be 0xFF
Sep 30 20:36:26 708719 [3DB9700] 0x01 -> mcmr_rcv_create_new_mgrp: ERR 1B22: Invalid requested MGID

And the above error messages went away when I rebooted and gave Dom0 more memory. I currently have 2GB allocated to it with autoballooning off. Unfortunately, they are back with no obvious reason. So I've asked a new question that relates to that here

I'm not really sure why it works in dom0 but in my case I have to have OpenSM running on the Dom0 which has the VF's. I presume this is because the OpenSM instance running on Dom0 knows about the VF's and can advertise them while a subnet manager on another node doesn't? that's my guess. I hope the other xen node will pick up it's VF's as well. That may end up becoming another question. For now it's working with a single Xen node.

I'm setting up my first Gluster 3.4 install and all is good up until I want to create a distributed replicated volume.

I have 4 servers 192.168.0.11, 192.168.0.12, 192.168.0.13 & 192.168.0.14.

From 192.168.0.11 I ran:

gluster peer probe 192.168.0.12
gluster peer probe 192.168.0.13
gluster peer probe 192.168.0.14

On each server I have a mounted storage volume at /export/brick1

I then ran on 192.168.0.11

gluster volume create gv0 replica2 192.168.0.11:/export/brick1 192.168.0.12:/export/brick1 192.168.0.13:/export/brick1 192.168.0.14:/export/brick1

But I get the error:

volume create: gv0: failed: Host 192.168.0.11 is not in 'Peer in Cluster' state

Sure enough if you run gluster peer status it shows 3 peers with the other connected hosts. i.e. Number of Peers: 3

Hostname: 192.168.0.12 Port: 24007 Uuid: bcea6044-f841-4465-88e4-f76a0c8d5198 State: Peer in Cluster (Connected)

Hostname: 192.168.0.13 Port: 24007 Uuid: 3b5c188e-9be8-4d0f-a7bd-b738a88f2199 State: Peer in Cluster (Connected)

Hostname: 192.168.0.14 Port: 24007 Uuid: f6f326eb-0181-4f99-8072-f27652dab064 State: Peer in Cluster (Connected)

But, from 192.168.0.12, the same command also shows 3 hosts and 192.168.0.11 is part of it. i.e.

Number of Peers: 3

Hostname: 192.168.0.11
Port: 24007
Uuid: 09a3bacb-558d-4257-8a85-ca8b56e219f2
State: Peer in Cluster (Connected)

Hostname: 192.168.0.13
Uuid: 3b5c188e-9be8-4d0f-a7bd-b738a88f2199
State: Peer in Cluster (Connected)

Hostname: 192.168.0.14
Uuid: f6f326eb-0181-4f99-8072-f27652dab064
State: Peer in Cluster (Connected)

So 192.168.0.11 is definitely part of the cluster.

The question is, why am I not able to create the volume on the first gluster server when running the gluster command. Is this normal behaviour or some sort of bug?

SSD drives have been around for several years now. But the issue of reliability still comes up.

I guess this is a follow up from this question posted 4 years ago, and last updated in 2011. It's now 2013, has much changed? I guess I'm looking for some real evidence, more than just a gut feel. Maybe you're using them in your DC. What's been your experience?

Reliability of ssd drives

UPDATE:

It's now 2016. I think the answer is probably yes (a pity they still cost more per GB though).

This report gives some evidence:

Flash Reliability in Production: The Expected and the Unexpected

And some interesting data on (consumer) mechanical drives:

Backblaze: Hard Drive Data and Stats

I have a remote client machine that is sending out DHCPDISCOVER's. The server is responding with a DHCPOFFER, but there is no DHCPACK.

This repeats about every 30 seconds from the same host. Is there something I can do remotely or do I need to get someone to reboot it? It's in a data centre so I may have to travel there to do it!

Thanks for the suggestions. I've had all the machines rebooted, but I still have issues. I think there is an issue with my configuration. Does this look correct?

#
# /etc/dhcpd.conf for primary DHCP server
#

authoritative;
ddns-update-style none;
deny duplicates;
default-lease-time 600;
max-lease-time 3600;

# Our fixed hosts
host host2  { hardware ethernet xx:xx:xx:xx:xx:xx; fixed-address x.x.x.202; }
host host3  { hardware ethernet xx:xx:xx:xx:xx:xx; fixed-address x.x.x.203; }
host host4  { hardware ethernet xx:xx:xx:xx:xx:xx; fixed-address x.x.x.204; }
host host5  { hardware ethernet xx:xx:xx:xx:xx:xx; fixed-address x.x.x.205; }

subnet x.x.x.128 netmask 255.255.255.128 {
  option subnet-mask 255.255.255.128;
  option broadcast-address x.x.x.255;
  option routers x.x.x.129;
  option domain-name-servers 8.8.8.8, 8.8.4.4;

  # Testing pool.
  pool {
    max-lease-time 300; # 5 minutes
    range x.x.x.250 x.x.x.254;
    deny known-clients;
  }

  # Our hosts - I didn't have this pool declaration before, do I need it if I want
  # the hosts to be running dhcp but always get the same address?
  pool {
    max-lease-time 1800;
    range x.x.x.200 x.x.x.220;
    deny unknown-clients;
  }
}

I have a linux router which has a 10GBe interface on the outside and bonded Gigabit ethernet interfaces on the inside.

We have currently budget for 2GBit/s. If we exceed that rate by more than 5% average for a month then we'll be charged for the whole 10Gbit/s capacity. Quite a step up in dollar terms.

So, I want to limit this to 2GBit/s on 10GBe interface.

TBF filter might be ideal, but this comment is of concern.

On all platforms except for Alpha, it is able to shape up to 1mbit/s of normal traffic with ideal minimal burstiness, sending out data exactly at the configured rates.

Should I be using TBF or some other filter to apply this rate to the interface and how would I do it. I don't understand the example given here: Traffic Control HOWTO

In particular "Example 9. Creating a 256kbit/s TBF"

tc qdisc add dev eth0 handle 1:0 root dsmark indices 1 default_index 0
tc qdisc add dev eth0 handle 2:0 parent 1:0 tbf burst 20480 limit 20480 mtu 1514 rate 32000bps

How is the 256K bit/s rate calculated? In this example, 32000bps = 32k bytes per second. Since tc uses bps = bytes per second. I guess burst and limit come into play but how would you go about choosing sensible numbers to reach the desired rate?

This is not a mistake. I tested this and it gave a rate close to 256K but not exactly that.

EDIT

After doing a lot of reading and testing, I've come to the conclusion that TBF is inappropriate because of the bandwidth involved. Whatever settings I tried I couldn't get TBF to provide bandwidth > ~50Mbit/s. According to lartc.org/lartc.pdf, the RED method is better for shaping bandwidth > 100Mbit/s so I will attempt to use that.

However, choosing the a value for min (i.e. Average queue size at which marking becomes a possibility). The example given is this:

You should set the min by calculating that highest acceptable base queueing latency you wish, and multiply it by your bandwidth. For instance, on my 64kbit/s ISDN link, I might want a base queueing latency of 200ms so I set min to 1600 bytes.

1. how would you choose the highest acceptable base queuing latency? The example is for 64kbit/s.

2. What would be acceptable for 2Gbit/s?

I'm after some clarification of the state/connection tracking in iptables.

• What is the difference between these two rules?

  iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

  iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

  Both appear to load the nf_conntrack module when either -m state or -m conntrack is specified. Both options turn on state or connection tracking.

Note: I am not asking what conntrack does, I'm asking just whether they are equivalent. I already know that the conntrack module has more features.

• If the above are equivalent, do you need to use the conntrack version when using conntrackd?

• Is connection tracking turned on when a packet is first matched containing -m state --state BLA , or is connection tracking always on for all traffic flows?

  e.g. Under FreeBSD PF you specify keepstate on a rule to track state. Is the same not true of netfilter? i.e. is it on for all flows as soon as the module is loaded?

• Can/Should connection tracking be used for fast matching like below? If not used like below, would it not mean that the firewall would step through the rule set again looking for a match for the packet rather that just hitting the first ESTABLISHED rule? [many examples do not seem to make use of that if true]

e.g. suppose this is some sort of router/firewall (no nat).

# Default DROP policy
iptables -P INPUT DROP
iptables -P OUTPUT DROP
iptables -P FORWARD DROP

# Drop invalid
iptables -A FORWARD -m state --state INVALID -j DROP

# Accept established,related connections
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow ssh through, track connection 
iptables -A FORWARD -p tcp --syn --dport 22 -m state --state NEW -j ACCEPT

• When the conntrack table fills, will your firewall start denying traffic, or will rules that have no state on them still work. In which case I should not DROP packets with INVALID state, correct?

See here: Shoot yourself in the foot with iptables and kmod auto-loading

I have a firewall/router (not doing NAT).

I've googled and seen conflicting answers. It seems UDP 500 is the common one. But the others are confusing. 1701, 4500.

And some say I need to also allow gre 50, or 47, or 50 & 51.

Ok, which ports are the correct ones for IPSec/L2TP to work in a routed environment without NAT? i.e. I want to use the built in windows client to connect to a VPN behind this router/firewall.

Perhaps a good answer here is to specify which ports to open for different situations. I think this would be useful for many people.

As I'm learning about iptables I've made a couple of mistakes and locked myself out.

What method(s) do you use to test rules without locking yourself out?

I'm using ubuntu server 12.04 LTS

All the answers below were helpful. In the end I used a combination of options. It also helps to have IPMI access to your remote server just in case! But ideally test the rules locally on a replicated environment and test that first. Vagrant helps in this regard to get test setups working quickly.

Under Linux you can combine multiple network interfaces into a "bonded" network interface to provide failover.

But there are several modes, some of which do not require switch support. I'm not constrained in my switch in that I can use any of the modes.

However, in reading about the different modes it's not immediately clear what the pros and cons of each one.

1. Do some modes provide a faster failover?
2. What about CPU load impact for each mode?
3. Which modes can combine the bandwidth rather than just provide redundancy?
4. Are there limitations to that?
5. Does balance-rr require switch support?
6. Reliability? What are your experiences running long term?

I'm building a failover firewall for a server cluster and started looking at the various options. I'm more familiar with carp on freebsd, but need to use linux for this project.

Searching google has produced several different projects, but no clear information about features they provide . CARP gave virtual interfaces that failover, I am not really clear on whether that's what corosync does, or is that what pacemaker does?

On the other hand I did get manage to get keepalived working. However, I noted that corosync provides native support for infiniband. This would be useful for me.

Perhaps someone could shed some light on the differences between:

1. corosync
2. keepalive
3. pacemaker
4. heartbeat

Which product would be the best fit for router failover?

EDIT: So I worked out a little more...

Pacemaker is the bigger project that can use Corosync & Pacemaker. It seems that Corosync & Heartbeat basically do the same thing. So you choose one or the other.

Heartbeat seems to be an older project but is still being worked on.

Keepalive on the other hand is an entirely different project and implements the VRRP protocol. It has lees features than the others. It appears to still be widely used but is missing recent documentation.

Unfortunately, for firewall/router failover there are very little examples. Has anyone found some nice howto's? I've found one written in Spanish.

Some years ago at my university, I recall that the labs there booted Windows NT over the network.

There was a shared drive for your own stuff and other than that any changes you did to the running OS were reset when you restarted the machine.

Now I'd like to be able to do the same thing with Windows 7.

I have found some how to's for this using iSCSI, but I don't want an iSCSI disk for every single PC, I want one image for multiple PC's. I've also found PXE Boot setup files for installing Windows locally, but that's not what I want either.

How would I go about setting up what I had at university but with Windows 7 as a OS to netboot?

i.e. How do I netboot Windows 7 images? I do not want to netboot a Windows 7 installer to a PC to install Windows locally, I want to run a Windows 7 image from memory/network.

I have some windows 7 virtual machines in xen that have devices showing up in "safely remove hardware".

I don't want users to ever be able to remove/eject any hardware at all. I'm told vmware has a hotplug option. xen doesn't seem to provide this for pci passthrough devices, therefore I'm looking for a reliable solution to prevent users from ejecting devices.

This issue is not necessarily related just to virtual machines but seems to be a common problem with devices that get wrongly reported as removable. I'm ideally looking for a way to prevent all devices from appearing or just prevent the safely remove hardware option from ever coming up.

I've tried setting device capabilities for specific devices on boot with a script but this for some reason doesn't always seem to work reliably.

Is there a way to prevent this icon from appearing in the notification area completely, either by registry key or group policy?

I should point out that setting this in group policy to "Administrators" did not seem to work.

[Computer Configuration>Windows Settings>Security Settings>Local Policies>Security Options>evices:Allowed to format and eject removable media]

Does NFS allow me to unlink/delete open files like I can do with local file systems?

The reason I ask is that I'm developing something that runs on multiple servers and may access NFS as a back end. It simplifies things somewhat if I can do an unlink on a file that is open.

This is a nice feature of the linux (unix?) filesystems that you can unlink a file that is open. It simply removes the directory entry. The file is still available to a process that holds it open as long as it is open. Once closed the inode is destroyed.

Does that same functionality hold true for files opened on NFS mounts?

Unfortunately, I have a database name with a dash in it. How do I grant access to that database as mysql reports a syntax error.

e.g.

GRANT SELECT,INSERT,UPDATE,DELETE ON astpp.* TO 'portal'@'localhost'
IDENTIFIED BY 'Ab7g12Xh35' WITH GRANT OPTION;

works, but

GRANT SELECT,INSERT,UPDATE,DELETE ON astpp-eth01.* TO 'portal'@'localhost'
IDENTIFIED BY 'Ab7g12Xh35' WITH GRANT OPTION;

Does not.

Neither does:

GRANT SELECT,INSERT,UPDATE,DELETE ON 'astpp-eth01'.* TO 'portal'@'localhost'
IDENTIFIED BY 'Ab7g12Xh35' WITH GRANT OPTION;

What is the difference between a Source NAT, Destination NAT and Masquerading?

For example, I thought IP Masqurading was what they used to call it in Linux? But what confuses me is that in our Astaro firewall there is IP Masquarading as well as NAT options. What's the difference between all these?

I'm reading around and it seems that 32 bit drivers do not work under 64 bit windows. Is this true? since 32-bit applications can run under 64 bit windows it seems ridiculous that 32-bit printer drivers cannot. Are printer drivers run at the kernel level?

Sounds like we're in for driver hell for our RDP environments.