Yarik Dot's questions

Our setup: 1 application, tens of instances. Each instance is one of Baremetal/KVM or vmware VM/LXC container (depending on required size).

We update all instances via puppet (masterless), which is executed by mcollective. This works great.

The issue is high hard drive load, when puppet is executed on all instances at one time,

When we execute mco rpc puppetrun run, puppet is immediately executed on all machines in the cluster, which leads to hard drive overload due to unpacking deb packages.

We would like to run mcollective sequentially on limited number of servers at the same time.

We have updated the mcollective execution to to: mco rpc puppetrun run --batch XXX --batch-sleep YYY which helps, but it is not ideal as sometimes we deploy small changes, which are processed within a second and sometimes large changes with database migrations, which takes tens of seconds to perform.

We will be happy for a hint of how to approach this issue.

And also we would like to skip all solutions requiring external inventory as number of instances changes dynamically. Mcollective is great in this way.

Problem:

When having VLAN without IPv6 address on L3 enabled switch, no RA is passed through VLAN unless there is IPv6 address assigned to the VLAN. Static assigments however are reachable fine.

Switch(config)#interface Vlan 3
  no ipv6 address autoconfig # RA from mikrotik not working
  ipv6 address autoconfig # RA from mikrotik suddenly starts working
  no ipv6 address autoconfig # IPMI immediately losts SLAAC IPv6 address and become unresponsive

Topology:

-> Uplink -> Cisco (L3 routing to VLANs) -> Servers

I receive non routed /27 IPv4 and /48 IPv6 routed via single /64 on uplink (vlan 1). Cisco therefore does IPv6 routing to VLANs, where each VLAN is assigned unique /64 range with RA enabled. Everyone who needs IPv4 is assigned second port with (mode access + vlan 1).

VLAN 3 is however special as it is delegated through mikrotik (firewall) and the same cisco is then used as L2 switch (access vlan 3). In that case traffic passes cisco twice.

-> Uplink -> Cisco -> (Gi0/2 VLAN 2 -> ether1) Mikrotik (ether2 -> Gi0/3 VLAN 3) -> Cisco -> IPMI

IPMI are then connected to VLAN 3 and delegated IPv6 is expected to be managed via mikrotik.

When we connect IPMI directly to mikrotik, it works without any issue.

show ver

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 52    WS-C3560G-48TS     15.0(2)SE4            C3560-IPSERVICESK9-M

show running-config

version 15.0
ipv6 unicast-routing
interface GigabitEthernet0/1  # uplink port
!
interface GigabitEthernet0/2  # ether1 to mikrotik
 switchport access vlan 2
 switchport mode access
!
interface GigabitEthernet0/3  # ether2 from mikrotik, which manages IPv6 RA
 switchport access vlan 3
 switchport mode access
!
....
interface GigabitEthernet0/27 # Example of server uplink
 switchport access vlan 10
 switchport mode access
!
interface GigabitEthernet0/28 # Example of IPMI uplink
 switchport access vlan 3
 switchport mode access
!
interface GigabitEthernet0/41 # Example of port with IPv4
 switchport access vlan 1
 switchport mode access
!
...
interface Vlan1 # Uplink VLAN
 no ip address
 ipv6 address fc00:0:0:ffff::1/64
 ipv6 nd ra suppress all
!
interface Vlan2 # Mikrotik prefix delegation network
 no ip address
 ipv6 address fc00:0:0:1::1/64
 ipv6 nd ra suppress all
!
interface Vlan3 # IPMI VLAN - IP addresses are managed by mikrotik
 no ip address
!
interface Vlan10 # VLAN for servers
 no ip address
 ipv6 address fc00:0:0:10::1/64
 ipv6 nd managed-config-flag
 ipv6 nd other-config-flag
 ipv6 dhcp relay destination .......
!
...
ipv6 route fc00:0:0:3::/64 fc00:0:0:1::2 # Prefix delegation to mikrotik
ipv6 route ::/0 fc00:ffff::2

ipv6 address autoconfig

show sdm prefer

The current template is "desktop IPv4 and IPv6 routing" template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

 number of unicast mac addresses:                  1.5K
 number of IPv4 IGMP groups + multicast routes:    1K
 number of IPv4 unicast routes:                    2.75K
   number of directly-connected IPv4 hosts:        1.5K
   number of indirect IPv4 routes:                 1.25K
 number of IPv6 multicast groups:                  1.125k
 number of directly-connected IPv6 addresses:      1.5K
 number of indirect IPv6 unicast routes:           1.25K
 number of IPv4 policy based routing aces:         0.25K
 number of IPv4/MAC qos aces:                      0.5K
 number of IPv4/MAC security aces:                 0.5K
 number of IPv6 policy based routing aces:         0.25K
 number of IPv6 qos aces:                          0.625k
 number of IPv6 security aces:                     0.5K

I would like to find out the best possible configuration/hardware to deliver 40Gbps from a single server in this question.

Situation

We have a video share proxy server that offloads peaks from slow storage servers behind it. All traffic is HTTP only. The server acts as a reverse proxy (files that are not cached on the server) and a webserver (files that are stored on local drives).

There are currently something like 100TB of files and growing on the backend storage servers.

The caching mechanism is implemented independently and this question is not about caching itself as it works very well - currently delivers 14Gbps, passes to the backend servers only 2Gbps. So the cache usage is good.

Goal

Achieve 40Gbps or even more throughput from a single machine.

Hardware 1

HW: Supermicro SC825, X11SSL-F, Xeon E3-1230v5 (4C/[email protected]), 16GB DDR4 RAM, 2x Supermicro 10G STGN-i1S (LACP L3+4)

SSD: 1x 512GB Samsung, 2x 500GB Samsung, 2x480GB Intel 535, 1x 240GB Intel S3500

System:

• irqbalancer stopped
• set_irq_affinity for each interface (via script in ixgbe driver tarball)
• ixgbe-4.3.15
• I/O scheduler deadline
• iptables empty (unloaded modules)
• Filesystem: XFS

Nginx:

• sendfile off
• aio threads
• directio 1M
• tcp_nopush on
• tcp_nodelay on

As seen on the graphs, we were able to push 12.5Gbps. Unfortunately the server was unresponsive.

There are 2 things that took my attention. The first one is high amount of IRQ. In this case I don't unfortunately have graphs from /proc/interrupts. The second thing was high system load, which I think was caused by kswapd0 having problems to work with 16G of RAM only.

Hardware 2

HW: Supermicro SC119TQ, X10DRW-i, 2x Xeon E5-2609v4 (8C/[email protected]), 128GB DDR4 RAM, 2x Supermicro 10G STGN-i1S

SSD, System configuration are the same as hardware 1. Nginx is sendfile on (aio/sendfile compared further).

This seems better, so now as we a have server, which works in peaks, we can try some optimizations.

Sendfile vs aio threads

I tried to disable sendfile and use aio threads instead.

• sendfile off
• aio threads
• directio 1M (which matches all files we have)

vs

• sendfile on

Then at 15:00 I switched back to sendfile and reloaded nginx (so it took a while to finish existing connections). It is nice that the drive utilization (measured by iostat) went down. Nothing has changed on the traffic (unfortunately zabbix decided not to collect the data from bond0).

sendfile on/off

Just tried to switch send on/off. Nothing has changed except Rescheduling interrupts.

irqbalancer as a server/cron/disabled

As @lsd mentioned I tried to setup irqbalancer to be executed via cron:

*/5 * * * *   root    /usr/sbin/irqbalance --oneshot --debug 3 > /dev/null

Unfortunately it didn't help in my case. One of the network cards started behaving strange:

I couldn't find what was wrong in graphs and as it happened the next day again, I logged in to the server and saw that one core was at 100% (system usage).

I tried to start irqbalance as a service, the result was still the same.

Then I decided to use the set_irq_affinity script and it fixed the problem immediately and server pushed 17Gbps again.

Hardware 3

We did upgrade to new hardware: 2U 24 (+2) drives chassis (6xSFF), 2x Xeon E5-2620v4, 64GB DDR4 RAM (4x16GB modules), 13x SSD, 2x Supermicro (with Intel chip) network cards. New CPUs improved the performance a lot.

Current setup remains - sendfile, etc. Only difference is that we let only a single CPU handle both network cards (via set_irq_affinity script).

20Gbps limit has been reached.

Next goal? 30Gbps.

Feel free to shoot at me ideas how to improve the performance. I will be happy to test it live and share some heavy graphs in here.

Any ideas how to deal with large amount of SoftIRQs on the cpu?

This is not a question about capacity planning - I already have the hardware and the traffic. I can always split the traffic to several servers (which I will have to do in the future anyway) and fix the problem with money. This is however a question about system optimization and performance tunning in a real live scenario.

We are currently preparing for IPv6 only network deployment and struggle to find a unique server identification.

In our previous setup, we used mac address as servers primary identification. We were then able to assign the server always the same address.

E.g. we needed to perform upgrade from wheezy to jessie and rather then apt-get dist-upgrade, we reinstalled the server. As we use puppet for configuration, it all took 15 minutes and server was up, fresh and perfectly running again with the same IP and DNS name.

Unfortunately, we are currently not able to achieve this with IPv6 as the server with every OS (Linux, PXEBoot kernel, ...) has different DUID, which means for every OS different IPv6 address.

In our setup, we use mikrotik as DHCP server.

How would you manage situation like this?

Update #1: It seems it is currently not possible:

• https://www.insinuator.net/2015/02/is-rfc-6939-support-finally-here-checking-the-implementation-of-the-client-link-layer-address-option-in-dhcpv6/
• http://blog.bronislavrobenek.com/post/64383333742/ipv6-security-in-enterprise-network

I was able to find a patch (http://alter.org.ua/soft/fbsd/isc_dhcpd/) for isc-dhcp-server that fixes this, but it is not global solution and I can't apply that on cisco or mikrotik devices.

Update #2: We have decided to leave mikrotik platform as we outgrown the capabilities of this platform. We are currently preparing new "infrastructure" server based on linux, that will run https://dhcpy6d.ifw-dresden.de/ (besides other services such as openvpn for ipmi, ...).

I have nginx server serving a lot of big files. I can see plenty of connections on the server. As it iw know, file to the accesslog is written after the request is completed.

Is there any way to log request before it is served? The problem is that it takes hours to download large files and I would like to know which files were hit.

In our infrastrcture, we have servers installed and managed by puppet. All our software is then installed from rpm. On each server is a small script that performs yum update if there is some change in the repository.

What I need to achieve is to perform automatic downgrade as well, so the package installed on all servers is the same as latest in the repository. We would like to perform rollbacks this way. So developers have some simple (web is a must) interface to move package live/remove package (perform rollback/downgrade).

I know debian supports this feature via priorities (or some speicific apt magick - I am not sure now, it's been a while when I configured that last time).

Is it possible to manage yum to perform this way?

I am trying to create automatic provisioning of my servers using pxeboot and netinstall. Everything works ok (RAID, LVM, ...), but I would need to create tmpfs for some directories.

I would like to use great option of partman - create partition that is as big as x% of memory. Therefore, for each server no matter how much memory it has, I'd like to use 20% as tmpfs.

Is that possible or I will have to create custom script and execute it as late_command?

We are currently having perfectly working IPv4 network with a typical layout - internal network with all servers splitted to VLANs in 10.0.0.0/8 (each vlan is /24) and external network with public ip addresses. IPv4 addresses are NATed to the internal network via 2 redundant linux boxes.

We got /48 IPv6 range and gateway (nothing more) from our ISP. I am able to configure static IPv6 on my firewall to test IPv6 connectivity, which works. However, I would like to split that /48 IPv6 range to smaller groups and advertise them further. I can imagine that I end up with configuration where each VLAN will have:

• 1x /24 IPv4 internal IP range within 10.0.0.0/8
• 1x /64 IPv6 external IP range within our public IP range

With IPv6, I would like to stop using IPv4, stop using internal IPv4 network and keep IPv4 addresses only on services that need to keep the compatibility. I want to avoid of NAT in the future.

The question is, what is the best practice to do this on linux?

Updated question

Is there any way, I can tell my provider next hop to pass the /64 ranges through? Does IPv6 support this? Or I need to ask him to set up a static route to my gateways?

I am going to install around 100 mysql servers and I would like to have different root password on every of them.

I am going to use puppet mysql (https://forge.puppetlabs.com/puppetlabs/mysql) which is amazing. Unfortunately, it supports only passing password from puppet. The great thing is that it stores password in /root/.my.cnf file, so I can connect to mysql locally without knowing the password. So that I don't see the need to have password stored in puppet anymore.

Is there any way to generate password on the server so I don't need to store it on my puppet master?

We are currently having ~100 servers and all of them are using remote syslog, so we have aggregated all logs on one server.

The most questioned problem from our support team is: Has an email from .... to ... been delivered?

I'd like to give to our support team access to some logging tool and some guide for searching in logs. What would you have recommended me? Or, do you know any other alternatives to test?

The problem of grepping logs is that there is not sender and recipient address on one line. So I supposed, there might by some aggregation by email id.

I am now looking for a way how to monitoring a server hardware such as fans/power supplies/etc.. The problem is, we have very dynamic environment - servers are automatically powered on/off - even several times a day, depending on load.

I created tetmplates for our supermicro servers (we have just 3-4 types of them, so they are very specific) that contains fan speed check (0 means fan is dead). However, everytime I turn off the server fan speed is also 0.

So I am now searching how to get power status (or any other indicator that server is running) over ipmi to send a zabbix alert only if the server is running.

Over ipmi is unortunately the requirement, because we monitor this way some servers that we don't have an access to.

I'd like to avoid writing a script that will run something like: ipmitool power status. Zabbix has an amaizing ipmi integration, so I'd like to use it as much as possible.

ipmitool sensor returns:

root@virt1:~# ipmitool sensor
System Temp      | 28.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 75.000    | 77.000    | 79.000
CPU Temp         | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na
FAN 1            | 8355.000   | RPM        | ok    | 400.000   | 585.000   | 770.000   | 29260.000 | 29815.000 | 30370.000
FAN 2            | 8355.000   | RPM        | ok    | 400.000   | 585.000   | 770.000   | 29260.000 | 29815.000 | 30370.000
FAN 3            | 8725.000   | RPM        | ok    | 400.000   | 585.000   | 770.000   | 29260.000 | 29815.000 | 30370.000
FAN 4            | na         | RPM        | na    | na        | na        | na        | na        | na        | na
CPU Vcore        | 1.144      | Volts      | ok    | 0.640     | 0.664     | 0.688     | 1.344     | 1.408     | 1.472
+3.3VCC          | 3.280      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712
+12 V            | 12.031     | Volts      | ok    | 10.494    | 10.600    | 10.706    | 13.091    | 13.197    | 13.303
DIMM             | 1.544      | Volts      | ok    | 1.152     | 1.216     | 1.280     | 1.760     | 1.776     | 1.792
+5 V             | 5.216      | Volts      | ok    | 4.096     | 4.320     | 4.576     | 5.344     | 5.600     | 5.632
+5VSB            | 5.056      | Volts      | ok    | 4.096     | 4.320     | 4.576     | 5.344     | 5.600     | 5.632
VBAT             | 3.232      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712
+3.3VSB          | 3.280      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712
AVCC             | 3.280      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712
Chassis Intru    | 0x0        | discrete   | 0x0000| na        | na        | na        | na        | na        | na
PS Status        | 0x1        | discrete   | 0x01ff| na        | na        | na        | na        | na        | na
root@virt1:~#

When providing dedicated services, what is your best practise to protect IPMI either from public network or through kernel module.

Firstly, I want to be sure that if someone scans my networks and he finds some IPMI cards, he won't be able to get control of it. There were a bug that allows to send emails via supermicro IPMI through Anonymous account. I know it's a good practice to use IPMI only in local networks without public, however clients won't be happy about using VPN to access IPMI much.

Secondly, you can use ipmitool command to manage IPMI configuration without user authentication. I'd like to prevent customers of changing IPMI setting - e.g. IP address, removing my monitoring users, ...

What's your best practise? How would you have solved this issue if you had faced it?

Just wanted to ask if somebody has experience with the combination of X9SCL-F and Celeron G470.

I wanted to buy it, unfortunately a seller hasn't tested this combination, so he didn't want to sell it, even though Celeron is listed in the supported cpu list: http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCL-F.cfm

The server is going to be backup server => goal is low power consumption and low price.

I have several supermicro servers (X8DTU-F, X7SPA-HF). I am using IPMI configuration with shared ethernet port (X7SPA-HF doesn't have dedicated port though). IPMI over ethernet is working perfectly. So servers network connectivity do.

The problem is that servers can't access to it's ipmi ip through shared port localy (it's working with dedicated). So, if I try to ping (also arping) from server to it's ipmi ip, it doesn't work.

Server and IP is on the same network and I can access to either server and ipmi without any issues. So the issue seems to be only somewhere in localswitch on SMC motherboard.

Any idea, where can be the issue?

Update

Wesley's answer is accurate. Thanks for that. To workaround this problem - use separate networks for the server and ipmi. Set the access port on the switch to trunk and allow only vlan for the server itself (untagged) and vlan for ipmi (tagged). On IPMI then set up packet tagging with specific vlan id. This way all packets (even though they use the same interface) will go through the gateway and will not be delivered directly.