ytjohn's questions

I'm using NSX/NCP Ingress in a dedicated VMWare PKS cluster. I am attempting to stand up ingress to Elasticsearch and the backend service uses HTTPS. I can not seem to find a way to have NSX Ingress talk to a backend HTTPS service.

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.0/ncp-kubernetes/GUID-E03D6EE5-9C6C-457F-AD81-25CF2056F4D8.html

The load balancer will terminate TLS and send HTTP requests to the default backend server if there is a TLS Ingress (in the cluster for the Kubernetes/PKS use case, or in the same namespace for the Project Pacific use case) with host which matches the host in the request.

Background:

1. In this environment, a k8s PKS cluster has no control over DNS. One IP is allocated per dedicated cluster and a wildcard DNS record is pointed to that IP.
2. Only the NSX class Ingress controller gets to use that dedicated IP (kuberneties.io/ingress.class: nsx)
3. NSX Ingress controllers do not support the ingress.kubernetes.io/secure-backends: 'true' annotation.
4. I can deploy other ingress controllers. For instance, I stood up an nginx ingress controller with the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation and that works. However, these ingress controllers get a randomly assigned IP (which may change) and no DNS records.
5. I have tried removing the tls section and setting the ncp/ssl-mode: reencrypt annotation and only get a 502 error. Would be willing to revisit this.

What I'm looking for:

1. Ideally, a way I missed to just have NSX ingress work with a secure backend.
2. Failing that, a way to make the ncp/ssl-mode: reencrypt option work for this scenario case. I don't want to set a default ingress (no host in the rule)
3. Failing that, the simplest solution to present the eventsink-opendistro-es-client-service port 9200 HTTPS as an HTTP service to an NSX ingress controller. I've considered standing up a separate nginx pod that connects to the service on the backend and presents it as HTTP on the frontend, which the ingress controller can then hit.

I also don't want to rebuild the Elasticsearch image to a non-https custom one, but if there's a helm chart method to make the exposed port 9200 HTTP only, I can do that.

Finally, here is my ingress definition as it stands:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: eventsink-opendistro-es-client
  annotations: 
    kubernetes.io/ingress.class: nsx
    ingress.kubernetes.io/secure-backends: 'true'
    # ncp/ssl-mode: reencrypt
  namespace: default
spec:
  rules:
  - host: elasticsearch.clustername.pks.example.net
    http:
      paths:
      - backend:
          serviceName: eventsink-opendistro-es-client-service
          servicePort: 9200
  tls:
  - hosts:
    - elasticsearch.clustername.pks.example.net
    secretName: clustername.pks.example.net-ssl

I have apache listening on port 443 running https and performing a ProxyPass back to a standalone jenkins process on the same server.

The bulk of the time, this works fine, but it often produces a 502 error. I can reproduce this by loading the jenkin's web page up and having it auto refresh. Often within 20 minutes of doing so, I can get a 502 page.

I enabled debug logging in Apache and provided my config + apache logs below. Jenkins logs reveal nothing (like it never received the request).

I need some more pointers on tracking this down (and ultimately resolve).

Apache/2.2.3 Jenkins ver. 1.451

    <Location />
    ProxyPass http://jenkins.example.com:8080/
    ProxyPassReverse http://jenkins.example.com:8080/
    </Location>


/usr/lib/jvm/jre-1.6.0/bin/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --daemon --httpPort=8080 --ajp13Port=8009 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20

Myself getting the error: 192.168.186.207 - - [13/Mar/2013:09:23:16 -0400] "GET /?auto_refresh=true HTTP/1.1" 502 473

[Wed Mar 13 09:23:16 2013] [info] Initial (No.1) HTTPS request received for child 10 (server jenkins.example.com:443)
[Wed Mar 13 09:23:16 2013] [debug] mod_proxy_http.c(56): proxy: HTTP: canonicalising URL //jenkins.example.com:8080/
[Wed Mar 13 09:23:16 2013] [debug] proxy_util.c(1505): [client 192.168.186.207] proxy: http: found worker http://jenkins.example.com:8080/ for http://jenkins.example.com:8080/?auto_re
fresh=true, referer: https://jenkins.example.com/?auto_refresh=true
[Wed Mar 13 09:23:16 2013] [debug] mod_proxy.c(986): Running scheme http handler (attempt 0)
[Wed Mar 13 09:23:16 2013] [debug] mod_proxy_http.c(1982): proxy: HTTP: serving URL http://jenkins.example.com:8080/?auto_refresh=true
[Wed Mar 13 09:23:16 2013] [debug] proxy_util.c(2007): proxy: HTTP: has acquired connection for (jenkins.example.com)
[Wed Mar 13 09:23:16 2013] [debug] proxy_util.c(2063): proxy: connecting http://jenkins.example.com:8080/?auto_refresh=true to jenkins.example.com:8080
[Wed Mar 13 09:23:16 2013] [debug] proxy_util.c(2189): proxy: connected /?auto_refresh=true to jenkins.example.com:8080
[Wed Mar 13 09:23:16 2013] [error] [client 192.168.186.207] (104)Connection reset by peer: proxy: error reading status line from remote server jenkins.example.com, referer: https://jenkins.example.com/?auto_refresh=true
[Wed Mar 13 09:23:16 2013] [debug] mod_proxy_http.c(1484): [client 192.168.186.207] proxy: NOT Closing connection to client although reading from backend server jenkins.example.com failed., r
eferer: https://jenkins.example.com/?auto_refresh=true
[Wed Mar 13 09:23:16 2013] [error] [client 192.168.186.207] proxy: Error reading from remote server returned by /, referer: https://jenkins.example.com/?auto_refresh=true
[Wed Mar 13 09:23:16 2013] [debug] proxy_util.c(2025): proxy: HTTP: has released connection for (jenkins.example.com)
[Wed Mar 13 09:23:16 2013] [debug] ssl_engine_kernel.c(1823): OpenSSL: Write: SSL negotiation finished successfully
[Wed Mar 13 09:23:16 2013] [info] [client 192.168.186.207] Connection closed to child 10 with standard shutdown (server jenkins.example.com:443)
[Wed Mar 13 09:23:17 2013] [info] [client 192.168.186.207] Connection to child 7 established (server jenkins.example.com:443)

Issue

rpm rollback is not working with a set of repackaged rpms created in the last couple days, but does work with more recent ones.

[root@host1 repackage]# ls -l zsh-4.2.6-*
-rw-r--r-- 1 root root 1788283 Apr 10  2011 zsh-4.2.6-3.el5.i386.rpm
-rw-r--r-- 1 root root 1788691 Aug 18 04:38 zsh-4.2.6-5.el5.i386.rpm
[root@host1 repackage]# rpm -q zsh
zsh-4.2.6-6.el5
[root@host1 repackage]# rpm --test -Uvh --rollback 'Aug 18 01:00'
[root@host1 repackage]# rpm -e zsh
[root@host1 repackage]# 
[root@host1 repackage]# ls -l zsh*
-rw-r--r-- 1 root root 1788283 Apr 10  2011 zsh-4.2.6-3.el5.i386.rpm
-rw-r--r-- 1 root root 1788691 Aug 18 04:38 zsh-4.2.6-5.el5.i386.rpm
-rw-r--r-- 1 root root 1789064 Aug 20 09:06 zsh-4.2.6-6.el5.i386.rpm
[root@host1 repackage]# cp zsh-4.2.6-6.el5.i386.rpm /tmp
[root@host1 repackage]# rpm --test -Uvh --rollback 'Aug 18 01:00'
Rollback packages (+1/-0) to Mon Aug 20 09:02:16 2012 (0x50323558):
Preparing...                ########################################### [100%]
Cleaning up repackaged packages:
    Removing /var/spool/repackage/zsh-4.2.6-6.el5.i386.rpm:
[root@host1 repackage]# ls -l zsh-4.2.6-*
-rw-r--r-- 1 root root 1788283 Apr 10  2011 zsh-4.2.6-3.el5.i386.rpm
-rw-r--r-- 1 root root 1788691 Aug 18 04:38 zsh-4.2.6-5.el5.i386.rpm    
[root@host1 repackage]# cp /tmp/zsh-4.2.6-6.el5.i386.rpm  .
[root@host1 repackage]# rpm  -Uvh --rollback 'Aug 18 01:00'
Rollback packages (+1/-0) to Mon Aug 20 09:06:05 2012 (0x5032363d):
Preparing...                ########################################### [100%]
   1:zsh                    ########################################### [ 50%]
Cleaning up repackaged packages:
    Removing /var/spool/repackage/zsh-4.2.6-6.el5.i386.rpm:
[root@host1 repackage]# rpm --test -Uvh --rollback 'April 9'
[root@host1 repackage]# 

Now, if I run my test commands with -Uvvh I get debug messages to stderror which shows me that rpm reads each of the rpm files in /var/spool/repackage. The only interesting bit is the "expected size" but after searching, the expected size should be different, as it records the files as they are on the filesystem.

D: opening  db environment /var/lib/rpm/Packages joinenv
D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: opening  db index       /var/lib/rpm/Installtid rdonly mode=0x0
D: opening  db index       /var/lib/rpm/Pubkeys rdonly mode=0x0
D:  read h#     769 Header sanity check: OK
D: ========== DSA pubkey id 53268101 37017186 (h#769)
D:  read h#      32 Header V3 DSA signature: OK, key ID 37017186
D:  read h#      40 Header V3 DSA signature: OK, key ID 37017186
...
D:  read h#    1753 Header V3 DSA signature: OK, key ID 37017186
D: Expected size:      3628918 = lead(96)+sigs(344)+pad(0)+data(3628478)
D:   Actual size:      3583695
D: /var/spool/repackage/Deployment_Guide-en-US-5.2-11.noarch.rpm: Header V3 DSA signature: OK, key ID 37017186
D: Expected size:      1100789 = lead(96)+sigs(344)+pad(0)+data(1100349)
D:   Actual size:      1109281
D: /var/spool/repackage/NetworkManager-0.7.0-10.el5_5.2.i386.rpm: Header V3 DSA signature: OK, key ID 37017186
D: Expected size:      1098167 = lead(96)+sigs(344)+pad(0)+data(1097727)
D:   Actual size:      1106179
D: /var/spool/repackage/NetworkManager-0.7.0-9.el5.i386.rpm: Header V3 DSA signature: OK, key ID 37017186
D: Expected size:        84351 = lead(96)+sigs(344)+pad(0)+data(83911)
D:   Actual size:        85378
...
D: Expected size:      1788276 = lead(96)+sigs(344)+pad(0)+data(1787836)
D:   Actual size:      1788691
D: /var/spool/repackage/zsh-4.2.6-5.el5.i386.rpm: Header V3 DSA signature: OK, key ID 37017186
D:      --- erase h#1758
D: closed   db index       /var/lib/rpm/Pubkeys
D: closed   db index       /var/lib/rpm/Installtid
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm/Packages
D: May free Score board((nil))

I am able to copy these rpms out of the repackage directory and if I run them through cpio, extract the files.

I also tried backing up and rebuilding the rpm database - no change.

System Information:
RHEL 5.8
rpm 4.4.2.3

/etc/yum.conf
tsflags=repackage

/etc/rpm/macros
%_repackage_all_erasures 1

mdadm, gpt issues, unrecognized partitions.

Simplified question: How do I get mdadm to recognize GPT partitions?

I have been attempting to convert/copy my Ubuntu 11.10 OS from a single drive to software raid 1. I have done similar in the past, but in this case, I was adding in a drive that has been configured for GPT and I tried to work with that without fully looking into the implications.

Currently, I have a non-booting mdadm RAID 1 array of /dev/md127 (the OS assigned that and it keeps picking up). I am booting off of live USB keys, currently System Rescue CD from sysresccd. While gdisk and parted can see all the partitions, most of the OS utilities do not, including mdadm. My main goal is just to make the raid array accessible so I can get pull the data and start fresh (without using GPT).

/dev/md127

/dev/sda
 /dev/sda1 <- GPT type partition
  /dev/sda1 <- exists within the GPT part, member of md127
  /dev/sda2 <- exists within the GPT part, empty

/dev/sdb
 /dev/sdb1 <- GPT type partition
  /dev/sdb1 <- exists within the GPT part, member of md127

History:

POINT A: The original OS was install on sda (actually /dev/sda6). I used a the Ubuntu live usb to add sdb. I got warning from fdisk about GPT so I used gdisk to create a raid partition (sdb1) and mdadm to create a raid1 mirror with a missing drive. I had many issues getting this working (including being unable to get grub to install) but I eventually got it to boot using grub on sda and /dev/md127 off of sdb. So at point A, I had copied my OS from sda6 to md127 on sdb. I then booted into a rescue mode and attempted to get a bootloader onto sdb, which failed. I then discovered my mistake: I had installed the raid onto sdb instead of sdb1, essentially overwriting the sdb1 partition.

POINT B: I now had two copies of my data- one on md127/sdb, and one on sda. I destroyed data on sda and created a new GPT table on sda. I then created sda1 for the raid array, and sda2 for a scratch partition. I added sda1 into the raid array and let it rebuild. md127 now covered /dev/sdb and /dev/sda1 as fully active and synced.

POINT C: I rebooted onto linux rescue again and was still able to access the raid array. I then removed /dev/sdb from the array and created /dev/sdb1 for the raid. I added sdb1 to the array and let it sync. I was able to mount and access /dev/md127 without issues. Once it completed, both /dev/sda1 and /dev/sdb1 were GPT partitions and actively syncing.

POINT D (current): I rebooted again to test if the array would boot and grub failed to load. I booted off of my live thumb drive and found that I can no longer assemble the raid array. mdadm doesn't see the required partitions.

--
root@freshdesk /root % uname -a
Linux freshdesk 3.0.24-std251-amd64 #2 SMP Sat Mar 17 12:08:55 UTC 2012 x86_64 AMD Athlon(tm) II X4 645 Processor AuthenticAMD GNU/Linux



===
/proc/partitions and parted look good:
root@freshdesk /root % cat /proc/partitions
major minor  #blocks  name

   7        0     301788 loop0
   8        0  976762584 sda
   8        1  732579840 sda1
   8        2  244181703 sda2
   8       16  732574584 sdb
   8       17  732573543 sdb1
   8       32    7876607 sdc
   8       33    7873349 sdc1


(parted) print all
Model: ATA ST31000528AS (scsi)
Disk /dev/sda: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size   File system  Name                Flags
 1      1049kB  750GB   750GB  ext4
 2      750GB   1000GB  250GB               Linux/Windows data


Model: ATA SAMSUNG HD753LJ (scsi)
Disk /dev/sdb: 750GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End    Size   File system  Name        Flags
 1      1049kB  750GB  750GB  ext4         Linux RAID  raid


Model: SanDisk SanDisk Cruzer (scsi)
Disk /dev/sdc: 8066MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      31.7kB  8062MB  8062MB  primary  fat32        boot, lba


===
# no sda2, and I double the sdb1 is the one shown in parted
root@freshdesk /root % blkid
/dev/loop0: TYPE="squashfs"
/dev/sda1: UUID="75dd6c2d-f0a8-4302-9da4-792cc7d72355" TYPE="ext4"
/dev/sdc1: LABEL="PENDRIVE" UUID="1102-3720" TYPE="vfat"
/dev/sdb1: UUID="2dd89f15-65bb-ff88-e368-bf24bd0fce41" TYPE="linux_raid_member"

root@freshdesk /root % mdadm -E /dev/sda1
mdadm: No md superblock detected on /dev/sda1.

# this is probably a result of me attempting to force the array up, putting superblocks on the GPT partition
root@freshdesk /root % mdadm -E /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 2dd89f15:65bbff88:e368bf24:bd0fce41
  Creation Time : Fri Mar 30 19:25:30 2012
     Raid Level : raid1
  Used Dev Size : 732568320 (698.63 GiB 750.15 GB)
     Array Size : 732568320 (698.63 GiB 750.15 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127

    Update Time : Sat Mar 31 12:39:38 2012
          State : clean
 Active Devices : 1
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 1
       Checksum : a7d038b3 - correct
         Events : 20195


      Number   Major   Minor   RaidDevice State
this     2       8       17        2      spare   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       0        0        1      faulty removed
   2     2       8       17        2      spare   /dev/sdb1


===

root@freshdesk /root % mdadm -A /dev/md127 /dev/sda1 /dev/sdb1
mdadm: no recogniseable superblock on /dev/sda1
mdadm: /dev/sda1 has no superblock - assembly aborted
    root@freshdesk /root % mdadm -A /dev/md127  /dev/sdb1
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has no superblock - assembly aborted

I have a server running cPanel/WHM with exim and SpamAssassin. I've been noticing an issue where emails coming in with forged spamassassin headers bypassing some of the filtering. I want to strip out all SpamAssassin headers before it goes through spamassassin and then filtered into the inbox/spam folders.

Searching the net, the only similar instance I could find was from 2004. However, the exim config by that user and by me are very different. I am not sure how to apply it. I can run formail against a file containing the message to remove the headers, but I don't know how to make exim do that.

Just to provide an example, a message will come in with headers like this:

X-Spam-Status: No, score=1.3
X-Spam-Score: 13
X-Spam-Bar: +
X-Ham-Report: Spam detection software, running on the system "serv02.example.com", has
        identified this incoming email as possible spam.  The original message *snip*
X-Spam-Flag: NO

My SpamAssassin will add these headers to the message:

X-Spam-Status: Yes, score=6.8
X-Spam-Score: 68
X-Spam-Bar: ++++++
X-Spam-Report: Spam detection software, running on the system "serv02.example.com", has
        identified this incoming email as possible spam.  The original message *snip*
X-Spam-Flag: YES

But because the exim vfilter rules read the first X-Spam headers, the email ends up in the user's inbox instead of in the spam folder.

I manage a campground wifi network with an average of 10 - 60 active users. I have encountered issues where the router starts acting flaky (failing to assign DHCP or failing to pass traffic) without any clear warning (low cpu utilization, etc). I upgraded the router a couple times and ended up with a Netgear ProSafe VPN router that seems to be handling the traffic. The interesting thing is that the Netgear has lower specs than the Buffalo router it replaced, indicating the issue is with the DD-WRT firmware. While I'll be pursuing this issue on the dd-wrt forums, I need a way to test routers.

My vision is having 1-2 computers connected on the LAN side and 1-2 computers connected on the WAN side. I want the LAN computers to be generating various type of traffic and connections, as well as requesting DCHP addresses.

A few notes:

• The wireless aspect should be a non-issue. Most clients would connect to a wireless bridge and come into the router through a network cable.
• I had a monitoring server with Nagios running check_dhcp against the router. This server was connected directly by a network cable, eliminating wifi bridges and other devices from the equation.
• This question is somewhat related, but not exactly: Load testing wireless LANs I am going to look at IxChariot.
• While I'd ideally like to use a 1 computer on each side running Linux and preferably free software, I can entertain running Windows, multiple computers, or non-free software.
• Total bandwidth doesn't seem to be the issue. I can transfer large files all day. Even on the busiest days, the users seemed to only pull ~5Mbps. There is very little "LAN to LAN traffic" and most of it might never have reached the main router.
• The issue I need to test for seems to be tied to active users, or more appropriately, active sessions.
• I know active users or active clients is a meaningless term from a router standpoint and wouldn't mind having more appropriate terms to use.

Summary: I need a way to test a routers ability in handling traffic from a large number of clients. My current strategy is to purchase a router, deploy it, and see how it fails in the live environment.