BaseZen's questions

FINAL EDIT 7/7 Multiple cable, port, and device substitutions have narrowed this to the Comcast modem, and on all 8 of its Ethernet ports. Given, as the comments say, this is a closed device, and unlikely to get much real information, we may never find out, but I will post the resolution as an answer nonetheless.

(Edit 1/6 motivation; underlying real issue) Based on user complaints of teleconference issues, I wanted to eliminate all possible causes. After verifying picture-perfect LAN and WiFi connectivity from laptop to server, then seeing very uneven end-to-end ping latency -- often well above the 100ms suggested limit -- with the Google Meet server, as directed here, I backtracked to the source of the latency. (This is not necessarily the root cause of Google Meet delays, of course, but I need to eliminate this as a possible cause.)

It turns out the uneven and high latency is coming from the direct link between a Sophos UTM 9 SG125 (Firmware: 9.703-3) to a Comcast CGA4341COM Gigabit modem (Manufacturer: Technicolor; Hardware revision: 2.3; Chipset: Broadcom).

Both link sides report Gigabit connection. Speedtest to speedtest.xfinity.com gives 400Mbps range results from a hardwired on-LAN server. (Edit 2/6 Additional evidence of issue This seems great except that when same server is wired directly to the modem, cutting out the router and the rest of the LAN entirely, the throughput is 930Mbps.)

After a long test on SSH cmd line on the Sophos to the directly-connected modem using a 10-foot Cat5e cable:

—— xx.xx.xx.134 ping statistics ---
756 packets transmitted, 756 received, 0% packet loss, time 755277ms
rtt min/avg/max/mdev = 0.162/21.789/199.543/34.605 ms

The long pings are quite densely interspersed:

64 bytes from xx.xx.xx.134: icmp_seq=1 ttl=64 time=58.2 ms
64 bytes from xx.xx.xx.134: icmp_seq=2 ttl=64 time=0.645 ms
64 bytes from xx.xx.xx.134: icmp_seq=3 ttl=64 time=72.4 ms

Trying this all day long changed nothing. All extraneous features of the modem are disabled: port forwarding, port triggering, firewall, MAC access control, dhcp, wifi, etc.

(Edit 3/6 Re: Possible traffic load accounting for delay) This occurred in the middle of the night as well, and so is not traffic-dependent. When the link utilization under 1%, despite the possible inaccuracy of ping, prioritization should not play a factor.

(Edit 4/6 Re: Possible low prioritization of ICMP) traceroute, using UDP, shows identical delay patterns:

traceroute -q 10 -w 1 10.1.10.1
traceroute to 10.1.10.1 (10.1.10.1), 30 hops max, 40 byte packets using UDP
 1  10.1.10.1 (10.1.10.1)  71.784 ms   70.684 ms * * *   66.310 ms * * * *
traceroute -q 10 -w 1 10.1.10.1
traceroute to 10.1.10.1 (10.1.10.1), 30 hops max, 40 byte packets using UDP
 1  10.1.10.1 (10.1.10.1)  1.218 ms   1.151 ms * * * * * * * *
traceroute -q 10 -w 1 10.1.10.1
traceroute to 10.1.10.1 (10.1.10.1), 30 hops max, 40 byte packets using UDP
 1  10.1.10.1 (10.1.10.1)  61.156 ms * * * *   55.497 ms   54.370 ms * * *

Edit 5/6 Re: Normal behavior for this ISP and modem At different customer site, identical modem hardware connected to a $65 EdgeRouter X, I see from the router, normalcy:

--- 10.1.10.1 ping statistics ---
60 packets transmitted, 60 received, 0% packet loss, time 59398ms
rtt min/avg/max/mdev = 0.278/1.201/2.175/0.554 ms

Similarly, after 100 traceroute UDP packets sent at this 2nd site, the slowest of all was 3.3ms.

(Edit 6/6 Re: Possible normalcy, generally) Between any modem and router I've never seen this delay pattern in years of working with broadband, both low and high end, with multiple vendors. I have not discounted the Sophos side; I will report when I can get onsite with a direct connect to a different device.

On the Sophos, no packet errors:

router:/var/log# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 7C:xx:xx:xx:xx:94  
          inet addr:96.xx.xx.129  Bcast:96.xx.xx.135  Mask:255.255.255.248
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:300119356 errors:0 dropped:0 overruns:0 frame:0
          TX packets:243077712 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:264200277517 (251961.0 Mb)  TX bytes:197347533783 (188205.2 Mb)

On eth0, the Sophos is pinging a server on-LAN with the typical 0.1-0.2ms very steady latency.

router:/# ping 192.168.1.5
PING 192.168.1.5 (192.168.1.5) 56(84) bytes of data.
64 bytes from 192.168.1.5: icmp_seq=1 ttl=128 time=0.198 ms
64 bytes from 192.168.1.5: icmp_seq=2 ttl=128 time=0.128 ms

There's no loading (CPU or memory or disk) on the Sophos at all, nor anything remarkable in the logs, nor anything in dmesg.

lshw reports:

      *-network:1
            description: Ethernet interface
            product: Ethernet Connection X553 1GbE
            vendor: Intel Corporation
            physical id: 0.1
            bus info: pci@0000:0b:00.1
            logical name: eth1
            version: 11
            serial: 7c:xx:xx:xx:xx:94
            size: 1Gbit/s
            capacity: 1Gbit/s
            width: 64 bits
            clock: 33MHz
            capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt-fd 100bt-fd 1000bt-fd autonegotiation
            configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=5.2.4 duplex=full firmware=0x80000878 ip=96.86.73.129 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
            resources: irq:17 memory:dfa00000-dfbfffff memory:dfe00000-dfe03fff memory:dc500000-dc57ffff

The modem has almost nothing to report when searching 90 days of logs. Today, only:

FW.WANATTACK DROP , 34 Attempts, 2020/6/16 15:58:01
Firewall Blocked

Detailed software stats on modem:

eMTA & DOCSIS Software Version: CM DOCSIS Application - Prod_18.1_d31 & MTA Application - Prod_18.1
Software Image Name: CGA4131COM_3.12p12s1_PROD_sey
Advanced Services: CGA4131COM
Packet Cable: 2.0

By accident, I have an expired intermediate certificate at the end of my chain file in my Dovecot server's SSL configuration. It's enough of a problem that my Android e-mail client refuses to use it, although Apple Mail lets it go (??!). Indeed, the expiration just happened hours ago. openssl x509 -in ... shows:

    Serial Number:
        13:ea:28:70:5b:f4:ec:ed:0c:36:63:09:80:61:43:36
    Signature Algorithm: sha384WithRSAEncryption
    Issuer: C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
    Validity
        Not Before: May 30 10:48:38 2000 GMT
        Not After : May 30 10:48:38 2020 GMT
    Subject: C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority

But this command:

openssl s_client -showcerts -verify_return_error -connect imap.example.com:993

fails to flag the problem (while outputting the expired certificate!). The OpenSSL package version is: 1.1.1g-1+ubuntu18.04.1+d

CONNECTED(00000003)
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
verify return:1
depth=0 CN = imap.example.com
verify return:1

How do I create an OpenSSL verification test to find and flag this? I have searched online already quite a bit and found nothing to address expiration down a few rungs in a public chain. The closest question is: Why is my SSL certificate untrusted on Android? but this only deals with a missing link in a 4-certificate chain. My guess as to why Apple Mail accepts the error is that MacOS has cached its own non-expired version of the same intermediate CA.

EDIT

On the server, the following:

/usr/share/ca-certificates/mozilla/USERTrust_RSA_Certification_Authority.crt

is now self-signed, so openssl must be silently substituting this one (edit: tested by hiding this cert; the expiration is now detected!) but my goal is to be sensitive enough to detect the Android's complaint. I am on Android 10 but one (May 4) update behind the latest.

I've successfully created a limited Management role, and assigned a user to it, that allows a junior admin only two things via the ECP web page:

-- Reset passwords in a particular Org Unit

-- Create new users in the same

The write privileges are no problem: you just pick the scope. However they really shouldn't see the AD data (users, groups) for any other OU. But when they log in, the Whole World is listed.

Everything I've read seems to give no flexibility to the Read Scope -- that it must be implicit and include the entire Organization: https://technet.microsoft.com/en-us/library/dd335146(v=exchg.150).aspx#ImplicitScopes

I know that's an irrational restriction, and there's a way to do what I want, and that it's going to be much more involved than I would ever want, but that it is possible, so I was hoping to be pointed in the right direction.

UPDATE Looks to be not possible: https://social.technet.microsoft.com/Forums/exchange/en-US/a063a190-89a4-4611-aa8d-772ab5a832f7/exchange-2013-rbac-read-scope?forum=exchangesvradmin&prof=required but this is just very surprising.

I have an Active Directory Windows 2012 (R2 Standard, 6.3.9600) server that has been working for years with no issues. All of a sudden, I have the following failure mode. It should be obvious from the evidence below that networking infrastructure -- packets, routing, firewall -- are not the problem. I've also tried restarting the DNS server several times, refreshing and reloading the zones, and sanity checking them.

This is PowerShell on the server talking to its own physical network interface:

Server> Resolve-DNSName printer.example.com -Server 10.1.1.1 -DNSOnly -Type A
Resolve-DNSName : printer.thecoop.com : This operation returned because the timeout period expired
At line:1 char:1
+ Resolve-DNSName printer.example.com -Server 10.1.1.1 -DNSOnly -Type A
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationTimeout: (printer.example.com:String) [Resolve-DnsName], Win32Exception
    + FullyQualifiedErrorId : ERROR_TIMEOUT,Microsoft.DnsClient.Commands.ResolveDnsName

What makes no sense is that when the request is initiated from anywhere else on the LAN or over the VPN that exact query works fine.

Further, if we switch to type ALL it also works fine (MX records are good too, for 'example.com'):

Server> Resolve-DNSName printer.example.com -Server 10.1.1.1 -DNSOnly -Type ALL

Name                                 Type   TTL   Section    IPAddress
----                                 ----   ---   -------    ---------
printer.example.com                    A    3600  Answer     10.1.1.202

And finally, switching to querying at localhost also works fine:

Server> Resolve-DNSName printer.example.com -Server 127.0.0.1 -DNSOnly -Type A

Name                                   Type   TTL   Section    IPAddress
----                                   ----   ---   -------    ---------
printer.example.com                    A      3600  Answer     10.1.1.202

DNS PACKET CAPTURE

Enabling packet debugging, I see, but mostly don't understand:

4/30/2018 2:27:41 PM 1D30 PACKET  000000764976A220 UDP Rcv 10.1.1.1     81ea   Q [0001   D   NOERROR] A      (15)printer(7)example(3)com(0)

4/30/2018 2:27:41 PM 1D30 PACKET  000000764976A220 UDP Snd 10.1.1.1     81ea R Q [8085 A DR  NOERROR] A      (15)printer(7)example(3)com(0)

4/30/2018 2:27:42 PM 1D30 PACKET  000000764ACB60A0 UDP Rcv 10.1.1.1     81ea   Q [0001   D   NOERROR] A      (15)printer(7)example(3)com(0)

4/30/2018 2:27:42 PM 1D30 PACKET  000000764ACB60A0 UDP Snd 10.1.1.1     81ea R Q [8085 A DR  NOERROR] A      (15)printer(7)example(3)com(0)

4/30/2018 2:27:43 PM 1D30 PACKET  00000076482420A0 UDP Rcv 10.1.1.1     81ea   Q [0001   D   NOERROR] A      (15)printer(7)example(3)com(0)

4/30/2018 2:27:43 PM 1D30 PACKET  00000076482420A0 UDP Snd 10.1.1.1     81ea R Q [8085 A DR  NOERROR] A      (15)printer(7)example(3)com(0)

I have a bridged OpenVPN setup on a Linux server in an Amazon EC2 VPC. (Spent hours on docs, reading similar problems, here, openVPN forums, no luck yet.)

The bridged interface is up and contains both sub-interfaces:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.0e7c15e787b0       no              eth0
                                                        tap0

Routing is obviously OK on the VPN server; I can SSH in, ping around, respond to the VPN request from the client:

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.0.1        0.0.0.0         UG        0 0          0 br0
10.0.0.0        0.0.0.0         255.255.0.0     U         0 0          0 br0

I can ping from both Windows & Mac clients to the VPN server's IP but not to any other IP's on the VPC subnet. (Those other IP's are OK; they are pingable from the VPN server.)

When I tcpdump on the bridge interface br0 on the VPN server it sees the "ARP who-has" requests from the Windows client. However they aren't going onto the VPC subnet! tcpdump on the destination IP does not see the ARP arrive. The Windows arp cache remains unfilled. (10.0.0.128 is the Windows client; 10.0.0.58 is the VPN server; 10.0.0.180 is the other IP on the subnet; the output below is from the VPN server.)

root@vpn:# tcpdump -i br0 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:00:21.092367 ARP, Request who-has 10.0.0.180 tell 10.0.0.128, length 28

[crickets]

I have disabled the Source/Dest. check within the EC2 console on the VPN server's sole network interface.

I have set up the IP tables as recommended in the bridging HOWTO, and generally followed these instructions exactly.

# iptables -L INPUT -v
Chain INPUT (policy ACCEPT 9 packets, 1008 bytes)
 pkts bytes target     prot opt in     out     source               destination
   38 12464 ACCEPT     all  --  tap0   any     anywhere             anywhere
10447 1297K ACCEPT     all  --  br0    any     anywhere             anywhere

# iptables -L FORWARD -v
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  918  167K ACCEPT     all  --  br0    any     anywhere             anywhere

https://openvpn.net/index.php/open-source/documentation/miscellaneous/76-ethernet-bridging.html

I don't think I need to dump the full configs because obviously a lot is working: authentication, certs, compression, address pool, connection set-up generally. Does the Amazon VPC simply refuse to forward packets and I should really be on a somewhat-less-virtual cloud to do this?

MORE EXPERIMENTS THE NEXT DAY: The VPC clearly isn't behaving like a true layer 2 subnet. In particular, ARP who-has broadcasts don't actually broadcast! When I ping a non-existent IP (say .5) from .180, .58 doesn't see the request. The VPC is obviously optimizing away ARP broadcasts and sending it only to .5, if a .5 has been configured in the VPC via management console / API. Leaving tcpdump -vv -i eth0 arp on for a while only shows traffic between the host and the gateway, for all hosts.

Further, pinging the broadcast address on the subnet doesn't work at all. This is backed up by the Amazon VPN FAQ.

So the VPC is likely refusing to recognizing the unknown MAC address of .129, since it doesn't exist in its own "virtual ethernet switch". I'll probably shift this as the answer in a week or so. To extend the VPC with your own VPN, it must be via the formal "VPC gateway", which is only designed to work as an extension of a corporate intranet backed by a dedicated hardware router and static IP, not the roaming laptop scenario I'm aiming for.

Got a disturbing crash while performing 'apachectl stop'. General system:

$ uname -a 
Linux www.example.com 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -d
Description:    Ubuntu 14.04 LTS

Lots of spare capacity in disk, memory, CPU. This is an Amazon EC2 cloud instance, running today at 1PM 5/7/2014, region us-east-1a, medium-size instance with 3.7GB mem/2CPU. My other instances on the same VPC and same region were fine.

I read elsewhere that in today's kernels, you don't get a crash like this unless the hardware is failing. Seems unlikely that Amazon would have faulty hardware in the cloud?? Or am I being pollyannish?

Anyway, the dump from dmesg (the system continued to operate by serving webpages and talking to the database, but new processes hung instantly, such as ps and ssh):

[27917995.400499] general protection fault: 0000 [#1] SMP 
[27917995.400515] Modules linked in: isofs crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd
[27917995.400537] CPU: 0 PID: 1672 Comm: apache2 Not tainted 3.13.0-24-generic #46-Ubuntu
[27917995.400545] task: ffff8800020117f0 ti: ffff88005f012000 task.ti: ffff88005f012000
[27917995.400551] RIP: e030:[]  [] devpts_kill_index+0x13/0x60
[27917995.400564] RSP: e02b:ffff88005f013d58  EFLAGS: 00010286
[27917995.400568] RAX: dc73af5e3df7dcab RBX: ffff880003f30400 RCX: 0000000181000079
[27917995.400574] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: ffff8800aab76ff8
[27917995.400579] RBP: ffff88005f013d68 R08: 0000000000000000 R09: 0000000000000001
[27917995.400583] R10: ffffea0003a01180 R11: ffffffff8144a320 R12: 0000000000000002
[27917995.400588] R13: ffff8800e87a8001 R14: 0000000000000002 R15: 0000000000000001
[27917995.400598] FS:  00007f8d8b320780(0000) GS:ffff8800ef600000(0000) knlGS:0000000000000000
[27917995.400605] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[27917995.400610] CR2: 00007f8d79aea7e0 CR3: 0000000001c0e000 CR4: 0000000000002660
[27917995.400616] Stack:
[27917995.400619]  ffff880003f30400 ffff880003f30800 ffff88005f013d78 ffffffff8144caa8
[27917995.400628]  ffff88005f013d90 ffffffff81440e47 ffff880003f30400 ffff88005f013e38
[27917995.400636]  ffffffff81443159 ffff880003f30610 ffff880003f30628 ffff880003f30630
[27917995.400645] Call Trace:
[27917995.400656]  [] pty_unix98_shutdown+0x18/0x20
[27917995.400662]  [] release_tty+0x37/0x140
[27917995.400668]  [] tty_release+0x4b9/0x600
[27917995.400678]  [] __fput+0xe4/0x260
[27917995.400684]  [] ____fput+0xe/0x10
[27917995.400693]  [] task_work_run+0xc4/0xe0
[27917995.400701]  [] do_exit+0x2ab/0xa50
[27917995.400708]  [] ? vtime_account_user+0x54/0x60
[27917995.400717]  [] ? context_tracking_user_exit+0x4f/0xc0
[27917995.400723]  [] do_group_exit+0x3f/0xa0
[27917995.400729]  [] SyS_exit_group+0x14/0x20
[27917995.400738]  [] tracesys+0xe1/0xe6
[27917995.400742] Code: 0f 1f 84 00 00 00 00 00 48 83 c4 08 b8 fb ff ff ff 5b 41 5c 5d c3 66 90 66 66 66 66 90 55 48 89 e5 41 54 41 89 f4 53 48 8b 47 28  81 78 58 d1 1c 00 00 74 0b 48 8b 05 44 bf d7 00 48 8b 40 08 
[27917995.400796] RIP  [] devpts_kill_index+0x13/0x60
[27917995.400803]  RSP 
[27917995.400811] ---[ end trace 5b24303912015285 ]---
[27917995.400815] Fixing recursive fault but reboot is needed!