Christian's questions

Has anyone seen the Linux time being stuck, it has been driving me nuts since yesterday.

# date
Thu Apr  2 16:35:04 UTC 2020
# date
Thu Apr  2 16:35:04 UTC 2020
# date
Thu Apr  2 16:35:04 UTC 2020
# date
Thu Apr  2 16:35:04 UTC 2020

Here was about 3 minutes break while I logged into servfault.

# date
Thu Apr  2 16:35:05 UTC 2020

RTC seems to tick, this is about ten one-thousands apart:

# timedatectl
      Local time: Thu 2020-04-02 16:35:06 UTC
  Universal time: Thu 2020-04-02 16:35:06 UTC
        RTC time: Thu 2020-04-02 16:42:36
       Time zone: America/New_York (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

# timedatectl
      Local time: Thu 2020-04-02 16:35:06 UTC
  Universal time: Thu 2020-04-02 16:35:06 UTC
        RTC time: Thu 2020-04-02 16:42:46
       Time zone: America/New_York (UTC, +0000)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a

# hwclock --show
Thu 02 Apr 2020 04:43:56 PM UTC  -0.001834 seconds
# hwclock --show
Thu 02 Apr 2020 04:44:04 PM UTC  -0.000916 seconds

These are also about ten secs apart:

# hwclock --hctosys
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020

Try NTPd

# systemctl start ntpd
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020
# date
Thu Apr  2 16:44:24 UTC 2020

# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-04-02 16:44:24 UTC; 27ms ago
  Process: 18008 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 18009 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─18009 /usr/sbin/ntpd -u ntp:ntp -g



# ntpq
ntpq> lpeers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 time.cloudflare .INIT.          16 u    -   64    0    0.000    0.000   0.977
 clock.sjc.he.ne .INIT.          16 u    -   64    0    0.000    0.000   0.977
 ntp4.doctor.com .INIT.          16 u    -   64    0    0.000    0.000   0.977
 tock.usshc.com  .INIT.          16 u    -   64    0    0.000    0.000   0.977

It just stays at .INIT. although peers are reacheable

# ping tock.usshc.com
PING tock.usshc.com (199.102.46.72) 56(84) bytes of data.
64 bytes from tock.usshc.com (199.102.46.72): icmp_seq=1 ttl=54 time=1.00 ms

Edit#1

# grep . /sys/devices/system/clocksource/clocksource*/[ac]*_clocksource
/sys/devices/system/clocksource/clocksource0/available_clocksource:tsc hpet acpi_pm refined-jiffies jiffies
/sys/devices/system/clocksource/clocksource0/current_clocksource:jiffies

# dmesg | grep clocksource
[    0.000000] Command line: vmlinuz nomodeset rw selinux=0 enforcing=0 raid=noautodetect nmi_watchdog=0 LAUNCH_INIT=/tmp/SUTup rdshell console=tty0 console=ttyS0,9600n1 HOMEBASE=192.168.1.1 biosdevname=0 clocksource=jiffies nmi_watchdog=0 consoleblank=0 vga=791 ramdisk_size=4000000 initrd=initramfs
[    0.000000] Kernel command line: vmlinuz nomodeset rw selinux=0 enforcing=0 raid=noautodetect nmi_watchdog=0 LAUNCH_INIT=/tmp/SUTup rdshell console=tty0 console=ttyS0,9600n1 HOMEBASE=192.168.1.1 biosdevname=0 clocksource=jiffies nmi_watchdog=0 consoleblank=0 vga=791 ramdisk_size=4000000 initrd=initramfs
[   51.409031] tsc: Refined TSC clocksource calibration: 2394.374 MHz

This server is getting really annoying. The running OS is a CentOS live image served via PXE. There's no FakeTime.

Check this out: this should take ~2 seconds but takes ten times that (running via SSH so I get an external 'time')

# time ssh 192.168.100.115 'for i in {0..1}; do date; sleep 1 ; done'
Thu Apr  2 23:39:22 UTC 2020
Thu Apr  2 23:39:23 UTC 2020
real    0m20.869s
user    0m0.026s
sys     0m0.007s

There's no firewall blocking NTP, it's a small network. It's a physical Intel server, not VM.

Edit#2

Paul suggested to check my clocksource, it was jiffies, switched to hpet, and things started working; not sure why it worked until yesterday (first time I noticed the time stuck), we've used this image for years on all sorts of hardware.

Clocksource: https://access.redhat.com/solutions/18627

Is there any way to find the physical topology of a drive on the SCSI bus in linux?

here's an example:

>: for drive in $(find /dev/disk/by-path -name \*0|sort); do udevadm info --query=path --name $drive  ; done
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:0/0:2:0:0/block/sda
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:1/0:2:1:0/block/sdb
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:2/0:2:2:0/block/sdc
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:3/0:2:3:0/block/sdd
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:4/0:2:4:0/block/sde
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:5/0:2:5:0/block/sdf
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:6/0:2:6:0/block/sdg
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:7/0:2:7:0/block/sdh
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:8/0:2:8:0/block/sdi
/devices/pci0000:00/0000:00:02.2/0000:03:00.0/host0/target0:2:9/0:2:9:0/block/sdj

I can trace them up to the RAID card

>: lspci | egrep 02.2\|03.00
00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07)
03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] (rev 01)

but I can't trace any further (port on the raid card or expander), the scsi target numbers are sequential, mind that I have four empty slots so I would have expected target numbers to "skip" a few.

thanks.