Timothy Miller's questions

I have one system running RHEL7 that mysteriously keeps coming up with a secondary IP address in addition to the one I set statically. When I do "ip addr show eno0", it describes this secondary address as being "scope global secondary dynamic". There are no files in "/etc/sysconfig/network-scripts/" that call for a dynamic IP. And in the past, this has been good enough to get only static IP.

All the searches I do on this lead to the suggestion to disable dhcpcd. But this is a new problem. This might solve it for now, right up until the point where I need to make eno1 have a dynamic IP address. Disabling dhcpcd shouldn't be necessary and hasn't been up until now.

The address it's getting is consistently the same thing, despite the fact that there is no DHCP server on this network. So it seems like it's getting this address from some config file local to the machine, but we can't identify it.

Any help would be appreciated. Thanks!

I have a PetaLinux machine (Embedded Linux on a Xilinx Zynq device, debian fork, kernel 4.19). If the two NICs are on different subnets, then I can disconnect one NIC, and the other will continue working. But if they are on the same subnet, then disconnecting eth0 will render both unreachable. (Disconnecting eth1 is fine.) Also, if the addresses are acquired by DHCP, then disconnecting the plug for eth0 is also fine.

Now, I understand that by default, Linux has a weak policy for choosing a NIC to respond through for any message, and this is something I dealt with a few years ago, in another article.

Unfortunately, this solution doesn't seem to be working on the version of Linux we have here. Is there something new we have to do with more recent kernels?

Thanks in advance.

Update

"ip route" while both cables are plugged in:

default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.195
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.196

"ip route" when eth0 is unplugged (failure case):

default via 192.168.1.1 dev eth0 linkdown
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.195 linkdown
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.196

"ip route" when eth1 is unplugged (works fine):

default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.195
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.196 linkdown

To reiterate, we are unable to ping 192.168.1.196 when the NIC for 192.168.1.195 is unplugged.

We recently bought a New Dell R340XL Server and installed CentOS 8. We have one SSD for boot on the BOSS controller (as sda) and 4 HDDS on the PERC H330 in RAID 5 (as sdb). It worked fine for a few days, when two days ago, we could no longer see the RAID volume. We got on the phone with Dell, and they helped us upgrade firmwares on multiple devices, but still we can't see the RAID volume. Any help would be much appreciated.

The device shows up when I do lspci:

02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] (rev 02)
        Subsystem: Dell PERC H330 Adapter
...
        Kernel driver in use: megaraid_sas
        Kernel modules: megaraid_sas

I downloaded MegaCli, and I can get all kinds of information. The command MegaCli64 -LdGetNum -aAll tells me this:

Number of Virtual Drives Configured on Adapter 0: 1

If we reboot the server over and over again, once in a while, we'll see the RAID array appear as a block device, but it appears as sda, making the boot drive sdb.

I've looked through dmesg and system logs, and there's not much I know how to recognize, but I do see things like this:

[    1.296976] megasas: 07.707.51.00-rc1
[    1.301092] megaraid_sas 0000:02:00.0: FW now in Ready state
[    1.301095] megaraid_sas 0000:02:00.0: 63 bit DMA mask and 32 bit consistent mask
[    1.301363] megaraid_sas 0000:02:00.0: firmware supports msix        : (96)
[    1.301364] megaraid_sas 0000:02:00.0: current msix/online cpus      : (12/12)
[    1.301365] megaraid_sas 0000:02:00.0: RDPQ mode     : (disabled)
[    1.301366] megaraid_sas 0000:02:00.0: Current firmware supports maximum commands: 928        LDIO threshold: 237
[    1.301477] megaraid_sas 0000:02:00.0: Configured max firmware commands: 927
[    1.303185] megaraid_sas 0000:02:00.0: FW supports sync cache        : No

...

[    1.680004] megaraid_sas 0000:02:00.0: FW provided supportMaxExtLDs: 0       max_lds: 32
[    1.680005] megaraid_sas 0000:02:00.0: controller type       : iMR(0MB)
[    1.680005] megaraid_sas 0000:02:00.0: Online Controller Reset(OCR)  : Enabled
[    1.680006] megaraid_sas 0000:02:00.0: Secure JBOD support   : No
[    1.680006] megaraid_sas 0000:02:00.0: NVMe passthru support : No
[    1.680007] megaraid_sas 0000:02:00.0: FW provided TM TaskAbort/Reset timeout        : 0 secs/0 secs
[    1.702120] megaraid_sas 0000:02:00.0: INIT adapter done
[    1.702121] megaraid_sas 0000:02:00.0: Jbod map is not supported megasas_setup_jbod_map 5371
[    1.728949] megaraid_sas 0000:02:00.0: pci id                : (0x1000)/(0x005f)/(0x1028)/(0x1f44)
[    1.728950] megaraid_sas 0000:02:00.0: unevenspan support    : yes
[    1.728950] megaraid_sas 0000:02:00.0: firmware crash dump   : no
[    1.728951] megaraid_sas 0000:02:00.0: jbod sync map         : no
[    1.729017] scsi host0: Avago SAS based MegaRAID driver
[    1.730804] scsi 11:0:0:0: Processor         Marvell  Console          1.01 PQ: 0 ANSI: 5
[    1.732057] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732076] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732094] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732112] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732131] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732149] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732167] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732185] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732206] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732224] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured
[    1.732242] scsi_alloc_sdev: Allocation failure during SCSI scanning, some SCSI devices might not be configured

That allocation failure looks concerning, but when I tried googling it, I didn't find much. One person was trying to do over 200 logical devices, but we're definitely not doing that here.

Thanks in advance.

I have a network appliance based on Busybox (on a Zynq device), and it has two NICs. One network has access to the internet, while the other is private. Both have DHCP, so both are reporting gateways, even though the private network doesn't have Internet access. For some reason, traffic for things like ntpd, rdate, and ping is being routed to the private network. We need to get NTP working, but the Busybox ntpd doesn't have a -I option. We've tried deleting the gateway (route del default eth1, IIRC) from the NIC for the private network, but that results in making the device completely inaccessible.

At a minimum, I need to figure out how to just get NTP to work. More generally, it would be nice (but not critical at the moment) to figure out how to set up a default route for all outgoing traffic to go to the public network if it doesn't match the IP range of the private one. Any help will be most appreciated.

I have a server from Red Barn that has been throwing errors accessing its local SATA flash drive. Here are some facts:

• I'm running Ubuntu 15.10, although two other similar Supermicro servers are as well, with no trouble.
• The root drive is a SATA flash drive.
• If I reboot the system, it'll seem fine at least a day and then gets stuck throwing these errors all the time.
• We tried reseating all the RAMs and running memtest86 for days, with no problems.
• We booted the system from a USB stick, with the root drive not attached and watched it have no problems for days.
• We booted from USB, mounted the drive in question, and had a script touch a file once every 5 seconds. This ran for several days with no errors.

We thought about OS corruption, but why would it wait so long before manifesting? If the drive is failing, why does SMART report nothing, and why does it seem to work for a long time before going weird?

What else can we do to investigate this failure? We're kinda stuck.

Here's a screenshot from the remote console. I see this if I try to login, and then it returns me to the login problem.

Although this question involves an embedded motherboard, serverfault seemed to me the best stackexchange forum to post to. My colleagues and I have been investigating this weird Linux boot problem for a few months now, and we're kinda stuck. Any suggestions are appreciated.

We have a Portwell motherboard (single Atom core with hyperthreading), running Centos 6.4. A customer has come back to us with a really weird boot problem that we've finally been able to reproduce.

Everything works fine if you do this:

• Boot up normally
• Halt the system
• Unplug power from the system for any period of time
• Plug in power
• Boot up

However, if you do the following, the usual fsck that is run as a normal part of boot will give us an error:

• Boot up normally
• Shut down normally
• DO NOT DISCONNECT POWER
• Wait 8 to 12 hours (shorter doesn't seem to cause the problem)
• Boot up again

The error we get is as in the following image:

We can press ctrl-D and reboot as many times as we want, and the error will keep coming back. But note that there's nothing wrong with the filesystem.

We can get the error to go away this way:

• Shut down
• Pull power for at least 10 minutes (much shorter, and the problem doesn't go away)
• Plug power back in and boot up.

Our hypothesis as of yesterday was that the hard drive might be spun down but not off, and over time, the disk cache might get degraded. However, that turns out to be false because the following procedure also makes the problem go away:

• Get the system to boot up so that it gives the error
• Do not disconnect power
• Go into the BIOS and change the system date to some time in the future
• Boot into Linux.

This motherboard does not have a battery, so when we first power on, it always comes up with the wrong date, some time in January of 2010. So in the usual case, the date is wrong, but the OS boots up normally. When the OS comes up, the date is set properly by NTP. If we leave it plugged in but off for 12 hours, the date gets reset again, but for some reason, fsck now cares about the date and wants to so a manual fsck because it considers that discrepancy to be a major problem. If we manually change the date to the future, it boots fine. If we change it back to the past, it errors again. But if we disconnect power for long enough and boot up, we get no error despite the fact that the date is wrong.

Can anyone help us reason through the various things that fsck might be looking at such that is decides sometimes to error out due to the date being wrong but never if the system date is in the future?

If we can program the BIOS to default to some date in the far future, that might solve this problem, but it's important to understand why it's happening, because we don't just want to stick on a bandaid and hope.

Thanks for any suggestions.

I have a 24-core server with 64G of DRAM, and it's running Ubuntu Server 15.10. I am getting some strange behavior with respect to swapping and physical memory use. I never had this trouble until I upgraded Ubuntu to 15.10.

You can see my problem in 'top':

top - 18:52:09 up 1 day, 2:25, 3 users, load average: 1.64, 1.30, 1.18
Tasks: 525 total, 2 running, 523 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 1.3 sy, 0.0 ni, 97.0 id, 1.4 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65937528 total, 37526160 used, 28411368 free, 14396 buffers
KiB Swap: 67071996 total, 67071724 used, 272 free. 104304 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
363 root 20 0 0 0 0 R 100.0 0.0 324:02.98 kswapd0 
6725 theosib 20 0 99.398g 0.034t 8920 D 12.0 55.1 59:08.71 common_shell_ex 

There's a single user logged in (me), and I have a single process using a large amount of virtual memory. However, something is limiting it to around half the physical memory, while the swap partition is basically full. I've watched these processes (Synopsys Design Compiler) run, and they don't break the 50% mark until swap fills. And another weird thing is that kswapd0 uses very high (usually 100%) CPU. AFAIK, kswapd0 should be I/O bound and therefore not use a lot of CPU time.

I've looked to see if there were any limits being imposed, but ulimit says otherwise:

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 257447
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 257447
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I've also tried setting swappiness to 10, but that didn't help.

Can anyone help me figure out why this system is misbehaving?

I'm developing for a headless embedded appliance, running CentOS 6.2. The user can connect a keyboard, but not a monitor, and a serial console would require opening the case, something we don't want the user to have to do. This all pretty much obviates the possibility of using a recovery USB drive to boot from, unless all it does is blindly reimage the harddrive. I would like to provide some recovery facilities, and I have written a tool that comes up on /dev/tty1 in place of getty to provide these functions.

One such function is fsck. I have found out how to remount the root and other file systems read-only. Now that they are read-only, it should be safe to fsck them and then reboot. Unfortunately, fsck complains to me that the filesystems are mounted and refuses to do anything.

How can I force fsck to run on a read-only mounted partition?

Based on my research, this is going to have to be something obscure. "-f" just means to force repair of a clean (but unmounted) partition. I need to repair a clean or unclean mounted partition. From what I read, this is something "only experts" should do, but no one has bothered to explain how the experts do it. I'm hoping someone can reveal this to me.

BTW, I've noticed that e2fsck 1.42.4 on Gentoo will let you fsck a mounted partition, even mounted read-write, but it seems only to do so if fsck is run from a terminal, so it can ask the user if they're sure they want to do something so dangerous. I'm not sure if the CentOS version does the same thing, but it appears that fsck CAN repair a mounted partition, but it flatly refuses to when not run from a terminal.

One last-resort option is for me to compile my own hacked fsck. But I'm afraid I'll mess it up in some unexpected way.

Thanks!

Note: Originally posted here.

Update: I didn't think it would matter at the time I wrote this, but in order to remount the fs read-only, I had to do this:

echo s > /proc/sysrq-trigger
echo s > /proc/sysrq-trigger
echo u > /proc/sysrq-trigger

That was the only way I could find to do this. Everything else complained about the file system being busy. As far as I know, this is 'safe', but it probably remounts a bit differently from the usual approach. And this may be a reason why fsck doesn't want to repair it. It still thinks it's mounted read-write.