I'd like to get a feel for how 'efficient' my deployment/jobs are at consuming the resources they request. i.e., if a job that only ends up using 1CPU at peak ends up requesting 320, I'd like to have a dashboard/alert/metric to chase down rogue pods that meet this criteria.
Does such a thing exist? Closest I've found is grafana + promql, but ideally a ready made dashboard or other solution would be great.
I'm running this on an on-premise kubernetes cluster.
I have a Dell Poweredge server and Netgear 48-port Gigabit switch, all attached to a CyberPower 1500 VA 900W UPS.
We occasionally get power outages here, and when we do, the systems shut down gracefully, but when the power comes back on, the UPS starts screaming and the devices stay off. The only solve I have right now is to disconnect the server, let the switch power up, and then plug back in the server.
Does this mean I have the wrong UPS for this? Can I somehow 'stage' startups of the devices?
The server is running Funtoo linux, if it helps.
I'm a bad person and haven't backed up my RAID elsewhere. I now have a RAID10 array that won't assemble, and I'm hopeful I can save it. Here's the details:
I have five hard drives set up as RAID10 (4+1 Spare). For unknown reasons, two failed and my spare was broken, and now I can't seem to reassemble them.
I have some information below, notably the output of mdstat after a reboot [1], the output of stopping the array and attempting to reassemble it [2] and detailed information from mdadm -E [3].
The literature seems to indicate that I can likely force recreate the RAID array using --create and --assume-clean, but I'm worried it will aggravate my data.
My current thoughts are to recreate the array in a degraded state using the drives with the highest matching events (sdc1, sdd1 and sde1, below). Is there a better solution?
megatron ~ # mdadm -E /dev/sd[bcdefghijklmnop]1 | egrep 'Event|/dev'
/dev/sdb1:
Events : 494734
/dev/sdc1:
Events : 502154
/dev/sdd1:
Events : 502154
/dev/sde1:
Events : 502154
/dev/sdf1:
Events : 494756
[1] This is the output of mdstat, after a reboot:
megatron ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid10] [raid6] [raid5] [raid4]
md0 : inactive sde1[3](S) sdf1[2](S) sdc1[1](S) sdd1[4](S) sdb1[0](S)
7814052444 blocks super 1.2
unused devices: <none>
[2] If I stop the array and reassemble it, I get the following:
megatron ~ # mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sde2
mdadm: /dev/sde2 has wrong uuid.
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sde has wrong uuid.
mdadm: no RAID superblock on /dev/sdf
mdadm: /dev/sdf has wrong uuid.
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdd has wrong uuid.
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdc has wrong uuid.
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sdb has wrong uuid.
mdadm: cannot open device /dev/sda3: Device or resource busy
mdadm: /dev/sda3 has wrong uuid.
mdadm: cannot open device /dev/sda2: Device or resource busy
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: /dev/sda1 has wrong uuid.
mdadm: cannot open device /dev/sda: Device or resource busy
mdadm: /dev/sda has wrong uuid.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot -1.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0.
mdadm: added /dev/sdb1 to /dev/md0 as 0
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: added /dev/sdf1 to /dev/md0 as -1
mdadm: added /dev/sdc1 to /dev/md0 as -1
mdadm: added /dev/sdd1 to /dev/md0 as 2
mdadm: /dev/md0 assembled from 2 drives and 2 spares - not enough to start the array.
[3] Lastly, the output of mdadm -E /dev/sd[bcdefghijklmnop]1 yields:
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 09ce480d:aad5f0ac:4cfbc777:7303d4a7
Update Time : Tue Jul 9 20:16:22 2013
Checksum : 8a17d4c9 - correct
Events : 494734
Layout : near=2
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAA ('A' == active, '.' == missing)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : d89673c0:5efc51a7:1589bbc2:d7002e7d
Update Time : Wed Jul 10 00:14:04 2013
Checksum : 57efa473 - correct
Events : 502154
Layout : near=2
Chunk Size : 512K
Device Role : spare
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 08542b46:f4cfa093:31650e72:dd781e74
Update Time : Wed Jul 10 00:14:04 2013
Checksum : aa71a2c3 - correct
Events : 502154
Layout : near=2
Chunk Size : 512K
Device Role : Active device 2
Array State : ..AA ('A' == active, '.' == missing)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 8c655b9d:4630e93b:32eb61b8:f7a5f513
Update Time : Wed Jul 10 00:14:04 2013
Checksum : 55dcae05 - correct
Events : 502154
Layout : near=2
Chunk Size : 512K
Device Role : Active device 3
Array State : ..AA ('A' == active, '.' == missing)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 01516d30:2e2c0bc4:b743e476:12445ecf
Name : megatron:1 (local to host megatron)
Creation Time : Thu Feb 3 23:38:56 2011
Raid Level : raid10
Raid Devices : 4
Avail Dev Size : 2930269954 (1397.26 GiB 1500.30 GB)
Array Size : 5860538368 (2794.52 GiB 3000.60 GB)
Used Dev Size : 2930269184 (1397.26 GiB 1500.30 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 3b54d74e:a1065dd6:d3e836ca:b340b56d
Update Time : Tue Jul 9 20:17:04 2013
Checksum : d63b6fa - correct
Events : 494756
Layout : near=2
Chunk Size : 512K
Device Role : spare
Array State : .AAA ('A' == active, '.' == missing)
I'm finding that on occasion my Linux box runs out of memory and it starts tearing down random processes to deal with it.
I'm curious what administrators do to avoid this? Is the only real solution to up the amount of memory (will upping the swap alone help?), or is there better ways to set up the box with software to avoid this? (i.e., quotas, or some such?).