King David's questions

from my understanding Using 20-30TB disks with HDFS can present some challenges, but it can also be managed effectively with proper configuration

using 20-30TB disks with HDFS is possible, it requires careful consideration of block size, rebuild times, data distribution, metadata management, and performance. Proper planning and configuration can help mitigate these challenges.

Performance: Large disks can lead to longer seek times and potentially impact performance, especially for workloads that require frequent random access.

based on above can we intend to use disks of 20T-30T on our new data nodes machines?

Note we intend to install from scratch 16 data nodes machines based on DELL HW , when each data node should contain 12 NON-RAID disks (when each disk size is ~22T)

we have some red-hat servers with the following details ( when OS disk size is 230G )

# lsblk
NAME             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                8:0    0  230G  0 disk
├─sda1             8:1    0    1G  0 part /boot/efi
├─sda2             8:2    0    1G  0 part /boot
└─sda3             8:3    0  228G  0 part
  ├─VG100-lv_root 253:0    0   20G  0 lvm  /
  ├─VG100-lv_swap 253:1    0   16G  0 lvm  [SWAP]
  └─VG100-lv_var  253:2    0   30G  0 lvm  /var


fdisk -l

Disk /dev/sda: 193.3 GB, 193273528320 bytes, 377487360 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1   209715199   104857599+  ee  GPT

the Goal is to increase the /var and root filesystem based on that disk size is 230G when current /var is 30G and root filesystem is 20G

so, we start with the following procedure that should gives us enough PFREE or enough “Physical Free” space.

parted /dev/sda resizepart 3 100%
Error: The backup GPT table is not at the end of the disk, as it should be.  This might mean that another operating system believes the disk is smaller.  Fix, by moving the backup to the end (and removing the old backup)?
parted: invalid token: 3
Fix/Ignore/Cancel?


sgdisk -e /dev/sda

note - sgdisk: The simplest way to fix the backup GPT table is by using the sgdisk command with the -e option, which moves the backup GPT data structures to the end of the disk

parted /dev/sda resizepart 3 100%
Information: You may need to update /etc/fstab.

partprobe /dev/sda

pvresize /dev/sda
  Failed to find physical volume "/dev/sda".
  0 physical volume(s) resized or updated / 0 physical volume(s) not resized

but from pvs Pfree is still 0

# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda3  VG100 lvm2 a--  66.00g    0

any idea about other approach that able to increase the /var and root filesystem partitions?

more info

# vgs
  VG   #PV #LV #SN Attr   VSize    VFree
  VG100  1   3   0 wz--n- <228.00g <162.00g

we also try this ( for example )

lvextend -l +100%FREE -r /dev/mapper/VG100-lv_root
  Size of logical volume VG100/lv_root unchanged from 20.00 GiB (5120 extents).
  Logical volume VG100/lv_root successfully resized.

xfs_growfs /dev/mapper/VG100-lv_root

but the size of root filesystem is the same

df -h | grep VG100-lv_root
/dev/mapper/VG100-lv_root   20G  3.8G   17G  19% /

 pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/sda3  VG100 lvm2 a--  66.00g    0

we have kafka cluster with 3 VM machines. , when each kafka machines use the sdb disk ( VMDK disk ) in order to store the data

on all machines we seen the following kernel messages

[1123783.849575] EXT4-fs (sdb): error count since last fsck: 9
[1123783.849582] EXT4-fs (sdb): initial error at time 1595958527: ext4_writepages:2414
[1123783.849586] EXT4-fs (sdb): last error at time 1613639279: ext4_put_super:791
[1210205.709917] EXT4-fs (sdb): error count since last fsck: 9
[1210205.709937] EXT4-fs (sdb): initial error at time 1595958527: ext4_writepages:2414
[1210205.709944] EXT4-fs (sdb): last error at time 1613639279: ext4_put_super:791
[1296627.570121] EXT4-fs (sdb): error count since last fsck: 9
[1296627.570141] EXT4-fs (sdb): initial error at time 1595958527: ext4_writepages:2414
[1296627.570147] EXT4-fs (sdb): last error at time 1613639279: ext4_put_super:791
[1383049.419003] EXT4-fs (sdb): error count since last fsck: 9
[1383049.419019] EXT4-fs (sdb): initial error at time 1595958527: ext4_writepages:2414
[1383049.419025] EXT4-fs (sdb): last error at time 1613639279: ext4_put_super:791
[1469471.269771] EXT4-fs (sdb): error count since last fsck: 9
.
.
.

red hat explain this messages as the following. ( from case - https://access.redhat.com/solutions/383993 )

Issue

I see the following lines in /var/log/messages:

kernel: EXT4-fs (sdd1): error count: 5
kernel: EXT4-fs (sdd1): initial error at 1369732760: ext4_lookup:1044: inode 11534366
kernel: EXT4-fs (sdd1): last error at 1369733705: ext4_lookup:1044: inode 11534366

Resolution These are not errors, they're informational messages; however, they may be referencing other possible historical errors. These error counts should be reset once a successfully fsck has been run; however prior to e2fsprogs-1.41.12-18 a bug was preventing the reset. This has been corrected in e2fsprogs-1.41.12-18 via errata.

the messages that we get on our Kafka cluster are little diff from the redhat case

so we are more worry about the sdb disks ,

from what red hat say they not so worry because they are explained the messages as they're informational messages

so about my kernel messages, I can umount the disk from mount-point and do fsck in order to fix the Erros ,

but my question is how much I need to be worry about the following messages:

[1123783.849575] EXT4-fs (sdb): error count since last fsck: 9
[1123783.849582] EXT4-fs (sdb): initial error at time 1595958527: ext4_writepages:2414
[1123783.849586] EXT4-fs (sdb): last error at time 1613639279: ext4_put_super:791
[1210205.709917] EXT4-fs (sdb): error count since last fsck: 9
[1210205.709937] EXT4-fs (sdb): initial error at time 1595958527: ext4_writepages:2414

how to know the number of concurrent connections per port on RHEL 7.x machines

for instance, let say we connect to port 9080 with XXXX concurrent connections.

wget  http://server_mngmnt.jupiter.com:9080 &
wget  http://server_mngmnt.jupiter.com:9080 &
wget  http://server_mngmnt.jupiter.com:9080 &
wget  http://server_mngmnt.jupiter.com:9080 &
wget  http://server_mngmnt.jupiter.com:9080 &
.
.
.

second - what could be the results of huge concurrent connections to port 9080 ?

from my machine

more /etc/security/limits.conf


# End of file
* soft nofile 25000
* hard nofile 25000

sysctl -n fs.file-max
500000

ulimit -n
63536

The swpd field indicates how much swap space has been used; this value increases when the systems physical memory is full and the Linux kernel starts to use the swap partition/file. When the systems physical memory and swap space has been exhausted

from our RHEL 7.2 machine we can see the following

vmstat 1 20
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0 1029076 6695152      4 49021608    1    1   520    32    2    2  9  1 90  0  0
 2  0 1029076 6694024      4 49022924    0    0 19612     0 5761 4269  7  1 91  0  0
 8  0 1029076 6665220      4 49021688    0    0 14812   707 22450 26191 13  3 84  0  0
 2  0 1029076 6656396      4 49021704    0    0 15748     0 17994 21252  9  2 88  0  0
 2  0 1029088 6649536      4 49022448    0   12 33972    12 13426 14773  8  2 90  1  0
 2  0 1029088 6687988      4 49022604    0    0 31212     0 6085 4391  7  2 91  0  0
 2  0 1029088 6688688      4 49022340    0    0 27040     9 7001 8469  4  1 95  1  0
 3  0 1029092 6689572      4 49022820    0    4 12136    28 3390 2773  4  0 96  0  0
 1  0 1029092 6689856      4 49022224    0    0 13776     0 3460 2841  4  0 96  0  0
 1  0 1029092 6690356      4 49021820    0    0 18444     0 3445 2819  4  0 96  0  0
 1  0 1029092 6689864      4 49021520    0    0 18768     0 3743 3005  4  0 96  0  0
 1  0 1029092 6693856      4 49021452    0    0 17544     0 3406 2732  4  0 96  0  0
 1  0 1029092 6694048      4 49021208    0    0 24244   292 4654 4816  4  1 95  0  0
 1  0 1029092 6695096      4 49021312    0    0 15572     0 3431 2835  4  0 96  0  0
 2  0 1029092 6694536      4 49022072    0    0 17476     0 5065 4033  7  1 91  0  0
 2  1 1029092 6685364      4 49022140    0    0 28112     9 24914 30798 10  3 86  1  0
 1  0 1029092 6684160      4 49021280    0    0 10356   104 14245 16378  5  2 93  0  0
 1  0 1029092 6671432      4 49022212    0    0 12816    20 11465 13620  5  1 94  0  0
 1  0 1029092 6707700      4 49022364    0    0 19840     0 4113 3603  4  0 95  0  0
 2  0 1029092 6706948      4 49022364    0    0 18128     0 3324 2833  4  0 96  0  0

dose high values of swpd indicate on problem?

 free -g
              total        used        free      shared  buff/cache   available
Mem:            122          69           6           8          46          43
Swap:            15           0          14


vmstat 1 20
    128195440 K total memory
     72465912 K used memory
     77700088 K active memory
     36918416 K inactive memory
      6707164 K free memory
            4 K buffer memory
     49022360 K swap cache
     16351228 K total swap
      1029076 K used swap
     15322152 K free swap
    137196351 non-nice user cpu ticks
          778 nice user cpu ticks
     21836716 system cpu ticks
   1437416566 idle cpu ticks
      7095049 IO-wait cpu ticks
            0 IRQ cpu ticks
      1280657 softirq cpu ticks
            0 stolen cpu ticks
   8346063429 pages paged in
    515040249 pages paged out
      3267044 pages swapped in
      5005632 pages swapped out
   4242627451 interrupts
   1713287115 CPU context switches
   1705353038 boot time
     31037695 forks



sar -B 2 5 


03:37:14 PM  pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
03:37:16 PM      0.00      6.00    133.50      0.00     93.00      0.00      0.00      0.00      0.00
03:37:18 PM      0.00      0.00     34.50      0.00     91.50      0.00      0.00      0.00      0.00
03:37:20 PM      0.00      4.50  50683.50      0.00  19022.00      0.00      0.00      0.00      0.00
03:37:22 PM     60.00      2.00  11028.50      0.00  10382.00      0.00      0.00      0.00      0.00
03:37:24 PM      0.00      0.00    805.00      0.00   1205.50      0.00      0.00      0.00      0.00
Average:        12.00      2.50  12537.00      0.00   6158.80      0.00      0.00      0.00      0.00


sar -d 

03:34:42 PM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
03:34:52 PM       sda    448.60  20769.60     66.30     46.45      0.60      1.35      0.15      6.73

we installed RHEL 7.9 version on physical machine., when sda device was the OS

machine have the following disks.

sda - OS disk
sdb - data disk
sdc - data disk

after reboot OS disk device changed to sdc as the following and sda became as part of data disk

sdc                  8:32   0 558.4G  0 disk
├─sdc1               8:33   0   500M  0 part /boot
└─sdc2               8:34   0 557.9G  0 part
  ├─VG-VOL_root   253:0    0   100G  0 lvm  /
  ├─VG-VOL_swap   253:1    0    16G  0 lvm
  ├─VG-VOL_var    253:2    0   200G  0 lvm  /var

we assume on some next reboots sdc maybe ? will change again to sda as should be

but how to prevent sda disk to change to another device? after reboot ?

according to red-hat doc they recommended to set the following:

users can modify kernel boot parameters by appending "scsi_mod.scan=sync" in the /etc/sysconfig/grub file to get more consistent disk lette It will not guarantee consistent device names in all cases. This option should be used with caution as it slows down disk discovery procedure and thus whole boot process.

but is this configuration to set scsi_mod.scan=sync in /etc/sysconfig/grub is really helps?

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=VG/VOL_root rd.lvm.lv=VG/lv_swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
scsi_mod.scan=sync            <---------------

what are the other options that we can to do in order to avoid sda disk device to change after reboot?

other some useful details from the server

 ls -ltr /dev/disk/by-id  | grep sdc
lrwxrwxrwx 1 root root  9 Dec 28 11:43 wwn-0x6588a5a0ef7f4d0023880046051e7a74 -> ../../sdc
lrwxrwxrwx 1 root root  9 Dec 28 11:43 scsi-36588a5a0ef7f4d0023880046051e7a74 -> ../../sdc
lrwxrwxrwx 1 root root 10 Dec 28 11:43 wwn-0x6588a5a0ef7f4d0023880046051e7a74-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Dec 28 11:43 scsi-36588a5a0ef7f4d0023880046051e7a74-part2 -> ../../sdc2
lrwxrwxrwx 1 root root 10 Dec 28 11:43 lvm-pv-uuid-2Ee6HA-9vVx-bUVq-noyG-U0Nq-AVMz-oI56ji -> ../../sdc2
lrwxrwxrwx 1 root root 10 Dec 28 11:43 wwn-0x6588a5a0ef7f4d0023880046051e7a74-part1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Dec 28 11:43 scsi-36588a5a0ef7f4d0023880046051e7a74-part1 -> ../../sdc1



cat /etc/fstab | grep boot
UUID=f57f0346-9e6c-429e-b80f-3871334bae35 /boot xfs defaults 0 0

We have 254 Physical servers when all machines are DELL servers R740.

servers are part of Hadoop cluster. most of them are holding HDFS filesystem and data node & node manager services, part of them are Kafka machines.

The OS that installed on the physical servers is RHEL 7.9

Now we want to add additional 52 physical servers to the cluster. but we can add only the DELL servers R760 That must be installed with RHEL 8.6

each physical server includes 256G and 64 Cores.

The problem is that we can't works with RHEL 8.6 version because Hadoop version. not fit RHEL 8.6

Therefore, we need to stay with RHEL 7.9 version but on other hand DELL server R760 can't be alignment with RHEL 7.9

so, we are thinking to use VM machines instead of DELL R760 physical machines.

it's clear for us that VM performance isn't like the physical machines.

but let's say. if we increase the default memory from 256G to 384G and increase the COREs from 64 to 80 Core's, on VM machine/s

So, in case we set VM machine as above spec with more memory and CPU's

then is it make sense to add the additional VM machines? to Hadoop cluster and not Worry about performance issues?

In essence the Page Cache is a part of Virtual File System (VFS) which main purpose, as you can guess, is improving IO latency of read and write operations.

In computing, a page cache, sometimes also called disk cache, is a transparent cache for the pages originating from a secondary storage device such as a hard disk drive (HDD) or a solid-state drive (SSD).

The operating system keeps a page cache in otherwise unused portions of the main memory (RAM), resulting in quicker access to the contents of cached pages and overall performance improvements.

Based on the above, what are the tools that already exists in order to monitoring the page cache? I wish to monitor usage, hits, and especially the dirty pages.

OS version is RHEL 7.X.

we have few DELL machines ( with RHEL 7.6) , and as we replaced the DIMM cards on machines because the Erros that we seen from kernel messages

after some time we checked again the kernel messages and we found the following and we can see the errors about the RAM memory ( also related to RHEL case - https://access.redhat.com/solutions/6961932 )

[Mon May  8 21:08:01 2023] EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1683580080 SOCKET 0 APIC 0
[Mon May  8 21:08:01 2023] EDAC MC0: 0 CE memory read error on CPU_SrcID#0_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x6f3c77 offset:0xc80 grain:32 syndrome:0x0 -  area:DRAM err_code:0000:009f socket:0 ha:0 channel_mask:2 rank:4)
[Mon May  8 21:08:21 2023] mce: [Hardware Error]: Machine check events logged
[Tue May  9 05:30:29 2023] {13}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4
[Tue May  9 05:30:29 2023] {13}[Hardware Error]: It has been corrected by h/w and requires no further action
[Tue May  9 05:30:29 2023] {13}[Hardware Error]: event severity: corrected
[Tue May  9 05:30:29 2023] {13}[Hardware Error]:  Error 0, type: corrected
[Tue May  9 05:30:29 2023] {13}[Hardware Error]:  fru_text: B6
[Tue May  9 05:30:29 2023] {13}[Hardware Error]:   section_type: memory error
[Tue May  9 05:30:29 2023] {13}[Hardware Error]:   error_status: 0x0000000000000400
[Tue May  9 05:30:29 2023] {13}[Hardware Error]:   physical_address: 0x000000446e0d5f00
[Tue May  9 05:30:29 2023] {13}[Hardware Error]:   node: 1 card: 1 module: 1 rank: 0 bank: 3 row: 64982 column: 888 
[Tue May  9 05:30:29 2023] {13}[Hardware Error]:   error_type: 2, single-bit ECC
[Tue May  9 05:30:29 2023] EDAC sbridge MC0: HANDLING MCE MEMORY ERROR
[Tue May  9 05:30:29 2023] EDAC sbridge MC0: CPU 0: Machine Check Event: 0 Bank 1: 940000000000009f
[Tue May  9 05:30:29 2023] EDAC sbridge MC0: TSC 30d2ef7e9bfda 
[Tue May  9 05:30:29 2023] EDAC sbridge MC0: ADDR 446e0d5f00 
[Tue May  9 05:30:29 2023] EDAC sbridge MC0: MISC 0 
[Tue May  9 05:30:29 2023] EDAC sbridge MC0: PROCESSOR 0:406f1 TIME 1683610228 SOCKET 0 APIC 0
[Tue May  9 05:30:29 2023] EDAC MC1: 0 CE memory read error on CPU_SrcID#1_Ha#0_Chan#1_DIMM#1 (channel:1 slot:1 page:0x446e0d5 offset:0xf00 grain:32 syndrome:0x0 -  area:DRAM err_code:0000:009f socket:1 ha:0 channel_mask:2 rank:4)
[Tue May  9 05:30:51 2023] mce: [Hardware Error]: Machine check events logged
[Tue May  9 17:52:21 2023] perf: interrupt took too long (380026 > 7861), lowering kernel.perf_event_max_sample_rate to 1000
[Wed May 10 06:27:17 2023] warning: `lshw' uses legacy ethtool link settings API, link modes are only partially reported

just to be sure that this above messages are not random messages we decided to reboot machines and see if the bad messages about memory are reproduced

but the Erros messages about the RAM memory, are still remained.

so we are confused about the problem that we seen from kernel messages

how it can be that we still get Erros about RAM in spite we replaced the DIMM cards

I must gives here additional information about what we see from IDRAC

as we can above IDRAC not completed about the DIMM cards or about the RAM memory

so the question is - how comes dmesg ( kernel messages ) are complained about the RAM memory in spite all DIMMs was replaced?

is it possible that something else is BAD and not the DIMM cards? for example the motherboard in DELL machine?

here is example from dmesg output from important production server ( RHEL 7.2 - DELL machine HW ) as we can see the sde disk in server is dying

[Wed Jun 30 11:24:58 2021] sd 0:2:4:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Jun 30 11:26:18 2021] sd 0:2:4:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Jun 30 11:26:18 2021] sd 0:2:4:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Jun 30 11:27:28 2021] sd 0:2:4:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Jun 30 11:27:46 2021] sd 0:2:4:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE

what is interesting is that these messages are old from 2021 , and we not seen this messages on 2022/2023

based on that facts, I want to ask if disk replacement should be considered based on faulty disk messages from 2021

second important question, is how to capture new fresh kernel messages by dmesg

is it possible to re-create new fresh kernel messages ?

as I know maybe reboot machine can helps about this , but I want to avoid machine reboot

We are using HDP cluster with 182 data node machines:

HDP version - 2.6.4 Ambari version 2.6.1

We note the following behavior on the data nodes machines (its happens on all data-node machines and on all disks).

When we perform the command as above example:

ps -eo s,user,cmd | grep ^[RD]
D hdfs     du -sk /grid/sdj/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
D hdfs     du -sk /grid/sdm/hadoop/hdfs/data/current/BP-1018134753-10.3.6.170-1530088122990
R root     ps -eo s,user,cmd

Note - each disk in the data node is 5.4 T Bytes.

We can see that HDFS is running the "du -sk" on the data node disks

We don't like this, because the meaning of that is consuming high load CPU Avrg and sometimes even bad performance.

We are understand that HDFS need to run the "du -sk" in order to verify the disks space, but on other hand its cost - high CPU load avrg and sometimes even poor performance.

Is it possible to tell HDFS in some way to disable this verification?

On our RHEL 7.2 server we saw a strange kernel messages with lines as:

Disabling DIF Type 2 protection
***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)

Above messages are relevant for all disks in the server ( DELL HW MACHINE )

Example of dmesg

[    4.637635] sd 0:2:0:0: [sdm] ***10DIGITS*** 512-byte logical blocks: (599 GB/558 GiB)
[    4.637668] sd 0:2:0:0: [sdm] Write Protect is off
[    4.637670] sd 0:2:0:0: [sdm] Mode Sense: 1f 00 00 08
[    4.637690] sd 0:2:0:0: [sdm] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[    4.637705] sd 0:0:5:0: [sdf] Disabling DIF Type 2 protection
[    4.637713] sd 0:0:5:0: [sdf] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.638024]  sdm: sdm1 sdm2
[    4.638914] sd 0:2:0:0: [sdm] Attached SCSI disk
[    4.639623] sd 0:0:0:0: [sda] Disabling DIF Type 2 protection
[    4.639630] sd 0:0:0:0: [sda] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.640577] sd 0:0:5:0: [sdf] Write Protect is off
[    4.640580] sd 0:0:5:0: [sdf] Mode Sense: d7 00 10 08
[    4.641773] sd 0:0:5:0: [sdf] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.642697] sd 0:0:0:0: [sda] Write Protect is off
[    4.642699] sd 0:0:0:0: [sda] Mode Sense: d3 00 10 08
[    4.643679] sd 0:0:7:0: [sdh] Disabling DIF Type 2 protection
[    4.643681] sd 0:0:7:0: [sdh] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.643969] sd 0:0:2:0: [sdc] Disabling DIF Type 2 protection
[    4.643975] sd 0:0:2:0: [sdc] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.644348] sd 0:0:8:0: [sdi] Disabling DIF Type 2 protection
[    4.644354] sd 0:0:8:0: [sdi] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.645693] sd 0:0:1:0: [sdb] Disabling DIF Type 2 protection
[    4.645696] sd 0:0:1:0: [sdb] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.645845] sd 0:0:4:0: [sde] Disabling DIF Type 2 protection
[    4.645848] sd 0:0:4:0: [sde] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.646360] sd 0:0:7:0: [sdh] Write Protect is off
[    4.646362] sd 0:0:7:0: [sdh] Mode Sense: d7 00 10 08
[    4.646780] sd 0:0:2:0: [sdc] Write Protect is off
[    4.646781] sd 0:0:2:0: [sdc] Mode Sense: d7 00 10 08
[    4.647030] sd 0:0:8:0: [sdi] Write Protect is off
[    4.647035] sd 0:0:8:0: [sdi] Mode Sense: d7 00 10 08
[    4.647087] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.647365] sd 0:0:3:0: [sdd] Disabling DIF Type 2 protection
[    4.647368] sd 0:0:3:0: [sdd] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.647622] sd 0:0:7:0: [sdh] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.647977] sd 0:0:2:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.648294] sd 0:0:8:0: [sdi] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.648389] sd 0:0:1:0: [sdb] Write Protect is off
[    4.648390] sd 0:0:1:0: [sdb] Mode Sense: d7 00 10 08
[    4.648514] sd 0:0:4:0: [sde] Write Protect is off
[    4.648516] sd 0:0:4:0: [sde] Mode Sense: d7 00 10 08
[    4.649369] sd 0:0:9:0: [sdj] Disabling DIF Type 2 protection
[    4.649372] sd 0:0:9:0: [sdj] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.649578] sd 0:0:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.649708] sd 0:0:4:0: [sde] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.650046] sd 0:0:3:0: [sdd] Write Protect is off
[    4.650052] sd 0:0:3:0: [sdd] Mode Sense: d7 00 10 08
[    4.650371] sd 0:0:16:0: [sdl] Disabling DIF Type 2 protection
[    4.650376] sd 0:0:16:0: [sdl] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.651296] sd 0:0:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.652099] sd 0:0:9:0: [sdj] Write Protect is off
[    4.652101] sd 0:0:9:0: [sdj] Mode Sense: d7 00 10 08
[    4.653189] sd 0:0:6:0: [sdg] Disabling DIF Type 2 protection
[    4.653192] sd 0:0:6:0: [sdg] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.653201] sd 0:0:16:0: [sdl] Write Protect is off
[    4.653203] sd 0:0:16:0: [sdl] Mode Sense: d7 00 10 08
[    4.653285] sd 0:0:9:0: [sdj] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.653495] sd 0:0:14:0: [sdk] Disabling DIF Type 2 protection
[    4.653498] sd 0:0:14:0: [sdk] ***10DIGITS*** 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    4.654397] sd 0:0:16:0: [sdl] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.654474] sd 0:0:5:0: [sdf] Attached SCSI disk
[    4.655990] sd 0:0:6:0: [sdg] Write Protect is off
[    4.655993] sd 0:0:6:0: [sdg] Mode Sense: d7 00 10 08
[    4.656204] sd 0:0:14:0: [sdk] Write Protect is off
[    4.656210] sd 0:0:14:0: [sdk] Mode Sense: d7 00 10 08
[    4.657189] sd 0:0:6:0: [sdg] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.657431] sd 0:0:14:0: [sdk] Write cache: disabled, read cache: enabled, supports DPO and FUA
[    4.660959] sd 0:0:8:0: [sdi] Attached SCSI disk
[    4.662148] sd 0:0:1:0: [sdb] Attached SCSI disk
[    4.662206] sd 0:0:7:0: [sdh] Attached SCSI disk
[    4.662350] sd 0:0:4:0: [sde] Attached SCSI disk
[    4.662387] sd 0:0:2:0: [sdc] Attached SCSI disk
[    4.665915] sd 0:0:9:0: [sdj] Attached SCSI disk
[    4.665985] sd 0:0:3:0: [sdd] Attached SCSI disk
[    4.666962] sd 0:0:16:0: [sdl] Attached SCSI disk
[    4.671749] sd 0:0:14:0: [sdk] Attached SCSI disk
[    4.672000] sd 0:0:6:0: [sdg] Attached SCSI disk
[    4.681089] sd 0:0:0:0: [sda] Attached SCSI disk

Do we need to be worry about those messages ?

Notes:

1. Server is include physical disks as sdb / sdc /sdd /sde etc ..
2. The reason for our investigation is CPU high load avrg

We have a VM Linux server, where the hot add feature is disabled in VCenter.

For now, we can't power off the machine, and the feature is disable, so we can't add memory resources to the machine.

So do you have any idea how to enable the hot add feature without powering off the Linux redhat machine ?

We have two option that we can use in order to install the docker-compose

We already installed docker , and now we want to install docker compose on production environment

One option is by binary docker-compose

curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

the second option is by pip as the following

$ yum install epel-release
Install python-pip
$ yum install -y python-pip
Then install Docker Compose:
$ pip install docker-compose

Are the first option that installed the docker-compose by binary is a good enough as the docker-compose that installed by PIP?

is it possible to run xfs repair by re-edit the fstab file?

/dev/mapper/vg-linux_root /                       xfs     defaults        0 0
UUID=7de1dc5c-b605-4a6f-bdf1-f1e869f6ffb9 /boot   xfs     defaults        0 0
/dev/mapper/vg-linux_var /var                     xfs     defaults        0 0
/dev/mapper/vg-linux_swap swap                    swap    defaults        0 0

I am not sure but by replace the last number from 0 to 1 , is it right?

we do the following on linux rhel machines

ethtool -K ifcfg-478476 tso off

now lets say that we want to verify the settings

how to do the check with ethtool or other cli?

we try

ethtool -k ifcfg-478476 | grep -i tso

but we not get status

any idea ?

note - ifcfg-478476 is the eth link

I do the following steps in order to increase the swap size

but when I start with swapoff /dev/mapper/rootvg-swap , then its stuck for along time and not return the prompt

why we can do to solve this

 swapoff /dev/mapper/rootvg-swap
 lvresize -L +5g  /dev/mapper/rootvg-swap
 mkswap /dev/mapper/rootvg-swap
 swapon /dev/mapper/rootvg-swap


 vmstat 1
 procs -----------memory---------- ---swap-- -----io---- --system-- ----- cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 2027848 127788  28804 18490972  105   38   180   211   25   59 17  6 64 13  0
 1  0 2027848 126944  28812 18490972    0    0     0   124 1073 1491 16  1  82  1  0
 0  0 2027848 126076  28828 18490956    0    0     0   104  727 1236  6  0    94  0  0
 0  0 2027848 126076  28828 18491008    0    0     0     0  452  979  2  0 98  0  0
 1  0 2027696 130960  28828 18486500  480    0   480    92  985 1415 21  4 69  5  0
 0  0 2027696 130984  28844 18486572    0    0     0    43  421  984  1  0 98  1  0

I have linux redhat machine - 6.8

and from the /var/log/message file I see the following lines:

from my point is seems that linux is hung because the bad connection to the server - 100.1.93.7 ,

is it correct ?

 Oct 30 04:39:40  guard_tap[18038]: Delayed server connection to 100.1.93.7 error: Connection timed out
 Oct 30 04:40:31  kernel: INFO: task oracle:16131 blocked for more than 120 seconds.
 Oct 30 04:40:31  kernel:      Tainted: G        W  -- ------------    2.6.32-642.4.2.el6.x86_64 #1
 Oct 30 04:40:31  kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
 Oct 30 04:40:31  kernel: oracle        D 0000000000000001     0 16131      1 0x00000000
 Oct 30 04:40:31  kernel: ffff8801012a7a98 0000000000000082 ffff8801012a79e8 ffffffffa03093ee
 Oct 30 04:40:31  kernel: ffff8801012a7a18 ffffffffa0309c90 ffff880700b057c0 ffff8801012a7a48
 Oct 30 04:40:31  kernel: ffff880700b05870 ffff88082ba2e448 ffff88010c6e3ad8 ffff8801012a7fd8
 Oct 30 04:40:31  kernel: Call Trace:

Why after installing from scratch the iptables or upgrading the iptables

Then iptables from chkconfig display on ?

How to avoid chkconfig iptables on , after installling/upgradin iptables ?

• remark we want to avoid to perform chkconfig iptables off after installing the iptables

Real example from my redhat machine version 6

service iptables status
Firewall is stopped. 

# chkconfig --list | grep iptables
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off

# chkconfig iptables off

# chkconfig --list | grep iptables
iptables        0:off   1:off   2:off   3:off   4:off   5:off   6:off



yum remove iptables
.
.
.


chkconfig --list | grep iptables

show nothing ( as should be )



yum install  iptables
.
.
.

After iptables was installed Successfully , We can see that chkconfig is on - why ?

# chkconfig --list | grep iptables
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off

The critical problem with this is:

After reboot iptables service will be running and we want to avoid this

Please advise how to avoid iptables chkconfig on after installing iptables package

I performed the following procedure in order to add 400m to swap,

I took the 400m from the sdb disk for the swap increasing

Everything was ok but before the last command , I performed

mkswap /dev/rootvg/swap

And I get the following warning messages from mkswap

mkswap: /dev/rootvg/swap: warning: don't erase bootbits sectors
    on whole disk. Use -f to force.
Setting up swapspace version 1, size = 2490364 KiB
no label, UUID=2f42fa37-987d-40a5-8cf6-5707e1d8f077

Finally swap was extend to 2.4G as expected

But I little worry about the warning message from mkswap command

Please advise what is meaning of the warning:

mkswap: /dev/rootvg/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.

And if this is OK?

My procedure on the linux redhat machine - version 6

# swapon -s
Filename                                Type            Size    Used      Priority
/dev/dm-1                               partition       2064380 0       -1

# free
        total       used       free     shared    buffers     cached
Mem:       5991892     539036    5452856        212      70444     270664
-/+ buffers/cache:     197928    5793964
 Swap:      2064380          0    2064380


# pvs
PV         VG      Fmt  Attr PSize  PFree
 /dev/sdb   rootvg  lvm2 a--u  7.97g 416.00m
 /dev/sdc   orahome lvm2 a--u 30.00g  15.00g

# lsblk /dev/sdb
NAME                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb                     8:16   0    8G  0 disk
|-rootvg-swap (dm-1)  253:1    0    2G  0 lvm  [SWAP]
|-rootvg-slash (dm-2) 253:2    0  576M  0 lvm  /
|-rootvg-usr (dm-3)   253:3    0  1.4G  0 lvm  /usr
|-rootvg-home (dm-4)  253:4    0  256M  0 lvm  /home
|-rootvg-opt (dm-5)   253:5    0  416M  0 lvm  /opt
`-rootvg-var (dm-6)   253:6    0    3G  0 lvm  /var

# swapon -s
Filename                                Type            Size    Used     Priority
/dev/dm-1                               partition       2064380 0       -1

# swapoff /dev/rootvg/swap

# lvresize /dev/rootvg/swap -L +400m
 Rounding size to boundary between physical extents: 416.00 MiB.
 Size of logical volume rootvg/swap changed from 1.97 GiB (63 extents) to  2.38 GiB (76 extents).
 Logical volume swap successfully resized.

# mkswap /dev/rootvg/swap
mkswap: /dev/rootvg/swap: warning: don't erase bootbits sectors
    on whole disk. Use -f to force.
Setting up swapspace version 1, size = 2490364 KiB
no label, UUID=2f42fa37-987d-40a5-8cf6-5707e1d8f077

# swapon /dev/rootvg/swap

# free
         total       used       free     shared    buffers     cached
 Mem:       5991892     539680    5452212        212      70536     270892
 -/+ buffers/cache:     198252    5793640
 Swap:      2490364          0    2490364