GP92's questions

We have few requests that takes http headers size more than the default accepted size in our tomcat application i.e., maxHttpHeaderSize = 8192 (https://tomcat.apache.org/tomcat-8.0-doc/config/http.html)

And also nginx, has a default limit for buffers that can be allocated for large client requests i.e., large_client_header_buffers size = 8K (http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers)

However, without increasing the large_client_header_buffers on nginx side, requests can still go through if just maxHttpHeaderSize parameter in tomcat is increased.

As per nginx documentation, "Buffers are allocated only on demand." I do not understand this. Are buffers not allocated for every request?

we collect heap dumps of the resource on failed node, before restarting the resource.

However, to minimize the downtime (as heapdump collection taking time), I'm checking if it is possible to start the resource in new node, without stopping it in the old node.

I could think of some dirty tricks that might work if the stop script uses a pid file to stop the process. But if monitoring of the resouce is successful, then it gets more complicated, as the resource gets blocked etc. etc.

So, is there any option in pacemaker itself to move resources and telling cluster not to stop a specific resource.

I thought the following is possible, but didn't work:

-> make the resource unmanaged by cluster -> make active node standby (all resources always runs in same node for our case) -> Now I expected, all resources to start again in the new node (with the unmanaged resource to not start), so that I could start it manually with pcs resource debug-start.

However, it didn't work as the particular resource is part of a resource group and it blocks the whole resource group from moving to new node.

Any ideas?

What I understood is that some of the Openstack components communicate using RabbitMQ service (which uses RPC protocol) while other communicate using REST APIs, DB queries etc.

But when I check rpcinfo -p, it doesn't list any service related to RabbitMQ/openstack.

What's the difference between the communication mechanism of services which are dispayed in rpcinfo (Eg: NIS, NFS etc.) and RabbitMQ/Openstack service?

What I understood is that with resource agents (which are exactly same on all nodes), pacemaker can start/stop resource on any node. But can we also use it to sync a configuraton file in all nodes in the cluster.

For Example, I want my drbd.conf file to be same on all nodes and want to maintain it with pacemaker cluster. Is it possible and if it is, how?

So, in other words, I do not want the drbd itself to be maintained by pacemaker i.e., stop, start etc, only the configuration file. I am not saying it's an ideal usecase, just checking if it's possible.

I have my system authentication managed with SSSD which uses Kerberos.

As kerberos support mutual authentication model i.e., both client and server should support Kerberos, how exactly does SSH to the server work from any client like putty or another linux machine, irrespective of whether it supports Kerberos?

Or is the sssd daemon acts as the actual client for the Kerberos authentication and if I configure SSH authentication with pure kerberos without SSSD, does the login still work or here does the SSHD daemon itself works as the client for Kerberos authentication.

I am confused here regarding what is client and what is server. Eg: SSH logins, Web based logins etc.

I understand queue depth which is the number of outstanding I/O requests that the storage controller can handle (https://www.tomshardware.com/reviews/ssd-gaming-performance,2991-3.html) i.e., this is the limitation on a storage controller which handles the I/O requests and sends the commands to disk (r/w) and it (not strictly?) drops the requests if there are more than which it can handle (which will be resubmitted by the clients presumably).

And the reason for having high outstading I/O requests could be multiple client connections requesting I/O or multiple processes even from a single host requesting I/O (which I thought, but it seems OS uses I/O scheduler merges the I/O requests - which are originated from buffer when doing periodic or on-demand sync and send only a fixed number of outstading requests, so that it won't overload the storage devices?)

Now, coming to the definition of iodepth in fio man page:

Number of I/O units to keep in flight against the file. Note that increasing iodepth beyond 1 will not affect synchronous ioengines (except for small degrees when verify_async is in use).

This aligns with my understanding of queue depth. If the IO is synchronous (blocking IO), we can have only one queue.

Even async engines may impose OS restrictions causing the desired depth not to be achieved. This may happen on Linux when using libaio and not setting `direct=1', since buffered I/O is not async on that OS.

Confused with this whole statement.

Keep an eye on the I/O depth distribution in the fio output to verify that the achieved depth is as expected. Default: 1.

I have run multiple tests for each iodepth and device type, with 22 parallel jobs as the CPU count is 24 and with rwtype: sequential read and sequential write. Iodepths are 1,16,256,1024,32768 ( I know 32 or 64 should be the maximum limit, I justed wanted to try anyway).

And the results are almost same for all depths and for all disks (RAID 6 SSD,NVME and NFS): except for sequential read on NVME disk with 32768 depth.

IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%

For NVME with 32768 depth,

complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%

I used libaio engine in fio (because I am not sure what IO engine I need to give for asynchronous I/O testing and libaio seemingly the right one. This is a different question altogether)

So, what's going on? Why is Submit and complete shows 1-4 (except for one run of NVME where it's >64)

[global]
lockfile=none
kb_base=1024
fallocate=posix
blocksize=64k
openfiles=100
ioengine=libaio
buffered=1
invalidate=1
loops=5
randrepeat=1
size=512M
numjobs=22

[sr-iodepth-1]
description="Sequential Write,Parallel jobs-22,IO depth-1,libaio"
readwrite=write
size=5G
iodepth=1

[sr-iodepth-16]
description="Sequential Write,Parallel jobs-22,IO depth-16,libaio"
readwrite=write
size=5G
iodepth=16

[sr-iodepth-256]
description="Sequential Write,Parallel jobs-22,IO depth-256,libaio"
readwrite=write
size=5G
iodepth=256

[sr-iodepth-1024]
description="Sequential Write,Parallel jobs-22,IO depth-1024,libaio"
readwrite=write
size=5G
iodepth=1024

[sr-iodepth-32768]
description="Sequential Write,Parallel jobs-22,IO depth-32768,libaio"
readwrite=write
size=5G
iodepth=32768


[sw-iodepth-1]
description="Sequential Read,Parallel jobs-22,IO depth-1,libaio"
readwrite=read
size=512M
iodepth=1

[sw-iodepth-16]
description="Sequential Read,Parallel jobs-22,IO depth-16,libaio"
readwrite=read
size=512M
iodepth=16

[sw-iodepth-256]
description="Sequential Read,Parallel jobs-22,IO depth-256,libaio"
readwrite=read
size=512M
iodepth=256

[sw-iodepth-1024]
description="Sequential Read,Parallel jobs-22,IO depth-1024,libaio"
readwrite=read
size=512M
iodepth=1024

[sw-iodepth-32768]
description="Sequential Read,Parallel jobs-22,IO depth-32768,libaio"
readwrite=read
size=512M
iodepth=32768

I would like to know what will happen to an ESTABLISHED connection between two hosts (say A:9999 & B:8888), after a firewall restriction enabled between those two hosts thus preventing any communication between the hosts or on specific ports.

I think we can expect same result when the network service is down.

My observation is the process (in my case: it's ldapsearch query running on 'host A' with a huge output) stops after few minutes, but also the subsequent commands in the parent process fails to run. So, I am confused what is happening and why the subsequent commands can't continue which doesn't have any network dependency like an echo command.

I understand that SNMP is a protocol for managing (view/update) system resources information remotely and LDAP is a protocol to access and use directory services information.

But is there any correlation between both protocols? Is one dependent on the other? Do I need to understand SNMP, if I need to understand LDAP thoroughly?

The reason I asked this is I am trying to understand what is LDIF and here when talking about it, there are references of SNMP syntax in this page: https://docs.oracle.com/cd/E10773_01/doc/oim.1014/e10531/schema_overview.htm.

I am quite confused if hal is really in use or just udev.

What I understood:

• In general, HAL is an abstraction layer that allows an operating system to interact with a hardware device.

• And daemon hald is different from HAL. It is a service which provides HAL and is used for identifying devices and then mounting them (and where they would be, under /dev?) or configuring them automatically for applications' use.

• And it is now deprecated by udev, which also does a similar thing i.e., automounting the devices as they are connected by reading messages from kernel and naming according to the predefined rules.

• And currently only few GUI based applications like GNOMe use hald, to get notification about newly connected devices (while the mounting is still taken care by udev?)

  So my question is if hal is only used for notifiying GUI based applications about newly connected hardware as it can communicate through DBUS, but udev doesn't have dbus implementation. And for automounting devices, only udev does that and no where hal is used?

And I am paricularly talking about Redhat 5,6 and 7.

Thanks.

Let's say we have a FUSE filesystem.

Will it have a drastic performance impact with the Kaiser patch?

According to my understanding as FUSE filesystem runs completely in userspace and hence are less impacted when compared to Kernel File systems?

It is mentioned here https://access.redhat.com/articles/3307751 that:

benchmarks with high kernel-to-user space transitions were measured to be impacted between 8-19%

HPC (High Performance Computing) CPU-intensive workloads that spend little time in the kernel were measured to have 2-5% performance impact. This is with jobs that run mostly in user space and are scheduled using cpu-pinning or numa-control.

I need to remove millions of directories recursively on a NFS share and to avoid any performance issues, I ran the command with:

ionice -c 3 -t find /dir -type f -exec rm {};

which will delete all the files and the leftover empty directories, I can remove with rm -rf.

But I am unable to say if ionice has any effect on the above command.

From man, ionice class 3:

A program running with idle io priority will only get disk time when no other program has asked for disk io for a defined grace period. The impact of idle io processes on normal system activity should be zero.

From source code, I see that ionice sets ioprio_set.

So, what is ioprio_set?

The ioprio_get() and ioprio_set() system calls respectively get and set the I/O scheduling class and priority of one or more threads.

Does it mean that the process thread carries additional properties for I/O scheduler to use them to schedule the I/O for the process?

It begs me the following questions:

• Does I/O scheduler in general knows I/O load on a disk or it just schedules I/O to disk?
• If it's the case i.e. I/O scheduler knows about load on a disk and takes decisions based on that, I think the IO scheduling classes and priorities will not have any effect on IO scheduling to a remote disk. Thus ionice can not have any effect.
• And if it's not, I think the IO scheduling classes and prioities can have effect on IO scheduling to a remote disk also like NFS. Thus ionice will work if multiple users/processes are using the same NFS disk, where the I/O scheduler schedules IO for a given thread based on if other user/procesess are using the same NFS disk.

Please correct me if I am utterly wrong.

There can be a way to perform the operation from server end also, but I want to first try from the client end.

I am trying to understand the configuration of nvme.

But I do not understand why there are two devices: nvme block and nvme character device:

crw------- 1 root root 243, 0 Dec 12 16:09 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Jan 14 01:30 /dev/nvme0n1

What is the purpose of each or when to use them?

I want to understand how the IO is sent to hard disk blocks from a virtual machine.

Does having a XFS or ext3 or ext4 etc. really matters inside a virtual machine, or it is just depended on the file system of the host machine?

What's the impact on IO when compared to Virtual machine and a physical machine?

The following are just my assumptions based on what I know. Please correct me if I'm wrong:

Let's say I am using a RAW image!

Example 1: Preallocation=full (Virtual Size = Disk Size)

• If I understood correctly, the disk blocks that would be used for Virtual machine are pre-allocated in this mode, and filled with Zeroes for unused data.
• And once a Guest OS is installed on that Raw disk, all these physical disk blocks are mapped to a virtual disk blocks (hard coded maps?).
• And any IO request made inside a virtual machine, the Guest OS kernel first processes it and sends a request to the virtual disk and that request is taken care by some kind of virtual abstraction service (I am guessing libvirtd) and is sent to hard disk controller. But what I don't get here is if the physical disk blocks are preallocted, the IO request should contain all the block addresses or the block maps so that the data is written only in one of those disk blocks. But how does such kind of an IO request is made to the hard disk controller and it seems really a major performance issue.

In this example, I completely excluded the filesystem of the host machine. I couldn't find a way to relate it.

Again all of this above is my giant assumption. I feel I am really wrong about the above idea. But I am hoping someone will correct me.

Example 2: preallocation=off (Disk Size = Actual usage)

• The disk blocks for Virtual Machine are not pre-allocated and are only allocated on actual disk usage.
• The same steps in example 1 applies here until the IO request is made to Hard disk controller, but here no need to request for specific blocks to be overwritten, as no preallocation is done here.

So, going back to my actual question. It seems the filesytem on virtual machine will do have an impact, as all the calls to IO are initiated from Guest OS kernel and later.

Huhhh confused!!!! :(

I am confused and not able to understand exactly the actual concept behind it. At switch level, what happens that prevents to use one globally accepted bond configuration policy?

And I don't understand why do we use specifically active-backup policy in our environment, instead of going for a better one that gives combined throughput, but not just fail-over configuration.

This website seems to have a good explanation: http://www.enterprisenetworkingplanet.com/linux_unix/article.php/3850636/Understanding-NIC-Bonding-with-Linux.htm

but I am unable to follow exactly. So, can someone explain it more clearer and try to explain in as much detail as possible what's happening at switch end and with the mac addresses?

Thanks.

Or you can also suggest a good book, where I can read everything about it and learn from scratch.

I am confused about the network speed outputs:

If the report says:

sar -n DEV 1 1
Linux 2.6.18-194.el5PAE (dev-db)        03/26/2011      _i686_  (8 CPU)

01:11:13 PM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
01:11:14 PM      eth0    342.57    342.57  93923.76 141773.27      0.00      0.00      0.00

Does it mean that there is only an average of 342 packets that could have transmitted in the last second or it mean there is an average of 342 packets that were actually transmitted in the last second?

I think the second sentence is correct. But how can it tell the network performance is good or bad? I mean it only tells how many packets actually were transmitted but doesn't tell the reason if there is demand/no demand for packet transmission or if the throughput is reduced due to external network traffic.

This question has been there in my mind for years.

I thought snmpbulkget acts like snmpget except that it can support multiple oid requests. But I see weird behavior. It gives next OID info instead of what is asked:

snmpbulkget -Cr1 servername IF-MIB::ifSpeed.7.0 IF-MIB::ifInOctets.7 IF-MIB::ifInUcastPkts.7 IF-MIB::ifInNUcastPkts.7 IF-MIB::ifInNUcastPkts.7 IF-MIB::ifInDiscards.7 IF-MIB::ifInErrors.7 IF-MIB::ifOutOctets.7 IF-MIB::ifOutUcastPkts.7 IF-MIB::ifOutNUcastPkts.7 IF-MIB::ifOutDiscards.7 IF-MIB::ifOutErrors.7
IF-MIB::ifSpeed.8 = Gauge32: 10000000
IF-MIB::ifInOctets.8 = Counter32: 3674120514
IF-MIB::ifInUcastPkts.8 = Counter32: 2462391943
IF-MIB::ifInNUcastPkts.8 = Counter32: 19169575
IF-MIB::ifInNUcastPkts.8 = Counter32: 19169575
IF-MIB::ifInDiscards.8 = Counter32: 0
IF-MIB::ifInErrors.8 = Counter32: 775777
IF-MIB::ifOutOctets.8 = Counter32: 3824588280
IF-MIB::ifOutUcastPkts.8 = Counter32: 2866781089
IF-MIB::ifOutNUcastPkts.8 = Counter32: 0
IF-MIB::ifOutDiscards.8 = Counter32: 0
IF-MIB::ifOutErrors.8 = Counter32: 0

Why?

So far as I understood,

• mem_free can be specified to submit a job in a host that has the memory free = mem_free, whereas
• h_vmem is the hard limit of the memory up to which the job can consume and if the job reaches the h_vmem, the job crashes? I think we can set the h_vmem of a host near to the total physical memory, so that the job won't start using swap and slow the server down.

Then what is h_rss? It seems to have the same definition as of h_vmem.

Or am I misinterpreting h_vmem? Is h_vmem used to reserve the extra memory that it might need than the minimum memory it's needed (mem_free)? But do not crash if it exceeds the memory, so the job can exceed h_vmem?

If my second interpretation of h_vmem is correct, then I guess, for a job to be submitted in a host, the job has to satisfy both mem_free and h_vmem (given h_vmem is not INFINITY).

And if my first interpretation of h_vmem is correct, then I guess, for a job to be submitted in a host, the job can satisfy mem_free alone and no need to satisfy h_vmem, as it only reserves the space available and if there is no space available, it doesn't matter?

How much time does the CPU wait to go to higher mode, let's say from C0 to C1? Or if it can be customized, where can I find the current value?

Also, does the the C-State changes in stages like first go to C1 mode then wait there for some time, and go to C2 etc. It seems so as it's the only possible way to go to all C-State modes. Otherwise, there is no reason to have intermediate C-State modes.

Also if you can answer this too: What does the following C-State output mean?

I am trying to find what raid level and what storage controllers are there in my remote HP server.

I can find out the following details which storage controller, size of the disk, model etc.

But how to find if the raid is active and if, which raid?

One more clarification, I need here: So, I read raid can be defined in two ways: Hardware and Software.

So, hardware I guess means the storage controller. Ours is:

             description: RAID bus controller
            product: Smart Array Gen9 Controllers

Some more info:

# hwinfo --storage
185: PCI 1f.2: 0106 SATA controller (AHCI 1.0)
  [Created at pci.319]
  Unique ID: w7Y8.AWGP+VY_LGA
  SysFS ID: /devices/pci0000:00/0000:00:1f.2
  SysFS BusID: 0000:00:1f.2
  Hardware Class: storage
  Model: "Intel SATA controller"
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x8d02
  SubVendor: pci 0x103c "Hewlett-Packard Company"
  SubDevice: pci 0x8030
  Revision: 0x05
  Driver: "ahci"
  Driver Modules: "ahci"
  I/O Ports: 0x4058-0x405f (rw)
  I/O Ports: 0x407c-0x407f (rw)
  I/O Ports: 0x4050-0x4057 (rw)
  I/O Ports: 0x4078-0x407b (rw)
  I/O Ports: 0x4020-0x403f (rw)
  Memory Range: 0x93100000-0x931007ff (rw,non-prefetchable)
  IRQ: 211 (2357394 events)
  Module Alias: "pci:v00008086d00008D02sv0000103Csd00008030bc01sc06i01"
  Driver Info #0:
    Driver Status: ahci is active
    Driver Activation Cmd: "modprobe ahci"
  Config Status: cfg=new, avail=yes, need=no, active=unknown

187: PCI 300.0: 0104 RAID bus controller
  [Created at pci.319]
  Unique ID: svHJ.ccZlDNM+DaD
  Parent ID: vSkL.TK0GkKDmr11
  SysFS ID: /devices/pci0000:00/0000:00:01.0/0000:03:00.0
  SysFS BusID: 0000:03:00.0
  Hardware Class: storage
  Model: "Hewlett-Packard Company RAID bus controller"
  Vendor: pci 0x103c "Hewlett-Packard Company"
  Device: pci 0x3239
  SubVendor: pci 0x103c "Hewlett-Packard Company"
  SubDevice: pci 0x21c0
  Revision: 0x01
  Driver: "hpsa"
  Driver Modules: "hpsa"
  Memory Range: 0x92f00000-0x92ffffff (rw,non-prefetchable)
  Memory Range: 0x93000000-0x930003ff (rw,non-prefetchable)
  I/O Ports: 0x3000-0x3fff (rw)
  Memory Range: 0x93200000-0x9327ffff (ro,prefetchable,disabled)
  IRQ: 16 (no events)
  Module Alias: "pci:v0000103Cd00003239sv0000103Csd000021C0bc01sc04i00"
  Driver Info #0:
    Driver Status: hpsa is active
    Driver Activation Cmd: "modprobe hpsa"
  Config Status: cfg=new, avail=yes, need=no, active=unknown
  Attached to: #153 (PCI bridge)

As you can see, there are two: SATA and RAID. From lshw output, I can confirm that all the storage drives are under RAID controller. SATA controller is for Optical Drives etc.

There are two drives sda (900GB) and sdb (4.1TB).

# dmraid -r
no raid disks
# dmraid -s
no raid disks

But,

# cat /sys/block/sda/device/raid_level
RAID 1(1+0)
# cat /sys/block/sdb/device/raid_level
RAID 5

Is the raid active?

I think like this. Please correct me if I am wrong:

Hardware raid is supported, but it is not active, as raid disks are not set. Software raid is active. So, RAID 1(1+0) in sda and RAID 5 in sdb.

If I am allowed to ask one more question:

I don't understand how software raid works and I guess its advantage is only for data recovery. As it uses only one disk, it can't do failover or have redundancy etc. right? And also, it seems unnecessary as it will only increase the data size and time.

Please correct me if I am wrong.

According to SGE 5.3 Manual,

Slots - The number of jobs which may be executed concurrently in that queue

I am new to these concepts and want to start by understanding one by one.

For suppose, if RAM is 10G and if there are 10 slots, and hence 1G per slot, then does only jobs less than 1G can be run? And if the job needs only some 0.5G, it will be wasting the remaining 0.5G in that slot right? And if so, what's the use of grid then if there is no optimization of the resources?

And if a job of 2G is shared among multiple slots, is this called a parallel job or normal job?

And is there any difference between queue and slot concept in SGE v5.3 and v6.0 & above?