exhuma's questions

I'm maintaining an application (written in Python) which polls information of a large number of hosts. As input I have a set of network ranges. Not all of the hosts in those ranges are always up. For the hosts which are down, I run into incredibly long TCP timeouts.

For this reason, I run an nmap fast ping-scan before the main application executes.

This means that the application has a dependency on nmap being installed on the system. All for only a simple ping-check.

The problem is that, as a normal user, I am not privileged enough to send out ICMP ECHO requests. Both nmap and ping have the setuid bit set IIRC. So those can be run as normal user.

Given that I write my own application, which is also a Python script which runs through the Python interpreter, I wonder what the best possibility is to run ICMP requests from there.

Setting the setuid bit on the Python interpreter seems like a sledge-hammer solution.

Note that the application runs as a separate user and from a Python virtual environment (in case that would make a difference).

I have some trouble getting the proper access right to work for a folder shared through samba and nfs. Connected like this:

┌────────────┐    ┌──────────────┐    ┌──────────────┐
│ NFS-Server ├────┤ NFS Client   ├────┤ Samba Client │
│            │    │ Samba Server │    │              │
└────────────┘    └──────────────┘    └──────────────┘

The machine in the middle is a shared development machine (running Linux). The Samba clients are usually Windows. The reason this is connected like this is because of the network policy / firewall. It would be strongly discouraged to access the NFS server by Samba directly.

Additionally, as a side-effect this setup gives everyone samba access to the development machine.

The exported NFS resources should be read/writable by a specific group. I implemented this using Unix ACLs. When accessing the share via the development machine (the NFS client) directly, this already gives me trouble. It seems that the mask is not properly kept.

I have found a couple of resource on the web covering this topic, but I still have trouble understanding exactly why/how this is happening.

I also came across NFSv4 ACLs which are different from the ACLs you set using setfacl. I tried to play around with those as well, but running the command, gave me an error:

[11:24:32] michel@BBS-extractor coftp $ nfs4_getfacl .
Operation to request attribute not supported.

I thought this might be a problem related to mount options. However, the underlying FS is an ext4 volume. Which has nothing to do with NFS... right?

Can someone explain to me what's happening here? Why is the default mask ignored when using NFS?

As a practical example, let me give you the ACLs of the root folder:

[11:30:26] michel@BBS-extractor coftp $ getfacl .
# file: .
# owner: coftp
# group: coftp
# flags: -s-
user::rwx
group::rwx
group:coftp:rwx
mask::rwx
other::r-x
default:user::rwx
default:group::rwx
default:group:coftp:rwx
default:mask::rwx
default:other::r-x

After creating a folder named test on the remote machine, I get this:

[11:30:26] michel@BBS-extractor coftp $ getfacl test
# file: test
# owner: michel
# group: coftp
# flags: -s-
user::rwx
group::rwx                      #effective:r-x
group:coftp:rwx                 #effective:r-x
mask::r-x
other::r-x
default:user::rwx
default:group::rwx
default:group:coftp:rwx
default:mask::rwx
default:other::r-x

This shows, that the group "coftp" won't have write access to this folder, although in the ACL on the root folder, it should have. The reason being the mask. The root folder specifies a default mask, but it seems to be ignored/changed when accessing the folder via NFS.

As stated in the title, when I run echo ::memstat | mdb -k on certain machines, it is very slow. My munin graphs show a vastly increased CPU load when I run this regularly. I intend to use ::memstat as input for munin graphs, so it needs to be run frequently.

While the process is running, I see this in the prstat output:

   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  6865 root       14M   12M cpu2     0    0   0:00:26  23% mdb/1
  6868 root       14M   12M cpu1     0    0   0:00:26  23% mdb/1

As you can see, it has already been running for 26 seconds. Monitoring prstat shows an ever increasing CPU usage during the processes run-time. It needs around 30 seconds to finish.

On another machine (granted, in terms of hardware it's beefier), it runs much faster. The point is, that it runs much faster as I would expect from comparing the hardware specs (8GB vs 12GB memory, 8 Cores vs 4 Cores, SunOS 5.10 Generic_147441-01 i86pc i386 i86pc vs SunOS 5.10 Generic_127128-11 i86pc i386 i86pc). But that's not the real point. My primary problem is that echo ::memstat | mdb -k takes 30 seconds to complete.

Can somebody please explain what's going on here? What could explain the "slowness"?

Last week we decided to add some SunOS (uname -a = SunOS bbs-sam-belair 5.10 Generic_127128-11 i86pc i386 i86pc) machines into our running munin instance. First off, the machines are pre-configured appliances, so, I want to avoid touching the system too much without supervision of the service provider.

But adding it to munin was fairly easy by writing a small socket-service (if anyone is interested, I put it up on github: https://github.com/munin-monitoring/contrib/tree/master/tools/pypmmn)

Yesterday, I implemented/adapted the required plugins for our machines. And here the questions start:

First, I have not found a way to determine detailed memory usage values. I get the total memory by running prtconf | grep Memory, and the free memory using vmstat. Fiddling together a munin-plugin, gives me the following graph:

This is pretty much uninformative. Compare this to the default plugin for linux nodes which has a lot more detail:

Most importantly, this shows me how much memory is actually used by applications.

So, first question: Is it possible to get detailed memory information on SunOS with the default system tools (i.e. not using top)?

Onto the next puzzle: Seeing the graphs, I noticed activity in the "Paging in/out" graphs, even though the memory graph still has unused memory:

Upon further investigation, I found out that df reports that /tmp is mounted on swap. Drilling around on the web, I understood that df will display swap, but in fact, it's mounted as a tmpfs. Now I don't know if this explains the swap activity.

The default munin-plugin for solaris uses kstat -p -c misc -m cpu_stat to get these values. I find it already strange that this is using the cpu_stat module. So maybe I simply misinterpret the "paging" graphs?

Second question: Do the paging graphs indicate that parts of the memory are paged to disk? Or is the activity caused by file operations in /tmp?

When installing munin, it activates a default set of plugins (at least on ubuntu). Alternatively, you can simply run munin-node-configure to figure out which plugins are supported on your system. Most of these plugins plot straight-forward data. My question is not to explain the nature of the data (well... maybe for some) but what is it that you look for in these graphs?

It is easy to install munin and see fancy graphs. But having the graphs and not being able to "read" them renders them totally useless.

I am going to list standard plugins which are enabled by default on my system. So it's going to be a long list. For completeness, I am also going to list plugins which I think to understand and give a short explanation as to what I think it's used for. Pleas correct if I am wrong with any of them.

So let me split this questions in three parts:

• Plugins where I don't even understand the data
• Plugins where I understand the data but don't know what I should look out for
• Plugins which I think to understand

Plugins where I don't even understand the data

These may contain questions that are not necessarily aimed at munin alone. Not understanding the data usually mean a gap in fundamental knowledge on operating systems/hardware.... ;) Feel free to respond with a "giyf" answer.

These are plugins where I can only guess what's going on... I hardly want to look at these "guessing"...

• Disk IOs per device (IOs/second)
  What's an IO. I know it stands for input/output. But that's as far as it goes.
• Disk latency per device (Average IO wait)
  Not a clue what an "IO wait" is...
• IO Service Time
  This one is a huge mess, and it's near impossible to see something in the graph at all.

Plugins where I understand the data but don't know what I should look out for

• IOStat (blocks/second read/written)
  I assume, the thing to look out for in here are spikes? Which would mean that the device is in heavy use?
• Available entropy (bytes)
  I assume that this is important for random number generation? Why would I graph this? So far the value has always been near constant.
• VMStat (running/I/O sleep processes)
  What's the difference between this one and the "processes" graph? Both show running/sleeping processes, whereas the "Processes" graph seems to have more details.
• Disk throughput per device (bytes/second read/written)
  What's thedifference between this one and the "IOStat" graph?
• inode table usage
  What should I look for in this graph?

Plugins which I think to understand

I'll be guessing some things here... correct me if I am wrong.

• Disk usage in percent (percent)
  How much disk space is used/remaining. As this is approaching 100%, you should consider cleaning up or extend the partition. This is extremely important for the root partition.
• Firewall Throughput (packets/second)
  The number of packets passing through the firewall. If this is spiking for a longer period of time, it could be a sign of a DOS attack (or we are simply recieving a large file). It can also give you an idea about your firewall performance. If it's levelling out and you need more "power" you should consider load balancing. If it's levelling out and see a correlation with your CPU load, it could also mean that your hardware is not fast enough. Correlations with disk usage could point to excessive LOG targets in you FW config.
• eth0 errors (packets in/out)
  Network errors. If this value is increasing, it could be a sign of faulty hardware.
• eth0 traffic (bits/second in/out)
  Raw network traffic. This should correlate with Firewall throughput.
• number of threads
  An ever-increasing value might point to a process not properly closing threads. Investigate!
• processes
  Breakdown of active processes (including sleeping). A quick spike in here might point to a fork-bomb. A slowly, but ever-increasing value might point to an application spawning sub-processes but not properly closing them. Investigate using ps faux.
• process priority
  This shows the distribution of process priorities. Having only high-priority processes is not of much use. Consider de-prioritizing some.
• cpu usage
  Fairly straight-forward. If this is spiking, you may have an attack going on, or a process is hogging the CPU. Idf it's slowly increasing and approaching max in normal operations, you should consider upgrading your hardware (or load-balancing).
• file table usage
  Number of actively open files. If this is reaching max, you may have a process opening, but not properly releasing files.
• load average
  Shows an summarized value for the system load. Should correlate with CPU usage. Increasing values can come from a number of sources. Look for correlations with other graphs.
• memory usage
  A graphical representation of you memory. As long as you have a lot of unused+cache+buffers you are fine.
• swap in/out
  Shows the activity on your swap partition. This should always be 0. If you see activity on this, you should add more memory to your machine!

I have a quite old box in a remote server farm. So I have no direct access. Only remote SSH (and via SSH to a serial console). I haven't updated this box in ages. Now, whenever I want to install a new package, a dependency to glibc appears. Unfortunately, the install of glibc depends on a 2.6 kernel and I am running a venerable 2.4 kernel (one more reason to upgrade).

The problem is, that the install of a new kernel has an indirect (over locales) dependency to glibc. So, to install glibc, I need a new kernel. For a new kernel, I need to upgrade glibc. Essentially I am blocked. What's the best way to proceed considering I have no "hardware" access?

Here's a quick transcript of the upgrade process:

[green:~]% sudo aptitude install linux-image-686
Reading Package Lists... Done
Building Dependency Tree
Reading extended state information
Initializing package states... Done
Reading task descriptions... Done
The following packages are unused and will be REMOVED:
  gcc-4.3-base
The following NEW packages will be automatically installed:
  dash libc6-i686 libparse-recdescent-perl linux-image-2.6-686
  linux-image-2.6.18-6-686 module-init-tools yaird
The following packages have been kept back:
  adduser apache2 apache2-mpm-prefork apache2-utils apache2.2-common apt
  apt-utils aptitude autoconf autotools-dev awstats base-files base-passwd
  [...snip...]
  util-linux vacation vim vim-common wamerican wbritish wget whiptail whois
  wwwconfig-common zlib1g
The following NEW packages will be installed:
  dash libc6-i686 libparse-recdescent-perl linux-image-2.6-686
  linux-image-2.6.18-6-686 linux-image-686 module-init-tools yaird
The following packages will be upgraded:
  hotplug libc6
2 packages upgraded, 8 newly installed, 1 to remove and 277 not upgraded.
Need to get 0B/22.7MB of archives. After unpacking 52.1MB will be used.
Do you want to continue? [Y/n/?]
Writing extended state information... Done
Preconfiguring packages ...
(Reading database ... 34065 files and directories currently installed.)
Preparing to replace libc6 2.3.6.ds1-13 (using .../libc6_2.7-18lenny2_i386.deb) ...
Checking for services that may need to be restarted...
Checking init scripts...
WARNING: init script for postgresql not found.

  [ --- libc6 config screen appears here --- ]

WARNING: POSIX threads library NPTL requires kernel version
2.6.8 or later. If you use a kernel 2.4, please upgrade it
before installing glibc.

The installation of a 2.6 kernel _could_ ask you to install a new libc
first, this is NOT a bug, and should *NOT* be reported. In that case,
please add etch sources to your /etc/apt/sources.list and run:
  apt-get install -t etch linux-image-2.6
Then reboot into this new kernel, and proceed with your upgrade
dpkg: error processing /var/cache/apt/archives/libc6_2.7-18lenny2_i386.deb (--unpack):
subprocess pre-installation script returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/libc6_2.7-18lenny2_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
Ack!  Something bad happened while installing packages.  Trying to recover:
dpkg: dependency problems prevent configuration of locales:
locales depends on glibc-2.7-1; however:
  Package glibc-2.7-1 is not installed.
dpkg: error processing locales (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
locales
Reading Package Lists... Done
Building Dependency Tree
Reading extended state information
Initializing package states... Done
Reading task descriptions... Done

Now, if I follow the instrunctions as promted I get the following. Note that I am using aptitude instead of apt-get to benefit from the better dependency tracking. I did try with apt-get first. But that let me to the same problem.

[green:~]% sudo aptitude install -t etch linux-image-2.6.26-2-686
Reading Package Lists... Done
Building Dependency Tree
Reading extended state information
Initializing package states... Done
Reading task descriptions... Done
E: Unable to correct problems, you have held broken packages.
E: Unable to correct dependencies, some packages cannot be installed
E: Unable to resolve some dependencies!
Some packages had unmet dependencies.  This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.

The following packages have unmet dependencies:
  linux-image-2.6.26-2-686: Depends: initramfs-tools (>= 0.55) but it is not installable or
                yaird (>= 0.0.13) but it is not installable or
                linux-initramfs-tool which is a virtual package.

Any ideas?