Matthew's questions

We are currently working across our environment and disabling all ways that an HP server can automatically reboot. This is in response to a massive outage which caused our servers to begin flapping, causing a service outage for several million customers. The request from "on high" is to have the servers shut down, but not reboot until a human can manually guide them back online when the "coast is clear" (we have several geographically redundant sites).

So far, I have identified the following possible causes:

1. HP ASR automatically reboots a host. This can be disabled by switching off the ASR timer.
2. Disable automatic-power-on in iLO. I believe this is only triggered when power is removed, and then re-applied to the host.

However, I assume there is yet another configuration that is applied when one of the server sensors passes a critical threshold, for example if the ambient temperature sensor exceeds 40 degrees C. That should absolutely shut down a host, but I'm unsure where the configuration lies to disable the automatic reboot after the ambient temperature drops. Or is this also controlled by HP ASR?

I just want to ensure that there aren't any scenarios that I have forgotten that could bite us in the butt in production.

Any help would be appreciated.

We have several Dl360 G6's in the field. They had 2x 146G disks in a raid 1. After LVM, this shows up as a 136G volume (7% loss to filesystem).

This was carved into 100M /boot, and the remainder an LVM PV. The volgroup contained 2 logical volumes, one 54G dedicated to swap (yes, it's a lot, yes, it's by design), and 81G dedicated to /.

As of today we added 2x 300G disks in a raid 1. This shows up in the OS, and after being added to LVM it shows as 279G (7% loss to filesystem). This showed up as /dev/cciss/c0d1.

I ran these commands -

Make new array an LVM volume -

pvcreate /dev/cciss/c0d1

Add the volume to the existing volume group -

vgextend VolGroup00 /dev/cciss/c0d1

Extend the root partition logical volume to use 100% of the new space -

lvextend -l 100%FREE /dev/VolGroup00/LogVol01

Resize the volume -

resize2fs -p /dev/VolGroup00/LogVol01

The problem is, we seem to be missing a lot of space. About 90G to be exact. The output of vgdisplay shows about 82G of physical extents free (Free PE / Size 2642 / 82.56 GB), as if it ignored that I had included 100%FREE.

Am I misreading something here? Did I do something incorrectly? Any help would be appreciated. I'm wondering if the possibility of losing space to LVM formatting, and then losing space a second time to the ext3 filesystem on top of that causes a "double" loss of usable space. But that doesn't seem to explain what I see in pvdisplay and vgdisplay.

Here are dumps of the LVM commands on the host:

[root@bass01 ~]# pvs
  PV                VG         Fmt  Attr PSize   PFree 
  /dev/cciss/c0d0p2 VolGroup00 lvm2 a-   136.59G     0 
  /dev/cciss/c0d1   VolGroup00 lvm2 a-   279.34G 82.56G
[root@bass01 ~]# lvs
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  LogVol00 VolGroup00 -wi-ao  54.00G                                      
  LogVol01 VolGroup00 -wi-ao 279.38G                                      
[root@bass01 ~]# vgs
  VG         #PV #LV #SN Attr   VSize   VFree 
  VolGroup00   2   2   0 wz--n- 415.94G 82.56G
[root@bass01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol01
                      271G   18G  240G   7% /
/dev/cciss/c0d0p1      97M   19M   73M  21% /boot
tmpfs                  36G     0   36G   0% /dev/shm
[root@bass01 ~]# free -g
             total       used       free     shared    buffers     cached
Mem:            70         12         58          0          9          0
-/+ buffers/cache:          1         68
Swap:           53          0         53
[root@bass01 ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/cciss/c0d0p2
  VG Name               VolGroup00
  PV Size               136.60 GB / not usable 7.45 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              4371
  Free PE               0
  Allocated PE          4371
  PV UUID               nh7cgk-3pr7-gZst-Pz6o-PAWt-X1wZ-GH4TmI

  --- Physical volume ---
  PV Name               /dev/cciss/c0d1
  VG Name               VolGroup00
  PV Size               279.37 GB / not usable 22.29 MB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              8939
  Free PE               2642
  Allocated PE          6297
  PV UUID               tHEAfR-HsNd-jNUT-mOwj-Ettb-N76X-fXIf0Q

[root@bass01 ~]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               415.94 GB
  PE Size               32.00 MB
  Total PE              13310
  Alloc PE / Size       10668 / 333.38 GB
  Free  PE / Size       2642 / 82.56 GB
  VG UUID               dcPGvL-SUCl-l99e-pQJN-e6sA-xEKZ-Q5ZT1G

[root@bass01 ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol01
  VG Name                VolGroup00
  LV UUID                p7of6A-poAR-1iXs-Tgpq-lJRh-qae3-VNxBTO
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                279.38 GB
  Current LE             8940
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol00
  VG Name                VolGroup00
  LV UUID                X4iAo4-kKd8-ZZvJ-vfqq-fdMo-HaEM-YIsGE7
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                54.00 GB
  Current LE             1728
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

We are trying to troubleshoot a DL360 G7 with a P410i controller.

As mentioned in the thread title, the site tech pulled all disks and forgot what order they were pulled in. We were under the expectation that the server was a cold spare, but apparently the vendor who performed the installation mislabelled the host. It was one of our application servers.

When we had the tech reseat the disks, we are now getting an error along the lines of:

"Physical disks reordered. Previously failed disks 3,4 now operational.

Press f1 to continue with logical drive disabled. Press f2 to accept data loss and continue."

We contacted HP, and they're useless. Their statement is that recovery is probably impossible, and we can try the different permutations of moving the disks around but there's no guarantee the OS (RHEL 5) will boot. They're pretty sure the OS is hosed.

What I am trying to figure out...

1. Why the heck would moving the disks around mean the OS is destroyed? All Array information is written to the first few sectors of the drive. The controller can look at it and say "Hey, this disk isn't in the right place. I'm going to freeze now." It seems like they write additional information to the disk marking it as failed which destroys its array association.
2. Has anyone been through this before? Is HP correct? Is the OS wrecked? Is there any way to recover?
3. HP is suggesting that pulling all disks at the same time, even with the server off, causes the controller to stop recognizing the array. But this doesn't make any sense because the array information is written to the disks themselves. That's why you could build an array on one server, then take all of those disks in the array and move them to a like server, and it should boot.
4. If we put the drives in the correct order, will the server just boot without prompting us? We have about 30 permutations to work through. It will be very hard to tell if we hit the right combo or not if we keep getting told disks are failed.

Edit: Following up on this item. It appears we will be unable to recover with any major chance of success. Our issue was exacerbated by some additional problems.

1. The drives were removed from the host and plugged into a different host out of their original order. This likely caused data corruption to that array information that cannot easily be recovered.
2. When the drives were put back into the server, their order wasn't maintained. This likely caused further issues with the array metadata.

After some testing in our lab, it appears that using a raid 1+0 and swapping drive order does not seem to impact the array's functionality, even when the drive order is severely modified. This would suggest the issue really is related to array metadata being damaged when moving the drives between hosts.

We have a poorly implemented mysql server that had been built by a vendor. It has consumed 100% of available disk space, and as soon as any space is cleared out mysql grows to consume it. This is, obviously, a problem.

We currently have 2 disks in a raid 1+0, and just added 2 additional. We have added them directly to the RAID, so disk size should be essentially double what it was previously.

Pvresize would be our next step, but it only seems to facilitate growing to specific numbers (pvresize --setphysicalvolumesize 20G /path/to/disk).

Is there an equivalent command that lets you specify to expand to all physical space on the disk (or in this case, the array)?

We have a mostly Cisco network, though for our access layer we usually deploy Juniper Virtual Chassis switches. To replicate this, we are interested in deploying a few juniper switches in our product development lab. The downside is we have well over a hundred vlans in that lab.

Is there any support for VTP? Did they license it for cisco? Or will I have to grab a copy of the vlan database off of one our other switches and implement it manually on the juniper switches?

We are using a custom repository with RHEL 5.x. We were running into a ton of issues and conflicts when trying to run an update (issues with yum 32bit packages conflicting with x64 packages). To get around this, I added

exclude=*.i386

to /etc/yum.conf. This worked great, until I had to install compat-libstdc++-296, which apparently is 32bit only. I checked on RH's site, and the only package listed verified that. So, when I try to specify the full package name compat-libstdc++-296.i386, the exclusion list blocks it from being installed.

Is there any way to force yum to temporarily ignore the exclusion list? Something like

yum install compat-libstdc++-296.i386 --ignore-exclude 

Not that it's hard to comment out the line, run the install, and then uncomment, my major concern is that we have around 300 servers this needs to be run on, and that's a lot of hands editing text files. I'd much prefer being able to copy and paste a command, if that makes sense.

Our company uses home brewed repositories and a system to configure them, which is conveniently not working at the moment. This is causing a problem for me since I can't add repositories, and therefore can't install software on this RHEL 5 box.

I can log in to red hat and download packages, but I can't find a listing of everything included in the package group when you run "yum install httpd." Can anyone provide a list of those packages? I'd appreciate it.

We had a server effectively go down this morning. SSH access cut out, and at least temporarily network access went down as well. We were able to log in using out-of-band access and were presented with a screen full of "Init: cannot fork, retry.." messages.

When trying to log in, when we typed in a userid and bad password, we got the normal "invalid user/pass" error. However if we typed in a correct userid and password, we were simply presented with the MOTD and the login screen again. It looks like the system was no longer able to launch any new processes (logging in successfully should launch a shell, if it can't I guess it drops you back at login?).

I found a description of the issue at Red Hat's knowledgebase (https://access.redhat.com/site/solutions/39497), but there is very little supplementary information on the error, just a suggested solution.

What exactly does nproc do? Is it a hard limit on the number of processes the system can have running at any point in time? When nproc is exceeded does it cause impacts like we saw? Is there any way to set it to unlimited? If not, how can we know what a safe or unsafe range is?

Any help or guidance would be very much appreciated, since it caused production issues and is now on the plate of several layer-8 folks :(

Edit: Also in /var/log/messages:

May 31 15:26:00 servername udevd[1637]: udev_event_run: fork of child failed: Resource temporarily unavailable
May 31 15:26:00 servername last message repeated 3 times
May 31 15:26:00 servername udevd-event[2461]: run_program: fork of '/lib/udev/udev_run_hotplugd' failed: Resource temporarily unavailable
May 31 15:26:00 servername udevd-event[2461]: run_program: fork of '/lib/udev/udev_run_devd' failed: Resource temporarily unavailable
May 31 15:26:00 servername udevd[1637]: udev_event_run: fork of child failed: Resource temporarily unavailable

Right now we are getting an intermittent error in iLO stating that "Power Supply 1" is failing. The issue is that there are two power supplies, and neither one is clearly labelled as "Power Supply 1." Since it is an intermittent issue, the error comes and goes (but appears resolved in between). We can't really look for an amber LED because right now the server doesn't detect a fault with the PSU. We want to pull the PSU and replace it with a spare we have on site. So, looking at the server from behind, is power supply 1 on the right or on the left?

Right now I have a user who is able to use his domain credentials to log in to any server on our network, except one. The server that he cannot log in to allows other domain users to connect, but when he tries to connect it says that he is unauthorized.

When a user is locked out in an Active Directory environment, is it at the domain level or is it on a particular server?

Right now we are attempting to build gold configurations for our environment. One piece of software that we use relies on large XML files to contain the bulk of its configuration. We want to take our lab environment, catalog it as our "gold configuration" and then be able to audit against that configuration in the future.

Since diff is a bytewise comparison and NOT a logical comparison, we can't use it to compare files in this case (XML is unordered, so it won't work). What I am looking for is something that can parse the two XML files, and compare them element by element. So far we have yet to find any utilities that can do this. OS doesn't matter, I can do it on anything where it will work. The preference is something off the shelf.

Any ideas?

Edit: One issue we have run into is one vendor's config files will occasionally mention the same element several times, each time with different attributes. Whatever diff utility we use would need to be able to identify either the set of attributes or identify them all as part of one element. Tall order :)

This is hopefully a simple question. Are there any tools out there to find out an enhanced description (beyond the normal rpm command) of the contents of an rpm, and possibly decompile it? We currently have an rpm from our company's security team which is supposed to be used for centralized authentication on our linux servers. The issue is that they can't supply source, and don't have a documented list of all changes that are made on the server. They did say that decompiling the rpm was ok.

So, any ideas??

This is hopefully a simple question. Right now we are deploying servers which will serve as data warehouses. I know with RAID 5 the best practice is 6 disks per RAID 5. However, our plan is to use RAID 10 (both for performance and safety). We have a total of 14 disks (16 actually, but two are being used for OS). Keeping in mind that performance is very much an issue, which is better - doing several RAID 1's? Do one large RAID 10? One large RAID 10 had been our original plan, but I want to see if anyone has any opinions I haven't thought of.

Please note: This system was designed for using RAID 1+0, so losing half of the raw storage capacity is not an issue. Sorry I hadn't mentioned that initially. The concern is more whether or not we want to use one large RAID 1+0 containing all 14 disks, or several smaller RAID 1+0's and then stripe across them using LVM. I know the best practice for higher RAID levels is to never use more than 6 disks in an array.

I received an email yesterday that one of our users was trying to make room for a heartbeat/clustering package which requires its own partition to act as a voting disk. To do this, he attempted to reduce the size of the root partition's logical volume, and then create a new logical volume for this purpose. However, he forgot to resize the filesystem first (or include the -r switch in the command). He also forgot to unmount the root partition by running this process from a rescue cd. The system is now refusing to boot into the OS with the following error:

Either the superblock or the partition table is likely to be corrupt! Unexpected Inconsistency; run fsck manually.

The system them drops the user into single user mode.

Is it possible to rescue the filesystem, or is it hosed? Its running ext3.

At first I thought this was a fluke, but I've now built well over 50 of these servers and I can't seem to find a solution. We are deploying HP DL360 G7's. Before you used to be able to hit F8 during boot up to enter into a ROM based setup utility for iLO. However now it simply shows a mention of iLO during POST, but you don't get a chance to press F8. Even if you spam the F8 key, you'll see the briefest flash of the "Press F8 to Configure" message which almost immediately disappears. It then continues on through POST. Has anyone else had this issue? Is there any way to resolve it? I have 17 servers I need to build today and the online config tool isn't an option (in an isolated build environment). Any help would be very much appreciated, thank you.

Just curious, what is the default remote shell used by rsync when you're not using it in daemon mode? According to the man page it can be run via rsh or ssh. However nowhere in the man page does it specify how to set which one to use. So, when running it it prompts me for a password, but it doesn't seem to indicate which remote shell is being used. I know this is probably a dummy question, but I could really use some clarification. Thanks!

I am just trying to figure out if this is possible to do. I am trying to resize a number of filesystems, and to be safe (these are production boxes as well as QA) I wanted to do the resize offline.

I have gotten into the server with single user mode and am having no issues unmounting /var, and /opt. However I cannot unmount /.

Any ideas would be appreciated, using a rescue cd is not an option.

What is the Solaris equivalent of the Linux hostname -f command? On Solaris, when I type hostname, I receive the short name but I need to get the FQDN.

Typing hostname -f sets the short name to be "-f", and the manpage for hostname is pitiful.

I am trying to use the windows version of dd to copy a RHEL iso to a USB stick. However, I wanted to zero out the drive first to ensure there is no filesystem on it before writing it out. Is there an equivalent of /dev/zero in windows that I can use as the infile?

I am trying to set up a python script to query a DB for all updates over the past 5 minutes. Given the number of updates, it must be as precise as possible (our records are timestamped to the microsecond). There are two ways I'm considering tackling this - first is using cron. However this relies on cron being precise enough to always run at the exact same interval every time it runs. So if the first execution time is 00:00:00.123456789, it would need to run again at 00:05:00.123456789. Otherwise, there is a possibility of records being missed in the gaps in between.

The other option is finding a way to "snap" the sql query to the nearest minute, rounded down. But if I can use cron, i'd prefer to do that to keep things as simple as possible.