ewwhite's questions

Is the default Ctrl-Alt-Delete shutdown -r functionality on Linux systems a dangerous feature?

Years ago, when I deployed physical systems with attached keyboards and monitors, I'd sometimes modify the /etc/inittab on Red Hat systems to disable the reboot trap. This usually happened after a local IT person or Windows admin accidentally used the magic key combination on the wrong terminal/keyboard/window and rebooted their server.

# Trap CTRL-ALT-DELETE
ca::ctrlaltdel:/sbin/shutdown -t3 -r now

I haven't done this since the RHEL4 days, but newer systems seem to have a /etc/init/control-alt-delete.conf file for this.

In the years since, most of my systems have been deployed headless or are running as virtual machines. This has reduced the frequency of unintended reboots... however, I've had a recent set of ctrl-alt-delete oopses from:

1). an IP KVM plugged into the wrong server by datacenter staff.
2). a Windows admin using the key combination in a VMware console, thinking it was needed for logon.
3). me using the ctrl-alt-delete macro in an HP ILO console to reboot a live CD... but it was actually the ILO for a very busy production server.

• Does it make sense to disable Ctrl-Alt-Delete reboot in Linux by default?
• Is this a common concern, or generally ignored?
• Are there any downsides to doing so?
• How do you handle this in your environment?

Edit: In fact, I just encountered this server, a virtual machine running for 1,115 days, root password unknown, and VMware tools were not installed (so Ctrl-Alt-Delete would be the only graceful shutdown option).

What is the common wisdom in 2014 about Active Directory authentication/integration for Linux servers and modern Windows Server operating systems (CentOS/RHEL-focused)?

Over the years since my first attempts with integration in 2004, it seems like the best-practices around this have shifted. I'm not quite sure which method currently has the most momentum.

In the field, I've seen:

Winbind/Samba
Straight-up LDAP
Sometimes LDAP + Kerberos
Microsoft Windows Services for Unix (SFU)
Microsoft Identity Management for Unix
NSLCD
SSSD
FreeIPA
Centrify
Powerbroker (née Likewise)

Winbind always seemed terrible and unreliable. The commercial solutions like Centrify and Likewise always worked, but seemed unnecessary, since this capability is baked into the OS.

The last few installations I've done had the Microsoft Identity Management for Unix role feature added to a Windows 2008 R2 server and NSLCD on the Linux side (for RHEL5). This worked until RHEL6, where the lack of maintenance on NSLCD and memory resource management issues forced a change to SSSD. Red Hat also seemed to back the SSSD approach, so that's been fine for my use.

I'm working with a new installation where the domain controllers are Windows 2008 R2 Core systems and do not have the ability to add the Identity Management for Unix role feature. And I'm told that this feature is deprecated is no longer present in Windows Server 2012 R2.

The benefit of having this role installed is the presence of this GUI, while allows easy one-step administration of user attributes.

But...

The Server for Network Information Service (NIS) Tools option of Remote Server Administration Tools (RSAT) is deprecated. Use native LDAP, Samba Client, Kerberos, or non-Microsoft options.

That makes it really difficult to rely on if it may break forward-compatibility. The customer wants to use Winbind, but everything I see from the Red Hat side points to the use of SSSD.

What's the right approach?
How do you handle this in your environment?

I have a client whose workforce is comprised entirely of remote employees using a mix of Apple and Windows 7 PCs/laptops.

The users don't authenticate against a domain at the moment, but the organization would like to move in that direction for several reasons. These are company-owned machines, and the firm seeks to have some control over account deactivation, group policy and some light data-loss prevention (disable remote media, USB, etc.) They are concerned that requiring VPN authentication in order to access AD would be cumbersome, especially at the intersection of a terminated employee and cached credentials on a remote machine.

Most services in the organization are Google-based (mail, file, chat, etc.) so the only domain services are DNS and the auth for their Cisco ASA VPN.

The customer would like to understand why it is not acceptable to expose their domain controllers to the public. In addition, what is a more acceptable domain structure for a distributed remote workforce?

Edit:

Centrify is in use for a handful of Mac clients.

I've been a longtime advocate for HP ProLiant servers in my system environments. The platform has been the basis of my infrastructure designs across several industries for the past 12 years.

The main selling points of ProLiant hardware have been long-lasting product lines with predictable component options, easy-to-navigate product specifications (Quickspecs), robust support channels and an aggressive firmware release/update schedule for the duration of a product's lifecycle.

This benefits the use of HP gear in primary and secondary markets. Used and late-model equipment can be given a new life with additional parts or through swapping/upgrading as component costs decline.

One of the unique attributes of HP firmware is the tendency to introduce new functionality along with bugfixes in firmware releases. I've seen Smart Array RAID controllers gain new capabilities, server platforms acquire support for newer operating systems, serious performance issues resolved; all through firmware releases. Reading through a typical changelog history reveals how much testing and effort goes into creating a stable hardware platform. I appreciate that and have purchased accordingly.

Other manufacturers seem to ship product as-is and only focus on correcting bugs in subsequent firmware releases. I rarely run firmware updates on Supermicro and Dell gear. But I deem it irresponsible to deploy HP servers without an initial firmware maintenance pass.

Given this, the early reports of an upcoming policy change by HP regarding server firmware access were alarming...

The official breakdown:

Access to select server firmware updates and SPP for HP ProLiant Servers will require entitlement and will only be available to HP customers with an active contractual support agreement, HP Care Pack service, or warranty linked to their HP Support Center User ID. As always, customers must have a contract or warranty for the specific product being updated.

Essentially, you must have active warranty and support on your servers in order to access firmware downloads (and presumably, the HP Service Pack for ProLiant DVD).

This will impact independent IT technicians, internal IT and customers who are running on older equipment the most, followed by people seeking deals on used HP equipment. I've provided many Server Fault answers that boil down to "updating this component's firmware will solve your problem". The recipients of that advice likely would not have active support and would be ineligible for firmware downloads under this policy.

• Is this part of a growing trend of vendor lock-in? HP ProLiant Gen8 disk compatibility was a precursor.
• Is HP overstepping bounds by restricting access to updates that some people have depended upon?
• Will the result be something like the underground market for Cisco IOS downloads?
• How does this sit with you, your organization or purchase decision makers? Will it impact future hardware decisions?
• Is this any incentive to keep more systems under official warranty or extend Care Packs on older equipment?
• What are other possible ill-effects of this policy change that I may not have accounted for?

Update:
A response on the HP Support Services Blog - Customers for Life

Update:

This is in effect now. I'm seeing the prompt when trying to download BIOS updates for my systems. A login using the HP Passport is not necessary to proceed with the download.

An SD (SDHC) card installed in an HP ProLiant DL380p Gen8 server running VMware ESXi just failed :(

I encountered some ominous looking messages on the vCenter console and in the HP ProLiant ILO event log...

Lost connectivity to the device ... backing the boot filesystem. As a result, host configuration changes will not be saved to persistent storage.

Embedded Flash/SD-CARD: Error writing media 0, physical block 848880: Stack Exception.

VMware advocates the use of USB and SD (SDHC) boot devices for ESXi. It was one of the main reasons the smaller footprint ESXi was developed (versus the older ESX). I've spent much time highlighting the differences between ESXi's installable and embedded modes to coworkers and clients. However, these failures do seem to happen. In this case, this is my third instance.

Luckily, this is a vSphere cluster with SAN storage. What steps should be taken to remediate this failure?

In my organization, I work with a group of NOC staff, budding junior engineers and a handful of senior engineers; all with a focus on Linux. One interesting step in the way the company grows talent is that there's a path from the NOC to the senior engineering ranks. Viewing the talent pool as a relative newcomer, I see that there's a split in the skill sets that tends to grow over time...

• There are engineers who know one or several particular technologies well and are constantly immersed... e.g. MySQL, firewalls, SAN storage, load balancers...
• There are others who are generalists and can navigate multiple technologies.
• All learn enough Linux (commands, processes) to do what they need and use on a daily basis.

A differentiating factor between some of the staff is how well they embrace scripting, automation and configuration management methodologies. For instance, we have two engineers who do the bulk of Amazon AWS CloudFormation work, and another who handles most of the Puppet infrastructure. Perhaps a quarter of the engineers are adept at BASH shell scripting.

Looking at this in the context of the incredibly high demand for DevOps skills in the job market, I'm curious how other organizations foster the development of these skills and grow their internal talent. Scripting doesn't seem like a particularly-teachable concept.

• How does a sysadmin improve their shell scripting?
• Is there still a place for engineers who do not/cannot keep up in the DevOps paradigm?
• Are we simply to assume that some people will be left behind as these technologies evolve? Is that okay?

VMware memory management seems to be a tricky balancing act. With cluster RAM, Resource Pools, VMware's management techniques (TPS, ballooning, host swapping), in-guest RAM utilization, swapping, reservations, shares and limits, there are a lot of variables.

I'm in a situation where clients are using dedicated vSphere cluster resources. However, they are configuring the virtual machines as though they were on physical hardware. In turn, this means a standard VM build may have 4 vCPUs and 16GB or more of RAM. I come from the school of starting small (1 vCPU, minimal RAM), checking real-world use and adjusting up as necessary. Unfortunately, many vendor requirements and people unfamiliar with virtualization request more resources than necessary... I'm interested in quantifying the impact of this decision.

Some examples from a "problem" cluster.

Resource pool summary - Looks almost 4:1 overcommitted. Note the high amount of ballooned RAM.

Resource allocation - The Worst Case Allocation column shows that these VMs would have access to less than 50% of their configured RAM under constrained conditions.

The real-time memory utilization graph of the top VM in the listing above. 4 vCPU and 64GB RAM allocated. It averages under 9GB use.

Summary of the same VM

• What are the downsides of overcommitting and overconfiguring resources (specifically RAM) in vSphere environments?

• Assuming that the VMs can run in less RAM, is it fair to say that there's overhead to configuring virtual machines with more RAM than they actually need?

• What is the counter-argument to: "if a VM has 16GB of RAM allocated, but only uses 4GB, what's the problem??"? E.g. do customers need to be educated that VMs are not the same as physical hardware?

• What specific metric(s) should be used to meter RAM usage. Tracking the peaks of "Active" versus time? Watching "Consumed"?

Update: I used vCenter Operations Manager to profile this environment and get some detail on the cluster stats listed above. While things are definitely overcommitted, the VMs are actually so overconfigured with unnecessary RAM that the real (tiny) memory footprint shows no memory contention at the cluster/host level...

My takeaway is that VMs should really be right-sized with a little bit of buffer for OS-level caching. Overcommitting out of ignorance or vendor "requirements" leads to the situation presented here. Memory ballooning seems to be bad in every case, as there is a performance impact, so right-sizing can help prevent this.

Update 2: Some of these VMs are beginning to crash with:

kernel:BUG: soft lockup - CPU#1 stuck for 71s! 

VMware describes this as a symptom of heavy memory overcommitment. So I guess that answers the question.

vCops "Oversized Virtual Machines" report...

vCops "Reclaimable Waste" graph...

Can I run reliably with a single Fusion-io card installed in a server, or do I need to deploy two cards in a software RAID setup?

Fusion-io isn't very clear (almost misleading) on the topic when reviewing their marketing materials Given the cost of the cards, I'm curious how other engineers deploy them in real-world scenarios.

I plan to use the HP-branded Fusion-io ioDrive2 1.2TB card for a proprietary standalone database solution running on Linux. This is a single server setup with no real high-availability option. There is asynchronous replication with a 10-minute RPO that mirrors transaction logs to a second physical server.

Traditionally, I would specify a high-end HP ProLiant server with the top CPU stepping for this application. I need to go to SSD, and I'm able to acquire Fusion-io at a lower price than enterprise SAS SSD for the required capacity.

• Do I need to run two ioDrive2 cards and join them with software RAID (md or ZFS), or is that unnecessary?
• Should I be concerned about Fusion-io failure any more than I'd be concerned about a RAID controller failure or a motherboard failure?
• System administrators like RAID. Does this require a different mindset, given the different interface and on-card wear-leveling/error-correction available in this form-factor?
• What IS the failure rate of these devices?

Edit: I just read a Fusion-io reliability whitepaper from Dell, and the takeaway seems to be "Fusion-io cards have lots of internal redundancies... Don't worry about RAID!!".

Considering the fact that many server-class systems are equipped with ECC RAM, is it necessary or useful to burn-in the memory DIMMs prior to their deployment?

I've encountered an environment where all server RAM is placed through a lengthy burn-in/stress-tesing process. This has delayed system deployments on occasion and impacts hardware lead-time.

The server hardware is primarily Supermicro, so the RAM is sourced from a variety of vendors; not directly from the manufacturer like a Dell Poweredge or HP ProLiant.

Is this a useful exercise? In my past experience, I simply used vendor RAM out of the box. Shouldn't the POST memory tests catch DOA memory? I've responded to ECC errors long before a DIMM actually failed, as the ECC thresholds were usually the trigger for warranty placement.

• Do you burn-in your RAM?
• If so, what method(s) do you use to perform the tests?
• Has it identified any problems ahead of deployment?
• Has the burn-in process resulted in any additional platform stability versus not performing that step?
• What do you do when adding RAM to an existing running server?

Recent versions of RHEL/CentOS (EL6) brought some interesting changes to the XFS filesystem I've depended on heavily for over a decade. I spent part of last summer chasing down an XFS sparse file situation resulting from a poorly-documented kernel backport. Others have had unfortunate performance issues or inconsistent behavior since moving to EL6.

XFS was my default filesystem for data and growth-partitions, as it offered stability, scalability and a good performance boost over the default ext3 filesystems.

There's an issue with XFS on EL6 systems that surfaced in November 2012. I noticed that my servers were showing abnormaly-high system loads, even when idle. In one case, an unloaded system would show a constant load average of 3+. In others, there was a 1+ bump in load. The number of mounted XFS filesystems seemed to influence the severity of the load increase.

System has two active XFS filesystems. Load is +2 following upgrade to the affected kernel.

Digging deeper, I found a few threads on the XFS mailing list that pointed to an increased frequency of the xfsaild process sitting in the STAT D state. The corresponding CentOS Bug Tracker and Red Hat Bugzilla entries outline the specifics of the issue and conclude that this is not a performance problem; only an error in the reporting of system load in kernels newer than 2.6.32-279.14.1.el6.

WTF?!?

In a one-off situation, I understand that the load reporting may not be a big deal. Try managing that with your NMS and hundreds or thousands of servers! This was identified in November 2012 at kernel 2.6.32-279.14.1.el6 under EL6.3. Kernels 2.6.32-279.19.1.el6 and 2.6.32-279.22.1.el6 were released in subsequent months (December 2012 and February 2013) with no change to this behavior. There's even been a new minor release of the operating system since this issue was identified. EL6.4 was released and is now on kernel 2.6.32-358.2.1.el6, which exhibits the same behavior.

I've had a new system build queue and have had to work around the issue, either locking kernel versions at the pre-November 2012 release for EL6.3 or just not using XFS, opting for ext4 or ZFS, at a severe performance penalty for the specific custom application running atop. The application in question relies heavily on some of the XFS filesystem attributes to account for deficiencies in the application design.

Going behind Red Hat's paywalled knowledgebase site, an entry appears stating:

High load average is observed after installing kernel 2.6.32-279.14.1.el6. The high load average is caused by xfsaild going into D state for each XFS formatted device.

There is currently no resolution for this issue. It is currently being tracked via Bugzilla #883905. Workaround Downgrade the installed kernel package to a version lower then 2.6.32-279.14.1.

(except downgrading kernels not an option on RHEL 6.4...)

So we're 4+ months into this problem with no real fix planned for the EL6.3 or EL6.4 OS releases. There's a proposed fix for EL6.5 and a kernel source patch available... But my question is:

At what point does it make sense to depart from the OS-provided kernels and packages when the upstream maintainer has broken an important feature?

Red Hat introduced this bug. They should incorporate a fix into an errata kernel. One of the advantages of using enterprise operating systems is that they provide a consistent and predictable platform target. This bug disrupted systems already in production during a patch cycle and reduced confidence in deploying new systems. While I could apply one of the proposed patches to the source code, how scalable is that? It would require some vigilance to keep updated as the OS changes.

What's the right move here?

• We know this could possibly be fixed, but not when.
• Supporting your own kernel in a Red Hat ecosystem has its own set of caveats.
• What's the impact on support eligibility?
• Should I just overlay a working EL6.3 kernel on top of newly-build EL6.4 servers to gain the proper XFS functionality?
• Should I just wait until this is officially fixed?
• What does this say about the lack of control we have over enterprise Linux release cycles?
• Was relying on an XFS filesystem for so long a planning/design mistake?

Edit:

This patch was incorporated into the most recent CentOSPlus kernel release (kernel-2.6.32-358.2.1.el6.centos.plus). I'm testing this on my CentOS systems, but this doesn't help much for the Red Hat-based servers.

I have a new HP ProLiant DL360 G7 system that is exhibiting a difficult-to-reproduce issue. The server randomly hangs at the "Power and Thermal Calibration in Progress..." screen during the POST process. This typically follows a warm-boot/reboot from the installed operating system.

The system stalls indefinitely at this point. Issuing a reset or cold-start via the ILO 3 power controls makes the system boot normally without incident.

When the system is in this state, the ILO 3 interface is fully accessible and all system health indicators are fine (all green). The server is in a climate-controlled data center with power connections to PDU. Ambient temperature is 64°F/17°C. The system was placed in a 24-hour component testing loop prior to deployment with no failures.

The primary operating system for this server is VMWare ESXi 5. We initially tried 5.0 and later a 5.1 build. Both were deployed via PXE boot and kickstart. In addition, we are testing with baremetal Windows and Red Hat Linux installations.

HP ProLiant systems have a comprehensive set of BIOS options. We've tried the default settings in addition to the Static high-performance profile. I've disabled the boot splash screen and just get a blinking cursor at that point versus the screenshot above. We've also tried some VMWare "best-practices" for BIOS config. We've seen an advisory from HP that seems to outline a similar issue, but did not fix our specific problem.

Suspecting a hardware issue, I had the vendor send an identical system for same-day delivery. The new server was a fully-identical build with the exception of disks. We moved the disks from the old server to the new. We experienced the same random booting issue on the replacement hardware.

I now have both servers running in parallel. The issue hits randomly on warm-boots. Cold boots don't seem to have the problem. I am looking into some of the more esoteric BIOS settings like disabling Turbo Boost or disabling the power calibration function entirely. I could try these, but they should not be necessary.

Any thoughts?

--edit--

System details:

• DL360 G7 - 2 x X5670 Hex-Core CPU's
• 96GB of RAM (12 x 8GB Low-Voltage DIMMs)
• 2 x 146GB 15k SAS Hard Drives
• 2 x 750W redundant power supplies

All firmware up-to-date as of latest HP Service Pack for ProLiant DVD release.

Calling HP and trawling the interwebz, I've seen mentions of a bad ILO 3 interaction, but this happens with the server on a physical console, too. HP also suggested power source, but this is in a data center rack that successfully powers other production systems.

Is there any chance that this could be a poor interaction between low-voltage DIMMs and the 750W power supplies? This server should be a supported configuration.

"Can we upgrade our existing production EL5 servers to EL6?"

A simple-sounding request from two customers with completely different environments prompted my usual best-practices answer of "yes, but it will require a coordinated rebuild of all of your systems"...

Both clients feel that a complete rebuild of their systems is an unacceptable option for downtime and resource reasons... When asked why it was necessary to fully reinstall the systems, I didn't have a good answer beyond, "that's the way it is..."

I'm not trying to elicit responses about configuration management ("Puppetize everything" doesn't always apply) or how the clients should have planned better. This is a real-world example of environments that have grown and thrived in a production capacity, but don't see a clean path to move to the next version of their OS.

Environment A:
Non-profit organization with 40 x Red Hat Enterprise Linux 5.4 and 5.5 web, database servers and mail servers, running a Java web application stack, software load balancers and Postgres databases. All systems are virtualized on two VMWare vSphere clusters in different locations, each with HA, DRS, etc.

Environment B:
High-frequency financial trading firm with 200 x CentOS 5.x systems in multiple co-location facilities running production trading operations, supporting in-house development and back-office functions. The trading servers are running on bare-metal commodity server hardware. They have numerous sysctl.conf, rtctl, interrupt binding and driver tweaks in place to lower messaging latency. Some have custom and/or realtime kernels. The developer workstations are also running a similar version(s) of CentOS.

In both cases, the environments are running well as-is. The desire to upgrade comes from a need for a newer application or feature available in EL6.

• For the non-profit firm, it's tied to Apache, the kernel and some things that will make the developers happy.
• In the trading firm, it's about some enhancements in the kernel, networking stack and GLIBC, which will make the developers happy.

Both are things that can't be easily packaged or updated without drastically altering the operating system.

As a systems engineer, I appreciate that Red Hat recommends full rebuilds when moving between major version releases. A clean start forces you to refactor and pay attention to configs along the way.

Being sensitive to business needs of clients, I wonder why this needs to be such an onerous task. The RPM packaging system is more than capable of handling in-place upgrades, but it's the little details that get you: /boot requiring more space, new default filesystems, RPM possibly breaking mid-upgrade, deprecated and defunct packages...

What's the answer here? Other distributions (.deb-based, Arch and Gentoo) seem to have this ability or a better path. Let's say we find the downtime to accomplish this task the right way:

• What should these clients do to avoid the same problem when EL7 is released and stabilizes?
• Or is this a case where people need to resign themselves to full rebuilds every few years?
• This seems to have gotten worse as Enterprise Linux has evolved... Or am I just imagining that?
• Has this dissuaded anyone from using Red Hat and derivative operating systems?

I suppose there's the configuration management angle, but most Puppet installations I see do not translate well into environments with highly-customized application servers (Environment B could have a single server whose ifconfig output looks like this). I'd be interesting in hearing suggestions on how configuration management can be used to help organizations get across the RHEL major version bump, though.

This is a simple issue that we all face and probably resolve manually without giving much thought.

As servers change, are re-provisioned, or IP addresses reallocated, we receive the SSH host verification message below. I'm interested in streamlining the workflow to resolve these ssh identification errors.

Given the following message, I typically vi /root/.ssh/known_hosts +434 and remove (dd) the offending line.

I've seen developers/users in other organizations delete their entire known_hosts file out of frustration in seeing this message. While I don't go that far, I know there's a more elegant way to handle this.

Tips?

[root@xt ~]# ssh las-db1

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
ed:86:a2:c4:cd:9b:c5:7a:b1:2b:cc:42:15:76:8c:56.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:434
RSA host key for las-db1 has changed and you have requested strict checking.
Host key verification failed.

One of my client's sites received a direct lightning hit last week (coincidentally on Friday the 13th!).

I was remote to the site, but working with someone onsite, I discovered a strange pattern of damage. Both internet links were down, most servers were inaccessible. Much of the damage occurred in the MDF, but one fiber-connected IDF also lost 90% of the ports on a switch stack member. Enough spare switch ports were available to redistribute cabling elsewhere and reprogram, but there was downtime while we chased down affected devices..

This was a new building/warehousing facility and a lot of planning went into the design of the server room. The main server room is run off of an APC SmartUPS RT 8000VA double-conversion online UPS, backed by a generator. There was proper power distribution to all connected equipment. Offsite data replication and systems backups were in place.

In all, the damage (that I'm aware of) was:

• Failed 48-port line card on a Cisco 4507R-E chassis switch.
• Failed Cisco 2960 switch in a 4-member stack. (oops... loose stacking cable)
• Several flaky ports on a Cisco 2960 switch.
• HP ProLiant DL360 G7 motherboard and power supply.
• Elfiq WAN link balancer.
• One Multitech fax modem.
• WiMax/Fixed-wireless internet antenna and power-injector.
• Numerous PoE connected devices (VoIP phones, Cisco Aironet access points, IP security cameras)

Most of the issues were tied to losing an entire switch blade in the Cisco 4507R-E. This contained some of the VMware NFS networking and the uplink to the site's firewall. A VMWare host failed, but HA took care of the VM's once storage networking connectivity was restored. I was forced to reboot/power cycle a number of devices to clear funky power states. So the time to recovery was short, but I'm curious about what lessons should be learned...

• What additional protections should be implemented to protect equipment in the future?
• How should I approach warranty and replacement? Cisco and HP are replacing items under contract. The expensive Elfiq WAN link balancer has a blurb on their website that basically said "too bad, use a network surge protector". (seems like they expect this type of failure)
• I've been in IT long enough to have encountered electrical storm damage in the past, but with very limited impact; e.g. a cheap PC's network interface or the destruction of mini switches.
• Is there anything else I can do to detect potentially flaky equipment, or do I simply have to wait for odd behavior to surface?
• Was this all just bad luck, or something that should be really be accounted for in disaster recovery?

With enough $$$, it's possible to build all sorts of redundancies into an environment, but what's a reasonable balance of preventative/thoughtful design and effective use of resources here?

I've run XFS filesystems as data/growth partitions for nearly 10 years across various Linux servers.

I've noticed a strange phenomenon with recent CentOS/RHEL servers running version 6.2+.

Stable filesystem usage became highly variable following the move to the newer OS revision from EL6.0 and EL6.1. Systems initially installed with EL6.2+ exhibit the same behavior; showing wild swings in disk utilization on the XFS partitions (See the blue line in the graph below).

Before and after. The upgrade from 6.1 to 6.2 occurred on Saturday.

The past quarter's disk usage graph of the same system, showing the fluctuations over the last week.

I started to check the filesystems for large files and runaway processes (log files, maybe?). I discovered that my largest files were reporting different values from du and ls. Running du with and without the --apparent-size switch illustrates the difference.

# du -skh SOD0005.TXT
29G     SOD0005.TXT

# du -skh --apparent-size SOD0005.TXT
21G     SOD0005.TXT

A quick check using the ncdu utility across the entire filesystem yielded:

Total disk usage: 436.8GiB  Apparent size: 365.2GiB  Items: 863258

The filesystem is full of sparse files, with nearly 70GB of lost space compared to the previous version of the OS/kernel!

I pored through the Red Hat Bugzilla and change logs to see if there were any reports of the same behavior or new announcements regarding XFS.

Nada.

I went from kernel version 2.6.32-131.17.1.el6 to 2.6.32-220.23.1.el6 during the upgrade; no change in minor version number.

I checked file fragmentation with the filefrag tool. Some of the biggest files on the XFS partition had thousands of extents. Running on online defrag with xfs_fsr -v during a slow period of activity helped reduce disk usage temporarily (See Wednesday in the first graph above). However, usage ballooned as soon as heavy system activity resumed.

What is happening here?

This seems basic, but I'm confused about the patching strategy involved with manually updating standalone VMware ESXi hosts. The VMware vSphere blog attempts to explain this, but the actual process is still not clear to me.

From the blog:
Say Patch01 includes updates for the following VIBs: "esxi-base", "driver10" and "driver 44". And then later Patch02 comes out with updates to "esxi-base", "driver20" and "driver 44". P2 is cumulative in that the "esxi-base" and "driver44" VIBs will include the updates in Patch01. However, it's important to note that Patch02 not include the "driver 10" VIB as that module was not updated.

This VMware Communities post gives a different answer. This one contradicts the other.

Many of the ESXi installations I encounter are standalone and do not utilize Update Manager. It is possible to update an individual host using the patches make available through the VMWare patch download portal. The process is quite simple, so that part makes sense.

The bigger issue is determining what exactly to actually download and install. In my case, I have a good number of HP-specific ESXi builds that incorporate sensors and management for HP ProLiant hardware.

• Let's say that those servers start with an ESXi build #474610 from 9/2011.
• Looking at the patch portal screenshot below, there is a patch for ESXi update01, build #623860. There are also patches for builds #653509 and #702118.
• Coming an old version of ESXi (e.g. vendor-specific build), what is the proper approach to bring the system fully up-to-date? Which patches are cumulative and which need to be applied sequentially? Is installing the newest build the right approach, or do I need to step back and patch incrementally?
• Another consideration is the large size of the patch downloads. At sites with limited bandwidth, downloading of multiple ~300mb patches is difficult.

Most of the Linux systems I manage feature hardware RAID controllers (mostly HP Smart Array). They're all running RHEL or CentOS.

I'm looking for real-world tunables to help optimize performance for setups that incorporate hardware RAID controllers with SAS disks (Smart Array, Perc, LSI, etc.) and battery-backed or flash-backed cache. Assume RAID 1+0 and multiple spindles (4+ disks).

I spend a considerable amount of time tuning Linux network settings for low-latency and financial trading applications. But many of those options are well-documented (changing send/receive buffers, modifying TCP window settings, etc.). What are engineers doing on the storage side?

Historically, I've made changes to the I/O scheduling elevator, recently opting for the deadline and noop schedulers to improve performance within my applications. As RHEL versions have progressed, I've also noticed that the compiled-in defaults for SCSI and CCISS block devices have changed as well. This has had an impact on the recommended storage subsystem settings over time. However, it's been awhile since I've seen any clear recommendations. And I know that the OS defaults aren't optimal. For example, it seems that the default read-ahead buffer of 128kb is extremely small for a deployment on server-class hardware.

The following articles explore the performance impact of changing read-ahead cache and nr_requests values on the block queues.

http://zackreed.me/articles/54-hp-smart-array-p410-controller-tuning
http://www.overclock.net/t/515068/tuning-a-hp-smart-array-p400-with-linux-why-tuning-really-matters
http://yoshinorimatsunobu.blogspot.com/2009/04/linux-io-scheduler-queue-size-and.html

For example, these are suggested changes for an HP Smart Array RAID controller:

echo "noop" > /sys/block/cciss\!c0d0/queue/scheduler 
blockdev --setra 65536 /dev/cciss/c0d0
echo 512 > /sys/block/cciss\!c0d0/queue/nr_requests
echo 2048 > /sys/block/cciss\!c0d0/queue/read_ahead_kb

What else can be reliably tuned to improve storage performance?
I'm specifically looking for sysctl and sysfs options in production scenarios.

I'm looking for a good solution to a VMware ESXi environment issue where there's no vCenter available.

What's the best way to move a VM from one datastore to another on a single ESXi host, while maintaining the VMDK thin-provisioning?

This is a standalone server that's been expanded with another drive array/datastore. I'd like to move the data contained in the old array to the new.

Edit: The destination datastore size is actually smaller than the source. I do not have enough room to copy the thick file.

Throughout my career, I've encountered quite a few users who were email hoarders...

I'm managing a 100-user environment where users previously had 100mb quotas and a basic Cyrus IMAP/Postfix solution. That led to lots of locally-stored mail and the related problems with data-retention and lost mail following PC failures.

I promptly moved towards Exchange 2010 with MailArchiva (for discovery/archiving) and have been humming along for nearly a year in this setup. Everything inbound and out is archived and retained for compliance purposes. I initially set 3GB mailbox limits per user. A handful of users needed extra space for their working sets of email and were able to justify it, so I've made some exceptions. However, I have one particularly-bad (but important) user whose Inbox is presently 22,000 items, of which 14,000 are unread. It's probably not the number of messages, but the nature of the email; lots of reports, statements and large PDF attachments. This is also a Blackberry user, so I suspect that they read mail selectively, based on subject. The user has hit the 3GB limit for the first time and is unwilling to prune the mailbox. I took care of their 3,000 deleted items to keep mail flowing, but am at a loss on how to educate users on how to organize their email. I can institute stringent policies and be a hardass, or I could sit with the user and help sort and purge mail. Either way, it doesn't address the real issue of how to help with education.

Does anyone here have any tips on how to deal with this type of situation? I'd like some example guidelines. If you're using policies in your firm, what types of policies help keep people under control?

I find that people use mail differently, but it's a challenge to force them into a particular way of organization. I may not even be the best example, but I also know what to keep and what to delete.

Update - The first battle is over, in that I've been able to keep the user under the Exchange mailbox limit by some reasonable pruning. However, I went to rebuild the user's computer and discovered a 17GB local PST file with just over 6,500 subfolders! The user had been copying (not moving) messages into an array of subfolders; sometimes with single messages being copied/filed into multiple folders. Underscores (_) and prefixes (a-z) were used to control the display order of the folders within the file! The user had also been CC'ing themselves on outbound email so as to have an Inbox record of messages to later be filed into one of the local PST's subfolders. This is an absolute misuse of the technology and is a scary way to organize company data. I am handing it over to the owners to address the situation, as it's now a company liability.

I have a system running a financial trading application at a remote facility. I do not have access to the ILO/DRAC, but need to disable hyperthreading. The system runs Intel Westmere 3.33GHz X5680 hex-core CPUs. I can reboot, but want to make sure that the system does not enable hyperthreading due to performance problems. Is there a clean way to do this from within Linux?

Edit: The noht directive added to the kernel boot command line did not work. Same for RHEL.

See: https://bugzilla.redhat.com/show_bug.cgi?id=440321#c9