Zac B's questions

Context:

We use a RabbitMQ 2.8.4 HA cluster for messaging across a mid-size (hundreds to thousands of servers) datacenter. The messaging functions are critical to our app.

We use the C client API for AMQP/RabbitMQ operations, and the HTTP API to the RabbitMQ management plugin to periodically monitor queue status and send alerts based on size/throughput/etc.

The only ports/connections allowed from clients to the messaging servers are on the AMQP ports and on HTTP for the management API.

For the near future, management has decided that downtime of the messaging cluster (which translates into downtime for certain requesting parts of the app) is unacceptable. In a few months it will be possible, but until then sales-driven concerns won't allow us to have maintenance of the messaging system. It sucks, but we (the infrastructure/devops people) don't get to make those calls.

Problem:

For awhile, we have been victimized by the problem discussed in this SO question. After a connection storm, or a long period during which new clients are connecting and using the AMQP system very frequently, the RabbitMQ management API shows hundreds of thousands of connections that it lists as "running" that no longer exist. netstat, lsof and other diagnostics prove that the sockets corresponding to these connections are not open on the clients; the management API is just unable to prune the entries for some reason. rabbitmqctl list_connections sometimes shows some of the nonexistent connections to be still open, but it always shows many fewer than the management API, and its "staleness" detection seems better than the API's and it eventually purges the false connections from its listing.

When we attempt to delete one of these "false" connections via the web UI, the page doesn't respond to our pressing "delete", and the connection stays in the listing. When we try to delete it by sending a DELETE in the JSON API, we get a 500 server error indicating that the connection isn't valid, so it can't be deleted (exactly as described in the linked SO question above).

After a long accumilation of these nonexistent connection objects, three bad things happen, in no particular order:

• The RabbitMQ server has massive memory spikes (that come out of nowhere; they don't correspond to the slow connection creep in the management API, though its memory consumption does grow slowly over time). This often leads to a crash.
• The management API stops responding to requests (times out on the API clients).
• The RabbitMQ server itself starts sporadically refusing connections from AMQP clients that are not doing anything with HTTP/the management API.

In cases where this problem begins breaking our app, we have to do one of two things: * Restart the whole messaging cluster, which is bad because it causes momentary downtime, and bad because messages can be dropped during this process. * Restart and purge the management API's database, which is bad because we can't use the management API, and worse because it often breaks the management API in such a way that it will not start properly again until we restart the whole cluster as in the first bullet.

Most people afflicted by similar problems resolve them by upgrading to RabbitMQ 3.*. We have neither the time, permission, nor downtime authorization for such a project. Even in the face of periodic several-minute-long outages when we restart the cluster due to this issue, we were not authorized to upgrade. It's a purely political restriction, but an unfortunate reality.

Question:

Given that we can't upgrade RabbitMQ in the near future, and that we need to continue using the management API, how can we resolve the issue of persistent, stale connections that cause messaging cluster failures?

We have the ability to develop new/different client code, and run jobs on the messaging servers themselves, but we don't have the ability to use rabbitmqctl locally or via ssh on any of the client machines.

What we've tried:

• We've tried switching AMQP client libraries in hopes that a better client would clean up after itself and not leave stale connections, to no avail. The issue persists.
• We've tried playing with the tcp_keepalive and exit_on_close TCP settings on the RabbitMQ servers themselves. No combination of these values resolves the issue.
• We've made a script which iterates over all connections returned from the JSON API's connections list and sends a DELETE for each one that has been open for longer than it would need to be if it were being used. All such requests are met with 500 server errors as described in the SO question linked above.

Question:

Is there a way to tell Postgres (9.2) to "merge all WAL files in pg_xlog back into the non-WAL data files, and then delete all WAL files successfully merged?"

I would like to be able to "force" this operation; i.e. checkpoint_segments or archiving settings should be ignored. The filesystem WAL buffer (pg_xlog) directory should be emptied, or nearly emptied. It's fine if some or all of the space consumed by the pg_xlog directory is then consumed by the data directory; our DBA has asked for file (i.e. data directory, rather than sql) database backups without any backlogged WALs, but space consumption is not a concern.

Having near-zero WAL activity during this operation is a fine constraint. I can ensure that the database server is either shut down or not connectible (zero user-generated transaction load) during this process.

Essentially, I'd like Postgres to ignore archiving/checkpoint retention policies temporarily, and flush all WAL activity to the core database files, leaving pg_xlog in the same state as if the database were recently created--with very few WAL files.

What I've Tried:

I know that the pg_basebackup utility performs something like this (it generates an almost-all-WALs-merged copy of a Postgres instance's data directory), but we aren't ready to use it on all our systems yet, as we are still testing replication settings; I'm hoping for a more short-term solution.

I've tried issuing CHECKPOINT commands, but they just recycle one WAL file and replace it with another (that is, if they do anything at all; if I issue them during database idle time, they do nothing). pg_switch_xlog() similarly just forces a switch to the next log segment; it doesn't flush all queued/buffered segments.

I've also played with the pg_resetxlog utility. That utility sort of does what I want, but all of its usage docs seem to indicate that it destroys (rather than flushing out of the transaction log and into the main data files) some or all of the WAL data. Is that impression accurate? If not, can I use pg_resetxlog during a zero-WAL-activity period to force a flush of all queued WAL data to non-WAL data? If the answer to that is negative, how can I achieve this goal?

Thanks!

We have a windows (7 pro) workstation LAN, managed via Group Policy. Lots of our users run Dropbox. Security concerns aside, one of our netadmins says that our routers/firewalls are handling and blocking tons of UDP broadcast traffic, and they want it to stop. We tracked the traffic to Dropbox's LAN Sync functionality.

Most users on our network don't use LAN Sync (they only have one workstation on the network), but almost everyone with Dropbox, it seems, has it turned on.

Is there a way to centrally disable LAN Sync for Dropbox, for all Windows workstations in a GPO-managed LAN?

The solution doesn't have to be nice and state-based like a true GPO; it can be as simple as a scheduled task that runs something on all workstations daily to disable LAN Sync. Heck, the solution doesn't even have to be a GPO--we can use PS_EXEC to push programs to workstations from the domain controllers. I am just hoping to avoid manually reconfiguring all users' Dropbox applications. Users are local admins, so if they really want to turn LAN Sync on, they can.

Aside: there is literally zero chance of getting management to agree to ban/remove either Dropbox or users' admin privileges.

What I've Tried:

Initially, I figured there would be a GPO-manageable registry key for it. No dice, it turns out; Dropbox keeps all of its configs in a SQLite file.

Then I tried using this script to modify the SQLite file, but newer versions of Dropbox don't seem to have an externally-modifiable config.

Context:

After a recent version upgrade of a bunch of our server software, I've found myself in a dilemma:

I have two sets of applications (Zend server and assorted utils, and a bunch of PostgreSQL management utilities) that both heavily use a certain library file (libpq.so). Zend ships with its own libpq, which is the only version (that I've found) that will work with all of the Zend stuff. Postgres also ships with its own version of the library. The two are mutually exclusive: if Postgres's version of the library comes first in the LD_LIBRARY_PATH of whatever user is doing something, all of the Postgres utils will work, but none of the Zend ones will. If Zend's comes first, the Zend stuff will work, but the Postgres stuff won't.

The libraries both have the same names. The PostgreSQL compatibility libs package doesn't work.

In our environment, both sets of applications are used under a lot of different, unpredictable user accounts on a lot of different CentOS machines, so using the "right way" and setting LD_LIBRARY_PATH per-user, and telling people to "only use X applications under Y account" will not work.

Question:

Is there a per application way to say "for these binaries/apps executed in X folder, link against a library at path Y, but for these other binaries, link against library Z"? Basically, a per-application library path, without having to set LD_LIBRARY_PATH or DYLD_* every time I access one of the applications.

I'd prefer to avoid having to recompile libraries, since that would sign me up for a significant additional amount of hassle and testing every time a new version of either set of software comes out. I already have libraries that work, they just won't work for both sets of apps.

Say I have an ESXi (5.0) host that runs a Linux distribution which hosts iSCSI targets, which contain the images for other VMs which the host will run. When it's used, I'll start the host first, then the iSCSI server, and then refresh all storage targets/HBAs in order to see the provided shares as online. I know it's a strange puzzle-box solution, but I was told to implement it.

The ESXi host itself has a gigabit NIC which connects to the outside world. The guest OS (CentOS) supports VMXNet3, however, and if I can, I'd like to use its VMXNET3 NIC to host iSCSI for the ESXi host. How should I go about doing this?

I went to create a new virtual network, and selected "VKernel", as it suggested that I use that type of network for SAN traffic, but it is apparently not set up for "self-hosted" SAN hosts, as the new network did not appear as an option to attach the CentOS box's VMXNET3 NIC to.

How should I best connect an iSCSI host out to its "parent" ESXi host, if I need a) a 10gb connection, and (optionally) b) a VMKernel network for it?

Context:

I'm an admin for a midsize corporate network with several overlapping wireless access points and an ethernet cabling infrastructure. To comply with a security audit, we were recently ordered to implement full device authorization on the network--not just a method for detecting rogue APs and the like, but a full "if your device isn't manually added by us, it can't connect to anything" system.

My first thought was: just create static DHCP MAC-to-IP bindings for everyone, manually, and have people with new devices come to me when they first set up their accounts. I told the auditors, and they said that whatever system we implement cannot be vulnerable to MAC address spoofing or static IP-ing, or else we will remain out of compliance.

I'm aware that there are products out there that will bind to your computer's network connection, and require you to authenticate against some centralized server in order to get network access, but my experience with those (and the Cisco AnyConnect VPN, which works similarly) has been that they can't run on all platforms. Since we have a BYOD environment (not my call), that wouldn't work.

Question:

What software can I run at the network level that prevents devices (any/all devices; arbitrary types/OSes on laptops, random mobile phones, tablets, etc.) from getting network access until I manually add them to our network (ideally without too much hassle during the addition process)? It needs to work similarly on both wired and wireless connections, and not be able to be bypassed by setting a device's MAC address or IP to that of an already-authorized device.

Because of the variety of devices we have, whatever solution we use should be centralized, without extra software requirements on the client. If there's something that fits the bill, but requires client software, I could probably convince management to use it if the client software worked on the majority of our devices (i.e. all *nix laptops, OSX, Windows (xp through 8), iOS, Android, Windows Mobile, BlackBerry), and the oddball users that weren't covered could just deal with it.

Context:

I have a SAN operating system (Nexenta) that runs on ESXi. It has a couple of HBAs passed through to it already, via VT-d. All of the HBAs for the SAN OS are full (no free connectors). I recently purchased some SATA SSDs for Nexenta and attached them directly to the motherboard's on-board SATA controller.

I can add those new SSDs to the Nexenta VM by adding "physical disks" to the VM profile in vSphere. Alternatively, I could connect then to one of the HBAs, but I'd have to disconnect/move existing disks, which entails considerable hassle.

Question:

My question is: assuming that my HBAs don't do any fancy caching etc., and have the same available bus bandwidth and SATA specification as the onboard controller (the one connected to the new SSDs), is there a performance difference between attaching the physical disks to the VM via the vSphere disk add functionality, and attaching them to an HBA that is passed through to the VM via VT-d? Does the vSphere disk add method impose some relaying/request forwarding behavior that could negatively impact disk performance compared to native speeds?

Anecdotes for answers are good, but statistics are better: I know I probably won't notice a performance difference at first, since SSDs are fast. But I've been in the field long enough to know that if there is a problematic performance difference, it will manifest during production-critical activities, at the worst possible time :)

I'm considering buying a bunch of these to upgrade our servers at work (with SATA drives, not true SAS). We are out of ports to plug disks into, and these cards are decently supported and cheap.

The one thing I can't seem to figure out is: what type of HBA-to-SATA cable/connector type do these cards use? In the pictures, they look like SFF-8484, but I'm not 100% sure, and after reading the specs, Googling around, and reading the manual, I haven't found out for sure. I'd rather not spend a lot of money on cables that don't work, so...does anyone have any experience with these SAS cards? What is their cable type?

Cheers!

Context:

I have an Intel DX79TO motherboard and a Sun SAS3081E-S/LSI 1060E-S HBA card with a PCIe x8 interface. I plug the HBA into my mobo next to my graphics card, and the HBA power lights illuminate, but the BIOS and OSes (tried Linux, ESXi, Win7) don't see the HBA at all.

Question:

Does the DX79TO motherboard support non-x16/non-GPU devices in its PCIe x16 slots? According to this question, some consumer motherboards don't support this, but I can't figure out whether or not this motherboard/family does. The answer will affect whether I buy a new motherboard or RMA the SAS card, with money attached to each course, so I figured I'd ask here first.

What I've Tried:

I've read the spec/manuals for the motherboard and the HBA, and I didn't see anything regarding whether or not the x16 slots were back-compatible to lower lane widths/non graphics-card devices, or whether or not the card could run in wider slots than x8. I've tried contacting Intel, but that was over a month ago and I haven't yet heard anything back except an automated "we got your email!" message.

Context:

We've got multiple custom-added yum repositories (on CentOS 6), and our users regularly install various/arbitrary software via yum to test things.

We have a package that is available on multiple repositories. It's named the exact same thing on two different repositories, call them Y and Z, but the packages are NOT identical. Both repositories are used for other things/installations, but the package in question should only ever be installed from repo Y.

What I've Tried:

When I manually install the package in question, I exclude/include repos to ensure that I'm getting the correct version. However, ordinary users a) forget to do that, and b) sometimes install other packages that depend on the package in question, and it gets installed from the wrong repo as part of the dependency installation process.

Question:

Is there any way to configure yum's defaults to say that "if package X is available from more than one source, prefer/insist on source Y over source Z?"

My goal is not to have to change the normal yum install process that our users currently have; they should still be able to install arbitrary software using the simplest syntax possible. It's just that specific packages (either installed manually or as part of dependency installation) should only be allowed to come from certain repos.

I've got a Dell SAS 6/iR PCI-E adapter. I don't have a multiplier backplane. I'm planning on connecting SATA (non SAS) drives. If I buy cable adapters only (ones that split a SAS connector on the card to a certain number of SATA cables), how many drives can I connect to this card?

The way I see it, there are two limitations: a limitation imposed by the theoretical max number of devices supported on the card (which I've dug through the specs to find, but haven't seen yet), and a limitation imposed by the number of SAS plugs on the card multiplied by the number of SATA cables that come out of the highest-multiplying splitter I can buy. The answer to my question would be the minimum of those two limitations.

I've seen 4x SATA coming out of some splitters; are there any that have more?

Alternatively, if this is an RTFM question, does anyone have a good link to a "this is how SAS works, this is how you figure out the max number of devices, and this is how the concepts of 'ports', 'lanes', 'endpoint devices', and 'connectors' all relate in SAS-land" document? I've looked around on the Dell docs, but haven't found anything that explains this to someone at my level of understanding of SAN/enterprise storage technologies.

Cheers!

Context:

We have a midsize business with a few dozen laptops. Whenever we get one in because someone leaves or gets an upgrade, standard procedure is to zero and then run SpinRite (level 4) on the hard drive to make sure it's in good shape before sending the laptop back out with a fresh OS. If the drive passes SpinRite (no bad sectors, minimal or no seek errors, no other kinds of errors)

Problem:

Recently, a bunch (six or seven) of these SpinRite-ed drives I've put in laptops have started making weird noises and going bad. They aren't headcrashing, but several have developed a persistent "tick", and then lots of read failures, and three others have developed high-pitched whining noises. None of the drives were exhibiting these issues before the SpinRite/zeroing process, and none of the laptops they were from were reported defective. These are different-sized drives, from differently-specced laptops that were purchased at very different times. The oldest drive is about 2.5yrs old, the youngest is just four months old. These laptops weren't used for anything beyond your average business user's tasks.

Question:

Is SpinRite somehow breaking my hard drives? If it were just one or two with this problem, I'd assume that they were just near the end of their lives already, and the intensive operations of SpinRite just pushed them over the edge. However, 7 hard drives is pushing the boundaries of coincidence. Is there any way that SpinRite could be causing this?

I'm speccing out sandbox VM servers for developers at my company. The requirements for the dom0 hosts are: local, deduplicated ZFS storage (no separate storage servers), and Xen 4.1+ (4.2 is ideal).

I checked out Oracle VM, but it only has Xen 4.0. I also looked at the Xen official list of supported dom0s, but none of the ZFS-supporting ones (openindiana/illumos and opensolaris) seem to be current enough to run Xen 4.1+.

Question:

What operating systems out there have robust (read: stable and decent performance) support for ZFS/dedup, and can smoothly (read: good hardware compatibility) work as dom0 hosts for Xen 4.1+?

If possible, I'd like to avoid compiling Xen and have a stable package, but if that's the way it has to be, I can deal with it. I'd similarly prefer ZFS support to be integrated, rather than run through one of the hack-on kernel extensions/FUSE modules available for non-Oracle Linuxes. Again, if that's not possible, I can deal.

If you care about the why: The devs need Xen 4.1 or better to do something weird I'm not clear on with CAD and passthrough, and management likes ZFS, so that's a "must have" as well. I know that dedup'd ZFS uses a load of hardware resources, but the powers that be are fine with dumping money into hardware.

I'm persistently getting the "Superblock last mount time is in the future" error when booting CentOS 6. I've seen other questions which ask how to resolve this error, but I know exactly why it's occurring: our development/testing VMs regularly have their date set to times far from the present, and have all of their filesystems remounted.

What I want to know is: how do I disable all consistency checking for superblock mount time in centOS?

I've tried tune2fs -i 0 <device> and setting buggy_init_scripts=1 in /etc/e2fsck.conf and neither has worked; the problem persists.

I don't feel like this is a product recommendation question, more of a "does this tech even exist and is it feasible" question, but if I'm wrong, feel free to give this question the boot.

Context:

Our company has a bunch of traveling employees who access the company's Exchange server via thier iDevices or android phones, but because of the data protection laws in the state where our company is based (and the nature of the data our company works with), a recent security audit found that all mobile devices (laptops, phones, etc) operated by our company need to have all company correspondence and related data encrypted all the time. For laptops, that was easy: BitLocker or TrueCrypt, problem solved. For phones and tablets, however, I'm stumped. Sure, you can put lock screens/passwords on the phones, but the data is still accessible via external extraction, as law enforcement authorities already know.

Question:

Are there any clients for Microsoft Exchange that run on iOS or Android which store local data encrypted? The people using our mobile devices do a lot of their work while offline, so just giving them OWA access with SSL connection security isn't enough. Are there apps/technologies that present an additional login credential prompt to decrypt locally stored data in the app's storage area on the phone? My gut reaction when I started looking into this was "that doesn't sound like something Apple would allow into the App Store", but I've been wrong before...

Context:

We have an AD/GPO-managed corporate network with a little over a hundred PCs running Windows 7 x64, and a few managed printers. Our Server2008R2 primary domain controller is configured as a print server for them all.

Problem:

After a recent windows update and restart (no printer driver updates were included) on the DC, a particular shared printer (Lexmark T650) has begin exhibiting some strange behavior. First, it prints a preceding and following blank page for almost every document, on jobs submitted by about half of client machines (no separator page is configured on the server or any of the clients I've seen). Second, whenever someone tries to access "Printing Preferences" on any client, they recieve the following error message (this happens everywhere, 100% of the time, and didn't happen before the update on the DC):

Once they click "OK", the prefs screen appears (with no separator page selected) and everything seems fine. I'm not even sure if these two issues are related, but everyone seems affected by one or both of those issues.

What I've Tried:

I've been hesitant to un-deploy the problem printer, or remove it via GPO, as it's pretty heavily used. I've tried updating (via MS update and our internal WSUS server) client machines and the DC. No printer driver updates have appeared, and no number of updates or restarts on the server or the client seems to have achieved anything other than my boss getting grumpy that I'm bouncing the domain controller so often. I've tried deleting the drivers on the server, and re-installing them from the original source that has worked for the past year...no change. I've tried selecting "New Driver" for one of the shared printers on a client machine, running as domain admin, and pushed the latest driver found by MSupdate back up to the DC. This changed the version number of the driver recorded in the print server manager, but caused no change--on the client I pushed from, or on any other. The error still appears.

Question:

Why the heck is this happening? Obviously, I got a bad driver from somewhere, but how do I get rid of it? I don't know of any "roll back drivers" functionality for centrally managed print drivers like Windows offers for other devices. How would I a) get this issue resolved on a client, and b) push the fix to the other members of the domain?

I'm not even sure if "robustly" is a word. Anyway.

Context:

We have a few hundred Windows 7 workstations on a LAN. We use AD/GPO management pretty heavily, but there are a lot of periodic and/or manual maintenance tasks we need to do that can't be done via GPO/scheduled task. For example, say I want to execute program X (which runs silently, in the background, and doesn't bother the user) on workstation Y, or say I want to execute task A on a workstation group B either on a schedule or on demand. Kicking the users off of their computers to do this (i.e. using RDP) is a no-no, and doesn't work on groups anyway.

Question:

What's the best way to do this that is robust enough that, after setup, I could give it to beginner support people (read: people who are phobic of the command line, and get confused with GUI interfaces more complicated than Firefox)? I'm a competent programmer, and, if there is a robust set of tools or framework out there for this type of task, I'd consider hacking something together myself if it didn't take too long. If there's some combination of tools or techniques that others use to make remote-workstation-administration doable by beginners, I have yet to find it.

For those who care about the "why": I'm midlevel IT, and was told to implement a remote management solution that allows arbitrary/scheduled remote execution, with confirmation that programs actually ran remotely, and the ability to view what they returned. "Why?" I asked, "Can't I just use PsExec and the task scheduler on a dispatcher machine?" "No," I was told, "'Joe' the second-week tech is going to be in charge of this one, and he needs something simple with a GUI."

What I've tried:

I've played with making a bunch of one-clickable "transfer files to remote computer and run them with PsExec" batch/VB scrips, but those tend to break down and don't easily support running on customizable groups. I've played a little bit with the Windows version of Puppet, but it doesn't support arbitrary-time remote execution (it's ability to group computers into a tree/node structure is really nice though). I've used an older version of Altiris, and, while it does a lot of what I want, it's interface is awful, it's slow, crashes a lot, and is probably too expensive for management. SwiftWater's DMS solution does some of what I want, but it's very underdeveloped, closed-source (not a deal breaker but not ideal), and I get the impression that support and reliability are lacking.

The title says it all. We use Zenoss in a midsize network with lots of different Linux boxes. A lot of them have SMTP on by default, but we don't use it for anything; not on a single box. The Zenoss monitoring template "Status/IPService" keeps reporting various errors/timeouts for SMTP. Increasing the timeout threshold (zStatusConnectTimeout) fixes some of the timeout-related SMTP errors, but others persist. Apparently some of our unrelated configs screw SMTP up in some way.

I know I can filter out SMTP-related errors via an event view, but that feels like a band-aid solution. How do I disable Zenoss monitoring the SMTP service entirely? We don't use it, and probably never will (our mailservers aren't Zenoss monitored).

I'm fine editing .py files if I need to, but I'd hope that Zenoss provides some way to turn off monitoring of single services. I've RTFM, but all documentation just pointed me to the IPService template, which doesn't seem to have configurable settings.

The Context:

We ran Zenoss 2.2 for way too long, and decided it was time to upgrade to the current version, 3.2. I followed the upgrade steps here, and checked to make sure everything was working after each incremental version upgrade. I made sure to restart when unsure, and verified that all of the database migrations (and the preupgrade ZenPack) applied properly. Everything worked, with the exception of a couple of outdated ZenPacks that we weren't using much anyway. We run CentOS 5, so I used the old rpm'd versions available on sourceforge for the intermediate upgrades, and then made the last "hop" to 3.2 using yum. For some reason yum didn't install the core zenpacks for 3.2, so I did that manually, and then everything was fine: zenoss worked great, no issues. Well...except one.

The Problem:

We schedule backups using cron/mysqldump of Zenoss' events database. Those backups go off daily, and usually take up around 7-800MB of space. After the Zenoss upgrade, those backups are taking up more than 200GB (with a 'G'!) of space, though I haven't seen one finish yet. To run the backup, we are using

mysqldump -A | gzip > dump-12-28-2011.sql.gz 

We have MySQL 5, so --extended-insert is a default parameter for mysqldump. Zenoss is the only thing using the database (and the "events" database is the only one present besides the mysql one), and I've turned Zenoss off for the duration of the dump. My ibdata1 (we only have 1) file is around 13GB now. I don't know how big it was before the upgrade. I have tried this solution to delete old events detail entries. The query ran forever, but afterwards the dumps balloon in size the same way they did before. Why is this happening? I have backups from before the upgrade, should I revert?

TL;DR:

Why are my Zenoss "events" database dumps 1000X larger after a multiversion upgrade?

We recently had a SAN failure that killed a very high-disk-activity VM while it was on. We fixed the failure, but can't boot the VM. When we attempt to, it displays this error message. I checked the datastore, and that file doesn't exist. However, the datastore for that VM has the same contents (with slightly different sizes) as it did before the crash, and all that are in it are snapshots, tiny (<5kb) .vmdk files, and one big -delta.vmdk file. There are no "flat" base images, and there is no vmdk file corresponding to the delta. Here is a picture of the datastore's content. It seems like every vmware forum/KB article I browse assumes that there are flat files or other VMDKs present in my file structure. How do I get this thing to boot if all I have left is a delta and some snapshots? (And no, there is no chance of a recovery from backup. The last screenshot displays literally all of the files I have to work with). Is there some way to rebuild a working (even if not current) VM image from the collection of files that I have?