ryandenki's questions

Situation:

We recently turned off DHCP in favor of fixed IP addresses, and in the middle of the old DHCP range for the legacy network segment, we found a device responding to ping but without any of our normal remote access services running (SSH, RDP, SMB/etc).

NMap version identification fails to find anything, and we get the following results:

PROTOCOL STATE         SERVICE
1        open          icmp
4        open|filtered ip
6        open          tcp
17       open          udp
66       open|filtered rvd
96       open|filtered scc-sp
136      open|filtered unknown
157      open|filtered unknown
214      open|filtered unknown
235      open|filtered unknown
251      open|filtered unknown
MAC Address: B8:AC:6F:95:06:64 (Dell)

We don't get anything back on connecting to any of these open ports, and there is no documentation that any device should be there. All of our known devices are documented, so the existence of this device is a mystery.

I don't like mysterious devices.

Questions:

Does anybody know what device might have all of these strange ports open?

We have a lot of DELL workstations and servers, but we've checked against every single known asset.

Can anybody suggest a way to access the device?

Or, can anybody suggest how to physically find it other than yanking cables from various hub segments until it becomes unreachable and narrowing down from there?

Problem machine is Windows Server 2008 R2 file server with the following drives, all NTFS:

C: OS

D: File shares

E: DFSR staging folders, temp files, and Volume Shadow Copies

The D: drive is configured to place volume shadow copies on E:, and the C: drive has volume shadow copies disabled.

A week ago, a shared folder was removed and the DFS referral deleted. This left an orphaned DFSR replication item, which was removed from display, and all of its staging folders deleted manually.

Several days later, Volume Shadow Copies stopped working.

Run as the standard backup user, an attempt to create a shadow copy fails with "Access Denied", but when run at an elevated command prompt, it fails with a "File system restriction".

Running CHKDSK on E: returns "The type of the file system is RAW." Disk Manager shows "NTFS". Using the "testdisk" tool, the partition table still shows NTFS.

All of the files on the E: drive can be accessed, and temp files are being read and written normally. Existing volume shadow copies can be accessed without trouble.

Using the C: as a test, I found that volume shadow copies can be enabled with the target volume set to C: or D:, but cannot be enabled with the target volume set to E:.

Any ideas what's going on, and could it be related to the earlier DFS issues where the sysadmin deleted settings in the wrong order?

Setup

Using vCenter and ESX 4.1, we installed VMware Data Recovery 2.0 using two appliances.

Once appliance has a single 1TB (892GB free) destination on a mapped raw iSCSI LUN.

The other appliance has two 500GB (440 and 450GB free) destinations on CIFS shares to two Windows 2003 servers.

The virtual machines to be backed up are on a 1.9TB (1.3TB free) VMFS iSCSI LUN.

The iSCSI network is segregated from the vConsole network, and routing between them is not possible. Name resolution for the ESX hosts works from the Data Recovery appliance VM, and vice versa.

Issue

We configured a daily backup job on 10/21 and set it running.

It ran correctly every day, although there were occasional error -3948 (vcb api exception) messages, though running the backups again would be successful.

Everything was fine until 11/4, on which every backup job failed with error -3902 (file access error). Since then, no job has completed--all jobs fail with the same error.

Status

Both appliances show the same issue starting on the same day: no backups are being made to any destination.

VM snapshots can be created and deleted manually, and no delta files are present in any VM folders.

The vmware.log for each VM shows that a snapshot was created successfully, and then immediately removed.

VMs newly added to the existing backup jobs show the same behavior.

New backup jobs show the same behavior.

All destination datastores pass integrity checks.

Creating a new CIFS datastore shows the same behavior.

Creating a new ESXi 4.1 trial version (fully licensed) server using local disk datastore, creating a new VM, and creating a new backup job, shows the same behavior.

No (known or intentional) configuration changes were made.

Additional oddities

Rebooting one of the two DR VMs caused its IP address to revert to DHCP, and all destination, backup job, and configuration information was lost and had to be re-input.

Question

• What does error -3902 (file access error) mean? I.E. what file, from where?

• What else should we look at? Are there more detailed logs somewhere?

• Does anybody have DR working successfully?

Our shop is primarily ESX 4.1, which I and others are very familiar with, but we also have a few test ESXi 4.1 servers running the free version of ESXi, originally installed using the 60-day evaluation version, but now using a "free ESXi" license key from our VMware management account.

All of these servers are Dell R610 with 32GB RAM, single X5450 CPU, and RAID1 136GB local disk. ESXi is installed to the local disk, with the remainder configured as a VMFS volume. No shared storage is being user.

As of Friday at 18:00, all of the servers were running properly.

As of Saturday at 15:30, one of the servers appeared to have been re-installed.

Two of these servers are located in our office, where the weekend admin staff performed a power-off test this Saturday. This test consisted of literally throwing the breaker for the entire building. None of the servers in question are attached to a UPS, though they do have write cache and batteries on their RAID controllers.

When the machines booted after the test, one of them lost the free license key (reverting to an expired evaluation license), and the other reverted to initial installation settings (DHCP, no password, empty inventory), and the evaluation license had reset, giving another 60-day evaluation period from Saturday at 15:30.

The first of these servers was fixed by simply re-entering the free license key through VIclient. All inventory and settings were in the same state they were in on Friday.

The second of these servers was in exactly the state expected after a new installation or re-install; namely, all settings had reverted to defaults, and there do not exist any log or configuration files dated prior to the power-off test. Logging in through the unsupported service console shows that the very folders in the root directory were also dated from after the power-off test.

However, the VMFS volume contents were intact, just as if somebody had performed a "repair" installation from CD-ROM.

This server was repaired by following our standard checklist for repair installations: configuring the network, adjusting server settings, and re-adding machines to inventory from the datastore browser.

Question: is there anything besides a manual repair install which will reset an ESXi server to its original installation defaults, and set all service console folders, configuration files, and log files to the date and time that the server booted?

Yes, I am aware that in a diskless installation, this is pretty much what happens on every boot; however, this is not a diskless installation, but rather is installed and boots from the local disk.

However, I am not familiar enough with ESXi to know if this is also normal for an installation on disk.

Tests: Since both servers are configured identically, we used the first server to try to find out what happened to the second.

1. I did another power-off test of that server only, to see if it also reverted to defaults when it booted. It did not; it retained all settings and booted normally, twice. (Unfortunately, we did not check to see if the folder, configuration, and log files were reset to the boot time.)

2. I did a repair install to verify that the configuration and log file dates would all be updated to the time of the re-install, and that all configurations would be reset to defaults as happened with the second server. It did; after a repair install the first machine was in exactly the same state as the second had been, and the evaluation license was similarly reset to a new 60-day period.

Follow-up question: Assuming this happened without user intervention, why did it happen and how can we prevent it from happening again?

Real question: should I believe the weekend admins who said they didn't do anything to the machine? None of them are certified on our VMware systems, but know enough to be dangerous if they got it in their head to try to fix a problem.

Please tell me that I'm wrong in thinking the weekend admins are hiding something they did wrong, and that this can happen for reasons other than manual intervention.

Our application runs 50% slower on a Xeon X5650 under ESX than on a bare-metal E5450.

A test task on the physical servers takes 17 minutes.

The same task on the virtual servers takes 25 minutes: 50% longer.

From everything I can tell, this should be impossible; the 5600 series is supposedly 10-20% faster than the 5400 series for single-threaded processes at the same clock speed, and virtualization overhead should similarly low for single-thread CPU-bound workloads. Performance should at least break even, shouldn't it? But instead of having equal or better performance, performance is cut by 1/3.

UPDATE: Solved. The same task on the (fixed) virtual servers takes 14 minutes: 15% faster.

It was the RAM configuration. The 50% performance drop was because the ESX host system memory installed wrong and only providing around half of the full possible bandwidth. For the CPU & memory-bound process, that loss of bandwidth translated into 50% worse performance than expected.

Application performance is now smack in the middle of the 10-20% improvement we originally expected.

There are two physical Windows Server 2003 R2 systems which run an application which consists of a single-threaded calculation run on one server (32-bit), talking back and forth to an SQL Server 2005 database on the other (64-bit).

Both physical boxes are single-CPU E5450 with 4GB RAM @800Mhz. The calculation server never uses more than 1.5GB physical memory, and the SQL Server never uses more than 2.5GB. CPU utilization on the calculation server never goes over ~15% (around 50% of a single core). CPU utilization on the DB server never goes above ~25% (a fully-utilized single core).

The physical ESX 4.1 hosts are dual-CPU X5650 with 64GB RAM @1333Mhz. The virtual machines are given 4 cores and 4GB RAM each, to mirror the physical environment. The test was made with a single VM running on each physical host, as well as with both running on the same host.

Interestingly, we get very nearly the same 25-minute test results on another pair of ESX servers using X5550 CPUs and RAM @1066Mhz.

Also, test results in the virtual systems do not vary by more than 10% either way giving the VMs 1, 2, or 4 CPUs, or 1, 2, 4, or 8GB RAM. There is very little network or disk activity, and as far as I can tell the process should be CPU bound.

Tests have been run using both local 15K SAS disks on the separate hosts, as well as to an gigabit iSCSI SAN also with 15K disks. There is negligible different in results for different storage.

From everything I can tell, the Xeon 5600-series should be 20-50% faster than the 5400-series for single-threaded workloads. Even considering that the X5650 is a 2.67GHz part and the E5450 a 3GHhz part, if per-core performance was equal at the same clock speed you would still expect to see at least 90% of the performance instead of 67%. This doesn't even take into account the fact that the memory clock is nearly twice the speed.

It should be said that I've done several virtualization projects in the past, and never seen a anything close to a 50% performance degredation even using the SAME physical CPU cores, let alone cores two generations newer with faster memory.

Any ideas on possible causes or any configuration settings I should check?

Most of our servers are Dell boxes running Windows 2003 R2, and we are planning an upgrade to Server 2008 across the board.

To that end we have already upgraded one of our servers using an in-place upgrade, and everything seems to be working fine--with Windows and our applications.

But Dell's OpenManage is not working well at all. It starts fine, and runs initially, but after a few days, the services stop responding and we can no longer use it properly. Battery and power status are reported, but storage, memory, fans, temperature, and voltage monitoring stop responding.

When this happens, "DSM SA Data Manager" service cannot be restarted, and remains in "stopping" state when stopped or restarted. Killing the process via Task Manager allows it to be started, and then things are fine for another few days.

We have tried uninstalling and reinstalling, but to no effect.

This is with OpenManage 6.3.0 in Windows Server 2008 R2 x64.

Ideas?

Last night we received a notification from vCenter saying that it couldn't connect to the agent on one of our hosts, and that the "Host and Power status" was in error and the server was disconnected.

There were no issues with any guests running on the host, so we left it for the morning.

But when checking the task & event and alert logs, we find nothing. The host logs also show no issues at that time.

No reference to anything having gone wrong, nothing to tie that notification back to.

Even if the issue was temporary and fixed itself, should't there be something in the log indicating that some type of trouble occurred?

Also, if it in fact automatically recovered, why didn't vCenter send its usual "Oh hai, everythings fine nao" notification when the system recovered?

We have recently set up a Terminal Services Gateway website to allow external access to a Remote Desktop server. Each user installs a client certificate on their personal computer, connects to the HTTPS TS Gateway site, logs in, then connects to the configured remote desktop machine.

All users have the same access rights and configurations on the server side, and all certificates are issued from our local CA; the certificate chain is also installed on their local PCs.

Out of 30 users, two specific users are unable to use the system. They connect to the TS Gateway site using their client certificate, can log into the the TS site using their user accounts, and can then click on the configured remote desktop machine. They are prompted for login username and password, then see a "Connecting" dialog. After a while, they receive the message:

The computer can't connect to the remote computer. 
The two computers couldn't connect in the amount of time allotted.

We've checked everything we know how to, even silly items that shouldn't be related: they are using the same Windows XP SP3 as several other users, their Windows Firewall is enabled with port 443 open, each of them is using the same internet service provider and plan as another user who is connecting successfully, it still happens even after trying to issue a new certificate, and their system clocks are synchronized to the same public NTP source as the server they are connecting to.

Unfortunately, diagnosis is difficult since we don't have direct access to their home machines to try for ourselves.

What else should we check?

One of our Windows 2003 R2 ISA firewall servers recently had a 1-hour spell of intermittent connectivity to our Internet WAN connection, which was fixed after disabling then re-enabling the network adapter. The error log showed multiple instances of the following error:

Event 4199:
The system detected an address conflict for IP address 192.0.2.123 
with the system having network hardware address 00:00:00:00:00:00. 
Network operations on this system may be disrupted as a result

As far as I know, a null MAC address is impossible. I only found one Google result for the same error with the same strangely empty MAC address, an experts-exchange question with no answer.

The problem has not recurred, but I'm nervous having a server with unexplained issues still running as part of our live network. Without knowing the cause, I have no way of knowing that it might not happen again.

Is this a driver failure? Hardware failure? Network stack issue? Could there have been some strange conflict where something somehow tried to take over our WAN address?

My company has a Windows 2003 root certificate authority server which is used to generate client certificates for Remote Desktop Services logins, as well as certificates for internal HTTPS websites.

It recently developed some problems, and we would like to reboot the server.

Those problems are the inability to login remotely via Remote Desktop due to a "RPC server not available" error, and the lost ability to create new certificates. We tried stopping and restarting some of the services, but several of them remain stuck in the "stopping" phase. The server uptime is something close to a year and a half, and the assumption is that a restart ought to bring everything back up fresh.

However, several IT staff members are claiming that if we reboot the CA, all services on all servers (IIS, SQL Server, etc) will stop working until the system is back online.

I can't find any Microsoft documentation to support that position, but neither can I find any documentation that proves that there will not be any impact to running services.

Does anybody here know for certain what potential impact there may be for rebooting the company root CA server?

Running ESX server 3 on four identical 4-CPU hosts, guests on Fibre SAN VMFS.

Guest OS is Fedora 10. Cloned it to create web, jboss, mysql, and memcached templates. Cloned each template into four guests, one for each server.

Out of these 16 guests, one jboss and one mysql guest run so slowly as to be unusable. By "slowly", I mean that no matter how CPU-intensive the process launched, they never utilize more than ~200Mhz CPU. Moving them between hosts has no effect--it appears to have something to do with those guests themselves.

BUT! Today I found that they will run at nearly full speed if I either:

• Hold down the spacebar in the console
• Open an SSH session and, hold down some repeating key
• Flood them with ICMP packets

In other words, any kind of I/O activity seems to "wake them up", and all processes run at perfectly normal speeds during this time. Stop that I/O activity, and they again slow to a crawl. So apparently, their processes aren't being scheduled unless there is some sort of interrupt activity.

Any ideas why?

All guests are fully patched as of today. openvm-tools is installed, guest time sync enabled, kernel parameters are "notsc" (but changing that doesn't affect this issue).

Have used rsync in --dry-run mode to verify that /bin, /usr/bin, /var/jboss and /var/lib/mysql are identical to the normally-behaving guests, and that /etc only varies in hostname, IP address, and other instance-specific settings.

Have tried setting their resource utilization to "high" with no effect. (All guest resource utilizations are "normal", except for memory reservations on all JBoss and MySQL guests. The total memory reservations per server are about half of the host memory, and all guest memory sizes added together only use about 70% of the host memory.

The *.vmx, *.vmxf, and *.vmdk files vary only in uuid, displayName, MAC address and disk/swap filenames.