Aaronaught's questions

I know that there are thousands of reports of people having trouble getting Integrated Windows Authentication to work with IIS, but they all seem to lead to web pages that don't apply or solutions that I've already tried. I've deployed dozens of sites like this before, so either there's something bizarre going on with the server/configuration, or I've been looking at this too long and not seeing the obvious.

Simply put, everything works perfectly on my local machine, but falls apart on the production server, which as far as I can tell has the exact same configuration.

On the local machine:

• The machine is running Windows 7 Ultimate, Service Pack 1, IIS 7.5.
• The site has been tested successfully, using both IIS and the VS Web Development Server.
• The IIS site config has all authentication methods disabled except Windows Authentication.
• The local machine is not on any domain.
• The Providers set up are Negotiate and NTLM (not Negotiate:Kerberos).
• Extended Protection is Off.
• All browsers tested (IE, Firefox, Chrome) show the challenge prompt and allow me to log in to the localhost domain with my (local) Windows account.
• All browsers tested also work using an opaque local IP address - so the browsers themselves don't seem to care whether the site appears "local" or "remote".
• I've added a display line to the web page which shows the currently-logged-in user and it shows exactly what I would expect (whichever local user I logged in with).

On the remote machine:

• The server is running Windows Server 2008 R2, IIS 7.5.
• Loading the web page results in an immediate 401.2 error: You are not authorized to view this page due to invalid authentication headers. No challenge prompt ever appears.
• The IIS site config has all authentication methods disabled except Windows Authentication.
• The remote machine is not on any domain.
• The Providers set up are Negotiate and NTLM (not Negotiate:Kerberos).
• Extended Protection is Off.
• On the remote machine (remote desktop session), the same error appears in Internet Explorer regardless of whether the domain is localhost or the external IP address.
• If I try to view the remote web site from my local machine, the error is still 401, but a slightly different 401. No subcode, with the text: Access is denied due to invalid credentials.
• The Windows Authentication IIS role feature is installed.
• The WindowsAuthentication Module is added (at the Server level).
• The exact same error occurs if I turn off Windows Authentication and enable Basic Authentication.
• The site does load if I turn off Windows Authentication and enable Anonymous (obviously).
• I've already followed all of the troubleshooting steps on Microsoft Support: Troubleshooting HTTP 401 errors in IIS
• I've already tried the workaround shown on another Microsoft support page (supposedly to force NTLM as the only method).

Last but not least, I tried turning on FREB for 401.2 errors and the results don't seem to tell me anything useful, all I see is the following warning:

MODULE_SET_RESPONSE_ERROR_STATUS

ModuleName IIS Web Core
Notification 2
HttpStatus 401
HttpReason Unauthorized
HttpSubStatus 2
ErrorCode 2147942405
ConfigExceptionInfo
Notification AUTHENTICATE_REQUEST
ErrorCode Access is denied. (0x80070005)

...this seems to just be telling me what I already know (that it's simply rejecting the request instead of negotiating the credentials).

The trace does indicate that the WindowsAuthentication module is correctly loaded because there is a NOTIFY_MODULE_START line with ModuleName = WindowsAuthentication (and various other ASP.NET follow-up events - [un]fortunately, no interesting errors or warnings here).

Can anyone tell me what I might be missing here?

Quick Update:

I'm a little uncomfortable sending a whole Wireshark dump as it would reveal IPs, URLs and other stuff, but I did a side-by-side comparison of the HTTP responses from localhost and the remote server in Fiddler, and it seems fairly self-evident what the problem is:

Localhost:

HTTP/1.1 401 Unauthorized
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
X-Powered-By: ASP.NET
Date: Sat, 17 Dec 2011 23:42:34 GMT
Content-Length: 6399
Proxy-Support: Session-Based-Authentication

Remote:

HTTP/1.1 401 Unauthorized
Content-Type: text/html
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Sat, 17 Dec 2011 23:43:13 GMT
Content-Length: 1293

Aside from a few seemingly-inconsequential differences like cache-control, the main difference is that the remote server is not sending the WWW-Authenticate headers back to the client.

So, I guess that narrows the question down to: Why is IIS not sending WWW-Authenticate headers when Windows Authentication appears to be installed, loaded, and exclusively enabled?

We set up a new server here a few weeks ago that I am informally responsible for managing.

Almost everything works perfectly except for one thing: Every so often it hangs without warning.

Some facts about this hang:

• It is not a single application or service; the entire system is non-responsive.
• Nothing is displayed (monitor acts as though there's no VGA signal).
• The power LED is on and the fans are running.
• Pressing the power button does nothing (normally it would shut the machine down).
• Pings generally time out; once it did respond, another time I got "destination host unreachable".
• Event logs show nothing (literally nothing at all) from before the hang until the hard reboot.
• There are no performance problems, strange errors, or other obvious signs of impending doom leading up to the eventual hang.
• The machine is generally not heavily loaded (it's for development, not production), and the hangs appear to be occurring at non-peak times of day (between midnight and 6 AM).

Some additional facts about the machine/environment:

• Windows Server 2008 R2
• Running SQL Server 2008 and IIS (not much else)
• All drivers up to date, patches installed, etc.
• No vendor-supplied diagnostics (not "top tier").
• The machine is completely new, not merely reformatted or repurposed. No recent changes although the machine is less than a month old to start with.

I don't expect any easy answers here. What I'd like to know his I can methodically determine the root cause of this problem, be it a misbehaving service, defective hardware, or something else.

Is there any kind of logging I can set up that will help me get to the bottom of this? Any hardware diagnostics or remote monitoring? Anything else I can do to help me discover what's actually happening, or at least be able to eliminate what isn't wrong?

Just to reiterate, I really don't want to start speculating about possible causes and take a trial-and-error approach, because it's going to be at least several days at a time before I would have conclusive results. I'm looking for solutions to reliably trace the problem to its source.

When I run DBCC CHECKDB, I get a slew of error messages like this:

Table error: object ID 2020918271, index ID 1, partition ID 72057594196590592, alloc unit ID 72057594190233600 (type In-row data), page (4:129574), row 0. The record check (valid UDT column) failed. The values are 3 and 0.

Right now I'm just trying to understand what this error means and how serious it is - it seems to be some checksum-related error and probably not that severe. Recent backups seem to have the same problem, but it's not a big deal either way because the data can be recreated from other sources.

Anyway, I'm trying to get to the bottom of this, maybe understand which rows are corrupted, if there's any particular pattern to it. When I run DBCC PAGE to see what's in there, such as the following statement:

DBCC PAGE('MyDB', 4, 129574, 3)

It shows nothing. Nada. Zip. Just the standard:

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

But no error and no page data. In fact, every single error that's coming out of CHECKTABLE has a file/page number for which I get this output from PAGE.

I'm also seeing the following error from the CHECKTABLE output, but only sporadically:

Table error: Object ID 2020918271, index ID 1, partition ID 72057594196590592, alloc unit ID 72057594190233600 (type In-row data). Page (4:129575) was not seen in the scan although its parent (4:129977) and previous (4:129574) refer to it. Check any previous errors.

It looks like it might be related, but I'm not really sure how. The UDT can be relatively large (about 5 KB), so maybe it's split across pages and one of the pages is missing? That's just a wild guess, though.

The number of errors coming out of CHECKTABLE also makes it look as if the entire table is like this, but I know that's not the case, because I can read the data just fine. In fact, there's an automated process that runs every day that will, over time, read just about all of the data in this table, and it hasn't reported a single error. Also if I run DBCC PAGE on one of the parent pages (which do exist, even though the "previous" pages don't), I can get the key columns, and I can SELECT all the data for all surrounding keys without any error.

Can anybody tell me what's going on here? Does it make sense for DBCC CHECKTABLE to be giving me these errors when the pages referred to don't even exist? Is it possible that CHECKTABLE itself is emitting spurious errors?