Why use Kerberos instead of NTLM in IIS?

From a Windows perspective only:

NTLM

• works with both external (non-domain) and internal clients
• works with both domain accounts and local user accounts on the IIS box
  • using domain accounts, only the server requires direct connectivity to a domain controller (DC)
  • using local accounts, you don't need connectivity anywhere :)
  • you don't need to be logged on as the user in question to use a credential
  • Aside: it's not that uncommon for a DC to be overwhelmed by a busy NTLM server (IIS, Exchange, TMG/ISA, etc) with the volume of NTLM requests (to mitigate: MaxConcurrentAPI, AuthPersistSingleRequest (false), faster DCs.) (Self-referential bonus.)
• requires client connectivity only to the IIS server (on the site port, nothing else. i.e. Everything happens over HTTP (or HTTPS).)
• can traverse any proxy supporting HTTP Keep-Alives
  • you may be able to use TLS/SSL to work around others
• requires multiple round-trips to authenticate, with small packets
  • (log pattern is 401.2, 401.1, 200 with username)
• cannot be used in scenarios where double-hop authentication is required
  • i.e. the user's credentials are to be forwarded to a service on another computer
• supports older clients (< Win2000)
• Is susceptible to LM Auth Level discrepancies (mismatched lmcompatibilitylevel)
• is used as a fallback by the Negotiate package if Kerb fails.
• (not "if access is denied with Kerb", Kerb must break for NTLM to be used - usually this looks like not getting a ticket. If the client gets a ticket and it's not perfect, that doesn't cause a fallback.)

Kerberos

• works with currently domain-joined clients only

  • requires client connectivity to an AD DC (tcp/udp 88) AND the server (tickets are retrieved by the client from the DC via the Kerb port, and then provided to the server using HTTP)

• might be able to traverse a proxy, but see DC point above: you still need to be on the same network as an active DC, as does the server.

  • so in theory if you had a domain in which internet-connected clients chatted directly to an internet-connected DC, it's workable. But don't do that unless you already knew that.
  • In reverse proxy scenarios (ISA/TMG), the protocol transition server needs to be on that network, i.e. not the client... but then the client isn't really the one doing the Kerberos bit (necessarily - think Forms auth to Kerb transition).

• ticket is long-lived (10h) meaning less DC communication during ticket lifetime - and to emphasise: this could save thousands to millions of requests per client over that lifetime - (AuthPersistNonNTLM is still a thing; Kerberos PAC validation used to be a thing)

• requires a single round-trip to authenticate, but the authentication payload size is relatively large (commonly 6-16K) (401, {(encoded) token size} 200)

• can be used with ("please always use Constrained") delegation to enable double-hop scenarios, i.e. Windows authentication of the connecting user to the next service

  • actually, N-hop - it stacks like Lego! Add as many hops as needed...
  • for example, to allow UserA to access IIS, and for IIS to impersonate that same Windows user account when it accesses a different SQL Server computer. This is "delegation of authentication".
  • (Constrained in this context means "but not anything else", eg Exchange or another SQL box)

• is currently the primary security package for Negotiate authentication

  • meaning Windows domain members prefer it when they can get it

• requires registration of SPNs, which can be tricky. Rules that help.

• requires use of a name as the target, not an IP address

• reasons Kerb might fail:

  • using an IP address instead of a name
  • no SPN registered
  • duplicate SPNs registered
  • SPN registered against wrong account (KRB_ERR_AP_MODIFIED)
  • no client DNS / DC connectivity
  • client proxy setting / Local Intranet Zone not used for target site

While we're at it:

Basic

• can multi-hop. But does so by exposing your username and password directly to the target web app
  • which can then do anything it wants with them. Anything.
  • "Oh, did a Domain Admin just use my app? And did I just read their email? Then reset their password? Awww. Pity"
• needs transport layer security (i.e. TLS/SSL) for any form of security.
  • and then, see previous issue
• works with any browser
  • (but see first issue)
• requires a single round-trip to authenticate (401, 200)
• can be used in multi-hop scenarios because Windows can perform an interactive logon with basic credentials
  • May need the LogonType to be configured to accomplish this (think the default changed to network cleartext between 2000 and 2003, but might be misremembering)
  • but again, see first issue.
  • Getting the impression that the first issue is really, really important? It is.

To sum up:

Kerb can be tricky to set up, but there are loads of guides (my one) out there that try to simplify the process, and the tools have improved vastly from 2003 to 2008 (SetSPN can search for duplicates, which is the most common breaking issue; use SETSPN -S anytime you see guidance to use -A, and life will be happier).

Constrained delegation is worth the cost of admission.

From the Microsoft Application Verifier, which detects common developer mistakes. One of those mistakes is the use of NTLM:

NTLM is an outdated authentication protocol with flaws that potentially compromise the security of applications and the operating system. The most important shortcoming is the lack of server authentication, which could allow an attacker to trick users into connecting to a spoofed server. As a corollary of missing server authentication, applications using NTLM can also be vulnerable to a type of attack known as a “reflection” attack. This latter allows an attacker to hijack a user’s authentication conversation to a legitimate server and use it to authenticate the attacker to the user’s computer. NTLM’s vulnerabilities and ways of exploiting them are the target of increasing research activity in the security community.

Although Kerberos has been available for many years many applications are still written to use NTLM only. This needlessly reduces the security of applications. Kerberos cannot however replace NTLM in all scenarios – principally those where a client needs to authenticate to systems that are not joined to a domain (a home network perhaps being the most common of these). The Negotiate security package allows a backwards-compatible compromise that uses Kerberos whenever possible and only reverts to NTLM when there is no other option. Switching code to use Negotiate instead of NTLM will significantly increase the security for our customers while introducing few or no application compatibilities. Negotiate by itself is not a silver bullet – there are cases where an attacker can force downgrade to NTLM but these are significantly more difficult to exploit. However, one immediate improvement is that applications written to use Negotiate correctly are automatically immune to NTLM reflection attacks.

By way of a final word of caution against use of NTLM: in future versions of Windows it will be possible to disable the use of NTLM at the operating system. If applications have a hard dependency on NTLM they will simply fail to authenticate when NTLM is disabled.

• Kerberos has the reputation of being a faster and more secure authentication mechanism than NTLM.
• It also has historically been easier to connect to through proxy servers than NTLM, due to the connection-based nature of NTLM.
• That said, as you note, Kerberos is more difficult to get up and running, and requires a connection to the AD that isn't always practical.

Another approach would be to set authentication to negotiate and use both rather than one instead of the other.

You should add a very important point:

Kerberos has been standard and open protocol in Unix over 20 years whereas NTLM is a purely proprietary solution from Microsoft and only known to Microsoft.

Kerberos is required if you need to impersonate the user to access resources that are not on the iis server.