TessellatingHeckler's questions

IOMeter testing says the IOPs of the storage subsystem can range from hundreds of IOPs for 100% random loads to tens of thousands IOPs for 100% sequential loads.

Is there any way I can measure or estimate the random/sequential distribution of the IO we actually have coming from a mix of servers on VMware ESX hosts?

From the SAN, I can get stats about Read Percentage, about Read cache hit rate, and about current IO counts. That says we've got ~40% read / 60% write distribution, and averaging 1,000 IOPs varying from lows of 350 to highs of 2,100.

The Read Cache hit rate is very variable, it goes <1% up to 80%, and averages 36%. Can I use that variability to conclude anything about the randomness of the IO, or is that not a good correlation?

My goal is to have Apache authenticate authenticate against AD, without prompting for a username and password, using Kerberos. It currently always shows "401 Unauthorized", and appears to be not trying Kerberos. I can't find any error logs, and can only find one kinit command that clearly doesn't work and gives an error - that error shows up in WireShark as:

Linux sends AS-REQ
Windows replies KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN

Clue or red herring?

Setup / background details

Windows Server 2008 R2 SP1, with Active Directory ("winserver").
CentOS 6.5 ("centos").

1. Configure hostname and DNS and add A and PTR entries in Windows DNS server. Name resolution appears OK.

2. Create /etc/krb5.conf and /etc/samba/smb.conf using authconfig-tui and tweak them.

krb5.conf

`/etc/krb5.conf`

[libdefaults]
 default_realm = SITE.EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 SITE.EXAMPLE.COM = {
  kdc = winserver.site.example.com
  admin_server = winserver.site.example.com
 }

[domain_realm]
 site.example.com = SITE.EXAMPLE.COM

and Samba

`/etc/samba/smb.conf` includes

workgroup = SITE
password server = winserver
realm = SITE.EXAMPLE.COM
security = ads
kerberos method = system keytab
winbind use default domain = true

From here, Kerberos setup steps:

1. net ads join createupn=HOST/[email protected] -k -U administrator asks me for a Windows password, and accepts it. It joins the domain (creates a computer object in AD).

2. net ads testjoin reports "join is OK".

3. net ads keytab create -k -U administrator creates /etc/krb5.keytab and fills it with HOST/centos.... permutations.

4. net ads keytab add HTTP -k -t /etc/krb5.keytab -U administrator adds HTTP/centos.site.example.com entries to the keytab. klist -ke shows a lot of SPN combinations, including HTTP/[email protected] and appears to match the KVNO with Windows' msDS-KeyVersionNumber attribute on the server object.

At this point, I can connect by SSH using PuTTY and be automagically logged in with SSO with no password prompt. I can connect by SSH and force a password prompt, use a Windows username and password and log in. After logging in using a Windows account, I can kinit and then klist see Kerberos tickets. I can use wbinfo -u and wbinfo -g and net ads search to query AD and get results.

Everything looks good. Let's move on to Apache:

1. chgrp apache /etc/krb5.keytab

2. chmod ugo+r /etc/krb5.keytab

3. yum install mod_auth_kerb

4. Edit /etc/httpd/conf/httpd.conf and add:

httpd.conf

LoadModule auth_kerb_module /usr/lib64/httpd/modules/mod_auth_kerb.so

AuthName "Kerberos Authentication"
AuthType Kerberos

Krb5Keytab /etc/krb5.keytab
KrbAuthRealms SITE.EXAMPLE.COM
KrbMethodNegotiate on
KrbMethodK5Passwd off
KrbServiceName Any

In Internet Explorer, "Local Intranet sites" includes http://*.site.example.com and is set to 'automatic logon only in intranet zone'. Automatic login works for other (IIS based) websites in the office using these settings. In FireFox, about:config has network.negotiate-auth.trusted-uris; .site.example.com.

In IE, FireFox or Chrome, I visit the site with the FQDN, get 401 Unauthorized. It doesn't appear that Kerberos is even being tried.

I can't find anything in the Linux log files, /var/log/httpd/error_log or access_log /var/log/secure or /var/log/audit/audit.log to show anything going wrong.

Trying to find things to test, the only thing I can find to create a clear and relevant-seeming error is the command:

$ kinit -k -t /etc/krb5.keytab HTTP/centos.site.example.com
kinit: Client not found in Kerberos database while getting initial credentials

I don't know exactly what it's doing, but if I WireShark it:

Linux sends AS-REQ to Windows with content
    cname name-string 2 items
    KerberosString: HTTP
    KerberosString: centosserver.site.example.com
    realm: SITE.EXAMPLE.COM
Windows replies KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN 
    error-code eRR-C-PRINCIPAL-UNKNOWN (6)

On Windows, if I use setspn -L centos it shows Registered Principal Names include HTTP/centos.site.example.com.

On my Windows desktop client, if I klist get HTTP/centos.site.example.com then I get a ticket showing up as Client: [email protected], Server: HTTP/centos.site.example.com.

It feels a bit like http://support.microsoft.com/kb/951191 - but that only applies to Windows Server 2008 original, pre Service Pack 2. This is Windows Server 2008 R2 Service Pack 1. I've tried pointing it to another domain controller, but that's also running the same Windows Server version and responds the same way.

On the Linux side: I've been working with CentOS 6.5 (Samba 3.x, Kerberos 5.0), but I've also fresh installed CentOS 7 (Samba 4.x and Kerberos 5.1) and tried the same setup and got the same results from kinit. I haven't tried Apache on CentOS 7.

Passwordless logins for SSH work. Login authentication against AD works with kinit. Kerberos authentication doesn't work in Apache, it doesn't even seem to try in Apache, or log any errors. What am I missing, why doesn't it try Kerberos authentication? What can I test or do to make it log more?

And the only thing I can clearly make fail is that command ^ which I found on the internet, but I don't know exactly what it's testing. What is it doing and why doesn't it work, when apparently, my desktop can query AD for the same service principal and get an answer?

[Edit update]

I followed natxo asenjo's answer to 'do it differently', and now I have a setup that works. I also accept that having different usernames in AD for each service, and then creating separate keytabs, is reasonable (e.g. if Apache is compromised and the HTTP/ keytab it can read is compromised, the HOST/ keytab is still safe for a bit).

However I have no answer to why the Kerberos principal lookup to windows was failing with my original approach. Which is frustrating, given that guides like http://requesttracker.wikia.com/wiki/Kerberos_SSO_with_Active_Directory_Integration and http://wiki.gentoo.org/wiki/Kerberos_Windows_Interoperability put several principals in one keytab tied to the computer account, using the Samba net ads commands, and (presumably) get a working setup out of it.

Server 2008 R2 SP1, with Remote Desktop Services.

Office 2013.

[Edit: Most] Microsoft updates installed.

Publish Outlook as a RemoteApp, and run it from a Windows 8 client, then open an email. What happens is this:

• The email opens and renders briefly, but the Outlook inbox becomes partly-visible through the email.
• Most of the email is invisible, as if it was in the background, except the border remains visible like an embossed imprint.
• The partly-invisible email is actually in the foreground, the visible Outlook main window is in the background.

Moving things around, clicking on them, causing a refresh, fixes the display. Sometimes waiting does. It happens pretty much every email, but not exactly every one.

Disabling hardware acceleration in the Outlook options doesn't make a difference, nor does disabling add-ins, nor does changing the LoadBehaviour in the add-in registry entries.

Using a full remote desktop session, Outlook works fine.

It happens on multiple Windows 8 clients, too. Not sure about Win 7.

Any ideas, please?

[Edit: Disabling themes, menu animations, bitmap cache persistance, font smoothing, etc. in the RDP connection - all make no difference].

[Edit: Also tried: - Disabling desktop composition, and menu/window animations on the server - Running Outlook in 'safe' mode (with Outlook.exe /safe) - Changing Outlook theme - Connecting to my PC using RDP, then opening the remote app, to try and get around it being my video drivers. - The latest round of Windows/Office updates on the servers. - Removing/recreating my RDS User Profile.

All no change.

Tried on 3x Windows 8 PCs, all exhibit this behavior. Tried on 3x Win 7 PCs, only one does this, the other two appear OK.

]

We have a DFS namespace at \\company.example.org\dfs\ and inside that are entries for half a dozen folders. There is one namespace server, and no replication. The shares are all now on one server. Very basic.

I use Windows 8 Pro as a desktop, with a drive mapped to \\company.example.org\dfs\companyshare\

Suddenly, and arbitrarily, it stops working, with the error "Windows cannot access \\company.example.org\dfs\companyshare\"

I browse to the \dfs\ namespace folder, only one of the endpoint entries is there, the rest are missing. I can ping the namespace server and browse to the share as \server\share.

After a few minutes ... they reappear. Or I reboot, and it works again. This seems to happen on more than one Windows 8 client, but not on Windows 7.

I have run the DFSDiag tests linked, and they all come back fine.

In the properties of the \company.example.org\dfs folder, on the DFS tab, I see one entry in the referral list, which is active and status "Okay".

There are three domain controllers, 2008 R2 twice and 2008. All service packed and updated.

No event log errors that I can find on server or client.

Summary: Win 8 drops most of the DFS endpoints, and then a few minutes later, refreshes them. Nothing else seems to be wrong.

Where can I look for more information?

[Update: So far I have used the dfsutil diagnostics from TheFiddlerWins' answer, and found absolutely nothing. The only progress I've made is realising that the virtual machine in question still had 1 core and 3GB of RAM assigned, and now I've moved lots of roles onto it (printers, several heavy file shares), that was rather underpowered. Since increasing the available CPU and RAM for the machine last week, I haven't noticed this issue occuring again. Too early to say for sure that it's gone, but it might be that simple].

Given an RDS cluster with NLB, users connect and login to one of the servers at random.

But there is one program that's only available on one of the servers (licensing reasons).

Can we force a group of users to always be logged on to one specific server, and let everyone else have logins allocated normally?

I have a folder with permissions:

• Administrators (group): Full.
• J. Bloggs: Full.

I'm logged in as a member of the Administrators group.

I can't open the folder in Explorer because "you do not have permission".

I suspect this is because normal processes do not have the administrator permission token because of UAC, unless you also 'run as administrator'. But I can't do that for Windows Explorer, can I?

So my options seem to be: - Click the button to take ownership (ruins ownership, takes ages on large folders, doesn't solve for other administrators) - Add each individual administrator account with full permissions so it works without the admin token (administrative mess, what's the point in groups)

This is a really annoying design, I must be missing something. How is it supposed to work? What's the 'right' way for an administrator to get into a folder that administrators have access to?

Assume an office of people, they want to limit HTTP downloads to a max of 40% bandwidth of their internet connection speed so that it doesn't block other traffic.

We say "it's not supported in your firewall", and they say the inevitable line "we used to be able to do it with our Netgear/DLink/DrayTek".

Thinking about it, a download is like this:

HTTP GET request
Server sends file data as TCP packets
Client acknowledges receipt of TCP packets
Repeat until download finished.

The speed is determined by how fast the server sends data to you, and how fast you acknowledge it.

So, to limit download speed, you have two choices:

1) Instruct the server to send data to you more slowly - and I don't think there's any protocol feature to request that in TCP or HTTP.

2) Acknowledge packets more slowly by limiting your upload speed, and also ruin your upload speed.

How do devices do this limiting? Is there a standard way?

We have a setup of ADSL line -> Cisco 837 ADSL router -> Zyxel ZyWall 35 firewall/NAT -> Switch -> LAN.

It has been fine for years, suddenly DNS resolution stopped working from the LAN to public DNS servers. No changes that I know of, so I can't revert anything.

Current behaviour:

• DNS requests from the LAN using TCP show up in the oubound firewall log, in the Cisco debug log, in the dns-server-firewall, in tcpdump on the DNS server, the answer comes back, it works fine.
• DNS requests from the LAN using UDP show up in the outbound firewall log, in the Cisco debug log, but does NOT show in the dns-server-firewall, not in tcpdump on the DNS server, times out.
• DNS requests from the Cisco using UDP show up in the dns-server-firewall and in tcpdump on the DNS server, answer received, works fine.
• netcat connections to port 53 or a random port by TCP show up in the dns-server-firewall
• netcat connections to port 53 or a random port by UDP do not show up in the dns-server-firewall

Summary: TCP seems fine throughought. UDP works from the Cisco over the ADSL, and it works from the LAN to the Cisco, but it doesn't seem to cross the Cisco 837 properly. Update: confirmed with netcat that any UDP traffic from the LAN is affected, not just traffic to port 53.

Update: If I change the firewall's external IP to any other IP in the subnet, this starts working. When I put it back, it stops working. I now suspect it's an ISP issue (does that sound plausible?), and am removing the Cisco config.

Have a single Sharepoint webserver backed by a SQL 2005 server. The sharepoint server has a messed up document in a document library and I have a SQL database backup from the day before.

What are my options to get just that document back?

Once upon a time I set about getting a Linux server to use our Active Directory for logins, and I got it so that I could login as myself and then work on the server, and SU to root to make system changes using the server's local root password.

However, part of the plan we wanted was to not have separate root accounts on every server with passwords which need tracking and updating.

What's a way around this? Can root be a centralised AD account like a Windows Administrator account? Can non-root users be given root permissions in a way that isn't a bodge?

I have a Windows 7 laptop (domain member), and when I connect a VPN to the office, within 5-10 minutes the main file server/domain controller reports a failed logon for the user account, then 3, then 5, then the user account locks.

It happens with no extra software running, so it's not an Outlook problem, for instance.

I have been through control panel and cleared out the Windows and .Net password cache. There are no mapped printers, there is a mapped drive but it appears to work, and there are no scheduled tasks I can find which run so often. I have reset the user's password and put the new one on the laptop.

I have also put Wireshark on the laptop, but all that can see is "encrypted traffic" going over the VPN.

A friend has said there are issues with Windows 7 and Windows 2003 Server, e.g. with the LM hash settings which needs a hotfix. He was quite vague and there's another user with Windows 7 for whom this doesn't happen.

My suspicion is that it's Windows doing this, not a third party installed program.... but how can I dig into this further and find what's causing it?

****Update: I have disconnected the mapped drive, and the problem still happened - 4 login failures within two minutes of connecting the VPN to the office.

Event logs on the server shows Source: Security, Category: Account Logon, ID 675, Pre-authentication failed. Username: (computer-name)$, Service Name: krbtgt/domain.fqdn.example.org, failure code: 0x19

Followed by Event Security / 680 / Account Logon, MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 / Error Code: 0xC000006A Account logoff / Unknown user name or bad password (both repeat 4 times)

Update

I think I have it - the VPN username is the same as the Windows account username, but the passwords are different (because the VPN connects to a firewall, not a Windows server), however when accessing network resources Windows seems to be trying the VPN credentials first - because the account name is the same, the password failures cause the Windows account to become locked.

I've changed it so the VPN uses a different username and in a few minutes of testing, that seems to workaround it at least.

Would be interested if anyone knows why the (Windows 7 PPTP) VPN might behave like this and if there's an official way to stop it.