Sobrique's questions

I have a process that relies on detecting file creation on an NFS server.

Usually it works fine, but there's been a few minor glitches recently, due to caching and an NFS rename operation.

We've now got as far as investigating lookupcache on the NFS server:

lookupcache=mode Specifies how the kernel manages its cache of directory entries for a given mount point. mode can be one of all, none, pos, or positive. This option is supported in kernels 2.6.28 and later.

The Linux NFS client caches the result of all NFS LOOKUP requests. If the requested directory entry exists on the server, the result is referred to as positive. If the requested directory entry does not exist on the server, the result is referred to as negative.

If this option is not specified, or if all is specified, the client assumes both types of directory cache entries are valid until their parent directory's cached attributes expire.

Now, my problem is - I don't really want to globally disable lookupcache - and I think I'd have to, because of the nature of this process (it hits files in a lot of locations across our NFS estate)

Is there a way I can programatically invalidate the lookupcache and force a LOOKUP? An ioctl or other system call ideally. What I'm ideally looking for is something that's documented as a reliable approach - I have seen indications that 'opendir; closedir' will trigger a cache-invalidate, but I can't track down a source for this information.

I've got a process that:

• writes a new '.tmp' file.
• uses a rename() syscall to replace an existing file.
• This file is being accessed from a remote NFS client.

We do this, because we want atomic file updates, and the rename() spec says:

If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing. However, there will probably be a window in which both oldpath and newpath refer to the file being renamed.

We rely on this behaviour.

But here's the gotcha - just recently, since migrating to a new NetApp (Cluster mode, from 7 mode) - we've had a process that very occasionally falls over with ENOENT - no such file or directory.

By 'very occasionally' I mean - 4 or 5 times in the last few weeks, on a process that happens every 5 minutes or so.

I'm investigating with the vendor as to whether this might be a bug or not with their NFS server.

But what I'm actually trying to figure out is whether that atomicity guarantee is actually applicable to NFS. Is anyone able to clarify for me if rename()'s atomicty guarantee applies to multi-client NFS scenarios? I'm not actually sure that this feature is one that has been working, but was never guaranteed to in the first place.

From: RFC1813

Procedure RENAME renames the file identified by from.name in the directory, from.dir, to to.name in the directory, to.dir. The operation is required to be atomic to the client.

In case it's relevant, we've got SL 6.5 clients hitting NFS datastores on ONTAP-CDOT 8.3.

This might be a bit esoteric, but I've got an odd sort of a problem with stat().

Specifically - on a Linux system, with Isilon storage NFS mounted. I have a process that is getting time differences when running fstat on the same file.

I can reproduce this with a bit of C, which does nothing more complicated than open-write-stat-close; open-stat-close

And these two numbers will - occasionally - differ by a few milliseconds - a file might appear 'older' than it actually is.

I'm unclear if the 'stat' is reading the time from the host, or from the server. And I'm also not clear if the problem is with the Isilon (although it doesn't seem to happen on NetApp storage).

Can anyone give me a bit of insight into what might be going on?

As an example:

time mismatch: 1468936451.862865611 != 1468936451.860183107

The first is the time read when the file handle is open. The second is the time read after closing (and fsync).

As you can see - the file is a bit older 'on the server' than on the client.

The C to test this is:

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>

int main ()
{
  while (1)
  {
    struct stat sb[2];

    /*
     * Open file
     */
    int fd = open ("test.dat", O_CREAT | O_RDWR | O_APPEND, 0644);
    if (fd < 0)
    {
      perror ("failed to open file (1)");
      return EXIT_FAILURE;
    }

    /*
     * Write to it
     */
    write (fd, "", 1);

    /*
     * Ensure it's sync'd to disk
     */
    if (fsync (fd) < 0)
    {
      perror ("failed to fsync file");
      return EXIT_FAILURE;
    }

    /*
     * Stat the file
     */
    if (fstat (fd, &(sb[0])) < 0)
    {
      perror ("failed to fstat file (1)");
      return EXIT_FAILURE;
    }

    /*
     * Close it
     */
    close (fd);

    /*
     * Open file
     */
    fd = open ("test.dat", O_RDONLY);
    if (fd < 0)
    {
      perror ("failed to open file (2)");
      return EXIT_FAILURE;
    }

    /*
     * Stat it again
     */
    if (fstat (fd, &(sb[1])) < 0)
    {
      perror ("failed to fstat file (2)");
      return EXIT_FAILURE;
    }

    /*
     * Close it
     */
    close (fd);

    /*
     * Check the times match
     */
    if (sb[0].st_mtime != sb[1].st_mtime ||
        sb[0].st_mtim.tv_nsec !=sb[1].st_mtim.tv_nsec)
    {
      printf ("time mismatch: %d.%09ld != %d.%09ld\n",
              (int)(sb[0].st_mtime), sb[0].st_mtim.tv_nsec,
              (int)(sb[1].st_mtime), sb[1].st_mtim.tv_nsec);
    }
  }
}

And you can trigger it by running this in an NFS mount, then calling 'stat' (shell command is fine) from another system. E.g. on the isilon itself:

while true
do 
    stat test.dat
done

This gets me errors like:

time mismatch: 1468936613.313045037 != 1468936613.310174576
time mismatch: 1468936613.547768543 != 1468936613.544878047
time mismatch: 1468936615.228495345 != 1468936615.225862947
time mismatch: 1468936619.036053897 != 1468936619.033362349
time mismatch: 1468936619.988151772 != 1468936619.985529620
time mismatch: 1468936664.541226541 != 1468936664.538549678
time mismatch: 1468936666.165674866 != 1468936666.163171366
time mismatch: 1468936666.787528027 != 1468936666.784797051
time mismatch: 1468936711.334376729 != 1468936711.331868927

And ... I don't think it should.

Edit:

The plot thickens - when running the above and capturing packets, the response from the NFS GETATTR call from the server is inconsistent. I get two different timestamps.

Mount options don't seem to make a difference - but we have tried setting noac and actimeo=0, sync lookupcache=none etc.

Our isilon nodes are all ntp-synced to the same sources. As usual, there's a small amount of drift between them.

We have found that we can apply a workaround in the form of isi export modify --zone <zoneid> <exportnumber> --time-delta 1

This sets the resolution of the timestamp to 1s rather than 1ns. However whilst it diminishes the frequency of the problem occuring, it will still happen - just less frequently, because it only occurs at a '1s' boundary.

I've been testing failover functionality on a new Isilon. Running OneFS 8.0.0 (A .0 release, but .1 is on the install queue).

Part of this is performing a failover operation, following a procedure on EMC's site - https://emcservice.force.com/CustomersPartners/kA2j0000000QXbrCAG (First of all, we tried it using Superna Eyeglass, but could reproduce the problem using the documented procedure).

And on issuing isi sync recovery allow-write policyname our target transitions to enabling_writes and then seems to get stuck.

The only errors we've had pop up are:

786883 05/20 10:22    W    (policy name: failovertest target: localhost) SyncIQ encountered a filesystem error on source cluster. Error at source cluster on node [Isilon02-6]:  Operation failed while constructing list of lins changed between snapshots 36902 and 36288, Local error : No snapshots found between 36902 and 36288: No such file or directory
    from snapset_min_max (/b/mnt/src/isilon/lib/isi_migrate/config/../migr/summ_stf.c:391)
    from migr_start_changeset (/b/mnt/src/isilon/bin/isi_migrate/pworker/stf_based.c:1073)

786554 05/19 15:30    W    (policy name: failovertest target: localhost) SyncIQ failed to take a snapshot on source cluster. Failed to open and lock new snapid 36288: lock failed: marked for delete

Now, I appreciate this is probably something for the support team (and a case has been raised) however I'm posting this in the hopes anyone has seen similar, and can offer me pointers of where to look to troubleshoot. (I know my way around storage generally, but Isilons are something I've not really looked into before).

I've got an nginx config that does ssl-pki authentication - I'm happily able to authenticate my users.

However, I want to take it a bit further - I want to allow/deny access to resources based on the ssl variables - specifically $ssl_client_s_dn (and verify that $ssl_client_verify is 'SUCCESS')

Based on this I want to selectively allow/deny groups of URLs based on a user list. In this scenario, Elasticsearch. I have:

• Some common APIs I want to access (server status, cluster health, 'ping').
• Some indicies I want one group of users to be able to 'see'
• Some indices I want another group of users to 'see'.

Now it looks like I can probably do this with http_auth_request - redirect to an 'auth' url - a bit like outlined in this blog

But this seems a bit overkill to me, as to do that I'll have to write my own 'engine' to accept/read http, validate it against what amounts to a regex, and spit out a response code.

What I'm trying to do 'whitelist' e.g. for me:

allow if /(logstash|filebeat|topbeat)-mygroup-.*/;

(where 'mygroup' is read from a file somewhere, and ideally allows to have multiple-group membership).

Now, is this best done with http_auth_request, and feeding it to a really cut down validator web app (either script standalone, or a cgi type script inside NGINX) or is there a better way?

This module is an extra for nginx, which means a recompile. That's not a disaster, but if it's not necessary (and I can continue using 'stock' nginx) that would be preferable.

I've got an elasticsearch instance, with an Nginx reverse proxy sat in front of it, implementing a URL level access control mechanism - approximately as described in this article

This is giving me per-index granularity, which is handy.

What I'm trying to figure out next though is - can I enforce a blanket filter criteria that excludes results from searches and direct document retrievals?

I'm thinking a 'group' model, where docs in my indices are tagged by group, and users can only see results and docs for their group.

Now, I know I could do a per-group index, and apply URL level controls. That may be my workaround if I can't do this.

I have had a look at aliasing - this seems to do 90% of what I want, in that I can restrict a search to an alias. But what I can't then do is inhibit a direct 'GET' request with an (unauthorised) document ID.

Is there a way of doing this, or am I just on a road to nowhere?

Note - part of my reasons for this, is I'm trying to use a fairly standard kibana setup, and I have overlapping groups of users.

I have a two server failover cluster, and for various reasons - they're on mismatched hardware.

One has Emulex HBAs installed, the other Qlogic.

I'd be twitchy about using two different makes/models of card for multipathing within a box. I'm therefore a bit uncomfortable doing it in a shared storage failover cluster.

Is there a particular reason why this might be a bad (or good?) idea, or am I just being unduly concerned?

I've a recently upgraded 3PAR array ( 3.2.1 MU1) , which overnight is getting some high iowaits - to the point where some paths to drop and servers to crash (particularly noticeable with e.g. Oracle). I think since the upgrade, but I'm not 100% sure.

When we look at 'total disk' chart on the 3PAR management tool (SSMC) we see that at 0100 BST service time shoots up (6k ms) and indicates that there are 8 near line disks that are on the same pair of nodes, all showing high service time.

This has happened several nights now - and it's different nodes and sets of disks each time - but not quite every night. It says in this state for a couple of hours.

Now this sounds like like some sort of scheduled task (media scrub maybe?) 'going off', but I'm at something of a loss to figure out what it might be. Is anyone able to point me in the right direction for how to figure out what's going on here?

I'm currently fiddling with a project using the ONTAPI (perf stats monitoring sort of thing).

What I'm wanting to do is reproduce the information from 'df' and 'df -s'.

Doing well so far, because the api 'volume-list-info' seems to have most of the information I want. There's just one thing missing - how much of the 'snap reserve' I'm actually using.

I can't seem to find it in either that, or the 'snapshot' counters. I would ideally be able to do this without having to do a per-volume calculation, because one of the things I'm hoping to do is support a lightweight proxy client that 'just' fetches source XML for processing on a server.

Or is there a way I can calculate this from the size/used/available in the volume-list-info?

I have a large multi user system, NFS mounting a large piece of shared storage. I can identify the source of all my network traffic - narrowing it down to a particular host - thanks to nfsstat.

However I'm still having a little difficulty tracing which of the users is doing this - there's a few hundred, and there aren't any obvious culprits in the process list. (I usually start with looking for instances of find)

But we do definitely have 2.5k IOPS coming from it, and it's causing resource issues on the host.

Is anyone able to offer me suggestions to figure out culprit processes/users?

Box is a RedHat Linux, talking to a NetApp filer.

I've been trying to figure out something.

I have a stack of SAS shelves (attached to a NetApp if that's relevant). Each devices within the shelf is 6G SAS. According to the vendor, the sustained transfer is somewhere around 200-250MB/sec.

So - with 10 shelves of 24 drives - attached 'top and bottom' to separate controllers on my filer head.

What's the fastest rate I could transfer data from my drives?

6G SAS implies 600MB/sec. 2 controllers therefore, 1200MB (in optimal circumstances). Or - about 6 drives worth, out of my 240 spindles. This seems oddly low - am I missing something? Do the SAS controllers have some sort of mulitplicative factor?

Or am I really honestly in a position where I'll never get anywhere near 'max throughput' of the drives in this stack? Certainly it looks like my historic peak has been around 2000MB/sec on 3 controllers. (So ~650MB/sec each).

Still, I suppose it makes the drive utilisation look low....

I have a bunch of NetApp filers, which I've been accessing via ssh with public-private key pairs.

I would like to extend what I'm doing - we're repeatedly hitting limitations where e.g. ssh is failing to complete commands (disconnnect before execute) and formatting of output from the commands we're running is 'human readable' which means it's annoying to machine parse.

So I'd like to query the API and for various reasons I'd like to do so using 'off the shelf' modules from CPAN, rather than the NetApp SDK.

Can this be done, and if so how?

(I had pondered whether this was a better fit for Stack Overflow, but concluded that it was probably more relevant to sysadmin than perl scripting).

I'm rearranging some NFS mounts - tidying up and decommissioning them.

As part of this, my 'management' server has gained a huge load average, because of nagios kicking off 'stat' on the NFS mounts.

All the processes have gone into state D - uninterruptible sleep.

Is there a way - short of rebooting the box - that I can convince these processes that the mount they're accessing is dead and gone? I know they're irrelevant, but the kernel masks interrupts to these processes.

I've started looking at switching storage charging models. Historically - it's been purely per-gig, in a big shared pool.

But I'm looking at extending the model, and looking at tiering storage offerings - mostly because we're looking at an investment in SSD - but under our previous charging model, we've not really been able to justify it. (We have got some as a 'controller cache upgrade').

My initial start point is to:

• Take a 'storage chunk'. Probably a RAID group, but maybe 'a shelf' or 'a controller'.
• Sum up the usable gigs.
• Sum up the theoretical IOPs of the spindles (random read and write penalty writes)
• Sum up theoretical MB/sec throughput of the spindles.

Divide IOPs and throughput by available TB, to set a 'performance allocation' based on each TB 'bought'. And look at a sample SATA configuration, sample FC/SAS configuration and SSD.

Now, this is rather a simplification I know - and is rather a worst case scenario, in a world of big fat caches and assorted other bottlenecks.

I've got a big pile of 'average usage' performance stats, and know what my real world cache hit ratios look like - but am now a bit stuck as to how to 'scale' this. As an example - my NetApp Filers are giving me 20-25% read cache miss rates, and does something entirely different with write cache and WAFL that makes that element hard to compare. (But I think assuming a high write cache-hit is not unreasonable, and thus allowing me to disregard write penalty and burst write latency).

So this is my question - what approach would you suggest for putting together say, 3 tiers of 'storage offering' (Archive, 'standard' and high performance)? How would you factor in the expected returns of caching and consolidation benefits?

Regarding storage tiering - that's an option, because it's effectively what we're doing already by big controller caches. But there's still a need for differentiating 'cheap' and 'fast' storage, and I'm needing and appraoch for that too.

I've a NetApp filer that's running exclusively CIFS. (/vol/vol0 is CIFS exported) Account logins via ssh are possible as DOMAIN\username, but we're having difficulty with configuring ssh public-private key pairs.

Normally we'd create an authorized_keys file in order to allow access, but can't figure out the account mapping. (Filer login as DOMAIN\username works, but not as just username)

How do we go about setting this up?

I've 4 NetApp 2240-4 filer heads. They're single chassis 'cluster in a box' so two separate units.

Over the last few days, at about the same time - all of them started logging a LOT of Low water mark consistency points.

Running wafl_susp -w gives me cp_from_low_water clocking up at a rate of 10/sec or more. Before this started, they were almost entirely cp_from_timer at a rate of 1 every 10s or so.

Two of my boxes have become unresponsive and been rebooted, and the problem has now gone again. I'm not 100% sure that's connected, but it seems a reasonable bet as to a culprit.

The other two - are completely idle, as in they have a base OS, and a couple of vfilers - and nothing else. But yet - Low watermark, suggests they're running out of memory, for some reason. I can only assume some sort of denial of service condition is occurring (perhaps 'failed SSH logins'?).

Can anyone offer an insight on how to troubleshoot this? Specifically from a NetApp perspective, I'm looking for some hints as to how to extract what's hogging my memory.

I've been reading: http://community.netapp.com/t5/Tech-OnTap-Articles/Back-to-Basics-RAID-DP/ta-p/86123

That basically describes RAID-DP as diagonal parity raid. But there's one thing that's bothering me a bit - this example must be simplified, because it describes taking a sum. That's fair enough - for 'normal' parity in RAID4 and RAID5 - you do XORs rather than straight additive sums and this works because XORs are quite efficient and you can reconstruct.

If your parity is:

A xor B xor C = P

Then

P xor B xor C = A

There's just one thing that's nagging at me a bit though - The DP sum can't work that way, because if you do an 'XOR style' dual parity... you end up with an ambiguous result. You create a bunch of simultaneous equations, that allow you to figure out the relationships between bits - e.g. you know which bits are equal and not equal to each other. However, you end up with two valid solutions - one the 'inverse' of the other.

I assume that's why the worked example uses additive sums... but the thing that bothers me about that is: Your additive sum will often be a bigger number of bytes than the source. If you 'sum' 10 values a byte long, and your sum might be bigger than a byte. You could 'wrap' your sum I guess, but you still risk getting an ambiguous result.

What your parity effectively tells you is which bits are equal to each other and which aren't. But the net result is ambiguous - you can either have the 'right' answer, or an inversion of the 'right' answer.

What am I missing?

(I suspect the answer might be similar to how RAID-6 does it).

I have an isolated network, into which I've built a vfiler. The point of this network is that it's a non routed 'test' network.

However, there's a need for LDAP/Kerberos and CIFS access to the filer, via domain level accounts.

So we have Read Only domain controllers deployed.

To join a Windows box to the RODC, we would:

• create a machine account by hand.
• join the domain, and specify the machine account password on the client.

A spot of googling finds me: https://kb.netapp.com/support/index?page=content&id=1012918

Where the advice is: Point the filer at a writable DC manually first.

I'd rather not do that if I can avoid it - I don't have writable DCs on this piece of the network deliberately. More importantly - my vfilers are on an ipspace, so I can't even temporarily 'jump over' to a network with the right access. (Which is sort of the point I guess, but even so...)

Does anyone have a suggestion for how I can accomplish this - I'm assuming I may need to extract some information from my DC and transfer it over, such as a servicePrincipal. Or perhaps just 'set' my CIFS password manually somewhere.

Since a sitewide upgrade to Windows 7 on desktop, I've started having a problem with virus checking. Specifically - when doing a rename operation on a (filer hosted) CIFS share. The virus checker seems to be triggering a set of messages on the filer:

[filerB: auth.trace.authenticateUser.loginTraceIP:info]: AUTH: Login attempt by user server-wk8-r2$ of domain MYDOMAIN from client machine 10.1.1.20 (server-wk8-r2).
[filerB: auth.dc.trace.DCConnection.statusMsg:info]: AUTH: TraceDC- attempting authentication with domain controller \\MYDC.
[filerB: auth.trace.authenticateUser.loginRejected:info]: AUTH: Login attempt by user rejected by the domain controller with error 0xc0000199: STATUS_NOLOGON_WORKSTATION_TRUST_ACCOUNT.
[filerB: auth.trace.authenticateUser.loginTraceMsg:info]: AUTH: Delaying the response by 5 seconds due to continuous failed login attempts by user server-wk8-r2$ of domain MYDOMAIN from client machine 10.1.1.20.

This seems to specifically trigger on a rename so what we think is going on is the virus checker is seeing a 'new' file, and trying to do an on-access scan. The virus checker - previously running as LocalSystem and thus sending null as it's authentication request is now looking rather like a DOS attack, and causing the filer to temporarily black list. This 5s lock out each 'access attempt' is a minor nuisance most of the time, and really quite significant for some operations - e.g. large file transfers, where every file takes 5s

Having done some digging, this seems to be related to NLTM authentication:

Symptoms

Error message:

System error 1808 has occurred.
The account used is a computer account. Use your global user account or local user     account to access this server.

A packet trace of the failure will show the error as:

STATUS_NOLOGON_WORKSTATION_TRUST_ACCOUNT (0xC0000199)

Cause

Microsoft has changed the functionality of how a Local System account identifies itself
during NTLM authentication.  This only impacts NTLM authentication.  It does not impact
Kerberos Authentication.

Solution

On the host, please set the following group policy entry and reboot the host.
Network Security: Allow Local System to use computer identity for NTLM: Disabled
Defining this group policy makes Windows Server 2008 R2 and Windows 7 function like Windows Server 2008 SP1.

So we've now got a couple of workaround which aren't particularly nice - one is to change this security option. One is to disable virus checking, or otherwise exempt part of the infrastructure.

And here's where I come to my request for assistance from ServerFault - what is the best way forwards? I lack Windows experience to be sure of what I'm seeing.

I'm not entirely sure why NTLM is part of this picture in the first place - I thought we were using Kerberos authentication. I'm not sure how to start diagnosing or troubleshooting this. (We are going cross domain - workstation machine accounts are in a separate AD and DNS domain to my filer. Normal user authentication works fine however.)

And failing that, can anyone suggest other lines of enquiry? I'd like to avoid a site wide security option change, or if I do go that way I'll need to be able to supply detailed reasoning. Likewise - disabling virus checking works as a short term workaround, and applying exclusions may help... but I'd rather not, and don't think that solves the underlying problem.

EDIT: Filers in AD ldap have SPNs for:

nfs/host.fully.qualified.domain
nfs/host
HOST/host.fully.qualified.domain
HOST/host

(Sorry, have to obfuscate those).

Could it be that without a 'cifs/host.fully.qualified.domain' it's not going to work? (or some other SPN? )

Edit: As part of the searching I've been doing I've found: http://itwanderer.wordpress.com/2011/04/14/tread-lightly-kerberos-encryption-types/

Which suggests that several encryption types were disabled by default in Win7/2008R2. This might be pertinent, as we've definitely had a similar problem with Keberized NFSv4. There is a hidden option which may help some future Keberos users: options nfs.rpcsec.trace on (This hasn't given me anything yet though, so may just be NFS specific).

Edit: Further digging has me tracking it back to cross domain authentication. It looks like my Windows 7 workstation (in one domain) is not getting Kerberos tickets for the other domain, in which my NetApp filer is CIFS joined. I've done this separately against a standalone server (Win2003 and Win2008) and didn't get Kerberos tickets for those either.

Which means I think Kerberos might be broken, but I've no idea how to troubleshoot further.

Edit: A further update: It looks like this may be down Kerberos tickets not being issued cross domain. This then triggers NTLM fallback, which then runs into this problem (since Windows 7). First port of call will be to investigate the Kerberos side of things, but in neither case do we have anything pointing at the Filer being the root cause. As such - as the storage engineer - it's out of my hands.

However, if anyone can point me in the direction of troubleshooting Kerberos spanning two Windows AD domains (Kerberos Realms) then that would be appreciated.

Options we're going to be considering for resolution:

• Amend policy option on all workstations via GPO (as above).
• Talking to AV vendor about the rename triggering scanning.
• Talking to AV vendor regarding running AV as service account.
• investigating Kerberos authentication (why it's not working, whether it should be).

I have a NetApp filer, with a CIFS export. The permissions have been locked down on it, to a point where it's no longer accessible. I need to reset the permissions on this - I've figured out I can probably do this by changing the qtree to Unix security mode and back again (provided I'm prepared to unexport the share temporarily).

However, I think I should be able to use the fsecurity command to do this. There's just one problem - the manpage example refers to 'applying ACLs from a config file': https://library.netapp.com/ecmdocs/ECMP1196890/html/man1/na_fsecurity_apply.1.html

But what it doesn't do, is give me an example of what a 'security definition file' actually looks like.

Is anyone able to give me an example? Resetting a directory structure to Everyone/Full Control is sufficient for my needs, as re-applying permissions isn't a problem.