Home / server / Questions / 79982
In Process
ibuys
Asked: 2009-10-31 11:08:02 +0800 CST

NFS Log Corruption

  • 772

We several WebSphere application servers logging to a central NFS mount. Both the app servers and the NFS server are running in VMWare, the app servers are RHEL 5, and the NFS server is Ubuntu 8.04. All servers are connected via gigabit ethernet, at a minimum. We are experiencing random corruption of the logs, where there will be large amounts of white space in them. We used to mount directly to a NetApp server, but migrated to this setup. The NetApp doesn't have this problem.

On the NFS server, I've set the following sysctl commands:

sysctl -w net.ipv4.tcp_wmem="4096 87380 8388608"
sysctl -w net.ipv4.tcp_rmem="4096 87380 8388608"
sysctl -w net.ipv4.tcp_sack=0
sysctl -w net.ipv4.tcp_dsack=0
sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sysctl -w net.ipv4.tcp_synack_retries=3
sysctl -w net.ipv4.tcp_retries2=5

The filesystem is exported with this command in /etc/exports:

/srv/export/logs        192.168.0.0/255.255.255.0(rw,no_root_squash,no_all_squash,sync,no_subtree_check)

And mounted with this command on the app servers:

mount -t nfs -o rsize=8192,wsize=8192 192.168.0.1:/srv/export/logs /webspherelogs/
linux

2 Answers

  1. I'm not sure there is enough information to answer this question. Have you tried writing large amounts of data from your various servers to the NFS mount, do those files exhibit corruption? If not this could be a websphere specific issue or an interaction between websphere and NFS. Maybe something in the way it writes data to log files....

    Either way I wouldn't recommend logging to an NFS mount if at all possible, you would be much better served with a centralized syslog-ng daemon with logging over tcp via the local syslog daemon. In general this would be a more flexible solution, would allow for greater redundancy and remove a lot of the complexity inherent in NFS.

    If this architectural change isn't possible then I would first rule out the NFS server itself, before you bring Websphere into the equation.

    • 1

  2. We had similar behavior with when two apps where logging to the same file. Especially if one application rolls the logfile while the other app tries to write to the logfile it might happen that they get the possition in the files mixed up.

    So make sure that only one process writes to each file. Also avoid that two different threads within one process write directly to the file. It only asks for trouble if two apps write to the same file.

    • 1