I have a process P that listens for incoming TCP connections on one port. I am observing that sometimes clients have to issue several SYN to get a SYN-ACK back and then the processing of their connection. I think the process P is not able to handle more than a dozen of incoming parallel connections per second.
Is this delay and retry coming from my process P that is not accepting new TCP connection fast enough ? How could I confirm this ?
On a NFS mount (standard options on RedHat 5.6 with ancient 2.6.18 kernel), it seems to me that big and multiple write operations delay smaller read operations. For example, doing a simple ls in a directory will takes seconds (or minutes) if there's a cp or dd running concurrently.
The issue is a bit mitigated because Linux caches metadata for a few seconds, but when there's a lot of data to write, the NFS mount becomes unusable.
At first I though that this was only a NFS server issue but running something like this:
for((i=0; i<60; i++)) do
strace -f -t -o strace.$i.log time stat /mnt/nfs/data > out.$i.log 2>&1
sleep 1
if ((i == 30)); then
dd if=/dev/zero of=/mnt/nfs/data bs=1M count=1000 &
fi
done
wait
and a tcpdump in parallel tells me the following:
1) whenever the dd starts, the next stat that does a cache miss takes 15s
23261 16:41:24 munmap(0x2ad024d0e000, 4096) = 0
23261 16:41:24 lstat("/mnt/fermat_emctest/data", {st_mode=S_IFREG|0600, st_size=1048576000, ...}) = 0
23261 16:41:40 open("/proc/filesystems", O_RDONLY) = 3
2) the tcpdump shows that while the dd is running and WRITE calls are issued, not a single GETATTR is sent.
Given that RPC is async, I would have expected to see GETATTR calls multiplexed with the WRITE, but that's not the case.
It's not the GETATTR that's slow (it takes a few us when submitted), it's the kernel that queues it after all the WRITEs.
That's why the stat takes ages, because it waits for the kernel to submit the GETATTR call.
Am I right ?
This looks like a bufferbloat issue, the kernel is starving my stat because the client-side operations queue for this mount (server ?) is full.
I think this is somehow related to my other question How to achieve multiple NFS/TCP connections to the same server? .
Is there a way to tune the kernel NFS ops queue ?
on Linux, using various benchmark tools (bonnie++, iozone, etc) on ext4 (but i guess it's the same on other FS), rewrite is always slower than write to new files.
For example, using bonnie++, with 24GB files (host with 12GB), block write is 167MB/s when rewrite is 64MB/s.
So it seems to me that rewrite is always slower than write. What could explain this ? Internal FS operations (free old blocks, allocate new ones ?) ? RAID design ? Thank you.
rsync detects files modification by comparing size and mtime. However, if for any reason, the mtime is unchanged, rsync won't detect the change, although it's possible to spot it by looking at the ctime.
Of course, I can tell rsync do compare the whole files' contents, but that's very very expensive.
Is there a way to make rsync smarter, for example by checking mtime+size are the same AND that ctime isn't newer than mtime (on both source and destination) ? Or should I open a feature request ?
Here's an example:
Create 2 files, same content and atime/mtime
benoit@debian:~$ mkdir d1 && cd d1
benoit@debian:~/d1$ echo Hello > a
benoit@debian:~/d1$ cp -a a b
Rsync them to another (non-exisiting) directory:
benoit@debian:~/d1$ cd ..
benoit@debian:~$ rsync -av d1/ d2
sending incremental file list
created directory d2
./
a
b
sent 164 bytes received 53 bytes 434.00 bytes/sec
total size is 12 speedup is 0.06
OK, everything is synced
benoit@debian:~$ grep . d*/*
d1/a:Hello
d1/b:Hello
d2/a:Hello
d2/b:Hello
Update file 'b', same size and then reset its atime/mtime
benoit@debian:~$ echo World > d1/b
benoit@debian:~$ touch -r d1/a d1/b
Attempt to rsync again:
benoit@debian:~$ rsync -av d1/ d2
sending incremental file list
sent 63 bytes received 12 bytes 150.00 bytes/sec
total size is 12 speedup is 0.16
Nope, rsync missed the change.
benoit@debian:~$ grep . d*/*
d1/a:Hello
d1/b:World
d2/a:Hello
d2/b:Hello
Tell rsync the compare the file content
benoit@debian:~$ rsync -acv d1/ d2
sending incremental file list
b
sent 144 bytes received 31 bytes 350.00 bytes/sec
total size is 12 speedup is 0.07
Gives the correct result:
benoit@debian:~$ grep . d*/*
d1/a:Hello
d1/b:World
d2/a:Hello
d2/b:World
I'm trying to maximize the NFS/TCP bandwidth between a Linux client and a single NFS server over a 10Gbps link. Although each NFS mount has its own device (major/minor), everything comes down to a single TCP connection and this is a bottleneck.
Using iperf and multiple TCP connections, I can reach ~8Gbps, but NFS is stuck on a single TCP stream and limited to ~2Gbps (no tcp tuning yet).
On Solaris, there's the system setting rpcmod:clnt_max_conns which allow to set how many concurrent TCP connection the system is able to use to a single NFS server.
Is there a Linux equivalent or a way to do so ?
As a workaround, I know I could set many IP on the NFS server, and then load-balance the mountpoints on them but that wouldn't be optimal.
I am trying to debug locks interactions between NFSv3 clients and server and i am unable to correctly interpret their exchanges (using wireshark).
rfc1813 NFS Version 3 Protocol Specification says nothing has changed:
6.2 NLM Procedures
The procedures in the NLM version 4 protocol are semantically
the same as those in the NLM version 3 protocol.
but the previous NFS: Network File System Protocol Specification doesn't say anything about NLM.
So far, I've ended up reading Linux kernel source code but I would like to find a documentation that explains how the protocol works, the list of procedures and their use, etc. I think I need the full specification but a statechart would also be nice?
Could someone point me to the correct RFC ?
I'm familiar with extending fabrics by adding more switches to it, but i don't get why sometimes it's better to do routing between fabrics (considering all the switches are from the same vendor, using exactly the same FC protocol, etc).
For example, i have a fabric with storage arrays from vendor A. The new storage from vendor B has been configured and instead of extending the current fabrics, it was chosen to route between the old and the new ones.
Is there a best practice about the maximum size of a fabric ? When is it best to use SAN routing instead of extending current fabrics ?
on a Linux or Solaris server, using automount/autofs, how do i create/populate a directory with automount entries without mounting them ?
For example, i have /data yp:auto.data and /data/ subdirectories are only created after the first access. This is not persistent across reboot on Linux.
Is there a way to populate my /data without mounting all the disks ? So that users/applications see /data/disk1..N even if they are not mounted.
when the Red Hat 5.x workstation of a user crashes and leaves its filesystems with errors, on the next boot, it ask for the root password to enter single mode and run fsck manually.
This is annoying because the user can't use his workstation and we need to send someone just to run fsck.
Users don't store any valuable data on their workstations, so is there a way to configure this fsck to automatically run on errors without any password ?
on my NFS mounts, there are thousands of .nfs* files. What are they ?
Are they the files left behind by stale NFS handle ?
Thanks
445074 20 -rw-rw-rw- 1 19654 5000 19864 Feb 10 11:16 ./.nfs000000000006ca9200000004
445075 4 -rw-rw-rw- 1 19654 5000 1253 Feb 8 12:38 ./.nfs000000000006ca9300000005
188056 4 -rw-rw-rw- 1 19654 5000 1254 Feb 14 13:44 ./.nfs000000000002de9800000007
645729 6556 -rw-rw-rw- 1 19654 5000 6693189 Feb 14 16:52 ./.nfs000000000009da6100000008
188058 4 -rw-rw-rw- 1 19654 5000 1254 Feb 14 13:45 ./.nfs000000000002de9a00000009
188055 68 -rw-rw-rw- 1 19654 5000 64114 Feb 14 16:42 ./.nfs000000000002de970000000a
645730 4 -rw-rw-rw- 1 19654 5000 2872 Feb 14 16:12 ./.nfs000000000009da620000000b
On my Windows Server 2003 server, all incoming connections are dropped. I can see them getting in using Wireshark, but even a single ping from another computer fails.
All locally initiated connection work fine (I'm asking from the server). This server is the DC/DHCP/DNS/File server, so computer clients are in the dark.
I've run varius antivirus and removal tools without any luck. The Windows Firewall is disabled.
I'm wild-guessing at some virus/worm.
How can i check why these incoming ICMP/TCP SYN/etc are dropped?
Anyone has any knowledge about such situations?
Thanks.
I'm trying to write a SMF manifest but i'm stuck because i can't find the complete documentation. Their DTD (/usr/share/lib/xml/dtd/service_bundle.dtd.1) is a joke, it's full of CDATA.
For instance, i'm looking for the complete specification for <service_fmri/>, which value attributes are valid, etc.
Where can i find the complete specification for writing Solaris SMF manifests ?
Thanks.
i have this auto.master:
/data yp:my_nis_map
on a particular linux workstation, i like to add a /data/special that is not present in my_nis_map.
/data yp:my_nis_map
/data/special filer:/path/to/special
but this fails ? How can i concatenate two entries ? I can't modify the NIS or the NFS exports.
Thansk.
I have a network of solaris/linux servers that have ntpd configured to use a single internal server of stratum 2 through a DNS alias/CNAME. This is server is down for some time and the client servers' clocks are out of sync.
Since we have another internal server of stratum 1 (PPS), the DNS CNAME has been modified to point to the new server (which is up).
But using ntpq -p i can see that the client servers are still pointing to the old server.
It looks like they are not resolving again the peer name, so they don't get the new server IP.
How do i smoothly update ntpd's peer list ?
If i restart (x)ntpd, it's going to create timejumps. I wish ntpd would have updated its peer list / configuration and smoothly synced with the new server.
on a solaris 8 server with a specific workload, how do i turn on memory overcommit (à la Linux) ?
# Memory size: 65536 Megabytes
sun[root] # vmstat 1
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr m0 m1 m1 m1 in sy cs us sy id
0 0 0 30679520 38600344 123 318 941 0 0 0 0 4 0 3 0 7640 11016 7196 5 2 93
sun[root] # swap -s
total: 18808072k bytes allocated + 48426552k reserved = 67234624k used, 972480k available
sun[root] # swap -l
swapfile dev swaplo blocks free
/dev/md/dsk/d1 85,1 16 33560432 33556592
Thanks
[ID 879645 kern.notice] NOTICE: /: unexpected free inode 211471, run fsck(1M) -o f
How can i run fsck cleanly (/ unmounted) ? The only solutions i can see are: - detach the SDS mirror, fsck on the other metadevice, reboot on it, and sync back the mirror; - boot on a CDROM and run fsck (hope the CD will handle the RAID).
Is there a ways to force Solaris 10 to run fsck on next reboot ? How do you fsck your / on Solaris 10 ?
NetApp provides block-level deduplication (ASIS). Do you know any filesystem (even FUSE-based) on Linux (or OpenSolaris, *BSD) that provides the same functionnality ?
(I'm not interested in false deduplication like hardlinks).