Marki's questions

I'm trying to obtain debug output of

1. what "find" does
2. compared to what happens on the network (tshark)

Therefore I want to run these commands in parallel and have output meticulously sorted by time.

I do this:

{
stdbuf -oL tshark -tad -l -n -s 9999 host x | ts '[TSRK %Y-%m-%d %H:%M:%.S]' &
sleep 5
stdbuf -oL find . -type d 2>&1 | ts '[FIND %Y-%m-%d %H:%M:%.S]' &
} >/root/out

Major problem: Tshark output is lagging

# stdbuf -oL tshark -tad -l -n | ts '[TSRK %Y-%m-%d %H:%M:%.S]'
[TSRK 2021-08-18 17:40:56.206744]     2 2021-08-18 17:40:55.477802058 00:04:96:xx:xx:xx → ff:ff:ff:ff:ff:ff ARP 64 Who has 172.31.x.x? Tell 172.31.x.x

Compare the timestamp of ts ([TSRK 2021-08-18 17:40:56.206744]) to the one of tshark (2021-08-18 17:40:55.477802058)

Output is lagging behind. The combined log outputs the result of find before showing the output of tshark. While - clearly - network activity must precede the result of find.

I've also tried parallel

parallel --line-buffer ::: \
'stdbuf -oL tshark -tad -l -n -s 9999 host x | ts "[TSRK %Y-%m-%d %H:%M:%.S]" ' \
'stdbuf -oL find . -type d | ts "[FIND %Y-%m-%d %H:%M:%.S]" ' \
>/root/outpar

but the outer tool doesn't seem to fix what's obviously wrong with the inner tool... The result is the same.

How do I do this correctly?

We use an explicit proxy to access the Internet.

This is what the HTTP header for "407 Proxy-Authentication required" looks like:

Proxy-Authenticate: BASIC realm="xxx"\r\n
Cache-Control: no-cache\r\n
Pragma: no-cache\r\n
Content-Type: text/html; charset=utf-8\r\n
Proxy-Connection: close\r\n
Connection: close\r\n
Content-Length: 813\r\n

Now when transmitting the Proxy-Authorize header (the following step when the browser transmits credentials), behavior seems to be different between browsers (here password=passwörd):

Using FF or IE (here IE):

Credentials: user:passw\366rd   <----------- ISO-8859
Accept: text/html, application/xhtml+xml, image/jxr, */*\r\n
Accept-Language: en-US,en;q=0.8,fr-LU;q=0.5,fr;q=0.3\r\n
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko\r\n
Accept-Encoding: gzip, deflate\r\n

Using Chrome

Credentials: user:passw\303\266rd   <----------- UTF-8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36\r\n
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3\r\n
Accept-Encoding: gzip, deflate\r\n
Accept-Language: en-US,en;q=0.9\r\n

Using FF or IE the proxy complains that the encoding is not what it expects. This means that it expects UTF-8 since auth using Chrome is working.

I found this online:

Previously, Firefox was using the ISO-8859-1 character encoding for the username and password in Basic HTTP authentication requests. Firefox 59 and later will use UTF-8 instead for the Authorization header to make sure non-ASCII characters like French accented letters are encoded properly.

However we are using FF 60.7.2ESR and it still seems to be sending ISO-8859 just like IE11 shown above.

There are several relevant RFCs: 7617, 7235, 7615.

https://www.rfc-editor.org/rfc/rfc7617#section-2 states that

For backwards compatibility reasons, this specification continues to leave the default encoding undefined, as long as it is compatible with US-ASCII (mapping any US-ASCII character to a single octet matching the US-ASCII character code).

Then, https://www.rfc-editor.org/rfc/rfc7617#section-2.1 states

In challenges, servers can use the 'charset' authentication parameter to indicate the character encoding scheme they expect the user agent to use when generating "user-pass" (a sequence of octets). This information is purely advisory. The only allowed value is "UTF-8" ...

It gives an example: WWW-Authenticate: Basic realm="foo", charset="UTF-8"

And suggests the same applies to "Proxy-Authenticate: ".

What does that mean for challenges when the password is not US-ASCII and no charset authentication parameter is sent? It must not be undefined but it doesn't state what it should be then.

We could force the proxy to interpret everything as ISO-8859 but that would break everything where the client sends UTF-8.

Usually when there is a problem, the filesystem mounts itself read-only:

[449551.919049] EXT3-fs (dm-2): error: ext3_journal_start_sb: Detected aborted journal
[449551.919056] EXT3-fs (dm-2): error: remounting filesystem read-only

Now I tried to catch that using monitoring via SNMP that we have in place anyway, but it didn't:

# snmpwalk -c public -v 2c localhost HOST-RESOURCES-MIB::hrFSTable
SNMP table: HOST-RESOURCES-MIB::hrFSTable

...
HOST-RESOURCES-MIB::hrFSAccess.11 = INTEGER: readWrite(1)
HOST-RESOURCES-MIB::hrFSAccess.12 = INTEGER: readWrite(1) ***

It seems to show the configured, not the current state.

Is there any other way to catch this, preferably using SNMP, without resorting to actually having to write a file and check the outcome?

BTW

• OS: SLES11-SP3
• Kernel 3.0.101
• net-snmp-5.4.2.1

How do I svMotion a single disk in PowerCLI 5.5?

Trying Set-HardDisk gives the following warning:

WARNING: Parameter 'Datastore' is obsolete. 'Datastore' parameter is obsolete. To move hard disk to another datastore
and change it's storage format, use 'Move-HardDisk' cmdlet instead.

Set-HardDisk fails anyway with a message that the disk is locked (hence I guess you can't use it for an online migration).

Move-HardDisk only seems to be able to move disks in offline mode too (cold migration), however I'd like to perform a Storage vMotion.

Move-VM can only move all disks of a VM, which is not what I want. (Too many and too big disks on a VM, will never fit on one destination LUN)

Let's say I have:

• public IP address A: sending host
• public IP address B: mail relay for A
• public IP address C: receiver MX

What IP does C look at?

Does he try to find the IP of A in the "Received" headers (e.g. by trying to find the first non-private IP)? Or does he use the source IP address of the connection established in this case from B?

I see that for example googlemail seems to do the latter. While it looks like one of our local providers seems to do the former.

https://www.rfc-editor.org/rfc/rfc7208 does not really enlighten me concerning this basic point. I did find the circumstantial phrase

Because SPF evaluation is based on the IP address of the "last" sending SMTP server

in section 10.3. but I guess there needs to be a clearer definition somewhere.

We have some ProxySGs. The version is SGOS 6.4. They are configured in explicit mode. Meaning for https requests: they are tunnelled through port 80 of the proxy's IP address.

Anyone using a similar setup please try the following CSS stylesheet address of a french webmail provider and explain to me why that takes 30 seconds or more to load across the proxy (if that is the case at your site):

https://webmail.laposte.net/css/images,common,dwt,msgview,login,zm,spellcheck,skin.css?v=131203105433&debug=&skin=laposteskin4&locale=fr

I have tried several ProxySGs behind different Internet connections. When I go direct, no problem. Doesn't matter what browser I use, tried IE in Windows, FF in Windows, FF in Linux. Via the proxy there are terrible timeouts to that site, making it impossible to use since they updated their site half a year or so ago.

I have also tried going through an explicit Squid proxy: no problem!

I am at a loss here.

• Are they using something inside HTTP that the ProxySG is choking on?
• Is there something wrong with SSL? I see their certificate does not always seem to match their actual hostname (laposte.net vs. laposte.fr)...
• I had complete policy tracing of the SG turned on yet saw no apparent errors, there was no blacklist match or the like (which wouldn't consistently take 30+ seconds to evaluate anyhow)

Funny thing: Using wget on the same Linux box with FF installed, both across the proxy, wget downloads the file without any delay.

Please advise :)

See packet capture below (left=PC to proxy, right=proxy to internet)

The NFS server is a SLES11SP3, here is the exports statement:

/media/data *domain.com(rw,sync,no_root_squash,fsid=1)

I suspect this is a caching issue on the client (RHEL6), however I believe I turned all caching off, still the problem remains.

How the problems shows:

On the server I do a touch /media/data/test. The file does not show on the client, even if I wait several minutes.

If I touch /media/data (the directory) on the server, then it shows. It also appears when I touch /data/test2 from the client.

What I tried on the client:

# original
server:/media/data   /data  nfs   rw,soft,vers=3,rsize=1048576,wsize=1048576,mountvers=3,sync,lookupcache=none
# mod 1 (remove parameters which have defaults: size, mountvers; add noac)
server:/media/data   /data  nfs   rw,soft,vers=3,sync,lookupcache=none,noac 0 0

I also tried mounting using hard and specify actimeo=1, no effect.

I have seen this and this. No help either.

Update #1

It seems that the underlying filesystem is special.

The underlying filesystem on the server is a Novell NSS volume.

NSS has certain unique properties like mapping the creation time of the file to the ctime in Linux (which natively is the change time there).

Meaning: Upon modifying the directory contents, only mtime gets updated and not ctime. Apparently the RHEL NFS client does not consider this a sufficient reason to update its cache while the SuSE NFS clients seem to be patched for this.

What are the options?

• Change the properties on the NSS volume (which can optionally map ctime to Change Time and not Creation Time, although one would have to investigate the repercussions of this filesystem-wide setting)
• Get Redhat to only consider mtime and not ctime (why are they anyway?)
• Workaround: touch the directories before listing them.......

Can you help me diagnose this OOM?

My remarks are in-between.

<4>[598175.284914] cifsd invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0

Whut? GFP mask lower byte 0xa indicates a request for a free page in highmem. This is a 64-bit system and therefore has no highmem zone.

<6>[598175.284919] cifsd cpuset=/ mems_allowed=0
<4>[598175.284921] Pid: 5529, comm: cifsd Tainted: G           E X 3.0.101-0.35-default #1
<4>[598175.284923] Call Trace:
<4>[598175.284934]  [<ffffffff81004935>] dump_trace+0x75/0x310
<4>[598175.284941]  [<ffffffff8145f2f3>] dump_stack+0x69/0x6f
<4>[598175.284947]  [<ffffffff810fc53e>] dump_header+0x8e/0x110
<4>[598175.284950]  [<ffffffff810fc8e6>] oom_kill_process+0xa6/0x350
<4>[598175.284954]  [<ffffffff810fce25>] out_of_memory+0x295/0x2f0
<4>[598175.284957]  [<ffffffff8110287e>] __alloc_pages_slowpath+0x78e/0x7d0
<4>[598175.284960]  [<ffffffff81102aa9>] __alloc_pages_nodemask+0x1e9/0x200
<4>[598175.284965]  [<ffffffff8113de60>] alloc_pages_vma+0xd0/0x1c0
<4>[598175.284969]  [<ffffffff81130bcd>] read_swap_cache_async+0x10d/0x160
<4>[598175.284972]  [<ffffffff81130c94>] swapin_readahead+0x74/0xd0
<4>[598175.284975]  [<ffffffff81120bfa>] do_swap_page+0xea/0x5f0
<4>[598175.284978]  [<ffffffff81121c21>] handle_pte_fault+0x1e1/0x230
<4>[598175.284982]  [<ffffffff81465bcd>] do_page_fault+0x1fd/0x4c0
<4>[598175.284985]  [<ffffffff814627e5>] page_fault+0x25/0x30
<4>[598175.285002]  [<00007f65a0891078>] 0x7f65a0891077

Ok, it wants to get sth from swap but fails because apparently there is no more physical memory. Let's proceed...

<4>[598175.285003] Mem-Info:
<4>[598175.285004] Node 0 DMA per-cpu:
<4>[598175.285006] CPU    0: hi:    0, btch:   1 usd:   0
<4>[598175.285007] CPU    1: hi:    0, btch:   1 usd:   0
<4>[598175.285008] Node 0 DMA32 per-cpu:
<4>[598175.285010] CPU    0: hi:  186, btch:  31 usd:   9
<4>[598175.285011] CPU    1: hi:  186, btch:  31 usd:   7
<4>[598175.285012] Node 0 Normal per-cpu:
<4>[598175.285013] CPU    0: hi:  186, btch:  31 usd:  35
<4>[598175.285014] CPU    1: hi:  186, btch:  31 usd:  31
<4>[598175.285017] active_anon:218 inactive_anon:91 isolated_anon:0
<4>[598175.285018]  active_file:187788 inactive_file:451982 isolated_file:896
<4>[598175.285018]  unevictable:0 dirty:0 writeback:69 unstable:0
<4>[598175.285019]  free:21841 slab_reclaimable:8417 slab_unreclaimable:132175
<4>[598175.285020]  mapped:8168 shmem:4 pagetables:2639 bounce:0

Here we see a little over 3G used although I wouldn't be able to say what the different entries are exactly.

<4>[598175.285021] Node 0 DMA free:15880kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolat
ed(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0k
B pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
<4>[598175.285027] lowmem_reserve[]: 0 3000 4010 4010
<4>[598175.285029] Node 0 DMA32 free:54600kB min:50368kB low:62960kB high:75552kB active_anon:860kB inactive_anon:308kB active_file:600716kB inactive_file:1576184kB
 unevictable:0kB isolated(anon):0kB isolated(file):3328kB present:3072160kB mlocked:0kB dirty:0kB writeback:248kB mapped:26800kB shmem:16kB slab_reclaimable:23552kB
 slab_unreclaimable:412540kB kernel_stack:752kB pagetables:2412kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:4169324 all_unreclaimable? yes
<4>[598175.285036] lowmem_reserve[]: 0 0 1010 1010
<4>[598175.285038] Node 0 Normal free:16884kB min:16956kB low:21192kB high:25432kB active_anon:12kB inactive_anon:56kB active_file:150436kB inactive_file:231744kB u
nevictable:0kB isolated(anon):0kB isolated(file):384kB present:1034240kB mlocked:0kB dirty:0kB writeback:28kB mapped:5872kB shmem:0kB slab_reclaimable:10116kB slab_
unreclaimable:116160kB kernel_stack:2848kB pagetables:8144kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:688103 all_unreclaimable? yes
<4>[598175.285044] lowmem_reserve[]: 0 0 0 0
<4>[598175.285046] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB
<4>[598175.285051] Node 0 DMA32: 12620*4kB 3*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 54600kB
<4>[598175.285056] Node 0 Normal: 3195*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 16884kB

There seems to be a lot of fragmentation. But since an order 0 page (4k) was requested (in highmem!?), and tons of those are available, that wouldn't matter, would it?

<4>[598175.285061] 375504 total pagecache pages

That's more than 1G of pagecache. Shouldn't it first lower that cache before throwing OOM?

<4>[598175.285062] 268 pages in swap cache
<4>[598175.285064] Swap cache stats: add 1266107, delete 1265839, find 3666696/3838636
<4>[598175.285065] Free swap  = 4641856kB
<4>[598175.285066] Total swap = 5244924kB

Almost no swap used. Shouldn't it swap before throwing OOM?

<4>[598175.285066] 1030522 pages RAM

It's a machine with 4G of RAM BTW.

Oh and FWIW here comes the process list

<6>[598175.285067] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
<6>[598175.285071] [  485]     0   485     4223       62   0     -17         -1000 udevd
<6>[598175.285073] [ 1434]     0  1434     1003       65   1       0             0 acpid
<6>[598175.285075] [ 1449]   100  1449     8585      112   0       0             0 dbus-daemon
<6>[598175.285077] [ 1475]     0  1475    36450      428   1       0             0 mono
<6>[598175.285079] [ 1772]     0  1772    21365      298   1       0             0 vmtoolsd
<6>[598175.285081] [ 1838]   101  1838    12322      180   0       0             0 hald
<6>[598175.285083] [ 1842]     0  1842    41067      187   1       0             0 console-kit-dae
<6>[598175.285085] [ 1843]     0  1843     4510       56   1       0             0 hald-runner
<6>[598175.285087] [ 1961]     0  1961     8691       17   0       0             0 hald-addon-inpu
<6>[598175.285107] [ 1984]     0  1984     8691       75   1       0             0 hald-addon-stor
<6>[598175.285109] [ 1992]   101  1992     9130        7   1       0             0 hald-addon-acpi
<6>[598175.285111] [ 1993]     0  1993     8691       77   0       0             0 hald-addon-stor
<6>[598175.285113] [ 2562]     0  2562    47184       78   1       0             0 httpstkd
<6>[598175.285115] [ 2581]     0  2581     5881      221   1       0             0 syslog-ng
<6>[598175.285117] [ 2584]     0  2584     1070       63   1       0             0 klogd
<6>[598175.285119] [ 2598]     0  2598    23796      104   1     -17         -1000 auditd
<6>[598175.285121] [ 2600]     0  2600    19995       87   1       0             0 audispd
<6>[598175.285123] [ 2621]     0  2621     2093       58   0       0             0 haveged
<6>[598175.285125] [ 2641]     0  2641     4728       81   1       0             0 rpcbind
<6>[598175.285127] [ 2680]     0  2680    77513      657   0       0             0 nsrexecd
<6>[598175.285129] [ 2753]     0  2753     4222       52   0     -17         -1000 udevd
<6>[598175.285131] [ 2832]     0  2832     2160       75   0       0             0 irqbalance
<6>[598175.285133] [ 2863]     0  2863     6778       53   1       0             0 mcelog
<6>[598175.285135] [ 3163]     0  3163    35027      170   1       0             0 gmond
<6>[598175.285137] [ 3177] 65534  3177    56670      185   0       0             0 gmetad
<6>[598175.285139] [ 3213]     0  3213    24991      107   1       0             0 sfcbd
<6>[598175.285141] [ 3214]     0  3214    16795        0   1       0             0 sfcbd
<6>[598175.285143] [ 3221]     0  3221    20445       78   1       0             0 sfcbd
<6>[598175.285145] [ 3222]     0  3222    41992      117   1       0             0 sfcbd
<6>[598175.285147] [ 3239]     0  3239    16092       58   1       0             0 pure-ftpd
<6>[598175.285149] [ 3240]     2  3240     6284       82   0       0             0 slpd
<6>[598175.285151] [ 3290]     0  3290    12855      120   0     -17         -1000 sshd
<6>[598175.285153] [ 3316]    74  3316     8070      152   0       0             0 ntpd
<6>[598175.285154] [ 3333]     0  3333    17945       90   1       0             0 cupsd
<6>[598175.285156] [ 3393]     0  3393    19365       31   1       0             0 sfcbd
<6>[598175.285158] [ 3395]     0  3395    21475      109   0       0             0 sfcbd
<6>[598175.285160] [ 3400]     0  3400    38331      129   1       0             0 sfcbd
<6>[598175.285162] [ 3479]     0  3479    38357      125   0       0             0 sfcbd
<6>[598175.285164] [ 3719]     0  3655   220311     2005   0       0             0 ndsd
<6>[598175.285166] [ 3893]    30  3893   177915      910   0       0             0 java
<6>[598175.285168] [ 3910]     0  3910    14968       97   1       0             0 nscd
<6>[598175.285170] [ 3961]     0  3961    47276      332   0       0             0 namcd
<6>[598175.285172] [ 4073]     0  4073    10998      104   0       0             0 master
<6>[598175.285174] [ 4099]    51  4099    14190      229   0       0             0 qmgr
<6>[598175.285176] [ 4135]     0  4135    33370       99   1       0             0 httpd2-prefork
<6>[598175.285178] [ 4136]    30  4136    35518       85   1       0             0 httpd2-prefork
<6>[598175.285180] [ 4137]    30  4137    35523      266   0       0             0 httpd2-prefork
<6>[598175.285182] [ 4138]    30  4138    35523      111   0       0             0 httpd2-prefork
<6>[598175.285184] [ 4139]    30  4139    35523      137   0       0             0 httpd2-prefork
<6>[598175.285186] [ 4140]    30  4140    35523      299   0       0             0 httpd2-prefork
<6>[598175.285188] [ 4168]     0  4168     5751       86   0       0             0 cron
<6>[598175.285190] [ 4349]     0  4349    43028      120   0       0             0 ndpapp
<6>[598175.285194] [ 4548]     0  4548    17722       33   0       0             0 adminusd
<6>[598175.285196] [ 4577]     0  4577    17136       26   1       0             0 jstcpd
<6>[598175.285198] [ 4580]     0  4580    12511        0   1       0             0 jstcpd
<6>[598175.285200] [ 4601]     0  4601    10976       42   1       0             0 vlrpc
<6>[598175.285202] [ 4621]     0  4621     4222       54   1     -17         -1000 udevd
<6>[598175.285204] [ 4672]     0  4672    21525       70   0       0             0 volmnd
<6>[598175.285206] [ 4693]     0  4693    48377      195   0       0             0 ncp2nss
<6>[598175.285208] [ 4942]    81  4942    40049       32   0       0             0 novell-xregd
<6>[598175.285210] [ 5195]     0  5195    90312      479   0       0             0 cifsd
<6>[598175.285212] [ 5240]     0  5240     9586        9   1       0             0 smdrd
<6>[598175.285214] [ 5279]     0  5279    55127      172   0       0             0 novfsd
<6>[598175.285216] [ 5327]   104  5327     9431       72   0       0             0 nrpe
<6>[598175.285218] [ 5337]     0  5337     3177       78   0       0             0 mingetty
<6>[598175.285219] [ 5338]     0  5338     3177       78   1       0             0 mingetty
<6>[598175.285221] [ 5339]     0  5339     3177       78   0       0             0 mingetty
<6>[598175.285223] [ 5340]     0  5340     3177       78   1       0             0 mingetty
<6>[598175.285225] [ 5341]     0  5341     3177       78   0       0             0 mingetty
<6>[598175.285227] [ 5342]     0  5342     3177       78   1       0             0 mingetty
<6>[598175.285229] [ 5520]     0  5520    67658       99   0       0             0 cifsd
<6>[598175.285231] [25139]     0 25139    17698      836   0       0             0 snmpd
<6>[598175.285233] [ 4842]    51  4842    14147      511   0       0             0 pickup
<6>[598175.285235] [ 7917]     0  7917    21027     2460   1       0             0 savepnpc
<3>[598175.285237] Out of memory: Kill process 3719 (ndsd) score 19 or sacrifice child
<3>[598175.285239] Killed process 3719 (ndsd) total-vm:881244kB, anon-rss:0kB, file-rss:8020kB

So it found almost 1G of memory to free but it goes on, many processes follow...

Something is not right here.

It's a kernel 3.0.x BTW

Compare this

# free -m
             total       used       free     shared    buffers     cached
Mem:         72363      68035       4328          0        522      66294
-/+ buffers/cache:       1218      71145
Swap:        12291          0      12291

and this:

# snmpwalk -c public -v 2c localhost .1.3.6.1.4.1.2021.4
UCD-SNMP-MIB::memIndex.0 = INTEGER: 0
UCD-SNMP-MIB::memErrorName.0 = STRING: swap
UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 12586888 kB
UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 12586784 kB
UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 74100516 kB
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 4429580 kB
UCD-SNMP-MIB::memTotalFree.0 = INTEGER: 17016364 kB
UCD-SNMP-MIB::memMinimumSwap.0 = INTEGER: 16000 kB
UCD-SNMP-MIB::memBuffer.0 = INTEGER: 534804 kB
UCD-SNMP-MIB::memCached.0 = INTEGER: 44238560 kB
UCD-SNMP-MIB::memSwapError.0 = INTEGER: noError(0)
UCD-SNMP-MIB::memSwapErrorMsg.0 = STRING:

Why does free show 66294MB for "cache" while snmp shows about 44238MB for "memCached"? Shouldn't that be the same?

Looking at the MIB I see that the "memCached" is "physical or virtual" memory used for caching. (Don't tell me it puts disk cache into swap) ^^

The goal is finding out the real free physical memory (i.e. here 71145 as shown by free) via snmp.

More info

# cat /proc/meminfo
MemTotal:     74100516 kB
MemFree:       4422092 kB
Buffers:        542168 kB
Cached:       44239460 kB
SwapCached:          4 kB
Active:       16455504 kB
Inactive:     28707308 kB
SwapTotal:    12586888 kB
SwapFree:     12586784 kB
Dirty:            2536 kB
Writeback:           0 kB
AnonPages:      381088 kB
Mapped:         252132 kB
Slab:         23961488 kB
SReclaimable: 23648768 kB
SUnreclaim:     312720 kB
PageTables:       7812 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:  49637144 kB
Committed_AS:        4 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    264124 kB
VmallocChunk: 34359474191 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     2048 kB
DirectMap4k:      7936 kB
DirectMap2M:  75481088 kB

Before we had the VCSA (Vcenter Server Appliance) we had a Windows Vcenter Server.

We had the PowerCLI installed locally.

There, we could say

Get-VM -Name * | Sort-Object | %{
....

to get a list of all VMs registered in Vcenter.

Now we have the Appliance.

I installed the vMA (management assistant) too.

I registered the vcenter appliance, I am successfully connected to it, but I am unable to simply retrieve a list of VMs like above.

It always wants me to connect to a single ESXi host to perform a listing.

Now to me that seems like a step backward.

How would you do this?

Would you use vMA at all? Maybe use vCLI? Or stick with PowerCLI, but then I would need again to install a Windows machine just to run my scripts.....

This is a FAS2240 8.2P3 7-Mode

Backup Software: EMC Networker 8.1

NDMP is configured and running inside a vFiler.

Let's start with the backup. The backup command in Networker is:

nsrndmp_save -T dump

You can also specify -M (DSA) but I think here it is implicit.

Application information is (just recreated the NDMP client in Networker using the New Client Wizard to be on the safe side):

USE_TBB_IF_AVAILABLE=Y
DIRECT=Y
BUTYPE=dump
HIST=Y
UPDATE=Y

Full backup runs at more or less wirespeed

> sysstat -x 5
CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
29%      0      0      0       1     485  94867   98060     11       0      0     0s    91%    0%  -    90%       1      0      0       0      0       0      0
31%      0      0      0       1     502  97711  101518    490       0      0     0s    89%   13%  T    90%       1      0      0       0      0       0      0
32%      0      0      0      57     530 103167  107344    251       0      0     0s    89%    5%  Zf   88%      57      0      0       0      0       0      0
30%      0      0      0      41     552 107049  110286    222       0      0     0s    89%    7%  :    87%      41      0      0       0      0       0      0

HOWEVER: We only get about 10MB/sec avg when restoring from tape. The backup to tape was made when nothing else was running, so the problem is not that data is interleaved on the tape.

The command and output from Networker is (restoring to an empty volume):

# nsrndmp_recover -S 1760972238 -m /vol/vfprod_xtest /vol/vfprod_x
42795:nsrndmp_recover: Performing recover from Non-NDMP type of device
85183:nsrndmp_recover: DSA is listening for an NDMP data connection on: 1.2.4.5, port = 8745
42690:nsrndmp_recover: Performing non-DAR Recovery..
86724:nsrdsa_recover: DSA listening at: host 'backupserv', IP address '1.2.4.5', port '8745'.
42937:nsrdsa_recover: Performing Immediate recover
42940:nsrdsa_recover: Reading Data...
42617:nsrndmp_recover: NDMP Service Log: RESTORE: Dump came from a SnapLock volume.
42617:nsrndmp_recover: NDMP Service Log: RESTORE: Incremental restores to SnapLock volumes are not supported.
All files will be correctly restored, but subsequent restores
of incremental backups will not recreate the file system as
it appeared during the final incremental backup.
42617:nsrndmp_recover: NDMP Service Log: RESTORE: ./.etc/gid_map: cannot create file: Read-only file system
42617:nsrndmp_recover: NDMP Service Log: RESTORE: Sat Feb  8 14:52:25 2014: Writing data to files.
42617:nsrndmp_recover: NDMP Service Log: RESTORE: Sat Feb  8 14:56:43 2014 : We have read 3361690 KB from the backup.
42617:nsrndmp_recover: NDMP Service Log: RESTORE: Sat Feb  8 15:01:43 2014 : We have read 7053433 KB from the backup.
42617:nsrndmp_recover: NDMP Service Log: RESTORE: Sat Feb  8 15:06:43 2014 : We have read 10908694 KB from the backup.
42617:nsrndmp_recover: NDMP Service Log: RESTORE: failed to find the file
42617:nsrndmp_recover: NDMP Service Log: RESTORE: Sat Feb  8 15:11:22 2014: Restoring NT ACLs.
42617:nsrndmp_recover: NDMP Service Log: RESTORE: RESTORE IS DONE
42942:nsrdsa_recover: Reading data...DONE.
42927:nsrndmp_recover: Successfully done

Here are stats on the filer during a restore

>sysstat -x 5
CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
91%      0      0      0      35   15364    143    5369  20946       0      0     0s    98%   56%  H    55%      35      0      0       0      0       0      0
91%      0      0      0      20   14668    126    5135  20617       0      0    59s    98%   56%  H    56%      20      0      0       0      0       0      0
91%      2      0      0       3   14407    119    5685  20954       0      0     1     98%   53%  H    52%       1      0      0       0      0       0      0
91%      0      0      0       1   15564    154    5454  20711       0      0     1     98%   56%  Hf   53%       1      0      0       0      0       0      0
91%      0      0      0       2   14447    121    6502  14358       0      0     1     98%   42%  Hf   56%       2      0      0       0      0       0      0
...
92%      6      0      0       6   19503    245    4696  15072       0      0     1     98%   46%  :    56%       0      0      0       0      0       0      0
93%      0      0      0       3   18902    253    7667  13669       0      0     0s    98%   22%  Hf   63%       3      0      0       0      0       0      0
91%      6      0      0       7   18099    216    1957  32432       0      0     0s    97%  100%  :f   45%       1      0      0       0      0       0      0
91%      6      0      0       6   16111    153    4427  23419       0      0     0s    98%   55%  :    56%       0      0      0       0      0       0      0
92%      7      0      0       7   15629    156    8155      0       0      0     1     98%    0%  -    68%       0      0      0       0      0       0      0
91%      0      0      0       3   14226    125    4427  32453       0      0     1     99%   79%  Hf   53%       3      0      0       0      0       0      0
94%      6      0      0       6   32264    594     744  38453       0      0     2     97%  100%  :f   45%       0      0      0       0      0       0      0
92%      6      0      0       6   14781    127    9846     59       0      0     2     98%    7%  Hn   61%       0      0      0       0      0       0      0
90%      6      0      0      63   11546     57    2073  42592       0      0     2     99%  100%  :f   50%      57      0      0       0      0       0      0

CPU usage seems high compared to backup, where disk util was high, as it should.

Doing the same thing between two filers gives about 40MB/sec avg.:

na1> ndmpcopy -sa root:xx -da root:xx /vol/vfprod_x/fs1 na2:/vol/test/xtest
Ndmpcopy: Starting copy [ 1 ] ...
Ndmpcopy: na1: Notify: Connection established
Ndmpcopy: na2: Notify: Connection established
Ndmpcopy: na1: Connect: Authentication successful
Ndmpcopy: na2: Connect: Authentication successful
Ndmpcopy: na1: Log: DUMP: creating "/vol/vfprod_x/../snapshot_for_backup.10825" snapshot.
Ndmpcopy: na1: Log: DUMP: Dumping from a SnapLock volume
Ndmpcopy: na1: Log: DUMP: Using subtree dump
Ndmpcopy: na1: Log: DUMP: Date of this level 0 dump: Sat Feb  8 15:23:04 2014.
Ndmpcopy: na1: Log: DUMP: Date of last level 0 dump: the epoch.
Ndmpcopy: na1: Log: DUMP: Dumping /vol/vfprod_x/fs1 to NDMP connection
Ndmpcopy: na1: Log: DUMP: mapping (Pass I)[regular files]
Ndmpcopy: na1: Log: DUMP: mapping (Pass II)[directories]
Ndmpcopy: na2: Log: RESTORE: Dump came from a SnapLock volume.
Ndmpcopy: na1: Log: DUMP: estimated 16178315 KB.
Ndmpcopy: na1: Log: DUMP: dumping (Pass III) [directories]
Ndmpcopy: na1: Log: DUMP: dumping (Pass IV) [regular files]
Ndmpcopy: na2: Log: RESTORE: Sat Feb  8 15:23:12 2014: Begin level 0 restore
Ndmpcopy: na2: Log: RESTORE: Sat Feb  8 15:23:12 2014: Reading directories from the backup
Ndmpcopy: na2: Log: RESTORE: Sat Feb  8 15:23:13 2014: Creating files and directories.
Ndmpcopy: na2: Log: RESTORE: Sat Feb  8 15:23:31 2014: Writing data to files.
Ndmpcopy: na1: Log: DUMP: Sat Feb  8 15:28:11 2014 : We have written 12577964 KB.
Ndmpcopy: na2: Log: RESTORE: Sat Feb  8 15:28:11 2014 : We have read 12575988 KB from the backup.
Ndmpcopy: na1: Log: ACL_START is '16565304320'
Ndmpcopy: na1: Log: DUMP: dumping (Pass V) [ACLs]
Ndmpcopy: na1: Log: DUMP: 16177066 KB
Ndmpcopy: na1: Log: DUMP: DUMP IS DONE
Ndmpcopy: na2: Log: RESTORE: Sat Feb  8 15:29:36 2014: Restoring NT ACLs.
Ndmpcopy: na1: Log: DUMP: Deleting "/vol/vfprod_x/../snapshot_for_backup.10825" snapshot.
Ndmpcopy: na1: Log: DUMP_DATE is '5686836680'
Ndmpcopy: na1: Notify: dump successful
Ndmpcopy: na2: Log: RESTORE: RESTORE IS DONE
Ndmpcopy: na2: Notify: restore successful
Ndmpcopy: Transfer successful [ 0 hours, 6 minutes, 41 seconds ]
Ndmpcopy: Done

Also tried options ndmpd.tcpnodelay.enable on, as suggested in https://communities.netapp.com/thread/15890, to no effect.

We even bought the filer with a 10GbE card so the filer at least has the potential to really deliver, but for now I'm not sure it does.

The volume I used for test is located on a snaplock aggreate with underlying 10x 7200rpm SATA disks (8 usable, double parity).

The day we would need to restore a larger amount of data would be hell in this scenario, because a day would not be enough time to restore a few TB...

Anyone have any useful ideas?

Thanks.

UPDATE #1

Ok unrelated to NDMP I've 10MB/s read almost all the time I can think of, so here are some performance stats which I got using this script

#!/bin/sh
#
if [ -z $1 ]; then
echo " "
echo "I need a filer target"
echo "An example syntax"
echo " $0 filer01.msg.dcn"
echo " "
exit 0
fi

SSH="ssh -i /root/.ssh/id_rsa-netapp"

FILER=$1
#
DATAFILE="${FILER}_$(date +%Y%m%d%H%M)"
echo "" >> $DATAFILE
date >> $DATAFILE
echo "------------------------------" >> $DATAFILE
$SSH $FILER 'priv set -q diag; statit -b' 2>/dev/null
echo "Starting statit sample" >> $DATAFILE
$SSH $FILER 'priv set -q diag; nfsstat -z' 2>/dev/null
echo "Zeroing nfsstat" >> $DATAFILE
$SSH $FILER 'priv set -q diag; nfs_hist -z' 2>/dev/null
echo "Zeroing nfs_hist" >> $DATAFILE
$SSH $FILER 'priv set -q diag; wafl_susp -z' 2>/dev/null
echo "Zeroing wafl_susp" >> $DATAFILE
$SSH $FILER 'sysstat -xs -c 30 1' >> $DATAFILE

# And we wait...

$SSH $FILER 'priv set -q diag; statit -en' >> $DATAFILE 2>/dev/null
$SSH $FILER 'priv set -q diag; nfsstat -d' >> $DATAFILE
$SSH $FILER 'priv set -q diag; nfs_hist' >> $DATAFILE
$SSH $FILER 'priv set -q diag; wafl_susp -w' >> $DATAFILE

echo " ** " >> $DATAFILE

and the output can be found here.

Note that this filer seems to have 768 MB of NVRAM and the aggregate we're working on has 10 7200rpm SATA disks in a RAID-5

UPDATE #2 Jun 10

I'm not sure why we were hitting that many high watermarks in the example earlier, but for now I have found out the following with the help of support:

• the 10MB/s reads seems to be normal as the filer is always scrubbing the disks
• disabling controller failover (cf disable) makes NDMP restores (thus writes) proceed at 5 or more times the speed (50-100MB/s) and this is expected

Still I'm not sure then why they sold us a 10G card for max. 100MB/s, but I'll keep at it. Especially, since as far as I understood it, WAFL uses all the disks all the time to spread writes across as many spindles as possible, and this filer that about 20 disks even though they're only "BSAS"

I am adding a tagged VLAN onto eth0:

#ip link add link eth0 name eth0.20 type vlan id 20

This results in:

#ip link
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
 link/ether 9c:c7:a6:95:65:1c brd ff:ff:ff:ff:ff:ff
....
12: eth0.20@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
 link/ether 9c:c7:a6:95:65:1c brd ff:ff:ff:ff:ff:ff

#ip -d link show eth0.20
70: eth0.20@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 9c:c7:a6:95:65:1c brd ff:ff:ff:ff:ff:ff
    vlan id 20 <REORDER_HDR>

#cat /proc/net/vlan/config
VLAN Dev name    | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
eth0.234       | 234  | eth0
eth0.20        | 20  | eth0

Now I start dhclient:

#dhclient -d -v -1 eth0.20

What I see in tcpdump is an untagged DHCP discovery frame:

#tcpdump -i eth0 -XX
0x0000:  ffff ffff ffff 9cc7 a695 651c 0800 4500 
                                       ^^^^

Why is it not tagged?

802.1q module seems to be used:

#lsmod | grep 8021q
8021q                  28324  0
garp                   14311  1 8021q

(OS: SLES11SP2 kernel 3.0.13-0.27-default)

BTW other traffic is not tagged either (at least tcpdump doesn't show it)...

 # tcpdump -Uw - | tcpdump -i eth0 -en -r - &
[1] 7310
 # tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

 # dhclient -d -v -1 eth0.20
Internet Systems Consortium DHCP Client 4.2.3-P2
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0.20/9c:c7:a6:95:65:1c
Sending on   LPF/eth0.20/9c:c7:a6:95:65:1c
Sending on   Socket/fallback
DHCPDISCOVER on eth0.20 to 255.255.255.255 port 67 interval 3
reading from file -, link-type EN10MB (Ethernet)
18:49:14.437882 9c:c7:a6:95:65:1c > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 347: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 9c:c7:a6:95:65:1c, length 305
                                                                       ^^^^^^

So, still no tag shown here.

But indeed the transmit counter for eth0.20 in /proc/net/dev does increment when running dhclient...

I need a fresh pair of eyes.

We're using a 15km fibre optic line across which fibrechannel and 10GbE is multiplexed (passive optical CWDM). For FC we have long distance lasers suitable up to 40km (Skylane SFCxx0404F0D). The multiplexer is limited by the SFPs which can do max. 4Gb fibrechannel. The FC switch is a Brocade 5000 series. The respective wavelengths are 1550,1570,1590 and 1610nm for FC and 1530nm for 10GbE.

The problem is the 4GbFC fabrics are almost never clean. Sometimes they are for a while even with a lot of traffic on them. Then they may suddenly start producing errors (RX CRC, RX encoding, RX disparity, ...) even with only marginal traffic on them. I am attaching some error and traffic graphs. Errors are currently in the order of 50-100 errors per 5 minutes when with 1Gb/s traffic.

Here is the power output of one port summarized (collected using sfpshow on different switches)

SITE-A      units=uW (microwatt)    SITE-B
**********************************************
FAB1
SW1   TX 1234.3       RX   49.1       SW3   1550nm (ko)
      RX   95.2       TX 1175.6
FAB2
SW2   TX 1422.0       RX  104.6       SW4   1610nm (ok)
      RX   54.3       TX 1468.4      

What I find curious at this point is the asymmetry in the power levels. While SW2 transmits with 1422uW which SW4 receives with 104uW, SW2 only receives the SW4 signal with similar original power only with 54uW.

Vice versa for SW1-3.

Anyway the SFPs have RX sensitivity down to -18dBm (ca. 20uW) so in any case it should be fine... But nothing is.

Some SFPs have been diagnosed as malfunctioning by the manufacturer (the 1550nm ones shown above with "ko"). The 1610nm ones apparently are ok, they have been tested using a traffic generator. The leased line has also been tested more than once. All is within tolerances. I'm awaiting the replacements but for some reason I don't believe it will make things better as the apparently good ones don't produce ZERO errors either.

Earlier there was active equipment involved (some kind of 4GFC retimer) before putting the signal on the line. No idea why. That equipment was eliminated because of the problems so we now only have:

• the long distance laser in the switch,
• (new) 10m LC-SC monomode cable to the mux (for each fabric),
• the leased line,
• the same thing but reversed on the other side of the link.

Here is a port config from the Brocade portcfgshow (it's like that on both sides, obviously)

Area Number:              0
Speed Level:              4G
Fill Word(On Active)      0(Idle-Idle)
Fill Word(Current)        0(Idle-Idle)
AL_PA Offset 13:          OFF
Trunk Port                ON
Long Distance             LS
VC Link Init              OFF
Desired Distance          32 Km
Reserved Buffers          70
Locked L_Port             OFF
Locked G_Port             OFF
Disabled E_Port           OFF
Locked E_Port             OFF
ISL R_RDY Mode            OFF
RSCN Suppressed           OFF
Persistent Disable        OFF
LOS TOV enable            OFF
NPIV capability           ON
QOS E_Port                OFF
Port Auto Disable:        OFF
Rate Limit                OFF
EX Port                   OFF
Mirror Port               OFF
Credit Recovery           ON
F_Port Buffers            OFF
Fault Delay:              0(R_A_TOV)
NPIV PP Limit:            126
CSCTL mode:               OFF

Forcing the links to 2GbFC produces no errors, but we bought 4GbFC and we want 4GbFC.

I don't know where to look anymore. Any ideas what to try next or how to proceed?

If we can't make 4GbFC work reliably I wonder what the people working with 8 or 16 do... I don't assume that "a few errors here and there" are acceptable.

Oh and BTW we are in contact with everyone of the manufacturers (FC switch, MUX, SFPs, ...) Except for the SFPs to be changed (some have been changed before) nobody has a clue. Brocade SAN Health says the fabric is ok. MUX, well, it's passive, it's only a prism, nature at it's best.

Any shots in the dark?

@Chopper3: This is the second generation of Brocades exhibiting the problem. Before we had 5000s, now we have 5100s. In the beginning when we still had the active MUX we rented a longdistance laser once to put it into the switch directly in order to make tests for a day, during that day of course it was clean. But as I said, sometimes it's clean just like that. And sometimes it's not. Alternative switches would mean to rebuild the entire SAN with those only to test. Alternative SFPs, well they're hard to come by just like that.

@longneck: The line is rented. It's a dark fibre (9um monomode) so there's noone else on it. Sure there are splices. I can't go and look but I have to trust they have been done correctly. As I said the line has been checked and rechecked (using an optical time-domain reflectometer). Obviously you don't have all this equipment yourself because it's way too expensive.

@mdpc: What would be the "wrong" type of cable according to you? Up to the switch everything is monomode, yes. The connectors are the correct ones too. Yeah I know there are the green ones where the fibre is cut off at a certain angle etc. But we have the correct ones for all that I know.

We have had two fabrics (=2x2 switches) with Brocade 5100s with FabricOS 6.4.1 and two fabrics (another 2x4 switches) on FabricOS 7.0.2.

On the longdistance ISLs (one in each fabric) it turned out that with FOS 6.4.1 setting it to long distance issues warnings about the VC Init setting and consequently the fill word. But those are only warnings. FOS 7.0.2 requires you to do modifications to VCI and the fillword for long distance links.

Setting FOS 6.4.1 to the LS (long-distance static distance) setting with wrong VCI and fillword setting made the whole fabric inoperational (stuck in an SCN loop, use fabriclog -s to see, you don't see it anywhere else, no port error counters or anything increasing).

Currently I'm giving the one fabric with the IMHO more correct settings a beating and it seems to do fine, whereas the other one without much traffic still has errors here and there.

In short:

• We have eliminated the active part of the MUX (the FC retimer).
• We are putting the long distance SFPs into the end equipment themselves.
• Just to be sure we bought new monomode cables to connect the end equipment to the remaining passive part of the MUX.
• We are now trying out several long distance configs.

It's almost black magic. Everything that happens is mostly empirical, noone seems to have a clue what are the exact reasons to do something. ("We have tried this, and it didn't work, then we tried that and it worked, so we stuck with that." But noone really seems to know why.)

I'll keep you updated.

We got the new lasers for one of the fabrics on warranty. It's ultra clean even on 4GbFC.

They're transmitting with roughly 2mW (3dBm) whereas the others are only at 1.5mW (1.5dBm) although that should really be enough.

The other fabric (where the lasers are apparently ok) still produces one or two CRCs infrequently.

Using sfpshow the SFP producing the actual RX errors shows

Status/Ctrl: 0x82
Alarm flags[0,1] = 0x5, 0x40
Warn Flags[0,1] = 0x5, 0x40

Now I'll have to find out what that means. Not sure if it was there before.

Well I'll first clear my head with a week of vacation. 8-)

We have an explicit (= not transparent) proxy setup using a BlueCoat ProxySG (software 6.4.3.1).

When I create Web Access Layer rules like:

• Protocol: HTTPS -> All HTTPS: Allow,
• Protocol: All TCP Tunneling: DENY, then

I still can do an SSH outside via the proxy.

Inversely when specifying something along the lines of

• Protocol: All Shell: DENY

(with no explicit statement of HTTPS) I can also do the SSH via the proxy.

The only thing that helps (with the SSH) is to specify All TCP Tunneling: DENY. However in that case HTTPS also no longer works.

HTTPS thus seems to be treated like TCP tunneling. That's also what the trace file shows on a HTTPS request:

 CONNECT tcp://www.xxx.com:443/

I know as long as someone is allowed to create outgoing encrypted connections they will probably be able to make almost anything work. But I don't want to make it too easy.

So how to recognize HTTPS while not allowing too much other stuff? (Probably impossible without setting up the MitM stuff etc.) And why is there an object called HTTPS and when I use it it doesn't change anything. I really don't get that part.

I read about cold and hot-cloning everywhere, but isn't the following possible:

• snapshot the machine while running including the memory
• restore the snapshot (including the memory) as a new VM (e.g. paused)
• be able to change the NIC to an isolated network etc. so the clone doesn't interfere with the original
• resume the clone/fork exactly at the snapshot state

This would not create any inconsistencies in the cloned VM. At least no filesystem-level inconsistencies...

Nobody talks about this, so probably it won't work. Does the snapshot that "Clone..." in Vcenter takes at least quiesce the disks then? There is no option in the "Clone" dialog to do this explicitely.

Thoughts on why this probably couldn't work: MAC address changes instantly => OS may not be happy.

What do you think?

After vMotion some guests went 2 hours ahead in time.

2 hours ahead of UTC is exactly our timezone.

Why would it do that? AFAICS both source and destination hosts have the same configuration.

But apparently not. What could be wrong?

What could be writing on the disk that iotop does not show?

# iotop -a
Total DISK READ: 8.19 M/s | Total ****DISK WRITE: 3.34 M/s****
  TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND
  428 be/4 root          0.00 B     84.00 K  0.00 %  0.02 % [kjournald]
 2600 be/3 root          0.00 B      8.00 K  0.00 %  0.01 % auditd -s disable
 2582 be/4 root          0.00 B      4.00 K  0.00 %  0.00 % syslog-ng

(iotop is NOT run using -o)

How do you interpret this?

# iostat
Linux 3.0.51-0.7.9-default (x)   03/10/13        _x86_64_

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       1.08    0.00    1.33    5.27    0.00   92.32

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               5.10       173.02        14.35  824994560   68417967
sdb              79.08      1028.23      1018.19 4902923384 4855045168
sr0               0.00         0.00         0.00        988          0
dm-0            175.34      1028.23      1018.19 4902916232 4855045168

Can someone explain to me the ds[fan_speed].value field in this RRD?

rrdtool info gives:

rrd_version = "0003"
step = 300
last_update = 1360341382
ds[fan_speed].type = "GAUGE"
ds[fan_speed].minimal_heartbeat = 600
ds[fan_speed].min = 0.0000000000e+00
ds[fan_speed].max = 1.0000000000e+02
ds[fan_speed].last_ds = "60"
ds[fan_speed].value = 4.9200000000e+03 ********* ???
ds[fan_speed].unknown_sec = 0
...
rra[2].cf = "AVERAGE"
rra[2].rows = 700
rra[2].cur_row = 295
rra[2].pdp_per_row = 6
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 6.0000000000e+01
rra[2].cdp_prep[0].unknown_datapoints = 0

(The same value is shown using 'rrdtool dump' in "PDP Status".)

What is it?

(Usually, the fans on this device are at 60(%), which is shown for example in rra[2].cdp_prep[0].value = 6.0000000000e+01)

Thanks.

The error is 2147746321.

I can see in the mailserver log that it tries, but the connection gets closed.

Wed 2012-10-10 15:55:25: Session 990590; child 1
Wed 2012-10-10 15:55:25: Accepting SMTP connection from [x:49161] to [y:25]
Wed 2012-10-10 15:55:25: --> 220 Mdaemon; Wed, 10 Oct 2012 15:55:25 +0200
Wed 2012-10-10 15:55:25: <-- EHLO x
Wed 2012-10-10 15:55:25: --> 250-Hello x, pleased to meet you
Wed 2012-10-10 15:55:25: --> 250-VRFY
Wed 2012-10-10 15:55:25: --> 250-EXPN
Wed 2012-10-10 15:55:25: --> 250-ETRN
Wed 2012-10-10 15:55:25: --> 250-AUTH LOGIN
Wed 2012-10-10 15:55:25: --> 250-8BITMIME
Wed 2012-10-10 15:55:25: --> 250 SIZE 20971000
Wed 2012-10-10 15:55:25: <-- AUTH LOGIN
Wed 2012-10-10 15:55:25: --> 334 VX......
Wed 2012-10-10 15:55:25: Connection closed
Wed 2012-10-10 15:55:25: SMTP session terminated (Bytes in/out: 26/212)

Googling does not reveal much except that it indeed "doesn't work" and Exchange pops up all over the place.

This is no Exchange server. I just want a plain and straight SMTP connection to work. How?

(I have tried running the task as normal user and as system account, no difference.)

Sending a mail manually using telnet to port 25 from that server is working.

(Windows 2008 R2)

can anyone tell me where the memory is gone: (no, this time neither buffers nor cache)

# free
             total       used       free     shared    buffers     cached
Mem:       3928200    3868560      59640          0       2888      92924
-/+ buffers/cache:    3772748     155452
Swap:      4192956     226352    3966604 

top, sorted by memory, descending:

top - 13:42:06 up 1 day,  3:47,  2 users,  load average: 0.08, 0.12, 0.36
Tasks: 228 total,   1 running, 227 sleeping,   0 stopped,   0 zombie
Cpu0  :  2.0%us,  4.0%sy,  0.0%ni, 90.1%id,  0.0%wa,  0.0%hi,  4.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,  0.0%id,100.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   3928200k total,  3868020k used,    60180k free,     2896k buffers
Swap:  4192956k total,   226048k used,  3966908k free,    82068k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3863 root      20   0  902m 199m 3296 S    7  5.2  99:08.77 ndsd
21906 root      20   0  138m 9076 2988 S    0  0.2   0:00.02 sfcbd
 2332 root      20   0  126m 4660 1332 S    0  0.1   0:17.72 mono
 4243 wwwrun    20   0  683m 4468  668 S    0  0.1   0:07.38 java
 2994 root      20   0  202m 2288 1660 S    0  0.1   6:10.02 httpstkd
 4338 root      20   0  184m 2240 1112 S    0  0.1   0:00.52 namcd
21898 root      20   0 32368 1832 1256 R    1  0.0   0:00.08 top

In fact, some time ago oom kicked in and crashed the system (kernel panic), and I'm afraid we're again not far from that point....

UPDATE

 # cat /proc/meminfo                                                     
 MemTotal:        3928200 kB                                             
 MemFree:           51336 kB                                             
 Buffers:            2964 kB                                             
 Cached:            72876 kB                                             
 SwapCached:        29128 kB                                             
 Active:           233440 kB                                             
 Inactive:          88040 kB                                             
 Active(anon):     188920 kB                                             
 Inactive(anon):    56752 kB                                             
 Active(file):      44520 kB                                             
 Inactive(file):    31288 kB                                             
 Unevictable:           0 kB                                             
 Mlocked:               0 kB                                             
 SwapTotal:       4192956 kB                                             
 SwapFree:        3966824 kB                                             
 Dirty:                32 kB                                             
 Writeback:             0 kB                                             
 AnonPages:        225112 kB                                             
 Mapped:            11356 kB                                             
 Shmem:                32 kB                                             
 Slab:            1624080 kB                                             
 SReclaimable:      13740 kB                                             
 SUnreclaim:      1610340 kB                                             
 KernelStack:        4176 kB                                             
 PageTables:        10500 kB                                             
 NFS_Unstable:          0 kB                                             
 Bounce:                0 kB                                             
 WritebackTmp:          0 kB                                             
 CommitLimit:     6157056 kB                                             
 Committed_AS:    2397684 kB                                             
 VmallocTotal:   34359738367 kB                                          
 VmallocUsed:      441372 kB                                             
 VmallocChunk:   34359246755 kB                                          
 HardwareCorrupted:     0 kB                                             
 HugePages_Total:       0                                                
 HugePages_Free:        0                                                
 HugePages_Rsvd:        0                                                
 HugePages_Surp:        0                                                
 Hugepagesize:       2048 kB                                             
 DirectMap4k:       10240 kB                                             
 DirectMap2M:     4184064 kB                                             

slabtop

 Active / Total Objects (% used)    : 9041019 / 9207548 (98.2%)          
 Active / Total Slabs (% used)      : 401132 / 401156 (100.0%)           
 Active / Total Caches (% used)     : 91 / 159 (57.2%)                   
 Active / Total Size (% used)       : 1491537.88K / 1519791.56K (98.1%)  
 Minimum / Average / Maximum Object : 0.02K / 0.17K / 4096.00K           

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME              
4240470 4240319  99%    0.12K 141349       30    565396K pid             
2245140 2219675  98%    0.25K 149676       15    598704K size-256        
2238090 2210087  98%    0.12K  74603       30    298412K size-128        
...