Marki555's questions

Few days ago we noticed some of our outging emails being delayed due to remote mailservers unable to resolve our IP (for example Client host rejected: cannot find your hostname, [92.240.244.176]). We did not make any changes to our DNS recently and our ISP also says they did not touch it.

We have 92.240.244.128/25 delegated from their ns1/ns2.lightstorm.sk to our ns1/n2.mojhosting.sk (our ns1 is running BIND 9.10, ns2 is running PowerDNS 4.0).

They claim the fault is on our side because our DNS servers return REFUSED on this query host 92.240.244.202 ns1.mojhosting.sk, but this must have been like this for a long time. Maybe it is because that command queries for 202.244.240.92.in-addr.arpa., which is only CNAME on our ISP DNS, but does not exist inside our delegated 128/25.244.240.92.in-addr.arpa. sub-tree)? Running dig +trace for both records seems to work for me every time.

MXtoolbox says: Warning: Received Non-Authoritative (lame) Answer from: 'ns1.lightstorm.sk', but I have no idea how to check it. Global DNS Propagation Checker shows our IP unresolvable from majority of locations.

Also our ISP has quite bad IPv6 connectivity and I see their ns1/ns2 having also IPv6 addresses, could this also add to the issue?

I'm trying to mount Azure File Share in Oracle Linux 7.7 and it doesn't work in the latest kernel 4.1.12-124.31.1.1.el7uek.x86_64 (released Sep 2019).

I know that I must use encryption when connecting from outside of Azure region, but I'm not sure if the File Share is in the same region as VM as I don't have access to the Azure Portal.

I have tried mounting with options vers=3.0 and vers=2.1, both give permission denied. I have tried with smbclient and that works without any issues.

Does anyone know if this Oracle UEK kernel supports the cifs protocol needed by Azure File Share? This article mentions kernel 4.11 is needed, but I thought Oracle backported it in their kernel.

Also is there any way to see which protocol version smbclient used to connect? Using -d 3 I can only see that it used NTLMSSP.

After some time (or amount of data written) after server reboot the write speed drops to less than 1 MB/s. This is regardless of which filesystem (also raw partition) and regardless of HDD (HW RAID) or SSD (SW RAID with SSD connected to motherboard AHCI ports, not the raid board). We are testing with command dd if=/dev/zero of=tst bs=1M oflag=dsync (I have tried also 1k and also without dsync, but performance was not better).

The only weird thing I noticed was that the avgrq-sz is only 8 in the iostat output (on other tested servers it was more than 600) and that the req/s is about 100 (also on SSD). Running more dd in parallel gave each of them 1 MB/s and each of them about 100 req/s.

Sample iostat -xN 1 output:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdc               0.00     0.00    0.00  125.00     0.00   500.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00  124.00     0.00   496.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     3.00    0.00  128.00     0.00   524.00     8.19     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     6.00    0.00  124.00     0.00   728.00    11.74     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00  125.00     0.00   500.00     8.00     0.00    0.00    0.00    0.00   0.00   0.00
sdc               0.00     3.00    0.00  128.00     0.00   524.00     8.19     0.00    0.00    0.00    0.00   0.00   0.00

Iostat output with 8x dd running:

sdc               0.00    64.00    0.00  959.00     0.00  7560.00    15.77     0.00    0.00    0.00    0.00   0.00   0.00

lsblk -O output is consistent with other servers, which do not have this problem (like MIN-IO, RQ-SIZE, LOG-SEC). Current kernel is 4.9.16-gentoo, but the issue started with older kernel. Running dd with oflag=direct is fast.

EDIT: Based on shodanshok's answer I now see that the requests are indeed small, but the question is why io scheduler does not merge them into larger requests? I have tried both cfq and deadline schedulers. Is there anything I can check (or compare against other servers)?

Output when running with oflag=direct (speed is OK):

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdc               0.00     0.00    2.00  649.00     8.00 141312.00   434.16     2.78    4.26    2.00    4.27   1.53  99.60
sdc               0.00     0.00    0.00  681.00     0.00 143088.00   420.23     2.71    3.99    0.00    3.99   1.46  99.20
sdc               0.00     0.00    2.00  693.00     8.00 146160.00   420.63     2.58    3.71    0.00    3.72   1.42  98.80
sdc               0.00    49.00    2.00  710.00     8.00 146928.00   412.74     2.68    3.76   22.00    3.71   1.39  99.20
sdc               0.00     0.00    1.00  642.00     4.00 136696.00   425.19     2.43    3.79   60.00    3.71   1.42  91.60

Server is Dell PowerEdge R330 with 32 GB RAM, LSI MegaRAID 3108 controller with HDDs, SSDs connected to onboard SATA, Intel E3-1270 CPU. Filesystem ext3, but the same happens with dd to raw partition.

Output of lsblk (sdc is HW RAID HDD, sda/sdb SW RAID SSDs):

NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sdb       8:16   0 223.6G  0 disk
|-sdb4    8:20   0     1K  0 part
|-sdb2    8:18   0   140G  0 part
| `-md1   9:1    0   140G  0 raid1 /
|-sdb5    8:21   0    25G  0 part
| `-md2   9:2    0    25G  0 raid1 /var/log
|-sdb3    8:19   0     1G  0 part
|-sdb1    8:17   0    70M  0 part
| `-md0   9:0    0    70M  0 raid1 /boot
`-sdb6    8:22   0  57.5G  0 part
  `-md3   9:3    0  57.5G  0 raid1 /tmp
sr0      11:0    1  1024M  0 rom
sdc       8:32   0   3.7T  0 disk
`-sdc1    8:33   0   3.7T  0 part  /home
sda       8:0    0 223.6G  0 disk
|-sda4    8:4    0     1K  0 part
|-sda2    8:2    0   140G  0 part
| `-md1   9:1    0   140G  0 raid1 /
|-sda5    8:5    0    25G  0 part
| `-md2   9:2    0    25G  0 raid1 /var/log
|-sda3    8:3    0     1G  0 part
|-sda1    8:1    0    70M  0 part
| `-md0   9:0    0    70M  0 raid1 /boot
`-sda6    8:6    0  57.5G  0 part
  `-md3   9:3    0  57.5G  0 raid1 /tmp

With oflag=direct the speed is OK, but the issue is that applications don't use direct io, so even plain cp is slow.

/sys/block/sdc/queue/hw_sector_size : 512
/sys/block/sdc/queue/max_segment_size : 65536
/sys/block/sdc/queue/physical_block_size : 512
/sys/block/sdc/queue/discard_max_bytes : 0
/sys/block/sdc/queue/rotational : 1
/sys/block/sdc/queue/iosched/fifo_batch : 16
/sys/block/sdc/queue/iosched/read_expire : 500
/sys/block/sdc/queue/iosched/writes_starved : 2
/sys/block/sdc/queue/iosched/write_expire : 5000
/sys/block/sdc/queue/iosched/front_merges : 1
/sys/block/sdc/queue/write_same_max_bytes : 0
/sys/block/sdc/queue/max_sectors_kb : 256
/sys/block/sdc/queue/discard_zeroes_data : 0
/sys/block/sdc/queue/read_ahead_kb : 128
/sys/block/sdc/queue/discard_max_hw_bytes : 0
/sys/block/sdc/queue/nomerges : 0
/sys/block/sdc/queue/max_segments : 64
/sys/block/sdc/queue/rq_affinity : 1
/sys/block/sdc/queue/iostats : 1
/sys/block/sdc/queue/dax : 0
/sys/block/sdc/queue/minimum_io_size : 512
/sys/block/sdc/queue/io_poll : 0
/sys/block/sdc/queue/max_hw_sectors_kb : 256
/sys/block/sdc/queue/add_random : 1
/sys/block/sdc/queue/optimal_io_size : 0
/sys/block/sdc/queue/nr_requests : 128
/sys/block/sdc/queue/scheduler : noop [deadline] cfq
/sys/block/sdc/queue/discard_granularity : 0
/sys/block/sdc/queue/logical_block_size : 512
/sys/block/sdc/queue/max_integrity_segments : 0
/sys/block/sdc/queue/write_cache : write through

Today I noticed unusual high request rate on Apache webserver and also quite high incoming network traffic. Upon checking Apache's mod_status page, I found the offending URLs to be from path www.server.com/www/wp-includes/js/tinymce/plugins/wpautoresize/. And indeed I have found several hacked (obfuscated) PHP scripts there.

Also noticed weird process executed by www-data user:

www-data  7300 10.8  0.1 2122900 18768 ?       Ssl  Jul11 121:47 /usr/bin/host

Checking /proc/7300/cmdline revealed that indeed this is the original /usr/bin/host binary. netstat -anp showed it has many HTTP connections opened, so somehow that binary is abused. debsums confirmed the binary checksum to be OK. As the process was run under www-data user, I had no reason to believe server itself was compromised.

How is that binary abused?

EDIT: This not broad "how to deal with compromised server" question. Rather a question (and already an answer) about one specific type of abuse how is it technically done, as this particular case is quite creative in how it works. It seems this is in wild for several years now (old threads & questions from 2012) and I encountered it this week.

Using postfix 2.9 on debian I have configured before-queue proxy filter for scanning of spam and viruses smtpd_proxy_filter=127.0.0.1:27. I am using proxy filter and not content filter, so that I can directly reject the email (with after-queue content filter, I can only discard the email and optionally send backscatter to forged sender address).

I also want to use policy service to limit the number of sent emails per SASL username. This is configured in smtpd_recipient_restrictions just before permit_sasl_authenticated. The policy daemon works OK, but postfix complains that it cannot HOLD the email with this message:

warning: access table inet:127.0.0.1:10031: with smtpd_proxy_filter
specified, action HOLD is unavailable

As per this thread, this is not directly possible:

To HOLD mail with smtpd_proxy_filter, specify the HOLD action with the smtpd proces AFTER the filter.

I have tried that, however when I put the policy service on the smtpd daemon after the proxy filter (smtpd on port 26 in my case), it doesn't get any SASL information (confirmed with tcpdump).

smtpd :25 ---> before-queue proxy localhost:27 ---> smtpd localhost:26

So how can I combine these two checks?

EDIT: Maybe nested before-queue filters would work? Then I would need to use different program for the SASL checks, or make some filter/policy protocol rewriting...

I have configured Policyd v2 (cluebringer) to apply quota on msg count for SASL and non-SASL users. It is easy in policyd [1]. However I don't know how to configure postfix (2.7.1). Right now I have this (non important rules removed):

smtpd_recipient_restrictions    =
    check_policy_service inet:127.0.0.1:10031,
    permit_sasl_authenticated,
    permit_mynetworks,
    reject_unauth_destination,
    reject_rbl_client zen.spamhaus.org,

check_policy_service must be before permit_sasl_authentificated to be able to check on SASL users (because permit_sasl_authentificated returns OK and no other permissions are checked after it).

But having it here means, that it would count every incoming email attempt in quota - regardless if it will be accepted by postfix or not (because greylist/blacklist and reject_unauth_destination are after check_policy_service). So all rejected spam would count on recipient users' quota.

How is it possible to have policyd be invoked for both SASL incoming email and for non-SASL email which is accepted for server?

1]: http://lists.policyd.org/pipermail/users_lists.policyd.org/2012-October/004002.html

I have some big files inside VM which I need to copy to ESXi host datastore. What is the best/fastest method? Right now I am using vSphere Client inside the VM. VM is running in virtual network with another VM acting as a router to the outside world. As ESXi has different public IP than te router, traffic goes out to the physical network switch and back in over the same NIC. This is connected only at 100 Mbit, so the real speed is only about 4 MB/s.

Is it somehow possible to connect a VM to the VMkernel management network, so that ESX would handle the traffic itself?

Or is it possible to mount the vmdk of VM via command-line and copy the files to datastore locally?

I will have 2 servers in different datacenters (different countries) and I want to use DNS load-balancing mainly for High Availability of website hosted on those 2 servers. It is just ad tracking site, which records hit in local database and returns few lines on html code.

I want to return 2 A records each time because of DNS pinning in browsers (if one server fails, browser will try second A record which it has already cached).

Both servers will be acting also as DNS servers for redundancy. Now comes my proposed solution: I will use BIND and have both servers as a master for that zone. On each server there will be running script, which will periodically test availability (http) of both servers and remove IP from DNS in case of failure.

Now the questions :)

1) Is BIND suitable for this solution? I think BIND performance is good and it is easy to manipulate the zone file via script. And as I will modify the zone only in case of failure/maintenance, the modifications (and thus bind reload) won't be often.

2) I plan to use TTL of 5 minutes. The website will have about 1000-3000 req/s but from distinct clients (each IP only 1-3 requests), so I think the DNS load won't be too much. I suppose their ISPs will cache the responses for those 5 mins. Is there any reason to lower the TTL even more?

3) Is my master-master approach good? Or should I make one of the servers master and the other one slave? Right now each server can monitor both itself and the other one. If only webservice fails, both DNS nodes will notice it. If the whole server fails, then the remaining DNS node will notice it and the failed node will not answer DNS queries anyway.

4) Is it a big issue when one NS server does not respond to queries? If yes, I can make a third DNS, so anytime at least 2 of them would accept queries...

5) Should I rewrite the zone file via script, or just use dynamic DNS update (for example via nsupdateutility)?