hazymat's questions

Recently one of our customers undertook an IT network audit from another (third party) IT audit firm. The results were generally good, although they pointed out that we had used iSCSI client on Windows Server as a means of connecting to the NAS, instead of creating an SMB share on the NAS. They suggested this was a bad idea:

"There is [also] a security risk with iSCSI and Ransomware attacks, where illicit encryption can be undertaken to the iSCSI disk leaving data unreadable. From a security perspective it is advised to retire this method of data sharing and adopt a share approach."

What do they mean by this? Is it referring to the fact that iSCSI operates on a lower OSI layer (session) than SMB does (application), and an iSCSI disk presents to the application layer the same way as a locally attached disk does, therefore easier to compromise?

If so, is that correct?

I am not an security forensics expert, although our work is often forensic in nature. My understanding is that ransomware is just as likely to attack data on an SMB share accessible to a given Win machine as it would be to attack an iSCSI disk.

Is my understanding correct, or have I missed something?

Additional context to the question

1. CHAP password is set on the iSCSI server, so I would presume the point they are making is related to a compromise of the Win Server that has the iSCSI client installed.

2. Only one iSCSI client is ever connected, and very strong "cyber hygiene" is adopted to ensure this password is not at any point entered into any other server or machine on or off the network.

3. Generally, our preference is to stick with iSCSI for making NAS disks available to the Windows Server We have found that when Windows takes care of the file system, we don't have any problems with advanced access control entries (ACEs) within the DACL. For example QNAP's implementation has in the past been buggy in relation to ACE ordering which can be problematic. We also found a bug with setting CONTAINER_INHERIT_ACE on child objects (communicated to QNAP, but to this day never resolved). This point is not strictly relevant to this question, but provides some context for why we prefer iSCSI.

4. Contrary to my above point, in the case of this particular customer, the iSCSI attached disk in question is formatted with ReFS, because it's used as a Veeam backup store. Although not technically required, Veeam recommends the use of ReFS over NTFS for performance reasons so we tend to prefer this option. (Here is a good article explaining ReFS vs NTFS for backup.) These gains are only possible if we use iSCSI, not if we moved the NAS to SMB.

5. I have read around this subject already a little bit, and have not been able to find any supporting evidence that iSCSI is more subject to ransomware than connecting over a network share, however I remain open-minded.

This question is subtly different from other questions I've found on Serverfault about how best to configure Windows Updates by GPO for Win10 machines on an AD Domain.

I'm running Server 2016, domain from scratch (i.e. latest Win10 Admin Templates for GPO), with Windows 10 clients (1703 - creators update).

I'm generally happy with the Windows Updates settings for workstations, aside from the below problem.

Non-administrator users logged onto Win10 PCs are receiving a blue notification on screen "Updates are available", and they are given the option to "View Updates" and nothing else. Users click on this button, it opens Windows Updates settings screen. It says "Updates are ready to install" and there's an "install now" button. Users are able to click this button.

I don't want this to happen, and I don't want users to be interrupted by the intrusive blue bar, which takes focus from the user and forces them to click the "View Updates" button.

I've made the following settings already in Group Policy:

• Allow automatic updates immediate installation > enabled
• Allow non-administrators to receive update notifications > disabled
• Turn off auto-restart during active hours > enabled 8am-5pm

I want to prevent this "View Updates" bar from showing on the screen, as it interrupts non-administrator users. The users on this network should not interact with Windows Update at all, and updates should be seamless.

I have Windows Server Backup connected to 2x NAS devices (via iSCSI), each one is situated in a different building for resilience.

I'd like to backup from WSB to both targets. WSB supports multiple targets, however the MS recommendation is to rotate the backup targets by taking one physical disk offsite, then subsequently switching them around periodically. Presumably suitable for USB disks rather than fixed network attached targets.

Instead, I'd obviously like to back up to both at the same time, but the above article suggests this isn't supported. An acceptable compromise might be to back up to each target alternately.

The above article states;

• To force a backup to be saved to a particular disk, you can detach or disable all other disks in the backup series.

Perhaps to achieve my goal, I could create a scheduled task that runs a script which takes one target offline (in disk management) and brings the other one back online.

Although possibly an acceptable solution, the above is somewhat clunky.

Is there a better way?

Please do not suggest alternative products. My question is specific to Windows Server Backup and I would like to find the most robust possible solution within this limitation.

Sysadmins have known for a long time that snapshots, or as they are now called, "Checkpoints" are excellent for dev / testing, but not suitable for use in production as they require a chain of VHDX files, making the server slower and more susceptible to corruption. And probably for other reasons.

But I'd like to know whether it's sensible to set up some VMs (running 2016 Server) in a lab environment, install AD and some other services, take a checkpoint of all at the same time, then when I'm happy with the configuration roll these out into production by deleting previous checkpoints which should merge the VHDX files.

I want to know if doing this will effectively remove all the possible performance issues that would otherwise exist with a machine that has checkpoints.

I've found no articles online that discuss this question.

This is for a school that has just moved classrooms to a new building on a new site (for emergency reasons), and we have the summer holiday to migrate their domain to a new one which I'm setting up from scratch, so I want to spend some time getting the configuration just perfect before pushing it to production. They've had enough upheaval already without a crappy implementation of a new AD domain!

NOTES

• I'm aware of Server 2016's new "Production" checkpoint, which uses VSS and allows for an application consistent checkpoint. To be honest, I don't fully understand what this means, but I gather it even makes it viable to roll back SQL or Exchange servers.

• I'm also aware Microsoft have advised never to use checkpoints in production with Active Directory domain controllers, although every bone in my body is sceptical of this advice. In 2007, I ran an Exchange server for 150 users on a VMWare virtual machine against all best practices advice, and because the hardware was up to it, it ran like a dream for years.

Background

We have taken over a Windows Domain from a previous support company. The company had implemented a proprietary desktop lockdown system (i.e. not Group Policy), to achieve certain goals such as

• preventing access to registry
• preventing access to command line
• redirecting UserShellFolders (user Desktop, user Start Menu etc.) to a shared network location

As part of their lockdown system, all users logging on to PCs were added to the local machine (builtin) Administrators group, which we consider to be poor practice, especially for preventing the propagation of viruses and general security of the network.

I have removed their lockdown system, and replicated many of the configurations made by this proprietary system using Group Policy. Including removing local administrator rights from Domain Users. (This was done using Group Policy Restricted Groups.)

The problem

Since doing this, the following happens.

Whilst users are able to access their mapped drives...

• if a user enters the UNC path to such a mapped drive in Windows Explorer, they get an error message
• Likewise, icons on their redirected desktops that point to network locations don't work. When double-clicking on an icon, nothing happens

Here's the error when entering a UNC path into Explorer:

Sure enough, if I make the user a member of the local Administrators group, there's no error.

As part of my testing I enabled the Group Policy (User) setting: "Remove Run menu from Start Menu", as from previous experience I know this also prevents browsing UNC paths. (Yes, this does seem a strange side-effect of this setting, but it's documented in GPO.) Doing the above gives me a subtly different error when I enter the UNC path:

I've also noticed that both of these errors happen even if you enter a network path that doesn't exist.

Because of the above point, and because the local machine (Builtin) Administrators group is completely unrelated to the domain SIDs applied to the shares in question, I conclude that it's not related to NTFS permissions or share permissions on the server share itself.

Given it seems to be a local machine policy issue, I've checked the local security policy and there are no related items set there. Likewise I've run a resultant set of policy to ensure that there are no other domain GPOs applying that I was unaware of.

Could it be UAC related?

EDIT: I checked this SE post about unhiding the network icon. No luck:

https://superuser.com/questions/323598/windows-7-unable-to-browse-to-network-shares-in-explorer-but-map-network-driv

I'm unable to connect remotely to one of a number of servers on the network running 2012 R2 using the Windows RDC client.

Here is the error:

So far I've done the following:

• Ping the server in question. Connectivity = fine.
• Checked Remote Desktop is enabled (system properties > remote > allow remote connections to this computer)
• Windows firewall is currently turned OFF for all profiles (domain, private, public)
• telnet from another machine on the network to the server on port 3389: I just get a blank cursor, no connection
• run netstat -o -a to check the server is listening on port 3389. It is, see below:

TCP 0.0.0.0:3389 servername:0 LISTENING 3236

• I have also run qwinsta to check the RDC service is listening, see pic below:

• I have rebooted the server, checked firewall on the client (and tested from multiple clients)
• Nothing in event logs
• I have changed the listening port (regedit > hklm\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp port) and done all of the above, same happens. netstat shows the node is listening on 3390 but can't connect by RDC client or telnet on that new port.

I followed this unusually thorough article on support.microsoft.com (despite it seems to refer to Server 2003) to troubleshoot further.

I'm concluding that I should somehow find a way to blat the RDC service and rebuild it. Is that possible?

I like the acronym RAT (remote access trojan), because it resembles another type of thing you need to get rid of from your kitchen, and because when you get rid of one and find the hole it made and fix the hole and set traps and invest your time getting into the mind of the rat, you still can't truly rid yourself of all until you rebuild your house out of concrete.

In .htaccess, a proposed temporary measure to deal with a compromised server might be;

RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{REMOTE_ADDR} !123.456.789.000
RewriteRule .* /nopost.html [R=301,L]

Some rats poo on your kitchen, some rats help other rats, and some steal your food.

Given access.log shows a whole load of POST requests taking place on these injected base64 encoded files, is it correct to assume that preventing POST requests as above will resolve the problem of those pests breeding further, not to mention they will not be allowed to POST poo on the kitchen floor in the meantime?

Or will it just stop the rats' deposits?

My question is regarding preventing php scripts from sending mail. It has been marked a duplicate of another more general question about server security but that's not what this question is about.

After a long and bitter fight against hackerspammers who are somehow injecting rogue base64 encoded php files into various web directories on my Debian / Apache / PHP server, (the bitter fight involving first patching existing scripts and changing ftp passwords, web service passwords, and mysql passwords, then rebuilding sites from scratch, installing maldet - which has curbed the problem but not completely eliminated it - then finally turning off postfix altogether by stopping the service (but not uninstalling), then blocking port 25 traffic from the server at the firewall) I am still having problems.

My problems went away for many months, and the server was automatically removed from blacklists according to mxtoolbox. But today I received an mxtoolbox email saying my server is again blacklisted by many services. I don't fully understand how this is possible given I've disabled outgoing port 25 traffic.

When there's a problem, my postfix mailq fills up with hundreds of thousands of emails from a given webuser on my server.

My questions are this:

1. Given I've disabled port 25 traffic using iptables -A OUTPUT -p tcp --dport 25 -j REJECT, how is it possible that mxtoolbox is reporting my server is still sending out spam? When I check the mailq, the mails are backed-up. When I start postfix, items in the mailq don't send as indeed I expect, and I see (delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]:10024: Connection refused) next to each entry.

2. Having identified the location of the RAT by looking at the X-PHP-Originating-Script line in a spammy mail in the mailq, I can find and destroy the file in question, which resolves the problem for anywhere between 5 days and many months. How do I completely prevent any php script from sending a mail? If I enter disable_functions = mail into my php.ini file, I understand this prevents the use of internal functions but not custom functions, which a spammer could take advantage of.

3. What else am I doing wrong?

Caveat: I know #2 doesn't solve my problem at its root, but having taken advice and hardening the security of my server in as many ways as I understand over a couple of years now, I'm working on "deal with the mail reputation issue" rather than "solve all security issues period".

This is a follow-up from my last related question here on ServerFault.

The Hypothetical Scenario. Our small business server has 2x 400GB SSDs and 6 slower 7,200rpm SAS drives. We create a RAID1 array for the SSDs (henceforth Fast Disk) and a RAID10 array for the 6x SAS drives (henceforth Big Disk).

The latter is clearly fast as well. But

• Fast Disk is also Expensive Disk - every megabyte is precious
• We know our physical host system drive requirements will never increase beyond about 50GB, as it was purchased with one destiny in mind: to run Hyper-V (we are around to ensure this remains the case)
• Therefore to best utilise Fast Disk we favour storing some fixed size VHDXs on the SSDs.

(It would be ideal, of course, to avoid storing any virtual disks on our host's C:\ drive, but this hypothetical world in which we find ourselves is strangely devoid of logic and our customer or boss has expressed that she doesn't have unlimited funds for their servers. Nothing like our real world; please use your imagination.)

The Hypothetical Solution. The physical server has 128GB of RAM, and we know that if we allocate much of this to our virtual machines, and opt to enable "Save state", Hyper-V will create large BIN files.

Of course, we don't want these 30-40GB files to take up our precious Fast Disk, so we set the default location for virtual machines as the Big Disk, safe in the knowledge that these BIN files will be stored here. Our Virtual Machines therefore have configuration files and VHDXs dedicated to user data on Big Disk, and VHDXs dedicated to guest system drives on Fast Disk.

The Question. Assuming we would prefer to keep Save State enabled, i.e. we do not wish to get rid of the BIN files altogether, will there be a performance hit in having these stored on a different disk? What actually happens when Hyper-V saves state? Does it dump the whole contents of memory into that file? Or was it already there waiting? Or does some other clever black magic / optimisation take place behind the scenes to make this process happen quickly?

I migrated a working Hyper-V guest server from one physical server to another by stopping the machine, copying the files manually, then importing by "registering in place".

The import completed and the newly registered machine booted fine on the new physical server.

But the newly moved machine cannot ping anything on the physical network except the host. And nothing on the physical network can ping the newly registered machine.

Yet I know the newly built physical server is accessible on the LAN as I can ping it from another machine on the LAN.

And I know the host and guest should be equally accessible from the LAN, that is to say, if one is accessible then the other certainly should be, because I made the setting in Hyper-V Virtual Network to share the "external" type virtual network with the host machine.

Firewall. I've temporarily disabled firewall on all machines mentioned, the new and old physical servers, the virtual (guest), and the remote machine on the network I'm using to test ping connectivity. I've disabled Windows firewall for all profiles on all machines mentioned!

VLANS. There aren't any. Not on the switches, not on the servers.

All hosts and guests are running Windows Server 2012 R2.

Further steps taken:

Throughout all of the below steps, I regularly checked to ensure newly created virtual NIC on new physical server has correct IP address, as did virtual NIC on guest.

• removed and re-added from scratch the virtual network on new physical server
• toggled the setting "share this connection with the host machine" in Virtual Switch settings.
• removed and re-added from scratch the guest's NIC from the Hyper-V setting and set its static IP address back to original (being careful to define the default gateway and furthermore remove the old hidden network device in guest OS Device Manager)
• repeat the above and use a "legacy network adapter" instead of the standard one
• tested by importing another virtual server due for migration. This time it's a Gen 2 Hyper V guest, as opposed to Gen 1 which I have been working on. No change: the same situation. Host and guest can talk, host and LAN can talk, guest and LAN can not talk!
• verified new Hyper-V configuration exactly mirrors old server's. Every check box in every screen double checked
• always ping IP address not hostname, to rule out DNS
• deleted stale arp records from client used to ping test from LAN
• rebooted physical and virtual repeatedly
• disabled and re-enabled virtual NIC on guest (despite also recreating virtual NIC as above)

Perhaps someone might enlighten me as to which one of the virtualisation gods I have royally enraged today, and how I might make things better?

I host a public-facing web server running Debian Wheezy, and latest versions of Postfix, Apache, PHP, Spamassassin, ClamAV, rootkit hunter. Apache is configured with a handful of vhosts, each tied to a user and secured with suExec, and Suhosin. The websites run Wordpress and ModX and by the law of averages given the number of installations on this one server at least 20% of the websites will, at any given time, have some kind of vulnerability be it from the CMS itself or from an out-of-date plugins.

I have notifications from the excellent MX Toolbox website which monitors IP addresses against 100+ blacklists.

When I hear that my IP address has yet again been added to a given blacklist, I ssh in immediately, pause Postfix

postfix stop

wait a few seconds, view the mail queue

mailq

and from this I can tell immediately the source user/vhost of the spam because all mails come from "[email protected]", where "mywebsite.com" is the domain hosted on the vhost that caused the problem.

Then I run a manual malware detection scan using the excellent maldet, and the problem goes away. If I patch all known plugins and software on the site, the problem goes away for c.6 months. If I don't it comes back within about a week.

For testing purposes I have left Postfix stopped for months on end, but some trojans apparently bypass the mail server and send mail directly. (I know this from server resource monitoring, blacklist watches, and bounced spam emails coming back to my domain. Not to mention the Postfix mailq fills up with e.g. 65,000 unsent mails.)

As I care more about mail authenticity than the ability to send emails through websites I host, I've taken a number of steps, namely ensuring my SPF records for each domain do not recognise my own server as an authoritative source of mail for that domain. At the very least this means my domain names aren't being automatically blacklisted.

My question. Is there a clever way to simply block all outgoing email using IPTABLES? I don't just mean blocking mail sent using the email server Postfix, but ALL traffic that could end up with my server being blacklisted?

Until I find other ways of solving this problem I don't mind disallowing websites from sending any mails out. This is NOT ideal as I use some to generate my own business, but I can find other solutions in the meantime.

First of all I've checked these two solutions on serverfault but I'm still having issues.

Here's my problem:

root@myserver:/# mail
/tmp: No space left on device

• Firstly I checked the filesystem usage by running df -k but all filesystem usage was under 25%
• Then I ran df -i to check inode usage; maximum was 4% usage
• I have emptied /tmp and /var/tmp and still get above error

Pertinent information:

• I'm running Debian 6, postfix
• Recently had to remove a number of rogue scripts which were planted in user www directories. They looked like this: https://stackoverflow.com/questions/3328235/how-does-this-giant-regex-work these were being used to send high volumes of spam, and I've checked the mail logs and mail queue to ensure spam is no longer being sent. As a result, mail logs had grown very quickly. I have archived the large mail logs to another machine / deleted them, and done the same with all other large files in /var/log

Here's the output of df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda              20G  4.0G   15G  22% /
tmpfs                 249M     0  249M   0% /lib/init/rw
udev                   10M  112K  9.9M   2% /dev
tmpfs                 249M     0  249M   0% /dev/shm
overflow              1.0M     0  1.0M   0% /tmp