Questions[veeam](server)

I want to create a remote repository for the backup of my local infrastructure. I'm using Veeam v10.

I think I need a remote machine with a Veeam component installed, then I was wondering if I can build a EC2 Windows machine with Veeam with S3 as storage.

The goal is to transfer from my site to EC2 only the incremental backup (I cannot transfer a full with my bandwith!) and let the machine compute and merge backup in a reverse incremental backup. Is this a good configuration? Is it possibile to use EC2 with a S3 storage for veeam repository?

Recently one of our customers undertook an IT network audit from another (third party) IT audit firm. The results were generally good, although they pointed out that we had used iSCSI client on Windows Server as a means of connecting to the NAS, instead of creating an SMB share on the NAS. They suggested this was a bad idea:

"There is [also] a security risk with iSCSI and Ransomware attacks, where illicit encryption can be undertaken to the iSCSI disk leaving data unreadable. From a security perspective it is advised to retire this method of data sharing and adopt a share approach."

What do they mean by this? Is it referring to the fact that iSCSI operates on a lower OSI layer (session) than SMB does (application), and an iSCSI disk presents to the application layer the same way as a locally attached disk does, therefore easier to compromise?

If so, is that correct?

I am not an security forensics expert, although our work is often forensic in nature. My understanding is that ransomware is just as likely to attack data on an SMB share accessible to a given Win machine as it would be to attack an iSCSI disk.

Is my understanding correct, or have I missed something?

Additional context to the question

1. CHAP password is set on the iSCSI server, so I would presume the point they are making is related to a compromise of the Win Server that has the iSCSI client installed.

2. Only one iSCSI client is ever connected, and very strong "cyber hygiene" is adopted to ensure this password is not at any point entered into any other server or machine on or off the network.

3. Generally, our preference is to stick with iSCSI for making NAS disks available to the Windows Server We have found that when Windows takes care of the file system, we don't have any problems with advanced access control entries (ACEs) within the DACL. For example QNAP's implementation has in the past been buggy in relation to ACE ordering which can be problematic. We also found a bug with setting CONTAINER_INHERIT_ACE on child objects (communicated to QNAP, but to this day never resolved). This point is not strictly relevant to this question, but provides some context for why we prefer iSCSI.

4. Contrary to my above point, in the case of this particular customer, the iSCSI attached disk in question is formatted with ReFS, because it's used as a Veeam backup store. Although not technically required, Veeam recommends the use of ReFS over NTFS for performance reasons so we tend to prefer this option. (Here is a good article explaining ReFS vs NTFS for backup.) These gains are only possible if we use iSCSI, not if we moved the NAS to SMB.

5. I have read around this subject already a little bit, and have not been able to find any supporting evidence that iSCSI is more subject to ransomware than connecting over a network share, however I remain open-minded.

Recently our MySQL server has been "going away" (ie. the client connection drops out). After weeks of trying different things (like adjusting packet size), we've discovered that it's our Veeam imaging backups which use the VMWare API to snapshot and copy the vmdks etc.

We are using ESXi 5 with a Centos 6.4 guest, running (pretty much) only MySQL 5.1.69-log.

The change which seemed to initiate this problem was increasing the physical disk size to 300GB, from about 100, and resizing the guest filesystem to use most of the new capacity. Ever since the disk was increased, we've been getting these problems during backups - presumably due to the increase time it takes to perform snapshot related functions.

The new disks are 2x300GB Gen8 15k SAS in RAID1. The old disks would have been similar only smaller. The target of the Veeam process is a ReadyNAS over a 1Gb dedicated ethernet (i.e. separated from general office traffic).

The host is an HP DL380P tower:

==server spec (BASE CHASSIS)==
SERIES DL380P GEN8
PROCESSOR TYPE Intel Xeon E5-2609 v2 (2.5GHz/4-core/10MB/6.4GT-s QPI/80W)
NUMBER OF PROCESSORS 2 
MEMORY 80GB
INTERNAL DRIVE BAYS 8 SFF HDD Bays
COMPATIBLE HDD SFF SAS/SATA
HARD DISK CONTROLLER SMART ARRAY P420I/ZERO MEMORY CONTROLLER (RAID 0/1/1+0)

My "IT guy" has made a few tweaks to the Veeam config including skipping empty blocks (the majority of the new disk is empty), but this didn't seem to help at all.

Veeam haven't been much help either, saying "reboot the target" or "we just use VMWare APIs".

I believe the "stun" means the virtual machine simply freezes for a time (around 30s) then continues normally.

VMWare.log example:

Line 7411: 2016-06-08T17:11:44.910Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 21068381 us
Line 7556: 2016-06-08T17:22:24.608Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 19819322 us
Line 7700: 2016-06-08T17:22:30.140Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 1130044 us
Line 7929: 2016-06-08T17:23:08.616Z| vcpu-0| I120: Checkpoint_Unstun: vm stopped for 30197618 us

So my problem has two likely solutions:

1. Is there a way to prevent or reduce the "stunning" of a VMWare guest during imaging.

2. Is there a way to reduce the impact of the stun onto MySQL or the virtual network or Centos.

The environment consists of two 2012R2 Virtual Machines running RabbitMQ in high availability (ha-all) on their Queues. I use Veeam to create snapshot backups that are sent offsite as part of the DR policy.

What I am seeing are intermittent failures of the cluster when the Veeam backup occurs. When the cluster breaks it causes Mnesia events to be logged or sometimes causes one node to turn itself off completely. I believe the issue is how the VM is blipped by Veeam where it essentially pauses the VM for a brief moment and then continues. When this blip occurs both Nodes see the other vanish and the secondary promotes itself to master immediately. With two masters running as soon as they see each other (literally seconds later) they butt heads and the cluster breaks.

I read about net_ticktime here and implemented 300 seconds thinking this would help make the cluster more resilient to the short Veeam blips, but it doesn't appear to have helped. When one node sees the other vanish the secondary promotes itself to master immediately and does not seem to utilize the net_ticktime setting.

Example Mnesia error:

Mnesia('rabbit@Node01'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@Node02'}

Has anyone else experienced this or something similar? Are there additional configuration setting with RabbitMQ or Erlang that might help make the cluster more resilient to small blips of connectivity loss between the nodes?

This information is available in the web management GUI, but I'd like to get this information for use in a Powershell script that will use those values, and I'd prefer to not have to figure out how to scrape the website if I don't have to. Does anyone know if there's a file on the Exagrid unit, or a command that can be run, that will expose this info? For each given share, I'd like to know the current dedupe ratio.

My desired end-result is to combine this with Veeam's dedupe numbers, so I can have a start-to-finish dedupe number for my backup jobs.