Stilez's questions

I notice a lot of U.2 adapters on sale, seem to show what looks like a U.2 to SFF-8643 cable, or SFF-8643 PCIe connector/adapter, instead of the expected U.2 to SFF-8639 cable or SFF-8639 PCIe connector/adapter. (Example 1 ; Example 2)

1. Are SFF-8643 and SAFF-8639 cables, connectors and adapters electrically compatible? Or are SFF-8643 deployed as 2 incompatible variants ( for use with SAS and U.2) that have the same physical form factor but are electrically incompatible?
2. If compatible, does this mean I can plug any U.2 2.5" SSD into any SFF-8643 baseboard/PCIe adapter (not just into SFF-8639 connectors/adapters), provided the cable/adapter has physically compatible terminations at both ends? Would HBAs such as LSI-93xx with SFF-8643 connectors be compatible? Alternatively, what criteria apply for compatibility, how to check?
3. SFF-8643 with SAS is typically used with (eg) 4 way fanout cables. What are the implications for SFF-8643 fanout if used with U.2 drives? Is fanout possible with U.2 drives (albeit at reduced individual bandwidth)? Or are fanout cables only made for SFF-8643 to SAS/SATA, not SFF-8643 to U.2?

I've looked but can't find a definitive answer, either on Google, or on this site, or on superuser.com. There are similar questions but none specifically clarify these points

Note: This question is real-world, but to analyse it, please note I've begun from a "theoretical" starting point of device and bus capability, which I acknowledge will not usually be at all representative of in-use bandwidth utilisation.

I have an array of 18 x SAS3 mixed 8TB and 10TB enterprise drives, being configured as 6 sets of 3 way mirrors under ZFS (FreeBSD). Currently they are all hanging off a single 24 port HBA (9305-24i).

It's hard to know how many drives work at peak together, but assuming they were all in use for reading, I get the following calculation worst case (may not be realistic?):

SAS3 simplex bandwidth: (12 gbits/sec) x (8/10 encoding) = 1.2 GB/sec raw data max
=> 18 x SAS3 maximum at peak: (1.2 x 18) = 21.6 GB/sec
But PCI-E 3.0 x 8 simplex bandwidth: 7.9 GB/sec

So at a first glance, it seems that the array could be throttled very badly under demand, because the link is limiting the array IO from 21.6 GB/sec down to 7.9 GB/sec each way: a loss of 64% of HDD I/O capability.

On the other hand, the file server is primarily used by 2 end-users: the file server itself which needs to read and write at highest speed as part of its file handling, and any other devices which are linked by 10 GbE, and hence can't consume more than 2 GB/sec simplex even with 2 link aggregation. Therefore potentially it can't use more than a fraction of the PCI-E link speed regardless, in any event.

(Even if I do some file management on the server itself via SSH, 2 GB/sec is still quite a good speed, and I might not complain.)

Also whatever SAS 3 might deliver in theory, 12 gbit = 1.2 GB/sec and even on maximum reading from its internal cache, it seems unlikely an enterprise HDD can utilise SAS bandwidth. SSDs yes, but HDDs? Less likely? Maximum read is usually quoted as around 200 - 300 GB/sec in datasheets.

My question is therefore, given the HBA can provide up to almost 8 GB/sec bandwidth across PCI-E, and the end users can consume at most 2 GB/sec, will there in fact be a throttling effect?

Put another way, does it matter that the disk array in theory is throttled from 22 GB/sec down to 8 GB/sec at the PCIE slot given the end users have a 2 GB/sec aggregated connection? Or will the PCI-E slot limitation still be an issue because the local system at times needs faster I/O than the end-device bandwidth would suggest?

If there is a limitation I can split the disks across 2 HBAs, but I'd like some idea how to assess if there's a real issue, before sacrificing a second PCIE slot to raise the bar on raw disk IO.

I just got an alert from one of my servers, running FreeBSD 11.2 on a Xeon E5-1620 v4 with ECC. Its the first time I've seen an error like this, and there's a lot of detail. But it's also a one-off item at this point, there's no previous or subsequent repeat other than copied below. Log timezone is UK GMT +0 if that helps, so it happened 24 hours ago.

Im trying to understand whether it arose within the CPU's memory controller or one of the RAM modules, and either way whether its a transient that was corrected and if it's safe to ignore?

Or, what's (probably) happened?

Nov 28 03:45:01 SVR-3 ZFS: vdev state changed, pool_guid=**** vdev_guid=****
Nov 28 10:50:57 SVR-3 MCA: Bank 7, Status 0xcc0004c000010092
Nov 28 10:50:57 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:57 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:57 SVR-3 MCA: CPU 0 COR (19) OVER RD channel 2 memory error
Nov 28 10:50:57 SVR-3 MCA: Address 0x2cb6ab7f80
Nov 28 10:50:57 SVR-3 MCA: Misc 0x140282886
Nov 28 10:50:57 SVR-3 MCA: Bank 11, Status 0x8c00004c000800c2
Nov 28 10:50:57 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:57 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:57 SVR-3 MCA: CPU 0 COR (1) MS channel 2 memory error
Nov 28 10:50:57 SVR-3 MCA: Address 0x2c80407fc0
Nov 28 10:50:57 SVR-3 MCA: Misc 0x122101100110188c
Nov 28 10:50:57 SVR-3 MCA: Bank 11, Status 0x8c00004c000800c2
Nov 28 10:50:57 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:57 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:57 SVR-3 MCA: CPU 0 COR (1) MS channel 2 memory error
Nov 28 10:50:57 SVR-3 MCA: Address 0x2c80417fc0
Nov 28 10:50:57 SVR-3 MCA: Misc 0x122101000100188c
Nov 28 10:50:57 SVR-3 MCA: Bank 11, Status 0xcc00008c000800c2
Nov 28 10:50:57 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:57 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:57 SVR-3 MCA: CPU 0 COR (2) OVER MS channel 2 memory error
Nov 28 10:50:57 SVR-3 MCA: Address 0x2c80437fc0
Nov 28 10:50:57 SVR-3 MCA: Misc 0x122111110110188c
Nov 28 10:50:57 SVR-3 MCA: Bank 11, Status 0xcc00010c000800c2
Nov 28 10:50:57 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:57 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:57 SVR-3 MCA: CPU 0 COR (4) OVER MS channel 2 memory error
Nov 28 10:50:57 SVR-3 MCA: Address 0x2c80477fc0
Nov 28 10:50:57 SVR-3 MCA: Misc 0x122111011000188c
Nov 28 10:50:58 SVR-3 MCA: Bank 11, Status 0xcc00020c000800c2
Nov 28 10:50:58 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:58 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:58 SVR-3 MCA: CPU 0 COR (8) OVER MS channel 2 memory error
Nov 28 10:50:58 SVR-3 MCA: Address 0x2c80c07f40
Nov 28 10:50:58 SVR-3 MCA: Misc 0x122101011001188c
Nov 28 10:50:58 SVR-3 MCA: Bank 11, Status 0xcc00040c000800c2
Nov 28 10:50:58 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:58 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:58 SVR-3 MCA: CPU 0 COR (16) OVER MS channel 2 memory error
Nov 28 10:50:58 SVR-3 MCA: Address 0x2c81507fc0
Nov 28 10:50:58 SVR-3 MCA: Misc 0x122110001101188c
Nov 28 10:50:58 SVR-3 MCA: Bank 11, Status 0xcc00080c000800c2
Nov 28 10:50:58 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:58 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:58 SVR-3 MCA: CPU 0 COR (32) OVER MS channel 2 memory error
Nov 28 10:50:58 SVR-3 MCA: Address 0x2c82627fc0
Nov 28 10:50:58 SVR-3 MCA: Misc 0x122100100001188c
Nov 28 10:50:58 SVR-3 MCA: Bank 11, Status 0xcc00100c000800c2
Nov 28 10:50:58 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:58 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:58 SVR-3 MCA: CPU 0 COR (64) OVER MS channel 2 memory error
Nov 28 10:50:58 SVR-3 MCA: Address 0x2c84057fc0
Nov 28 10:50:58 SVR-3 MCA: Misc 0x122111101111188c
Nov 28 10:50:58 SVR-3 MCA: Bank 11, Status 0xcc00200c000800c2
Nov 28 10:50:58 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:58 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:58 SVR-3 MCA: CPU 0 COR (128) OVER MS channel 2 memory error
Nov 28 10:50:58 SVR-3 MCA: Address 0x2c884b7fc0
Nov 28 10:50:58 SVR-3 MCA: Misc 0x122110100100188c
Nov 28 10:50:59 SVR-3 MCA: Bank 11, Status 0xcc00400c000800c2
Nov 28 10:50:59 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:50:59 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:50:59 SVR-3 MCA: CPU 0 COR (256) OVER MS channel 2 memory error
Nov 28 10:50:59 SVR-3 MCA: Address 0x2c90cd7f40
Nov 28 10:50:59 SVR-3 MCA: Misc 0x122111011110188c
Nov 28 10:51:00 SVR-3 MCA: Bank 11, Status 0xcc00800c000800c2
Nov 28 10:51:00 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:51:00 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:51:00 SVR-3 MCA: CPU 0 COR (512) OVER MS channel 2 memory error
Nov 28 10:51:00 SVR-3 MCA: Address 0x2ca1557fc0
Nov 28 10:51:00 SVR-3 MCA: Misc 0x122111001100188c
Nov 28 10:51:04 SVR-3 MCA: Bank 11, Status 0xcc01000c000800c2
Nov 28 10:51:04 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:51:04 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:51:04 SVR-3 MCA: CPU 0 COR (1024) OVER MS channel 2 memory error
Nov 28 10:51:04 SVR-3 MCA: Address 0x2ccbf77f40
Nov 28 10:51:04 SVR-3 MCA: Misc 0x122100100010188c
Nov 28 10:51:10 SVR-3 MCA: Bank 11, Status 0xcc02000c000800c2
Nov 28 10:51:10 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 10:51:10 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 10:51:10 SVR-3 MCA: CPU 0 COR (2048) OVER MS channel 2 memory error
Nov 28 10:51:10 SVR-3 MCA: Address 0x2d11dc7f40
Nov 28 10:51:10 SVR-3 MCA: Misc 0x122110010010188c
Nov 28 11:26:34 SVR-3 MCA: Bank 11, Status 0xcc03e28c000800c2
Nov 28 11:26:34 SVR-3 MCA: Global Cap 0x0000000007000c16, Status 0x0000000000000000
Nov 28 11:26:34 SVR-3 MCA: Vendor "GenuineIntel", ID 0x406f1, APIC ID 0
Nov 28 11:26:34 SVR-3 MCA: CPU 0 COR (3978) OVER MS channel 2 memory error
Nov 28 11:26:34 SVR-3 MCA: Address 0x2e5fbf7f40
Nov 28 11:26:34 SVR-3 MCA: Misc 0x122111011110188c
Nov 29 00:00:00 SVR-3 syslog-ng[2846]: Configuration reload request received, reloading configuration;

I've read about this system in a white paper somewhere, but can't remember where, or any details, and haven't found it since.

The comment was in regard to GPU virtualisation with Type 1 (bare metal) hypervisors. It noted that some system or other had used a different path from vGPU hardware. Instead of needing a card that could provide native vGPU or shared GPU, it used a different method. From memory, one installed an OS that was already able to support multiple desktop users (it suggested Windows Server 2016) as one VM, and then with some kind of mediating driver or shim in the hypervisor+server VM, the other VMs could redirect just their GPU calls to that VM, accelerating their GPU needs.

The difference being that in a normal shared user context (Windows Server/RDS), the multi user OS hosts and manages the users sessions+processes. As described, in this design, the only things the Windows Server VM handles, are requests initiated by the hypervisor to set up effectively "null" user accounts as required (to leverage the multiuser graphical sharing inherent in Windows Server), and GPU calls relayed from the hypervisor on behalf of VMs which are presented as graphics calls from those users, whose results are presumably forwarded back to the originating VMs via the hypervisor, or perhaps even directed to the VM operator via RDS - I'm not sure, that part wasn't described.

(Other than that, none of the other Windows Server VM capabilities are used at all. So its role sounds more like it's being leveraged to create a GPU sharing appliance able to "convert" a single user GPU to a multiuser GPU, via Windows' native full multiuser GPU sharing ; it isn't being used as an actual "server OS".)

The overall benefit was to provide what sounds like a "poor man's vGPU" leveraging existing GPU sharing mechanisms baked into Windows Server - a system that could take almost any decent consumer graphics card supported by Windows Server, and share/virtualise it among other VMs, without needing a specialist high-end card that had sGPU/vGPU built-in.

It also sounded like an approach that could be less susceptible to prohibitive licensing/HW cost (AMD/nVidia vGPU), EOL drivers (nVidia K1/K2), limited main core count due to inclusion of GPU cores (Iris Pro), or narrowed range of graphics APIs, and would also be considerably future-proof. So it sounded ideal for a small home VM server/home lab.

Does anyone know what this might refer to, or a system like this? I think it might have been when I was looking into Xen/Citrix, but I can't find a specific reference there either.

Short version - how well do current hypervisors support vGPU with VM suspend/resume capability?

Longer version:

I've used VMware Workstation for many years in my home lab, with the GPU (used for graphics not compute) shared between VMs, and VMs suspended and resumed at will. I'm interested in moving to a bare metal (Type 1) hypervisor, due to limitations in running a hypervisor on top on a full OS. For example, my setup struggles with > about 3 VMs, struggles with memory and resource sharing, isn't quite as stable as a bare metal hypervisor might be, etc.

But getting information on how the various hypervisors work with vGPU is opaque to say the least. I've got use of an early nVidia GRID K2 card, and hoped to use ESXi, but so far this is the sum of my knowledge:

• GRID/ESXi - My existing K2 is EOL. It'll work up to ESXi 6.5 only, and suspend requires >= ESXi 6.7 (great!). I can use ESXi 6.7 only with newer GRID, but all newer GRID requires commercial cost-level licensing which is totally out of my league. Running unlicensed would be a breach of the EULA and therefore not okay, but even if I did it briefly, for test purposes and evaluation only, the implications seem to shift every few months as new GRID software is released - there's no stability to it. If I could use my K2 I'd be happy, but its compatibility is very limited right now, it seems, and will vanish in a while.

• Non-ESXi hypervisors - Other reputable and solid hypervisors exist too (KVM, open source + proprietary (Citrix) Xen, bhyve, Hyper-V, QEMU, VirtualBox). But not all are equally robust/efficient at running VMs, and finding with certainty, which of these supports vGPU (on the K2 or any other GPU), and also has suspend/resume capability with vGPU in use, is incredibly tricky.
  
  Its also usually unclear up front exactly what's supported, what incurs extra licensing fees, etc. (For example, finding out how much is possible with Hyper-V and RemoteFX without a separate remotefx/RDS license). Or perhaps, if I use a different type 2 hypervisor, I will find I don't need a type 1 hypervisor to get around my current setup headaches.

• Other vGPUs - There are also other vGPU providers. Intel has its Iris Pro, AMD has a few too. Iris Pro has advantages (cost built into CPU, likely won't have support for older versions withdrawn quickly), and AMD seems costly. But again, Iris Pro is fairly newish and finding which hypervisors can/will be able to utilise it and allow suspend/resume is not easy at all. Also Iris Pro comes at a cost of CPU cores, which is exactly what you don't want for a hypervisor, and isn't available on Xeon either (which have more cores) AFAIK.

• RemoteFX - RemoteFX is a question mark all on its own. I use VMs for (mainly) Windows and (sometimes) BSD desktops, and rarely, family use them for very light Windows gaming, so Hyper-V + Gen1 VM is interesting. But figuring RemoteFX support/leveraging of vGPU isn't easy, in addition to finding about suspend/resume, plus perhaps options for other transports (Teridici cards?). And of course, RemoteFX is being retired and we have little info on how its successor will work. So clarity about the state of play is really hard to find.

I'm not looking for a recommendation here.

I'm just trying to understand more clearly, how things actually stand - which mature/robust hypervisors both support vGPU and also support suspend/resume of vGPU based VMs.

And also any significant conditions/restrictions/limitations which would apply, if any? (Number/size of displays and VMs, accelerates some things not others, etc)

Thanks for any help in clarifying this!

Ideally I'd like to know the GPU families they can do vGPU with, or any other major limitations/requirements - I imagine "Windows guests only" will be a common one and that's ok. But that's a bonus, compared to the main point.

I just came across this page ("Direct hosting of SMB over TCP/IP"). It's quite old, so I can't be sure if it's the standard these days. It refers to disabling NetBIOS over TCP and disabling WINS, which suggests that it's still not the standard or default way SMB works these days, at least in many smaller networks.

Is this page still relevant, and if it is, how do I know whether it's a good idea?

My network here is Win 8.1/10 talking SMB 3.x to FreeNAS 11 (Samba 4.5+) if it helps.

Update:

Based on comments below I've updated the question to reflect the point of confusion:

If it's now standard to disable it, clearly none of the Win 8.1/10 PCs and Server 2016/Samba 4.5+ servers on my local network ever got the memo. (Nor did I.) It's all still running on ports 137-139 despite being 100% modern devices that should apparently be preferring a newer approach.

I can imagine that when NetBIOS over TCP is disabled in a small local network without AD, the host announce/locate functions used to enumerate devices offering shares/printers migrates to DNS (or host file lookup), and the actual client-server traffic runs much as before only on a different port and more modern protocol.

I can add specific hosts to my local DNS resolver and I'm running DHCPD locally, if that helps. I might run RADIUS in future but not at the moment.

On reflection, I'm confused about server enumeration, and what I need to do, to move off the old approach (= changes needed to how I currently configure the clients/servers).

1. How it works? If the old BROWSER functions on ports 137-139 are disabled, how does enumeration by a client of local shares work? Where do available servers and peers offering shares make themselves known? How do potential clients "know" which other local devices offer file or printer shares, and if a device starts to offer a newly created share, or starts to offer a share when it didn't previously, how do existing LAN devices discover this?
2. What changes I have to make, so it'll work that way? On Windows 8.1/10, apart from the checkbox "Disable NetBIOS over TCP", what else do I have to do, to move my local SMB file shares off the old protocols, off ports 137-139, and onto port 445 or whatever the current practice is?

I have a fairly common setup - vSphere and ESXi hosts using FreeNAS as the VM store. The servers can see each other (obviously) so I want to segregate system admin traffic and user traffic onto different VLANs, and restrict the management IPs on both boxes.

Configuring management access on ESXi is easy, but I can't figure how to do it on FreeNAS. At the moment the relevant FreeNAS config is that it has one active NIC (10G Chelsio) with IP of say 192.168.1.2, and no VLANs have been set up on the network yet. What I'd like is to do one or more of the following:

1. Create two VLANs, say 1 and 2, with any VLAN able to access sharing services on the sharing ports, but only VLAN 2 able to reach the admin IP/port

2. Create two IPs on the one NIC, say 192.168.1.2 and 192.168.1.3, with only 192.168.1.3 able to reach the management login.

3. Blocking the management access ports (80,443 etc) for VLAN != 2 and/or IP != 192.168.1.3.

As FreeNAS isn't a router or firewall it doesn't have much built in to do this, so I'm not sure how to go about doing these things. It can't be uncommon to have it directly connected to the general LAN, so I'm hoping there's a straightforward helpful answer to the above 3 approaches, so I can choose which works best for me and figure out how to combine them if useful.

In ESXi 5.5, the host config has a number of similar sounding settings related to swapfiles and caches of various types.

I understand the principles of write-back/write-through disk cache and swapping VM swap files to the host instead of their remote VMDKs, and I'm reading the docs, but it's still pretty confusing which of these is which, and how they inter-relate within ESXi.

For information, my system has a single standalone host with 96GB RAM and three datastores - a local boot store, an iSCSI main store, and a 250 GB NVMe SSD for swap/cache use.

A clear explanation of the differences and how they work together would be really useful right now :)

I'm using ESXi 5.5 and vCenter 6.5 with vSphere Update Manager. I successfully staged about 20 updates (new host deploy) and set it to remediate. I know this can take a long time, but the percentages are static for many hours. Is there a way to check (via command line or otherwise) if it's actually hung or if it's doing anything?

It would be nice to also get a more realistic idea about how much it's got to go, but that isn't essential, I mainly want to be sure that I can leave it be, and it's working properly on the patches, even if slowly.

On Windows I can watch update-related files being accessed and update-related processes I/O counts, in task manager/perfmon, as they do their disk and process I/O, and on Linux or BSD I can use various well-known tools (but they aren't included in ESXi). What's the equivalent with command line or other tools, in vSphere?

I am trying to be clear about how zfs dedup handles the case where some (but not all) datasets in a pool are deduped, from a dedup table/RAM impact perspective. I found this quote from the FreeBSD mailing list in 2012:

"Note that only file systems that you enabled dedup for will actually participate in dedup. File systems that have dedup=off won't go through the dedup."

As an example, suppose we have two zpools, A and B. Pool A has 4 datasets containing 21 TB of data:

• Datasets #1 and #2 each contain 0.5 TB data with dedup on
• Dataset #3 and #4 each contain 10 TB data with dedup off

Pool B has one dataset containing 1 TB data with dedup on.

It's clear that the dedup functionality applies to the entirety of each pool. What isn't clear, is whether the RAM impact of dedup is based only on the deduped datasets? In other words, all other things being equal, will the dedup table size and RAM impact be similar for pool A and pool B, or far larger for pool A than pool B?

I think the dedup table has to be similar for both (set poolwide but no impact on size from any non-deduped datasets), mainly because if it was much larger, it would be equivalent to forcing dedup on the whole pool not just specific datasets. However it isn't clear to me whether this is actually so.

I'm virtualising a small setup of half a dozen mixed PCs and small servers. The aim is to consolidate hardware resources (one high spec server + thin clients rather than 8 medium-to-high spec individual devices which aren't all used at the same time), to enhance mobility and session start-stop-suspend-move, to allow resource sharing (inactive machines can be suspended and the resources used for other things, rather than having dedicated PCs left idle, also the occasional heavy workload can be "averaged out" across VMs and doesn't needed every machine able to handle it), and to allow session snapshots on all devices. I've been trialling on a small scale using VMware Workstation for a year or so to see if the approach benefits the setup and it very clearly does, enough to move to a VM server more fully.

Everything is straightforward, but my confusion is about the video handling aspects and how they interact (VDI, SVGA capable video cards, PCoIP/RDP, and pros+cons between the more generalist ESXi vs the more specialist View/Horizon for the desktops). This area is completely confusing me and holding me back.

USAGE AND REST OF NETWORK - The active VMs at any given time would be a mix of about 3-5 Windows desktops and a mix of 3-4 small internal *nix servers (shell tinkering, tiny radius server, etc). The desktops are mostly used for desktop and "productivity" work on Windows 8.1/10 (heavy duty multitasking on Office suite, browsing, coding/development, video viewing, small amount of Photoshop now and then), but the desktop "windowing" use can be intense and multitasked. Most modern software can also use 2D hardware rendering if available to offload desktop GUI compositing and controls in applications. The servers are all light load *nix. There's a separate robust file server + offsite replication in place with enough capacity/hardware spec to support 1 or more VM servers, and 10G LAN for the file server/VM server link.

My focus for this question is the graphics handling aspect. I'm reluctant to rely purely on soft (CPU) desktop and graphics handling due to the excessive CPU load it imposes even for moderate use, so I'd like to plan and spec for a bit beyond that. I want some flexibility in video resource sharing, and the usage varies, so if CPU alone isn't enough, I'm really looking to a VSGA or similar style solution, not dedicated-card-per-VM passthrough solutions.

My question is borne of ignorance, openly admitted. I don't know what options make sense to consider in relation to graphics/desktop use. Virtualization is usually discussed in terms of a single purpose and on a larger scale, rather than a heterogeneous mix like this is. My points of confusion are things like these -

• Are View/Horizon so specialised that I can only run VDI on them, or can I also use them to run general purpose VMs hosting the non-desktop servers, as I would with ESXi? Conversely if I use standard ESXi VMs to host the desktops, how much is ESXi missing compared to View/Horizon in desktop-GUI-related optimisations that can't be made up for in other ways?

• I want to accelerate VDI and offload much of it from the host CPU. But does this force me down the View/Horizon route, or if not, what hardware would be relevant for ordinary ESXi + client? (VSGA capable card such as quadro 6000 or would I need GRID (I'd like too avoid GRID due to additional cost over VSGA)? Teridici thin client cards as well, or not?)

• Does moving to VDI and any perceived latency (mostly LAN but occasionally remote) force my hand in choosing the remote desktop/PCoIP protocol and system, or thin-client hardware, and my choice of VM system and video related hardware?

I've done a fair bit of troubleshooting but I'm completely at a loss what could be going on.

Hardware / platform

• Supermicro X10SRi-F motherboard
• EVGA 850W G2 PSU (one of the highest rated for power quality at Jonnyguru.com)
• 128GB Crucial DDR4 RDIMM
• LSI 9211-8i PCIe HBA flashed to latest P20-IT (from LSI/Avago website)
• 8087-to-quad-SAS connector cable (new: see this picture, the kind of cable where the power side accepts a SATA PSU connector)
• Seagate 6TB SAS drive (new: ST6000NM0054)
• Various other Seagate 3TB - 6TB SATA drives (test purposes)

Problem / troubleshooting so far

This is a new server being set up, so all components are new, although some have been tested already before now.

On booting, the HBA didn't recognise or report the 6TB SAS drive (either via the main BIOS or via its own OROM -> SAS topology) and the 6TB drive was cold and not spinning up. No other drives were connected. The rest of the system works fine, so on the face of it the issue is limited to one or more of bad HBA, bad cable, or bad drive.

Troubleshooting steps so far:

1. Connected 6TB SAS drive using different terminator on the quad cable, and the quad cable to both 8087 ports. No change - implies the issue isn't one specific terminator or port.
2. Connected various Seagate 3TB-6TB SATA drives using same cable (same manufacturer and similar modern range to eliminate subtle compatibility issues if any). All recognised, reported, and spun up perfectly as normal on boot, on both 8087 ports and on all 4 terminators, and over multiple reboots - implies HBA and cable both work fine, at least for SATA. (Would be odd if they worked perfectly for SATA but not SAS.)
3. Kept identical connections but replaced SATA drives by 6TB SAS drive, not changing anything else. As before, 6TB SAS drive wasn't recognised or reported by HBA, and wasn't spun up.
4. Tried exactly the same with a different card and platform m- LSI 9260-8i RAID controller on an ASUS based desktop. Again all SATA drives immediately recognised and spun up, but 6TB SAS drive isn't/doesn't.
5. Reluctantly concluded that however unlikely, the most likely issue was 6TB SAS drive DOA and RMA'ed it. ("Reluctantly" is because I've never actually had a DOA before, the drives are usually reliable, and if it is dead then by far more usual/expected would be that it's at least recognised but non-functional. I just couldn't figure a more likely issue than complete DOA.)
6. Just received the warranty replacement - and getting exactly identical symptoms with the replacement as well: (a) When the 6TB SAS drive and any SATA drive are connected to 2 terminators and the system boots, the SATA drive is immediately recognised, reported and spins up, while the 6TB SAS drive stays cold and still. (b) When the 6TB SAS and any SATA drive are connected to the 9260-8i RAID card in the other ASUS desktop the SATA drive is likewise immediately recognised but the 6TB SAS drive stays cold.
7. As a last step, re-re-read the 9211-8i HBA user guide in case I missed anything first time, and re-checked the BIOS. Can't find anything that would seem to explain this, or any statement that SAS drives will not be recognised unless/until <some action/content>.

Didn't really believe it was DOA first time. Definitely don't believe it's a DOA now. But if not, then what is it, and what can I be missing?

I've tested everything (AFAIK) in the component chain, the HBA just doesn't have much OROM interface that can go wrong, or any options to recognise SATA/SAS/both, or anything like that, and the main PC/server in both cases just leave detection to the HBA/RAID card. I've tested on two completely different platforms, with two different models of controller cards, with SAS vs SATA drives, and I'm utterly stumped.

(Note: I'm slightly limited as I'm starting to transition from SATA to SAS, with the intent being to replace SATA by SAS as they wear out, so at the moment I don't have any other SAS disks or cables to test with, which I would otherwise have done too. But I think I've probably covered that by testing the cards+cables while varying SATA/SAS)

Updated for more accurate title to help others, now more info obtained. See answer.

I have an LSI 9211 HBA and an Intel 910 SSD. I updated the firmware on the HBA, using LSI's usual sas2flash utility. I removed all other LSI/HBA cards first.

Unfortunately it now looks like the SSD also has the same controller internally, which I had no realistic way to know in advance, and now Intel's SSD toolkit won't recognise the SSD so I can't figure how to put it back as it was. Sas2flash -listall shows the SSD with P20 LSI bios.

What do I do?

I'm building a 3 server setup (FreeNAS, FreeNAS replication, ESXi) for a small network with a handful of users and a couple of "power users". The aim, briefly, is to migrate from multiple machines and windows file shares to more centralised and better resource usage, and higher quality and reliability.

The hardware is adequately specced (ESXi on octocore Xeon + Supermicro, and FreeNAS on quad fast Xeon for CIFS with a ton of RAM and Intel SSDs). But I'm expecting a considerable learning curve because of the differences of what I'm moving them from and to.

I was intending to run my file shares as NFS (for ESXi) and CIFS (mostly Windows + a couple of Mac clients). Then I came across VAAI and ODX and thought "wow", as the workloads look like they would really benefit from them. But as with any iSCSI, they'd replace NFS by one initiator/one target per share only unless one really loves corrupt data. So I'm wondering whether I have to give up on the idea of these nice shiney protocols, or whether I can get at least some benefits they offer if not all of them:

• If I ran ESXi (but not CIFS) as iSCSI, then the question becomes how to copy VM store files off the iSCSI pool if I wanted to, which will be useful, or indeed, how to upload extra files onto the pool. I don't want constant access but I would like to feel I have a way to access the iSCSI based store either read-only, or read/write when the usual initiator isn't connected.

• Is there a way to maintain a second iSCSI initiator running as read-only within FreeNAS itself, so that I can use FreeNAS itself to pull data that I know is idle (old files etc) from an iSCSI target on FreeNAS, to a separate machine?
  
  Ideally it would mean that when ESXi is disconnected, I can locally copy files to/from/between the VM store and an NFS share, or make the files on the VMFS store accessible and shared read-only to other machines

I understand that iSCSI doesn't 'know' about files, and that a file and its metadata which was in use wouldn't be guaranteed to be served correctly to a second 'interloper' read-only initiator if it was changed halfway through the request or something, but I'm thinking here about reading from old ZFS snapshots or when the main initiator isn't connected, or files known to be idle - situations where there may not be the usual risk of conflicting instructions and corruption, and for a small setup it would be helpful.

I'm planning a small ESXi standalone server to host a number of VMs I use here. I haven't decided what facilities to use from the wider vSphere system. The underlying VM storage is local HDD RAID using enterprise HDDs and LSI megaraid, with the LSI card's onboard battery backed ram +ssd caching systems enabled.

My concern relates to data corruption and bit rot in the VM store over time - I don't really know what my options are, and I'd like to be sure that stored VMDKs and snapshots, and other VM files, don't get corrupted over time and can be set to be periodically scrubbed and any bit-level corruption (within reason) detected and repaired.

As background, for casual desktop use, I've tended to use RAID 1 (mirroring) rather than higher levels (reasons: fast read speeds, complete portability of drives without tie-in to specific brands or cards, no disruption if a drive fails). For my file server I use ZFS on a mirrored volume. But ESXi and VMware's suite use their own data store design for local storage. So I don't know how resilient against silent corruption, a setup would be "out of the box", especially when it holds many TBs of large files that might sometimes only be accessed years later, and with a local store rather than a dedicated separate storage system. I also gather VMFS uses a journaled filing system but not one with the self correcting capability of ZFS.

Are the inbuilt capabilities of ESXi (and if necessary other parts of their suite) sufficient to protect against routine data corruption concerns? If not, what are my options for peace of mind?

Update

@mzhaase - I didn't feel confident about passing through to a second server that would act as a file store, because then every file access and snapshot has to be done remotely across a LAN or a second device and even if 10G was used (which is still cost prohibitive for most home setups) the slowdown would be a major concern.

Part of the whole reason for getting this specific card is to get true cache-on-write for speed, so that bulk writes or rollbacks are less likely to slow everything down by "chugging" the main HDDs, which should be helpful whatever the file store location. Issues with latency impact sound like they would also happen with any remote data store, whethet a server appliance or a home build such as a second FreeNAS box (although if I had to choose, I'd use a second FreeNAS).

I'm perhaps overlooking using a dedicated NIC port and multiple parallel 1Gb ports as a way round this, but latency and traffic implications for snapshots and rollbacks are a big concern. I'm also possibly overlooking running a FreeNAS VM on a small dedicated disk, which services the main VM store array off the raid card as a passthrough device, which keeps it local. (Meaning that ESXi can boot and can load the FreeNAS VM off one disk, once that's running it can act as a ZFS based file server for any other VMS with - hopefully - low latency). But running the file server virtualized might increase latency more than keeping it local would reduce it, and latency and disk bottlenecks are already an issue I'm trying to overcome.

I will however look up the LSI card info, and - can you install file integrity checking/repairing software on the underlying ESXi platform itself to check and repair VM files? I didn't know that. And would iSCSI be that much of a latency-killer to make a remote store usable? Once a VM is up and running, how much does access speed/latency to the VM store affect the running of ESXi or other VMs currently running on it?

As the question says. There's a lot of threads around on NFS vs. Samba/SMB, but a lot of them are outdated or refer to old security models, or just give a one-line "use SMB with Windows".

Both modern Windows clients and modern *nix file servers can handle both of NFS or SMB/Samba. Whichever protocol is chosen, one will be using a "native" protocol and the other won't. So in mixed environments (*nix server/Windows clients) it's not as simple as "for X use Y".

So I'm interested in actual pros, cons and experience. The few threads I can find that cover modern incarnations of these protocols suggest as possible differences:

• Threading? - Samba is single threaded (for example FreeNAS forum posts suggest that a single core at higher speed is better than multi-core for Samba serving Windows clients, for this reason). So presumably during congested periods or with high or complex levels of demand or locking/unlocking activity, this might be a factor;

• Single file performance? - some threads suggest NFS is more efficient or serves faster with smaller files <= a few MB (which is many/most of them?) and comparable for larger files.

• Security? - some threads suggest that NFS doesn't implement as effective a security model as SMB, or at least hasn't historically.

What's the current state of play on this for a modern *nix file server with Windows 8.1/10 x64 clients, if NFS is also enabled on the clients?

At the moment I'm managing a small home office setup - a few PCs and a soft open source x86 *nix based router. I'd like to move traffic scanning / inspection onto an edge device.

The motive is to increase the barrier for undetected malware and suspect traffic, have the main traffic scan separated from any end-user devices, and increase certainty by scanning at the LAN gateway.

At present they use consumer antivirus which effectively scans SSL by creating a fake CA on the PCs. It's a decent brand AV and works but has issues. If it's not too hard they'd like the reassurance that it's approached as an enterprise issue using 'proper' means (their term) rather than PC-based consumer software.

It's clear that malware is increasingly moving to encrypted traffic which - if not scanned - can be indistinguishable from normal https traffic. I can't filter based on endpoint (acceptable endpoints too wide to whitelist, and even 'good' sites can inadvertently host malware) and you don't get much other solid data for encrypted traffic on usual ports.

There's trust and good dialog with the people using it (3 partners, no employees), and they all want this as well as well, so no ethics/consent/privacy issues arise internally. The traffic is low enough (web/email) not to need a high-volume or specialised UTM.

The practical issue in ssl scanning is that SSL is precisely designed to detect and prevent MITM (which is what this is), and therefore scanning needs to rewrite packets with its own CA. That's a pretty bad idea and "breaks" the cert chain so it's best not to... except the only alternative seems to be no scanning at all of any SSL/https which is seen as more risky and potentially worse. Like democracy, it's the worst solution except all the others ;-) For example, as the browser or local device wouldn't be able to verify cert chains, the edge device would have to do this for them, and block or warn if not correct. But in theory if the external cert chain was reliably checked before discarding, then MITM inspection at or next to the gateway could be a safer bet than not scanning SSL at all.

I've looked around and this does indeed seem to be "out there" and used in the enterprise, but usually as a standalone device.

Is there a recognised way to do it without a specialised device?

Update 1:

They are older people, who didn't grow up with computers and still despite their efforts prone to clicking first and worrying after, or the alternative of phoning every time they are in doubt. As with other SMB they also use their work PCs for shopping/browsing/social etc, which adds to the risk - and they know it. SSL scan has caught quite a few things that apparently weren't blocked by the PC based blacklist setup, including apparently legit sites or "nobody knows how". Probably quite sensibly they want to have more comfort going forward even if blacklists would catch "most" things. I think this might be their best way, if I can work out whether it's feasible and how I might do it. If the main objection is "not worth the hassle" then I'd still like to research it so they can think about cost/benefit. But so far all I can find is high-end devices, I can't find if software/FOSS solutions exist for this function, even though one would expect them to. But I haven't been able to find any if so, hence the question.