I'm looking for ways to encrypt Microsoft SMB data traffic between a client and Windows server or filer appliance. This is not about encrypting the authentication but the actual data transfer.
Is IPSec from the client to a Windows Server my only option?
Windows Server 2012 includes a new SMB (3.0) transport encryption option. http://blogs.technet.com/b/filecab/archive/2012/05/03/smb-3-security-enhancements-in-windows-server-2012.aspx
CIFS/SMB doesn't have any protocol-level encryption options as of SMBv2, so you're stuck encapsulating the traffic in an encrypted envelope. Which in all practicality means a VPN of some kind. Be it IPSEC, SSL, PPTP.
There's vague references to CIFS/SMB the TLS or SSL on the internet, but nothing obvious and not from Microsoft, so I'd assume it's not all that possible, common, easy or all three.
In that case, if you can't encrypt the protocol, you'll have to do encryption at a lower level, which means a VPN of some form. IPSec is just one way. You could use an encrypted pptp tunnel or something similar.
If you're not talking about adding other computers or software to the mix then, yes, IPsec or the built-in VPN functioinality in Windows is the only built-in way to encrypt CIFS/SMB traffic between a Windows Server computer and a client.
You could, obviously, stick hardware-based encryption devices between the client and the server (VPN gateways, routers doing IPsec tunnels, etc). You could install third-party VPN software onto the client and/or server, as well. You can do anything you want to encapsulate, encrypt, or otherwise slice-and-dice the packets once they're on the wire, as long as by the time they make it to the server computer they're decapsulated, decrypted, and glued back together.
A simple solution is to create a transparent SSL tunnel with stunnel by installing the client on all end points
As for Windows 2012 and SMB 3.0, encryption is now supported. But disabled by default. You can activate it :
To enable encryption for an individual share, run the following on the server:
To create a new share with Encryption turned on:
To enable encryption for the entire server, run the following on the server:
Beware that clients not supporting SMB3 (prior to Windows 8) won't be able to connect using this version of the protocol.