I have a site-to-site VPN set up with two SonicWall's (a TZ170 and a Pro1260). It was suggested to me that turning off encryption (so the VPN is tunneling only) would improve performance. (I'm not concerned with security, because the VPN is running over a trusted line.)
Using FTP and HTTP transfers, I measured my baseline performance at about 130±10 kB/s. The Ipsec (Phase 2) Encryption was set to 3DES, so I set it to "none". However, the effect was opposite -- the performance dropped to 60±30 kB/s, and the transfers stall for about 25 seconds before any data comes down the line. I tried AES-128 and the throughput went UP to 160±5 kB/s. The rated speed of my line is 193 kB/s (it's a T1).
Contrary to what I would think, stronger Ipsec encryption seems to improve throughput. Can anyone explain what might be going on here? Why would no encryption cause poor and highly variable performance, and cause transfers to stall? Why does AES-128 improve performance?
AES is faster than 3DES because of the algorithm design (number of rounds, etc.), not because of the key size/encryption strength. I don't know much about SonicWall products, but I'd assume that the firewall product should be able to pass traffic at line speed for a T1, so there may be some issues there.
I'm not sure why you'd see performance that's worse when you turn off encryption, but if you don't need encryption, as Antoine Benkemoun said, you don't really need IPSec, especially not ESP (tunnel mode).
I don't know your exact setup, but one common explanation for worse performance turning off encryption behaviour is that you don't only use encryption, but also compression. Turning off both encryption and compression significantly reduces performance, especially if your packets start going over the MTU and getting fragmented frequently. Did you check so you don't usually run with IPComp?
You should also check if there is anything weird on the line when you turn off the encryption. I recommend sticking a sniffer such as wireshark in there and having a look both with and without encryption turned on. It should give you a much better idea what is happening.
The speeds we are speaking about here are so slow that almost any hardware can do the encryption without noticeable delay, so I would guess the encryption overhead is a red herring.
My first guess would be MTU mismatch, when no encryption is present. when there is no encryption you may have to manually set the MTU value for the interface to match the network it's on. This mismatch would cause significant fragmentation which would translate into lower speeds.
Encryption is a tedious process that requires CPU time. This is even more true in the case of 3DES which is DES done three times. Disabling it will give you a little performance boost by deleting all the overhead headers and encryption. The difference should be rather minimal (10% at most I'd say).
It surprises me that you can disable encryption in IPSec. Anyways, I am pretty sure that it's not meant to be used like that.
If you want to have a tunnel without encryption, you shouldn't be using IPSec but PPTP or L2TP.
MTU shouldn't be a problem in this situation, but you should take a look anyways.
I'm not familiar with the Sonicwall VPN, so I'm just guessing but it looks to me like it may be a QOS thing. Just maybe the unencrypted traffic is falling into the "all others" category, which often has a low setting.