Is there any reason to NOT have a swap file on CentOS (DigitalOcean)

Is there any reason to NOT have a swap file on CentOS

Yes:

• You have some type of horizontal scaling in place which increases your number of servers based on their memory usage, thus eliminating the possibility of swap being used.

• You have a specific performance requirement that means you cannot allow your program to use swap for processing as it is significantly slower than memory, so rather than use swap, you choose to monitor memory usage closely and increase memory as needed.

Will my system just die the first time it gets seriously loaded up with users and memory demands?

If you have no idea how much load your server is going to experience, and you don't have enough memory, and you have no swap space, your application will stop.

Will swap save me from that, but only by imposing an unpleasant performance hit?

Correct.

What exactly happens when the application stops? Which application stops?
If I had 3 processes and 1 was leaking memory, would all 3 die or the 1 that was leaking die?
Would 1 force the other 2 to die?

Think of it like this:

If you make a program and tell the program to fill up a variable with an infinite amount of integers, very soon that application will encounter a memory error and it will exit.

It's the same. Whichever application is trying to access more memory for it's operations is going to fail. So if you have 1 byte left and your application creates a 32 bit integer, you need 2 bytes, the operation fails because there's not enough memory, and depending on how well the application handles memory errors, it will either fix itself or exit.

The same goes for 3 applications at once. If by coincidence they all request more memory when the system has only 1 byte of memory left, they should technically all fail at once.

Somebody can correct me on this if they see any errors.

Will swap save me from that, but only by imposing an unpleasant performance hit?

Check out this page regarding the swappiness setting: http://www.scottalanmiller.com/linux/2012/09/03/controlling-swappiness-in-linux/

You can change your swappiness so that your system isn't using swap space too often, and thus not taking as much of a performance hit.

First install sysstat (if not already installed) and monitor the resource usage for few days. Run "sar -r" and monitor the memory usage frequently. Most of the time memory usage will be near 90%, watch the "kbcached" and "kbbuffers" column. If they are using most of the memory, then you are fine as it means the applications are cached. But if those values are less, it means caching is less due to more actual memory usage which might also indicate your applications needs more RAM. So you can increase the RAM or setup swap space.

Now to swap, I believe digital ocean disks are on SSD. If that is the case, setup a swap file and you should not mostly see any performance hit since SSDs are faster than HDDs.

HTH.