Server problems, running out of RAM, really high load average

Congratulations, you've managed to use nearly all of your swap space.

The first obvious problem here is that you went very deep into swap. This is probably what's causing the system to thrash so hard (lots of time spent in system, I/O wait and software interrupts).

First thing to do is to reduce the number of Apache processes that are running. You don't need that many for a small site, and it's just going to throw you deep into swap and kill your performance...which is what already happened. I would recommend you start very small and increase when it becomes necessary. An example:

StartServers            1
MinSpareServers         1
MaxSpareServers         2
MaxClients              5

This limits you to only serving 5 simultaneous requests (everyone else has to wait in line). If at this point you get warnings from Apache about running out of servers, and you still have RAM to spare, then you can increase them, but you are eventually going to run into a point where your VPS simply hasn't got enough RAM to handle all your traffic. At that point you should upgrade the VPS.

Before anything, based on the screenshot you have posted with htop output, it seems you have 512MB of RAM on a site running WordPress? I have never seen WordPress happy on servers less than 1GB of RAM. Maybe if you are running a test or development site, 512MB is adequate, but for a production site you need 1GB of RAM. That is the root of your problem. That said, here are some ideas to help you squeeze out better performance from the setup you have:

I have no idea how to figure out what's going on. Is Apache or MySQL not tuned properly? Maybe someone is attacking the server with repeated hits (how would I know?). I installed htop but even if I saw that Apache or MySQL was eating up a ton of resources, how would I figure out why?

First, I would not panic about an attack happening. The reality is that your server is probably just taking on a high load of legitimate traffic, but the server itself is not configured/tuned for your usage. Of course bad configurations can bring your sit down during DDoS (Distributed Denial of Service) attack, but it can all make your life miserable when nice/normal traffic suddenly shows up in high numbers.

I posted a nice list of items you can review to improve your L.A.M.O. stack performance on another similar question, and will repost here for your reference:

1. Properly configure Apache: Apache is a good piece of software, but right out of the box it’s a memory hog. For example, I believe the default is to allow 255 connections per second? I can assure you most simple sites barely get 40 connections per second on a good day. So adjusting Apache to be realistic to your traffic will help. Also, there is a KeepAlive setting in Apache that works great! But out of the box, I believe it’s set to a MaxKeepAliveRequests of 100 which is fairly nuts. I usually set this to about 30 connections with a small KeepAliveTimeout of 2 to 3 seconds. The key is to have the KeepAliveTimeout to match speed it takes for an average page to download with a little bit of room for overhead/slowness. So if a page loads in 1 second, do a KeepAliveTimeout of 2 seconds.
2. Review the code for your WordPress sites for potential bottlenecks: Concentrate on the PHP core of it & clear up what you can. Look out for excessive MySQL calls & file system calls. This is where you will be able to make the app fly! Also, check the memory_limit in your php.ini and make sure it’s not higher than necessary. The default is 64M, but in many cases that can be lowered to 32M.
3. MySQL tuning or moving it onto it’s own server: After writing about MySQL above I realized you might be hosting your MySQL instance on the same box. Look into optimizing MySQL performance by running a script like MySQL tuning primer. Without tuning, MySQL will eat up all resources & big the system down. With tuning, MySQL will run better/faster & resources can be freed for other purposes. Also, consider moving your MySQL DB to a standalone server. You might have to learn how to properly network & firewall the server to allow your servers access but protect against hackers, but the performance benefit will be great.

Regarding the MySQL tuning, that is something that can take a few weeks to nail down at the beginning. The reason being tuning scripts are based on real traffic MySQL sees. So you basically make your site live to the world, wait 2 days (at least), run the tuning scripts & then wait a few more days to tune some more. After a week or so you should be able to tune MySQL to work as well as it can with your setup.

First install these plugins: super cache (htaccess mode), wpbase, widget cache. Wordpress is known for having these issues. IF that doesn't help, it might be a memory leak on one of your themes (especially if mobile view is enabled).

Post your access and error logs.

I strongly recommend working with nginx instead of apache.