The OCD part of me wants to delete all of the groups that don't have any members. The part of me that doesn't want to break things says leave them alone. Right now I have all my real users in a specific OU.
I've left these accounts in the Users and the built-in groups so far - What would break if I delete the built-in groups?.
You don't actually ask a question, but I'm assuming that you're wondering about deleting the built-in groups like
Account Operators.Instead of asking yourself "What will break if I do this?" you should be asking yourself "What will I gain?" The answer is that you will gain absolutely nothing, so there's no reason to do it. This is a classic case of asking "Why not?" instead of asking "why?"
If you really want to see what happens (or if it will even let you), set up a test environment and test. Realistically as others have pointed out, there is no gain to doing so and I can't imagine your AD being so inundated with objects that the few built-in groups/ users really make a difference. If you have your AD setup with proper OU structure you shouldn't really see these objects on a regular basis...
As far as what it will affect, read up on the AdminSDHolder object. There is a process that runs that checks the ACLs of objects that are members of Protected (built-in) groups against the AdminSDHolder object and overwrites changes made to the objects with that of the AdminSDHolder object. If you were somehow able to delete those groups, I imagine you'd have problems on your PDC Emulator FSMO Role Holder, where this background operation runs (every hour).
Additionally, those built-in groups logically correspond to the local built-in groups on your DCs.
It's probably best that you don't try deleting them as they are there for a reason.