Mail Server running well but incoming traffic kills my internet connection

I am running a Postfix - Dovecot - MySQL mail server on my personal Ubuntu dedicated server which I'm hosting it in my home.

Everything is working amazing except the fact that I am getting loads of incoming connection that I can see in /var/log/mail.log and these connections are coming from different IPs all over the world.

It looks like I'm getting DDoS'd and when the connections are kicking in, my internet connection drops for few seconds then it goes back on. It looks like someone is trying to take down my mail server.

Can anyone give me an idea how can I stop this? How can I improve the security of my Ubuntu dedicated machine? Any other advice that you guys can provide? I'm kinda new at this and Google didn't solve my problem.