I would like to ask you what is performance of iptables.
Let's say that I whitelist and blacklist some IP addresses, I block about 10 000+ IP addresses, so every IP is like new rule.
I don't know how fast it is, because let's say that someone DDoS me width bandwidth of some Gbps, can iptables handle it?
CPU: 1 core of Intel ® Xeon® E5-2650L v4
And is possible to make iptables faster? For example that I would "bundle" all these IP addresses into 1 rule.
Iptables can handle anything aslong as you give it enough resources to work with, and it will probably outlast your physical connection anyway.
This is called address summarization, and yes you can do that.
Iptables is very effective, if you would like to test this i would probably advice allowing everything, put the machine on a private network and setup a loader. What is system process load at 1 Gbp/s? what is it a 2, 3, 4 and so on? And from their get an idea of what Iptables can and cannot.
If you truly are under a denial of service attack, when traffic gets to the host it is too late. They already have used your bandwidth. See our canonical question about DoS and DDoS mitigation.
Use the host firewall to prevent a small number of annoying clients from scanning your applications. Use service providers with DDoS migrations services, CDNs scrubbers and large bandwidth, to counter large attacks.