There is a BAN chain with this rule:
-A BAN -m recent --name ping --rcheck --seconds 5 --hitcount 3 --rsource -j RETURN
it will match when there will be 2 pings in 5 seconds but if i add remove addresses from ping list which didn't match this upper rule will never match why ?
-A BAN -m recent --name ping --remove
Since your first rule doesn't trigger until the third time it has been traversed and your second rule deletes the table entry entirely, the first rule will only ever observe what it thinks is the first packet from that IP address.
Do not use the "remove" rule, and the "ping" table will be managed just fine.
EDIT: Here is one way to use iptables and the recent module to have a different ban time than the time to become banned. I have used 5 pings in 10 seconds as the ban criteria and only 120 seconds ban time (just to make it easier to check).