I just know there is two kind of firewalls. Stateless and stateful. It's hard to determine what kind of firewall I have to use. Currently I have to run firewall within same machine runs services, Basically I want stateless because of its less resource consumption. However if it is not sufficient for security, it's meaningless. I'll run HTTP, SSH, NFS (only over SSH), and some custom made server on several TCP/UDP ports. Should I use stateful firewall?
(edit) Maybe the question can be assumed as 'Should I use stateful rules?'.
These days, there is a very simple answer to this question: all proper firewalls are now stateful.
stateful rules greatly simplify design of the firewall policy and improve security. HTTP and SSH work with since tcp sessions that use single port. Any stateful firewall can handle this and all you need to do is write one rule to permit original packet that opens the session and another rule to permit reply packets that match known states. In case of iptables , the latter looks like
(assuming the firewall runs on the server, hence chain INPUT).
More complex protocols, such as for example ftp, may require several sessions and state engine in the firewall needs to be able to track these separate sessions to match packets. Usually stateful firewalls support most of the popular protocols but it is unlikely that they support custom protocols. The problem with custom protocols is not that they use unique tcp or udp ports, but that they may use multiple connections that are dynamically created and torn down. The firewall can dynamically open and close "holes" for the ftp data channel because it can track and interpret commands flowing through the fto command channel. If the protocol is proprietary and unique, the firewall won't be able to interpret it and to make it work you'll have to write stateless rules and try to make them as tight as the protocol allows.
However if the protocol uses single tcp connection, then stateful rule will work just fine.
The only time you might possibly need a stateless firewall nowadays, is for better speed performance.