Does nftable drop invalid state connections if I only allow established and related state?

TL;DR

If you never use the reject statement in any rule, it doesn't matter much to drop or not drop invalid packets. If you do use a reject statement, you should most certainly drop invalid states.

Explanation

In many cases, dropping an packet in invalid state isn't mandatory: the receiving network stack when receiving such packet will itself ignore it. Anyway one can imagine there are bugs, and that when the system is a router, doing this will protect systems behind if their implementation can't cope correctly.

There's one case where it becomes mandatory to drop invalid packets: that's to not reject them. This has been recently added to iptables' documentation:

Warning: You should not indiscriminately apply the REJECT target to packets whose connection state is classified as INVALID; instead, you should only DROP these.

Consider a source host transmitting a packet P, with P experiencing so much delay along its path that the source host issues a retransmission, P_2, with P_2 being successful in reaching its destination and advancing the connection state normally. It is conceivable that the late-arriving P may be considered not to be associated with any connection tracking entry. Generating a reject response for a packet so classed would then terminate the healthy connection.

So, instead of:

-A INPUT ... -j REJECT

do consider using:

-A INPUT ... -m conntrack --ctstate INVALID -j DROP
-A INPUT ... -j REJECT

It's exactly the same behaviour when using nftables, since both rely on conntrack. Tools like firewalld always got it correctly and do drop invalid states before doing any reject in their generated rules. Users not knowing about this might not have done so when manually building rules.

Just rewriting above with nftables in mind:

Netfilter's conntrack does a lot more checking than people usually think. For example, with TCP flows, it also checks the validity of the sequence numbers with regard to the current TCP window (PDF). When a packet gets delayed (perhaps temporarily routed over Internet to an alternate slower path) it's possible it arrives after such packet got already retransmitted and ACK-ed successfully, thus becoming now out of window and tagged as invalid state. Now consider what happens with these rules (in a type filter hook input chain of a client's firewall hosting no service):

ct state established accept
tcp reject with tcp reset
reject

Such invalid packet (eg: occuring during a lengthy and huge download) will trigger a TCP RST, aborting the download. Worse, if the rule was not on the client system but an equivalent rule was on a firewalling router's forward chain, the administrator of such client system would have no clue about what happened (except maybe seeing some retransmits before the connection fails in captured traffic).

Now with this:

ct state established accept
ct state invalid drop
tcp reject with tcp reset
reject

If you drop such invalid packet, nothing happens, download goes on unaffected. With no firewall rules at all that's what would have done the TCP stack: ignore such packet, not react over it with a TCP RST.

If you don't explicitly drop invalid connections, then nftables will go through the following rules. If there is no matching rule, it will apply the default policy of the chain (which by default is 'accept').

So it depends on what your following rules and default policy are. But if say, one of your rule is to accept incoming packets to port 22, you really want to drop invalid connections first.