This is a proposed Canonical Question about understanding and debugging the software firewall on Linux systems.
In response to EEAA's answer and @Shog's comment that we need a suitable canonical Q&A for closing common relatively simple questions about iptables.
What is a structured method to debug problems with the Linux software firewall, the netfilter packet filtering framework, commonly referred to by the userland interface iptables?
What are common pitfalls, recurring questions and simple or slightly more obscure things to check that an occasional firewall administrator might overlook or otherwise benefit from knowing?
Even when you use tooling such as UFW, FirewallD (aka firewall-cmd), Shorewall or similar you might benefit from looking under the hood without the abstraction layer those tools offer.
This question is not intended as a How-To for building firewalls: check the product documentation for that and for instance contribute recipes to iptables Trips & Tricks or search the tagged iptables ufw firewalld firewall-cmd questions for existing frequent and well regarded high-scoring Q&A's.
In general:
Viewing and modifying the firewall configuration requires administrator privileges (
root) as does opening services in the restricted port number range. That means that you should either be logged in asrootor alternatively usesudoto run the command as root. I'll try to mark such commands with the optional[sudo].Contents:
-Iand-Aiptables -L -v -n1. Order matters or the difference between
-Iand-AThe thing to remember is that firewall rules are checked in the order they are listed. The kernel will stop processing the chain when a rule is triggered that will either allow or dis-allow a packet or connection.
I think the most common mistake for novice firewall administrators is that they follow the correct instructions to open a new port, such as the one below:
and then discover that it won't take effect.
The reason for that is that the
-Aoption adds that new rule, after all existing rules and since very often the final rule in the existing firewall was one that blocks all traffic that isn't explicitely allowed, resulting inOr equivalent in iptables-save:
and the new rule opening TCP port 8080 will never be reached. (as evidenced by the counters stubbornly remaining at 0 packets and zero bytes).
By inserting the rule with
-Ithe new rule would have been the first in the chain and will work.2. Display the current firewall configuration
My recommendation for the firewall administrator is to look at the actual configuration the Linux kernel is running, rather than trying to diagnose firewall issues from userfriendly tools. Often once you understand the underlying issues you can easily resolve them in a matter supported by those tools.
The command
[sudo] iptables -L -v -nis your friend (although some people likeiptables-savebetter). Often when discussing configurations it is useful to use the--line-numbersoption as well to number lines. Refering to rule #X makes discussing them somewhat easier.Note: NAT rules are included in the
iptables-saveoutput but have to be listed separately by adding the-t natoption i.e,[sudo] iptables -L -v -n -t nat --line-numbers.Running the command multiple times and checking for incrementing counters can be a useful tool to see if a new rule actually gets triggered.
Alternatively the output of
iptables-savegives a script that can regenerate the above firewall configuration:It is a matter of preference what you'll find easier to understand.
3. Interpreting the ouput of
iptables -L -v -nThe Policy sets the default action the chain uses when no explicit rule matches. In the
INPUTchain that is set to ACCEPT all traffic.The first rule in the INPUT chain is immediately an interesting one, it sends all traffic (source 0.0.0.0/0 and destination 0.0.0.0/0) destined for TCP port 22 (
tcp dpt:22) the default port for SSH to a custom target (fail2ban-SSH). As the name indicates this rule is maintained by fail2ban (a security product that among other things scans system log files for possible abuse and blocks the IP-address of the abuser).That rule would have been created by an iptables commandline similar to
iptables -I INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSHor is found in the output of iptables-save as-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH. Often you'll find either of those notations in documentation.The counters indicate that this rule has matched 784'000 packets and 65 Megabytes of data.
Traffic that matches to this first rule is then processed by the
fail2ban-SSHchain that, as a non-standard chain, gets listed below the OUTPUT chain.That chain consists of two rules, one for each abuser (source ip-address 117.253.221.166 or 58.218.211.166) that is blocked (with a
reject-with icm-port-unreachable).SSH packets that aren't from those blocked hosts are neither allowed nor dis-allowed yet and will now that the custom chain is completed will be checked against the second rule in the INPUT chain.
All packets that weren't destined for port 22 passed the first rule in the INPUT chain and will also be evaluated in INPUT rule #2.
The INPUT rule number 2 means this is intended to be a statefull firewall, which tracks connections. That has some advantages, only the packets for new connections need to be checked against the full rule-set, but once allowed, additional packets belonging to an established or related connection are accepted without further checking.
Input rule #2 matches all open and related connections and packets matching that rule will not need to be evaluated further.
Note: rule changes in the configuration of a stateful firewall will only impact new connections, not established connections.
In contrast, a simple packet filter tests every packet against the full rule-set, without tracking connection state. In such a firewall no state keywords would be used.
INPUT rule #3 is quite boring, all traffic connecting to the loopback (
loor 127.0.0.1) interface is allowed.INPUT rules 4, 5 and 6 are used to open TCP ports 22, 80 and 443 (the default ports for resp. SSH, HTTP and HTTPS) by granting access to NEW connections (existing connections are already allowed by INPUT rule 2).
In a stateless firewall those rules would appear without the state attributes:
or
The final INPUT rule, #7 is a rule that blocks all traffic that was NOT granted access in INPUT rules 1-7. A fairly common convention: everything not allowed is denied. In theory this rule could have been omitted by setting the default POLICY to REJECT.
Always investigate the whole chain.
4. Know your environment
4.1 . The settings in a software firewall won't effect security settings maintained elsewhere in the network, i.e. despite opening up a network service with
iptablesthe unmodified access control lists on routers or other firewalls in your network may still block traffic...4.2 . When no service is listening you won't be able to connect and get a connection refused error, regardless of firewall settings. Therefore:
[sudo] netstat -plnutor alternatively usess -tnlp.[sudo] nc -l -p 123oropenssl s_server -accept 1234 [options]if you need a TLS/SSL listener (checkman s_serverfor options).telnet <IP of Server> 123orecho "Hello" | nc <IP of Server> 123or when testing TLS/SSL secured serviceopenssl s_client -connect <IP of Server>:1234, before trying the same from a remote host.4.3 . Understand the protocols used by your services. You can't properly enable/disable services you don't sufficiently understand. For instance:
/etc/servicesdo not necessarily match with the actual service using a port.4.4 . The kernel packet filter is not the only thing that may restrict network connectivity:
getenforcewill confirm if SELinux is running.ldd /path/to/service |grep libwrapand the/hosts.[allow|deny]control files.5.
INPUTorFORWARDChainsThe concept of chains is more thoroughly explained here but the short of it is:
The
INPUTchain is where you open and/or close network ports for services running locally, on the host where you issue the iptables commands.The
FORWARDchain is where you apply rules to filter traffic that gets forwarded by the kernel to other systems, actual systems but also Docker containers and Virtual guest Servers servers when your Linux machine is acting as a bridge, router, hypervisor and/or does network address translation and port forwarding.A common misconception is that since a docker container or KVM guest runs locally, the filter rules that apply should be in the INPUT chain, but that is usually not the case.
6. Kernel modules
Since the packet filter runs within the Linux kernel it can also be compiled as dynamic module, multiple modules actually. Most distributions include netfilter as modules and the required netfilter modules will get loaded into the kernel as needed, but for some modules a firewall administrator will need to manually ensure they get loaded. This primarily concerns the connection tracking modules, such as
nf_conntrack_ftpwhich can be loaded withinsmod.The modules currently loaded into the running kernel can be displayed with
lsmod.The method to ensure modules are loaded persistently across reboots depends on the Linux distribution.
Iptables/Firewall "introduction"
A Firewall is basically a policy-based network filter. Linux firewalls are built around Netfilter; the kernel's network packet processing framework which is made of several kernel modules performing specific tasks:
Users configure the Netfilter framework to suit their firewall needs using iptables from the command line. With iptables we define rules that instruct the kernel what to do with IP packets when they arrive into, pass through, or leave our Linux box. Each Netfilter main process is represented by a TABLE (FILTER, NAT, MANGLE) on iptables lingo. They have several specific hook points on the network packet flow map where they are invoked by the kernel to perform their duties. Certain specifically located sequences of TABLE calls are generically called built-in CHAINS receiving the names of PREROUTING, INPUT, FORWARD, OUTPUT, and POSTROUTING. It is easy to remember if we associate a TABLE with a "type of process" and a CHAIN with the "location" on the network packet flow map where instances of those processes are invoked.
Since an IP packet is received on a network interface, or created by a local process, until it is finally delivered or discarded the Netfilter engine will sequentially test and apply the rules contained along the network packet flow map. At each block identified by a TABLE@CHAIN pair the user can add one or more of these consecutive rules containing an IP packet matching criteria and a corresponding course of action. There are actions (i.e. ACCEPT, DROP, etc.) that can be performed by more than one TABLE and other actions (i.e. SNAT, DNAT, etc.) that are TABLE specific.
i.e. when an IP packet arrives from a network interface it is first processed by the PREROUTING chain invoking the MANGLE table user defined rules if any. If there are not rules that match the current packet the corresponding MANGLE@PREROUTING default course of action or "policy" applies. At this point if the packet was not dropped the process will continue now invoking the rules of the NAT table at the PREROUTING chain (see the map) and so on. In order to facilitate the rules layout, users can also create their own custom chains and "jump" into them from different points of the map as they wish.
While built-in chains can have user defined policies of either ACCEPT or DROP packets, user defined chains have allways an unchangeable default policy of RETURN to the caller to continue the process.
Iptables commands
The iptables main commands populate the network packet flow map with the required processing rules.
The generic iptables rule can be written as:
# iptables <table> <Add/Insert/Delete> <CHAIN> <PKT_MATCHING_CRITERIA> <ACTION>It could be read like:
The iptables auxiliary commands complete the scenario setting default conditoins, listing rules, flushing rules, etc.
Iptables loads our commands into Netfilter engine at runtime, Netfilter inmediatelly enforces the loaded rules and settings but they are not persistant. Afeter reboot all previously loaded Netfilter rules and settings will be lost. For this reason there are iptables utilities that allow to save the currently active ruleset to a file and to reload it later.
Iptables Summary
Netfilter is an extremely flexible and powerful framework but there is a price to pay for it; Iptables is complex. From an user point of view certain terms like TABLE, CHAIN, TARGET do not really match very well the concept they represent and do not make much sense at first. The topic is long, commands seem to have an endless list of parameters. To make things worse there is not a single book that really masters Iptables. They mostly fall into two categories: "recipe book" or "manpage book". I think this introduction gives you a snapshot of the Netfilter/Iptables landscape plus the necessary dose of pre-digested manpage stuff. If you are new at iptables, after reading these paragraphs a couple of times you will be ready to read iptables examples. With some practice you will soon find yourself writting your own rules.
Firewalls
A firewall is mainly designed to dynamically permit or deny network traffic based upon a set of rules. At this point it is easy to understand why the Linux Netfilter/Iptables framework is perfect for firewall construction. Looking at the network packet flow map we find two particularly interesting spots on the FILTER table at the INPUT and FORWARD chains; We can decide there upon IP source address, IP protocol (UDP/TCP), destination port (80, 21, 443, etc), etc, if we ACCEPT, REJECT, or just DROP a particular IP packet. This is what a firewall does 80% of the time when i.e. protects a web server from unauthorized network requests. The other 20% of the time is manipulating (NAT, MANGLE) network packets.
Firewalls Scenarios
There are hundreds of different firewall layouts addresing different needs but 3 of them could be considered the most typical firewall scenarios.
I have written this for: http://www.vercot.com/~jeoss/howto/JeossEasyFirewall.html
Common issues with different protocols
DNS: DNS uses port 53 UDP by default, but messages that won't fit in a single UDP datagram will be transmitted using TCP instead (typically zone transfers and such) requiring port 53 TCP to be opened as well when you run a name server.
Email: Many consumer ISP's block SMTP traffic (or at least the default port TCP 25), making it impossible to directly receive or send email and their customers are forced to use the ISP's SMTP relay for all outgoing email and sometimes for incoming email as well. Relates to §1.1.
FTP: FTP is an odd protocol in the regard that two connections are used. The first is the control connection, by default an FTP server will listen on TCP port 21 for that. The control connection is used for authentication and issueing commands. The actual file transfers and things such as the output of a directory listing go over a second TCP connection, the DATA connection. In active FTP that DATA connection would be initiated from the FTP server from TCP port 20 and connect to the FTP client. Active FTP doesn't work too well with users behind firewalls and NAT gateways so it has mostly fallen into disuse. Most FTP servers suppport Passive FTP instead. With Passive FTP the FTP server opens a listener for the DATA connection on a second port, to which FTP client can then connect. The problem for a firewall is that the DATA port can be any available unprivilged port between 1024-65536.
In a stateless firewall that is typically resolved by restricting the number of passive ports that the FTP server may assign and then explicitly opening those ports. i.e.
In a stateful firewall you do not need to explicitly open the DATA port, the netfilter helper module will recognize the dynamic port that gets assigned and dynamically open that port for the correct client by marking the DATA connection as
RELATEDafter which it will match the generic rule:This requires that the correct kernel module is loaded, in the FTP case manually by running for instance
insmod nf_conntrack_ftp, making that persistent depends accross reboot depends on the distribution.Note: The FTP connection tracking module will fail when FTP is used with SSL, as the control connection will be encrypted and the nf_conntrack_ftp won't be able to read the PASV repsonse anymore.
NFS and similar RPC services: The problem with RPC services is that by design they don't use a specific fixed port. They can pick any available port at random, which will then be registered with the RPC Portmap daemon. A client trying to connect will query the Portmap daemon and then connect directly to the correct port. That solved the problem of running out of reserved ports...
From a firewall perspective TCP/UDP port 111 needs to be opened and and the actual port that the RPC service is currently using. The problem of opening such a random port in a firewall is typically solved by restricting the RPC service, such as the NFS server, to use a predfined fixed port.