openvpn ping: packet response is done on a different (?) port

I am having difficulty pin pointing the exact location of a problem with my openvpn which is running on a Linux server, which sits behind a modem that is in turn connected to the Internet through a dynamic ip. For an extended history of the problem from openvpn side and steps taken to find it see this question.

The problem lies somewhere between the server firewall, the client firewall and the intermediate modems involved

on the server cat /etc/rc.d/rc.firewall | egrep -v '^(#|$)' | grep "\-t nat" gives:

$IPT -t nat -A PREROUTING -p tcp -i $LOCAL_IFACE --destination-port 80 -j REDIRECT --to-ports 3128 #TRANSP PROXY
$IPT -t nat -A POSTROUTING -o $INET_IFACE -j MASQUERADE

on the client the same command:

$IPT -t nat -A POSTROUTING -o $INET_IFACE -j MASQUERADE [/quote]

again one more tcpdump -i eth1 udp and host <server_or_client_inet_ip> of a TLS negotiation: on client

15:07:43.028277 IP 192.168.233.165.37073 > server_ip.32456: UDP, length 14
15:07:43.064518 IP server_ip.29199 > 192.168.233.165.37073: UDP, length 22

and on the server

15:07:43.046535 IP client_ip.15312 > 192.168.231.165.32456: UDP, length 14
15:07:43.046618 IP 192.168.231.165.32456 > client_ip.15312: UDP, length 22

this connection topographically looks like this:

  client         <--> modem of client <--i-n-t-e-r-n-e-t--> modem of server <-->     server
192.168.233.165       192.168.233.117                       192.168.231.117      192.168.231.165.

sorted by time , an openvpn ping between a client and the server goes like this :

[client]  15:07:43.028277 IP 192.168.233.165.37073 > server_ip.32456: UDP, length 14
[server]  15:07:43.046535 IP client_ip.15312 > 192.168.231.165.32456: UDP, length 14
[server]  15:07:43.046618 IP 192.168.231.165.32456 > client_ip.15312: UDP, length 22
[client]  15:07:43.064518 IP server_ip.29199 > 192.168.233.165.37073: UDP, length 22

SO if I read that correctly

1. the client sends out a packet from port port 37073 to the server openvpn port (32456)
2. in this step the packet passes through the client's modem, then through the internet and then through the server's modem.
3. then the server receives this packet on a different port (15312) than the originating one. destination is 32456 as expected though.
4. the server responds correctly from the port 32456 and with destination port the one is received the packet from (15312). The response packet is on its way again towards the client.
5. the packet is again going through the server modem, internet and client modem
6. the packet seems to arrive at the client on port on which this communication attempt originated (37073), but it does not seem to be coming from the port 32456. instead , somewhere in step 5 the source port was changed to 29199.
7. this source port 29199 is stated in no firewall rule and so the client firewall rule drops this packet.

Is it correct to assume that in step 2) and most importantly 5) some NAT is happening in the modems, because of the source port change of the packet? Or should I stick to the assumption that the problem lies in my firewalls somewhere?

I mean this seems like a NAT problem, but looking into the server and client firewall NAT tables I cannot pin point the error from my own setup firewalls.

If I assume that I have messed something up in my custom server and client firewalls (entirely possible), what netfilter commands should I look for that alter the source port of a packet?

As far as modems are concerned, let me state that in my server modem's config webpage, the firewall and SPI are disabled, no NAT rule has been set, and only a DMZ host has been setup so that all internet traffic is forwarded to the server's firewall. On the client side modem settings are exactly the same.

Is this packet sniffing with tcpdump enough ? could I use tcpdump with some other command in order to get more info as to where the source port is changed?

Thank you in advance for your help.