Iptables - source or destination check first?

So I'm running a server with iptables config and I've only been checking for destinations. Here is what the config structure is. Top of config has a list of ip addresses to drop followed by rate-limiting on specific destination ports then accepting the one's rate limited. In the end I blocked all the ports. So the one's not accepted are blocked.

Now I need to fix a source port match somewhere in the config. I want to block udp source port X so where would this come?

I placed it at the end of the config but I am not sure if it works. Would it check the destination of that source port first and allow it? or would it just block the source port.

UPDATE:

Here is what it sort of looks like.

# Block Source Port 53 - The commands below slow the SSH down
#-A INPUT -p udp --sport 53 -j DROP
#-A INPUT -p tcp --sport 53 -j DROP 

#
# Destinations
#

# Accept PPTPD VPN Ports 
// Commands go here

# Accept IP Rules and Port Range
// Commands go here

# Rate-Limit ICMP Ping Requests
// Commands go here - Used with the iptables recent module

# Rate-Limit Game Servers Ports
// Commands go here - Used with the iptables recent module

# Accept the Ports Rate-Limited above
// Commands go here

# Drop All Other TCP & UDP Ports
-A INPUT -p tcp -j DROP
-A INPUT -p udp -j DROP
-A INPUT -j DROP

I tried the source port 53 block at the end of the config where it didn't seem to work possibly because the destination match of that source came first. Now I put it on top of the config or even in the middle somewhere and it seems to work but my SSH/SFTP login seems awfully delayed, such that it even times-out sometimes after I enter the username although my own IP is accepted among the IP addresses accepted. I also just tried blocking source port 53 at UDP because that's what I actually need but it would still cause that delay. Not sure what UDP has to do with ssh/sftp. Maybe the delay is caused by the source port checking actually?