How do I setup an iptable setting to drop all IP packets that don't have my IP address on it? (do I even need to do this?)
This is the closest I can think of, but I think it doesn't quite work because I somehow need to be able to specify NOT destination AND source.
* filter
-A INPUT -i eth0 -d ! 1.2.3.4 -j DROP
COMMIT
You don't need to do this; the networking stack ignores packets that aren't destined for your machine. Only hosts on your local network would be able to send you packets with the wrong destination address anyway. This would also block broadcasts, which may be a problem.
You shouldn't need a rule to do this. Your system normally will ignore packets not addressed for it. Unless your system is setup for bridging, or routing.
If you did want to block traffic, you should explicitly allow your IP address, and then simply set the policy of the table to DROP.