Is using Free/Open BSD + pf a workable option for filtering DDoS? Which of the two would perform better under heavy load? (SYN flood maxing a 1 gbit pipe)
Is this even an option to consider, or is a full hardware DDoS filter needed to get fast enough performance?
I think that pf can handle (synproxy,urpf and syncache tuning) this correctly on decent hardware without a problem using Freebsd or OpenBSD. I'll tend to use OpenBSD because i'm more familiar with it.
Using any firewall as DDoS protection is a bad idea. DDoS attacks hit the most resource-intensive portion of any firewall, evaluating its ruleset for huge numbers of new connections. DDoS attacks melt down any firewall very quickly. How quickly and how big of an attack one can handle depends on how big of a firewall. In general, you don't want to look at a firewall as a solution to help with DDoS attacks as it will most likely become the most susceptible thing on your network to succumbing to those attacks.