FreeBSD load splitting and failover with 2 WANs (dynamic routing)

This is NOT possible, given only the configuration you've described. Link aggregation (or "bonding", as it's sometimes called) cannot provide load balancing or failover between multiple upstream providers. Your current configuration does not appear to serve any useful purpose, unless you are leaving out some important routing and/or IP configuration details.

Link aggregation is strictly a layer 2 (Ethernet) protocol. It can provide load balancing and/or failover when you have multiple physical Ethernet links to a single upstream switch or host. But two different DSL uplinks (even if they link to the same provider) are two different devices. Bonding multiple DSL lines is possible, and so is bonding multiple T1 lines, if the router/bridge devices that terminate the telco loops (i.e., the DSL box you plug into your phone line) support the same protocol-specific (e.g., T1, DSL, etc.) bonding technique.

Your FreeBSD host cannot bond multiple DSL links unless you install DSL cards in it that offer that feature. And even then, your ISP would have to support bonding at their DSLAM (the equipment at the other end of your DSL lines, in the phone company CO).

Load balancing and/or failover of IP traffic, using multiple upstream links, MUST be handled with routing techniques (Layer 3, IP).

To be fair, I may have assumed too much about the details you left out. Could you update your question with some more information about your configuration?

• What IP addresses are bound to re0, re1, and/or lagg0?
• Have you configured a single upstream gateway route, or are you using a more complex routing setup (dynamic routing daemons, IP forwarding/header rewriting, source IP-based routing rules, etc.)?
• If you're using a complex routing setup, could you describe it in detail?
• Do your two DSL phone lines terminate in a single device (AKA modem, router, bridge, etc.), or does each DSL line have its own box? If the two lines share one box, do you have both re0 and re1 plugged directly into that box?
• Do your DSL lines come from the same ISP? If so, does your ISP support DSL bonding? If so, is your current service plan set up for bonding? (i.e., did you order bonded DSL lines, and have they set you up for bonding?)

I may be misunderstanding your situation and I've certainly never done anything of this sort with freebsd, but...

A quick examination of the lagg makes it look like a layer2 redundancy mechanism where the IP address is floating between a number of interfaces but the IP address never changes.

On your DSL links -- are they to different providers? If they are then I think you'd need a layer 3 aware mechanism.

If they are to different providers, you will need to do something with your routing tables so that link 1 goes to one set of remote ip addresses and the other gets the rest, or maybe one interface is used for outgoing web traffic and the other is used for everything else. Typically this is done with "policy routes" for small organizations and BGP for larger organizations.

Lastly, I'd look into pf instead of ipfw. It's super sophisticated and fully awesome. Example pages that discuss what you're looking for

http://www.openbsd.org/faq/pf/pools.html

The above article has a discussion of how to load balance outgoing traffic.

I only looked at the pf docs for a second but it doesn't seem as though there is a link tracking or ping tracking mechanism directly in the pf functionality.

I'd just make 3 pf config files -- only.link1, only.link2 and balance.links. Then, make sure that in the balance links config you don't balance traffic to the next hop (so traffic to the next hop of link1 never goes through link2 and the reverse). Then, run a script that does a ping test to the next hop of each interface and if that fails switch pf configs to the other interface exclusively. Networking equipment that does vrrp often has this sort of functionality, but built into the device not as part of a hacked on perl script, but it should work just the same.