This might be a highly daft question so apologies in advance... I have the following set up:
100 Mbit on 1Gbit carrier Fibre line into comms room,
Fibre <-> EDD (ethernet demarcation device)
EDD <-> [switch A]
[switch A] <-> router 1 <-> [switch stack Z]
[switch A] <-> router 2 <-> [switch stack Z]
The routers are running pfSense with router 2 acting as the failover for router 1, floating IP's, synced interfaces etc.
My questions is, what sort of switch (if any) should I be using between the routers and the EDD?
The EDD only has one port activated on it (and is controlled by our supplier) so I initially tried putting a crappy SoHo 10/100 8-port netgear unmanaged switch as [switch a] but suffered horrendous packet loss (4 - 16%) and no single TCP connection could get above 80 Kbps.
So I binned that and replaced it with a slightly higher grade 100/1000 8-port linksys which seems to be doing a better job (no TCP throttling, only very occasional packet loss (2% max)).
However this still feels like the wrong solution to the problem of connecting both routers to the single EDD, any suggestions anyone to how this could better or at least anything very wrong that I'm doing?
If you can't get a second port off of the demarc, then your current solution is about as good as it can get.
It always hurts to have added single points of failure, but it doesn't seem like there's much of anything you can do about it.
The remaining packet loss may be due to packet buffering switches either upstream, or in your network. You may find configuring bandwidth limitations at 80 to 90% of the network capacity reduces the packet loss.
Try verifying the packet loss with the backup router out of the configuration and the other router connected directly to the EDD. If you still get occasional 2% packet loss the problem isn't your switch. The switch may be your most likely single point of failure, so document procedures for removing it from the configuration if it fails.