Home / server / Questions / 540632
Accepted
abstrask
Asked: 2013-09-21 16:31:05 +0800 CST

Designing segmented LAN with fairly shared hi-speed internet access on a tight budget

  • 772

With another member of the owners' association, I've been tasked with designing and setting up shared, hi-speed, internet access, for our apartment building. We have very little budget and hope to be able to do this, with the hardware already at hand (meaning not state of the art).

I have more than a decade's worth of experience as a sys admin, but with focus on the server side of things. While no stranger to (most of) the terms, and with at least some practical experience in setting up smaller networks, designing such a setup definitely falls outside my main competence.

I have an idea on how to accomplish this, but I've probably not taken everything into consideration, and a second opinion and sanity check is certainly welcome.

While this question is based on my specific challenge, I would think the answers will form a generally sound approach, to setting up fairly shared, segmented, multi-tenant, internet access on a tight budget. I hope this is acceptable for this site.

How is this best done?

Scope

Environment

  • 54 apartments with existing CAT5e cabling
  • Apartments split roughly 50/50 into two patch rooms, with a single CAT5e cable between (might add another later)
  • Fibre based internet connection, initially capped at 300 Mbps, will terminate in one patch room
  • No administrative access to internet router

Hardware & Software

All hardware is at least 5 years old. Some was bought by us a while ago, some was given to us, because it was too old for that company's production use (expensive service, performance and all that).

  • HP Proliant DL380 G6, dual Xeon CPU, 32 GB RAM, 4 gigabit NICs
  • 2 x Dell PowerConnect 5324 gigabit managed switches
  • 2 x HP ProCurve 2524 managed 100 Mbit switches
  • pfSense as gateway/firewall preferred, due to existing knowledge and experience
  • Running firewall virtual on VMWare ESXi preferred, due to manageability (snapshots, full image backups, hardware abstraction), and the possibility to run a small webserver, without the need for extra kit

Requirements & Goals

  • We must collectively be able to utilise the internet bandwidth we pay for
  • We'd like to be able to support higher bandwidth in the future
  • Being 100 Mbps switches, we need to be able to trunk/team at least two uplinks from the HP to the Dell switches
  • No access should be possible between apartments
  • IP addresses must be handed out to apartments by DHCP
  • We will run a web server and mail server as well, which must be accessible both from the inside and outside
  • Anything obvious missing?

Rough Design Idea

Physical Network

  • Install a Dell gigabit switch in each patch room
  • Connect the two Dell switches, by the single cable running between the patch rooms
  • Connect an HP 100 Mbit switch by two uplinks using link aggregation to each of the Dell switches
  • Connect all server NIC's to gigabit ports, reserve a few additional gigabit ports for future use
  • Connect as many apartments as possible to available gigabit ports, the rest to 100 Mbit ports

LAN Setup

  • Create a VLAN for each apartment
  • Assign individual VLANs untagged to each apartment switch port
  • Assign the necessary VLANs tagged on the (aggregated) uplink ports
  • Assign all apartment VLANs tagged on the uplink to a server NIC for "Apartments LAN"
  • Assign the default VLAN (ID 0) untagged on the uplink to a server NIC for "Administrative LAN"

Server

  • Dedicate one physical server NIC for "Apartment LAN", one for "Administrative LAN", one for "WAN" and one for "DMZ"
  • Install VMWare ESXi, create virtual NICs for each apartment (using their VLAN IDs), linked to the "Apartment LAN" physical NIC
  • Connect the "WAN" server NIC to the internet router
  • Install and configure pfSense to handle routing, adding all apartment vNICs,

Questions

  • Does our idea pass your sanity check? Any obvious flaws or pitfalls in the design?
  • Is it reasonable to expect 300+ Mbps routing from pfSense running virtualised?
  • What sort of link aggregation can/should we use for linking the two different brands of switches together - preferably providing both fault tolerance and n times 100 Mbps uplink
  • Can we expect VLAN to work as expected, combining tagged and untagged, across different brands of switches, through link aggregation and passed through to VMWare?
  • Given the proposed design, I'm not quite sure how we would go about handling routing and allowing access, to the administrative VLAN from selected apartments (of those who are managing the setup), but in the big picture this is a minor thing.

Edit: What We Ended Up With

Sorry for the radio silence, but we struggled quite a bit to get the network setup as we wanted. It turned out to be a real PITA, trying to setup segregation for each switch port, across various makes of switches.

The particular model of Dell switches we got, was one of the few that didn't support Private VLAN's. Instead we acquired an old 48-port Allied Telesis switch, but while "private" ports could not talk to each other, each port was not segregated from the admin network, which we required. The HP's "port-isolation" feature worked as advertised, but they did not have enough ports.

After som effort, we managed to get our hands on 1 x 48 port and 2 x 24 port Cisco Catalyst 2950 - each with 2 gigabit uplink ports, the rest 100 Mbit. What a difference that made! No more fiddling with getting different vendor variants of the same feature to work together.

We run pfSense virtual, along with a few low-intensity VM's, and can easily route 300 Mbps without the server breaking sweat.

In summarized bullet form, this is what we did to accomplish our goals, and is what we're currently running. Hope this can prove useful to someone. Thanks for the input!

Physical Network

  • 48 port switch in same room as server and internet - one gigabit uplink to tenant NIC in server, the other to 24 port switch in other room
  • Gigabit uplink between the two 24 port switches in the other room
  • 100 Mbit port connected to admin NIC in server
  • Internet router connected to WAN NIC in server

Switch Configuration

  • Free Cisco Network Assistant software used to configure all devices, from the same interface. Highly recommended!
  • VLAN 1 was designated our admin network
  • Protected port (segregration) and port security (limited MAC addresses per port) configured on each tenant port
  • As the switches were simply Ethernet linked, as opposed to stacked, "Protected Port" has no effect between switches. That is, a tenant port in one switch, can talk to all tenant ports on other switches. To get around that, we created a separate tenant VLAN, (VLAN ID matching the fourth octet of the switch IP address) for each switch
  • Ports for uplinks between switches configured with "Smartports", to enable trunking. Native VLAN set to 1, allowing all other VLANs (tagged)
  • In switch A, tenant uplink to server NIC configured to run tenant VLAN A native, VLAN B and C tagged

pfSense (and a Little VMWare) Config

  • Separate vSwitches (VMWare) setup for WAN, Admin LAN, Tenant LAN server NIC's
  • Tenant vSwitch (VMWare) changed from VLAN ID 0 (none) to 4095 (all)
  • Interfaces assigned in pfSense, labeled accordingly (WAN, Admin, Tenant)
  • Two VLAN's added to the Tenant NIC, with the ID's of Tenant VLAN B and C (as VLAN A already runs untagged) - now we have 3 tenant interfaces in pfSense
  • An interface group for the three tenant interfaces added, to simplify administration of firewall rules

That's pretty much the configuration pertaining to my questions. We've also added OpenVPN for remote admin access, auto config backup from a Windows PC, added DMZ interface and continue to fine-tune and improve our now super powerful, 6 interface firewall/router! :-)

networking

1 Answers

  1. Best Answer
    • Does our idea pass your sanity check?

    I would consider handling the VLAN interfaces in pfSense rather than mess up ESXi with 54 extra NIC's that it doesn't really need to know about (personal preference).

    You didn't mention anything about your addressing, but assuming you're using RFC1918 private addresses (so you have full control), make sure the addressing makes sense. Personally, I like to make sure my VLAN tags match my subnets. In this instance, I would do something like this:

    10.0.0.0/24   => Admin
10.1.101.0/24 => Apartment 1 (VLAN 101)
10.1.102.0/24 => Apartment 2 (VLAN 102)
...
10.1.154.0/24 => Apartment 54 (VLAN 154)

    I added 100 to the VLAN's since you don't want to use VLAN 1 for Apartment 1. I also kept the admin and apartment subnets in a separate /16 so you can cleanly route the apartment VLAN's (in a single CIDR) separately in the future if need be.

    How will you handle when tenants want to have inter-apartment traffic? (eg, I'm really good friends with my neighbour and I want to share the videos on my NAS with him/her). This isn't so much a technical question as a policy question.

    • Any obvious flaws or pitfalls in the design?

    You're going to end up with a lot of NIC's in pfSense that you're going to have to make sure can't be routed between. I haven't used pfSense for years, but in principal you need to setup a default DENY policy. This will prevent Apt X routing traffic to Apt Y, without having to manually ensure that policy is always in place. Then create the individual ALLOW policies to only let each apt out via the internet connection.

    • Is it reasonable to expect 300+ Mbps routing from pfSense running virtualised?

    I don't see why not when you're using gigabit NIC's. Virtualization doens't add that much overhead.

    • What sort of link aggregation can/should we use for linking the two different brands of switches together - preferably providing both fault tolerance and n times 100 Mbps uplink

    LACP.

    • Can we expect VLAN to work as expected, combining tagged and untagged, across different brands of switches, through link aggregation and passed through to VMWare?

    802.1q is a "standard"... Depending how old the switches are, if they were designed when 802.1q was still being finalized they might do something "weird", but you should be pretty safe.

    • Given the proposed design, I'm not quite sure how we would go about handling routing and allowing access, to the administrative VLAN from selected apartments (of those who are managing the setup), but in the big picture this is a minor thing.

    The default deny policy will prevent this, so you just need to add explicit ALLOW's to permit it.

    • 5