I have ran into a 10.10.50.0/20 network that is having issues with packets dropping. They run all unmanaged switches and I imagine the huge amount of devices on one subnet is causing issues.
I've looked at breaking the network down into separate VLANS but the cost of Layer3 switches is too high. What options do I have? They use all unmanaged switches and an NSA 2400 sonicwall.
How many computers are actually in the subnet?
The size of the subnet has no effect on broadcast traffic. You can run a /8 with 12 hosts in it and you'll see the same composition of traffic as a /24 with the same number of hosts. It's the number of computers that matters.
Personally, I wouldn't put more than about 512 Windows PCs into a single subnet. If you're below that, though, you're probably better off figuring out what the root cause of the problems is because you're likely going to find out, after wasting a lot of time and spending a lot of money, that the problem will still be with you.
Well, first you don't need L3 switches to VLAN, you can use normal L2 switches and then have a router, a single L3 switch or just use the NSA 2400 to route and firewall between subnets.
You are going to have to buy some form of equipment to segregate the networks. Your options are to either A) go the old school way with physical LANs separated by a router, or B) use VLANs.
The first option allows you to avoid spending on a whole new set of switches, BUT will be a management nightmare.
I wouldn't want to guess as to what's causing the problem, I would want to know what's causing the problem. My recommendation would be to run some packet captures at various points in the network and see what the evidence shows you.