Home / server / Questions / 117043
In Process
z8000
Asked: 2010-02-26 21:17:34 +0800 CST

IPC between multiple processes on multiple servers

  • 772

Let's say you have 2 servers each with 8 CPU cores each.

The servers each run 8 network services that each host an arbitrary number of long-lived TCP/IP client connections.

Clients send messages to the services.

The services do something based on the messages, and potentially notify N>1 of the clients of state changes.

Sure, it sounds like a botnet but it isn't. Consider how IRC works with c2s and s2s connections and s2s message relaying.

  • The servers are in the same data center.
  • The servers can communicate over a private VLAN @1GigE.
  • Messages are < 1KB in size.

How would you coordinate which services on which host should receive and relay messages to connected clients for state change messages?

There's an infinite number of ways to solve this problem efficiently.

  • AMQP (RabbitMQ, ZeroMQ, etc.)
  • Spread Toolkit
  • N^2 connections between allservices (bad)
  • Heck, even run IRC!
  • ...

I'm looking for a solution that:

  • perhaps exploits the fact that there's only a small closed cluster
  • is easy to admin
  • scales well
  • is "dumb" (no weird edge cases)

What are your experiences?

What do you recommend?

Thanks!

networking service messaging architecture

1 Answers

  1. If:

    1. You're opposed to using an existing middleware bus.
    2. Your switch supports IGMP snooping (basically any Cisco switch you can buy will have this turned on by default---I assume Dell/HP switches will also have this feature)
    3. You're all on the same VLAN

    Then IPv4 multicast will work out-of-the-box for exactly what you're trying to do. Receivers subscribe to a channel (multicast group), senders send a UDP datagram to the group's multicast address, the switch works out who gets what. It makes your network equipment handle the message routing intelligently.

    If your switch doesn't support IGMP snooping, then it will treat the (Ethernet) multicast frames as broadcasts, and send them to all hosts on the VLAN, regardless of whether the host has asked for them or not. So you'll clog the tube between your switch and the hosts, even if the host OS is just dropping the packets before it gets to any application.

    You can still use IPv4 Multicast if they aren't all on the same VLAN, but you've got to do more configuration on your gear to make it go, but it will still work just fine.

    • 1