Home / server / Questions / 772953
Accepted
MiquelB
Asked: 2016-04-27 02:28:32 +0800 CST

How to maximize utilization of fluentd server?

  • 772

I have a fluentd server that is processing an average of 1000 req/s. This server is composed by 32 cpus. In this server are running 32 fluentd docker containers with the same configuration. The entry point of the server is an nginx as a load balancer configured with round-robin algorithm and send it to the differents containers. Both, containers and nginx, listen on 24224/tcp.

The input of this nodes is 6 C# applications that use fluent-logger-csharp to send to fluentd server. So, for each app open one connection to the Fluentd Server and due to the large traffic, the app keeps open the socket. This causes that only 6 of 32 docker is processing requests.

How you could better use of server utilization without modifying the C# client?

logging load-balancing nginx docker fluentd

1 Answers

  1. Best Answer

    First, I would check what are the bottlenecks:

    If your app is not overloading the fluentd service, then why even use 32 cores?

    If fluentd output is the bottleneck, you can use multi threading with the num_thread option; that way you may want to use like 5 threads on 6 fluentd instances, adding up to 30 cores, instead of 32 single instances where only 6 are used.

    As for input, if your servers keep connections open, then indeed this is your bottleneck, and then you might want to deploy more of those services to increase the number of logging outputs to your fluentd inputs.

    • 1