We have deployed multiple Haproxys TCP mode haproxy for the DB front and layer 7 haproxy for app front in production. I am looking for which is the best option to take one backend node out of traffic and do update/release/maintenance on applications. either I use Set server state Drain or set weight 0 which one will be effective and useful to slowly redirect all the clients connection to the other backends for app with out breaking and redirect all tcp connections as well ? also let me know whats meant by persistent connections ?
Use Drain.
That prevents new sessions being directed to the backend in question, however existing sessions will still be directed to that backend. Hence you should wait a reasonable amount of time for users to end their session (depending on the type of workload this could be a couple of minutes up to a day).
This of course is assuming that you have sticky sessions configured in haproxy, so that a user will always be directed to the same backend. If this is not the case, then it makes no difference if you use weight 0 or drain.
Persistent connections are connections that stay open. If you want to use those, then ensure that haproxy timeouts are long enough to cater to those connections; otherwise haproxy may close a connection that is still in use (but idle).