Questions[scale](server)

I have a single OpenVPN server instance running on a box running Ubuntu with a large number of clients (~500) connected over UDP. The client connections are stable till I restart the OpenVPN service on the server side. After that, the clients keep connecting and disconnecting, never able to achieve the same number of connections again. What could be the reason ? The connections on the client side keep timing out, never receiving any replies to their initial messages. Logs from one of the clients:

Wed Apr 22 10:16:14 2020 us=209975 [31956] Control Channel MTU parms [ L:1570 D:138 EF:38 EB:0 ET:0 EL:0 ]
Wed Apr 22 10:16:14 2020 us=210010 [31956] Socket Buffers: R=[229376->131072] S=[229376->131072]
Wed Apr 22 10:16:14 2020 us=210029 [31956] Data Channel MTU parms [ L:1570 D:1450 EF:70 EB:135 ET:0 EL:0 AF:3/1 ]
Wed Apr 22 10:16:14 2020 us=210052 [31956] Local Options String: 'V4,dev-type tun,link-mtu 1570,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-128-CBC,auth SHA256,keysize 128,key-method 2,tls-client'
Wed Apr 22 10:16:14 2020 us=210060 [31956] Expected Remote Options String: 'V4,dev-type tun,link-mtu 1570,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-128-CBC,auth SHA256,keysize 128,key-method 2,tls-server'
Wed Apr 22 10:16:14 2020 us=210074 [31956] Local Options hash (VER=V4): 'ec542dd5'
Wed Apr 22 10:16:14 2020 us=210087 [31956] Expected Remote Options hash (VER=V4): '00bc425f'
Wed Apr 22 10:16:14 2020 us=210098 [31956] UDPv4 link local: [undef]
Wed Apr 22 10:16:14 2020 us=210107 [31956] UDPv4 link remote: [AF_INET]10.146.23.218:8443
Wed Apr 22 10:16:14 2020 us=210130 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.23.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:16:16 2020 us=460234 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.23.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:16:20 2020 us=957985 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.23.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:16:28 2020 us=968424 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.23.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:16:44 2020 us=729143 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.23.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:17:16 2020 us=330992 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.23.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:18:20 2020 us=293321 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.23.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:19:14 2020 us=324898 [31956] [UNDEF] Inactivity timeout (--ping-restart), restarting
Wed Apr 22 10:19:14 2020 us=325114 [31956] TCP/UDP: Closing socket
Wed Apr 22 10:19:14 2020 us=325172 [31956] SIGUSR1[soft,ping-restart] received, process restarting
Wed Apr 22 10:19:14 2020 us=325217 [31956] Restart pause, 10 second(s)

And sometimes, it goes a few steps ahead and then times out:

Wed Apr 22 10:00:18 2020 us=961904 [31956] UDPv4 WRITE [14] to [AF_INET]10.146.7.218:8443: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Wed Apr 22 10:00:22 2020 us=447412 [31956] UDPv4 READ [26] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_HARD_RESET_SERVER_V2 kid=0 [ 0 ] pid=0 DATA len=0
Wed Apr 22 10:00:22 2020 us=447487 [31956] TLS: Initial packet from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2), sid=99bdf93f 4d4f0c73
Wed Apr 22 10:00:22 2020 us=447606 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 0 ]
Wed Apr 22 10:00:22 2020 us=447813 [31956] UDPv4 WRITE [114] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=1 DATA len=100
Wed Apr 22 10:00:22 2020 us=447990 [31956] UDPv4 WRITE [114] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=2 DATA len=100
Wed Apr 22 10:00:22 2020 us=448131 [31956] UDPv4 WRITE [100] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=3 DATA len=86
Wed Apr 22 10:00:24 2020 us=789806 [31956] UDPv4 WRITE [114] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): 
...
...
Wed Apr 22 10:00:29 2020 us=952755 [31956] UDPv4 READ [126] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ 3 ] pid=1 DATA len=100
Wed Apr 22 10:00:29 2020 us=952860 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 1 ]
Wed Apr 22 10:00:29 2020 us=952978 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=2 DATA len=100
Wed Apr 22 10:00:29 2020 us=953016 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 2 ]
Wed Apr 22 10:00:29 2020 us=953074 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=3 DATA len=100
Wed Apr 22 10:00:29 2020 us=953107 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 3 ]
Wed Apr 22 10:00:29 2020 us=953168 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=4 DATA len=100
Wed Apr 22 10:00:29 2020 us=953201 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 4 ]
Wed Apr 22 10:00:30 2020 us=411565 [31956] UDPv4 READ [22] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 1 ]
Wed Apr 22 10:00:36 2020 us=566024 [31956] UDPv4 READ [126] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ 1 ] pid=1 DATA len=100
Wed Apr 22 10:00:36 2020 us=566119 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 1 ]
Wed Apr 22 10:00:36 2020 us=566243 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=2 DATA len=100
Wed Apr 22 10:00:36 2020 us=566302 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 2 ]
Wed Apr 22 10:00:36 2020 us=566396 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=3 DATA len=100
Wed Apr 22 10:00:36 2020 us=566450 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 3 ]
Wed Apr 22 10:00:36 2020 us=566657 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=4 DATA len=100
Wed Apr 22 10:00:36 2020 us=566769 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 4 ]
Wed Apr 22 10:00:38 2020 us=981036 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=5 DATA len=100
Wed Apr 22 10:00:38 2020 us=981102 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 5 ]
Wed Apr 22 10:00:38 2020 us=981167 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=6 DATA len=100
Wed Apr 22 10:00:38 2020 us=981204 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 6 ]
Wed Apr 22 10:00:38 2020 us=981286 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=7 DATA len=100
Wed Apr 22 10:00:38 2020 us=981328 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 7 ]
Wed Apr 22 10:00:38 2020 us=981371 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=8 DATA len=100
Wed Apr 22 10:00:38 2020 us=981406 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 8 ]
Wed Apr 22 10:00:49 2020 us=79299 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=6 DATA len=100
Wed Apr 22 10:00:49 2020 us=79459 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 6 ]
Wed Apr 22 10:00:49 2020 us=79750 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=7 DATA len=100
Wed Apr 22 10:00:49 2020 us=79874 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 7 ]
Wed Apr 22 10:00:49 2020 us=80081 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=8 DATA len=100
Wed Apr 22 10:00:49 2020 us=80191 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 8 ]
Wed Apr 22 10:00:49 2020 us=80388 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=9 DATA len=100
Wed Apr 22 10:00:49 2020 us=80507 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 9 ]
Wed Apr 22 10:00:49 2020 us=80636 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=10 DATA len=100
Wed Apr 22 10:00:49 2020 us=80717 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 10 ]
Wed Apr 22 10:00:49 2020 us=80884 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=11 DATA len=100
Wed Apr 22 10:00:49 2020 us=80977 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 11 ]
Wed Apr 22 10:00:49 2020 us=81130 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=12 DATA len=100
...
...
Wed Apr 22 10:00:59 2020 us=796622 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 15 ]
Wed Apr 22 10:00:59 2020 us=796741 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=16 DATA len=100
Wed Apr 22 10:00:59 2020 us=796788 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 16 ]
Wed Apr 22 10:00:59 2020 us=796885 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=17 DATA len=100
Wed Apr 22 10:00:59 2020 us=796933 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 17 ]
Wed Apr 22 10:00:59 2020 us=797016 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=18 DATA len=100
Wed Apr 22 10:00:59 2020 us=797061 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 18 ]
Wed Apr 22 10:00:59 2020 us=797139 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=19 DATA len=100
Wed Apr 22 10:00:59 2020 us=797172 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 19 ]
Wed Apr 22 10:00:59 2020 us=797271 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=20 DATA len=100
Wed Apr 22 10:00:59 2020 us=797313 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 20 ]
Wed Apr 22 10:01:08 2020 us=418632 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=17 DATA len=100
Wed Apr 22 10:01:08 2020 us=418683 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 17 ]
Wed Apr 22 10:01:08 2020 us=418754 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=18 DATA len=100
Wed Apr 22 10:01:08 2020 us=418794 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 18 ]
Wed Apr 22 10:01:08 2020 us=418848 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=19 DATA len=100
Wed Apr 22 10:01:08 2020 us=418876 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 19 ]
Wed Apr 22 10:01:08 2020 us=418929 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=20 DATA len=100
Wed Apr 22 10:01:08 2020 us=418960 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 20 ]
Wed Apr 22 10:01:08 2020 us=419048 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=21 DATA len=100
Wed Apr 22 10:01:08 2020 us=419097 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 21 ]
Wed Apr 22 10:01:08 2020 us=419170 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=22 DATA len=100
Wed Apr 22 10:01:08 2020 us=419220 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 22 ]
Wed Apr 22 10:01:08 2020 us=419298 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=23 DATA len=100
Wed Apr 22 10:01:08 2020 us=419348 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 23 ]
Wed Apr 22 10:01:08 2020 us=419416 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=24 DATA len=100
Wed Apr 22 10:01:08 2020 us=419449 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 24 ]
Wed Apr 22 10:01:16 2020 us=361894 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=21 DATA len=100
Wed Apr 22 10:01:16 2020 us=361993 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 21 ]
Wed Apr 22 10:01:16 2020 us=362261 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=22 DATA len=100
Wed Apr 22 10:01:16 2020 us=362341 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 22 ]
Wed Apr 22 10:01:16 2020 us=362585 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=23 DATA len=100
Wed Apr 22 10:01:16 2020 us=362643 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 23 ]
Wed Apr 22 10:01:16 2020 us=362855 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=24 DATA len=100
Wed Apr 22 10:01:16 2020 us=362910 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 24 ]
Wed Apr 22 10:01:16 2020 us=363146 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=25 DATA len=100
Wed Apr 22 10:01:16 2020 us=363213 [31956] UDPv4 WRITE [22] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 25 ]
Wed Apr 22 10:01:16 2020 us=363463 [31956] UDPv4 READ [114] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=26 DATA len=100
...
...
UDPv4 WRITE [114] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=10 DATA len=100
Wed Apr 22 10:02:10 2020 us=955007 [31956] UDPv4 READ [22] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 12 ]
Wed Apr 22 10:02:38 2020 us=998361 [31956] UDPv4 WRITE [114] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=9 DATA len=100
Wed Apr 22 10:02:40 2020 us=213582 [31956] UDPv4 WRITE [114] to [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_CONTROL_V1 kid=0 [ ] pid=10 DATA len=100
Wed Apr 22 10:02:46 2020 us=59600 [31956] UDPv4 READ [22] from [AF_INET]10.146.7.218:8443 (via [AF_INET]10.146.7.195%Port2): P_ACK_V1 kid=0 [ 10 ]
Wed Apr 22 10:03:24 2020 us=700713 [31956] TLS Error: TLS key negotiation failed to occur within 200 seconds (check your network connectivity)
Wed Apr 22 10:03:24 2020 us=700760 [31956] TLS Error: TLS handshake failed
Wed Apr 22 10:03:24 2020 us=700962 [31956] TCP/UDP: Closing socket
Wed Apr 22 10:03:24 2020 us=700992 [31956] SIGUSR1[soft,tls-error] received, process restarting
Wed Apr 22 10:03:24 2020 us=701025 [31956] Restart pause, 10 second(s)

I'm trying to integrate an openFire XMPP server to my current company Spring server but have two major questions I cannot find the answer to -

I'll start with my current architecture first -
1. The xmpp server have a DB-server of it's own seperated from the Spring server DB, This is a dedicated machine to keep the users char history etc
2. The spring server have a DB of it's own where it keeps the user credentials (md5 encrypted) and also client applications data
3. The spring server is dedicated to serve HTTP requests (a dedicated REST server)

All in all I have 2 DB servers once chat server and one Rest server

Now for the questions -
1. Can I forbid registration to the xmpp server (i.e. whitelist the rest server ip and let it be the only one who can create users after a user registers on it)?
2.For security reasons the Rest server switch the session for a logged in user every 2 days the iOS and Android clients deal with session managment locally - How can I use those session with the XMPP server?
To clarify - I want the users to be able use the xmpp server only for chat purposes but only after they logged in to the application itself since the user session may expire the chat client will also have to re-authenticate against the REST server, how can I achieve this?
3. Won't it create an overload on the REST server? (i.e. the Rest server will now have to handle client requests and also XMPP server requests)
4. What is the best architecture to achieve this kind of a system (chat server, db server for chat server, rest server, db server for rest server) so that the system can scale horizontally?

I searched google for an article or something related to describe the general architecture but couldn't find nothing relevant, since I'm not "inveneting the wheel" here I would love to hear a good advice or be directed to an article that explains the How-To's

Thanks in advance.

I have an application which needs to scale horizontally to cover web and service nodes (at the moment they're all on one) but interact with the same set of databases and source files (both application code and custom assets). Database is no problem, it's handled already with replication in MongoDB.

Also, the configuration of the servers are the same (100% linux). This question is literally about sharing a filesystem between machines so that its content is always correct, regardless of the node accessing it.

My two thoughts have so far been NFS and SAN - SAN being prohibitively expensive and NFS seeing some performance issues on the second node with regards to glob()ing in PHP.

Does anyone have recommended strategies or other techniques that don't involved sharding data across nodes or any potential gotchas in NFS that may cause slow disk seek times?

To give you an idea of the scale, the main node initialises it's application modules in ~ 0.01 seconds. The secondary is taking ~2.2 seconds. They're VM's inside a local virtual network in ESXi and ping time between them is ~0.3ms

What sort of computer specs (CPU, RAM, disk speed) should I use for running queries on a database of 200+ million records? The queries are for a research project, so there is only one "user" and only one query will be running at a time.

I tried it on my own laptop with SQL Server with an i3 processor, 2GB RAM, 5400 RPM disk and a simple query didn't finish even after 8+ hours. I have an option to connect a SSD via eSata and upgrade to 4GB RAM, but not sure if this will be enough...

Thanks!

Edit: The database is about 25 GB and the indexes are not setup properly. When I tried to add an index, I let it run for about 8 hours and it still hadn't finished so I gave up. Should I have more patience :)?
In general, the queries will run once in a while and its ok even if it takes a couple hours to complete....

Also, the queries will produce probably about 10 million records which I need to process using Stata/Matlab and I'm concerned that my current laptop is not strong enough, but unsure of the bottleneck....

I'm looking for resources on how to grow our server setup.

We currently have one dedicated server with Rackspace in the UK of the following spec:

HPDL385_G2_PrevGen
HP Single Dual Core Opteron 2214 (2.2Ghz)
4GB RAM
2x 10,000 SCSI Drives in RAID 1

Our traffic is up to 550,000 UVs per month.

The site runs off a PHP and MySQL setup. The database gets an absolute hammering, we have many complex queries joining multilpe tables.

We are using APC for PHP caching.

I'm getting to the stage where I've done as much DB and query optimisation as I can and wonder what the next step should be......

I've looked at memcache, but I've got the impression that his requires a large amount of RAM and ideally a dedicated box....

So is the next step to have two boxes; one for database, one for Apache? Or is there a step I've overlooked.

Our load is usually around the 2 mark, but right now it's up at 20!

Some graphs from Munin: