Why does Apache need such a long time for initial connection?

I am running apache 2.4.18 on ubuntu. Serving about 500.000 PI per month. No big deal without peaks. Somehow it started now to take a long time for the initial connection, after restarting apache it is back to normal.

Chrome tells me, that the initial connection takes so long:

After a few minutes I can come back with a new browser session and it looks OK:

MySQL, PHP and the rest seem not to be the problem as restarting apache will fix it and all query logs show high performance.

My guess is that the apache configuration is not made for 2.4 but 2.2 as there seem to be some naming changes on concurrent connetions.

This is a part of my config:

# 
# Timeout: The number of seconds before receives and sends time out.
# 
Timeout 300

#       
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#       
KeepAlive On

#       
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#   
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
# 
KeepAliveTimeout 5

What could be the issue here?