Elijah Lynn's questions

We are experiencing intermittent "Address already in use: AH00072: make_sock: could not bind to address" on port 80 failures on our servers. I have looked at other answers and they are mostly based around it being a consistent failure and a LISTEN being defined twice, which is not our issue.

Our issue only happens shortly after boot of the instance where we "sudo service httpd restart" the service to load a new config we lay down depending on the environment.

I was able to catch the issue live and see that it was in fact httpd that is bound. The issue appears to be that Apache/httpd is hung and can't restart and then the restart doesn't wait for it.

We are using Apache 2.4 on Amazon Linux 1.

I have a server that is currently accepting HTTPS requests and the only thing that shows with netstat -plnt is this:

elijahlynn@web:~$ sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 10.218.108.1:53         0.0.0.0:*               LISTEN      10242/dnsmasq   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1111/sshd       
tcp6       0      0 fd42:9324:ab98:50fb::53 :::*                    LISTEN      10242/dnsmasq   
tcp6       0      0 fe80::c024:c5ff:fe68:53 :::*                    LISTEN      10242/dnsmasq   
tcp6       0      0 :::22                   :::*                    LISTEN      1111/sshd       

And sudo lsof -i :443 returns nothing as well.

I have tried variations of the ss tool as well but no luck in seeing anything listening.

This server uses iptables to forward traffic to an LXC "haproxy" container but from my understanding, first the OS needs to bind a port to a dameon or application to begin with. The fact that this isn't showing any ports in the LISTEN state yet still serves traffic is really challenging my understanding of how things work.

Is there any way that this is possible for a program to be listening on a port but not showing with netstat, lsof or ss?

UPDATE:

elijahlynn@web:~$ sudo iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps /* generated for LXD network lxdbr0 */
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network lxdbr0 */
ACCEPT     all  --  anywhere             anywhere             /* generated for LXD network lxdbr0 */
ACCEPT     all  --  anywhere             10.0.0.0/8           state NEW,RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp spt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp spt:domain /* generated for LXD network lxdbr0 */
ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps /* generated for LXD network lxdbr0 */

UPDATE2:

elijahlynn@web:~$ sudo ss -lptn | column -t
State   Recv-Q  Send-Q  Local                                Address:Port  Peer                                 Address:Port
LISTEN  0       32      10.218.108.1:53                      *:*           users:(("dnsmasq",pid=10242,fd=9))
LISTEN  0       128     *:22                                 *:*           users:(("sshd",pid=1111,fd=3))
LISTEN  0       32      fd42:9324:ab98:50fb::1:53            :::*          users:(("dnsmasq",pid=10242,fd=13))
LISTEN  0       32      fe80::c024:c5ff:fe68:999e%lxdbr0:53  :::*          users:(("dnsmasq",pid=10242,fd=11))
LISTEN  0       128     :::22                                :::*          users:(("sshd",pid=1111,fd=4))   

UPDATE3:

elijahlynn@web:~$ sudo lsof -i -n  | column -t
COMMAND  PID    USER        FD   TYPE  DEVICE     SIZE/OFF  NODE  NAME
sshd     1111   root        3u   IPv4  786        0t0       TCP   *:ssh                                 (LISTEN)
sshd     1111   root        4u   IPv6  788        0t0       TCP   *:ssh                                 (LISTEN)
dnsmasq  10242  lxd         4u   IPv4  343469983  0t0       UDP   *:bootps
dnsmasq  10242  lxd         6u   IPv6  343469985  0t0       UDP   *:dhcpv6-server
dnsmasq  10242  lxd         8u   IPv4  343469988  0t0       UDP   10.218.108.1:domain
dnsmasq  10242  lxd         9u   IPv4  343469989  0t0       TCP   10.218.108.1:domain                   (LISTEN)
dnsmasq  10242  lxd         10u  IPv6  343469990  0t0       UDP   [fe80::c024:c5ff:fe68:999e]:domain
dnsmasq  10242  lxd         11u  IPv6  343469991  0t0       TCP   [fe80::c024:c5ff:fe68:999e]:domain    (LISTEN)
dnsmasq  10242  lxd         12u  IPv6  343469992  0t0       UDP   [fd42:9324:ab98:50fb::1]:domain
dnsmasq  10242  lxd         13u  IPv6  343469993  0t0       TCP   [fd42:9324:ab98:50fb::1]:domain       (LISTEN)

I am having trouble getting Timeout directive to work, I have reproduced the issue in the below instructions and official Docker container. The server is using the standard MPM prefork model with default configs. My understanding is that the Timeout should return a 500 error after the timeout or some sort of 4xx or 5xx response code but it doesn't. Is my understanding incorrect?

Setup:

mkdir /tmp/test; cd /tmp/test
echo "<?php sleep(10);" > index.php
docker run -d -p 80:80 --name apache-php -v "/tmp/test":/var/www/html php:5.6-apache
time curl http://localhost/ # should take 10 seconds

Now, lower timeout to 1 second:

docker cp apache-php:/etc/apache2/apache2.conf . # Copy to host, since container doesn't have editor.  
echo "Timeout 1" >> apache2.conf # Set Timeout to 1, default is 300.  
docker cp apache2.conf apache-php:/etc/apache2/apache2.conf # Copy conf file back to container  
docker exec -it apache-php /bin/bash
service apache2 reload
exit # exit docker container. 

Test for error:

time curl http://localhost/ # Here is where I expect a timeout error, but it works just the same as before and takes 10 seconds.

p.s. I have read that AcceptFilter can affect things until the request gets past it. We do have the below config as well:

AcceptFilter http none
AcceptFilter https none

I copied a large amount of text to the clipboard and then pasted it into the terminal. I didn't realize it would be so large that it would continue filling the screen for ~5 minutes.

ctrl + c seems to work most of the time but not all the time and I couldn't quite get the timing right it seemed.

How can I stop a runaway/rogue clipboard paste from rendering the terminal useless until it is done?

update: I posted an answer below that uses htop to find it and then a SIGINT to kill it but that kills the terminal or pane (tmux) too. Would be nice if it didn't kill/shutdown the process. I tried SIGQUIT (pause), SIGABRT (abort) and SIGHUP (hang up) but no success (it doesn't affect the rogue paste).

update 2: Ahh, in my case it is an HTML paste and ctrl + c is actually working but since their are line break/returns in the paste it is starting a new paste every second or so. So ctrl + c is just affecting the current one. I don't think there is going to be an option in this scenario other than to kill the process with a SIGINT.

How can I tell if memcache was flushed programatically?

If I issues a echo flush_all | netcat localhost 11211 to memcache how can I tell it actually worked? I am writing some code to flush all servers and want to make sure they all received the command.