n.r.'s questions

Good'day

Just out of curiosity I did a tcpdump port 80 on one of my remote virtual machines. I see a huge amount of requests to this http-port, 95% going to some AWS EC2 instance. Strange thing: The Web-Server was not on high load at all. The packets from tcpdump look like this:

20:57:16.860028 IP user123.example.net.http > ec2-55-155-123-123.ap-northeast-2.compute.amazonaws.com.12345: Flags [S.], seq 1234567, ack 12345567, win 12300, options [mss 1460], length 0

(user123.example.net.http is the address of my remote machine)

All of them looks basically the same, only thing that changes, is the actual EC2-adress (aka IP-adress) and the destination port. Sometimes it's an incoming packag (to port 80, too), but most of them are outgoing.

I run this tcpdump on port 80 today, from the office now. Now the traffic is between the remote virtual machine and another IP-adress, not the AWS-instance anymore:

08:34:46.369161 IP user123.example.net.http  > 123.12.123.132.37176: Flags [R.], seq 0, ack 2821777444, win 0, length 0
08:34:46.388933 IP 123.12.123.132.55539 > user123.example.net.http : Flags [S], seq 2934790784, win 29200, length 0

In this case, the IP-Adress does not change.

Only thing that changed: I remotely logged in from my working place now, using another IP-Adress (not the one you can see in tcpdump) and another SSH-client.

Investigation

What I tried so far, without success:

• I checked the logfile of nginx, they don't show any traffic (I am not running huges sites, just some little private projects)
• I stopped nginx, the only apparent port80-daemon in this setup
• I stopped all other non-system services: no changes

Current setup

I am running nginx on this server on port 80 and 443, but traffic is generally routed to HTTPS. Also I am running PHP 7.3 with FPM, but on sockets, not via TCP.

Besides that, I am running some e-mail services (dovecot, postfix, ...), Unbound, a *MySQL-*Server and an *ELK-*stack. This is what netstat -tulpn confirms: (I x'ed my public IP-Adress and my SSHD-port, which is not the default one)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      858/unbound
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2806/master
tcp        0      0 x.x.x.x:25              0.0.0.0:*               LISTEN      2806/master
tcp        0      0 127.0.0.1:8953          0.0.0.0:*               LISTEN      858/unbound
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      7942/nginx -g daemo
tcp        0      0 0.0.0.0:4190            0.0.0.0:*               LISTEN      628/dovecot
tcp        0      0 0.0.0.0:993             0.0.0.0:*               LISTEN      628/dovecot
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      491/mysqld
tcp        0      0 127.0.0.1:587           0.0.0.0:*               LISTEN      2806/master
tcp        0      0 x.x.x.x:587             0.0.0.0:*               LISTEN      2806/master
tcp        0      0 0.0.0.0:143             0.0.0.0:*               LISTEN      628/dovecot
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      7942/nginx -g daemo
tcp        0      0 0.0.0.0:???             0.0.0.0:*               LISTEN      27952/sshd
tcp6       0      0 ::1:53                  :::*                    LISTEN      858/unbound
tcp6       0      0 ::1:25                  :::*                    LISTEN      2806/master
tcp6       0      0 ::1:8953                :::*                    LISTEN      858/unbound
tcp6       0      0 :::443                  :::*                    LISTEN      7942/nginx -g daemo
tcp6       0      0 :::4190                 :::*                    LISTEN      628/dovecot
tcp6       0      0 :::993                  :::*                    LISTEN      628/dovecot
tcp6       0      0 ::1:587                 :::*                    LISTEN      2806/master
tcp6       0      0 :::143                  :::*                    LISTEN      628/dovecot
tcp6       0      0 :::80                   :::*                    LISTEN      7942/nginx -g daemo
tcp6       0      0 :::???                  :::*                    LISTEN      27952/sshd
udp        0      0 127.0.0.1:53            0.0.0.0:*                           858/unbound
udp6       0      0 ::1:53                  :::*                                858/unbound

After stopping all of the above mentioned servies, this is the output - Packets on port 80 still remains:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:???             0.0.0.0:*               LISTEN      27952/sshd
tcp6       0      0 :::???                  :::*                    LISTEN      27952/sshd

That's the output of service status-all|grep + (marked some with an * - those didn't stop when stopping the parent process, still trying to figure that out)

 [ + ]  amavis-mc *
 [ + ]  amavisd-snmp-subagent *
 [ + ]  apache-htcacheclean *
 [ + ]  apparmor
 [ + ]  cron
 [ + ]  fail2ban
 [ + ]  gdomap
 [ + ]  ip6tables
 [ + ]  iptables
 [ + ]  lm-sensors
 [ ? ]  modules_dep.sh
 [ + ]  nscd
 [ ? ]  php-fpm-chroot-setup.sh (make PHP available after reboot)
 [ + ]  procps
 [ + ]  quota
 [ + ]  rc.local
 [ + ]  resolvconf
 [ + ]  rsyslog
 [ + ]  ssh
 [ + ]  sysstat
 [ + ]  udev
 [ + ]  unattended-upgrades
 [ + ]  urandom
 [ + ]  uwsgi

Question is..

Why is there traffic on port 80, even if nginx is not running? Where does this traffic comes from (what process)?

I am using nginx with PHP-FPM and ELK as log file analysis.

When a PHP script causes an error the interpreter the error will be send back to nginx and nginx puts the error into the error.log file.

Problem is: Sometimes those error logs contains line breaks which logstash cannot handle, because line breaks are considered as a new log line.

2019/04/17 19:23:00 [error] 8356#8356: *4403 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function wp_using_themes() in /htdocs/wp-includes/template-loader.php:7
Stack trace:
#0 /htdocs/wp-blog-header.php(19): require_once()
#1 /htdocs/index.php(17): require('/htdocs/wp-blog...')
#2 {main}
  thrown in /htdocs/wp-includes/template-loader.php on line 7" while reading response header from upstream, client: 123.123.123.123, server: foobar.de, request: "GET /2014/11/foobar/ HTTP/1.1", upstream: "fastcgi://unix:/run/php/php-fpm-foobar.sock:", host: "foobar.de"

How to I either handle those linebreaks with logstash or format those error messages to remove the line breaks?

I am apparently not the only one with this problem, but after trying every suggested solution for couple I reached I point where I don't know what to do.

I am running Ubuntu 16.04, NGinx and PHP-FPM.

The nginx.conf is the default one, I only set a more informative log format.

The user rights seem to be okay: NGinx and PHP_FPM run under user www-data. Document-Root owner is www-data. FPM-Pool owner is www-data. Socket-file is there and writable.

I already tried setting folder- and file-permissions to the weakest (chmod 777): No result.

This is my server.conf, as you can see from commented lines, I tried a bunch of tricks - with no effect:

server {
    listen         8080;
#    listen         8080 default_server;
 #   listen         [::]:8080 default_server;
    server_name    example.com www.example.com
    root           /var/www/nginx/;
    index          index.php;
        access_log /var/log/nginx/scripts.log scripts;


    gzip             on;
    gzip_comp_level  3;
    gzip_types       text/plain text/css application/javascript image/*;

    location ~ \.php$ {
        if ($uri !~ "^/uploads/") {
            fastcgi_pass unix:/run/php/php-fpm-www.sock;

        }

    include fastcgi.conf;

#    include         snippets/fastcgi-php.conf;
    include         fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param SCRIPT_NAME $fastcgi_script_name;

  }

    # Block access to .htaccess
    location ~ \.htaccess {
        deny all;
    }

}    

That's the fpm-pool config:

[www]
listen = /run/php/php-fpm-$pool.sock
user = www-data
group = www-data
listen.owner = www-data
listen.group = www-data
php_admin_flag[allow_url_fopen] = off
php_admin_flag[allow_url_include] = off
php_admin_value[memory_limit] = 128M
pm = dynamic
pm.max_children = 5
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /
chroot = /var/www/nginx/
php_flag[display_errors] = on
php_admin_value[error_log] = /var/log/fpm-php.www.log
php_admin_flag[log_errors] = on
catch_workers_output = yes

This is the output of the nginx' access.log:

/var/www/nginx/index.php > GET /index.php HTTP/1.1

And that's the actual error.log:

2018/08/29 17:34:27 [error] 24020#24020: *47 FastCGI sent in stderr: "Unable to open primary script: /var/www/nginx/index.php (No such file or directory)" while reading response header from upstream, client: 213.61.37.18, server: example.com, request: "GET /index.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php-fpm-www.sock:", host: "example.com:8080"

today I had problems accessing the SMB-share on my Linkstation. My NAS has two hard drive disks, configured as raid0. this raid was mounted to /mnt/array1/intern - the folder that i am missing.

My first problem is, I really dont know, where to look for some error reporting.

Lets start with /var/log/messages, its says:

/usr/local/bin/hdd_check_normal.sh: mount -t xfs /dev/md2 /mnt/array1 failed.

Ok. I googled this message and tried the following:

cat /proc/mdstat
md2 : inactive sda6[1](S)
      1938311476 blocks super 1.2

md1 : active raid1 sda2[1]
      4999156 blocks super 1.2 [2/1] [_U]

md10 : active raid1 sda5[1]
      1000436 blocks super 1.2 [2/1] [_U]

md0 : active raid1 sda1[1]
      1000384 blocks [2/1] [_U]

unused devices: <none>

OK... from df -h i know, that md0 is my boot-partition and md1 is the root-partition. i guess md2 is my missing raid - but whats raid10 for? However, i tried to refresh mdadm's configuration and reassembling the raids with:

mdadm --examine --scan > /etc/mdadm.conf
mdadm --assemble --scan -v

This leads to a couple of error messages, like:

cannot open device /dev/md/1: Device or resource busy
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/mtdblock0

for sda, sda1, sda2, md/1, md/2 and son on. Its around 50 lines, I dont want to post them all. What i dont understand is "wrong uuid" - didn't i recently added the current UUIDs to mdadm.conf?

Back to my /var/log/messages i found a script. i tried to manually start them, hoping to get some more error messages:

/etc/init.d/start_data_array.sh

It gives me a whole bunch of messages, the most important are - IMHO:

mount: mounting /dev/md2 on /mnt/array1 failed: Input/output error
umount: forced umount of /mnt/array1 failed!
umount: cannot umount /mnt/array1: Invalid argument

So, the problem that i have is, as far is i know, something is wrong with my raid0-array named md2.

The main question is: What is wrong? How do I activate /dev/md2? (mdadm --detail /dev/md2 gives "device is not active?) Do i manually have to re-create the array? Will i lose my data?

The error, that this device is not active seems kind of generic to me, when looking it up i find a lot of posts and advices that are not really related to my problem.

Any help is appreciated, thanks a lot!

// UPDATE

Its getting strange - for me. This is what fdisk -l is saying for /sda and /sda6:

root@OoompaLoompa:~# fdisk -l /dev/sda

Disk /dev/sda: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1      243202  1953514583+ ee EFI GPT
Disk /dev/sda6: 1984.8 GB, 1984832000000 bytes
255 heads, 63 sectors/track, 241308 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sda6 doesn't contain a valid partition table

/sda6 has no partition table, because its party of my array, i guess. /sda has an partition table, but no superblock:

mdadm --examine /dev/sda
mdadm: No md superblock detected on /dev/sda

But its one of the 2 GB hdd. I am really confused. This is the output of --examine for both of those devices:

/dev/sda1:
        mdadm: No md superblock detected on /dev/sda.
/dev/sda6:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 41e67f06:3b93cda0:46ac3bd7:96702dae
           Name : UNINSPECT-EMC36:2
  Creation Time : Thu Oct 18 01:43:39 2012
     Raid Level : raid0
   Raid Devices : 2

 Avail Dev Size : 3876622952 (1848.52 GiB 1984.83 GB)
  Used Dev Size : 0
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 1dd1c5c5:220d14bf:5b0b1fc5:dbcc3f9c

    Update Time : Thu Oct 18 01:43:39 2012
       Checksum : 5d34dcac - correct
         Events : 0

     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing)

I'm still kinda confused. should /sda be the boot partition? I guess the solution is to somehow re-create the superblock and then re-assemble /md2.

Stil, any help is highly appreciated :)