Fox's questions

root@openwrt:~# ip -s -s -4 neigh show dev lan
10.64.42.121 lladdr b8:20:00:00:00:00 used 6387/6341/6313 probes 1 STALE
10.64.42.157 lladdr b8:20:00:00:00:00 used 24/813/19 probes 1 STALE
10.64.42.12  used 29066/30229/29063 probes 6 FAILED
10.64.42.1 lladdr e8:00:00:00:00:00 ref 1 used 10/5/5 probes 1 REACHABLE


root@openwrt:~# cat /proc/sys/net/ipv4/neigh/default/gc_interval 
30

root@openwrt:~# cat /proc/sys/net/ipv4/neigh/default/gc_stale_time
60

root@openwrt:~# cat /proc/sys/net/ipv4/neigh/lan/gc_stale_time
60

A host in the lan (b8:20:00:00:00:00) had IP address 10.64.42.121. This IP is now invalid and this same host's IP is now 10.64.42.157 (new DHCP lease).

I try to figure out when the old arp cache entry will change state to FAILED (providing nobody attempts to contact the IP).

The last time the entry has been confirmed is 6341s ago (1h45 ago). This is greater than 60s. Why is this entry still in STALE state and when will it change to FAILED state (or be deleted) (if nobody ever tries to use the entry)?

I read everywhere that it is dangerous to do traffic filtering in the nat table because the nat table is only consulted for connections whose state is "NEW" (later packets bypass the table).

Does it mean that nat table counters are only incremented for the first packet of every connection?

Should I then use the chain PREROUTING of the table RAW if I need reliable traffic information?

I'm looking for a nice way to generate a backup tar.gz of what is going to be overridden by the extraction of another tar.gz.

tar -ztf patch.tar.gz | grep -v "/$" | tar -T- -zcvf backup.tar.gz

This works perfectly! However, sometimes, the new patch (patch.tar.gz) will not only contain files already existing in the HDD, but also may add new files which don't already exist. These files which don't exist will be impossible to backup, and the second tar will generate an error. The exit code is 2.

tar: folder/file.txt: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous error

I'm looking for shell command which only checks for file existence (stdin) et send existing files on stdout. Does it exist? I would need something like this:

tar -ztf patch.tar.gz | grep -v "/$" | filterfileexist | tar -T- -zcvf backup.tar.gz

I need one shell command and not a shell script.
Obviously, I could implement this shell command in C myself, because it's very simple, but I hope to find something generic which will be found on other UNIX platforms.

The MX server in my domain example.com point to my Postfix server. However, I have also configured a Google Apps account responsible for emails of example.com.

I want all emails to go to my Postfix server (this is already done by the MX setting in the DNS zone), but want some users of this domain to be forwarded to the MX aspmx.l.google.com.

I did this with the transport_maps parameter. Here is the file /etc/postfix/transport.

[email protected] smtp:[aspmx.l.google.com]

The problem is, smtpd_reject_unlisted_recipient is set to yes by default, but postfix checks existence of user before applying the transport map. My Postfix server doesn't know the user test for this domain, so it simply rejects the email.

I would like to make transport routing happen before recipient existence check (in my current setting, only Google SMTP is aware of existence of user test). Is that possible?

Note: local_recipient_maps doesn't work for me because I use the virtual transport with dovecot. So the recipient table is virtual_mailbox_maps.

The problem

The ifconfig command shows more and more dropped packets in the RX section. So, there seem to be a problem to some packets arriving from the Internet to my server.

The questions

1. What kind of packets does this drop counter takes into account? Does it take all packets arriving, before reaching the iptables firewall, or after the packets have been accepted by iptables?

2. How to solve the situation so that the ipconfig drop packets counter stops to increase?

Useful troubleshooting infos

Since I don't know what my problem really is, feel free to ask me to complete this section if you think some other info would be needed.

ifconfig

eth0      Link encap:Ethernet  HWaddr 00:cc:cc:cc:cc:cc  
          inet adr:90.0.0.2  Bcast:90.0.0.255  Masque:255.255.255.0
          adr inet6: fe80::21c:c0ff:feb9:829c/64 Scope:Lien
          adr inet6: 2001:a100:1:bbbb::1/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:113264620 errors:0 dropped:2523 overruns:0 frame:0
          TX packets:168526529 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          RX bytes:59171827564 (55.1 GiB)  TX bytes:223993117711 (208.6 GiB)

Note the "dropped:2523" in the RX section. This is the most important. This number is continuously increasing.

ip -4 route show

default via 90.0.0.254 dev eth0 
90.0.0.0/24 dev eth0  proto kernel  scope link  src 90.0.0.2

ip -6 route show

2001:a100:1:bbbb::1/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
default via 2001:a100:1:bbff:ff:ff:ff:ff dev eth0  metric 1024

munin graph of plugin if_err_eth0_day

• I'm using cronolog to let the apache log files rotate.

• I'm using fail2ban to monitor those log files and ban in case of abuse.

• There's always an access.log symlink which is created by cronolog and points to the current log file.

• Each night, I run a cron to compress yesterday's logfiles :

  find /var/log/apache2/ -daystart -mtime +0 \( -name "*access*.log" -or -name "*error*.log" \) -type f -exec gzip {} \;
  

The problem is that, for website with low traffic, the access.log symlink now is dead because the file it was pointing to has been renamed by gzip. The result is that fail2ban gives up those jails because it cannot stat the files anymore.

The solution is to only compress the log files for which there is no symlink pointing to.

I found myself a way of doing so, and it works, but I would like to know if there is an easier way to do, because this one is really complicated, and not very fast (a find in a find).

    find /var/log/apache2/ -daystart -mtime +0 \( -name "*access*.log" -or -name "*error*.log" \) -type f -exec sh -c 'test `find /var/log/apache2/ -lname {} | wc -l` -eq 0' \; -exec gzip {} \;

I'm using Debian Wheezy on a dedicated server which uses IPv4 and IPv6 at the same time.

ifconfig eth0

eth0      Link encap:Ethernet  HWaddr 11:11:11:11:11:11  
          inet adr:90.0.0.1  Bcast:90.0.0.255  Masque:255.255.255.0
          adr inet6: fe80::21c:c0ff:feb9:829c/64 Scope:Lien
          adr inet6: 2001:a100:1:bbbb::1/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1544 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2199 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          RX bytes:304146 (297.0 KiB)  TX bytes:1376282 (1.3 MiB)

(Several minutes later, the RX dropped counter increases.)

cat /etc/network/interfaces

I first used the "gateway" keyword, but it didn't work, so I then tried to add the route in the file, but it didn't work either.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 90.0.0.1
    netmask 255.255.255.0
    network 90.0.0.0
    broadcast 90.0.0.255
    gateway 90.0.0.254

iface eth0 inet6 static
    address 2001:a100:1:bbbb::1
    netmask 64
#   gateway 2001:a100:1:bbff:ff:ff:ff:ff
    up ip -6 route add ::/0 via 2001:a100:1:bbff:ff:ff:ff:ff dev eth0
    down ip -6 route del ::/0 via 2001:a100:1:bbff:ff:ff:ff:ff dev eth0

ip -6 route show

These are the routes shown several minutes after rebooting the server.

2001:a100:1:bbbb::/64 dev eth0  proto kernel  metric 256 
2001:a100:1:bb00::/56 dev eth0  proto kernel  metric 256  expires 2591959sec
fe80::/64 dev eth0  proto kernel  metric 256 
default via fe80::224:c3ff:fe84:400 dev eth0  proto ra  metric 1024  expires 1759sec
default via fe80::224:c3ff:fe84:800 dev eth0  proto ra  metric 1024  expires 1759sec

Questions

1. Why isn't my default route taken into account (no matter using keyword gateway or ip -6 route add in the /etc/network/interfaces, during boot time? Note that if I add the route manually, it works perfectly and I can TX and RX to and from the Internet using IPv6. (The problem is to make it automatic.)

2. What are those strange kernel routes, that will expire? Do I need them? My server is connected directly to the same network as a rooter (2001:a100:1:bbff:ff:ff:ff:ff), which is itself connected directly to the internet.

EDIT

ifdown --force eth0; ifup --verbose eth0;

Configuring interface eth0=eth0 (inet)
run-parts --verbose /etc/network/if-pre-up.d
ip addr add 90.0.0.1/255.255.255.0 broadcast 90.0.0.255       dev eth0 label eth0
ip link set dev eth0   up
 ip route add default via 90.0.0.254  dev eth0 
run-parts --verbose /etc/network/if-up.d
run-parts: executing /etc/network/if-up.d/avahi-daemon
run-parts: executing /etc/network/if-up.d/bind9
run-parts: executing /etc/network/if-up.d/mountnfs
run-parts: executing /etc/network/if-up.d/ntpdate
run-parts: executing /etc/network/if-up.d/openssh-server
run-parts: executing /etc/network/if-up.d/openvpn
run-parts: executing /etc/network/if-up.d/postfix
run-parts: executing /etc/network/if-up.d/upstart
Configuring interface eth0=eth0 (inet6)
run-parts --verbose /etc/network/if-pre-up.d
modprobe -q net-pf-10 > /dev/null 2>&1 || true # ignore failure.


sysctl -q -e -w net.ipv6.conf.eth0.autoconf=0
ip link set dev eth0   up
ip -6 addr add 2001:a100:1:bbbb::1/64  dev eth0 
 ip -6 route add default via 2001:a100:1:bbff:ff:ff:ff:ff dev eth0 
RTNETLINK answers: File exists
Failed to bring up eth0.

I use munin, the monitoring program, to alert me by email when something goes wrong on my server. It sends emails as munin.

Normally, when an email like this is to be delivered on the Internet, when postfix/cleanup is called, it asks trivial-rewrite to do a rewrite, which should append realdomain.tld to incomplete sender address.

It seams it does not, because my Postfix SMTP client then tries to send the email containing the From: munin@thehostname header, which Google doesn't really like:

Jul 22 10:00:15 thehostname postfix/pickup[3167]: 979BD1E447E: uid=110 from=<munin@thehostname>
Jul 22 10:00:15 thehostname postfix/cleanup[9609]: 979BD1E447E: message-id=<[email protected]>
Jul 22 10:00:15 thehostname postfix/qmgr[3276]: 979BD1E447E: from=<munin@thehostname>, size=454, nrcpt=1 (queue active)
Jul 22 10:00:16 thehostname postfix/smtp[9611]: 979BD1E447E: to=<[email protected]>, relay=ASPMX.L.GOOGLE.COM[173.194.67.27]:25, delay=0.49, delays=0.07/0/0.12/0.3, dsn=5.7.1, status=bounced (host ASPMX.L.GOOGLE.COM[173.194.67.27] said: 550-5.7.1 [9.8.7.6       1] Our system has detected an unusual rate of 550-5.7.1 unsolicited mail originating from your IP address. To protect our 550-5.7.1 users from spam, mail sent from your IP address has been blocked. 550-5.7.1 Please visit http://www.google.com/mail/help/bulk_mail.html to review 550 5.7.1 our Bulk Email Senders Guidelines. x12si10440172wia.27 - gsmtp (in reply to end of DATA command))
Jul 22 10:00:16 thehostname postfix/cleanup[9609]: 372F91E4483: message-id=<[email protected]>
Jul 22 10:00:16 thehostname postfix/qmgr[3276]: 372F91E4483: from=<>, size=3167, nrcpt=1 (queue active)
Jul 22 10:00:16 thehostname postfix/bounce[9614]: 979BD1E447E: sender non-delivery notification: 372F91E4483
Jul 22 10:00:16 thehostname postfix/qmgr[3276]: 979BD1E447E: removed
Jul 22 10:00:16 thehostname postfix/local[9617]: 372F91E4483: to=<munin@thehostname>, relay=local, delay=0.13, delays=0.05/0.01/0/0.06, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION")
Jul 22 10:00:16 thehostname postfix/qmgr[3276]: 372F91E4483: removed

I don't know why the trivial-rewrite doesn't work as expected for the munin user and why doesn't it get a proper sender address before the email is sent on the internet.

Other programs like fail2ban succeed to send emails and are rewrote to [email protected]...

Here is the interesting part of my configuration:

myhostname = realdomain.tld
mydomain = thehostname
mydestination = localhost.localdomain, localhost, $mydomain, $myhostname
myorigin = $myhostname
mynetworks =
relayhost =

I think there must be something I don't understand in the rewrite process!

EDIT

I found this on an article talking about postfix addresses rewriting.

Rewrite user to user@$myorigin

This feature is controlled by the boolean append_at_myorigin parameter (default: yes). The purpose is to get consistent treatment of user on every machine in $myorigin.

I haven't changed the append_at_myorigin parameter, which should be set "yes". $myorigin is set to realdomain.tld, but the rewriting goes wrong and choose $mydomain instead.

Recently, I decided to DROP packets that want to go out through the port 80. It seams my configuration has a problem, because some unwanted packets are dropped.

Excerpt of my configuration:

iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -p tcp -s [PUBLIC IP OF MY SERVER] --sport 80 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Question 1: isn't the second rule useless, since I already said in the first one that I accept all packets with "ESTABLISHED" state?

Question 2: Why aren't these two rules enough to accept the following dropped packets:

Jul 14 18:47:18 [HOSTNAME] kernel: iptables output: IN= OUT=eth0 SRC=[PUBLIC IP OF MY SERVER] DST=[A WWW CLIENT PUBLIC IP] LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=8408 DF PROTO=TCP SPT=80 DPT=50085 WINDOW=123 RES=0x00 ACK FIN URGP=0 
Jul 14 18:47:53 [HOSTNAME] kernel: iptables output: IN= OUT=eth0 SRC=[PUBLIC IP OF MY SERVER] DST=[A WWW CLIENT PUBLIC IP] LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=8409 DF PROTO=TCP SPT=80 DPT=50085 WINDOW=123 RES=0x00 ACK FIN URGP=0 
Jul 14 18:48:08 [HOSTNAME] kernel: iptables output: IN= OUT=eth0 SRC=[PUBLIC IP OF MY SERVER] DST=[A WWW CLIENT PUBLIC IP] LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=54091 DF PROTO=TCP SPT=80 DPT=25780 WINDOW=16616 RES=0x00 ACK FIN URGP=0

N.B:

• There is no rule above those in the chain that drops packets.
• The default policy is DROP.

EDIT I looked at this post, and also enabled logging of INVALID packets by the kernel:

echo 255 >/proc/sys/net/netfilter/nf_conntrack_log_invalid

Now it seams I have several kinds of errors:

Jul 14 22:00:40 [HOSTNAME] kernel: nf_ct_tcp: invalid RST IN= OUT= SRC=[ONE_CLIENT_IP] DST=[SERVER_IP] LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=47149 PROTO=TCP SPT=993 DPT=51364 SEQ=1043042446 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 
Jul 14 21:57:11 [HOSTNAME] kernel: nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=[SERVER_IP] DST=[ONE_CLIENT_IP] LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=3782 SEQ=474588492 ACK=2243291425 WINDOW=14600 RES=0x00 ACK SYN URGP=0 OPT (020405B401010402)
Jul 14 21:57:25 [HOSTNAME] kernel: nf_ct_tcp: invalid packet ignored in state LAST_ACK IN= OUT= SRC=[SERVER_IP] DST=[ONE_CLIENT_IP] LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=3782 SEQ=474588492 ACK=2243291425 WINDOW=14600 RES=0x00 ACK SYN URGP=0 OPT (020405B401010402) 
Jul 14 21:57:41 [HOSTNAME] kernel: nf_ct_tcp: invalid packet ignored in state TIME_WAIT IN= OUT= SRC=[SERVER_IP] DST=[ONE_CLIENT_IP] LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=3782 SEQ=474588492 ACK=2243291425 WINDOW=14600 RES=0x00 ACK SYN URGP=0 OPT (020405B401010402)
Jul 14 21:58:52 [HOSTNAME] kernel: nf_ct_tcp: invalid packet ignored in state SYN_RECV IN= OUT= SRC=[SERVER_IP] DST=[ONE_CLIENT_IP] LEN=48 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=50488 SEQ=3804975135 ACK=229029122 WINDOW=14600 RES=0x00 ACK SYN URGP=0 OPT (020405B401010402) 

A website I'm hosting on my dedicated server has been hacked. Some pictures have been infected with things like <?php eval(base64_decode(....));?>. I'm looking for a generic shell command which would able to read the headers of the most common image types (jpg, gif, tiff, etc.).

I tried jhead and others, jut jhead can only read EXIF data from JPG.

The aim is to find all infected images like this :

find -type f \( -iname "*jpg" -or -iname "*.jpeg" -or ... \) -exec sh -c 'magicimgheadersdisplay "{}" | grep eval" \;

I have three questions related to the "ctime" information of a file. They are in bold (if you don't have time you can jump directly to them).

A website on my dedicated server has been hacked. Lots of files have been edited. I noticed that some hackers were clever enough to reset the mtime (some not).

atime and mtime can be changed with PHP functions. ctime can also be changed by using the chmod function. But ctime is handled by the kernel, so it's only possible to set it to the current system date/time. It's impossible to set it back to the past.

I found some backdoors whose mtime had been reset to the ctime (date in the past, like 1 or 2 years ago), so that I can't find it using recursive search tool based on that information.

The key to be able do find quickly all infested files is of course the ctime information. The problem is that I didn't think of it at first, so I first "chmoded" recursively all the files so that the website itself doesn't have write permissions anymore. In doing this, I lost the precious ctime information.

I have lots of backups, in form of .tar.gz files. I want to know if there is a way to extract from the tar.gz file the ctime of the files of the moment they were added the the tar.

I read the whole GNU tar documentation, and found out that the "GNU" format of tar (which is the default used by my Linux server) stores the ctime of its archived files. I tried following way to extract it:

tar -zxf Friday.tar.gz --to-command=./script

./script is a bash script and looks like this:

#!/bin/bash
echo $TAR_CTIME

The problem is that this gives the current date and time, maybe because tar gives out the informations of the file it would itself create if I hadn't used --to-command

If tar cannot handle this, is there backup tools which can keep original ctime information?

How to list every file whose mtime and ctime differ? I read the find manual but only found out the -newerXY, which (if I understand well), can only compare a ctime with a ctime, a mtime with a mtime, etc. Is there a way to compare a ctime and a mtime of the same file?