BotskoNet's questions

I had some very unimportant, largely abandoned mongo-based sites running on machine that's typically fairly well locked down, but after migrating them to this machine, the default mongo port was left open.

Unfortunately, I had never enabled query logging, so I'm trying to find any other possible avenues of finding out if this was the case.

I've looked through ~/.dbshell though I double that will capture commands/queries run by remote clients, since it's tied to the system user. There's no evidence there.

I don't see any indication that mongo keeps an access log, so is there any other possible way for me to tell:

• Whether anyone connected from outside localhost
• Which commands they may have run

I doubt there's any hope at this point.

Suddenly I'm no longer able to resume screens I have running on a remote machine. I've never had this happen in a few years of running this machine. I don't know that it's connected but I did run yum update last night.

I can see the screens listed and they're properly detached:

18709.bungee    (Detached)
5462.mc (Detached)
7045.ftb    (Detached)

Yet when I try to resume any of them, all I get is the cursor on the next line. Any attempt to exit or detach fails and I have no choice but to close my ssh connection entirely.

I have some services running in those screens that I need to gracefully shutdown before I try rebooting the server itself.

Is there any known cause of this? Or any way for me to get into a screen?

We have a 64GB SSD drive in a tower server with a local colocation company. This drive and the enter system was built about six months ago, brand new parts.

Until this weekend the SSD/system were working perfectly. We're running CentOS 6.2

After booting perfectly, the system can be used about 20-30 minutes (no real consistency with time) before the drive starts acting funny.

Libraries start saying they can't load, ssh starts denying public key logins. Shutdown starts saying "input/outout error". Some programs start indicating the drive is read-only.

Only 25GB of the 64GB are used.

I can't find any errors that indicate what happened. I tried running fsck from a live cd on the drive and it showed no problems and most of the time boot works fine. There was one boot that said "couldn't find os" but that's not happening anymore.

Where can I look to find logs about what happens? Are there any other disk checks I should do? It seems like a repairable problem, and not that I need a new drive.

Update:

I enabled SMART after rebooting the server. After 1 hour of uptime and normal system operation (running services are httpd, mysql, but very little to no traffic), suddenly things just stop working. During the hour of uptime it responded with a PASS for the smart health check. After the hour I tried it again (through webmin) and it now says SMART is disabled.

The hard drive is now showing the same issues I've seen before - trying most commands show "input/output error".

Running a smart health check now shows:

Log Sense failed, IE page [scsi response fails sanity test]

What can I do to figure out what's causing this to fail after a random period of time? It runs perfectly for 30-60 minutes and then it starts acting odd like this.

Update 2

Some requested that I try dmesg and this was the result: http://www.pastie.org/private/hk7jfhxilj7ypy828irna. Someone else recommended that I not assume it's the drive, but possibly the drive controller. I don't understand how to determine whether the error is the controller versus the drive - aside from trying a different drive. If I have to buy a replacement motherboard or drive I need to know which is failing first.

Running fsck shows:

fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/mapper/vg_192-lv_root

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

I've been running a CentOS6 server with a local colocation company for several months. I had OpenSSL setup to allow only my keys when logging in and I had no issue until recently. I also have webmin and apache running on this system.

Last night, all of the services shut down and now my ssh keys are being rejected. However, communication to the server is still there - so I know that the network wasn't disconnected, etc.

I'm the only user who is allowed to login, and it's via private key. I'm not an expert but I had the server locked down fairly well, but when I try to login i see:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I'm stuck with what to do. I can't access webmin (it's not running) so I can't reboot the server. Being a weekend, and a night/early morning it's impossible to get a hold of someone at this colocation center to reboot it for me.

Do I have any choices other waiting until I can get someone to reboot it, or Monday when I can drive down there and access it locally myself? Could some sort of software crash/error be causing this private key error or should I be concerned that someone gained access to the server and removed my keys?

P.S. The last thing I saw before I closed my last connection yesterday, was when I tried to resume a screen, it said:

screen: error while loading shared libraries: libutempter.so.0: cannot open shared object file: No such file or directory

I've never seen that before. After that I disconnected and attempted to re-connect and have been getting the permission denied ever since.

I've been trying to figure this out for an hour... how to get a total bandwidth graph (showing all network traffic) in Munin? I just installed Munin, got several plugins added but what I'd really like to see is this:

http://www.schettino.us/monitor/schettino.us/BandwidthMon/totband.html

I don't want to just track apache, because it's for a game server. I've been using vnstat but I really to see long-term graphs.

A previous question was closed because of this:

how to get a total bandwidth graph in munin

But the info here doesn't work. I've tried it already - the first example only looks at http bandwidth and it's not what I want and the second example kills my munin install as an invalid config option.