Low Kian Seong's questions

I am running an ETL process using cron on a daily basis on my server every 2 hourly. The ETL process populates a reporting database which runs Greenplum. I have noticed that the ETL process normally will get stalled and "Idle in Transactions" usually is the one holding it up. For these kinds of processes, how do I go about investigating what kinds of services it originated from ? I am guessing but I am not really sure, as when I run "sudo /etc/init.d/apache2 graceful" it usually clears the Idle transactions off.

I am running the ETL process on an Ubuntu server using the Sun Java. Would appreciate some debugging techniques or solutions to improve the process.

I am running a production gentoo Linux machine, and recently there was a situation where the server hung in my co-located premises and when I got there I noticed that the server was hung on what appeared to be a kernel panic hang. I rebooted the machine with a hard reboot and was disappointed to find out that I could not find a shred of evidence anywhere on why the machine hung.

Is it true that when I do a hard reboot the messages itself will get lost or is there a setting I can do somewhere say in syslog-ng or maybe in sysctl to at least preserve the error log so that I can prevent such mishaps from happening in the future ? I am running a 2.6.x kernel by the way.

Thanks in advance.

I have a server which is running a few discrete Python, Java application which most of the time imports data into a PostGreSQL database. I would like to know from people out there who have experience tuning enterprise grade servers how do i go about calculating in a holistic way the amount of tuning needed for my server for example vm.swappiness, vm.overcommit_ratio and other numerical tunings needed for my server.

I tried to enable sar on my server to capture daily numbers but these are more along the lines of total numbers and I can't figure out how to allocate memory for my applications. Help would be appreciated.

Thanks.

I have been reading about this: https://github.com/epa/lrzip and it seems that lrzip gets more bang for your money. My initial tests however showed takes quite long using lrztar. Any one here have experience using this on live servers?

Thanks.

I run a few applications which creates their own logs. Then I run cron scripts on the same server to do importing of data for my app. When these cron errors out, the default is it sends emails to the user that runs the cron job.

There are just too many places that I need to check the logs and mails for stuff that might have potentially went wrong. My question is, what is the best way to do this or even better is like a log parser application which will go through all the system logs when something really goes wrong instead of me having to go through it daily?

I am running a wsgi based application that has concurrent users accessing it. For my logs needs I tried logrotate but found that logrotate is not too friendly to Python applications, so I tried using RotatingFileHandler and even worse found my logs all chopped up and part of it went missing!

I am considering ConcurrentRotatingFileHandler, my question is, has anyone out there experienced the same thing and better yet do you have any battle tested solution for Python wsgi, concurrently accessed applications?