Questions[logs](ubuntu)

Ubuntu 18.04 lsb, I'm trying to figure out the best way to diagnose whats happening when the Amazon Ec2 instance (free tier) hangs.

There are experimental services running and there may be / is a memory leak.

For quality of life I'm using a utility called lnav to help me browse system logs. Also I've installed a utility called monitorix to visiualise whats happening.

Can I / how do I to identify the specific process causing the problem from system logs ? Which log might help me? (/var/log/syslog does not help)

These charts show high CPU load associated with system swap space being consumed until catastrophic failure occurs.

But this does not tell me the specific process. How can I do this through the terminal?

Is there some other process monitoring I could to configure ?

Any help appriciated...

Edit: Thanks to the hint from @Rinzwind sar is now installed and cron is running it every 2 mins... but it doesn't give process level info. So with help from this other answer:

pidstat 5 > pidhist.log pipes out to a text file, and running it in persistant session will aid diagnosis when the event happens again.

@heynnema suggested iotop

Running iotop -P -a which is top for file I/O as a totaliser. It indicated that the experimental process (a mono service) was the one consuming the most swap with SWAPIN
****

It's more visible in top

We see can see the same pattern of consumption, and then after restarting the process return to normal ~20% from monitorix.

The system is stable for weeks on end between these random events. The evidence from iotop proves the underlying issue is within the experimental process!

Yet, this is still a run time diagnosis. Is there a way to determine from existing logs which process was at fault after the fact? to do that without without preemptive monitoring and logging.

That proof of what went wrong is the critical issue to be resolved. how can we do that without waiting for it to reoccur if no logging is enabled? kernel logs???

Thanks for any help.

I'm trying to figure out when a command I entered stopped executing, for example

sudo apt-get update

I want information like:

started execution at : 12:00
finished execution at : 12:15

So that is my question. I googled it but nothing came. I see that the logs I have only covers 24 hours (if the time is not wrong, because it resets on every boot)

I am very very new to Ubuntu. I wish to know how to see user logs in Ubuntu. I realized that our son has been using his computer on days he's not supposed to. Apparently, he configured the computer that user logs are deleted so I don't see user logs except on the day the computer is turned on. Is there a way to see computer activity and which dates and time? I will really appreciate your responses.

I had installed the Xpad package on Ubuntu 13.10 and had taken a few notes in it. I recently upgraded to Ubuntu 14.04 and have taken the backup of the entire home folder. At which location is the content of the Xpad stored on the disk? If it is somewhere in the home folder, maybe I could retrieve it. Thanks.