lashgar's questions -server

lashgar

Asked: 2012-12-25 01:00:32 +0800 CST

Execution time differs for different users

1

I am running simple R job by root and another limited user. The execution time differs significantly. What can be the source of problem?

Further information

Here is how I compare the run time:

# time /share/binary/R/bin/R CMD BATCH s1n\=50.R

real    0m0.278s
user    0m0.217s
sys 0m0.032s
# su john
$ time /share/binary/R/bin/R CMD BATCH s1n\=50.R

the run under john user takes a long time and never finishes! The output of perf during these interval is:

   PerfTop:     906 irqs/sec  kernel:19.3%  exact:  0.0% [1000Hz cycles],  (all, 8 CPUs)
-------------------------------------------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                      DSO
             _______ _____ _____________________________ _______________________________

              598.00 14.5% __GI_vfprintf                 /lib64/libc-2.12.so            
              194.00  4.7% intel_idle                    [kernel.kallsyms]              
              176.00  4.3% read_hpet                     [kernel.kallsyms]              
              170.00  4.1% bcEval                        /usr/local/R/lib64/R/bin/exec/R
              141.00  3.4% ___printf_fp                  /lib64/libc-2.12.so            
              138.00  3.4% __strchrnul                   /lib64/libc-2.12.so            
              121.00  2.9% Rf_cons                       /usr/local/R/lib64/R/bin/exec/R
              120.00  2.9% R_gc_internal                 /usr/local/R/lib64/R/bin/exec/R
               91.00  2.2% _IO_default_xsputn_internal   /lib64/libc-2.12.so            
               88.00  2.1% Rf_allocVector                /usr/local/R/lib64/R/bin/exec/R
               84.00  2.0% _IO_file_xsputn_internal      /lib64/libc-2.12.so            
               82.00  2.0% scientific                    /usr/local/R/lib64/R/bin/exec/R
               72.00  1.7% MatrixSubset                  /usr/local/R/lib64/R/bin/exec/R
               71.00  1.7% duplicate1                    /usr/local/R/lib64/R/bin/exec/R
               68.00  1.7% floor                         /lib64/libm-2.12.so            
               64.00  1.6% __strcmp_sse42                /lib64/libc-2.12.so            
               53.00  1.3% Rf_findVarInFrame3            /usr/local/R/lib64/R/bin/exec/R
               53.00  1.3% Rf_protect                    /usr/local/R/lib64/R/bin/exec/R
               50.00  1.2% _IO_str_init_static_internal  /lib64/libc-2.12.so            
               50.00  1.2% Rf_eval                       /usr/local/R/lib64/R/bin/exec/R
               43.00  1.0% Rf_formatReal                 /usr/local/R/lib64/R/bin/exec/R
               43.00  1.0% Rf_matchArgs                  /usr/local/R/lib64/R/bin/exec/R
               41.00  1.0% _int_malloc                   /lib64/libc-2.12.so            
               36.00  0.9% _itoa_word                    /lib64/libc-2.12.so            
               33.00  0.8% __ieee754_log                 /lib64/libm-2.12.so            
               31.00  0.8% Rf_EncodeReal                 /usr/local/R/lib64/R/bin/exec/R
               29.00  0.7% Rf_mkPROMISE                  /usr/local/R/lib64/R/bin/exec/R
               29.00  0.7% do_bind                       /usr/local/R/lib64/R/bin/exec/R
               28.00  0.7% Rf_install                    /usr/local/R/lib64/R/bin/exec/R
               27.00  0.7% __vsnprintf                   /lib64/libc-2.12.so            
               27.00  0.7% _IO_no_init                   /lib64/libc-2.12.so            
               23.00  0.6% _IO_old_init                  /lib64/libc-2.12.so            
               22.00  0.5% __GI__nss_files_parse_servent /lib64/libnss_files-2.12.so    
               22.00  0.5% Rconn_printf                  /usr/local/R/lib64/R/bin/exec/R
               22.00  0.5% finite                        /lib64/libm-2.12.so            
               22.00  0.5% findVarLocInFrame             /usr/local/R/lib64/R/bin/exec/R
               21.00  0.5% Rf_getAttrib                  /usr/local/R/lib64/R/bin/exec/R

I suspect to ulimit and ``disk quota. After disabling the disk quota, the problem is still exist. Unfortunately, limits are equal underrootandjohn. Here is the output ofulimit -a` (thanks to @Eric DANNIELOU):

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127383
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
# su john
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127383
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Operating System: CentOS 6.2

HW: Intel Core-i7 16GB RAM

Thank you in advance!

lashgar

Asked: 2012-11-25 10:55:00 +0800 CST

Timely automatic unexpected reboot on ubuntu desktop

1

We have a remote linux server (ubuntu desktop). The system log indicates the system has been restarted on the timely fashion. Here is a part of the last output:

ut       pts/0        192.169.50.2-sta Sat Nov 24 22:17   still logged in
reboot   system boot  2.6.32-21-generi Sat Nov 24 22:04 - 22:17  (00:13)
ut       pts/0        server.local     Sat Nov 24 21:36 - crash  (00:27)
reboot   system boot  2.6.32-21-generi Sat Nov 24 15:55 - 22:17  (06:21)
reboot   system boot  2.6.32-21-generi Fri Nov 23 18:02 - 22:17 (1+04:14)
reboot   system boot  2.6.32-21-generi Fri Nov 23 10:39 - 22:17 (1+11:38)
reboot   system boot  2.6.32-21-generi Fri Nov 23 04:18 - 22:17 (1+17:59)
reboot   system boot  2.6.32-21-generi Fri Nov 23 03:57 - 22:17 (1+18:20)
reboot   system boot  2.6.32-21-generi Thu Nov 22 20:38 - 22:17 (2+01:38)
reboot   system boot  2.6.32-21-generi Thu Nov 22 11:13 - 22:17 (2+11:03)
reboot   system boot  2.6.32-21-generi Thu Nov 22 08:12 - 22:17 (2+14:05)
reboot   system boot  2.6.32-21-generi Wed Nov 21 11:16 - 22:17 (3+11:00)
reboot   system boot  2.6.32-21-generi Tue Nov 20 22:36 - 22:17 (3+23:41)
reboot   system boot  2.6.32-21-generi Tue Nov 20 14:12 - 22:17 (4+08:05)
reboot   system boot  2.6.32-21-generi Tue Nov 20 11:32 - 22:17 (4+10:44)
reboot   system boot  2.6.32-21-generi Tue Nov 20 01:52 - 22:17 (4+20:25)
reboot   system boot  2.6.32-21-generi Tue Nov 20 00:22 - 22:17 (4+21:55)
reboot   system boot  2.6.32-21-generi Mon Nov 19 17:27 - 22:17 (5+04:50)

It looks the system is set to be restarted at 22:17. Can anyone guide me why this happens?

Thanks in advance.

lashgar

Asked: 2012-11-16 00:39:06 +0800 CST

Quota: apply the same limit to all individual users of a group

4

I need to limit the disk usage of 10s of users. All the target users belong to the same group.

It looks edquota -g <group-name> enforces the limit across all users of the group. In other words, that command let all users of the group altogether allocate specified amount of disk. I want the restriction to be applied to each individual user independently. Is there any option in edquata for this goal?

The naive solution: Repeating the edquoata -u <user-name> for each user one-by-one.

Thanks in advance.

lashgar

Asked: 2012-11-14 00:38:31 +0800 CST

Linux: Finding the system processes which are at the top

2

I am running a multithreaded program on my 8-core processor. top command shows 8 processes are running and each one utilizes a core by 100%. The problem is that only 21% are marked as user cpu and my application is running so slow:

Cpu(s): 20.6%us, 79.4%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

How can I find further information about the system processes those which have taken 79% of the CPU time? Specifically, I want to know what are these system calls.

Thanks in advance!

EDIT: This is what top returns:

top - 11:23:04 up 83 days,  1:20, 10 users,  load average: 8.00, 8.00, 8.00
Tasks: 267 total,  10 running, 257 sleeping,   0 stopped,   0 zombie
Cpu(s): 40.6%us, 17.4%sy,  0.0%ni, 41.9%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  16324640k total,  5645008k used, 10679632k free,   285368k buffers
Swap:  2047992k total,    32064k used,  2015928k free,  4584140k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                         
27262 ahmad     20   0  308m  32m  18m R 102.2  0.2   5022:32 Espresso                                                       
26872 ahmad     20   0  176m  12m 4540 R 100.0  0.1   5087:05 Espresso                                                       
27263 ahmad     20   0  175m  13m 4824 R 100.0  0.1   5022:35 Espresso                                                       
27265 ahmad     20   0  175m  12m 4508 R 100.0  0.1   5022:35 Espresso                                                       
26869 ahmad     20   0  308m  31m  18m R 98.3  0.2   5086:20 Espresso                                                        
26870 ahmad     20   0  175m  13m 4864 R 98.3  0.1   5087:05 Espresso                                                        
26871 ahmad     20   0  176m  12m 4592 R 98.3  0.1   5086:58 Espresso                                                        
27264 ahmad     20   0  176m  12m 4552 R 98.3  0.1   5022:06 Espresso                                                        
    1 root      20   0 19404 1164  940 S  0.0  0.0   0:02.12 init                                                            
    2 root      20   0     0    0    0 S  0.0  0.0   0:02.81 kthreadd

lashgar

Asked: 2012-11-14 00:02:32 +0800 CST

Disk speed test linux

0

Is there any simple command (or tool) to test the speed (MB/sec) of disk access?

I've found hdparm but It look it only works for local disk. I need to evaluate the speed of remote NFS-mounted disk.

Thanks in advance!

lashgar

Asked: 2012-10-08 13:42:19 +0800 CST

Does 'ssh <remote> -t <command>' has a timeout? killed by Signal 15

1

I want to run a job on remote server with no login. I found that I can run my job using ssh as:

ssh remotenode -t /path/to/myjob &> /path/to/log

However, the job would be killed by Signal 15 after a while. Does SSH force some timeout?

NOTE: My jobs takes a few hours to be completed.

lashgar

Asked: 2012-09-25 01:39:48 +0800 CST

Block brute-force attack using lastb and iptables

2

Using linux lastb command, I found that my server is brute-force attacked from many different IPs around the world! I have developed an script to detect brute-force attackers by lastb and block them by iptables. Here is the script:

#!/bin/bash

cd /root/
windowSize=100
tresh=10
lastb | head -n $windowSize | awk '{print $3}' | uniq -c > .ips
nlines=`wc .ips -l | awk '{print $1}'`
END=`expr $nlines - 1 `
for i in `seq 0 $END`;
do
        range=`expr $nlines - $i`
        count=`tail .ips -n $range | head -n 1 | awk '{print $1}'`
        if [ $count -gt $tresh ] ; then
                IP=`tail .ips -n $range | head -n 1 | awk '{print $2}'`
                if [ ! -z .blips ] ; then
                        touch .blips
                fi ;
                blocked=`cat .blips | grep $IP -c`
                if [ $blocked = '0' ] ; then
                        echo blocking $IP
                        iptables -A INPUT -s $IP -j DROP
                        echo $IP >> .blips
                fi ;
        fi;
done
rm .ips

Can it cause any problem if I run this script by crond every hours?

lashgar

Asked: 2012-09-22 23:16:29 +0800 CST

Finding latest successful logins and failed attempts to a CentOS server

37

I'm looking for a log file or any service to report the latest login attempts that have failed due to username/password mismatch. Are there any such utilities available for CentOS? (built-in is preferred)

My second question, and more generally, I need a log file of penetration attempts to my server. Ideally, this log should contain all attempts including logins, httpd activities, and other conventional open ports.

lashgar

Asked: 2012-09-22 23:56:08 +0800 CST

Reporting Penetration Attempts to My Server

2

I need a log file of penetration attempts to my server. Ideally, this log should contain all attempts including logins, HTTPD activities, FTP activities, NFS, and other conventional open ports. However, it is enough for me to have HTTPD, FTP, and NFS.

Is there such service (or log files) available in CentOS (built-in preferred)?

lashgar

Asked: 2012-08-17 01:35:26 +0800 CST

Ubuntu vs. Ubuntu Server

0

I am familiar with ubuntu desktop. What additional features Ubuntu server provides? Is it competitive to high-end servers like RedHat or CentOS?

lashgar

Asked: 2012-07-15 04:00:56 +0800 CST

VMWare ESXi datastore disabled while the system is up

0

Yesterday, I had a ESXi server working. Total memory required by the deployed virtual machines is higher than the physical RAM. The disk also has a few Gigabytes of free capacity.

Today my server stopped normal operation! According the to the performance reports, memory usage exceeded the physical memory and the disk does not have enough free space to swap it.

In 'Configuration > Storage' , the datastore is disabled and is not accessible. SSH connection to ESXi is possible, however, any disk command takes a long time to execute (e.g. df). I tried to halt some VMs to free up some memory. But the 'Power Off' commands stalled at 95% in Recent Task window. I looks the disk faced to critical problem.

Is there anyway to bring the system to normal state without reboot/halt?

Any help is highly appreciated. Thank you so much.

Execution time differs for different users

Timely automatic unexpected reboot on ubuntu desktop

Quota: apply the same limit to all individual users of a group

Linux: Finding the system processes which are at the top

Disk speed test linux

Does 'ssh <remote> -t <command>' has a timeout? killed by Signal 15

Block brute-force attack using lastb and iptables

Finding latest successful logins and failed attempts to a CentOS server

Reporting Penetration Attempts to My Server

Ubuntu vs. Ubuntu Server

VMWare ESXi datastore disabled while the system is up

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?