mhost's questions

I have a php script that is trying to use exec (or shell_exec) to execute a binary on the system. The exec is failing with return code 127.

Return code 127 normally means command not found. So I made sure to use the absolute path to the binary. No change.

Apache is configure to run in a chroot using apache's ChrootDir.

I've made sure to copy the binary into the proper path in the chroot, as well as /bin/sh, and all the linked libraries needed for both of those.

Apache (and therefore php) are running as www-data. I've confirmed that www-data has read and execute permission to the binaries (including /bin/sh) and all the parent folders. To confirm that it is not a file permission issue, I've run the command using /bin/sh -c using sudo:

sudo -u www-data /chrootdir/bin/sh -c /chrootdir/path/to/binary

And that works without problems.

Using strace, I get this:

execve("/bin/sh", ["sh", "-c", "/path/to/binary"], 0x7ffe436b3618 /* 11 vars */) = -1 EACCES (Permission denied)

Just to confirm that the permission issue is for the sh binary (and the one in the chrootdir) I tried renaming /chrootdir/bin/sh to something else and did the strace again and now it complained about file not found.

So, I now know that the problem is with the access to /chrootdir/bin/sh when run via php through apache, but is not a permission of the www-data user.

I'm not sure what to try next.

This is running on Debian 10, apache 2.4.38, and php 7.3.11.

I've cleared open_basedir, and I've also cleared disable_functions.

I've confirmed that apache is unconfined by apparmor, but disabled it anyways.

Finally, if I disable the apache chroot, this does work.

So my question is there any other restriction somewhere that might be stopping apache from doing this?

I have an NFS server with a mount point that is mounted on several servers. I can access the files on all of them.

On all of the clients, I am getting the following behaviour.

$ ls
_index_ssl.html_old
$ ls -i _index_ssl.html
2966711 _index_ssl.html
$ ls -i _index_ssl.html_old
2966711 _index_ssl.html_old

Notice that ls alone only displays the .html_old file. However, I know the .html file is there because the web server is properly serving the file. And if I vi _index_ssl.html I see the contents of the file properly. Also, notice they have the same inode number.

The old file is created by the web server. This is a Wordpress site that has w3 total cache. And when the cache gets cleared/emptied, it renamed all the files to _old. So the old file was previously .html and then is renamed. Eventually when the cache is created again, a .html file should be visible there.

Another very strange point is that on the NFS server itself, only the .html_old file is there.

$ ls -i _index_ssl.html
ls: cannot access '_index_ssl.html': No such file or directory

Once again, I want to point out that all the NFS clients are showing the same files. So the folder is properly mounted and the file does exist on each client. But on the server, the file is not there. Also, the web server is properly retrieving the _index_ssl.html file. It does not user .html_old files.

The exports file on the server:

/folder <subnet>(rw,sync,no_subtree_check,no_root_squash)

On the clients, the mount command output:

<server_ip>:/folder on /folder type nfs4 (rw,relatime,vers=4.2,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=<client_ip>,local_lock=none,addr=<server_ip>)

I also just tested running the following on one of the web servers and the file (and contents are visible on the nfs server and all the clients.

date > newfile

I've also tried renaming the file and the change appears everywhere as well.

I'm not using the pseudo file system (or parent export) that NFS4 added.

I have setup exim to get autoreply text from an sqlite database. It is sending the autoreply just fine with the exception that I can't get new lines in the autoreply.

This is the line in the transport:

text     = ${lookup sqlite{SQLDB \
              SELECT body FROM autoreply WHERE \
                (username='$local_part' AND domain='$domain') \
              AND (sender ='$sender_address' or sender ='$reply_address' or sender ='*') \
              AND enabled=1 \
              AND (valid_from <= strftime('%s', 'now') OR valid_from IS NULL) \
              AND (valid_to >= strftime('%s', 'now') OR valid_to IS NULL) \
              LIMIT 1\
            }\
          }

As I mentioned, this is returning the proper text, the email just contains the actual characters \n instead of new lines.

I've tried things like \\n and nothing works.

I've been searching online for quite a while and I can't find anything about doing this or anything saying it can't be done.

This morning our database became extremely slow for no apparent reason. There were 2 active queries, one took a little over 30 minutes in "Sending data" and another took almost 10 minutes in "storing result in query cache". There were also many queries "Waiting for table level lock". Once those 2 queries completed, all queries finished up very quickly.

I don't think the queries need optimizing because after this happened, I re-ran those queries manually, and the 30 minute one completed in 14 seconds and the 10 minute one completed in 8 seconds. So it seems like it is just a temporary server issue however, it happened last week as well and once about a month ago.

This server runs nothing except mysql. Any help would be greatly appreciated.

This is a brand new server that we bought because we expect a lot more traffic soon. The old server which had much lower specs never had this problem.

Here are some commands that I ran while it was in this state:

pidstat -dru 1 3:

Linux 3.8.0-29-generic (db1)    02/28/2014      _x86_64_        (32 CPU)

10:16:06 AM       PID    %usr %system  %guest    %CPU   CPU  Command
10:16:07 AM      2907  103.88    1.94    0.00  105.83    17  mysqld
10:16:07 AM     29727    0.97    1.94    0.00    2.91    17  pidstat

10:16:06 AM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
10:16:07 AM      2907    230.10      0.00 17380572 756892   1.15  mysqld
10:16:07 AM     29727    868.93      0.00    7636   1268   0.00  pidstat

10:16:06 AM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command

10:16:07 AM       PID    %usr %system  %guest    %CPU   CPU  Command
10:16:08 AM        10    0.00    1.00    0.00    1.00     5  rcu_sched
10:16:08 AM       462    0.00    1.00    0.00    1.00    31  kworker/31:1
10:16:08 AM      2907  105.00    1.00    0.00  106.00    17  mysqld
10:16:08 AM     29727    1.00    2.00    0.00    3.00    17  pidstat

10:16:07 AM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
10:16:08 AM      2907    177.00      0.00 17380572 756892   1.15  mysqld
10:16:08 AM     29619     28.00      0.00   23304   4616   0.01  bash
10:16:08 AM     29727    895.00      0.00    7636   1300   0.00  pidstat

10:16:07 AM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command

10:16:08 AM       PID    %usr %system  %guest    %CPU   CPU  Command
10:16:09 AM      2907  103.00    2.00    0.00  105.00    17  mysqld
10:16:09 AM     29727    2.00    2.00    0.00    4.00    17  pidstat

10:16:08 AM       PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
10:16:09 AM      2907    162.00      0.00 17380572 756892   1.15  mysqld
10:16:09 AM     29619      2.00      0.00   23304   4616   0.01  bash
10:16:09 AM     29727    887.00      0.00    7636   1300   0.00  pidstat

10:16:08 AM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
10:16:09 AM      2907      0.00   2000.00      0.00  mysqld

Average:          PID    %usr %system  %guest    %CPU   CPU  Command
Average:           10    0.00    0.33    0.00    0.33     -  rcu_sched
Average:          462    0.00    0.33    0.00    0.33     -  kworker/31:1
Average:         2907  103.96    1.65    0.00  105.61     -  mysqld
Average:        29727    1.32    1.98    0.00    3.30     -  pidstat

Average:          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
Average:         2907    190.10      0.00 17380572 756892   1.15  mysqld
Average:        29619      9.90      0.00   23304   4616   0.01  bash
Average:        29727    883.50      0.00    7636   1289   0.00  pidstat

Average:          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
Average:         2907      0.00    660.07      0.00  mysqld

vmstat 1 2:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0    416 322964 191692 63370180    0    0     5     5    0    0  0  0 100  0
 1  0    416 322908 191692 63370188    0    0     0     0  567 2422  3  0 97  0

mpstat -P ALL 1 2:

Linux 3.8.0-29-generic (db1)    02/28/2014      _x86_64_        (32 CPU)
10:16:02 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:16:03 AM  all    3.16    0.00    0.03    0.00    0.00    0.00    0.00    0.00   96.81
10:16:03 AM    0  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
10:16:03 AM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM    9    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   10    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   11    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   13    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   15    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   16    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   17    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   18    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   19    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   20    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   21    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   22    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   23    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   24    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   25    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   26    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   27    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   28    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   29    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   30    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:03 AM   31    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

10:16:03 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:16:04 AM  all    3.22    0.00    0.06    0.00    0.00    0.00    0.00    0.00   96.72
10:16:04 AM    0   98.02    0.00    0.99    0.00    0.00    0.00    0.00    0.00    0.99
10:16:04 AM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM    4    1.98    0.00    0.99    0.00    0.00    0.00    0.00    0.00   97.03
10:16:04 AM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM    9    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   10    2.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.00
10:16:04 AM   11    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   13    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   15    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   16    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   17    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   18    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   19    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   20    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   21    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   22    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   23    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   24    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   25    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   26    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   27    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   28    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   29    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   30    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:16:04 AM   31    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

iostat -x 1 2:

Linux 3.8.0-29-generic (db1)    02/28/2014      _x86_64_        (32 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.17    0.00    0.04    0.00    0.00   99.79

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     1.29    1.41    3.11   151.57   151.91   134.28     0.08   16.85    0.54   24.28   0.31   0.14
dm-0              0.00     0.00    1.41    4.40   151.57   151.91   104.44     0.08   14.06    0.54   18.41   0.24   0.14
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     8.00     0.00    2.27    0.54    5.68   1.22   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.25    0.00    0.03    0.00    0.00   96.72

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

free -m:

             total       used       free     shared    buffers     cached
Mem:         64385      64070        315          0        187      61884
-/+ buffers/cache:       1998      62387
Swap:        65491          0      65491

SHOW GLOBAL STATUS:

Variable_name   Value
Aborted_clients 32
Aborted_connects    2088
Binlog_cache_disk_use   0
Binlog_cache_use    0
Binlog_stmt_cache_disk_use  2
Binlog_stmt_cache_use   2798217
Bytes_received  1724697475
Bytes_sent  10912829491
Com_admin_commands  103619
Com_assign_to_keycache  0
Com_alter_db    0
Com_alter_db_upgrade    0
Com_alter_event 0
Com_alter_function  0
Com_alter_procedure 0
Com_alter_server    0
Com_alter_table 113
Com_alter_tablespace    0
Com_analyze 0
Com_begin   1095437
Com_binlog  0
Com_call_procedure  0
Com_change_db   393259
Com_change_master   0
Com_check   15189
Com_checksum    0
Com_commit  1095434
Com_create_db   1
Com_create_event    0
Com_create_function 0
Com_create_index    0
Com_create_procedure    1
Com_create_server   0
Com_create_table    42
Com_create_trigger  0
Com_create_udf  0
Com_create_user 0
Com_create_view 0
Com_dealloc_sql 24
Com_delete  9989
Com_delete_multi    2
Com_do  0
Com_drop_db 1
Com_drop_event  0
Com_drop_function   0
Com_drop_index  0
Com_drop_procedure  2
Com_drop_server 0
Com_drop_table  1
Com_drop_trigger    0
Com_drop_user   0
Com_drop_view   0
Com_empty_query 0
Com_execute_sql 24
Com_flush   4
Com_grant   0
Com_ha_close    0
Com_ha_open 0
Com_ha_read 0
Com_help    0
Com_insert  711847
Com_insert_select   6
Com_install_plugin  0
Com_kill    0
Com_load    0
Com_lock_tables 0
Com_optimize    55
Com_preload_keys    0
Com_prepare_sql 24
Com_purge   0
Com_purge_before_date   0
Com_release_savepoint   0
Com_rename_table    1
Com_rename_user 0
Com_repair  0
Com_replace 491733
Com_replace_select  31
Com_reset   0
Com_resignal    0
Com_revoke  0
Com_revoke_all  0
Com_rollback    0
Com_rollback_to_savepoint   0
Com_savepoint   0
Com_select  2842695
Com_set_option  388138
Com_signal  0
Com_show_authors    0
Com_show_binlog_events  0
Com_show_binlogs    0
Com_show_charsets   0
Com_show_collations 0
Com_show_contributors   0
Com_show_create_db  0
Com_show_create_event   0
Com_show_create_func    0
Com_show_create_proc    0
Com_show_create_table   0
Com_show_create_trigger 0
Com_show_databases  3
Com_show_engine_logs    0
Com_show_engine_mutex   0
Com_show_engine_status  0
Com_show_events 0
Com_show_errors 0
Com_show_fields 231871
Com_show_function_status    0
Com_show_grants 0
Com_show_keys   0
Com_show_master_status  0
Com_show_open_tables    0
Com_show_plugins    0
Com_show_privileges 0
Com_show_procedure_status   0
Com_show_processlist    4103
Com_show_profile    0
Com_show_profiles   0
Com_show_relaylog_events    0
Com_show_slave_hosts    0
Com_show_slave_status   0
Com_show_status 4099
Com_show_storage_engines    0
Com_show_table_status   0
Com_show_tables 1516319
Com_show_triggers   0
Com_show_variables  6
Com_show_warnings   0
Com_slave_start 0
Com_slave_stop  0
Com_stmt_close  24
Com_stmt_execute    24
Com_stmt_fetch  0
Com_stmt_prepare    24
Com_stmt_reprepare  0
Com_stmt_reset  0
Com_stmt_send_long_data 0
Com_truncate    3
Com_uninstall_plugin    0
Com_unlock_tables   0
Com_update  1584610
Com_update_multi    22
Com_xa_commit   0
Com_xa_end  0
Com_xa_prepare  0
Com_xa_recover  0
Com_xa_rollback 0
Com_xa_start    0
Compression OFF
Connections 445201
Created_tmp_disk_tables 1736238
Created_tmp_files   90
Created_tmp_tables  1795650
Delayed_errors  0
Delayed_insert_threads  0
Delayed_writes  0
Flush_commands  1
Handler_commit  3893481
Handler_delete  9342
Handler_discover    0
Handler_prepare 0
Handler_read_first  329186
Handler_read_key    4104519415
Handler_read_last   12543
Handler_read_next   5423345858
Handler_read_prev   27590297
Handler_read_rnd    5211189
Handler_read_rnd_next   42554732813
Handler_rollback    0
Handler_savepoint   0
Handler_savepoint_rollback  0
Handler_update  39411585
Handler_write   274236845
Key_blocks_not_flushed  1
Key_blocks_unused   0
Key_blocks_used 319666
Key_read_requests   17637822343
Key_reads   13404466
Key_write_requests  192935745
Key_writes  1202353
Last_query_cost 0.000000
Max_used_connections    412
Not_flushed_delayed_rows    0
Open_files  9779
Open_streams    0
Open_table_definitions  4767
Open_tables 5000
Opened_files    7094126
Opened_table_definitions    49482
Opened_tables   55504
Performance_schema_cond_classes_lost    0
Performance_schema_cond_instances_lost  0
Performance_schema_file_classes_lost    0
Performance_schema_file_handles_lost    0
Performance_schema_file_instances_lost  0
Performance_schema_locker_lost  0
Performance_schema_mutex_classes_lost   0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost  0
Performance_schema_rwlock_instances_lost    0
Performance_schema_table_handles_lost   0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost  0
Performance_schema_thread_instances_lost    0
Prepared_stmt_count 0
Qcache_free_blocks  15231
Qcache_free_memory  49970792
Qcache_hits 2495975
Qcache_inserts  2762600
Qcache_lowmem_prunes    182256
Qcache_not_cached   80858
Qcache_queries_in_cache 11811
Qcache_total_blocks 39192
Queries 13324094
Questions   13324094
Rpl_status  AUTH_MASTER
Select_full_join    10253
Select_full_range_join  278
Select_range    5736
Select_range_check  18
Select_scan 1905346
Slave_heartbeat_period  0.000
Slave_open_temp_tables  0
Slave_received_heartbeats   0
Slave_retried_transactions  0
Slave_running   OFF
Slow_launch_threads 0
Slow_queries    445
Sort_merge_passes   44
Sort_range  66431
Sort_rows   5055362
Sort_scan   67580
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher
Ssl_cipher_list
Ssl_client_connects 0
Ssl_connect_renegotiates    0
Ssl_ctx_verify_depth    0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts    0
Ssl_finished_connects   0
Ssl_session_cache_hits  0
Ssl_session_cache_misses    0
Ssl_session_cache_mode  NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size  0
Ssl_session_cache_timeouts  0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries  0
Ssl_verify_depth    0
Ssl_verify_mode 0
Ssl_version
Table_locks_immediate   5984350
Table_locks_waited  21734
Tc_log_max_pages_used   0
Tc_log_page_size    0
Tc_log_page_waits   0
Threads_cached  0
Threads_connected   412
Threads_created 30180
Threads_running 38
Uptime  250436
Uptime_since_flush_status   250436

My mailman server is holding all posted messages from users even though I've confirmed that they do not have the mod bit set.

I've confirmed that the from email matches the subscribed member (the message in the moderation list says its from a member).

The server was just setup, and the members just joined. Is there maybe something like for the first 24 hours they are still moderated?

Or is there some other setting, like a list or even global server setting that is forcing all posts to get moderated?

Thanks.

We have a server with a Quad-Core AMD Opteron Processor 2378. It acts as our firewall for several servers. I've been asked to block all IPs from China.

In a separate network, we have some small VPS machines (256MB and 512MB). I've been asked to block china on those VPS's as well.

I've looked online and found lists which requires 4500 block rules. My question is will putting in all 4500 rules be a problem? I know iptables can handle far more rules than that, what I am concerned about is since these are blocks that I don't want to have access to any port, I need to put these rules before any allow. This means all legitimate traffic needs to be compared to all those rules before getting through. Will the traffic be noticeably slower after implementing this? Will those small VPS's be able to handle processing that many rules for every new packet (I'll put an established allow before the blocks)?

My question is not How many rules can iptables support?, its about the effect that these rules will have on load and speed.

Thanks.

A few days ago my linux apache server ran out of memory. The server is a xen guest. The server killed all my processes except ssh (even cron, monit, syslog).

A minute before it ran out of memory, a script of mine saved the output of various commands. Here was the output:

# free -m
             total       used       free     shared    buffers     cached
Mem:          2003       1866        137          0          3       1159
-/+ buffers/cache:        703       1300
Swap:          511          0        511

# vmstat 1 2
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  1    144 140944   4192 1186964    0    0     0     0    1    0  1  0 98  1
 3  1    144 142816   3704 1186896    0    0  4188 10132 3242 3242  0 19 59 22

# pidstat -dru 1 3
Linux 2.6.39-linode33 (web1)    09/10/11        _i686_

01:07:15          PID   %user %system    %CPU   CPU  Command
01:07:16          271    0.00   10.78   10.78     0  kswapd0
01:07:16         9871    0.00  100.00  100.00     1  flush-0:17
01:07:16        13082    0.00    1.96    1.96     0  apache2
01:07:16        15100    0.98    5.88    6.86     0  rm
01:07:16        15250    0.00    0.98    0.98     0  pidstat

01:07:15          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
01:07:16        13082      1.96      0.98   62872  29720   1.45  apache2
01:07:16        15250    224.51      0.00    1704    636   0.03  pidstat

01:07:15          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
01:07:16        13082     78.43      7.84      0.00  apache2
01:07:16        15100   1317.65      0.00      0.00  rm
01:07:16        15250      3.92      0.00      0.00  pidstat

01:07:16          PID   %user %system    %CPU   CPU  Command
01:07:17            3    0.00    0.99    0.99     0  ksoftirqd/0
01:07:17           12    0.00    0.99    0.99     2  ksoftirqd/2
01:07:17          271    0.00    1.98    1.98     2  kswapd0
01:07:17         9871    0.00  100.00  100.00     1  flush-0:17
01:07:17        15100    0.00   17.82   17.82     2  rm
01:07:17        15232   17.82    0.99   18.81     0  apache2
01:07:17        15250    0.99    1.98    2.97     0  pidstat
01:07:17        19944    0.00    0.99    0.99     0  kworker/0:2

01:07:16          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
01:07:17        15232    298.02      0.00   63096  12820   0.62  apache2
01:07:17        15250    231.68      0.00    1704    664   0.03  pidstat

01:07:16          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
01:07:17        15100   4261.39      0.00      0.00  rm
01:07:17        15232      3.96      7.92      0.00  apache2
01:07:17        15250     11.88      0.00      0.00  pidstat

01:07:17          PID   %user %system    %CPU   CPU  Command
01:07:18          271    0.00   11.11   11.11     2  kswapd0
01:07:18         9871    0.00  100.00  100.00     1  flush-0:17
01:07:18        15100    1.01   27.27   28.28     2  rm
01:07:18        15250    0.00    1.01    1.01     0  pidstat
01:07:18        19944    0.00    1.01    1.01     0  kworker/0:2

01:07:17          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
01:07:18        15250    229.29      0.00    1704    664   0.03  pidstat

01:07:17          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
01:07:18        15100   4230.30      0.00      0.00  rm
01:07:18        15250     16.16      4.04      0.00  pidstat

Average:          PID   %user %system    %CPU   CPU  Command
Average:            3    0.00    0.33    0.33     -  ksoftirqd/0
Average:           12    0.00    0.33    0.33     -  ksoftirqd/2
Average:          271    0.00    7.95    7.95     -  kswapd0
Average:         9871    0.00  100.00  100.00     -  flush-0:17
Average:        13082    0.00    0.66    0.66     -  apache2
Average:        15100    0.66   16.89   17.55     -  rm
Average:        15232    5.96    0.33    6.29     -  apache2
Average:        15250    0.33    1.32    1.66     -  pidstat
Average:        19944    0.00    0.66    0.66     -  kworker/0:2

Average:          PID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
Average:        13082      0.66      0.33   62872  29720   1.45  apache2
Average:        15232     99.67      0.00   62927  12456   0.61  apache2
Average:        15250    228.48      0.00    1704    655   0.03  pidstat

Average:          PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
Average:        13082     26.49      2.65      0.00  apache2
Average:        15100   3256.95      0.00      0.00  rm
Average:        15232      1.32      2.65      0.00  apache2
Average:        15250     10.60      1.32      0.00  pidstat

Monit reported these stats (also just one minute before it ran out of memory):

System 'local'
  status                            Resource limit matched
  monitoring status                 monitored
  load average                      [3.57] [2.63] [1.30]
  cpu                               0.5%us 18.5%sy 37.1%wa
  memory usage                      726464 kB [35.4%]
  swap usage                        148 kB [0.0%]
  data collected                    Sat Sep 10 01:06:06 2011

Here is an excerpt of what was logged in kern.log:

Sep 10 01:07:31 web1 kernel: Out of memory: Kill process 2190 (portmap) score 1 or sacrifice child
Sep 10 01:07:31 web1 kernel: Killed process 2190 (portmap) total-vm:1820kB, anon-rss:100kB, file-rss:360kB
Sep 10 01:07:31 web1 kernel: apache2 invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0, oom_score_adj=0
Sep 10 01:07:31 web1 kernel: Pid: 31143, comm: apache2 Tainted: G        W   2.6.39-linode33 #5
Sep 10 01:07:31 web1 kernel: Call Trace:
Sep 10 01:07:31 web1 kernel: [<c0185c10>] ? T.663+0x80/0xd0
Sep 10 01:07:31 web1 kernel: [<c0185cbe>] ? T.661+0x5e/0x160
Sep 10 01:07:31 web1 kernel: [<c013d8ac>] ? has_capability_noaudit+0xc/0x20
Sep 10 01:07:31 web1 kernel: [<c0185a1c>] ? oom_badness+0xdc/0x130
Sep 10 01:07:31 web1 kernel: [<c0185f76>] ? out_of_memory+0x1b6/0x2f0
Sep 10 01:07:31 web1 kernel: [<c0189715>] ? __alloc_pages_nodemask+0x6b5/0x6d0
Sep 10 01:07:31 web1 kernel: [<c0131c35>] ? dup_task_struct+0x55/0x140
Sep 10 01:07:31 web1 kernel: [<c0132728>] ? copy_process+0x88/0xb80
Sep 10 01:07:31 web1 kernel: [<c013326f>] ? do_fork+0x4f/0x290
Sep 10 01:07:31 web1 kernel: [<c0156084>] ? getnstimeofday+0x44/0x110
Sep 10 01:07:31 web1 kernel: [<c010f510>] ? sys_clone+0x30/0x40
Sep 10 01:07:31 web1 kernel: [<c06900cd>] ? ptregs_clone+0x15/0x48
Sep 10 01:07:31 web1 kernel: [<c068f7b1>] ? syscall_call+0x7/0xb
Sep 10 01:07:31 web1 kernel: [<c0680000>] ? sctp_icmp_proto_unreachable+0x20/0xc0
Sep 10 01:07:31 web1 kernel: Mem-Info:
Sep 10 01:07:31 web1 kernel: DMA per-cpu:
Sep 10 01:07:31 web1 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Sep 10 01:07:31 web1 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Sep 10 01:07:31 web1 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Sep 10 01:07:31 web1 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Sep 10 01:07:31 web1 kernel: Normal per-cpu:
Sep 10 01:07:31 web1 kernel: CPU    0: hi:  186, btch:  31 usd:  36
Sep 10 01:07:31 web1 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Sep 10 01:07:31 web1 kernel: CPU    2: hi:  186, btch:  31 usd:  18
Sep 10 01:07:31 web1 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Sep 10 01:07:31 web1 kernel: HighMem per-cpu:
Sep 10 01:07:31 web1 kernel: CPU    0: hi:  186, btch:  31 usd:  54
Sep 10 01:07:31 web1 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Sep 10 01:07:31 web1 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Sep 10 01:07:31 web1 kernel: CPU    3: hi:  186, btch:  31 usd:  22
Sep 10 01:07:31 web1 kernel: active_anon:7174 inactive_anon:3515 isolated_anon:0
Sep 10 01:07:31 web1 kernel: active_file:156159 inactive_file:133601 isolated_file:0
Sep 10 01:07:31 web1 kernel: unevictable:0 dirty:12 writeback:0 unstable:0
Sep 10 01:07:31 web1 kernel: free:39375 slab_reclaimable:166268 slab_unreclaimable:3027
Sep 10 01:07:31 web1 kernel: mapped:1913 shmem:8277 pagetables:95 bounce:0
Sep 10 01:07:31 web1 kernel: DMA free:2908kB min:88kB low:108kB high:132kB active_anon:0kB inactive_anon:0kB active_file:292kB inactive_file:296kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15808kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:988kB slab_unreclaimable:132kB kernel_stack:72kB pagetables:20kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:11075 all_unreclaimable? yes
Sep 10 01:07:31 web1 kernel: lowmem_reserve[]: 0 702 2008 2008
Sep 10 01:07:31 web1 kernel: Normal free:5376kB min:4004kB low:5004kB high:6004kB active_anon:0kB inactive_anon:0kB active_file:2316kB inactive_file:2264kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:719320kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:664084kB slab_unreclaimable:11976kB kernel_stack:512kB pagetables:360kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:84892 all_unreclaimable? yes
Sep 10 01:07:31 web1 kernel: lowmem_reserve[]: 0 0 10444 10444
Sep 10 01:07:31 web1 kernel: HighMem free:149216kB min:512kB low:2372kB high:4236kB active_anon:28696kB inactive_anon:14060kB active_file:622028kB inactive_file:531844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1336932kB mlocked:0kB dirty:48kB writeback:0kB mapped:7652kB shmem:33108kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Sep 10 01:07:31 web1 kernel: lowmem_reserve[]: 0 0 0 0
Sep 10 01:07:31 web1 kernel: DMA: 231*4kB 126*8kB 21*16kB 4*32kB 2*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2908kB
Sep 10 01:07:31 web1 kernel: Normal: 1310*4kB 18*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5384kB
Sep 10 01:07:31 web1 kernel: HighMem: 18118*4kB 7805*8kB 836*16kB 27*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 149216kB
Sep 10 01:07:31 web1 kernel: 298087 total pagecache pages
Sep 10 01:07:31 web1 kernel: 35 pages in swap cache
Sep 10 01:07:31 web1 kernel: Swap cache stats: add 2601, delete 2566, find 2926/3087
Sep 10 01:07:31 web1 kernel: Free swap  = 524144kB
Sep 10 01:07:31 web1 kernel: Total swap = 524284kB
Sep 10 01:07:31 web1 kernel: 1050608 pages RAM
Sep 10 01:07:31 web1 kernel: 865282 pages HighMem
Sep 10 01:07:31 web1 kernel: 537706 pages reserved
Sep 10 01:07:31 web1 kernel: 7195 pages shared
Sep 10 01:07:31 web1 kernel: 468537 pages non-shared
Sep 10 01:07:31 web1 kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Sep 10 01:07:31 web1 kernel: [ 1059]     0  1059      552      184   3     -17         -1000 udevd
Sep 10 01:07:31 web1 kernel: [ 2196]   104  2196      470      175   0       0             0 rpc.statd
Sep 10 01:07:31 web1 kernel: [ 2323]     0  2323     1044      575   2       0             0 syslog-ng
Sep 10 01:07:31 web1 kernel: [ 2334]     0  2334     1332      252   3     -17         -1000 sshd
Sep 10 01:07:31 web1 kernel: [ 2360]   101  2360     1047      303   3       0             0 ntpd
Sep 10 01:07:31 web1 kernel: [ 2426]     0  2426      425      125   0       0             0 getty
Sep 10 01:07:31 web1 kernel: [32223]     0 32223      523      220   0       0             0 cron
Sep 10 01:07:31 web1 kernel: [ 8378]     0  8378     3478      566   0       0             0 monit
Sep 10 01:07:31 web1 kernel: [31143]     0 31143    15446     2689   0       0             0 apache2
Sep 10 01:07:31 web1 kernel: [15030]     0 15030      606      228   2       0             0 cron
Sep 10 01:07:31 web1 kernel: [15034]     0 15034      622      248   3       0             0 sh
Sep 10 01:07:31 web1 kernel: [15100]     0 15100      429       58   2       0             0 rm
Sep 10 01:07:31 web1 kernel: Out of memory: Kill process 2196 (rpc.statd) score 1 or sacrifice child
Sep 10 01:07:31 web1 kernel: Killed process 2196 (rpc.statd) total-vm:1880kB, anon-rss:124kB, file-rss:576kB

Also, since it was 1 AM, there were almost no apache connections at the time.

As you can see, swap wasn't being used at all. There was still quite a bit of free ram (according to monit).

Any ideas how I can figure out the cause?

Also, what is the "flush-0:17" that was using so much CPU according to pidstat

This question is similar to Network port open, but no process attached?

I've tried everything from there, reviewed the logs, etc... and can't find anything.

My netstat shows a TCP listening port and a UDP port without a pid. When I search lsof for those ports nothing comes up.

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:44231           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:55234           0.0.0.0:*                           - 

The following commands display nothing:

lsof | grep 44231
lsof | greo 55234
fuser -n tcp 44231
fuser -n udp 55234

After rebooting, those "same" two connections are there except with new port numbers:

netstat -lntup
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:45082           0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:37398           0.0.0.0:*                           - 

And once again, the lsof and fuser commands show nothing.

Any ideas what they are? Should I be concerned about them?

Is it possible to allow a user to run a command using sudo, but prevent them from adding any arguments to the command?

For example, if I want a user to be able to run:

/sbin/mii-tool

but not:

/sbin/mii-tool -r

Thanks.

I have 2 web servers that share files over NFS. When we update our php code to a new version, our update script moves the existing directory and copies in the new version in its place. We make the change directly on the NFS server.

Every time we do this, web1 stops working. It gives HTTP 500 errors and the PHP log says it couldn't include fileX.php. However, when if I manually go to the directory in the command line, I can see the file and its the updated version.

In order to fix it, I need to remount the NFS directory and restart apache. Any idea why this is happening? It only happens on web1, not web2 and they both have identical configurations. And, like I said, I can see the files on web1 in the command-line, only apache doesn't see them.

Here are the mount options (/proc/mounts):

192.168.1.1:/var/exports/www/ /var/www nfs rw,sync,noatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=65535,timeo=600,retrans=2,sec=sys,local_lock=none,addr=192.168.1.1 0 0

Thanks.

When you run a full backup in backuppc and you use rsync as the transfer method, does it actually transfer the full backup source? Or does it only transfer the changes? The docs seem to imply that it would transfer the full thing and only an incremental would transfer the changes.

If this is the case, could I simply use incrementals only, and never do a full backup? The way the backups are stored (using hard links to make each incremental appear full), I would think that this would be the best method. Incrementals will only transfer the changes, yet each backup will appear full.

Thanks.

I am writing a script that will allow me to retrieve a file from all of my servers at once. I have SSH keys in place in order to log into my servers. My SSH key however requires a password.

The script I am writing will not be automated, it will only ever be run manually. So my script prompts the user for the SSH key password.

How can I send the password to the SSH key as it connects to each server. I am trying to avoid having to type my password in for each server.

I know I could use 'expect', but am hoping there is a simple way to do this. Maybe some environment variable?

Thanks.

I am looking to do some backups to S3 and want to reduce the bandwidth as much as possible. I am looking at a few options. It seems like librsync is the best solution for low bandwidth remote backups.

I've been reading on how librsync works. And it seems like the remote end needs to calculate the checksum on the blocks of the files it is comparing (as well as the local end). I would assume that S3 can't do those checksum calculations since it only serves files.

I've also read that S3 doesn't support separating files into chunks. It can only offer the whole file or nothing.

If both (or either) of these statements are true, would librsync be essentially useless? Can someone shed some light on this for me?

Thanks.

Our MySQL server has been using a lot of CPU lately (it's reached 100% several times and stays there for a while) and I noticed that it the CPU load is all on one core of one cpu. I was hoping to spread that out to all 4 on my server.

I have been tweaking the MySQL settings to use more ram and less cpu, but it still occasionally reaches very high CPU usage.

It seems like everything about the topic refers to thread_concurrency (which I've read is a solaris only setting). What can I do in Linux?

Thanks.

We have many sites in our nginx config file and we would like to use a custom error_page, but the same one for all sites.

I tried setting it in http{} to /path/to/my/404.html. I noticed that it looks inside the 'root' directory (so /path/to/my/root/path/to/my/404.html)

Is there a way that I can use one 404 file for all my virtual hosts without having to symlink a file in each directory?

I've setup my own repository which I want to use SSH as the protocol. I managed to get everything working with an SSH key using port 22.

Now, I would like to change the SSH port. I've already changed it on the SSH/repo server. Now I can't figure out how to change apt to use a custom port on the client computer.

My sources.list file line which worked over port 22:

deb ssh://[email protected]/home/user/repo lenny main contrib non-free 

I've tried:

deb ssh://[email protected]:12345/home/user/repo lenny main contrib non-free 

and it fails and actually says "failed connecting to port 22"

Can this be done? I've searched google for hours and I getting nothing but unrelated data. I've read the man pages. The man page for apt.conf specified that you can set the port this way for HTTP, but doesn't mention anything about ssh ports.

Also, a secondary question:

Can I somehow specify which SSH identity/key file to use for apt?

Thanks.

I am using qmail and libdkimtest to sign outgoing emails. The setup is almost exactly as per this tutorial (http://patchlog.com/security/qmail-and-dkim/).

Emails I send out have a signature added, and it looks fine to me however, spamassassin reports:

X-Spam-Report: 
    *  0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily
    *      valid
    *  0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid

I've added the DNS records exactly as many tutorials suggest. I've tested the DNS with http://www.sendmail.org/dkim/checker and it doesn't report any problems, and displays the public key correctly.

I would like to get more detailed errors in order to figure out why spamassassin says it isn't valid.

Thanks.

I need to configure Exim to require TLS when sending emails to a specific domain.

I know that I can use

 hosts_require_tls = host

in the smtp transport. But it sounds like that requires that I set the host of the receiving mail server. I would instead like to specify the domain.

Also, if I use this, will all other hosts/domains work without TLS? Just want to confirm before I implement.

Thanks.

Is there a way to reserve a specific amount of memory for a specific service/process?

I would like to guarantee that OpenSSH always has enough memory available to it that it can accept a new SSH connection in case the server's memory fills up.

Thanks.

We have a VPS server (it's using Virtuozzo). On a few occasions now, our VPS memory was fully used up and no new connections could be made to the server on SSH, SMTP, or POP. The only thing that works is connecting to the web service. Luckily, plesk is running on the VPS and we have been able to reboot it through plesk (as well as see that the RAM is 100% used).

I would like to find what process is causing this. I have a feeling it's MySQL, but don't really know. Is there some sort of logging I could implement that would help me find out what was the cause of this next time it happens?

Thanks.