On a machine (AWS m5.large) that only runs nice'd background processing jobs (i.e. no web/DB/etc servers present), are there disadvantages to consistently running the CPU at 100%?

I understand that running the system such that it consumes 100% of the available memory is not a good idea. Without swap, the system will simply kill processes when it runs out of memory. Even with swap, the system will start swapping out pages which slows the entire system down dramatically.

However, my understanding is that a system with nice'd processes running at 100% CPU usage will function without dramatic slowdowns. Is this correct?

Or, would it be better to try to configure the background processes such that the system stays within the range of 60% - 90% CPU usage?

Facebook alerted me that some SSL certificates have been issued that potentially phish one of our domains:

The detailed view for one of these entries is as follows:

When I access one of these domains, it forwards to our actual domain.

There are plenty of resources online about the risks that are involved if this is, indeed, the beginnings of a phishing attack. However, we're a low-reward target, so it's unlikely that someone is attempting to perform a phishing attack.

We don't use Microsoft's cloud, and I'm not very familiar with it. Is there a simple, innocuous explanation for the possible purpose of these domains and associated certificates? We provide a SaaS product and do work with partners to fulfill certain parts of our system. Perhaps one of our partners uses Microsoft cloud and their system caused these domains and certificates to be generated?

I'd ask Microsoft, but since we're not a customer of theirs, they won't help us... ?‍♂️

I have a long-running CLI cron job written in PHP. After an inconsistent period of time, the script stops writing output to its log file, indicating that it has stopped executing.

However, the script is still running as confirmed with ps aux.

The script isn't throwing any exceptions or outputting any errors. When it stalls, it doesn't seem to consume any more CPU.

Memory isn't an issue - the script uses ini_set("memory_limit", "1G"); and seems to freeze when the script is only using about 73 MB. The script is running on a capable dedicated machine (8 GB RAM, 8 CPUs, PHP 5.6.25) with plenty of free memory, disk space, inodes, /tmp space, available CPU, etc.

Using xdebug's function trace logging, I can see that the script simply stops at an unremarkable function call. A "normal" execution of xdebug writes something like TRACE END [2019-03-26 13:02:55] to the end of the logfile and then exits. However, when the script freezes, this line does not appear and the script does not exit. For example:

"Stalled" xdebug function log, Example A:

 1481.7659   73187512               -> in_array() /home/xyz/inc/util/email.inc:156
 1481.7659   73187056               -> strpos() /home/xyz/inc/util/email.inc:170
 1481.7659   73187208               -> mail() /home/xyz/inc/util/email.inc:181

The "stall" does not always happen at the same function.

"Stalled" xdebug function log, Example B:

 1422.5710   73306784                 -> strrpos() /home/xyz/inc/util/external-common.inc:106
 1422.5710   73306840                 -> substr() /home/xyz/inc/util/external-common.inc:108
 1422.5710   73307008                 -> shell_exec() /home/xyz/inc/util/external-common.inc:112

"Normal" xdebug function log:

55190.4859   35183552   -> WP_Object_Cache->__destruct() /home/xyz/public_html/wp/wp-includes/cache.php:0
55190.8278   34830224   -> wpdb->__destruct() /home/xyz/public_html/wp/wp-includes/wp-db.php:0
55193.1823       8672
TRACE END   [2019-03-26 13:02:55]

This might be mere coincidence, but the only common thread that I can see is that it might have something to do with C's popen() - it (or a relative) seems to be used by both PHP's mail() and shell_exec(). I don't think I have too many open processes (verified recently; don't have the command I used on-hand). The script only executes at most 5 shell_exec() calls per second and each call takes less than a second to execute. (Plus, if this were a problem, wouldn't I receive some sort of error instead of this "stalling" behavior?)

What could be going on?

What should my next steps be?

The problem

In short, using the following line in my Exim transport does not result in the string expansion (insertion) of the TESTVAR environment variable:

headers_add = "X-TESTVAR: ${env{TESTVAR}{$value}{The TESTVAR environment variable was not found}}"

I never get $value; I always get "The TESTVAR environment variable was not found".

What am I doing wrong?

Environment:

[email protected] [~]# cat /etc/redhat-release
CentOS release 6.10 (Final)

Exim version:

[email protected] [~]# exim --version
Exim version 4.91 #1 built 11-May-2018 09:49:25
Copyright (c) University of Cambridge, 1995 - 2018
(c) The Exim Maintainers and contributors in ACKNOWLEDGMENTS file, 2007 - 2018
Berkeley DB: Berkeley DB 4.7.25: (September 12, 2013)
Support for: crypteq iconv() IPv6 PAM Perl OpenSSL Content_Scanning DKIM DNSSEC Event I18N OCSP PRDR SPF Experimental_SRS
Lookups (built-in): lsearch wildlsearch nwildlsearch iplsearch dbm dbmjz dbmnz dnsdb passwd sqlite
Authenticators: cram_md5 dovecot plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir autoreply lmtp pipe smtp
Malware: f-protd f-prot6d drweb fsecure sophie clamd avast sock cmdline
Configure owner: 0:0
Size of off_t: 8
2019-03-08 19:04:22 cwd=/home/account 2 args: exim --version
Configuration file is /etc/exim.conf

The relevant transport from my exim.conf:

dovecot_virtual_delivery:
  driver = lmtp
  socket = /var/run/dovecot/lmtp
  batch_max = 200
  rcpt_include_affixes
  delivery_date_add
  envelope_to_add
  return_path_add
  headers_add = "X-TESTVAR: ${env{TESTVAR}{$value}{The TESTVAR environment variable was not found}}"

The commands I executed to send the email:

[email protected] [~]# export TESTVAR=hello
[email protected] [~]# echo $TESTVAR
hello
[email protected] [~]# exim -t
to:[email protected]
Hi there, this email should have a header named "X-TESTVAR" and it should have a value of "hello".
[email protected] [~]# 

The resulting email:

Return-Path: <[email protected]>
Delivered-To: [email protected]
Received: from host.example.com
    by host.example.com with LMTP id IH+3MCUQg1waLwAAx771+g
    for <[email protected]>; Fri, 08 Mar 2019 19:00:21 -0600
Return-path: <[email protected]>
Envelope-to: [email protected]
Delivery-date: Fri, 08 Mar 2019 19:00:21 -0600
Received: from account by host.example.com with local (Exim 4.91)
    (envelope-from <[email protected]>)
    id 1h2QLV-0004Mo-Ee
    for [email protected]; Fri, 08 Mar 2019 19:00:21 -0600
to:[email protected]
Message-Id: <[email protected]>
From: [email protected]
Date: Fri, 08 Mar 2019 19:00:20 -0600
X-TESTVAR: The TESTVAR environment variable was not found

Hi there, this email should have a header named "X-TESTVAR" and it should have a value of "hello".

Environment: cPanel/WHM using Exim 4.91.

I noticed that our "Global Email Filters" (cPanel parlance) weren't working correctly on emails sent by certain ISPs.

To make a long story short, I discovered that email headers that incorrectly use newlines are causing the filtering to not work correctly. Some examples are below.

Filter works

X-Custom-Header: allOneLine
To: "[email protected]" <[email protected]>

This message IS correctly filtered by Exim.

Filter does not work

X-Custom-Header: line1
line2
To: "[email protected]" <[email protected]>

This message is NOT correctly filtered by Exim due to the newline in X-Custom-Header that is not followed immediately by whitespace.

Filter works

To: "[email protected]" <[email protected]>
X-Custom-Header: line1
line2

This message IS correctly filtered by Exim. There is an invalid newline in the X-Custom-Header, but the To filter precedes the newline, so filtering succeeds.

Filter works

X-Custom-Header: line1
 line2
To: "[email protected]" <[email protected]>

This message IS correctly filtered by Exim. Although there is a newline in X-Custom-Header, the subsequent line begins with whitespace.

I've reviewed the actual Exim filters that cPanel creates in /etc/vfilters/example.com, and I don't see any obvious problems. So, it seems that this may be a bug in Exim. If a header contains a newline that is not followed immediately by whitespace, filtering will not be performed on subsequent headers.

I do understand that using a newline that is not followed by whitespace is syntactically invalid (RFC 2822; search for "folding"). The problem is that we obviously don't have control over other senders' headers. It seems that legitimate senders are guilty of committing this sin. Yahoo, for example, includes an "Outgoing Spam Guard" header (X-YMail-OSG) that contains newlines not followed by whitespace.

Does this indeed appear to be a bug in Exim?

What are the potential workarounds?

An SPF record is a DNS record that essentially states which IPs/domains are allowed to send email on behalf of the domain.

When using a third-party email service, the service often recommends using a "catch-all" SPF record for all of their servers. Here's what Sendgrid recommends:

v=spf1 include:sendgrid.net ~all

The domain that this record is associated with (e.g. example.com) will "allow" all email from all of sendgrid.net's domains.

It seems to me that an attacker could simply register for a Sendgrid account and send emails on behalf of example.com that would "pass" the SPF check.

Is this true?

I know that SPF isn't designed to verify message authenticity. It just seems a bit odd that in the right situation, such a widely-lauded technology has such an easily-exploited weakness. And yes, I do realize that requesting a private IP from the third-party email service would alleviate this concern.

As you can see in the excerpted log file below (from /var/log/lfd.log), lfd is prematurely removing temporary blocks that it imposes on IPs:

Apr  7 13:07:59 host lfd[32117]: (wordpressxmlrpc) Request of xmlrpc.php.  None of our users legitimately use this file. 92.255.223.83 (RU/Russian Federation/92x255x223x83.dynamic.kirov.ertelecom.ru): 1 in the last 300 secs - *Blocked in csf* for 86400 secs [LF_CUSTOMTRIGGER]
[...]
Apr  7 13:19:35 host lfd[7062]: Incoming IP 92.255.223.83:80 temporary block removed
Apr  7 13:19:35 host lfd[7062]: Incoming IP 92.255.223.83:443 temporary block removed

The first line shows that the IP will be blocked for 86400 seconds (one day). However, about 11 minutes later, lfd removes the temporary block. What's going on?

If it helps, the relevant portion of /etc/csf/regex.custom.pm is:

if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /(\S+).*] "POST \/xmlrpc\.php.*" 200/)) {
    return ("Request of xmlrpc.php.  None of our users legitimately use this file.",$1,"wordpressxmlrpc","1","80,443","86400");
}

On our web server, we perform an incremental backup of our files every 4 hours using rsnapshot. To perform the actual copying of files, I've configured it to use a custom script, cp-nice that has the following contents:

nice -n19 ionice -c3 cp "$@"

As you can see, this calls cp via nice and ionice to keep it from overloading the system. As far as I can tell, this has the desired effect (backups are made without bringing the system to a halt).

However, our NewRelic monitoring system freaks out every time a backup is made, since it detects that system IO is at 100%. Is there a way to perform these backups without causing NewRelic to think there's an issue on the server? Perhaps there's some way to "whitelist" a process in NewRelic so that it doesn't count against IO?

This seems like a pretty common use case - certainly, someone must have encountered and solved a situation like this before!

We have a somewhat complex LAMP stack running a variety of websites using Litespeed instead of Apache on Centos 5.

Once everything is up and running, the memory usage of the system is fairly consistent - most of the memory is taken up by MySQL's caches, with the remainder of it for Litespeed (webserver and lsphp5) and OS-related operations.

Every once in a while, we experience a huge reduction in memory consumed by processes (around 3:55 am and 9:25 am in the screenshot below).

Everything continues working just fine - in fact, performance increases. I've checked the OOM killer and can't see any processes being killed, and MySQL's caches appear to remain intact (e.g. no deviation in MyISAM or InnoDB hit rate).

So, my question is, "How can we determine what memory is being freed (e.g. which processes it belonged to)?"

Note that it's ideal if we can do this analysis retroactively (e.g. on historical data). However, I'd also be happy to find a solution we can put in place to track future memory usage.

I've noticed that our app does not respond to requests when a demanding request is in execution. Running top seems to identify the source of the problem:

top - 13:54:25 up 1 day, 13:43,  2 users,  load average: 1.02, 0.98, 0.83
Tasks: 110 total,   1 running, 109 sleeping,   0 stopped,   0 zombie
Cpu(s): 11.9%us,  1.1%sy,  0.0%ni, 86.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:   3145728k total,  2329220k used,   816508k free,        0k buffers
Swap:   131072k total,   128164k used,     2908k free,  1585060k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
26073 mysql     20   0  397m 209m 3452 S 99.1  6.8   3:02.49 mysqld             
16419 mailnull  20   0  9848 3288 2664 S  2.3  0.1   1:17.63 exim               
 2085 nobody    20   0 44312  10m 3436 S  1.3  0.3   4:50.98 litespeed          
24727 nobody    20   0  320m  50m  41m S  0.3  1.7   0:06.86 lsphp5             
26314 root      20   0  2428 1104  832 S  0.3  0.0   0:00.36 top                

It appears to me that mysql is hogging the entire CPU that it is running on (99.1%). This means that one of our cores is pegged at 100% while the other 7 (seven!) sit idle at 0%.

I understand that if our tables were InnoDB, the load would be distributed amongst the cores. Is this correct?

Is there any way to distribute the workload amongst the cores our tables are using MyISAM?

Am I looking in the wrong place entirely? While the resource-heavy query is hogging one CPU, shouldn't MySQL be able to utilize the other CPU's for separate queries? Or, is this limited by our usage of MyISAM?

I've got a fun challenge for you all. A company I'm working with is setting up its development/release workflow and environments. The eventual production environment is a VPS hosted with a popular host running CentOS within a Virtuozzo virtualization environment.

We'd like to install a virtualized instance of the production environment (or something close to it) to each developer's local system. This will allow each developer to test changes locally on his/her own system without tainting anyone else's system when things go wrong.

Clearly, there are many ways to do this, but I imagine there's a specific approach that is superior. This is where I need your help.

So far, the best options seem to be as follows.

Export/Back up the Virtuozzo VM. Download it to a local system and install it into a Virtuozzo Container.

This is the cleanest/purest solution, but it may not work. The cloned VM would still have all of the network configuration from the VPS' network, for example. Would it be hard to write a script to update these (and any other old configuration values) so they are more appropriate for the local VM?

Deploy the codebase to a similarly configured VM created in an arbitrary virtualization product.

This approach isn't as clean, but I know it can work (I've done it before). For example, we could install the same version of CentOS, Apache, MySQL, etc. to a local VMWare VM. The environment wouldn't be identical to the production environment, but it might be close enough to make this workflow feasible.

Another option?

What other options do you think there may be? Or, is one of these options the best? I'd love to hear your thoughts! :)

Edit:

Multiple cPanel/user accounts on the production server

I contacted our VPS' customer support to see if I could obtain access to the Virtuozzo backups; he denied our request but suggested that we might try creating a separate cPanel/user account for each developer. This would allow the environment to be identical to that of the server, but would be a little more inconvenient since the environment would still be running remotely instead of locally. Still, a decent option.