futureal's questions

Running Apache 2.4.7, our error logs are cluttered with messages like:

 [core:error] ... AH00011: ap_cookie: client submitted cookie 'PHPSESSID' more than once: ...

We haven't been able to determine any specific failure in behavior due to this, but also have not had success duplicating it.

Is this something we could be causing with our application configuration? If so, is there a preventative measure we could be taking to eliminate it?

Or, is it not worrisome, and if so, how would we prevent these messages from cluttering the log? (I am guessing it is worrisome, but just in case...)

I have a master and hot standby setup with PostgreSQL 9.3, and I'm attempting to monitor the state of replication on the standby using the check_postgres tool and the "hot_standby_delay" action. This seems to work by calculating the difference in bytes between the xlog position on the master and the standby.

In numerous online examples I have seen warning and critical thresholds for this in the < 1MB range. The exact command we are using in Nagios is:

/usr/local/bin/check_postgres.pl --action=hot_standby_delay --host=$HOSTNOTES$,$HOSTADDRESS$ --port=5432 --dbname=monitoring --dbuser=monitoring --dbpass=monitoring --warning=1000000 --critical=5000000

Which should set a warning at roughly 1MB and an outage at roughly 5MB. However, on our servers we routinely see it spike to a high level, like this:

[1417719713] SERVICE ALERT: host;PostgreSQL: Replication Delay;CRITICAL;SOFT;1;POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB "monitoring" (host:host.example.com) 121175880
[1417719773] SERVICE ALERT: host;PostgreSQL: Replication Delay;CRITICAL;SOFT;2;POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB "monitoring" (host:host.example.com) 132780968
[1417719833] SERVICE ALERT: host;PostgreSQL: Replication Delay;CRITICAL;SOFT;3;POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB "monitoring" (host:host.example.com) 21412936

Followed up on the next Nagios check with:

[1417719893] SERVICE ALERT: host;PostgreSQL: Replication Delay;OK;SOFT;4;POSTGRES_HOT_STANDBY_DELAY OK: DB "monitoring" (host:host.example.com) 0

So in the general sense it seems that replication is working (and indeed, performing a data update on the master sees immediate results on the standby).

Unfortunately this scenario makes the monitoring useless since it is triggering a false positive many times a day. From what I've found between the documentation and other examples of using this, this result is not typical, and most people are able to set a threshold of 1MB or less and only see errors when there are in fact errors.

Does anybody have any idea of what I could try with the configuration to remedy this? On this particular install we have changed only a few parameters, and of those, only wal_keep_segments seems even remotely related (and we have that set to 128).

Both master and standby are hosted in EC2 in the same availability zone and there don't seem to be any communication delays between them. This is also a very low-traffic database so I am uncertain as to how the xlog delta could be that far off to begin with, unless I am missing some very critical fact.

I am attempting to configure exim4 under Debian 6 or 7 as follows:

• For mails being forwarded for a "virtual" domain, send directly via SMTP
• For all other mails, send via Amazon SES

The first bullet seems necessary because SES will not allow the sending of a message whose "From" address does not match a valid configured domain for that AWS account (if anybody does know a way to do this with SES, that would be a preferred solution, although I am pretty sure it is not possible).

So, right now this system works fine with SES; everything sends correctly, exim is configured correctly, etc. From that point, I defined a some new virtual domain I'd like to forward for, for example in /etc/exim4/virtual/example.com:

user : [email protected]

I then created a new router, for example /etc/exim4/conf.d/router/090_exim4-config_virtual_redirect containing:

sender_redirect:
  driver = redirect
  domains = dsearch;/etc/exim4/virtual
  data = ${lookup{$local_part}lsearch{/etc/exim4/virtual/$domain}}

So far, this works fine -- exim attempts to relay messages addressed to [email protected] to [email protected] instead, but it fails because of the SES restriction mentioned above.

At this point, I want this router to send messages directly server-to-server, without affecting the sending of any other mails on the server.

Is this possible? I have a feeling there is one small addition I am missing here, but I haven't been able to find it.

I am attempting to do the following in a bash script and am wondering if it is possible:

• Time the output of a command (in this case, mysql) using time
• Suppress the stdout output of the command
• Capture the stderr output of the command in case of an error
• Check the exit status code of the command after running it

This is a complex chain and I haven't had much luck getting it all to work together. I can get the status code to work if I cease using time.

This is what I had so far:

# the setup
TIMEFORMAT=%R
MYSQL=mysql <options>

# execute and time a query (does more than is listed here)
function executeQuery {

  TIME=$( time ( $MYSQL "$1" 2>&1 | tee /tmp/somefile.txt > /dev/null ) 2>&1 )

  # do something with $?

}

I am redirecting any error response from the command to a file using tee, and then sending the resulting stdout to /dev/null. I am then redirecting the time command's stderr to stdout, which should end up in $TIME.

Now if I change that line to something like:

  TIME=$( time ( $MYSQL "$1" 2>&1 | tee /tmp/somefile.txt > /dev/null; exit ${PIPESTATUS[0]} ) 2>&1 )

It correctly sends back the exit code from mysql, but breaks the time command.

Is this even possible? Have I missed something? Hopefully the goal is clear.

Thanks!

Suppose that I have some instance running in Amazon EC2:

 somehost.example.com

This instance hosts some number of applications or websites for other domains:

 www.example.com
 web.example.org
 etc.

This host is configured to use exim4 to interface with Amazon SES for sending mail. In Amazon SES, each domain is separately allowed as a valid sender:

www.example.com
web.example.org
etc.

However, as one would expect, the FQDN of the instance and the executing user are being used as the sender of each email, so all emails appear to originate from:

[email protected]

I can certainly configure Amazon SES to allow mails from somehost.example.com, and I can further add a 'From: "Whoever" <[email protected]>' envelope to the emails at the application level.

What I am wondering is, (a) is this an acceptable solution, and if not, (b) what configuration might I explore to show that mail for each domain (e.g. example.com) appears to originate directly from a "something.example.com" server rather than the actual host. I assume this involves creating a hostname.example.com DNS entry and pointing it to the server, but beyond that I am lost in how to set up exim or the application to allow this.

I have configured nagiosgrapher correctly, as far as I can tell, but the logs continue to report a lack of data. For example, for load, in /var/log/nagiosgrapher/ngraph.log:

 2012-08-02 22:30:16 PIPE: myhost    CPU Load        OK - load average: 0.00, 0.00, 0.00     load1=0.000;15.000;30.000;0; load5=0.000;10.000;25.000;0; load15=0.000;5.000;20.000;0;
 2012-08-02 22:30:16 VALUES: [myhost][CPU Load]:No matching output values found...

I have configured Nagios and Nagiosgrapher on Debian systems before and have never run into an issue like this, and thus far have been unable to solve it.

The systems are using only default plugins. The relevant /etc/nagios3/nagios.cfg values are:

 process_performance_data=1

and

 service_perfdata_command=ngraph-process-service-perfdata-pipe

I have tried restarting both Nagios and Nagiosgrapher in varying order, which was something I found on Google searches, but that didn't seem to have an effect.

Is there any idea of what I might be doing wrong or where to look? All other Nagios functionality seems to be behaving normally on both local and remote hosts.

Suppose that in a simplified scenario, I have a server hosting a single virtual host under Apache 2, and I would like any other attempt to access the server via IP or any other host to be denied. Further, I'd like an attempt to access the virtual host by HTTP to redirect to HTTPS.

Suppose the host is myhost.company.com, so the scenario is:

• https://myhost.company.com should work
• http://myhost.company.com should redirect to https://myhost.company.com
• any other web access by host, IP, etc. to the box should be denied

This seems like it shouldn't be too difficult to do (should it?) but thus far the only way I've been able to make it work is by defining the host and the redirect, and then sending all other traffic to some dummy directory on the server that has been manually forbidden.

I'd like to be able to do it in the Apache configuration though, if possible. Is it? In all of my other attempts, with "Deny" etc. it still allows other IP/host accesses through.

Thanks!

I am configuring a few domain names in Route53 and am looking for some advice on best practices for setting things up. Here is the scenario:

Let's say domain1.com is my master corporate domain, and I also have domain2.com and domain3.com.

Suppose I have two Amazon EC2 instances defined as a web and database server, with elastic IPs of 11.11.11.11 and 22.22.22.22 respectively, and I want to address them externally as:

 prod-web01.domain1.com
 prod-db01.domain1.com

Let's further suppose that the website for domain1.com is hosted on one of the above instances, which means that I must set an A record for domain1.com (right?) to the elastic IP of the web instance. So I create record sets in Route53 like:

 domain1.com.                A         11.11.11.11
 prod-web01.domain1.com.     A         11.11.11.11
 prod-db02.domain1.com.      A         22.22.22.22
 www.domain1.com.            CNAME     prod-web01.domain1.com

Now, both domain2.com and domain3.com are also hosted on that web01 instance above, so I create record sets for them like:

 domain2.com.                A         11.11.11.11
 www.domain2.com.            CNAME     prod-web01.domain1.com.

and

 domain3.com.                A         11.11.11.11
 www.domain3.com.            CNAME     prod-web01.domain1.com.

This all works just fine, but it is not as elegant as I was hoping for. I am wondering if there is some way to set up this type of configuration in which there are fewer records pointing to direct IP addresses.

My understanding is that I can't use a CNAME record for the root of a domain (right?), so is there some other way to set things so that I only have one "hard" reference to each IP in my DNS infrastructure?

If not, does this seem like a smart setup? Or a dumb one? Feel free to tell me I'm dumb. :)

Looking for a "best practice" answer to managing externally-addressable hosts using the combination of Amazon EC2 and Amazon Route 53, without using Elastic IPs for each host. In my scenario I will have 30+ hosts that need to be accessible from outside EC2, so directly using internal DNS will not work.

In the past, I have addressed hosts by assigning an elastic IP to that host (let's say, 55.55.55.55) and then creating an associated A record. For example, let's say I want to create "ec2-corp01.mydomain.com" I might do:

ec2-corp01.mydomain.com.     A     55.55.55.55     300

Then on that EC2 instance, I would assign the Elastic IP of 55.55.55.55, and everything works fine. Of course, to make this work, I need to have one Elastic IP per instance, which is something I'd like to avoid if possible; I'd like the infrastructure to be more dynamic.

So my thought is to try something like:

• Create a script that queries the internal EC2 tools to determine an instance's private hostname

• On instance boot, call that script to determine its hostname, and then using the command-line Route 53 interface to find and update that hostname to its current internal hostname

• Since the host will have a relatively low TTL (let's say 300 as above, or 5 minutes) it should take effect pretty quickly

Is this a good idea? Is there a better or more widely accepted way to handle it?

If it IS a good idea, what type of record should I be creating? A CNAME that points to the internal host, like ec2-55-55-55-55.compute-1.amazonaws.com? Is an A record better or worse?

Thanks!

I am attempting to reconfigure the "View Issue" screen on a basically default JIRA 4.4 installation. So far, I am unable to actually see this new screen I have created in action on any issues, and I am unsure what I am doing wrong -- though it is probably something simple.

The steps I took were:

• Created a new screen
• Created a new screen scheme
• Attached my new screen to the "View Issue" action of the new scheme
• Attached my screen scheme to a new issue type scheme
• Attached my issue type scheme to my project
• Performed a re-index (not sure this is necessary)

However, I still do not see my new view screen in place of the old one; all of the normal defaults are still there. I also tried restarting JIRA to see if that changed anything, but it did not.

Is there anything I am missing? I assume it is just something minor, but maybe JIRA can't do what I am trying to do? Everything I read seems to indicate you can fully configure the issue view via the administrative tool rather than hacking around in the code, but maybe I am just misinterpreting it.

I am looking to understand some server performance problems I am seeing with a (for us) heavily loaded web server. The environment is as follows:

• Debian Lenny (all stable packages + patched to security updates)
• Apache 2.2.9
• PHP 5.2.6
• Amazon EC2 large instance

The behavior we're seeing is that the web typically feels responsive, but with a slight delay to begin handling a request -- sometimes a fraction of a second, sometimes 2-3 seconds in our peak usage times. The actual load on the server is being reported as very high -- often 10.xx or 20.xx as reported by top. Further, running other things on the server during these times (even vi) is very slow, so the load is definitely up there. Oddly enough Apache remains very responsive, other than that initial delay.

We have Apache configured as follows, using prefork:

StartServers          5
MinSpareServers       5
MaxSpareServers      10
MaxClients          150
MaxRequestsPerChild   0

And KeepAlive as:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5

Looking at the server-status page, even at these times of heavy load we are rarely hitting the client cap, usually serving between 80-100 requests and many of those in the keepalive state. That tells me to rule out the initial request slowness as "waiting for a handler" but I may be wrong.

Amazon's CloudWatch monitoring tells me that even when our OS is reporting a load of > 15, our instance CPU utilization is between 75-80%.

Example output from top:

top - 15:47:06 up 31 days,  1:38,  8 users,  load average: 11.46, 7.10, 6.56
Tasks: 221 total,  28 running, 193 sleeping,   0 stopped,   0 zombie
Cpu(s): 66.9%us, 22.1%sy,  0.0%ni,  2.6%id,  3.1%wa,  0.0%hi,  0.7%si,  4.5%st
Mem:   7871900k total,  7850624k used,    21276k free,    68728k buffers
Swap:        0k total,        0k used,        0k free,  3750664k cached

The majority of the processes look like:

24720 www-data  15   0  202m  26m 4412 S    9  0.3   0:02.97 apache2                                                                       
24530 www-data  15   0  212m  35m 4544 S    7  0.5   0:03.05 apache2                                                                       
24846 www-data  15   0  209m  33m 4420 S    7  0.4   0:01.03 apache2                                                                       
24083 www-data  15   0  211m  35m 4484 S    7  0.5   0:07.14 apache2                                                                       
24615 www-data  15   0  212m  35m 4404 S    7  0.5   0:02.89 apache2            

Example output from vmstat at the same time as the above:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 8  0      0 215084  68908 3774864    0    0   154   228    5    7 32 12 42  9
 6 21      0 198948  68936 3775740    0    0   676  2363 4022 1047 56 16  9 15
23  0      0 169460  68936 3776356    0    0   432  1372 3762  835 76 21  0  0
23  1      0 140412  68936 3776648    0    0   280     0 3157  827 70 25  0  0
20  1      0 115892  68936 3776792    0    0   188     8 2802  532 68 24  0  0
 6  1      0 133368  68936 3777780    0    0   752    71 3501  878 67 29  0  1
 0  1      0 146656  68944 3778064    0    0   308  2052 3312  850 38 17 19 24
 2  0      0 202104  68952 3778140    0    0    28    90 2617  700 44 13 33  5
 9  0      0 188960  68956 3778200    0    0     8     0 2226  475 59 17  6  2
 3  0      0 166364  68956 3778252    0    0     0    21 2288  386 65 19  1  0

And finally, output from Apache's server-status:

Server uptime: 31 days 2 hours 18 minutes 31 seconds
Total accesses: 60102946 - Total Traffic: 974.5 GB
CPU Usage: u209.62 s75.19 cu0 cs0 - .0106% CPU load
22.4 requests/sec - 380.3 kB/second - 17.0 kB/request
107 requests currently being processed, 6 idle workers

C.KKKW..KWWKKWKW.KKKCKK..KKK.KKKK.KK._WK.K.K.KKKKK.K.R.KK..C.C.K
K.C.K..WK_K..KKW_CK.WK..W.KKKWKCKCKW.W_KKKKK.KKWKKKW._KKK.CKK...
KK_KWKKKWKCKCWKK.KKKCK..........................................
................................................................

From my limited experience I draw the following conclusions/questions:

• We may be allowing far too many KeepAlive requests

• I do see some time spent waiting for IO in the vmstat although not consistently and not a lot (I think?) so I am not sure this is a big concern or not, I am less experienced with vmstat

• Also in vmstat, I see in some iterations a number of processes waiting to be served, which is what I am attributing the initial page load delay on our web server to, possibly erroneously

• We serve a mixture of static content (75% or higher) and script content, and the script content is often fairly processor intensive, so finding the right balance between the two is important; long term we want to move statics elsewhere to optimize both servers but our software is not ready for that today

I am happy to provide additional information if anybody has any ideas, the other note is that this is a high-availability production installation so I am wary of making tweak after tweak, and is why I haven't played with things like the KeepAlive value myself yet.

I am looking to see if anything exists that would allow us to capture all outgoing email on a machine -- for example, in a staging environment -- and drop it in a single place, which ideally would be something we could check with a mail client.

Currently we're doing this on the software level (if environment is staging, rewrite address) which is a bit ugly and leads to errors.

The servers are currently on Debian Linux, using exim as the mail transport.

I have a domain that sends two different types of emails: ones that are created via automated processes like notifications that go out to users, and ones that are generated from Request Tracker (the ticket system).

We set up DKIM awhile back and I made the mistake of never verifying that both types of emails were working, so I never checked the RT emails. More recently we realized that occasionally RT-generated messages are hitting spam filters, and it feels like failed DKIM checks may have a hand in it.

RT adds some headers to each message, for example:

X-RT-Loop-Prevention: Support
RT-Ticket: Support #3165
Managed-by: RT 3.8.2 (http://www.bestpractical.com/rt/)
RT-Originator: [email protected]
X-RT-Original-Encoding: utf-8 

However, those headers do not appear in the DKIM signature. For example, here is a hardfail string from Gmail this morning:

Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of [email protected] designates xxx.xxx.xxx.xxx as permitted sender) [email protected]; dkim=hardfail [email protected]
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xxx.com; s=mta;
    h=Date:Content-Type:Content-Transfer-Encoding:MIME-Version:To:Message-ID:References:In-Reply-To:Reply-To:From:Subject; bh=GDIpYEyFTXB3RPUtFDKxW+iBpkOYngdUELnMw316Ohk=;
    b=A6iZYrFUZ68gszu/KeTyMoUUE0jbGlZ+yxcz72gq7Bdxe+jAkcgFoExN+duxLPIZqJm87Gz+XCB9IwnQbKC5lsVKK8cwUzQTHZx6E8ZPyynkv0NvC8MStDgOswFnjdcy;

As you can see, the RT headers are not included in the DKIM signature. When a mail is sent from the site outside of RT, Gmail validates the signature properly.

My understanding was that "custom" headers (X-xxxxxxxx) are ignored by DKIM but others are not, for example the RT-Ticket, Managed-by, and RT-Originator above.

Does anybody have any detailed experience with this or know where I might look to get DKIM to include those headers in the signature? I have searched through the RT support a bit and cannot find much. I am using exim as the MTA, and the system is Debian Lenny.

edit: I may be barking up the wrong tree here with the header fields, I am not sure. I modified the exim configuration to explicitly tell it which headers to look at, by taking the list of headers from RFC4871 (https://www.rfc-editor.org/rfc/rfc4871) and adding the extra headers that RT is adding, like so:

  dkim_sign_headers = From:Sender:Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type:Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:Precedence:X-RT-Loop-Prevention:RT-Ticket:Managed-by:RT-Originator:X-RT-Original-Encoding

That resulted in all of the RT headers being properly added to the signature, however, Gmail still reported a hardfail on the signature when it came via RT. As far as I can tell RT is using exim in the same way as any other external program, so I am at a loss to explain why those messages are failing.

We are having Apache processes get hung up after seeing a POST request, and wondering what the best path might be to troubleshooting this. When I look at the extended server status for the requests that hang, I see entries like:

53-31 28616 9/13/74232 W 0.71 7174 0 7.6 0.01 961.47 xxx.xxx.xxx.xxx www.mysite.com POST xxxxx HTTP/1.1

27-31 7629 1/34/107074 W 0.96 4480 0 0.0 0.39 1394.10 xxx.xxx.xxx.xxx www.mysite.com POST xxxxx HTTP/1.1

The process itself is always doing something, usually consuming significant CPU resources until we kill it.

Originally I assumed it was some bad code -- everything is running PHP scripts -- but after awhile we saw hung processes resulting from a variety of different requests, many of them simple, but always showing POST as the last type of request handled.

I don't expect that the community would instantly be able to solve this without seeing the actual code running; what I am really looking for is the best path to debug this as I am out of ideas. Some things that do come to mind:

• Is there any kind of a tool similar to jstack that will allow us to dump and inspect a stack trace of a hung process, that would give useful information?

• Is there a way to track useful input that might help troubleshoot this? I have tried setting the Apache logs to debug and having it output POST query parameters but the amount of data logged is overwhelming and I have a hard time knowing what to look for. Any best practice here?

• The server is using prefork with MaxClients set to 150, and a KeepAlive of 5. Under normal load scenarios there are usually 30-50 processes up and running; at high load closer to 100. We have seen the hung process behavior occur under any load scenario. Is there any piece of the configuration that I might specifically look at?

The server is Apache 2 running under Debian Lenny on EC2.

Many thanks!