Bobby Jack's questions

I have a cron job that runs a shell script that executes a php script. It runs every day, sends some email, and writes to stdout (which is redirected to a log file in the shell script).

It looks as though this process has finished, today, but not exited (or whatever the correct terminology is!).

$ ps -ejH
...
10756 10756 10756 ?        00:00:00   sh
10760 10756 10756 ?        00:00:00     automail.sh
10766 10756 10756 ?        00:03:57       php
...

The process with id 10766 has consumed just under 4 minutes of CPU time. From an interactive top session, I get the following:

10766 root      20   0 40640 6024    4 S  0.0  0.2   3:57.48 php

Those hundredths haven't changed since I've looked at it. So my conclusion is that it's doing ... almost nothing, at the very most.

$ ls -ld /proc/10766/
dr-xr-xr-x 7 root root 0 2016-03-09 08:55 /proc/10766/

tells me it's been going for some time; server time is now:

$ date
Wed Mar  9 11:08:29 GMT 2016

The last line of the php script writes to a log file, and that line is present in the log file. Execution of the php script is the final thing in the shell script.

How do I go about diagnosing why this process hasn't quit?

UPDATE

Here's an edited version of the shell script I'm running:

#!/bin/sh

DATE=$(date +%Y-%m-%d)
PHP=/usr/bin/php
SCRIPT=/path/to/script.php
LOG=/path/to/log.file.$DATE.log

$PHP $SCRIPT >> $LOG

I'm having issues setting up HTTPS on AWS via a load balancer. I've managed to get as far as proxying my standard :80 server, via HTTPS on the load balancer, so requests to https://www.example.com are working. So if AWS can handle HTTPS traffic at the ELB level, do I actually need to start a separate web server to handle the HTTPS traffic, or can I just leave it at that?

What are the disadvantages of proxying HTTPS -> HTTP using an AWS load balancer?

I'm building a site that needs to include a 'check' procedure, to do several initiation tasks for a user's session. Examples include checking whether they're accepting cookies, determining if their IP address grants them specific privileges, etc.

Once the check is complete, I need to redirect the user back to the page they originally requested.

The plan is to use RewriteCond and map all URLs to an 'initiator' if the user doesn't have a specific cookie set.

Let's say I want to rewrite all URLs (ultimately, with some conditions, of course) to:

/foo?original_url=...

Where the ... is the original URL requested, URL-encoded.

The closest I've got is this:

RewriteRule ^(.*)$ http://localhost/php/cookie.php$1 [R=301]

I can then inspect the original URL, captured in the backreference, via PATH_INFO. However, this is pretty messy - I would much prefer to pass that value as a URL parameter

I'm trying to configure a URL scheme that a) allows the ".html" extension to be omitted and b) sends 403 (Forbidden) errors under certain conditions. However, with the following setup:

Options +MultiViews
RewriteRule ^foo/bar$ - [F]

If I request /foo/bar I get a 404 (Not Found) error complete with a 'None could be negotiated' line in the error log. Requesting /foo/bar.html does work, so I'm guessing that the content-negotiation is taking 'priority' over the RewriteRule and keeps looking for alternative URLs until it finds a 200. In fact, I've pretty much confirmed that by testing with a bar.txt file and a RewriteRule that only targets the .html file.

So, is there a way of modifying mod_negotiation's behaviour to 'give up' as soon as it encounters a Forbidden error?

UPDATE

NB: It looks as though I can workaround the problem by disabling MultiViews and using mod_rewrite instead e.g.

RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule !.*\.html$ %{REQUEST_FILENAME}.html [L]

but I'm guessing that's less efficient, and it's a lot more config to deal with

I have a domain managed via Amazon's Route53 service. I have a Google Apps (free) account with 10 full users. To facilitate more than 10 users (we need about 12-14), I've been using a setup whereby extra users are represented as 'groups'. From time-to-time, however, they have problems sending/receiving mail, which is marked as spam. For each 'group', a normal gmail account is set up, and configured to receive mail from the group's email address, and send mail via Google's SMTP server.

For email sent from our site, we use Amazon's SES service.

I've tried a couple of online-spf checking services and the results look generally OK. However, I do have the following warning from Google's CheckMX tool:

There SHOULD be a valid SPF record.

SPF record specifies Gmail's IP addresses as valid for sending domain's messages If no other servers send mail on behalf of this domain then this record SHOULD be set to "v=spf1 include:_spf.google.com ~all"

My current SPF record reads:

"v=spf1 include:_spf.google.com include:amazonses.com ?all"

I'm not sure if this could be the cause of the problem; is there an adjustment I should make?

I'm running a site on AWS with the following setup:

1. Single m1.small instance (web server)
2. Single RDS m1.small db
3. Joomla 1.5

Generally, the site is performant, but is fairly low-traffic - say around 50-100 visits / hour. However, at peak time, we see about double that traffic. During peak time, pretty much every day:

1. CPU usage on the web server slowly climbs to 100%
2. CPU usage on the RDS server climbs quite quickly to about 30%, from an average of about 15
3. Database connections shoot up to about 140, from a normal average of about 2 or 3

The site is then occasionally unreachable, certainly according to pingdom monitoring.

Does anyone recognise this behaviour? Can you point me in the right direction to begin investigating? Of course, RDS makes it difficult to do things like slow query logging, so I've started by regularly dumping the mysql process list into a file to see if there's anything I can spot there, but it would be good to have something more concrete to investigate.

UPDATE

At least, can someone confirm that I'm definitely right in saying that the level of traffic implies the problem must be a specific type of query taking way longer than it should to execute? This would happen if a table gets locked, and many queries need to write to it, right? For this very reason, I've already changed the __session table type to InnoDB.

If I have the following directory structure:

foo/
    foo.txt
bar/
    foo/
        hum.txt

how do I exclude the top-level foo directory without excluding bar/foo? When I run:

rsync --exclude-from exclude-file ...

and the contents of exclude-file is merely:

foo

both directories are excluded.

I'm trying to optimise a page, and I'm seeing some strange behaviour. Each time I click on a link to the page, all resources are fetched from the server, responding with 200s. However, when I refresh the page (specifically, F5 in Firefox), all resources return a 304 and - of course - the page loads much faster as a result.

The main page returns a 200 in both cases. In the refresh case, If-Modified-Since headers are sent with the requests to the resources. However, in the 'clicking a link' case, they are not. What's the reason for that, and can I control it?

I've configured Amazon SES and obtained the first level of increased sending rate. So my limits are now 10,000 emails/day, 5 emails/second. My question is: do I need to enforce those rates within my own application (i.e. ensure no more than 5 emails are sent to the SMTP server per second) or can I rely on the server queuing any mail beyond the rate and sending it with a delay?

Of course, I recognise the need to still average out to those rates, but I have a function in my app that sends out ~ 1,500 emails in a 'batch', and I just need to know if I need to throttle it, or if I can just let it run through as quickly at it can.

I want to include a header in a bunch of pages, like so:

header.html:

<html>
  <head>
    <title>My site</title>
  </head>

To enable a page-specific title, I'm trying to use an SSI variable that I set in each page:

page1.html:

<!--#set var="TITLE" value="first page" -->
<!--#include file="header.html" -->

Then I”m modifying header.html to use that variable:

<title>My site - <!--#echo var="TITLE" --></title>

This works fine but, of course, it has the unfortunate effect that, if TITLE is not set, the result is:

<title>My site - (none)</title>

So I'm trying a variety of attempts at conditionally echoing that variable depending on whether it's none or not (e.g., <!--#if expr="TITLE != \(none\)" --> ... <!--#endif-->) …but nothing seems to work.

Seems like this would be quite a common requirement. Does anyone have a reference to a working solution?

I'm trying to set up a subdomain using name-based Virtual Hosts, according to the configuration format in this question.

As I understand it, this:

Finally, you can fine-tune the configuration of the virtual hosts by placing other directives inside the containers. Most directives can be placed in these containers and will then change the configuration only of the relevant virtual host. To find out if a particular directive is allowed, check the Context of the directive. Configuration directives set in the main server context (outside any container) will be used only if they are not overridden by the virtual host settings.

should mean that the task of moving from a single host to 1 host + 1 subdomain should be fairly straightforward. However, when I add the VirtualHost directives, I run into problems.

First off (and pretty catastrophic) is that the home page for the main domain returns a 403 (Forbidden). Could any of the following be causing that:

• The existing main domain has a large configuration file, with many settings. I'm placing the VirtualHost directives near the end of httpd.conf, just after the commented out instructions for virtual hosts
• There are lots of Redirects and Rewrites in a file that's included after the VirtualHosts - could they be clashing?
• I've put the absolute minimum in the VirtualHost directives, because I want to keep it as simple as possible. Would it be better to move the contents of the old httpd.conf into a separate file (e.g. main-domain.conf), Include that in one VirtualHost and create a separate conf file for the sub-domain, including that in the second VirtualHost directive?
• Finally, is there any simple way of debugging this, bearing in mind I'm working with a live site, so any changes have to be tested very quickly!

Update

OK, the error is, more specifically:

Directory index forbidden by rule

Other pages appear to work OK, so it must be some kind of problem with a redirect from the homepage. I guess this was partly why I asked if there are known clashes between VirtualHost setups and Redirects / Rewrites.

Should a <Directory "/path/to/docroot"> block that matches the DocumentRoot in a VirtualHost block be before or after the latter (or doesn't it matter)?

Update 2

OK, so it looks like what's not happening is the following redirect:

RewriteRule ^/$ /path/to/something/else [PT]

which I'm guessing is happening for one of two reasons:

1. The current config isn't even attempting to process this rewriterule. The rule itself is in an Included file, after the VirtualHost blocks.

2. The rule is no longer matching because the VirtualHost directive is causing the pattern match to fail. This seems, to me, by far the most likely ...

Update 3

Anyone think this might be relevant?

Note that, by default, rewrite configurations are not inherited. This means that you need to have a RewriteEngine on directive for each virtual host in which you wish to use it.

In other words, does that mean I need to:

<VirtualHost *:80>
    RewriteEngine on
</VirtualHost>

for each VH that needs to handle Rewrite* stuff, rather than just relying on RewriteEngine on in the globally-included file?

Update 4

Sadly, the change referred to in Update 3 hasn't fixed the problem at all

I'm investigating the configuration of Name-based virtual hosts, under apache, for the provisioning of a main domain, and its sub-domain. I've found one example on the web that suggests a configuration as follows:

Listen 80
NameVirtualHost *

<VirtualHost *>
ServerName www.domain.com
DocumentRoot /home/httpd/htdocs/
</VirtualHost>

<VirtualHost *>
ServerName subdomain.domain.com
DocumentRoot /home/httpd/htdocs/subdomain/
</VirtualHost>

which makes sense until the DocumentRoot directives. Am I correct in saying that it's less than wise to have one virtual hosts's document root, a descendant of another? Doesn't that configuration mean that the "subdomain" directory will server up content at both domains?

Using trac, I'm getting a response with the following header:

Cache-control: must-revalidate

Moreover, no 'Expires' header is being sent. Our local proxy, however, is caching these responses, so when an edit is made, pages need to be 'hard refreshed' to update. Is the proxy misbehaving? Other headers that might be relevant:

Connection          Keep-Alive
Proxy-Connection    Keep-Alive
Keep-Alive          timeout=15, max=100

I have the following entry in cron:

0 0 * * * /path/to/script.sh

in script.sh, there's a call to:

date +%Y%m%d

Even if this executed in zero-time, I'm assuming that the value of date will be for the date after midnight, rather than the date before. Is that correct?

How do I determine the primary IP on a Windows Server 2003 box?