Tom's questions

I'm new to Kubernetes and I'm trying to figure out what resource limits I should set for my php webapp.

My first guess is that if I have configured php fpm to use a maximum of 256M of memory then should I configure kubernetes to limit to 256M as well.

The setup would be like this:

1. In the Dockerfile that builds the php container, I set php memory_limit to 256M:

   sed -i '/^;php_admin_value\[memory_limit\]/cphp_value[memory_limit] = 256M' /usr/local/etc/php-fpm.d/www.conf
   

2. In my Kubernetes Deployment I set resource limits like this on the fpm container:

   resources:
     requests:
       cpu: "500m"
       memory: "128Mi"
     limits:
       cpu: "1"
       memory: "256Mi"
   

The thing that makes me ask the question is that PHP.net's documentation says:

memory_limit integer

This sets the maximum amount of memory in bytes that a script is allowed to allocate. This helps prevent poorly written scripts for eating up all available memory on a server.

php fpm can serve many requests at the same time and I guess each request runs a script completely independent from the others. They do not share the memory_limit setting. If that's the case, maybe my Kubernetes container limit should be higher than the php memory_limit setting?

Ok, but how much higher!? If I limit cpu: 1 I guess it means there can't be real parallel-processing going on, but there can still be many processes sharing the CPU, with the OS switching when blocked by IO. I think the fpm pm.max_children setting will tell me how many scripts can be run so how much memory can be taken at the maximum. But this is where it gets confusing because in our non-Kubernetes setup we choose the max_children based on how much memory the instance has. The formula we use is:

max_children = (RAM - 2Gb) / 80Mb

# Where:
#  * We Reserve 2Gb for the system.
#  * We saw that each request needs on average 40Mb of memory, so we chose 80Mb to be safe.

This equation goes out the window in the Kubernetes world as the RAM is prescribed at runtime and you don't need to reserve any for the system because the container ONLY runs php.

I'm thinking to use:

memory_limit = 256M
max_children = 10

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "1"
    memory: "2Gi"

• Normal case: 10 children using 40M each. Memory usage: 400M
• High case: 10 children using 80M each Memory usage: 800M
• Max case: 10 children using 256M each Memory usage: 2560M. Very unlikely to happen but if it did, Kubernetes would set Pod to OOMKilled I think because it's more than the limit.

Does this thinking seem reasonable? I don't think it'll allow for good packing because 10 children will hardly ever need the 2Gi limit. We will use HPA to add more pods if resources are stretched.

What did you do in your Kubernetes setup?

We use a service who's API will reject requests unless the source IP has been previously whitelisted. They only give us 3 slots which is a problem because we have more than 3 machines that need to use the API.

What is the most common technique to workaround this issue?

Note: I'm not trying to do anything against the Terms & Conditions of the 3rd-party API. We are using ResellerClub and I contacted them to ask for more slots but they replied:

I request you to kindly route your servers to a few set of IPs.

Hence this question.

Thoughts:

• I was thinking we could solve the problem by running a sort of proxy that acts as a man-in-the-middle. Instead of making API requests to the 3rd-party we make them to our proxy which bounces the request to the 3rd-party so that all requests seem to all come from one IP in their eyes. Is there common software for doing this kind of thing? It seems simpler to do this than the below thoughts, but am I wrong?
• Is using "a NAT instance" something I should be looking into? e.g. http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html . It looks complicated - Is there not a simpler solution? (Running a extra instance with extra networking complexity is a shame).
• Since we use Docker, could Weave be relevant?
• Could we attach a static IP to the VPC gateway? I saw it's possible with the AWS Storage Gateway (source) - not sure about a regular vpc igw though?

Our architecture: We use AWS and have our instances in a VPC running behind a ELB. We frequently bring up new instances without knowing the IP addresses in advance. We run identical software on all machines. The machines run CoreOS and our app runs in Docker containers.

If I use a environment variable in the nginx config, and nginx is configured to serve only static files (html,js,css - e.g. a AngularJs app), is there any way I can use the environment variable within a JS file that nginx serves? Or is the only way to do this to run a non-static server, e.g. io.js, php, etc.

Btw you can't use environment variables natively in nginx's configuration.

When I talk about environment vars in the nginx config I mean like in this post: How can I use environment variables in Nginx.conf where they use env APP_WEB_1_PORT_5000_TCP_ADDR; and $ENV{"APP_WEB_1_PORT_5000_TCP_ADDR"};

Explain exactly your use case

My specific use case is that I have a nginx-powered AngularJS app in a docker container. The app is a "Single Page Application" that consumes a API running on another system altogether. Currently I run a different docker container between production and staging because the app has some config that is different, for example the Google-Analtyics code. This environment-specific data is kept in a config.js file and the values are currently hardcoded, one value for the master branch in git, and different value for the staging branch. I want to change the design so I can use the same container for both production and staging. I want to pass a ENV var into the container when I run it (docker run -e GACODE=UA-12345-6 ...) and have nginx use the ENV var (via env GACODE; and $ENV{"GACODE"} so the config.js file can use the GoogleAnaltyics codes that are passed in, rather than hardcoding them). I don't know whether this is possible or not (hence the question ;)). Using only nginx makes my container a single-process, whereas if I have to use io.js then I'll need multiple linked containers and more moving parts is more complex).

I have php-fpm in a docker container and in the Dockerfile I edit the fpm config file (/etc/php5/fpm/pool.d/www.conf) to set up access logs to go to /var/log/fpm-access.log and error logs to go to /var/log/fpm-php.www.log:

# Do some php-fpm config
#  Redirect worker stdout and stderr into main error log
#  Activate the fpm access log
#  Enable display errors
#  Enable the error log
RUN sed -i '/^;catch_workers_output/ccatch_workers_output = yes' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;access.log/caccess.log = /var/log/fpm-access.log' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;php_flag\[display_errors\]/cphp_flag[display_errors] = off' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;php_admin_value\[error_log\]/cphp_admin_value[error_log] = /var/log/fpm-php.www.log' /etc/php5/fpm/pool.d/www.conf && \
    sed -i '/^;php_admin_flag\[log_errors\]/cphp_admin_flag[log_errors] = on' /etc/php5/fpm/pool.d/www.conf

This works fine - I can get a shell into the container to see the logs. But... it is not best-practice.

The problem is when I try to use the docker log collector - I need php-fpm to log to stdout or stderr so docker can capture them and provide them to the docker logs command.

I tried to do this in the Dockerfile (which is a idea I copied from the official nginx docker image):

# Redirect fpm logs to stdout and stderr so they are forwarded to the docker log collector
RUN ln -sf /dev/stdout /var/log/fpm-access.log && \
    ln -sf /dev/stderr /var/log/fpm-php.www.log

This is not working - no access logs are seen from docker logs - I'm trying to figure out why? Did anyone else that uses fpm in docker manage to get logging working to the docker log collector?

The official nginx docker image (Dockerfile) uses the following trick to hand off its logs to stdout and stderr so that they are captured by the docker log collector and viewable using docker logs <container-name>:

# forward request and error logs to docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log

I want to do the same, however I have supervisord as PID 1 in my container and it supervises the nginx process and captures stdout and stderr and puts it in it's own log file. Therefore the logs do not reach the docker log collector.

Here is the relevant block in my supervisord.conf

[program:nginx]
command=/usr/sbin/nginx -g "daemon off;"
priority=990
; NOTE: We do not want to redirect stdout and stderr of a nginx process to a logfile because we want docker log collector to get them.
stdout_logfile= NOT SURE WHAT TO PUT HERE?
stderr_logfile= NOT SURE WHAT TO PUT HERE?
username=www-data
autorestart=true

The supervisord docs are great (http://supervisord.org/configuration.html) but for stdout_logfile they don't give me the answer I need - namely, how do I configure supervisord to not intercept a processes stdout / stderr ( or to capture to a log file but also forward to normal stdout / stderr).

What I tried already:

1. stdout_logfile=NONE - Get no logs neither stdout or to a file
2. stdout_logfile=/var/log/supervisor/%(program_name)s.log - Get logs to a file but not stdout.
3. Not defining stdout_logfile - Get logs to a randomly named file but not stdout.

4. stdout_logfile=/dev/stdout - Supervisord gives a error:

   CRIT uncaptured python exception, closing channel (stdout)> (:[Errno 29] Illegal seek [/usr/lib/python2.7/dist-packages/supervisor/supervisord.py|runforever|233] [/usr/lib/python2.7/dist-packages/supervisor/dispatchers.py|handle_read_event|231] [/usr/lib/python2.7/dist-packages/supervisor/dispatchers.py|record_output|165] [/usr/lib/python2.7/dist-packages/supervisor/dispatchers.py|_log|141] [/usr/lib/python2.7/dist-packages/supervisor/loggers.py|info|273] [/usr/lib/python2.7/dist-packages/supervisor/loggers.py|log|291] [/usr/lib/python2.7/dist-packages/supervisor/loggers.py|emit|186] [/usr/lib/python2.7/dist-packages/supervisor/loggers.py|doRollover|195])

I can host a dynamic website through Amazon CloudFront because they have CNAME Wildcard Support. However, some pages of my Site use HTTPS. Amazon have some documentation about how to associate your SSL certificate with a CloudFront distribution but the pricing shows that it costs $600 per month, too much for me.

My question is... is it possible to configure CloudFront so HTTP requests are served from CloudFront but HTTPS requests are ignored by CloudFront and passed directly to the origin (in my case, a Elastic Load Balancer which can decrypt SSL at no extra cost)?

We use the HttpLimitReqModule nginx mondule for rate throttling and have found that it conflicts with our "maintenance mode" because both components are using http status code 503.

When throttling is activated (via the limit_req directive), nginx would normally serve a 503, but unfortunatly our maintenance mode location is used, which results in a 302 to our Amazon S3 hosted maintenance page. A 302 for a throttled request is not a good outcome.

I was wondering how other people deal with this issue? Should I for example use a different status code for our maintenance page, but if so, what?

Ideally for throttled requests, I don't want any page served, only a 503 response header - it needs to be as lightweight as possible because the whole point is to stop the server being overwhelmed.

For reference, this is the nginx config that we use for "Maintenance Mode":

server {
    ...

    # Redirect processing of 503 error pages into a named location:
    error_page 503 @maintenance;

    # "Maintenance Mode" is off by default - Use a nginx variable to track state.
    set $maintenance off;

    # Switch on "Maintenance Mode" if a certain file exists.
    if (-f /var/www/mysite/shared/maintenanceON) {
        set $maintenance on;
    }

    if ($maintenance = on) {
        return 503; # 503 - Service unavailable
    }

    location @maintenance {
        # Redirect the request to our maintenance page in Amazon S3.
        rewrite ^(.*)$ http://mysite.s3-website-us-east-1.amazonaws.com/ break;
    }
    ...
    # Process the php files - pass to php-fpm.
    location ~ \.php {
        # Limit aggressive bots and crawlers to 30 requests per minute.
        limit_req zone=bots;

        fastcgi_pass 127.0.0.1:$fastcgi_port;
    }
    ...

We use capistrano for website deployments and our Apache document root is a symlink to a particular code release. The deployment procedure switches the symlink from the old release to the new release as the final step of the deployment.

We are migrating our webservers from real servers running RHEL 5.6 to Amazon EC2 virtual machines running Ubuntu 11.10 and the new servers are suffering from a problem where Apache doesn't immediately notice the change to it's document root when the symlink is switched. It can take a second or so (and I think I've even seen it take a couple of minutes). It's kind of like Apache has cached the physical path of the symlink for some time.

Does anyone know some Apache settings I could look at to get it to "scan" for changes to it's served files quicker.

Thoughts:

• I read that the disks on virtual machines are much slower (since they are network attached storage). Perhaps the filesystem cache somehow works differently too? If so, is there anything that can be done?
• The website runs PHP code. Perhaps there is some PHP config differences between RHEL and Ubuntu? I checked realpath_cache_ttl but both servers have it commented out:

e.g.

; Duration of time, in seconds for which to cache realpath information for a given
; file or directory. For systems with rarely changing files, consider increasing this
; value.
; http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-ttl
;realpath_cache_ttl = 120

• We do use the APC opcode cache but don't think it's the issue due to experimentation. The PHP code is in different file paths for each deployment and we ensure stat=1.
• Here is a similar question that is very interesting: 294107 - but doesn't provide an answer for me.
• One solution would be to reload Apache everytime we modify the document root symlink. I'll do this if we can't find another solution.

I'm seeing a few of these messages in the Apache error log (/var/log/httpd/error_log) at the rate of about one per half hour, often in small batches:

(process:23210): libgnomevfs-WARNING **: Unable to create ~/.gnome2 directory: Permission denied

It's not a problem but it's ugly and clouds the error log from useful errors.

Notes:

• I don't know the url that triggers these warnings. If I did, I'd be most of the way to solving why they occur. I guess they are triggered by a crawler guessing at a particular URL which triggers the error.
• Our Apache uses the "nobody" user. The "nobody" user is an unprivileged user. This means that the user doesn't own anything, doesn't have a home folder and doesn't have execute permissions to anything.

Does anyone know what causes this error?

> php -v
PHP 5.3.10 (cli) (built: Feb  2 2012 17:34:38) 
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies
    with Suhosin v0.9.33, Copyright (c) 2007-2012, by SektionEins GmbH
> httpd -v
Server version: Apache/2.2.3
Server built:   Oct  6 2011 12:11:58
> cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.7 (Tikanga)

I want to have a SQL script run whenever mysql starts but I can't get it working in Ubuntu 11.10.

I added a "init-file" option to the mysql config file:

> sudo emacs -nw /etc/mysql/my.cnf
...
[mysqld]
init-file=/etc/mysql/mysqlinit.sql
...

But when I restart mysql, it fails with a "File not found" error:

> tail /var/log/mysql/error.log
111111  7:41:06 [ERROR] /usr/sbin/mysqld: File '/etc/mysql/mysqlinit.sql' not found (Errcode: 13)
111111  7:41:06 [ERROR] Aborting

But the file certainly exists and is readable:

> ls -l /etc/mysql/mysqlinit.sql
-rwxr-xr-x 1 mysql mysql 30 2011-11-09 05:06 /etc/mysql/mysqlinit.sql

Any ideas how to solve this? Is this a Ubuntu oddness or did I do something dumb?

Info:

I'm running Ubuntu 11.10 and MySQL 5.1.

> mysqld --version
mysqld  Ver 5.1.58-1ubuntu1 for debian-linux-gnu on x86_64 ((Ubuntu))

In different server environments, the PHP $_SERVER['DOCUMENT_ROOT'] super global sometimes has a trailing slash and sometimes it does not. I would have thought this issue is directly related to how the Apache DocumentRoot is defined in the httpd.conf file:

i.e. I would have thought that if httpd.conf contains no trailing slash:

<VirtualHost *:8880>
    DocumentRoot /var/www/live/current
    ...

then echo $_SERVER['DOCUMENT_ROOT'] should give /var/www/live/current

and if httpd.conf does contain a trailing slash:

<VirtualHost *:8880>
    DocumentRoot /var/www/live/current/
    ...

then echo $_SERVER['DOCUMENT_ROOT'] should give /var/www/live/current/

This is the case on Ubuntu 10.04 but on RHEL 5.5 a trailing slash is added to $_SERVER['DOCUMENT_ROOT'] even if none was defined on Apache.

Any idea why this happens? Is there a configuration parameter that I'm missing?

For reference:

• PHP 5.3.3 of RHEL (issue occurs): PHP 5.3.3 (cli) (built: Jul 23 2010 16:26:53)
• PHP version of Ubuntu (no issue): PHP 5.3.2-1ubuntu4.2 with Suhosin-Patch (cli) (built: May 13 2010 20:03:45)

We are moving from a 1 webserver setup to a two webserver setup and I need to start sharing PHP sessions between the two load balanced machines. We already have memcached installed (and started) and so I was pleasantly surprized that I could accomplish sharing sessions between the new servers by changing only 3 lines in the php.ini file (the session.save_handler and session.save_path):

I replaced:

session.save_handler = files

with:

session.save_handler = memcache

Then on the master webserver I set the session.save_path to point to localhost:

session.save_path="tcp://localhost:11211"

and on the slave webserver I set the session.save_path to point to the master:

session.save_path="tcp://192.168.0.1:11211"

Job done, I tested it and it works. But...

Obviously using memcache means the sessions are in RAM and will be lost if a machine is rebooted or the memcache daemon crashes - I'm a little concerned by this but I am a bit more worried about the network traffic between the two webservers (especially as we scale up) because whenever someone is load balanced to the slave webserver their sessions will be fetched across the network from the master webserver. I was wondering if I could define two save_paths so the machines look in their own session storage before using the network. For example:

Master:

session.save_path="tcp://localhost:11211, tcp://192.168.0.2:11211"

Slave:

session.save_path="tcp://localhost:11211, tcp://192.168.0.1:11211"

Would this successfully share sessions across the servers AND help performance? i.e save network traffic 50% of the time. Or is this technique only for failovers (e.g. when one memcache daemon is unreachable)?

Note: I'm not really asking specifically about memcache replication - more about whether the PHP memcache client can peak inside each memcache daemon in a pool, return a session if it finds one and only create a new session if it doesn't find one in all the stores. As I'm writing this I'm thinking I'm asking a bit much from PHP, lol...

Assume: no sticky-sessions, round-robin load balancing, LAMP servers.

If a NFS client reads a file, does any network traffic happen? or is it just writes (by any node) that result in network traffic?

In particular, I'm interested in whether doing a file_exists() call in a PHP script will be more expensive if the target file is on NFS than if the target file is on the native file system?

For reference, this is my NFS configuration:

Server /etc/exports:

/var/www/staging/uploads 192.168.0.1(rw,sync,no_root_squash)

Client /etc/fstab:

192.168.0.1:/var/www/staging/uploads /var/www/staging/uploads nfs soft,intr,rsize=8192,wsize=8192

Possible Duplicate:
stop apache from asking for SSL password each restart

When I created my SSL certificate I used a pass phrase in the key. Whenever I restart my web server (Apache or Nginx) they ask for a password:

Apache:

Some of your private key files are encrypted for security reasons. In order to read them you have to provide the pass phrases.

Server www.example:443 (RSA)

Enter pass phrase:

Nginx:

Starting nginx: Enter PEM pass phrase:

Entering the password each time is fast getting annoying and I'm worried about downtime when the machine is next rebooted.

Is there a way to automatically provide the PEM pass phrase when the webserver is restarted? or do I have to get the SSL certificate re-issued using a key where the pass phrase has been removed?

If I remove the pass phrase, what are the security implications? Is it anything to worry about?

I'm using the "page speed" extension for Firebug to try to optimise a website and I'm currently working on the following suggestion: "Serve static content from a cookieless domain".

I have created a separate sub-domain for some content so that I have www.example.com and images.example.com but how do I specify that images.example.com is cookieless? Can I enforce that it is cookieless in a webserver such as Nginx or Apache or is it simply a matter of making sure not to set any cookies in this domain in the serverside code (e.g. PHP)?

The reason why I'm asking is because "Page Speed" is still showing the same recommendation even after I tried to fix it - so I guess some cookies must have slipped through. I can't see any cookies in my browser cookie search but if I examine the HTTP headers of the resource I can see:

Cookie  __utma=73051794.676740941.1271792323.1277710025.1277900715.20; __utmz=73051794.1271792323.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmx=73051794.00009825591030858779:3:0; __utmxx=73051794.00009825591030858779:2295429:2592000; __gads=ID=0a768e3407302ff8:T=1272608001:S=ALNI_MZ-GKhg3ETniU0TVftk0DdGyUypkQ

Anyone know how I can stop cookies from my sub-domain?

On a Linux server, I need to find all files with a certain file extension in the current directory and all sub-directories.

Previously, I have always used the following command:

find . -type f | grep -i *.php

However, it doesn't find hidden files, for example .myhiddenphpfile.php. The following finds the hidden php files, but not the non-hidden ones:

find . -type f | grep -i \.*.php

How can I find both the hidden and non-hidden php files in the same command?

The following command is useful to capture the current processes that are taking up the most CPU and store to a file:

top -c -b -n 1 > top.log

The -c flag is particularly useful because it gives you the command line arguments of each process rather than just the process name.

Without -c:

2497 root      18   0 11264 5888 1524 S  0.0  0.1   0:03.31 miniserv.pl

With -c (additional useful command info):

2497 root      18   0 11264 5888 1524 S  0.0  0.1   0:03.31 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf

The problem is that each line of output is truncated to fit on the current terminal window. This is ok if you can have a wide terminal because you have a lot of the output but if your terminal is only 165 characters wide, you only get 165 characters of information per process and it is often not enough characters to show the full process command. This is a particular problem when the command is executed without a terminal, for example if you do it via a cron job.

Does anyone know how to stop top truncating data or force top to display a certain number of characters per line?

This is not urgent because there is an alternative method of getting the top 10 processes ordered by CPU usage:

ps -eo pcpu,pmem,user,args | sort -r -k1 | head -n 10

I have a problem with the sendmail DNS timeout period when trying to sent to a email address which has a DNS issue. Let's say the problematic email address is [email protected]

I can see there is a DNS issue with this domain using:

>host -t mx example.com
;; connection timed out; no servers could be reached

I can test and time the sendmail call to the problematic email address using:

> time echo "This is a test message" | /usr/lib/sendmail [email protected] [email protected]
real 2m36.252s
user 0m0.004s
sys  0m0.005s

Currently, it takes 2m36s for sendmail to timeout! This causes our PHP webpages that try to send to this email address to hang and eventually return an error to the browser.

My question is: How do I reduce this timeout?

I have seen a few sendmail related timeout configurations:

> grep resolver /etc/mail/sendmail.cf
Timeout.resolver.retrans=5s
Timeout.resolver.retry=4

Sendmail seems to be set to retry 4 times, with a 5s delay each re-transmission.

But I haven't found the actual DNS timeout period?

For reference: This is occurring on Red Hat Enterprise Linux Server release 5.2, sendmail 8.13.8, Linux Kernel: 2.6.18, Apache version: 2.2.3, PHP version: 5.3.0. Note: This question is a follow up from Sendmail delays on some email addresses

I have scheduled a cron job to run every minute but sometimes the script takes more than a minute to finish and I don't want the jobs to start "stacking up" over each other. I guess this is a concurrency problem - i.e. the script execution needs to be mutually exclusive.

To solve the problem I made the script look for the existence of a particular file ("lockfile.txt") and exit if it exists or touch it if it doesn't. But this is a pretty lousy semaphore! Is there a best practice that I should know about? Should I have written a daemon instead?

I want to be able to flip back and forwards between two buffers in Emacs in the same way I can flip between tabs in some other editors. The following Ctrl + TAB key binding works nicely in XEmacs:

(defun buffer-shimmy ()
  "Back to the previous buffer"
  (interactive)
  (let ((buff (car (buffer-list))))
    (message "Previous buffer: %s" buff)
    (switch-to-buffer (other-buffer buff))
   )
)
(global-set-key (kbd "<C-tab>") 'buffer-shimmy)
;; or (define-key global-map [(control tab)] 'buffer-shimmy) 

However, it doesn't work when I open Emacs using "emacs -nw" (or if I'm using Emacs in an SSH session).

After finding this link - I think perhaps what I want is not possible?

Apparently, in the terminal (i.e. outside the Windowing system) adding a control modifier to the ASCII character for 'TAB' is nonsensical. Whereas, if going via a windowing system, Emacs can see you pressing Ctrl + TAB as <tab> with a control modifier, which can be represented in the .emacs file as C-<tab> or <C-tab> or [(control tab)].

Is there a workaround that fixes this issue?

Reference software: Emacs version: 22.2.1 on Ubuntu 9.04 using gnome-terminal with the default profile.