APZ's questions

I am facing this problem where my uWSGI error logs show "UAAAAAAH my master disconnected: i will kill myself !!!".

This has happened twice in 2 weeks and surfaces only when we deploy code, which does a uwsgi --reload /tmp/X.pid. uWSGI is controlled by supervisor but this step just reloads uWSGI, no errors are shown on our Jenkins box but munin shows that CPU usage hits 100%, high disk IO and disk latency and requests to this box are timing out. I have to manually go in and restart uWSGI and that has fixed the issue on both occassions.

I would appreciate if someone can please share ideas in how to handle this issue.

uWSGI version: 2.0

supervisor: 3.0b1

Thanks

I am switching to debian from ubuntu and setting up RAID0 on my AWS boxes using user-data. Things were working fine on ubuntu but on wheezy

apt-get -y install mdadm --no-install-recommends

throws a user input box

If the system's root file system is located on an MD array (RAID), it needs to be started early during the boot sequence. If it is located on a logical volume (LVM), which is on MD, all constituent arrays need to be started. If you know exactly which arrays are needed to bring up the root file system, and you want to postpone starting all other arrays to a later point in the boot sequence, enter the arrays to start here. Alternatively, enter 'all' to simply start all available arrays. If you do not need or want to start any arrays for the root file system, leave the answer blank (or enter 'none'). This may be the case if you are using kernel autostart or do not need any arrays to boot. Please enter 'all', 'none', or a space-separated list of devices such as 'md0 md1' or 'md/1 md/d0' (the leading '/dev/' can be omitted).

I want to select all automatically. I have tried echo -e 'all' | apt-get -y install mdadm --no-install-recommends and also this but no success till now, can any one please help me out here.

I need advice/ ideas about how can I ensure data consistency in my s3 bucket. I have 2-3 users pushing and reading resources from s3. As one can easily see this leads to data inconsistency( User A is downloading that resource from S3 while User B is still writing to it).

We are syncing saltstack code from s3. We have one bucket setup per environment(prod, staging) and are using AWS IAM roles to control access. Problem happens when I am trying to grab data from s3 and the other developer is pushing code to that bucket, we end up with downloaded resource in a inconsistent state.

I would really appreciate if you guys can share your thoughts, ideas, experiences in dealing with such kind of problem.

Thanks

I am using Python Boto to launch new server in AWS VPC, I have SNS topic setup and I plan to get notified when my new server's System reachability check and Instance reachability check fails. For achieving this I am getting list of metrics(this is all a part of creating a new alarm as a part of server provision script):

cw_conn.list_metrics(dimensions={'InstanceId':instance_id}, metric_name=StatusCheckFailed)[0]

Problem is that when a instance starts all the metrics are not available immediately and I get IndexError: 'list index out of range' error.

I would really appreciate if anyone who has a solution to this problem can guide me or share a work around this issue.

Thanks

I have uWSGI set up through supervisor on a ubuntu 12.04 box and lately I noticed that when trying to tail the logs (all the logs in that user account) I get message that too many files opened , uWSGI is running as a not root user, I changed the hard and soft limit for this user and rebooted the machine and that fixed the issue. This box has just uwsgi running and my code deployment process stop supervisor which sends a INT signal to the uwsgi process and then start it after performing some operations. I want to conform that sending INT does make uWSGI close the files it has opened over time?

If not what is a better way to stop uWSGI so that it takes care of its opened files?

uWSGI docs uWSGI Docs says that SIGHUP would reload the server and would close the files it has opened but my requirements are stopping and starting uwsgi.

supervisor-uwsgi section

[program:uwsgi]
command=uwsgi --ini abcd.ini --gevent 200
autostart=False
stderr_logfile=/mnt/log/uwsgi.err.log
stdout_logfile=/mnt/log/uwsgi.log
stopsignal=INT

Any thoughts please?

I am working on a project which requires rolling out new celery servers if the broker(redis) queue is consistently higher than a predetermined threshold size and killing the new boxes when the queue size comes down. I have scripts to take care of introducing new boxes and making sure right configuration is running on them. I plan to check the queue by redis-cli after frequent(15 mins) intervals and if size is greater then threshold bring down the time gap between check(say 3 mins) check twice and then roll out new boxes and follow the same approach to kill the new servers.

I am new to celery and it would be great if I can get some advise on this approach.

Thanks

I am using HAProxy to load balance a bunch of PHP servers and want to introduce Varnish in the scene now.
HAP sends request to Varnish iff app-login cookie is not available, Varnish doesnt have to do anything here except to serve request(Cache HIT) or send it back to HAP incase of Cache MISS, HAP then selects a PHP server & resources are fetched from it and served to client by HAP through Varnish. I have the following config file.
I don't understand 2 things here;

1. in cache miss scenario varnish gives request back to HAP which checks for the app-login cookie, doesn't find it and sends it back to varnish (end less loop situation), I can have Varnish set a cookie and make HAP to check for that and select PHP servers backend on this basis(suggestions welcome).
2. Secondly, how can I achieve that when resources are fetched by HAP in case of MISS it sends to varnish and then it is served to client , this way Varnish builds up the cache eventually.

Please also let me know if some thing critical is missed out here.

Thanks in advance

Config File

#BE for Varnish is HAP in this machine 
backend default {
.host = "127.0.0.1";
.port = "80";
}

sub vcl_recv {
# HAP sends request to Varnish iff app-login cookie is not available
# Varnish doesnt have to do anything here except to serve request(Cache HIT) or
# send it back to HAP incase of Cache MISS, resouces are then fetched from PHP servers 
# and served to client by HAP through Varnish

# We unset the cookies here as they dont affect the response
 unset req.http.cookie;
# Lighttpd is already compressing resources, so we dont do it here.
 return (lookup); # Control is passed to vcl_hit or vcl_miss 
}
sub vcl_hit {
 return (deliver);
}

sub vcl_miss {
 return (fetch);
}

sub vcl_fetch {
  set obj.ttl = 1m;
  return (deliver);
}

sub vcl_deliver {
 if (obj.hits > 0) {
            set resp.http.X-Cache = "HIT";
    } else {
            set resp.http.X-Cache = "MISS";
    }
 return (deliver);
}

sub vcl_init {
  return (ok);
}

sub vcl_fini {
   return (ok);
}

I am using Ubuntu 10.04 and use puppet to change the timezone from UTC to PST when the server comes up. Puppet's timezone recipe which I am using is this: Puppet Time Recipe

Things were working fine with Ubuntu 10.04, now we are moving to Ubuntu 12.04 and this Puppet recipe doesn't work. Reason I figured out is in 10.04 /etc/localtime is a file with contents PST8PDT whereas in 12.04 it is a symlink(because of puppet recipe) and it is linked to SystemV/ but the path of SystemV directory is not correct. When I do a ls -l in /etc

lrwxrwxrwx 1 root root 18 Oct 18 18:38 localtime -> ../SystemV/PST8PDT

But there is no SystemV folder one level up of /etc. I changed the puppet recipe to this:

class timezone::pacific inherits timezone {
file { "/etc/localtime":
    require => Package["tzdata"],
 #   source => "file:///usr/share/zoneinfo/US/Pacific",
    source => "file:///usr/share/zoneinfo/SystemV/PST8PDT",

and it worked and localtime was no longer a link. Can anyone please help me understand why the puppet recipe which I inherited and was working fine on 10.04 didnt work on 12.04 and why changing the recipe to the new SystemV destination fixed it .

I attached a Second ENI to my Ubuntu OS machine on AWS VPC, the internal IP of this new ENI is 192.168.12.24. When I try to SSH in to this machine from another machine in the same VPC I get a connection time out. I am able to SSH in to the first ENI of the same machine both from inside and from outside the VPC.

Route command shows the following

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     *               255.255.240.0   U     0      0        0 eth0
192.168.0.0     *               255.255.240.0   U     0      0        0 eth1
default         192.168.0.1     0.0.0.0         UG    100    0        0 eth0

I am new to iptables and iprules, any help would be really appreciated.

I am trying to associate a secondary internal IP with my Amazon VPC small machine, I am using a Ubuntu based image with AMI ID:ami-6936fb00. Problem is that I am unable to figure out how to configure my OS(Ubuntu) to record the second private IP address ( this is required as mentioned in the AWS Docs: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#using-instance-addressing-eips)!

I have single ENI and have associated one extra private IP with it.

Any help would be appreciated.

Thanks

I have a mail server which was listed by Spamhaus and hence emails were not going out, I have removed the server from the list now my question is would postfix resend the emails which were not being sent out as the IP was listed?

Thanks

am using the haproxy socat to get the sessions data to plot it in cacti, /var/run/socket-haproxy is owned by www-data (the user which cacti uses) but when I try to execute this command as www-data am getting permission denied, any help would be much appreciated.

sudo su - www-data echo show stat | socat unix-connect:/var/run/socket-haproxy stdio | grep inbound | cut -d, -f 5
2012/01/11 15:58:18 socat[5448] E connect(3, AF=1 "/var/run/socket-haproxy", 25): Permission denied
-su: Can't open echo

Am new to cacti so please forgive me if my question sounds stupid, I have tried my best to find the solution for the problem but still needs help. I want to gather number of lines in the haproxy.log file from a remote front end(FE) machine, I have added this device successfully in cacti and graphs like load average are being drawn. I followed this: http://www.cacti.net/downloads/docs/html/how_to.html to add a graph to this device using shell script, I added a Data input method selecting Script/Command as the input type and /home/ubuntu/script.sh as the input string, added a output field too. Next I added a data source with out a template and selecting FE as the host, then as mentioned in the link I added the graph and things look fine till now, problem is that nothing is being plotted in the graph, cacti log files(Debug mode) shows the following:

CMDPHP: Poller[0] Host[5] DS[29] CMD: /home/ubuntu/script.sh, output: U 12/30/2011 12:35:07 AM - CMDPHP: Poller[0] Host[5] DS[29] WARNING: Result from CMD not valid. Partial Result: U

Now I know that script should print out the output only as this is one output field, my script contains the following

temp=$(sudo ssh -i /home/ubuntu/key.pem [email protected] '/var/log/haproxy.log | wc -l') echo $temp

the script is owned by ubuntu as there is no cacti user on cacti server, someone else installed cacti server and I have now been asked to take care of it.

If I execute the script from cacti server command line it works fine and am executing it as user ubuntu only.

Please help me I dont know where am I going wrong.

Thanks

I am trying to use EC2 tools to get all the machines with a particular tag in some type of array in /config/deploy/prod.rb file in Capistrano. Something like this:

In prod.rb file:

//untested command
workers-array[]=$(ec2-describe-instances -F vpc-id=1234 -F tag:Env=prod -F     tag:SystemType=worker)
 for(i=0;i<workers-array.len;i++){
     role :worker-A, workers-array[i]
   }

I am not sure how we can do this in capistrano, am new to ruby too. Guys any help on this would be really appreciated.

Am trying to detect disk thrashing by monitoring si, so from the vmstat command. I am monitoring other services using nagios and service check happens after every 5 minutes. For this thrashing service I want that nagios should check it after every 20 minutes and if the status returned is not OK(ie warning or critical) then thrashing service should be checked after every 3 minutes till the status returned by the service becomes OK. The service check time for all other service remains unchanged.

I am new to Nagios and any help on this would be really appreciated.

I am writing a shell script to check the vmstat si and so data at various time intervals

vmstat 1 sample output:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
0  0  45820 899252  86700 468520    0    0     0    60  127 5821 20  7 34  0
0  0  45820 899252  86704 468504    0    0     0    32   44  104  0  0 100  0

I want to use awk, sed to extract si and so into different variables for further usage. I Am new to awk, sed and am still struggling to find my way through. Can you show me how I can do this ?

I have the following script to update file based on MD5 comparison, script works perfectly from command line but when i put it in cron nothing happens. I only have root user on the system, script is owned by root, in crontab am calling the script like this: /bin/bash /home/papa/systems/sron.sh

Here is my script, I have tried all the solutions I have read till now but no progress, please help.

#!/bin/bash


PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"

MD5FILE=/tmp/.md5oldjar
MD5FILENEW=/tmp/.md5newjar
FILE_TO_CHECK=/tmp/rose.jar
URL=http://xxxx
DESTINATION=/tmp

rm -rf $FILE_TO_CHECK

wget $URL -P $DESTINATION


#echo generating MD5 for $FILE_TO_CHECK  

echo `md5sum $FILE_TO_CHECK` > $MD5FILENEW
MD5PRINT=`cat $MD5FILENEW`

if [ -z "$MD5PRINT" ]; then
    echo "ERROR Recived an empty MD5PRINT thats not valid, aborting"
    exit 1


fi

OLDMD5PRINT=`cat $MD5FILE`



if [ -z "$OLDMD5PRINT" ]; then
    echo "Got an empty string from the oldfile, aborting"
    echo $OLDMD5PRINT 
    exit 1
fi


if [ "$OLDMD5PRINT" != "$MD5PRINT" ]; then 
    echo "File has changed"
    echo "updating MD5 file...."
    echo $MD5PRINT > $MD5FILE
    echo "file content:"
    cat $MD5FILE
    echo `date` >> /home/log
    echo done
else
    echo nothing >> /home/log
fi