Home / server / Questions

Questions[god](server)

Martin Hope
99miles
Asked: 2012-02-11 15:08:44 +0800 CST
  • 8

Most of the time when I reboot my Ubunutu server, I get "Waiting for Redis to shutdown…" over and over and it never stops until I force reboot. What could cause this?

I have http://godrb.com/ monitoring that resque stays running, so it could be related to that.

ubuntu redis god
Martin Hope
injekt
Asked: 2011-04-25 12:00:41 +0800 CST
  • 2

I'm currently using Nginx to proxy requests to a Unicorn server running a Sinatra application. The application only has a couple of routes defined, those of which make fairly simple (non costly) queries to a PostgreSQL database, and finally return data in JSON format, these services are being monitored by God.

I'm currently experiencing extremely slow response times from this application server. I have another two Unicorn servers being proxied via Nginx, and these are responding perfectly fine, so I think I can rule out any wrong doing from Nginx.

Here is my God configuration:

# God configuration

APP_ROOT = File.expand_path '../', File.dirname(__FILE__)

God.watch do |w|
  w.name = "app_name"
  w.interval = 30.seconds # default

  w.start = "cd #{APP_ROOT} && unicorn -c #{APP_ROOT}/config/unicorn.rb -D"

  # -QUIT = graceful shutdown, waits for workers to finish their current request before finishing
  w.stop = "kill -QUIT `cat #{APP_ROOT}/tmp/unicorn.pid`"

  w.restart = "kill -USR2 `cat #{APP_ROOT}/tmp/unicorn.pid`"

  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = "#{APP_ROOT}/tmp/unicorn.pid"

  # User under which to run the process
  w.uid = 'web'
  w.gid = 'web'

  # Cleanup the pid file (this is needed for processes running as a daemon)
  w.behavior(:clean_pid_file)

  # Conditions under which to start the process
  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
    end
  end

  # Conditions under which to restart the process
  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.above = 150.megabytes
      c.times = [3, 5] # 3 out of 5 intervals
    end

    restart.condition(:cpu_usage) do |c|
      c.above = 50.percent
      c.times = 5
    end
  end

  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Here is my Unicorn configuration:

# Unicorn configuration file

APP_ROOT = File.expand_path '../', File.dirname(__FILE__)

worker_processes 8

preload_app true

pid "#{APP_ROOT}/tmp/unicorn.pid"

listen 8001

stderr_path "#{APP_ROOT}/log/unicorn.stderr.log"
stdout_path "#{APP_ROOT}/log/unicorn.stdout.log"

before_fork do |server, worker|
  old_pid = "#{APP_ROOT}/tmp/unicorn.pid.oldbin"

  if File.exists?(old_pid) && server.pid != old_pid
    begin
      Process.kill("QUIT", File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
end

I have checked God status logs but it appears CPU and Memory Usage are never out of bounds. I also have something to kill high memory workers, which can be found on the GitHub blog page here.

When running a tail -f on the Unicorn logs I see some requests, but they're far and few between, when I was at around 60-100 a second before this trouble seemed to have arrived. This log also shows workers being reaped and started as expected.

So my question is, how would I go about debugging this? What are the next steps I should be taking? I'm extremely baffled that the server will sometimes respond quickly, but at others time it's very slow, for long periods of time (which may or may not be peak traffic times).

Any advice is much appreciated.

ruby nginx postgresql unicorn god
Martin Hope
organicveggie
Asked: 2011-04-23 16:20:19 +0800 CST
  • 7

I have god installed on at least a dozen (or more) servers running CentOS 5.5 in both i386 and x86_64 flavors that work perfectly. I just setup two new CentOS 5.5 x86_64 servers and installed God, but I'm getting an event system error: 

$ tail /var/log/god.log 
E [2011-04-22 12:33:17] ERROR: Condition 'God::Conditions::ProcessExits' 
requires an event system but none has been loaded 

$ god check 
using event system: none 
[fail] event system did not load 

$ uname -a 
Linux server2.example.com 2.6.18-238.9.1.el5 #1 SMP Tue Apr 12 18:10:13 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

I can't find any cn or netlink kernel module on any of my CentOS servers. Yet I have other servers that work fine: 

$ god check 
using event system: netlink 
starting event handler 
forking off new process 
forked process with pid = 17559 
killing process 
[ok] process exit event received 

$ uname -a 
Linux server1.example.com 2.6.18-194.el5 #1 SMP Fri Apr 2 14:58:14 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

All servers run ruby v1.8.6-399:

# ruby -v
ruby 1.8.6 (2010-02-05 patchlevel 399) [x86_64-linux]

Ruby comes from the ELFF repo:

# rpm -qi ruby
Name        : ruby                         Relocations: (not relocatable)
Version     : 1.8.6.399                         Vendor: Bravenet ELFF <[email protected]>
Release     : 2.el5                         Build Date: Fri Apr 16 18:53:48 2010
Install Date: Thu Mar 24 11:23:48 2011         Build Host: el-build.local
Group       : Development/Languages         Source RPM: ruby-1.8.6.399-2.el5.src.rpm
Size        : 1738695                          License: Ruby or GPLv2
Signature   : DSA/SHA1, Fri Apr 16 19:07:49 2010, Key ID 551751dfe8b071d6
Packager    : Bravenet ELFF <[email protected]>

I did a little digging and can see the exception getting thrown when God tries to load the Netlink event handler:

no such file to load -- netlink_handler_ext

What could possibly be different between my servers? Am I missing something simple?

centos monitoring god