Home / server / Questions / 582655
In Process
raphink
Asked: 2014-03-18 08:59:24 +0800 CST

MCollective with RabbitMQ stops responding

  • 772

Using MCollective (2.2.3) with RabbitMQ (3.1.5) on RedHat 6.5, MCollective stops responding after a first successful ping.

When I restart RabbitMQ, it works again.

root@rabbitmqserver# /etc/init.d/rabbitmq-server restart
Restarting rabbitmq-server: SUCCESS
rabbitmq-server.
root@rabbitmqserver# rabbitmqctl list_consumers -p /mcollective  |  grep directed
amq.gen-1amX9Z9w6jBKNP7IQrMfjg  <[email protected]>  Q_/exchange/mcollective_directed/srv114 false
amq.gen-dxFvTuxvQZOU0d1e7XuJJg  <[email protected]>  Q_/exchange/mcollective_directed/srv98  false
amq.gen-lwttnRCCTON9hgPg7X2zZw  <[email protected]> Q_/exchange/mcollective_directed/srv99  false
amq.gen-pG4o5csOGcam9mxvIQvQ7g  <[email protected]>  Q_/exchange/mcollective_directed/srv57  false
amq.gen-xtRFm6aAoeV2mDO9dTAyyA  <[email protected]>  Q_/exchange/mcollective_directed/srv115 false


user@mco-client$ mco ping

srv114                        time=66.49 ms
srv57                         time=69.34 ms

---- ping statistics ----
2 replies max: 69.34 min: 66.49 avg: 67.91 
user@mco-client$  mco ping


---- ping statistics ----
No responses received

root@rabbitmqserver# rabbitmqctl list_consumers -p /mcollective  |  grep directed
amq.gen-1amX9Z9w6jBKNP7IQrMfjg  <[email protected]>  Q_/exchange/mcollective_directed/srv114 false
amq.gen-dxFvTuxvQZOU0d1e7XuJJg  <[email protected]>  Q_/exchange/mcollective_directed/srv98  false
amq.gen-lwttnRCCTON9hgPg7X2zZw  <[email protected]> Q_/exchange/mcollective_directed/srv99  false
amq.gen-pG4o5csOGcam9mxvIQvQ7g  <[email protected]>  Q_/exchange/mcollective_directed/srv57  false
amq.gen-xtRFm6aAoeV2mDO9dTAyyA  <[email protected]>  Q_/exchange/mcollective_directed/srv115 false

If I try using another command (such as package), it fails right after the discovery (after restarting RabbitMQ that is):

user@mco-client$ mco package status bash
Do you really want to operate on packages unfiltered? (y/n): y

 | [ >                                                             ] 0 / 2
[... stalled ...]

After a while, I start seeing this in my MCollective server logs:

E, [2014-03-18T11:51:01.332969 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T11:55:35.759653 #24456] ERROR -- : agents.rb:138:in `dispatch' Execution of discovery failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T11:55:35.760040 #24456] ERROR -- : agents.rb:139:in `dispatch' /usr/lib/ruby/gems/1.8/gems/stomp-1.2.2/lib/stomp/connection.rb:331:in `publish'
        /usr/libexec/mcollective/mcollective/connector/rabbitmq.rb:184:in `publish'
        /usr/lib/ruby/site_ruby/1.8/mcollective/message.rb:230:in `publish'
        /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
        /usr/lib/ruby/site_ruby/1.8/mcollective/message.rb:218:in `publish'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:129:in `reply'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:83:in `agentmsg'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:132:in `dispatch'
        /usr/lib/ruby/1.8/timeout.rb:67:in `timeout'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:125:in `dispatch'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `initialize'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `new'
        /usr/lib/ruby/site_ruby/1.8/mcollective/agents.rb:121:in `dispatch'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:82:in `agentmsg'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:55:in `run'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:50:in `loop'
        /usr/lib/ruby/site_ruby/1.8/mcollective/runner.rb:50:in `run'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:30:in `daemonize_runner'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:13:in `daemonize'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `fork'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:5:in `daemonize'
        /usr/lib/ruby/site_ruby/1.8/mcollective/unix_daemon.rb:20:in `daemonize_runner'
        /usr/sbin/mcollectived:43
E, [2014-03-18T11:56:01.336954 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:01:01.343415 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:06:01.347218 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection
E, [2014-03-18T12:11:01.354069 #24456] ERROR -- : base.rb:29:in `run' Sending registration message failed: Stomp::Error::NoCurrentConnection

even though the machine is still listed as a consumer on the RabbitMQ server.

What could be going wrong here?

rabbitmq

1 Answers

  1. RabbitMQ had disk_free_limit set to 1000000000 in rabbit.app. This caused RabbitMQ to drop connections when available disk space in /var was lower than 950MB…

    • 0