Questions[erlang](server)

I recently installed RabbitMQ on a Debian 11 server and noticed that the vast majority of threads now belong to the same process - beam.smp. From what I found on the Internet, this process is Erlang's runtime or VM, a dependancy of RabbitMQ. The output of ps indicates that it is a single process with 1200 threads.

$ ps -e | grep beam.smp | wc -l
1
$ ps -eLF | grep beam.smp | wc -l
1200

CPU usage is very low, so all these threads must be sleeping. However, I find it very weird for a single process to have that many threads.

Is this normal or is it a sign of misconfiguration? Or could it be a bug? It doesn't seem to cause any immediate issues but could it lead to problems at a later time?

Normally I would provide the contents of a configuration file but in this case I can't find the configuration file that is responsible for beam.smp.

UPDATE:

If I do:

yum upgrade

I see:

Skip Packages With Broken Dependencies When 

How do I fix this?

Older post:

I found the file erl_crash.dump here:

/var/lib/rabbitmq/erl_crash.dump 

If I do this:

head -n 50 /var/lib/rabbitmq/erl_crash.dump

I get:

=erl_crash_dump:0.5
Tue Mar  1 20:00:36 2022
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<185,
System version: Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]
Compiled: Fri Sep  6 18:47:55 2019
Taints: crypto
Atoms: 13923
Calling Thread: scheduler:1
=scheduler:1
Scheduler Sleep Info Flags: 
Scheduler Sleep Info Aux Work: THR_PRGR_LATER_OP
Current Port: 
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | HALFTIME_OUT_OF_WORK | NONEMPTY | EXEC
Current Process: <0.0.0>
Current Process State: Running
Current Process Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING
Current Process Program counter: 0x0000ffff5cd5d020 (init:printable_list/1 + 24)
Current Process CP: 0x0000ffff5cd5dcc8 (init:boot_loop/2 + 1944)
Current Process Limited Stack Trace:
0x0000ffff5a93b9f0:SReturn addr 0xACD16DC0 (<terminate process normally>)

I'm not sure how to read this. Is failed_to_start_child the error? If yes, what does it mean? How do I fix it?

I've got a new EC2 instance, and a fresh install of Redhat8, with everything just installed with yum or dnf but I keep getting errors on RabbitMQ, which tries to start but fails instantly. If I do:

tail -n 500 /var/log/messages

Then I see this message, repeating over and over again:

Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: {"Kernel pid terminated",application_controller,"{application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,84,120,32,178,183,28,70,220,34,20,13,145,69,180,207,145,18,30,76,255,205,110,198,166,173,68,94,39,2,201,5,116,139,211,47,32,125,29,33,161,98,58,78,127,162,252,146,192,110,153,88,149,58,190,136,74,66,191,238,168,79,190,2,120,197,45,101,48,216,168,33,102,142,44,160,25,38,229,27,132,117,91,23,171,84,244,84,230,129,67,131,57,152,36,238,207,222,137,164,79,90,128,114,75,111,48,205,252,239,253,68,193,32,61,190,149,117,145,224,144,191,59,109,247,145,102,240>>,[<<167,114,26,126,33,160,146,70,65,253,176,24,5,41,155,193>>,<<0,0,0,1>>],32],[]},{credentials_obfuscation_pbe,xor_sum,7,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,160}]},{credentials_obfuscation_pbe,blocks,10,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,152}]},{credentials_obfuscation_pbe,make_key,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,94}]},{credentials_obfuscation_pbe,encrypt,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,76}]},{credentials_obfuscation_svc,check,3,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,163}]},{credentials_obfuscation_svc,init_state,0,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,133}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,374}]}]}}},{credentials_obfuscation_app,start,[normal,[]]}}}"}
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,8
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: #015
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Crash dump is being written to: erl_crash.dump...done
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: Failed to start RabbitMQ broker.

I'm struggling to think what the problem is. I believe I've opened up all the ports necessary.

Crash dump is being written to: erl_crash.dump...done

I have read this:

https://stackoverflow.com/questions/63053451/distribution-failed-failed-to-start-child-rabbitmq-docker-compose-issue

I have opened up port 4369 on the security group, but that did not help.

If I run:

epmd -names

I get:

epmd: up and running on port 4369 with data

So that is working, but RabbitMQ is not.

If I do this:

rabbitmqctl start_app 

I get:

Starting node rabbit@ip-172-31-56-22 ...
Error: unable to perform an operation on node 'rabbit@ip-172-31-56-22'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@ip-172-31-56-22
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@ip-172-31-56-22']

rabbit@ip-172-31-56-22:
  * connected to epmd (port 4369) on ip-172-31-56-22
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on ip-172-31-56-22
  * suggestion: start the node

Current node details:
 * node name: 'rabbitmqcli-871-rabbit@ip-12-131-56-22'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Gug6Jfak+f3/k+ww6vhA==

I have the impression that something failed to install. Whenever I use, I see something about RabbitMQ, even when my command has nothing to do with RabbitMQ. For instance, I do this:

sudo yum install screen

And part of the output is:

rabbitmq_rabbitmq-server                                                                    5.2 kB/s | 1.8 kB     00:00    

rabbitmq_rabbitmq-server-source                                                                   3.4 kB/s | 951  B     00:00 

I see those 2 lines whenever I use yum install and it makes me uneasy that RabbitMQ is mentioned with every use of yum install as if there is some weird failed cache issue.

I also see this:

 [/usr/lib/tmpfiles.d/rabbitmq-server.conf:1] Line references path below legacy directory /var/run/, updating /var/run/rabbitmq → /run/rabbitmq; please update the tmpfiles.d/ drop-in file accordingly.

I'm testing a rabbitmq instance upgrate scenario that looks good but throws an Erlang error when I try to stop the new version instance.

"Old version" of RabbitMQ RMQ 3.8.5 erlang 23.0.2

"New version" of RabbitMQ RMQ 3.9.10 erlang 23.2.7

The new version start well. I can send/receive messages but when i stopped the service some error comes into logs:

021-12-09 13:23:53.489319+01:00 [info] <0.4848.0> Successfully stopped RabbitMQ and its dependencies
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0> Halting Erlang VM with the following applications:
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     eldap
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     amqp10_common
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     seshat
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     observer_cli
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     stdout_formatter
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     gen_batch_server
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     aten
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     ranch
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     inets
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     systemd
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     enough
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     cuttlefish
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     credentials_obfuscation
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     recon
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     jsx
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     xmerl
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     tools
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     syntax_tools
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     ssl
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     public_key
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     asn1
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     crypto
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     compiler
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     sasl
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     stdlib
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>     kernel
2021-12-09 13:23:53.489568+01:00 [error] <0.4848.0>

im little confused. RabbitMQ seems to stopped well but i dont kown if the error is relevant and need to be resolved or if i can ignore this type of error.

Is anyone have already be confronted to that ?

Thank you

I need to install Erlang 24.1 on Aamazon Linux 2. I am using package cloud script and repos for this but it fails with following message.

(venv) [CORP\shiv.d@a-1njrk968l4brs yum.repos.d]$ sudo yum install erlang
Loaded plugins: amzn_workspaces_filter_updates, halt_os_update_check, priorities, update-motd
rabbitmq_erlang/x86_64/signature                                                                                                                                      |  833 B  00:00:00     
rabbitmq_erlang/x86_64/signature                                                                                                                                      | 1.8 kB  00:00:00 !!! 
Not using downloaded rabbitmq_erlang/repomd.xml because it is older than what we have:
  Current   : Fri Oct  1 10:15:14 2021
  Downloaded: Tue Sep 14 08:38:56 2021
rabbitmq_erlang-source/signature                                                                                                                                      |  819 B  00:00:00     
rabbitmq_erlang-source/signature                                                                                                                                      |  951 B  00:00:00 !!! 
rabbitmq_rabbitmq-server-source/signature                                                                                                                             |  836 B  00:00:00     
rabbitmq_rabbitmq-server-source/signature                                                                                                                             | 1.0 kB  00:00:00 !!! 
356 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package erlang.x86_64 0:24.1-1.el8 will be installed
--> Processing Dependency: libz.so.1(ZLIB_1.2.7.1)(64bit) for package: erlang-24.1-1.el8.x86_64
--> Finished Dependency Resolution
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
(venv) [CORP\shiv.d@a-1njrk968l4brs yum.repos.d]$

It complains about zlib 1.2.7.1 which is already installed.

We are running rabbit v3.8.3-1.el7, erlang v23.3.3.el7, kernel 3.10.0-1062.12.1.el7.x86_64, release Centos 7.7

I have three nodes that I would like in disc mode, cdvlhbqr23[1-3]

However I'm running into an issue after attempting to enable TLS on erlang.

[ cdvlhbqr231:rabbitmq ]
10.128.3.231 :: root -> rabbitmqctl status
Error: unable to perform an operation on node 'rabbit@cdvlhbqr231'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@cdvlhbqr231
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: [rabbit@cdvlhbqr231]

rabbit@cdvlhbqr231:
  * connected to epmd (port 4369) on cdvlhbqr231
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: check if the Erlang cookie identical for all server nodes and CLI tools
  * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
  * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
   * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more


Current node details:
 * node name: 'rabbitmqcli-23412-rabbit@cdvlhbqr231'
 * effective user's home directory: /var/lib/rabbitmq/
 * Erlang cookie hash: MudCW7tn3FA5sTmC1FlR0g==

I've double checked the cookie file and it's identical across all nodes. All of the hostnames are correct and consistent across the nodes. So I assume this has to be directly the result of ssl / tls

Here's what the node config looks like:

[ cdvlhbqr231:rabbitmq ]
10.128.3.231 :: root -> cat /etc/rabbitmq/rabbitmq.config
[
    {rabbit,
        [
            {vm_memory_high_watermark, 0.4},
            {vm_memory_high_watermark_paging_ratio, 0.5},
            {memory_alarms, true},
            {disk_free_limit, 41686528},
            {cluster_partition_handling, autoheal},
            {tcp_listen_options,
                [binary,
                    {packet, raw},
                    {reuseaddr, true},
                    {backlog, 128},
                    {nodelay, true},
                    {exit_on_close, false},
                    {keepalive, true}
                ]
            },
            {cluster_nodes, {['rabbit@cdvlhbqr231', 'rabbit@cdvlhbqr232', 'rabbit@cdvlhbqr233'], disc}},
            {loopback_users, []},
            {tcp_listeners, [{"0.0.0.0",5672}]},
            {ssl_listeners, [{"0.0.0.0",5671}]},
            {ssl_options, [
                {cacertfile,"/etc/pki/tls/certs/ca-bundle.crt"},
                {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                {verify,verify_peer},
                {versions, ['tlsv1.2']},
                {fail_if_no_peer_cert,false}
            ]}

        ]
    },
    {rabbitmq_management,
        [{
            listener, [
                {port, 15672},
                {ip, "0.0.0.0"},                
                {ssl, true},
                {ssl_opts, [
                    {cacertfile,"/etc/pki/tls/certs/ca-bundle.crt"},
                    {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                    {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                    {verify,verify_peer},
                    {versions, ['tlsv1.2']}
                ]}
            ]}
        ]
    }
].

The private key is generated on the host and signed by an intermediate CA whose pub key is in the systems extracted cert bundle. We generate an "/etc/rabbitmq/ssl/allfile.pem" which is a bundle of the servers private key and signed cert.

The ssl environment for erlang is defined as the following:

[ cdvlhbqr231:rabbitmq ]
10.128.3.231 :: root -> cat rabbitmq-env.conf
# Obtaining of an Erlang ssl library path
export HOME=/var/lib/rabbitmq/
ERL_SSL_PATH=/usr/lib64/erlang/lib/ssl-9.6.2/ebin

# Add SSL-related environment vars for rabbitmq-server and rabbitmqctl
SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile '/etc/rabbitmq/ssl/allfile.pem' \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

# CLI
CTL_ERL_ARGS="-pa $ERL_SSL_PATH \
  -proto_dist inet_tls \
  -ssl_dist_opt server_certfile /etc/rabbitmq/ssl/allfile.pem \
  -ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"

I'm not entirely clear what's causing the issue. I thought I had followed the documentation to a T. Can anyone help me review this and see if there is anything obvious I'm missing, or any suggestions on how to trace down this problem?