On a fresh install of Redhat8, on a new EC2 instance, RabbitMQ fails, failed_to_start_child

UPDATE:

If I do:

yum upgrade

I see:

Skip Packages With Broken Dependencies When 

How do I fix this?

Older post:

I found the file erl_crash.dump here:

/var/lib/rabbitmq/erl_crash.dump 

If I do this:

head -n 50 /var/lib/rabbitmq/erl_crash.dump

I get:

=erl_crash_dump:0.5
Tue Mar  1 20:00:36 2022
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<185,
System version: Erlang/OTP 22 [erts-10.4.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]
Compiled: Fri Sep  6 18:47:55 2019
Taints: crypto
Atoms: 13923
Calling Thread: scheduler:1
=scheduler:1
Scheduler Sleep Info Flags: 
Scheduler Sleep Info Aux Work: THR_PRGR_LATER_OP
Current Port: 
Run Queue Max Length: 0
Run Queue High Length: 0
Run Queue Normal Length: 0
Run Queue Low Length: 0
Run Queue Port Length: 0
Run Queue Flags: OUT_OF_WORK | HALFTIME_OUT_OF_WORK | NONEMPTY | EXEC
Current Process: <0.0.0>
Current Process State: Running
Current Process Internal State: ACT_PRIO_NORMAL | USR_PRIO_NORMAL | PRQ_PRIO_NORMAL | ACTIVE | RUNNING
Current Process Program counter: 0x0000ffff5cd5d020 (init:printable_list/1 + 24)
Current Process CP: 0x0000ffff5cd5dcc8 (init:boot_loop/2 + 1944)
Current Process Limited Stack Trace:
0x0000ffff5a93b9f0:SReturn addr 0xACD16DC0 (<terminate process normally>)

I'm not sure how to read this. Is failed_to_start_child the error? If yes, what does it mean? How do I fix it?

I've got a new EC2 instance, and a fresh install of Redhat8, with everything just installed with yum or dnf but I keep getting errors on RabbitMQ, which tries to start but fails instantly. If I do:

tail -n 500 /var/log/messages

Then I see this message, repeating over and over again:

Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: {"Kernel pid terminated",application_controller,"{application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,84,120,32,178,183,28,70,220,34,20,13,145,69,180,207,145,18,30,76,255,205,110,198,166,173,68,94,39,2,201,5,116,139,211,47,32,125,29,33,161,98,58,78,127,162,252,146,192,110,153,88,149,58,190,136,74,66,191,238,168,79,190,2,120,197,45,101,48,216,168,33,102,142,44,160,25,38,229,27,132,117,91,23,171,84,244,84,230,129,67,131,57,152,36,238,207,222,137,164,79,90,128,114,75,111,48,205,252,239,253,68,193,32,61,190,149,117,145,224,144,191,59,109,247,145,102,240>>,[<<167,114,26,126,33,160,146,70,65,253,176,24,5,41,155,193>>,<<0,0,0,1>>],32],[]},{credentials_obfuscation_pbe,xor_sum,7,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,160}]},{credentials_obfuscation_pbe,blocks,10,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,152}]},{credentials_obfuscation_pbe,make_key,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,94}]},{credentials_obfuscation_pbe,encrypt,5,[{file,\"src/credentials_obfuscation_pbe.erl\"},{line,76}]},{credentials_obfuscation_svc,check,3,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,163}]},{credentials_obfuscation_svc,init_state,0,[{file,\"src/credentials_obfuscation_svc.erl\"},{line,133}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,374}]}]}}},{credentials_obfuscation_app,start,[normal,[]]}}}"}
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Kernel pid terminated (application_controller) ({application_start_failure,credentials_obfuscation,{{shutdown,{failed_to_start_child,credentials_obfuscaton_svc,{undef,[{crypto,macN,[hmac,sha256,<<80,8
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: #015
Mar  1 18:23:47 ip-172-31-56-22 rabbitmq-server[350248]: Crash dump is being written to: erl_crash.dump...done
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
Mar  1 18:23:47 ip-172-31-56-22 systemd[1]: Failed to start RabbitMQ broker.

I'm struggling to think what the problem is. I believe I've opened up all the ports necessary.

Crash dump is being written to: erl_crash.dump...done

I have read this:

https://stackoverflow.com/questions/63053451/distribution-failed-failed-to-start-child-rabbitmq-docker-compose-issue

I have opened up port 4369 on the security group, but that did not help.

If I run:

epmd -names

I get:

epmd: up and running on port 4369 with data

So that is working, but RabbitMQ is not.

If I do this:

rabbitmqctl start_app 

I get:

Starting node rabbit@ip-172-31-56-22 ...
Error: unable to perform an operation on node 'rabbit@ip-172-31-56-22'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@ip-172-31-56-22
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@ip-172-31-56-22']

rabbit@ip-172-31-56-22:
  * connected to epmd (port 4369) on ip-172-31-56-22
  * epmd reports: node 'rabbit' not running at all
                  no other nodes on ip-172-31-56-22
  * suggestion: start the node

Current node details:
 * node name: 'rabbitmqcli-871-rabbit@ip-12-131-56-22'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: Gug6Jfak+f3/k+ww6vhA==

I have the impression that something failed to install. Whenever I use, I see something about RabbitMQ, even when my command has nothing to do with RabbitMQ. For instance, I do this:

sudo yum install screen

And part of the output is:

rabbitmq_rabbitmq-server                                                                    5.2 kB/s | 1.8 kB     00:00    

rabbitmq_rabbitmq-server-source                                                                   3.4 kB/s | 951  B     00:00 

I see those 2 lines whenever I use yum install and it makes me uneasy that RabbitMQ is mentioned with every use of yum install as if there is some weird failed cache issue.

I also see this:

 [/usr/lib/tmpfiles.d/rabbitmq-server.conf:1] Line references path below legacy directory /var/run/, updating /var/run/rabbitmq → /run/rabbitmq; please update the tmpfiles.d/ drop-in file accordingly.