user5416's questions

RAID1 array created with mdadm; array was rebuilding when there was a power failure. On coming back up, the array seems to be dead.

# cat /proc/mdstat
Personalities : [raid1]
md0 : inactive sdb1[2](S) sda1[3](S)
15627751424 blocks super 1.2

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble /dev/md1 /dev/sda1 /dev/sdb1
mdadm: /dev/md1 assembled from 0 drives and 1 rebuilding - not enough to start the array.

I don't know why it says 0 drives - I expected the 1 rebuilding but the other drive should have been fine. I really don't want to accidentally do anything destructive - what should I do to get the array back together to a state I can run fsck on it?

/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x3
     Array UUID : c4ea2289:c63bc8ce:e6fe5806:5bebe020
           Name : ******:0  (local to host ******)
  Creation Time : Thu Aug 20 20:48:53 2020
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 15627718656 (7451.88 GiB 8001.39 GB)
     Array Size : 7813859328 (7451.88 GiB 8001.39 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
Recovery Offset : 7331083264 sectors
   Unused Space : before=264112 sectors, after=0 sectors
          State : clean
    Device UUID : ae984a41:f3e421f4:f10e1fac:d7955178

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Oct 31 17:13:53 2020
  Bad Block Log : 512 entries available at offset 40 sectors
       Checksum : 4be47968 - correct
         Events : 1079809


   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c4ea2289:c63bc8ce:e6fe5806:5bebe020
           Name : ******:0  (local to host ******)
  Creation Time : Thu Aug 20 20:48:53 2020
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 15627784192 (7451.91 GiB 8001.43 GB)
     Array Size : 7813859328 (7451.88 GiB 8001.39 GB)
  Used Dev Size : 15627718656 (7451.88 GiB 8001.39 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=65536 sectors
          State : active
    Device UUID : 0abf70e2:7e0e43ca:a22548a2:ef87e9c0

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Oct 31 17:13:31 2020
  Bad Block Log : 512 entries available at offset 40 sectors
       Checksum : 4cf65df4 - correct
         Events : 1079799


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Looking at the drives with xxd, both do still seem to contain data. Is this something I could reasonable manage myself or is it time for a data recovery company?

Edit: Wazoox's answer worked.

#  mdadm --assemble --scan --force
mdadm: forcing event count in /dev/sda1(1) from 1079799 upto 1079809
mdadm: /dev/md0 has been started with 1 drive (out of 2) and 1 rebuilding.
# cat /proc/mdstat
Personalities : [raid1]
md0 : active (auto-read-only) raid1 sdb1[3] sda1[2]
      7813859328 blocks super 1.2 [2/1] [_U]
      bitmap: 59/59 pages [236KB], 65536KB chunk

I am currently setting up a RabbitMQ cluster where a requirement is that all communication within the cluster be encrypted.

I followed the guide at https://www.rabbitmq.com/clustering-ssl.html - created a file with the cert and key, and passed the SSL arguments to rabbitmq-server as environment variables:

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='-pa /usr/lib/erlang/lib/ssl-7.1/ebin -proto_dist inet_tls -ssl_dist_opt server_certfile /etc/ssl/certs/rabbit.pem'
RABBITMQ_CTL_ERL_ARGS='-pa /usr/lib/erlang/lib/ssl-7.1/ebin -proto_dist inet_tls -ssl_dist_opt server_certfile /etc/ssl/certs/rabbit.pem'

The server starts up fine and listens with TLS enabled, but I can not issue any commands to it via rabbitmqctl:

# rabbitmqctl status
Status of node rabbit@rabbit01 ...
Error: unable to connect to node rabbit@rabbit01: nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@rabbit01]

rabbit@rabbit01:
  * connected to epmd (port 4369) on rabbit01
  * epmd reports node 'rabbit' running on port 47965
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?
  * suggestion: is the Erlang distribution using TLS?

On the server's side, I don't get anything much in the logs other than the thoroughly unhelpful:

=ERROR REPORT==== 30-Dec-2015::13:08:58 ===
SSL: hello: tls_handshake.erl:167:Fatal error: insufficient security

A bit of searching around has indicated this can be due to a cipher suite mismatch, but my understanding here was that they would both be using the same SSL library.

Erlang dues have SSL support enabled as checked on https://www.rabbitmq.com/troubleshooting-ssl.html, and I can make a secure connection with the certificate with openssl's s_server and s_client.

I'm not sure what the next step would be to troubleshooting this, and couldn't find any known issue anywhere around clustering - unfortunately, most of RabbitMQ's documentation regarding TLS is for the AMQP protocol and not for the internal clustering.

I have tested using openSSL to connect to the running RabbitMQ instance as well and get a very similar error:

# openssl s_client -connect localhost:47965
CONNECTED(00000003)
140004605863584:error:1407742F:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert insufficient security:s23_clnt.c:770:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 295 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

 openssl s_client -connect localhost:47965 -tls1_2
CONNECTED(00000003)
140400037775008:error:1409442F:SSL routines:SSL3_READ_BYTES:tlsv1 alert insufficient security:s3_pkt.c:1262:SSL alert number 71
140400037775008:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:598:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1451944018
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

Postgres appears to be losing connections from clients. The clients get an error, normally that the connection was closed by the server, while at the same time the server gets a log entry. Example:

Nov 14 17:54:15 dev-db0 postgres[29523]: [219-1] 2014-11-14 17:54:15 GMT [29523]: [2-1] user=******,db=******,remote=10.0.13.1(51491) LOG:  could not receive data from client: Connection timed out

As far as I can see, this primarily happens for Windows clients, and is happening across several different database servers, physical and virtual. I first became aware of this issue when a user attempting to access postgres over a VPN was having the problem, but checking the logs shows it happens with local clients too, although the VPN seems to make it happen more often.

I have tried all the keepalive settings in postgresql.conf (tcp_keepalives_idle, tcp_keepalives_interval and tcp_keepalives_count) as well as tunnelling the database connection over SSH, but none seem to make any difference.

While it says timeout on the server side, on the client side it normally says connection reset by peer or similar. The most reliable way to reproduce it is to run large/long queries; pgadmin and tomcat seem to be more susceptible to it than psql.