Questions[galera](server)

I am trying to setup a mariadb-galera cluster through the bitnami helm chart in my kubernetes cluster (1 master, 3 nodes). I have modified the myvalues.yaml to include an existingClaim: dbstorage and the storageClass: "nfs-storage". The image repository is 10.5.9-debian-10-r52, I added a root password and outcommented the accessModes: as well as the size as that was defined in the existing persistence volume claim. I did not define anything in the section db: and left that as the defaults. I also did not define anything under the galera.mariabackup section like password and left the defaults.

As soon as I run the helm chart with helm install helm install mariadb-galera-cluster -f mariadb-galera.values.yaml bitnami/mariadb-galera --namespace database and describe the pod I get the error message

Readiness probe failed: mysqladmin: connect to server at 'localhost' failed error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)' 
Check that mysqld is running and that the socket: '/opt/bitnami/mariadb/tmp/mysql.sock' exists!

When checking the container for that mysql.sock it is true, that it is not available in that location.

I am using a NFS provisioner to provision the persistent storage which works fine. On my nfs server I can see the directory being created and data being stored in it. It is a NFS3 directory that is used by the container.

When I access the container and try to run the scrips "run.sh" or "entrypoint.sh" in the folder /opt/bitnami/scripts/mariadb-galera I am getting an error The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable or does not exist. Configurations based on environment variables will not be applied for this file. but the file is right in the folder where it should be. All components, like the stateful set are created and started properly as I can tell just the container, in my case it was obviously called mariadb-galera-cluster-0, is not finishing starting up because of the socket it can't find.

Version of Helm:

version.BuildInfo{Version:"v3.5.4", GitCommit:"1b5edb69df3d3a08df77c9902dc17af864ff05d1", GitTreeState:"clean", GoVersion:"go1.15.11"}

Version of Kubernetes

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

version of the values.yaml file

## Please, note that this will override the image parameters, including dependencies, configured to use the global v$
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
# global:
#   imageRegistry: myRegistryName
#   imagePullSecrets:
#     - myRegistryKeySecretName
#   storageClass: myStorageClass

## Bitnami MariaDB Galera image
## ref: https://hub.docker.com/r/bitnami/mariadb-galera/tags/
##
image:
  registry: docker.io
  repository: bitnami/mariadb-galera
  tag: 10.5.9-debian-10-r52
  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistryKeySecretName

  ## Set to true if you would like to see extra information on logs
  ##
  debug: false

## String to partially override common.names.fullname template (will maintain the release name)
##
# nameOverride:

## String to fully override common.names.fullname template
##
# fullnameOverride:

## Use an alternate scheduler, e.g. "stork".
## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
##
# schedulerName:

## Specifies the Kubernetes Cluster's Domain Name.
##
clusterDomain: cluster.local

## StatefulSet controller supports relax its ordering guarantees while preserving its uniqueness and identity guaran$
## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
##
podManagementPolicy: OrderedReady

## Deployment pod host aliases
## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
hostAliases: []

## MariaDB Gallera K8s svc properties
##
service:
  ## Kubernetes service type and port number
  ##
  type: ClusterIP
  port: 3306
  # clusterIP: None

  ## Specify the nodePort value for the LoadBalancer and NodePort service types.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
  ##
  # nodePort: 30001

  ## Specify the externalIP value ClusterIP service type.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips
  ##
  # externalIPs: []

  ## Set the LoadBalancer service type to internal only.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
  ##
  # loadBalancerIP:

  ## Load Balancer sources
  ## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-$
  ##
  # loadBalancerSourceRanges:
  # - 10.10.10.0/24

  ## Provide any additional annotations which may be required. This can be used to
  ## set the LoadBalancer service type to internal only.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
  ##
  annotations: {}

  ## Headless service properties
  ##
  headless:
    ## Additional annotations for headless service.
    ## Can be useful in case peer-finder is used in a sidecar,
    ## e.g.: service.alpha.kubernetes.io/tolerate-unready-endpoints="true"
    ##
    annotations: {}

## Pods Service Account
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
##
serviceAccount:
  ## Specifies whether a ServiceAccount should be created
  ##
  create: false
  ## The name of the ServiceAccount to use.
  ## If not set and create is true, a name is generated using the common.names.fullname template
  ##
  name: ""

## An array to add extra environment variables
## For example:
## extraEnvVars:
##  - name: TZ
##    value: "Europe/Paris"
##
extraEnvVars:

## ConfigMap with extra env vars:
##
extraEnvVarsCM:

## Secret with extra env vars:
##
extraEnvVarsSecret:

## Role Based Access
## Ref: https://kubernetes.io/docs/admin/authorization/rbac/
##
rbac:
  ## Specifies whether RBAC rules should be created
  ##
  create: false

## Pod Security Context
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
##
securityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001

## Database credentials for root (admin) user
##
rootUser:
  ## MariaDB admin user
  ##
  user: root
  ## MariaDB admin password
  ## Password is ignored if existingSecret is specified.
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#setting-the-root-password-on-first-run
  ##
  password: "ObviouslyIChangedThis"
  ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
  ## If it is not force, a random password will be generated.
  ##
  forcePassword: false

## Use existing secret (ignores rootUser.password, db.password, and galera.mariabackup.password)
##
# existingSecret:

## Custom db configuration
##
db:
  ## MariaDB username and password
  ## Password is ignored if existingSecret is specified.
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#creating-a-database-user-on-first-run
  ##
  user: ""
  password: ""
  ## Database to create
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#creating-a-database-on-first-run
  ##
  name: my_database
  ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
  ## If it is not force, a random password will be generated.
  ##
  forcePassword: false

## Galera configuration
##
galera:
  ## Galera cluster name
  ##
  name: galera

  ## Bootstraping options
  ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#bootstraping
  ##
  bootstrap:
    ## Node to bootstrap from, you will need to change this parameter in case you want to bootstrap from other node
    ##
    bootstrapFromNode:
    ## Force safe_to_bootstrap in grastate.date file.
    ## This will set safe_to_bootstrap=1 in the node indicated by bootstrapFromNode.
    ##
    forceSafeToBootstrap: false

  ## Credentials to perform backups
  ##
  mariabackup:
    ## MariaBackup username and password
    ## Password is ignored if existingSecret is specified.
    ## ref: https://github.com/bitnami/bitnami-docker-mariadb-galera#setting-up-a-multi-master-cluster
    ##
    user: mariabackup
    password: ""
    ## Option to force users to specify a password. That is required for 'helm upgrade' to work properly.
    ## If it is not force, a random password will be generated.
    ##
    forcePassword: false

## LDAP configuration
##
ldap:
  ## Enable LDAP support
  ##
  enabled: false
  uri: ""
  base: ""
  binddn: ""
  bindpw: ""
  bslookup:
  filter:
  map:
  nss_initgroups_ignoreusers: root,nslcd
  scope:
  tls_reqcert:

## TLS configuration
##
tls:
  ## Enable TLS
  ##
  enabled: false
  ## Name of the secret that contains the certificates
  ##
  # certificatesSecret:
  ## Certificate filename
  ##
  # certFilename:
  ## Certificate Key filename
  ##
  # certKeyFilename:
  ## CA Certificate filename
  ##
  # certCAFilename:

## Configure MariaDB with a custom my.cnf file
## ref: https://mysql.com/kb/en/mysql/configuring-mysql-with-mycnf/#example-of-configuration-file
## Alternatively, you can put your my.cnf under the files/ directory
mariadbConfiguration: |-
  [client]
  port=3306
  socket=/opt/bitnami/mariadb/tmp/mysql.sock
  plugin_dir=/opt/bitnami/mariadb/plugin
  [mysqld]
  default_storage_engine=InnoDB
  basedir=/opt/bitnami/mariadb
  datadir=/bitnami/mariadb/data
  plugin_dir=/opt/bitnami/mariadb/plugin
  tmpdir=/opt/bitnami/mariadb/tmp
  socket=/opt/bitnami/mariadb/tmp/mysql.sock
  pid_file=/opt/bitnami/mariadb/tmp/mysqld.pid
  bind_address=0.0.0.0
  ## Character set
  ##
  collation_server=utf8_unicode_ci
  init_connect='SET NAMES utf8'
  character_set_server=utf8
  ## MyISAM
  ##
  key_buffer_size=32M
  myisam_recover_options=FORCE,BACKUP
  ## Safety
  ##
  skip_host_cache
  skip_name_resolve
  max_allowed_packet=16M
  max_connect_errors=1000000
  sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTI$
  sysdate_is_now=1
  ## Binary Logging
  ##
  log_bin=mysql-bin
  expire_logs_days=14
  # Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
  sync_binlog=0
  # Required for Galera
  binlog_format=row
  ## Caches and Limits
  ##
  tmp_table_size=32M
  max_heap_table_size=32M
  # Re-enabling as now works with Maria 10.1.2
  query_cache_type=1
  query_cache_limit=4M
  query_cache_size=256M
  max_connections=500
  thread_cache_size=50
  open_files_limit=65535
  table_definition_cache=4096
  table_open_cache=4096
  ## InnoDB
  ##
  innodb=FORCE
  innodb_strict_mode=1
  # Mandatory per https://github.com/codership/documentation/issues/25
  innodb_autoinc_lock_mode=2
  # Per https://www.percona.com/blog/2006/08/04/innodb-double-write/
  innodb_doublewrite=1
  innodb_flush_method=O_DIRECT
  innodb_log_files_in_group=2
  innodb_log_file_size=128M
  innodb_flush_log_at_trx_commit=1
  innodb_file_per_table=1
  # 80% Memory is default reco.
  # Need to re-evaluate when DB size grows
  innodb_buffer_pool_size=2G
  innodb_file_format=Barracuda
  ## Logging
  ##
  log_error=/opt/bitnami/mariadb/logs/mysqld.log
  slow_query_log_file=/opt/bitnami/mariadb/logs/mysqld.log
  log_queries_not_using_indexes=1
  slow_query_log=1
  ## SSL
  ## Use extraVolumes and extraVolumeMounts to mount /certs filesystem
  # ssl_ca=/certs/ca.pem
  # ssl_cert=/certs/server-cert.pem
  # ssl_key=/certs/server-key.pem
  [galera]
  wsrep_on=ON
  wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
  wsrep_sst_method=mariabackup
  wsrep_slave_threads=4
  wsrep_cluster_address=gcomm://
  wsrep_cluster_name=galera
  wsrep_sst_auth="root:"
  # Enabled for performance per https://mariadb.com/kb/en/innodb-system-variables/#innodb_flush_log_at_trx_commit
  innodb_flush_log_at_trx_commit=2
  # MYISAM REPLICATION SUPPORT #
  wsrep_replicate_myisam=ON
  [mariadb]
  plugin_load_add=auth_pam
  ## Data-at-Rest Encryption
  ## Use extraVolumes and extraVolumeMounts to mount /encryption filesystem
  # plugin_load_add=file_key_management
  # file_key_management_filename=/encryption/keyfile.enc
  # file_key_management_filekey=FILE:/encryption/keyfile.key
  # file_key_management_encryption_algorithm=AES_CTR
  # encrypt_binlog=ON
  # encrypt_tmp_files=ON
  ## InnoDB/XtraDB Encryption
  # innodb_encrypt_tables=ON
  # innodb_encrypt_temporary_tables=ON
  # innodb_encrypt_log=ON
  # innodb_encryption_threads=4
  # innodb_encryption_rotate_key_age=1
  ## Aria Encryption
  # aria_encrypt_tables=ON
  # encrypt_tmp_disk_tables=ON
## ConfigMap with MariaDB configuration
## NOTE: This will override mariadbConfiguration
##
# configurationConfigMap:

## Specify dictionary of scripts to be run at first boot
## Alternatively, you can put your scripts under the files/docker-entrypoint-initdb.d directory
##
# initdbScripts:
#   my_init_script.sh: |
#      #!/bin/sh
#      echo "Do something."

## ConfigMap with scripts to be run at first boot
## Note: This will override initdbScripts
##
# initdbScriptsConfigMap:

## MariaDB additional command line flags
## Can be used to specify command line flags, for example:
##
## extraFlags: "--max-connect-errors=1000 --max_connections=155"
##

## Desired number of cluster nodes
##
replicaCount: 3

## updateStrategy for MariaDB Master StatefulSet
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
##
updateStrategy:
  type: RollingUpdate

## Additional labels for MariaDB Galera pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
##
podLabels: {}

## Additional annotations for MariaDB Galera pods
## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
##
podAnnotations: {}

## Pod affinity preset
## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
## Allowed values: soft, hard
##
podAffinityPreset: ""

## Pod anti-affinity preset
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
## Allowed values: soft, hard
##
podAntiAffinityPreset: soft

## Node affinity preset
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
## Allowed values: soft, hard
##
nodeAffinityPreset:
  ## Node affinity type
  ## Node affinity type
  ## Allowed values: soft, hard
  ##
  type: ""
  ## Node label key to match
  ## E.g.
  ## key: "kubernetes.io/e2e-az-name"
  ##
  key: ""
  ## Node label values to match
  ## E.g.
  ## values:
  ##   - e2e-az1
  ##   - e2e-az2
  ##
  values: []

## Affinity for pod assignment. Evaluated as a template.
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
## Note: podAffinityPreset, podAntiAffinityPreset, and nodeAffinityPreset will be ignored when it's set
##
affinity: {}

## Node labels for pod assignment. Evaluated as a template.
## ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}

## Tolerations for pod assignment. Evaluated as a template.
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []

## Enable persistence using Persistent Volume Claims
## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  ## If true, use a Persistent Volume Claim, If false, use emptyDir
  ##
  enabled: true
  # Enable persistence using an existing PVC
  existingClaim: dbstorage
  # Subdirectory of the volume to mount
  # subPath:
  mountPath: /bitnami/mariadb
  ## selector can be used to match an existing PersistentVolume
  ## selector:
  ##   matchLabels:
  ##     app: my-app
  ##
  selector: {}
  ## Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "nfs-storage"
  ## Persistent Volume Claim annotations
  ##
  annotations:
  ## Persistent Volume Access Mode
  ##
  #accessModes:
  #  - ReadWriteOnce
  ## Persistent Volume size
  ##
  #size: 8Gi

## Priority Class Name
#
# priorityClassName: 'priorityClass'

## Additional init containers
##
extraInitContainers: []
# - name: do-something
#   image: bitnami/minideb
#   command: ['do', 'something']

## Additional containers
##
extraContainers: []

## extraVolumes and extraVolumeMounts allows you to mount other volumes
## Example Use Cases:
##  mount certificates to enable data-in-transit encryption
##  mount keys for data-at-rest encryption using file plugin
# extraVolumes:
# - name: mariadb-certs
#   secret:
#     defaultMode: 288
#     secretName: mariadb-certs
# - name: mariadb-encryption
#   secret:
#     defaultMode: 288
#     secretName: mariadb-encryption
# extraVolumeMounts:
# - name: mariadb-certs
#   mountPath: /certs
#   readOnly: true
# - name: mariadb-encryption
#   mountPath: /encryption
#   readOnly: true

## MariaDB Galera containers' resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  limits: {}
  #   cpu: 0.5
  #   memory: 256Mi
  requests: {}
  #   cpu: 0.5
  #   memory: 256Mi

## MariaDB Galera containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
##
livenessProbe:
  enabled: true
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
readinessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
startupProbe:
  enabled: false
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  ## Let's wait 600 seconds by default, it should give enough time in any cluster for mysql to init
  ##
  failureThreshold: 48

## Pod disruption budget configuration
##
podDisruptionBudget:
  ## Specifies whether a Pod disruption budget should be created
  ##
  create: false
  minAvailable: 1
  # maxUnavailable: 1

## Prometheus exporter configuration
##
metrics:
  enabled: false
  ## Bitnami MySQL Prometheus exporter image
  ## ref: https://hub.docker.com/r/bitnami/mysqld-exporter/tags/
  ##
  image:
    registry: docker.io
    repository: bitnami/mysqld-exporter
    tag: 0.12.1-debian-10-r416
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ##
    # pullSecrets:
    #   - myRegistryKeySecretName
  ## MySQL exporter additional command line flags
  ## Can be used to specify command line flags
  ## E.g.:
  ## extraFlags:
  ##   - --collect.binlog_size
  ##
  extraFlags: []
  ## MySQL Prometheus exporter containers' resource requests and limits
  ## ref: http://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources:
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
    limits: {}
    #   cpu: 0.5
    #   memory: 256Mi
    requests: {}
    #   cpu: 0.5
    #   memory: 256Mi
  ## MySQL Prometheus exporter service parameters
  ##
  service:
    type: ClusterIP
    port: 9104
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9104"

  ## Prometheus Operator ServiceMonitor configuration
  ##
  serviceMonitor:
    enabled: false
    ## Namespace in which Prometheus is running
    ##
    # namespace: monitoring

    ## Interval at which metrics should be scraped.
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
    ##
    # interval: 10s

    ## Timeout after which the scrape is ended
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
    ##
    # scrapeTimeout: 10s

    ## ServiceMonitor selector labels
    ## ref: https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator#prometheus-configuration
    ##
    selector:
      prometheus: kube-prometheus
    ## RelabelConfigs to apply to samples before scraping
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
    ## Value is evalued as a template
    ##
    relabelings: []

    ## MetricRelabelConfigs to apply to samples before ingestion
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
    ## Value is evalued as a template
    ##
    metricRelabelings: []
    #  - sourceLabels:
    #      - "__name__"
    #    targetLabel: "__name__"
    #    action: replace
    #    regex: '(.*)'
    #    replacement: 'example_prefix_$1'

  ## Prometheus Operator PrometheusRule configuration
  ##
  prometheusRules:
    enabled: false

    ## Additional labels to add to the PrometheusRule so it is picked up by the operator.
    ## If using the [Helm Chart](https://github.com/helm/charts/tree/master/stable/prometheus-operator) this is the $
    ##
    selector:
      app: prometheus-operator
      release: prometheus

    ## Rules as a map.
    ##
    rules: {}
    #  - alert: MariaDB-Down
    #    annotations:
    #      message: 'MariaDB instance {{ $labels.instance }} is down'
    #      summary: MariaDB instance is down
    #    expr: absent(up{job="mariadb-galera"} == 1)
    #    labels:
    #      severity: warning
    #      service: mariadb-galera
    #    for: 5m

EDIT adding log file from container initialization:

mariadb 13:59:39.44
mariadb 13:59:39.44 Welcome to the Bitnami mariadb-galera container
mariadb 13:59:39.45 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb-galera
mariadb 13:59:39.45 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb-galera/issues
mariadb 13:59:39.46
mariadb 13:59:39.47 INFO  ==> ** Starting MariaDB setup **
mariadb 13:59:40.30 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 13:59:40.41 DEBUG ==> Set Galera cluster address to gcomm://
mariadb 13:59:40.42 INFO  ==> Initializing mariadb database
mariadb 13:59:40.43 DEBUG ==> Ensuring expected directories/files exist
mariadb 13:59:40.46 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable or does not exist. Configurations based on environment variables will not be applied for this file.
mariadb 13:59:40.49 DEBUG ==> Cleaning data directory to ensure successfully initialization
Installing MariaDB/MySQL system tables in '/bitnami/mariadb/data' ...

Unfortunately that is it. The container is now in a "CrashLoopBackOff" state. It restarted 7 times already and will not finish. I can see on my NFS server the "data" directory is created and it contains 2 files, "aria_log_control" and "mysql-bin.index". The my.cnf looks like being owned by root but as far as I could find in the documentation the container is a non-root container so the file would not be writable. But it is definitely there. I checked that several times. If I describe the pod I get the error message mentioned above "Readiness probe failed".

END EDIT

EDIT 2 here are the details on the readiness, liveness probe.

## MariaDB Galera containers' liveness and readiness probes
## ref: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
##
livenessProbe:
  enabled: true
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
readinessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  failureThreshold: 3
startupProbe:
  enabled: false
  ## Initializing the database could take some time
  ##
  initialDelaySeconds: 120
  periodSeconds: 10
  timeoutSeconds: 1
  successThreshold: 1
  ## Let's wait 600 seconds by default, it should give enough time in any cluster for mysql to init
  ##
  failureThreshold: 48

END EDIT

EDIT 3 I have no choice but to respond to my original post. I would otherwise exceed the character limit of stackoverflow. End EDIT

I would be grateful for any help or hint you can provide to get the cluster up and running.

realshadow

So I followed your advise @WytrzymałyWiktor and disabled the readiness and liveness probes. That way all 3 instances are spun up right away. The problem is they are not communicating among each other. Logs from the So despite having the pods running they are not working and communicating among each other.

I'm setting up four servers across four locations in an attempt to create a geo-redundant MySQL Galera cluster. Two of the servers are behind NAT, and I'm currently attempting to use WireGuard to overcome this inconvenience.

Servers one and two are hosted in the cloud, and three and four are behind NAT in my home lab. I'm having three and four connect via WireGuard to BOTH one and two, so that the cluster can still operate if a cloud server goes down.

Each server has two IP addresses on every other server:

• 10.1.0.x (where x is one, two, three, or four) being routed through WireGuard on server one
• 10.2.0.x (where x is one, two, three, or four) being routed through WireGuard on server two

How would I be able to bond these two IP addresses together, so that if one WireGuard tunnel were to fail, the other one would seamlessly take over without MySQL Galera noticing?

It would be great to have 10.0.0.x (where x is one, two, three, or four) be unified IPs, and then Linux would route to whatever host(s) are currently online. I could then later add more WireGuard servers by just adding more routes, and 10.0.0.x would remain the same.

I've already tried to do this with iproute2 and route metrics, however, the WireGuard interface is never marked as "DOWN" when the remote server fails, so the switchover never happens and packet loss occurs.

I'm migrating a Galera cluster from MySQL to MariaDB 10.3.27, and as part of the process I'd like to change the backup method from the somewhat fiddly (and blocking) garbd+rsync to garbd+mariabackup. Changing the SST method to mariabackup worked fine from the start, so I suppose the nodes are set up correctly, but the way I call garbd is wrong:

garbd --address 'gcomm://10.2.96.145:4567,10.2.96.146:4567?gmcast.listen_addr=tcp://0.0.0.0:4444' --group m1t-shared-mariadb --donor m1t-shared-db-5 --sst mariabackup

Port 4444 is copy/pasted from documentation, I think it pertains to the local garbd's listening port? Anyway it doesn't matter whether I use 4444 or the rest of the cluster's 4567.

On the donor node I see:

2021-01-25 17:34:26 0 [Note] WSREP: Member 0.0 (garb) requested state transfer from 'm1t-shared-db-5'. Selected 3.0 (m1t-shared-db-5)(SYNCED) as donor.
2021-01-25 17:34:26 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 19280263)
2021-01-25 17:34:26 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-01-25 17:34:26 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'donor' --address '' --socket '/var/lib/mysql/mysql.sock' --datadir '/data/mysql/' --gtid 'c12b9eef-1780-11eb-bac6-da564ff9aacc:19280263' --gtid-domain-id '0' --mysqld-args --basedir=/usr'
2021-01-25 17:34:26 2 [Note] WSREP: sst_donor_thread signaled with 0
2021-01-25 17:34:26 0 [Note] WSREP: 0.0 (garb): State transfer from 3.0 (m1t-shared-db-5) complete.
WSREP_SST: [INFO] Streaming with xbstream (20210125 17:34:26.604)
WSREP_SST: [INFO] Using socat as streamer (20210125 17:34:26.606)
WSREP_SST: [ERROR] Upgrade joiner to 5.6.21 or higher for backup locks support (20210125 17:34:26.609)
WSREP_SST: [ERROR] The joiner is not supported for this version of donor (20210125 17:34:26.611)
WSREP_SST: [ERROR] Cleanup after exit with status:93 (20210125 17:34:26.613)
WSREP_SST: [INFO] Cleaning up temporary directories (20210125 17:34:26.617)
2021-01-25 17:34:26 0 [ERROR] WSREP: Failed to read from: wsrep_sst_mariabackup --role 'donor' --address '' --socket '/var/lib/mysql/mysql.sock' --datadir '/data/mysql/' --gtid 'c12b9eef-1780-11eb-bac6-da564ff9aacc:19280263' --gtid-domain-id '0' --mysqld-args --basedir=/usr
2021-01-25 17:34:26 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'donor' --address '' --socket '/var/lib/mysql/mysql.sock' --datadir '/data/mysql/' --gtid 'c12b9eef-1780-11eb-bac6-da564ff9aacc:19280263' --gtid-domain-id '0' --mysqld-args --basedir=/usr: 93 (Protocol not supported)

The error Upgrade joiner to 5.6.21 or higher for backup locks support seems to be a red herring stemming from the fact that the required version is parsed from the --address parameter and that's conspicuously empty.

Grasping for clues I took to reading MariaDB's sql/wsrep_sst.cc file to find out how --address is filled before calling the wsrep_sst_* plugins. If the wsrep_sst_receive_address variable is set to anything but "AUTO", it's supposed to just use that; otherwise it uses wsrep_node_address or, failing that, some guessing procedure (and yes, that's just as documented). However, that doesn't seem to work. --address is always empty, no matter what I set wsrep_sst_receive_address to. Also, an error should be logged if wsrep_sst_receive_address is "AUTO" and wsrep_node_address is not a valid IP address. I tried to provoke that to find out what code path is taken but wasn't able to either.

Any idea what's going wrong here?

Let's say I have a 3 nodes Galera cluster with MariaDB composed of servers A, B and C. Server A and server B can't communicate between each other but both servers can communicate with Server C. How would the cluster react to this situation ?

From what I tested, it seems that when the connection is lost between A and B, the cluster is still working. I could not however restart the cluster. Is there any documentation on this particular case ? I can't seem to find any.

I'm trying to install a Galera cluster with 3 mariadb VM's (all of them CentOS 7). I've never done this before, so I was following this guide:

http://tunnelix.com/mariadb-galera-cluster-installation/

The problem is that after I enter my settings on the /etc/my.cnf.d/server.cnf file (step 6 on the guide), I can't start MariaDB:

[root@galera1 ~]# systemctl start mariadb
Job for mariadb.service failed. See 'systemctl status mariadb.service' and 'journalctl -xn' for details.

[root@galera1 ~]# systemctl status -l mariadb
mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: failed (Result: exit-code) since Thu 2016-10-27 12:23:42 WEST; 5s ago
  Process: 3435 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=1/FAILURE)
  Process: 3433 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)

Oct 27 12:23:39 galera1.syone.int sh[3435]: 2016-10-27 12:23:39 140598747605120 [Note] InnoDB: Waiting for purge to start
Oct 27 12:23:39 galera1.syone.int sh[3435]: 2016-10-27 12:23:39 140598747605120 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.32-78.1 started; log sequence number 1617490
Oct 27 12:23:39 galera1.syone.int sh[3435]: 2016-10-27 12:23:39 140598747605120 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
Oct 27 12:23:39 galera1.syone.int sh[3435]: 2016-10-27 12:23:39 140598747605120 [Note] Plugin 'FEEDBACK' is disabled.
Oct 27 12:23:39 galera1.syone.int sh[3435]: 2016-10-27 12:23:39 140598747605120 [Note] Server socket created on IP: '0.0.0.0'.
Oct 27 12:23:39 galera1.syone.int sh[3435]: 2016-10-27 12:23:39 140598747605120 [Note] WSREP: Recovered position: 00000000-0000-0000-0000-000000000000:-1
Oct 27 12:23:42 galera1.syone.int sh[3435]: WSREP: Failed to recover position: ''
Oct 27 12:23:42 galera1.syone.int systemd[1]: mariadb.service: control process exited, code=exited status=1
Oct 27 12:23:42 galera1.syone.int systemd[1]: Failed to start MariaDB database server.
Oct 27 12:23:42 galera1.syone.int systemd[1]: Unit mariadb.service entered failed state.

Here's the content of the my.cnf file:

[galera]
Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address='gcomm://'
wsrep_cluster_name='galera'
wsrep_node_address='10.1.40.83'
wsrep_node_name='galera1'
wsrep_sst_method=rsync

binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
Allow server to accept connections on all interfaces.
bind-address=0.0.0.0

I've been searching for a solution about this error and the only thing that I found was a bug (https://jira.mariadb.org/browse/MDEV-10396), but I confirmed the /usr/bin/galera_recovery script, and it already has that fix.

The weird part is these 2 lines on the logs:

Oct 27 12:23:39 galera1.syone.int sh[3435]: 2016-10-27 12:23:39 140598747605120 [Note] WSREP: Recovered position: 00000000-0000-0000-0000-000000000000:-1
Oct 27 12:23:42 galera1.syone.int sh[3435]: WSREP: Failed to recover position: ''

He gets the WSREP position, but it fails to get it...? This doesn't make any sense for me. What am I missing?

Thanks!