We have configured vault to run as a pod in the cluster. In the below deployment YAML file, we have included the vault initialisation and unsealing to happen when the pod comes up initially. But when the pod gets restarted, the pod is going to crashLoopBackOff state because the vault is getting reinitialised. This is because we have included both the initialisation and unsealing command in the postStart lifecycle command of the deployment file. Is there any way in which we could initialise the pod only once and later when pod restarts, unseal the vault using the existing keys?
Deployment file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: vault
name: vault
spec:
replicas: 1
template:
metadata:
labels:
app: vault
spec:
containers:
- image: vault
name: vault
imagePullPolicy: Always
ports:
- containerPort: 8200
name: vaultport
protocol: TCP
securityContext:
capabilities:
add:
- IPC_LOCK
env:
- name: VAULT_ADDR
value: "http://0.0.0.0:8200"
command: ["vault", "server"]
args:
- "-config=/vault/config/config.hcl"
volumeMounts:
- name: vault-unseal
mountPath: /vault/file/unseal.sh
subPath: unseal.sh
- name: vault-config
mountPath: /vault/config/config.hcl
subPath: config.hcl
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh" ]
volumes:
- name: vault-unseal
configMap:
name: vault-unseal
- name: vault-config
configMap:
name: vault-config
imagePullSecrets:
- name: regcred
Output of kubectl describe pod:
Name: vault-677bfd9c9c-dwsgv
Namespace: xxx
Priority: 0
Node: xxxxxxx-5b587f98-ljf4/10.0.0.11
Start Time: Thu, 30 Jan 2020 06:26:21 +0000
Labels: app=vault
pod-template-hash=677bfd9c9c
Annotations: <none>
Status: Running
IP: 10.4.2.10
IPs: <none>
Controlled By: ReplicaSet/vault-677bfd9c9c
Containers:
vault:
Container ID: xxxxxxxxxxx
Image: xxxxxxxxxxxxxxxx
Image ID: xxxxxxxxxxxxxxxxxxxxxxxxx
Port: 8200/TCP
Host Port: 0/TCP
Command:
vault
server
Args:
-config=/vault/config/config.hcl
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 30 Jan 2020 06:26:26 +0000
Finished: Thu, 30 Jan 2020 06:26:27 +0000
Ready: False
Restart Count: 1
Environment:
VAULT_ADDR: http://0.0.0.0:8200
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-kxfdb (ro)
/vault/config/config.hcl from vault-config (rw,path="config.hcl")
/vault/file from vault-data (rw)
/vault/file/unseal.sh from vault-unseal (rw,path="unseal.sh")
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
vault-unseal:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-unseal
Optional: false
vault-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
vault-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: vault-data
ReadOnly: false
default-token-kxfdb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-kxfdb
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18s default-scheduler Successfully assigned xxx/xxxxxxxxxx
Normal Pulling 13s (x2 over 15s) kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4 pulling image "xxxxxxxxx"
Normal Pulled 13s (x2 over 15s) kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4 Successfully pulled image "xxxxxxx"
Normal Created 13s (x2 over 15s) kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4 Created container
Normal Started 13s (x2 over 14s) kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4 Started container
Warning FailedPostStartHook 12s (x2 over 14s) kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4 Exec lifecycle hook ([/bin/sh -c vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh]) for Container "vault" in Pod "vault-677bfd9c9c-dwsgv_xxx(6ebdc17a-4329-11ea-9fc1-4201c0a80004)" failed - error: command '/bin/sh -c vault operator init > /vault/file/keys.txt; sh /vault/file/unseal.sh' exited with 2: Error initializing: Error making API request.
URL: PUT http://0.0.0.0:8200/v1/sys/init
Code: 400. Errors:
* Vault is already initialized
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was: file descriptor 0 is not a terminal
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was: file descriptor 0 is not a terminal
An error occurred attempting to ask for an unseal key. The raw error message
is shown below, but usually this is because you attempted to pipe a value
into the unseal command or you are executing outside of a terminal (tty). You
should run the unseal command from a terminal for maximum security. If this
is not an option, the unseal key can be provided as the first argument to the
unseal command. The raw error was: file descriptor 0 is not a terminal
Token (will be hidden):
Error authenticating: An error occurred attempting to ask for a token. The raw error message is shown below, but usually this is because you attempted to pipe a value into the command or you are executing outside of a terminal (tty). If you want to pipe the value, pass "-" as the argument to read from stdin. The raw error was: file descriptor 0 is not a terminal
, message: "Unseal Key (will be hidden): \nUnseal Key (will be hidden): \nUnseal Key (will be hidden): \nKey Value\n--- -----\nSeal Type shamir\nInitialized true\nSealed true\nTotal Shares 5\nThreshold 3\nUnseal Progress 0/3\nUnseal Nonce n/a\nVersion 1.3.2\nHA Enabled false\n++++++++++++ Vault Status +++++++++\nKey Value\n--- -----\nSeal Type shamir\nInitialized true\nSealed true\nTotal Shares 5\nThreshold 3\nUnseal Progress 0/3\nUnseal Nonce n/a\nVersion 1.3.2\nHA Enabled false\nError initializing: Error making API request.\n\nURL: PUT http://0.0.0.0:8200/v1/sys/init\nCode: 400. Errors:\n\n* Vault is already initialized\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was: file descriptor 0 is not a terminal\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was: file descriptor 0 is not a terminal\nAn error occurred attempting to ask for an unseal key. The raw error message\nis shown below, but usually this is because you attempted to pipe a value\ninto the unseal command or you are executing outside of a terminal (tty). You\nshould run the unseal command from a terminal for maximum security. If this\nis not an option, the unseal key can be provided as the first argument to the\nunseal command. The raw error was: file descriptor 0 is not a terminal\nToken (will be hidden): \nError authenticating: An error occurred attempting to ask for a token. The raw error message is shown below, but usually this is because you attempted to pipe a value into the command or you are executing outside of a terminal (tty). If you want to pipe the value, pass \"-\" as the argument to read from stdin. The raw error was: file descriptor 0 is not a terminal\n"
Normal Killing 12s (x2 over 14s) kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4 Killing container with id docker://vault:FailedPostStartHook
Warning BackOff 10s (x2 over 11s) kubelet, gke-cluster-testing--np-testing-featu-5b587f98-ljf4 Back-off restarting failed container
Yes, in this case you need to make your script smarter to check first if the vault is already initialized. Here is a very simple example with a bash script.
Also, a common pattern for achieving this is by using a sidecar (that eventually goes to "sleep" indefinitely), but the
postStarthook should also work as long as you make your script resilient enough considering that in some cases it will run before the vault container is up and will end up killing it and restarting until they both eventually sync up.I'd highly recommend this talk by Seth Vargo where auto-initialization is done with a more elaborate Golang program.