Home / server / Questions / 809153
In Process
Nathan Basanese
Asked: 2016-10-15 13:15:57 +0800 CST

How do I secure the access token, on Linux, to remote, automated secrets stores like Hashicorp Vault?

  • 772

There seems to be a bit of a "chicken and egg" problem with the passwords to the password managers like Hashicorp Vault for Linux.

While researching this for some Linux servers, someone clever asked, "If we're storing all of our secrets in a secrets storage service, where do we store the access secret to that secrets storage service? In our secrets storage service?"‡

I was taken aback, since there's no point to using a separate secrets storage service if all the Linux servers I'd store the secrets on anyway have its access token.

For example, if I move my secrets to Vault, don't I still need to store the secrets to access Hashicorp Vault somewhere on the Linux server?

There is talk about solving this in some creative ways, and at least making things better than they are now. We can do clever things like auth based on CIDR or password mashups. But there is still that trade-off of security For example, if a hacker gains access to my machine, they can get to vault if the access is based on CIDR.

This question may not have an answer, in which case, the answer is "No, this has no commonly accepted silver bullet solution, go get creative, find your tradeoffs bla bla bla"

I want an answer to the following specific question:

Is there a commonly accepted way that one secures the password to a remote, automated secrets store like Hashicorp Vault on modern Linux servers?

Obviously, plaintext is out of the question.

Is there a canonical answer to this? Am I even asking this in the right place? I considered security.stackexchange.com, too, but this seemed specific to a way of storing secrets for Linux servers. I'm aware that this may seem too general, or opinion based, so I welcome any edit suggestions you might have to avoid that.

‡We laugh, but the answer I get on here may very well be "in vault". :/ For instance, a Jenkins server or something else has a 6-month revokable password that it uses to generate one-time-use tokens, which they then get to use to get their own little ephemeral (session limited) password generated from Vault, which gets them a segment of info.

Something like this seems to be along the same vein, although it'd only be part of the solution: Managing service passwords with Puppet

linux authentication automation password-management keystore

1 Answers

  1. // , First of all, the problem discussed here goes beyond mere delivery of secret 0 or what they call "secure introduction" in ops parlance.

    Rather, this is about securing the secret, once received.

    I don't have a silver bullet solution to this. But there are a few defense-in-depth options:

    1. Use response wrapping for delivery of the secret.
    2. Place CIDR restrictions around token to the Secret Store, such that the token is only usable from a specific set of IP addresses, and use reliable protocols like PROXY (NOT X-Forwarding) to pass the IP addresses to the secret store (e.g. set token_bound_cidrs in a way that only one subnet can ever use the token.)
    3. Store the secret in memory only, and lock the memory with mlock.
    4. If possible, place time limits on the secret itself, or even allow the secret to only be used once
    5. Monitor for unusual use of the secret, e.g. the regular client should alert if its one-time-use secret doesn't work (because it's already been used) and the server should alert if someone is trying to use the secret from outside the allowed CIDR range
    6. This is kind of going out on a limb, here, but you might allow a "honeypot" secret to exist on the server along with the regular secret, if possible, that gives "access" to a set of credentials to a system that just records access and alerts.
    7. Require re-authentication for each use of the locally stored secret, which would mean additional authentication factors aside from locally stored secrets need to be applied with each use, e.g. signed meta-data unique to the compute instance or workload, or, in Vault, an Approle
    8. Disable any sort of disk caching, to prevent the secret from touching any potentially persistent storage
    • 0