Setup: I have an application that I currently deploy manually on a server. It requires several credentials (client secrets of external services, tokens, and AES keys and IVs etc.) that I currently have stored in a file encrypted with gpg. Whenever I restart the application I unlock the gpg-key in the console and the service will then decrypt the file again from within the application script (as gpg-agent persists the passphrase in memory for a limited time) and parse them through a pipe.
The advantage of this method is that neither the credentials nor the gpg-passphrase needed to access them are NEVER persisted to disk and only kept in memory. So even with full root access it is impossible to recover the credentials. The only chance is to gain access during the short time-window during which the gpg-key stays unlocked.
The disadvantages are that (a) no other user can start the service (unless we share my account) and (b) the application cannot be started by any automated means.
There is the creds project on github that is similar to what I'm doing, but which still shares the above shortcomings. It may be possible to encrypt the file with multiple gpg keys and then attempt to decrypt it with all keys until one succeeds.
Question: What are better ways to handle the credentials that is (a) accessible by multiple users without key or account sharing, (b) accessible by a CI/CD system (where I do not have a console every time a deployment happens), and (c) only keeps password data in memory? The "workaround" with multiple gpg keys looks like a complex hack and doesn't work with CI/CD systems.
I'll talk about secrets instead of credentials, as there might be other sensitive information you'd like to protect. It doesn't matter that your question is worded specifically towards CI/CD systems, the issue is the same whether we're talking about using X.509 certificates for authentication, saving database credentials, or protecting an access token for a build agent.
There's no canonical way to handle this, as application and organizations' needs differ on what constitutes secrets and how to handle them. Some applications might offer no other way to store secrets than in a file.
Some applications encrypt secrets on disk, but as they often have to be symmetric that is more or less decorative.
So, what can you do?
Your first line of defense is your operating system's DAC (discretionary access control) and MAC (mandatory access control) - if we're talking Linux, POSIX permissions are DAC and LSMs like App Armor, SELinux or GRSecurity are MAC.
Auditing your OS to proper standards like CIS or DISA STIG.
Using a HSM or OpenPGP smartcard to store the PGP key you use to encrypt the secrets on disk springs to mind. Devices like these guarantee that the key never leaves the hardware.
Keep in mind that no single measure keeps the keys in a HSM secure - physical access can still compromise them. Hardware key storage devices must be combined with physical security and proper operational procedures to enforce their security.
Check the major browser's root CA policies (Chrome, Firefox, Edge/IE). They enforce the usage of hardware crypto devices and several restrictions and rules on how to operate them and what audits you have to pass.