What Is Secrets Management? Challenges and Best Practices

To help keep secrets safe, this article explains what secrets are, how they work and how organizations can manage secrets effectively on modern platforms – such as Kubernetes and GitHub. We also cover secrets management best practices for helping to protect secrets no matter where they live or what you use them for.

In this article:

• What is secret management?
• Why is secret management important for security?
• The challenge of secrets management
• Best practices for secure secrets management
• Secrets management use cases: Kubernetes and Github

What is secrets management?

Secret management is the practice of responsibly storing secrets used by software systems.

In this context, a secret is any type of information that helps provide access to sensitive data or systems. Common examples of secrets include

• Passwords that a team uses to log into an application or service.
• SSH keys, which can be used to log into remote servers.
• Encryption keys, which unlock encrypted data to make it readable.
• API keys or tokens, which allow to authenticate and authorize access to cloud applications.
• Certificates, authenticating, and authorizing communication between Kubernetes clusters.

Effective secrets management ensures that data like this is stored and shared securely, minimizing the risk of it falling into the wrong hands.

Why is secret management important for security?

From a security perspective, secret management is important because secrets are effectively the keys to an organization’s digital kingdom. If malicious parties find a way to access secrets that should not be available to them, they could abuse the access to log into private systems or steal sensitive information.

Indeed, according to a 2023 report, stolen secrets account for 50 percent of all breaches.

For example, if attackers find a password and username embedded into a plain-text configuration file, they could use those credentials to connect to the application or service that is supposed to be protected by the login credentials. This effectively allows them to impersonate legitimate users.

As another example, imagine that threat actors obtain an encryption key that can unlock a database containing the personal data for a business’s customers. This would enable access to private information. It could also potentially trigger compliance violation for the business, which regulators might deem liable for failing to secure personally identifiable information.

The challenge of secrets management

Note, too, that secrets management is important because without a proper secrets management strategy in place, it can be easy for teams to make mistakes that expose their secrets to malicious parties.

This is true for two main reasons. One is the sheer number of secrets that exist in modern environments. With hundreds or thousands of passwords, access keys and other secrets floating around the various applications, services, configuration files and databases that an organization has running, it can be difficult to keep track of all of them effectively and ensure they are stored in a secure, central location. This leads to what’s known as secrets sprawl, meaning that secrets end up scattered across systems in ways that make it challenging to protect them all.

The second major challenge of effective secrets management is the fact that in some cases, secrets are stored by default in insecure locations. A common example is passwords that are embedded into plain-text configuration files. Engineers might store passwords this way because it’s simpler than other approaches, such as setting up password hashes. They may also believe that it’s not that risky to store sensitive secrets in plain text so long as the files are only intended for use in private settings, like a dev/test environment. Mistakes like these can be especially challenging to avoid in organizations that haven’t adopted a SecDevOps approach, which means IT engineers and developers don’t collaborate closely with security experts.

But the reality is that even data that is not intended to be available to the public can sometimes fall into the wrong hands due to risks like malicious insider attacks and configurations that accidentally expose internal files to third-party users.

Secrets management addresses these challenges by ensuring that all of an organization’s secrets are stored in a secure fashion. In addition, secrets management provides visibility into secrets, allowing a business to know at all times which secrets it has configured, which systems they unlock and which users are able to access which secrets.

Best practices for secure secrets management

Every organization’s approach to secrets will be a bit different because every company manages different secrets and uses them to support different use cases. In general, however, organizations can maximize the security of their secrets by adopting practices like the following.

Store secrets centrally

Storing all secrets in a central location, such as a secrets manager, is a best practice because it helps prevent secret sprawl. To do this, organizations should establish clear policies that require developers and IT engineers to place all types of secrets inside a central repository that is locked down with granular access controls to ensure that each user inside the organization can only access the specific secrets that he or she should be able to unlock.

Define clear secrets governance rules

More generally, organizations should establish clear governance policies that define how to manage secrets. Instead of leaving it up to individual engineers to decide how to approach secrets management – a strategy that is likely to lead to inconsistent and insecure practices – the business should make it easy for practitioners to determine which tools they are supposed to use to store secrets, how often to update secrets, when to delete secrets and so on.

Delete obsolete secrets

When secrets are no longer needed, they should typically be deleted. For example, if a user leaves the organization, any secrets associated with his or her personal accounts should be removed.

This is important from a security perspective because the fewer secrets you have to manage, the lower your risk of exposure to attack – and given that there is no reason to keep obsolete secrets on hand, you should delete them to reduce your exposure.

Update secrets regularly

Along similar lines, regularly updating secrets is a best practice that can help reduce the impact of an attack involving stolen secrets. This is because when you update a secret, malicious actors who obtained access to the secret can no longer use it.

The exact schedule for updating secrets should balance the user experience with security priorities; you don’t want to force employees to update secrets so frequently that it disrupts their workflows. But in general, practices like updating passwords on a monthly basis are wise.

In addition, when dealing with highly sensitive access credentials, businesses should consider configuring secrets so that they can be used only one time before becoming invalid.

Secret auditing

To validate that employees follow secrets management policies, organizations should monitor secrets management practices by maintaining logs and audit trails that track access and modifications to secrets. These resources can also help to demonstrate compliance and support forensic investigations in case of security incidents.

Automate secrets management

When you have a large volume of secrets to manage and you need to make changes to your secrets configurations on an ongoing basis, managing everything manually is impractical. Instead, organizations should employ automations to enforce their secrets management policies. Automations can do things like automatically require users to update secrets after they reach a preset expiration threshold and delete secrets for users when they leave the organization. Automation also makes it possible to integrate with DevOps pipelines so that secrets generated at every stage of the software delivery lifecycle are securely managed.

Secrets management use cases: Two examples

To illustrate what effective secrets management looks like in practice, here are two examples of how an organization might manage secrets for two key types of modern systems – Kubernetes and GitHub.

Managing Kubernetes secrets

Kubernetes supports a built-in object called (you guessed it) a secret. This makes it possible to create a secret using a tool called kubectl with a command like:

# kubectl create secret mysecret --from-literal=password='myp@ssw@rd!' -n mynamespace
# secret/mysecret created

This creates a secret that is not secure inside Etcd, Kubernetes’s configuration repository and cluster’s state, as Kubernetes secrets are base64-encoded and stored as plain text in Etcd The secret can then be retrieved  using a command like:

# kubectl get secret mysecret -n mynamespace -o jsonpath='{.data.password}' | base64 --decode

# myp@ssw@rd!

To make secrets management in Kubernetes more scalable and secure, you can also opt to use a third-party secrets management tool instead of Etcd to store secrets. This allows you to store secrets for other systems (not just Kubernetes) in the same place. Secrets managers also typically provide more automation features and other capabilities than you’ll get from using kubectl and Etcd alone to manage Kubernetes secrets.

That said, even if you set up a secure secrets management process within Kubernetes, Kubernetes won’t automatically detect or alert you to insecure secrets, such as passwords embedded into configuration files. That’s why scanning for Kubernetes secrets to detect secrets that are not stored properly is also important.

Managing Github Secrets

GitHub provides a feature for managing secrets for use within software development. Using this feature, you can set up secrets to your pipeline by navigating to the Settings > Security > Secrets and variables, and selecting the Actions tab.

Keep in mind, however, that this feature won’t prevent teams from engaging in insecure secrets management practices, such as embedding secrets into configuration files that they then upload to a GitHub repository. To protect against these risks, you should also perform GitHub secrets scanning.

Secret detection from code to cloud 

While establishing strong secrets management policies and automations helps reduce risks, it doesn’t guarantee that insecure secrets won’t lurk somewhere in your code.

This is why you should also employ a solution like Aqua, which can scan source code, container images, configuration files, repositories and a variety of other sources to detect insecure secrets – such as the “phantom” secrets that Aqua researchers uncovered by scanning 50,000 public repositories on GitHub.

By delivering an extra layer of defense against insecure secrets management, Aqua helps minimize the risk that a stolen password, encryption key, API token or other secret might become the weakest link in your organization’s security strategy and trigger a massive breach.