Integrating Hashicorp Vault with OKE (Oracle Container Engine for Kubernetes)
Welcome to this guide to integrating Vault with Oracle Container Engine for Kubernetes (OKE).
Introduction
First I'll introduce you to Hashicorp Vault, a comprehensive tool designed for the secure management of secrets.
Then I'll provide an overview of the Vault Kubernetes Auth Method, which is used to facilitate authentication with Vault by using a Kubernetes Service Account Token. This method of authentication leverages native Kubernetes identity and access management, and makes it easy to introduce a Vault token into a Kubernetes Pod.
Finally, I'll provide an overview of a Vault deployment scenario on OKE, whereby Vault will be both deployed to OKE, and integrated with the OKE cluster control plane using the Vault Kubernetes Auth Method. This deployment scenario is fully documented in this work instruction, including a step-by-step guide and necessary configuration files.
Oracle Cloud Infrastructure Container Engine for Kubernetes (often abbreviated as `OKE`) is a fully-managed, scalable, and highly available service that you can use to deploy your containerized applications to the cloud. Use OKE when your development team wants to reliably build, deploy, and manage cloud-native applications. OKE uses Kubernetes - the open-source system for automating deployment, scaling, and management of containerized applications across clusters of hosts.
Vault
Secrets management is one of the core use cases for Vault. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any type of secret, while providing important features such as an API driven framework delivering tight access control, version control, and detailed audit log.
Many organisations have credentials hard coded in source code, littered throughout configuration files and configuration management tools, and stored in plaintext in version control, wikis, and shared volumes. Vault provides a central place to store these credentials, ensuring they are encrypted, access is audit logged, and exposed only to authorized clients.
Vault provides a wide array of features across secrets management, data protection, identity-based access, collaboration & operations, and governance and compliance.
Vault is able to provide tight control over access to secrets and encryption keys by authenticating against trusted sources of identity, such as Active Directory, LDAP, Kubernetes, and cloud platforms. Vault enables fine grained authorization across which users and applications are permitted access to secrets and keys.
Kubernetes Auth Method
When working with Vault, successful authentication is a pre-requisite step for actors to store/retrieve secrets and perform cryptographic operations. Tokens are the core method for authentication within Vault which means that the secret consumer must first acquire a valid token. The Vault authentication process verifies the client's identity (secrets consumer), and then generates a token to associate with that identity.
Vault provides a range of Auth Methods to address application requirements where running on a variety of platforms. The Kubernetes Auth Method works well for Kubernetes based orchestrators such as OKE.
The Kubernetes Auth Method is used to authenticate with Vault using a Kubernetes Service Account Token. The token for a pod’s service account is automatically mounted within a pod at /var/run/secrets/kubernetes.io/serviceaccount/ token when a pod is instantiated. When using the Kubernetes Auth Method, it is this token which is sent to Vault for authentication.
Vault is configured with a service account that has permissions to access the Kubernetes TokenReview API. This service account is used by Vault to make authenticated calls to the Kubernetes API Server in order to verify service account tokens presented by pods that want to connect to Vault to access secrets.
The Vault deployment scenario illustration describes the solution architecture implemented in the referenced work instruction.
Flow 2, represented in orange provides a high level representation of the authentication workflow implemented via the Kubernetes Auth Method.
On successful completion, Vault returns a token to the application with pre-configured policies attached. From step 3 onward, the application can use the token to retrieve secrets from Vault’s Key/Value secrets engine.
Deployment Scenario
The following provides a break-down of the components provisioned in the deployment scenario:
- OKE Kubernetes cluster
- etcd and Vault Kubernetes operators
- Vault cluster
- Vault configured to use Kubernetes auth method
- Vault Key/Value (KV) store
secret/testapp - Vault role 'testapp' associated with the Kubernetes service account 'testapp' in the 'default' namespace
- Vault policy 'testapp-kv-crud' associated with 'testapp' role (providing CRUD access to the
secret/testappKV store) - etcd cluster serving as persistent storage tier for the Vault cluster
- Test application authenticated to vault (via Kubernetes auth) using the
testappservice account - Test application creating and reading secrets from the Vault
secret/testappKV store
Both the etcd and Vault clusters will be created by their respective Kubernetes operators. The Vault operator deploys and manages Vault clusters on Kubernetes. Vault instances created by the Vault operator are highly available and support automatic failover and upgrade. For each Vault cluster, the Vault operator will also create an etcd cluster for the storage backend.
The following is a high level outline of the process described in the work instruction to install and integrate Vault with OKE:
- Deploy the Vault & etcd operators
- Deploy the Vault & etcd clusters
- Configure Vault Kubernetes auth
- Create the Vault Key/Value (KV) store & associated policy for the test application
- Deploy the test application and authenticate to Vault using Kubernetes auth
- Create and read secrets from the KV store
The work instruction is a great way to quickly get up and running with a comprehensive, HA deployment of Vault on OKE.
Summary
Integrating Vault authentication with Kubernetes using the Kubernetes Auth Method serves to simplify the process of authenticating Vault clients by using a Kubernetes Service Account Token.
Some challenges do remain around solving how to manage the lifecycle of tokens in a standard way, without having to write custom application logic. The Vault team are planning a number of features which are designed to address these challenges.
One of the proposed mechanisms involves integrating Vault with the Kubernetes Secrets mechanism via a periodically running sync process. Another involves a Container Storage Interface plugin to inject secrets into a running pod. There's also mention of injecting Vault secrets into Pods via a sidecar container - each very interesting approaches to solving for this complexity.
The Kubernetes and cloud native space is moving fast - watch this space as we will continue to share solutions, tutorials, best practices, and more, all designed to spark inspiration, drive hands-on experiences, and unleash our industry’s potential.