Bozobooks.com: Fullstack k8s application blog series

Chapter 2: Secure secrets with Hashicorp Vault

8 min readOct 6, 2022

In this blog, we will walk through the complete process of setting up Vault in the Kubernetes cluster and using Secret Injection to consume the secrets in our Pods.

In Chapter 1, we went through the high-level architecture and the design of the Bozobooks.com application. Before we start building our application components, it's important to set up a Secret Management solution to store all secrets. We will be using Vault to store our secrets.

The following diagram shows a high-level architecture of Vault deployment in our k8s cluster.

Let's now go through the setup and configuration

Installing Vault into the cluster

We will be using helm vault operator-based installation, to install and set up the vault in our BozoBook.com cluster.

Step 1: Add the Hashicorp helm repository, and update the repository

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Step 2: Define values.yaml, to be used to install helm

server:
  affinity: ""
  ha:
    enabled: true
    raft:
      enabled: true

We really don't have to enable HA in the Dev environment, just to demonstrate I have enabled it. This will spin off 3 vault pods. (We will be doing Prod in AWS separately)

We are also enabling Raft storage. Vault supports various storage backends; however, configuring and managing them is operationally complex, a HA setup. Let's use the integrated storage solution, that vault provides called Raft. For more information check this site Vault HA Cluster with Integrated Storage

Step 3: Install Vault

Let's now install vault with helm, passing the values yaml we just created

helm install vault hashicorp/vault -n vault --create-namespace --values helm-vault-raft-values.yml

This will set up the vault cluster.

Step 4: Initialize Vault

Once the vault pods in vault namespaces are started, we need to initialize them. This is only the first time. This process will configure the number of key shares and threshold (Vault uses Shamir’s Secret Sharing algorithm, which provides a distributed key-sharing approach). Since this is in a development environment, let's generate 1 key share with 1 threshold (which means there is only 1 key, and the vault can be unsealed, with just 1 key). As we move to production (AWS), in later blogs, we will be using Auto-unsealing.

Just to understand the concept better, please see the diagram and the explanation below

When we initialize the vault we provide key-sharesand key-threshold values. The vault operator will then generate key-shares number of unseal-keys. These keys are shards of the root key or master key. The root key/master key is what is used to encrypt the “Encryption Key”. The Encryption key is actually used to encrypt the data store, which stores all our secrets.
During the UNSEAL process: Vault will need key-threshold number of keys to generate the root key, this root key is then used to decrypt the Encryption Key. This Encryption Key is then used to decrypt the data stored.

When we move to production, we will not be using this unseal process. We will be using the “Auto-Unseal” process, which works almost in a similar way, but we will not have so many shards/unseal keys. The unseal key is stored in the cloud native key management system (KMS). See the picture below

Let's now implement the dev environment approach. (I will be covering the KMS-based solution in later blogs, as we move to production)

To initialize the vault cluster, we can use the vault operator init command. The following command logs into vault-0 (one of the pods), and executes the vault operator init command, and the generated json is stored in cluster-keys.json. This json has to be secured, as this has the root key and unseal-keys. Let's execute the command

kubectl exec -n vault vault-0 -- vault operator init \
                                               -key-shares=1 \
                                               -key-threshold=1 \
                                               -format=json >        cluster-keys.json

Here is the copy of the cluster-keys.json that is generated. As you can see, we have the unseal_keys which has the base64 encoded unseal key

Step 5: Unseal the Vault

The vault cluster is sealed by default, and we need to unseal it to start accessing the secrets. To unseal, we can use vault operator unseal command, using the unseal key that is generated during the initialization process.

export UNSEAL_KEY=zzr+oQ0MF5uFsg0AJlbUKm5SnZpcOjOK3tlsv6pkVpc=kubectl exec -n vault -ti vault-0 -- vault operator unseal $UNSEAL_KEY

The following screenshot shows the output after unsealing the vault-0 pod

Step 6: Bring other pods to the RAFT cluster

Once vault-0 pod is unsealed, let's bring vault-1, vault-2 into the raft cluster. we can do that using the vault operator raft join command. The following are the commands

kubectl exec -n vault -ti vault-1 — vault operator raft join http://vault-0.vault-internal:8200kubectl exec -n vault -ti vault-2 — vault operator raft join http://vault-0.vault-internal:8200

The following screenshot shows the output after running these commands

Step 7: Unseal vault-1 and vault-2

Now we can unseal the other vault pods in the cluster

kubectl exec -n vault -ti vault-1 -- vault operator unseal $UNSEAL_KEY
kubectl exec -n vault -ti vault-2 -- vault operator unseal $UNSEAL_KEY

The following screenshot shows the output after unsealing the vault pods

The following diagram shows a high-level architecture of the deployment.

Once the Vault servers are unsealed, we will have to configure the vault server to work with our applications and configure access and policy.

The following diagram shows how the secrets are injected by vault and how these secrets are retrieved by our application pods. we will be executing as part of “Step 7” to configure the Kubernetes cluster access.

Step 7: Configure Vault to be used in our K8s cluster

To use the vault in our k8s cluster, we have to do the following configurations.

7.1 Login to the vault -0 pod

kubectl exec -n vault --stdin=true --tty=true vault-0 -- /bin/sh

Once you are in the vault shell, you might want to login into vault using the root_token that is generated in cluster_keys.json that we generated in the previous step while running init

vault login hvs.5pUdXyuHhoaov3sS3ZOJjAFl

7.2 Enable Kubernetes authentication by executing

vault auth enable kubernetes

7.3 Configure the Kubernetes config key with the Kubernetes host URL.

vault write auth/kubernetes/config \          kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

7.4 Create a vault policy for our key stores. This command creates a policy, which allows “read” access to the key stores

bozobooks/data/googleapi-config (where we will be storing our Google Books API — which will be used by book-info-service microservice to query google books API to get books information)
bozobooks/data/postgres-config (where we will store the Postgres username, and password. This will be used by book-library-service microservice to store the book preferences in a Postgres database)

vault policy write bozobooks-app-policy - <<EOF
path "bozobooks/data/googleapi-config" {
  capabilities = ["read"]
}
path "bozobooks/data/postgres-config" {
  capabilities = ["read"]
}
EOF

let's check if the policy has been created

$ vault policy list
bozobooks-app-policy
default
root$ vault policy read bozobooks-app-policy
path "bozobooks/data/googleapi-config" {
  capabilities = ["read"]
}
path "bozobooks/data/postgres-config" {
  capabilities = ["read"]
}

7.5 We need to create a vault role and attach a k8s service account to the policy that we just created. This will provide allow all the pods, that use the service account, to access the vault secrets as per the policy.

vault write auth/kubernetes/role/bozobooks-app \
bound_service_account_names=bozobooks-app \
bound_service_account_namespaces=bozo-book-library-dev \
policies=bozobooks-app-policy \
ttl=24h

we are binding the policy with a service account bozobooks-app in bozo-book-library-dev namespace. We will be using this service account for all the pods, that require access to the secrets.

Now vault is ready to be used. We will be using a vault to store the following secrets

Lets now configure the secrets in the vault for our application

Google API key
postgres database username and password

both the secret configurations, we will store them as key-value pairs. Let's create these secrets.

kubectl exec -n vault --stdin=true --tty=true vault-0 -- /bin/sh

The following screenshot shows the output of executing the command

Let's now create a secret with the path bozobooks We will be storing the googleapikey key-value under googleapi-config kv and POSTGRES_USER, POSTGRES_PASSWORD under postgres-config kv.

Please refer to Google Books API documentation on to generate a google API key

Following are the commands to configure these key-values

vault secrets enable -path=bozobooks kv-v2vault kv put bozobooks/googleapi-config googleapikey=<googleapikey>
vault kv put bozobooks/postgres-config POSTGRES_USER="postgres" POSTGRES_PASSWORD=""

Let's check if the kv is created

Now we have all the secrets configured, and they are ready to be used. We will be using the Vault injection technique to use these secrets. I will be covering that in the respective microservices chapters of the blog series.

Auto-Unseal using AWS KMS will be covered as we move to Production.

See you soon… take care

Bozobooks.com: Fullstack k8s application blog series

Chapter 2: Secure secrets with Hashicorp Vault

Installing Vault into the cluster

Written by A B Vijay Kumar