Why Vault and Kubernetes is the perfect couple

The (not so) secret flaws of Kubernetes Secrets

When you’re starting learning and using Kubernetes for the first time you discover that there is this special object called Secret that is designed for storing various kinds of confidential data. However, when you find out it is very similar to ConfigMap object and is not encrypted (it can be optionally encrypted at rest) you may start wondering – is it really secure? Especially when you use the same API to interact with it and the same credentials. This, combined with a rather simple RBAC model, can create many potential risks. Most people would stick with one of three default roles for regular users – view, edit, and admin – with view as the only one that forbids viewing Secret objects. You need to be very careful when assigning roles to users or deciding to create your custom RBAC roles. But again, this is also not that easy since RBAC rules can only whitelist API requests – it is not possible to create exceptions (i.e. create blacklists) without using the external mechanism such as Open Policy Agent.

Managing Secrets is Hard

On top of that managing Secret object definitions (e.g. yaml files) is not an easy task. Where should you store it before sending it to your Kubernetes cluster – in a git repo? Outside of it? Who should have access to view and modify it? What about encryption – should it be encrypted with a single key shared by the trusted team members or with gpg (e.g. git-secret, git-crypt)?
One thing is for sure – it is hard to maintain Secret object definitions in the same way as other Kubernetes objects. You can try to come up with your own way of protecting them, auditing changes and other important things you’re not even aware of, but why reinvent the wheel when there’s something better? Much, MUCH better.

HashiCorp Vault to the Rescue

Now some may say I am a HashiCorp fanboy which… might be partially true 🙂 I not only love their products but more their approach towards managing infrastructure and the fact that most features they provide are available in open source versions.
It is not a surprise that the best product they have on offer (in terms of commercial success) is Vault. It is a project designed to help you store and securely access your confidential data. It is designed for this purpose only and has many excellent features among which you will also find many that are specific for Kubernetes environments.

Best features of Vault

I’m not going to list out all of the features – they are available in the official documentation. Let me focus on the most important ones and the ones also related to Kubernetes.

One security-dedicated service enterprise features

The fact that it’s a central place where you store all your confidential data may be alarming at first, but Vault offers many interesting functionalities that should remove any doubts in its security capabilities. One of them is the concept of unsealing Vault after start or restart. It is based on a Shamir’s Secret Sharing concept which requires the usage of multiple keys that should be owned and protected by different people. This definitely decreases the chance of interfering or tampering with stored data, as the whole process imposes transparency of such actions.
Of course there’s audit, high availability and access defined with well-documented policies.

Ability to store various type of data

The first thing that people want to store in places like Vault are passwords. This is probably because we use them most often. However, if you want to deploy Vault only for this purpose you should reconsider it, cause it’s much more powerful and it’s like driving a Ferrari using 1st gear only. Vault has many secret engines designed for different kind of data. The basic one – KV (Key-Value) – can be used for store any arbitrary data with advanced versioning. It can also act as your PKI or Time-based One Time Passwords(similar to Google Authenticator). But that’s not all. In my opinion, the real power of Vault lies in dynamic secrets.

Forget your passwords with dynamic secrets

It’s my personal opinion and I think that many people will agree with me that dynamic secrets are the best feature of Vault. If there was a single reason for me to invest my time and resources to implement Vault in my organization that would be it. Dynamic secrets change the way you handle authentication. Instead of configuring static passwords you let Vault create logins and passwords on the fly, on-demand, and also with limited usage time. I love the fact that Vault also rotates not only users passwords but also administrators as well, cause let’s be honest – how often do you change your password to your database and when the last time you did it?
Vault can manage access to your services instead of only storing static credentials and this is a game-changer. It can manage access to databases, cloud (AWS,Azure,Google), and many others.

No vendor lock-in

There are various cloud services available that provide similar, however limited in features, functionality. Google recently announced their Secret Manager, AWS has Parameter Store, and Azure offers Key Vault. If you’re looking for a way to avoid vendor lock-in and keep your infrastructure portable, multi-cloud enabled and feature-rich then Vault will satisfy your needs. Let’s not forget about one more important thing – not every organization uses cloud and since Vault can be installed anywhere it also suits these environments perfectly.

Multiple authentication engines with excellent Kubernetes support

In order to get access to credentials stored in Vault you need to authenticate yourself and you have plenty of authentication methods to choose from. You can use simple username and password, TLS certificates but also use your existing accounts from GitHub, LDAP, OIDC, most cloud providers and many others. These authentication engines can be used by people in your organization and also by your applications. However, when designing access for your systems, you may find other engines to be more suitable. AppRole is dedicated to those scenarios and it is a more generic method for any applications, regardless of the platform they run on. When you deploy your applications on Kubernetes you will be better off with native Kubernetes support. It can be used directly by your application, your custom sidecar or Vault Agent.

Native Kubernetes installation

Since Vault is a dedicated solution for security, proper deployment can be somewhat cumbersome. Fortunately, there is a dedicated installation method for Kubernetes that uses a Helm Chart provided and maintained by Vault’s authors (i.e. HashiCorp).
Although I really like and appreciate that feature I would use it only for non-production environments to speed up the learning process. For production deployments, I would still use traditional virtual machines and automate it with Terraform modules – they are also provided by HashiCorp in Terraform registry (e.g. for GCP).

Why it is now easier than ever

Until recently using Vault with Kubernetes required some additional, often complicated steps to provide secrets stored in Vault to an application running on Kubernetes. Even with Vault Agent you’re just simplifying only the token fetching part and leaving you with the rest of the logic i.e. retrieving credentials and making sure they are up to date. With additional component – Agent Sidecar Injector – the whole workflow is very simple now. After installing and configuring it (you do it once) any application can be provided with secrets from Vault in a totally transparent way. All you need to do is to add a few annotations to your Pod definitions such as these:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld"
        vault.hashicorp.com/role: "myapp"

No more writing custom scripts, placing them in a separate sidecar or init containers – everything is managed by Vault components designed to offload you from these tasks. It has really never been that easy! In fact, this combined with dynamic secrets described earlier, creates a fully passwordless solution. Access is managed by Vault and your (or your security team’s) job is to define which application should have access to particular services. That’s what I call a seamless and secure integration!

Conclusion

I’ve always been a fan of HashiCorp products and at the same time I’ve considered Kubernetes Secrets as an imperfect solution for providing proper security for storing credentials. With the excellent support for Kubernetes in Vault now we finally have a missing link in a form of a dedicated service with proper auditing, modularity and ease of use. If you think seriously about securing your Kubernetes workloads, especially in an enterprise environment, then HashiCorp Vault is the best solution there is. Look no further and start implementing – you’ll thank me later.

How to increase container security with proper images

Security is a major factor when it comes to a decision of whether to invest your precious time and resources in new technology. It’s no different for containers and Kubernetes. I’ve heard a lot of concerns around it and decided to write about the most important factors that have the biggest impact on the security of systems based on containers running on Kubernetes.
This is particularly important, as it’s often the only impediment blocking potential implementation of container-based environment and also taking away chances for speeding up innovation. That’s when I decided to help all of you who wants strengthen security of their containers images.

Image size (and content) matters

The first factor is the size of a container image. It’s also a reason why container gained so much popularity – whole solutions, often quite complex are now consisting of multiple containers running from images available publicly on the web (mostly from Docker hub) that you can run in a few minutes.
In case your application running inside a container gets compromised and the attacker gets access to an environment with a shell, he needs to use his own tools and often it’s the first thing he does – he downloads them on the host. What if he can’t access tools that will enable him to do so? No scp, curl, wget, python or whatever he could use. That would make things harder or even impossible to make harm to your systems.
That is exactly why you need to keep your container images small and provide libraries and binaries that are really essential and used by the process running in a container.

Recommended practices

Use slim versions for base images

Since you want to keep your images small choose “fit” versions of base images. They are often called slim images and have less utils included. For example – the most popular base image is Debian and it’s standard version debian:jessie weights 50MB while debian:jessie-slim only 30MB. Besides less exciting doc files (list available here) it’s missing the following binaries

/bin/ip
/bin/ping
/bin/ping6
/bin/ss
/sbin/bridge
/sbin/rtacct
/sbin/rtmon
/sbin/tc
/usr/bin/lnstat
/usr/bin/nstat
/usr/bin/routef
/usr/bin/routel
/usr/sbin/arpd

They are mostly useful for network troubleshooting and I doubt if ping is a deadly weapon. Still, the less binaries are present, the less potential files are available for an attacker that might use some kind of zero-day exploits in them.
Unfortunately, neither official Ubuntu nor CentOS don’t have slim versions. Looks like Debian is a much better choice, it contains maybe too many files but it’s very easy to fix – see below.

Delete unnecessary files

Do you need to install any packages from a running container? If you answered yes then you probably doing it wrong. Container images follow a simple unix philosophy – they should do one thing (and do it well) which is run a single app or service. Container images are immutable and thus any software should be installed during a container image build. Afterward, you don’t need it and you can disable it by deleting binaries used for that.

For Debian just put the following somewhere at the end of your Dockerfile

RUN rm /usr/bin/apt-* /usr/bin/dpkg* 

Did you know that even without binaries you may still perform actions by using system calls? You can use any language interpreter like python, ruby or perl. It turns out that even slim version of Debian contains perl! Unless your app is developed in perl you can and should delete it

RUN /usr/bin/perl*

For other base images you may want to delete the following file categories:
Package management tools: yum, rpm, dpkg, apt
Any language interpreter that is unused by your app: ruby, perl, python, etc.
Utilities that can be used to download remote content: wget, curl, http, scp, rsync
Network connectivity tools: ssh, telnet, rsh

Separate building from embedding

For languages that need compiling (e.g. java, golang) it’s better to decouple building application artifact from building a container image. If you use the same image for building and running your container, it will be very large as development tools have many dependencies and not only they will enlarge your final image but also download binaries that could be used by an attacker.

If you’re migrating your app to container environment you probably already have some kind of CI/CD process or at least you build your artifacts on Jenkins or other server. You can leverage this existing process and add a step that will copy the artifact into the app image.

For full container-only build you can use multistage build feature. Notice however that it works only with newer version of Docker (17.05 and above) and it doesn’t work with OpenShift. On OpenShift there is similar feature called chained builds, but it’s more complicated to use.

Consider the development of statically compiled apps

Want to have the most secure container image? Put a single file in it. That’s it. Currently, the easiest way to accomplish that is by developing it in golang. Easier said than done, but consider it for your future projects and also prefer vendors that are able to deliver software written in go. As long it doesn’t compromise quality and other factors (e.g. delivery time, cost) it definitely increases security.

Maintaining secure images

There are three types of container images you use:

  1. Base images
  2. Services (a.k.a. COTS – Common off-the-shelf)
  3. Your applications

Base images aren’t used to run containers – they provide the first layer with all necessary files and utilities
For services you don’t need to build anything – you just run them and provide your configuration and a place to store data.
Application images are built by you and thus require a base image on top of which you build your final image.

Container image types

Of course, both services and applications are built on top of a base image and thus their security depends largely depends on its quality.
We can distinguish the following factors that have the biggest impact on container image security:

  • The time it takes to publish a new image with fixes for vulnerabilities found in its packages (binaries, libraries)
  • Existence of a dedicated security team that tracks vulnerabilities and publish advisories
  • Support for automatic scanning tools which often needs access to security database with published fixes
  • Proven track of handling major vulnerabilities in the past

In a case where there are many applications built in an organization, an intermediary image is built and maintained that is used as a base image for all the applications. It is used to primarily to provide common settings, additional files (e.g. company’s certificate authority) or tighten security. In other words – it is used to standardize and enforce security in all applications used throughout the organization.

Use of intermediary images

Now we know how important it is to choose a secure base image. However, for service images, we rely on a choice made by a vendor or an image creator. Sometimes they publish multiple editions of the image. In most cases it is based on some Debian/Ubuntu and additionally on Alpine (like redis, RabbitMQ, Postgres).

Doubts around Alpine

And what about alpine? Personally, I’m not convinced it’s a good idea to use it in production. I mean there are many unexpected and strange behaviors and I just don’t find it good enough for a reliable, production environment. Lack of security advisories, a single repo maintained by a single person is not the best recommendation. I know that many have trusted this tiny, Linux distribution that is supposed to be “Small. Simple. Secure.”. I’m using it for many demos, training and testing but still, it hasn’t convinced me to be as good as good old Debian, Ubuntu, CentOS or maybe even
Oraclelinux.
And just recently they had a major vulnerability discovered – an empty password for user root was set in most of their images. Check if your images use the affected version and fix it ASAP.

Recommended practices

Trust but verify

As I mentioned before there sometimes official images don’t offer the best quality and that’s why it’s better to verify them as a part of CI/CD pipeline.
Here are some tools you should consider:
Anchore (opensource and commercial)
Clair (opensource)
Openscap (opensource)
Twistlock (commercial)
jFrog Xray (commercial)
Aquasec (commercial)

Use them wisely, also to check your currently running containers – often critical vulnerabilities are found and waiting for next deployment can cause a serious security risk.

Use vendor provided images

For service images, it is recommended to use images provided by a vendor. They are available as so-called “official images”. The main advantage is that not only they are updated when a vulnerability is found in the software but also in an underlying OS layer. Think twice before choosing a non-official image or building one yourself. It’s just a waste of your time. If you really need to customize beyond providing configuration, you have at least two ways to achieve it:
For smaller changer you can override entrypoint with your script that modifies some small parts; script itself can be mounted from a configmap.

Here’s a Kubernetes snippet of a spec for a sample nginx service with a script mounted from nginx-entrypoint ConfigMap

spec:
  containers:
  - name: mynginx
    image: mynginx:1.0
    command: ["/ep/entrypoint.sh"]
    volumeMounts:
    - name: entrypoint-volume
        mountPath: /ep/
  volumes:
  - name: entrypoint-volume
    configMap:
      name: nginx-entrypoint

For bigger changes, you may want to create your image based on the official one and rebuild it when a new version is released

Use good quality base images

Most official images available on Docker Hub are based on Debian. I even did a small research on this a couple of years ago (it is available here) and for some reason, Ubuntu is not the most popular distribution. So if you like Debian you can feel safe and join the others who also chose it. It’s a distribution with a mature ecosystem around
If you prefer rpm-based systems (or for some reasons your software requires it) then I would avoid CentOS and consider Red Hat Enterprise Linux images. With RHEL 8, Red Hat released Universal Base Image (UBI) that can be used freely without any fees. I just trust more those guys, as they invest a lot of resources in containers recently and these new UBI images should be updated frequently by their security team.

Avoid building custom base images

Just don’t. Unless your paranoia level is close to extreme, you can reuse images like Debian, Ubuntu or UBI described earlier. On the other hand, if you’re really paranoid I don’t think you trust anyone, including guys who brought us containers and Kubernetes as well.

Auto-rebuild application and intermediary images when vulnerabilities are found in a base image

In the old, legacy vm-based world the process of fixing vulnerabilities found in a system is called patching. Similar practice takes place in container-based world but in its specific form. Instead of fixing it on the fly, you need to replace the whole image with your app. The best way to achieve it is by creating an automated process that will rebuild it and even trigger a deployment. I found OpenShift ImageStreams feature to address this problem in the best way (details can be found in my article), but it’s not available on vanilla Kubernetes.

Conclusion

We switched from fairly static virtual machines to dynamic, ephemeral containers with the same challenges – how to keep them secure. In this part, we have covered the steps required to address them on the container image level. I always say that container technology enforces the new way of handling things we used to accomplish in rather manual (e.g. golden images, vm templates), error-prone processes. When migrating to containers you need to embrace an automatic and declarative style of managing your environments, including security-related activities. Leverage that to tighten security of your systems starting with basics – choosing and maintaining your container images.