Originally posted by AmberWolf at https://blog.amberwolf.com/blog/2025/september/kubernetes_namespace_boundaries/

Introduction

Multi-tenancy is hard.

- Literally everyone who has tried to implement multi-tenancy in Kubernetes

Kubernetes is great for many things, but performing secure multi-tenancy in a single cluster is not one of its strengths. With organisations looking to get the most out of their hardware and to reduce administrative overhead, I’ve seen my fair share of platforms which aim to provide secure, isolated spaces for individual tenants1. As someone whose job is to treat secure development platforms as an escape room, I’ve broken out of most of those to the point that I’ve had full control of the underlying Kubernetes cluster, and access to information from every tenant.

This post aims to serve as both a collection of attack paths to watch out for when reviewing Kubernetes clusters, but also as a reference to point at when people claim that multi-tenancy is easy.

Setting the Scene

Let’s define our platform. Platform admins generally want users to have as much freedom as possible, but without compromising overall platform security. This helps reduce the number of support tickets raised by users trying to do actions they don’t have permissions to perform. A bit of reading tells us that Kubernetes makes use of namespaces to provide isolation between resources, so it seems reasonable for platform administrators to rely on namespaces to provide a security control.

Whether namespaces are “officially” a security boundary is a matter of some debate, but the Kubernetes docs state the following about RBAC’s Roles and Rolebindings:

In a multi-team environment, RBAC must be used to restrict tenants’ access to the appropriate namespaces, and ensure that cluster-wide resources can only be accessed or modified by privileged users such as cluster administrators.

- https://kubernetes.io/docs/concepts/security/multi-tenancy/#access-controls

So What Does This Mean?

From this text, the impression is that namespaces are segmented off from each other, so granting a tenant full control of a single namespace is safe. On a number of engagements over the last few years, I’ve reviewed clusters where “tenant administrators”, i.e. users who are trusted to have either limited or full control over the namespace, have been granted full RBAC access within that namespace. This is occasionally granted directly through a RoleBinding to the default cluster-admin ClusterRole, or through a CI implementation which has high levels of permission in the namespace. Either way, it’s not unusual to encounter clusters where a tenant admin has what boils down to wildcard permissions for all namespaced resources in a single namespace.

To understand the impact of discussions later in this post, we’ll need a basic understanding of Kubernetes Authorization. This section serves as an introduction: if you’re already familiar with this topic, feel free to skip to skip ahead

Resource Scopes

Resources in Kubernetes are considered either namespaced or global.

We can validate this by checking the resources available in a cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

➜  /tmp kubectl api-resources
NAME                                SHORTNAMES   APIVERSION NAMESPACED   KIND
bindings                                         v1         true         Binding
componentstatuses                   cs           v1         false        ComponentStatus
configmaps                          cm           v1         true         ConfigMap
endpoints                           ep           v1         true         Endpoints
events                              ev           v1         true         Event
limitranges                         limits       v1         true         LimitRange
namespaces                          ns           v1         false        Namespace
nodes                               no           v1         false        Node
persistentvolumeclaims              pvc          v1         true         PersistentVolumeClaim
persistentvolumes                   pv           v1         false        PersistentVolume
pods                                po           v1         true         Pod
podtemplates                                     v1         true         PodTemplate
replicationcontrollers              rc           v1         true         ReplicationController
resourcequotas                      quota        v1         true         ResourceQuota
secrets                                          v1         true         Secret
serviceaccounts                     sa           v1         true         ServiceAccount
services                            svc          v1         true         Service
...

Note the fourth column, which shows if a resource is namespaced or not. Of particular interest is the fact that resources are shown to have a namespaced value of false, meaning they are considered global resources, or true, showing that they are bounded by a namespace.

The general rule of namespaced resources is that multiple of the same resource with the same name can exist cluster-wide. For example, a deployment called application can exist in the project-1 and project-2 namespaces at the same time. These resources are typically tenant- or developer-facing.

Cluster-Wide Resources

Globally scoped resources tend to relate to cluster administration. These include Nodes, CustomResourceDefinitions, and some storage-related resources, among other things.

Role-Based Access Control

When trying to interact with an object in Kubernetes, a principal (normally a user or a service account), needs to have permission on that object. To accomplish this, Kubernetes uses the following resources:

Roles: A role consists of a series of rules, granting verb on object

For example, the following role allows reading of secrets:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secrets-reader
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list

RoleBindings: A link between a principal (i.e. a user, group, or service account) and a Role

To allow the user ‘alice’ to read secrets, we could create the following RoleBinding

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: allow-alice-to-read-secrets
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: secrets-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: alice

ClusterRoles and ClusterRoleBindings: The same as roles and rolebindings, but applied cluster-wide rather than existing in a single namespace.

These resources can all be combined in the following ways:

  • A RoleBinding to a Role in the same namespace will grant permissions only in a single namespace
    • A RoleBinding will only grant permissions on namespaced objects
  • A RoleBinding to a ClusterRole will grant the permissions of the ClusterRole, but only in the namespace in which the RoleBinding is created
    • A RoleBinding will only grant permissions on namespaced objects
    • This is useful for templating, as a single ClusterRole can grant permissions to multiple principals in multiple namespaces without creating the same Role every time
  • A ClusterRoleBinding to a ClusterRole will apply permissions cluster-wide
    • A ClusterRoleBinding will also grant permissions on global resources

Additional Controls

RBAC doesn’t aim to solve every security problem in a Kubernetes cluster. Without additional restrictions, it’s relatively simple for a user with the ability to create a pod to gain full control of an underlying node, including control plane nodes2.

Admission Control

Preventing this behaviour requires an additional step before any resources are successfully modified, and this is generally provided by Admission Controllers. Well configured admission control can prevent malicious (or accidental) security misconfigurations, but as we’ll see in the next section, it’s also relatively easy to overlook something when locking down a cluster.

The main examples of admission control which aim to prevent these behaviours work in one of three ways. OpenShift’s Security Context Constraints and Kubernetes’ now deprecated Pod Security Policies relied on RBAC, granting users the permission to use a policy in their namespace, relying on the boundaries of RoleBindings in relevant namespaces.

Kubernetes’ built-in Pod Security Admission, which replaced Pod Security Policy, uses namespace labels to determine which of three policies (restricted, baseline, or privileged) should be applied to resources created in a single namespace. Workloads created in namespaces labelled with pod-security.kubernetes.io/enforce: restricted will be put under the most stringent of restrictions, and those create in a namespace labelled with pod-security.kubernetes.io/enforce: privileged can do what they want.

Alternatives such as Kyverno or OPA GateKeeper rely on policies being created, either in a single namespace or cluster-wide. Where cluster-wide policies are used, these controllers often rely on namespace selectors to determine which namespaces a rule should be applied to.

Network Policy

Another built-in3 protection against hostile tenants is network policy, which is effectively an implementation of firewall rules between workloads. These work on a podSelector and namespaceSelector basis, and only allow traffic explicitly permitted by the configured policies.

Sharp Edges

Anyone who’s spoken to me for any period of time about Kubernetes, or had the misfortune of being vaguely near me when I’m ranting about it, probably knows my feelings on multi-tenancy in Kubernetes. RBAC does work, and mostly effectively, but there are a plethora of sharp edges and unexpected behaviours in a platform as complicated as a Kubernetes cluster. This section details just some of the ways overly permissive RBAC rules can allow an attacker to gain more access than intended.

Namespaced Controls

It should be clear that making use of namespaced resources to enforce security controls would not be a good idea, if users are going to end up with access to modify those resources.

For example, the (again, now deprecated) Pod Security Policy, or the (not deprecated) OpenShift Security Context Constraints, which create cluster-wide policies. These policies can not be modified (or even viewed) by a user with access to a single namespace.

To create a pod under a specific policy, the creating entity4 needs the use verb on the policy, in the same namespace as the pod was created. Allowing a user wildcard permissions in a namespace grants that user the ability to use the global resource PodSecurityPolicy in that namespace, and allows them to compromise nodes in the cluster.

Namespace Labels

At first glance, it appears that anything which relies on labels on a Namespace resource is not vulnerable to this same attack. After all, namespaces are global resources, as shown by the namespaced field being set to false:

1
2
3

laborant@dev-machine:~$ kubectl api-resources
NAME       SHORTNAMES   APIVERSION NAMESPACED   KIND
namespaces ns           v1         false        Namespace

However, due to an additional oddity in the Kubernetes RBAC model, namespaces are themselves a cluster-wide (i.e. non-namespaced) resource, but also sometimes considered namespaced. Specifically, a namespace is both global and namespaced to its own namespace. This means our tenant admin can modify the labels on their own namespace. A minimum permission for this is GETand PATCH on namespace resources5.

By modifying namespace labels, a huge range of attacks become possible. Regardless of whether or not namespaces should be considered a security boundary in Kubernetes, the reality is that their attributes are often used in security controls in a cluster. The most basic of attacks using these attributes would be to relabel a namespace with a higher PodSecurityAdmission label, then compromise a node, but we’ve also seen other scenarios.

We can see this working below, with a user initially being prevented from running a privileged debug pod, but being permitted after re-labelling the namespace.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Try to create a privileged container, which is rejected
laborant@dev-machine:~$ kubectl debug node/node-01 -n default --image alpine -it -- /bin/sh
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
Creating debugging pod node-debugger-node-01-p2mwk with container debugger on node node-01.
Error from server (Forbidden): pods "node-debugger-node-01-p2mwk" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), allowPrivilegeEscalation != false (container "debugger" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "debugger" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host-root" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "debugger" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "debugger" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

# Re-label the namespace
laborant@dev-machine:~$ kubectl label ns/default pod-security.kubernetes.io/enforce=privileged --overwrite
namespace/default labeled

# Try again, successfully
laborant@dev-machine:~$ kubectl debug node/node-01 -n default --image alpine -it -- /bin/sh
--profile=legacy is deprecated and will be removed in the future. It is recommended to explicitly specify a profile, for example "--profile=general".
Creating debugging pod node-debugger-node-01-456sg with container debugger on node node-01.
If you don't see a command prompt, try pressing enter.
> cat /etc/hostname
node-01

This namespace re-labelling technique doesn’t only help get around admission control either. Ingress resources in Kubernetes often run in their own namespace, and the ingress controller needs to be able to reach services in the cluster, so it is common to see a network policy allowing this. Network policy can’t use a namespace name as a selector and until comparatively recently a namespace’s name wasn’t added as a label, so it isn’t unusual to see a policy like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-from-controller
  namespace: default  # Change to your target namespace
spec:
  podSelector: {}  # Applies to all pods in the namespace
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          function: ingress-control

This policy allows ingress from a namespace labelled as ingress-control, which by a platform admin’s design would only be applied to the dedicated ingress-control namespace. However, as we’ve demonstrated above, a tenant admin would be able to add this label to their own namespace and gain access to network-restricted workloads.

Controllers and Operators

We’ve also been able to abuse the CREATE verb on wildcard resources. Kubernetes is heavily reliant on the concept of a control loop, essentially having controllers which are constantly monitoring for the current and desired state. If they don’t match, actions are performed to make the states match again. This might be something relatively simple, like the pod controller making pods for a replicaset, or something much more involved. Basically, most of Kubernetes is a bunch of for loops in a trenchcoat trying to keep everything running.

As a highly specific situational example, OpenShift clusters ship with an Operator Hub, powered by the Operator Lifecycle Manager (OLM). This is like an app store for Kubernetes clusters, with a central controller monitoring for resources to be created.

A resource can be anything from a basic workload to an entire complex software installation, such as ArgoCD or ElasticSearch, which are made up of multiple separate Kubernetes components.

It’s Operators All The Way Down

A slightly simplified user flow looks something like this:

  1. The OLM operator watches for “subscriptions”, which are requests for an operator to be installed. Here, an operator is a piece of software from the Operator Hub
  2. When certain conditions are met, that operator is installed, either in a user namespace or in a system one
  3. That operator then monitors namespaces for a resource of a certain type
  4. When those resources are detected, the operator creates the additional resources to make up that piece of software

Ultimately, this means that we have components in the cluster which are susceptible to a Confused Deputy attack. While platform administrators may assume that operators are safe to install, this is not always the case.

The most damaging possibility with these permissions is that users can create an OperatorGroup in their own namespace, which would allow the OLM Operator to install any subscription in their own namespace, making use of that operator’s service account permissions. This was reported to the project in 2022, and was deemed to not be a security issue in the Operator Lifecycle Manager itself.

This is a shining example of a namespaced resource being considered privileged, with software authors stating that permissions should not be granted, but platform administrators not recognising a risk.

Without Permissions

So far, we’ve discussed what’s possible to a user with permissions, but what about a workload with no permissions? There are no cross-tenant boundaries which can be crossed there, right?

I added this section in the blog, so you know where this is going.

While a modern cluster is unlikely to simply allow anyone with network access the ability to compromise the entire cluster, it’s still generally possible to get information about other tenants in the same cluster. Kubernetes creates a DNS entry for every service in the cluster, by default in the format of servicename.namespace.svc.cluster.local. By brute-forcing the built-in DNS server with requests for service names associated with an IP address, it’s possible for any workload in a cluster to obtain a full list. Various tools exist to make this easier, including the wonderful jpts/coredns-enum.

Knowing a service exists doesn’t inherently break a tenancy model, as network policies can be used to restrict access between tenants, but it does introduce potential privacy concerns under some multi-tenancy models.

Conclusion

This is far from a comprehensive view into Kubernetes breakouts, and we haven’t even touched on non-access control considerations like node breakouts or container runtime vulnerabilities. RBAC is hard, and has a bunch of footguns which can be granted in the belief that they are safe. multi-tenancy is not simple, and the accidental application of a broad permission can result in a single tenant gaining full admin control over a multi-tenanted cluster.

The key takeaway for platform administrators: namespace boundaries are not security boundaries by default. Achieving true multi-tenancy requires defence in depth:

  • Carefully scoped RBAC with no wildcards
  • Admission control to supplement RBAC
  • Use of immutable labels, such as kubernetes.io/metadata.name where namespace selectors are used
  • Regular security audits and penetration testing
  • For high-security clusters where better isolation is a hard requirement, use of separate clusters in preference to a single multi-tenanted cluster

And finally, as much as it’s tempting to just grant all the permissions, remember the advice from Ian Coldwater.

We’re all made of stars, but your RBAC shouldn’t be

  1. For our purposes, a tenant can be anything from an internal dev team to a paying customer renting compute cycles. Either way, tenants aren’t meant to have access to each others’ stuff. ↩︎

  2. The simplest way to accomplish this is to use “The most pointless Kubernetes command ever” ↩︎

  3. Assuming your chosen Container Network Interface supports it ↩︎

  4. Which is normally not a user directly, and led to a good deal of confusion ↩︎

  5. Proof of concept here ↩︎