Support for ECK with a PSP

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I am trying to use ECK with PSP enabled in my self-hosted kubernetes cluster. It does not seem to work, and it has a weird deployment method of using a stateful set to deploy additional privileged containers.

So, what permissions should I give for ECK to work and to which service account?

There seems to be an issue open for this. Would prefer to have a sample PSP shipped with the operator as this would make sense for users who enable PSP by default during installation!

The existing PSP used are mentioned on the Github issue. I finally had to allow all serviceaccounts to create a privileged pod in the Elasticsearch namespace which was definitely an overkill and a security risk. Am I missing something here?

2

Hi @NerdSec, thanks for moving to Discuss.

When I glanced over Calico I was under an impression that it's a k8s distribution and I worded my response like it mattered. Apologies!

To reiterate - ECK operator does not create Pods directly (privileged or not). Instead, we create a StatefulSet resource. There is a built-in StatefulSet controller in k8s that creates Pods based on that StatefulSet resource. This is important, because it means that elastic-operator is not the right Service Account to give use permission for a PSP.

So what's the right way? Well, there are two:

  • give the use permission to the StatefulSet controller
  • have the Pods use serviceAccount that has the right use permission

As you correctly noted, the first one is more permissive than needed. I suggested that one as I felt it can help unblocking and verifying if that's really the issue. I think that my response might've lacked context :slight_smile:

The second way is preferred, but requires to create a Service Account, the right Binding and providing it in the Elasticsearch manifest podTemplate.

Our E2E test suite runs on PSP enabled clusters and the PSP is in our repo, if you'd like to take a look.

Btw, where you able to check kubernetes.io/psp annotation on the Pod? It should tell you which PSP was the Pod created with.

Let me know if the above allowed you to unblock, I'll be happy to help further if not.

3

Hi David,

So, yes, by default I have allowed all authenticated users to use the z-restricted PSP. This is stopping the init container from being created.

image
image1550×355 23.5 KB

Adding a more permissive use permission for all SA in the elasticsearch namespace allows the pods to use the z-privileged policy, and thereby the containers get created.

I will try using an appropriate role binding and provide it in the podTemplate. Thanks for clarifying this up!

However, I remember trying out the BETA ECK without having to make any such changes, did something change? Why does the init container need to run as a root user with access to a writable rootfs?

4

So, I have not created a cluster role as such, I simply allowed the default serviceaccount the use permission for my z-privileged PSP. The deployment happened and I can see the pods use the privileged PSP.

The PSP you shared, is similar to the restricted PSP available on my cluster. Using that PSP, the pod is not allowed to load.

Labels:       common.k8s.elastic.co/type=elasticsearch
              controller-revision-hash=brb-es-default-fffd7cb5b
              elasticsearch.k8s.elastic.co/cluster-name=brb
              elasticsearch.k8s.elastic.co/config-hash=1841883488
              elasticsearch.k8s.elastic.co/http-scheme=https
              elasticsearch.k8s.elastic.co/node-data=true
              elasticsearch.k8s.elastic.co/node-ingest=true
              elasticsearch.k8s.elastic.co/node-master=true
              elasticsearch.k8s.elastic.co/node-ml=true
              elasticsearch.k8s.elastic.co/statefulset-name=brb-es-default
              elasticsearch.k8s.elastic.co/version=7.9.1
              statefulset.kubernetes.io/pod-name=brb-es-default-0
Annotations:  cni.projectcalico.org/podIP: 192.168.193.59/32
              cni.projectcalico.org/podIPs: 192.168.193.59/32
              co.elastic.logs/module: elasticsearch
              container.apparmor.security.beta.kubernetes.io/elastic-internal-init-filesystem: runtime/default
              container.apparmor.security.beta.kubernetes.io/elasticsearch: runtime/default
              kubernetes.io/psp: elastic.restricted
              seccomp.security.alpha.kubernetes.io/pod: runtime/default

Events:

Events:
  Type     Reason     Age                            From                    Message
  ----     ------     ----                           ----                    -------
  Normal   Scheduled  30s                            default-scheduler       Successfully assigned elastic/cdc-es-default-0 to sidcirmkube02
  Normal   Pulled     <invalid>                      kubelet, sidcirmkube02  Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:7.9.1" in 4.951464499s
  Normal   Pulled     <invalid>                      kubelet, sidcirmkube02  Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:7.9.1" in 3.733738989s
  Normal   Pulling    <invalid> (x3 over <invalid>)  kubelet, sidcirmkube02  Pulling image "docker.elastic.co/elasticsearch/elasticsearch:7.9.1"
  Warning  Failed     <invalid> (x3 over <invalid>)  kubelet, sidcirmkube02  Error: container has runAsNonRoot and image will run as root
  Normal   Pulled     <invalid>                      kubelet, sidcirmkube02  Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:7.9.1" in 3.52929003s

Is there a different image being used over here in the E2E environment?

5

As you mentioned correctly, had to set the securityContext in the podTemplate. After setting the Context it worked absolutely fine.

For someone who wants to do this, here are the sample yamls:

# Elasticsearch
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: cdc
spec:
  version: 7.9.1
  nodeSets:
  - name: default
    count: 3
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: longhorn
    podTemplate:
      spec:
        securityContext:
          fsGroup: 1000
          runAsUser: 1000
          runAsGroup: 1000

# Kibana
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: cdc
spec:
  version: 7.9.1
  count: 1
  elasticsearchRef:
    name: cdc
  http:
    service:
      spec:
        ports:
        - protocol: TCP
          port: 443
          targetPort: 5601
        type: LoadBalancer
        loadBalancerIP: 10.31.24.22
    tls:
      selfSignedCertificate:
        subjectAltNames:
        - ip: 10.31.24.22
        - dns: kibana.xio.com
  podTemplate:
    spec:
      securityContext:
        fsGroup: 1000
        runAsUser: 1000
        runAsGroup: 1000

@dkow Thanks for the inputs! By any chance can we add this to the ECK documentation? This would be easier for someone like me who is new to Kubernetes & ECK to start off in a much more secure manner.

Also, why is this not included in the ECK operator? Shouldn't this be like the default setting? The way I see it, specifying the USER in the Dockerfile could also solve this?