Beats, ECK, and Openshift 3.11

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi,

I started looking into deploying out the various beats based on the examples in the documentation to our OpenShift 3.11 cluster and have run into some issues. I wonder if there are quickstart examples for OpenShift given the block is generally a no-go:

    podTemplate:
      spec:
        securityContext:
          runAsUser: 0

I'm setting up heartbeat to get started, it seemed the simplest option and have not been able to get the pod rolling:

6m          3h           51        heartbeat-beat-heartbeat-667744cc.1627d1f5b16c4a1d     ReplicaSet               Warning   FailedCreate              replicaset-controller                Error creating: pods "heartbeat-beat-heartbeat-667744cc-" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000360000, 1000369999]]
4m          4m           8         heartbeat.1627dbf513377bbf                             Beat                     Warning   AssociationError          beat-controller                      Association backend for elasticsearch is not configured
4m          4m           1         heartbeat.1627dbf524a8af1b                             Beat                     Normal    AssociationStatusChange   beat-kibana-association-controller   Association status changed from [] to [Established]
4m          4m           1         heartbeat.1627dbf539331c72                             Beat                     Normal    AssociationStatusChange   beat-es-association-controller       Association status changed from [] to [Established]
4m          4m           1         heartbeat-beat-heartbeat.1627dbf53d7f0ba7              Deployment               Normal    ScalingReplicaSet         deployment-controller                Scaled up replica set heartbeat-beat-heartbeat-8964c6c44 to 1
2m          4m           15        heartbeat-beat-heartbeat-8964c6c44.1627dbf53e810d28    ReplicaSet               Warning   FailedCreate              replicaset-controller                Error creating: pods "heartbeat-beat-heartbeat-8964c6c44-" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used]

When I saw these event errors I ran:

oc adm policy add-scc-to-user hostaccess heartbeat
oc adm policy add-scc-to-user privileged heartbeat

Which lead to the logs:

4s          4s           1         heartbeat-beat-heartbeat.1627dc3081d5a1f7              Deployment               Normal    ScalingReplicaSet         deployment-controller                Scaled up replica set heartbeat-beat-heartbeat-7cfd79499d to 1
4s          4s           8         heartbeat.1627dc30531d642d                             Beat                     Warning   AssociationError          beat-controller                      Association backend for elasticsearch is not configured
4s          4s           1         heartbeat.1627dc3063753bd1                             Beat                     Normal    AssociationStatusChange   beat-kibana-association-controller   Association status changed from [] to [Established]
4s          4s           1         heartbeat.1627dc3079a61db7                             Beat                     Normal    AssociationStatusChange   beat-es-association-controller       Association status changed from [] to [Established]
1s          4s           10        heartbeat-beat-heartbeat-7cfd79499d.1627dc3082bff32d   ReplicaSet               Warning   FailedCreate              replicaset-controller                Error creating: pods "heartbeat-beat-heartbeat-7cfd79499d-" is forbidden: unable to validate against any security context constraint: [spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000360000, 1000369999]]

Not sure if I'm missing something. I didn't see anything in the github issues or the documentation for the beats about provisions for running on OpenShift, so any help would be great.

2

If heartbeat is a service account, and not a regular user, then you should use the -z flags:

oc adm policy add-scc-to-user privileged -z heartbeat

It also assumes that the service account has been set in the podTemplate:

    podTemplate:
      spec:
        serviceAccountName: heartbeat
        securityContext:
          runAsUser: 0
1 Like
3

@michael.morello Thank you for the reply!

Sorry, I did run the full servicename path in my command, just took it out for obscuring the namespace name, should've referenced that better. What I didn't do and didn't think to even check on, was whether the servicename was being made from the yaml (I added a similar section to the metricbeat yaml example), the addition to the podTemplate showed me the error that the servicename didn't exist - not sure why that didn't work. After creating the serviceaccount like the APM documentation for OpenShift, a pod started - unfortunately, it's in crashbackoffloop with the following error:

2020-08-04T13:26:48.357Z	INFO	instance/beat.go:647	Home path: [/usr/share/heartbeat] Config path: [/usr/share/heartbeat] Data path: [/usr/share/heartbeat/data] Logs path: [/usr/share/heartbeat/logs]
2020-08-04T13:26:48.357Z	INFO	instance/beat.go:390	heartbeat stopped.
2020-08-04T13:26:48.357Z	ERROR	instance/beat.go:958	Exiting: Failed to create Beat meta file: open /usr/share/heartbeat/data/meta.json.new: permission denied
Exiting: Failed to create Beat meta file: open /usr/share/heartbeat/data/meta.json.new: permission denied

Here's our full yaml:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: heartbeat
spec:
  type: heartbeat
  version: 7.8.0
  elasticsearchRef:
    name: observability-elasticsearch
  kibanaRef:
    name: observability-kibana
  config:
    heartbeat.monitors:
    - type: tcp
      schedule: '@every 5s'
      hosts: ["elasticsearch-observability-es-http.observability-namespace.svc:9200"]
    - type: tcp
      schedule: '@every 5s'
      hosts: ["observability-kibana-kb-http.observability-namespace.svc:5601"]
    - type: tcp
      schedule: '@every 5s'
      hosts: ["vendor-elasticsearch-es-http.app-namespace.svc:9200"]
    - type: tcp
      schedule: '@every 5s'
      hosts: ["vendor-kibana-kb-http.app-namespace.svc:5601"]
    - type: tcp
      schedule: '@every 5s'
      hosts: ["vendor-elasticsearch-staging-es-http.ingest-namespace.svc:9200"]
    - type: tcp
      schedule: '@every 5s'
      hosts: ["vendor-kibana-staging-kb-http.ingest-namespace.svc:5601"]
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccount: heartbeat
        serviceAccountName: heartbeat
        securityContext:
          runAsUser: 0
...

I ran these commands to create the serviceaccount:

 1134  oc create serviceaccount heartbeat -n observability-namespace
 1135  oc adm policy add-scc-to-user anyuid -z heartbeat -n observability-namespace
 1138  oc adm policy add-scc-to-user privileged -z heartbeat

Any thoughts?

Thanks for your time.

4

@michael.morello Thanks again for your time, I was able to get an answer to get heartbeat running on OpenShift 3.11 with ECK through the support portal that mapped to much of what I did plus an extra step of setting a SELinux label on the App Nodes. I'm going to post it below for people looking for answers and finding this thread.

In this example elastic is the name of the Namespace and beats the name of the ServiceAccount , replace those values by the desired ones

Beware that if SELinux is enabled, you may hit another issue, the Pod will fail with the following message:

> kubectl logs heartbeat-beat-heartbeat-5cdf958d86-n77f2 -n elastic -f
2020-08-04T06:14:41.519Z    INFO    instance/beat.go:647    Home path: [/usr/share/heartbeat] Config path: [/usr/share/heartbeat] Data path: [/usr/share/heartbeat/data] Logs path: [/usr/share/heartbeat/logs]
2020-08-04T06:14:41.519Z    INFO    instance/beat.go:390    heartbeat stopped.
2020-08-04T06:14:41.519Z    ERROR    instance/beat.go:958    Exiting: Failed to create Beat meta file: open /usr/share/heartbeat/data/meta.json.new: permission denied
Exiting: Failed to create Beat meta file: open /usr/share/heartbeat/data/meta.json.new: permission denied

In order to fix this error the label svirt_sandbox_file_t must be applied to the directory /var/lib/elastic/heartbeat/heartbeat-data/ :

chcon -Rt svirt_sandbox_file_t /var/lib/elastic/heartbeat/heartbeat-data/

This should be done on all the hosts where the heartbeat Pod can be deployed.