Trying to setup beats

knagasri · December 7, 2020, 4:57am

I had deployed Elasticsearch cluster along with kibana in GCP. I was trying to deploy beats as well. I had used the beats yaml files which are in the following link:

I just changed the elastic and kibana references. Everything remains the same as in that examples. For the first few minutes, beats pods are in running, later they ran into crashloop backoff or error state with the below error:

ERROR   instance/beat.go:951    Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/auditbeat/kibana: failed to import Kibana index pattern: 1 error: error loading index pattern: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/auditbeat/kibana: failed to import Kibana index pattern: 1 error: error loading index pattern: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

knagasri · December 7, 2020, 5:07pm

Any one answer this?
Elasticsearch health also becomes red

knagasri · December 17, 2020, 3:19am

I am directly using the sample examples that are given from the above link. Can anyone answer Do I need to change anything. Below is my file-beat config

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 7.9.2
  elasticsearchRef:
    name: elasticsearch-sample
  kibanaRef:
    name: kibana-sample
  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints:
            enabled: true
            default_config:
              type: container
              paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-operator
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

mtojek · December 17, 2020, 8:46am

It looks like HTTP 500 which should be present in logs. Did you check Elasticsearch or Kibana for any more details?

knagasri · December 24, 2020, 6:31am

When I see kibana logs, I found below error:

{"type":"log","@timestamp":"2020-12-24T06:25:54Z","tags":["info","plugins","security","authentication"],"pid":6,"message":"Authentication attempt failed: [security_exception] unable to authenticate user [es-upgrade-filebeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
{"type":"log","@timestamp":"2020-12-24T06:25:54Z","tags":["info","plugins","security","authentication"],"pid":6,"message":"Authentication attempt failed: [security_exception] unable to authenticate user [es-upgrade-auditbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
{"type":"response","@timestamp":"2020-12-24T06:25:54Z","tags":["api"],"pid":6,"method":"get","statusCode":401,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana-config-kb-http.es-upgrade.svc:5601","user-agent":"Go-http-client/1.1","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"10.90.0.119","userAgent":"10.90.0.119"},"res":{"statusCode":401,"responseTime":46,"contentLength":9},"message":"GET /api/status 401 46ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-24T06:25:54Z","tags":["api"],"pid":6,"method":"get","statusCode":401,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana-config-kb-http.es-upgrade.svc:5601","user-agent":"Go-http-client/1.1","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"10.90.0.40","userAgent":"10.90.0.40"},"res":{"statusCode":401,"responseTime":58,"contentLength":9},"message":"GET /api/status 401 58ms - 9.0B"}
{"type":"log","@timestamp":"2020-12-24T06:25:54Z","tags":["info","plugins","security","authentication"],"pid":6,"message":"Authentication attempt failed: [security_exception] unable to authenticate user [es-upgrade-packetbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
{"type":"log","@timestamp":"2020-12-24T06:25:54Z","tags":["info","plugins","security","authentication"],"pid":6,"message":"Authentication attempt failed: [security_exception] unable to authenticate user [es-upgrade-auditbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
{"type":"response","@timestamp":"2020-12-24T06:25:54Z","tags":["api"],"pid":6,"method":"get","statusCode":401,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana-config-kb-http.es-upgrade.svc:5601","user-agent":"Go-http-client/1.1","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"10.90.0.104","userAgent":"10.90.0.104"},"res":{"statusCode":401,"responseTime":42,"contentLength":9},"message":"GET /api/status 401 42ms - 9.0B"}
{"type":"response","@timestamp":"2020-12-24T06:25:54Z","tags":["api"],"pid":6,"method":"get","statusCode":401,"req":{"url":"/api/status","method":"get","headers":{"host":"kibana-config-kb-http.es-upgrade.svc:5601","user-agent":"Go-http-client/1.1","accept":"application/json","content-type":"application/json","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"10.90.0.134","userAgent":"10.90.0.134"},"res":{"statusCode":401,"responseTime":43,"contentLength":9},"message":"GET /api/status 401 43ms - 9.0B"}
{"type":"log","@timestamp":"2020-12-24T06:25:54Z","tags":["info","plugins","security","authentication"],"pid":6,"message":"Authentication attempt failed: [security_exception] unable to authenticate user [es-upgrade-auditbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}
{"type":"log","@timestamp":"2020-12-24T06:25:54Z","tags":["info","plugins","security","authentication"],"pid":6,"message":"Authentication attempt failed: [security_exception] unable to authenticate user [es-upgrade-filebeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\\"\" & 1=\"ApiKey\" & 2=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } }"}

knagasri · January 4, 2021, 5:16pm

Has anyone know why beats pods are in 3 or 4 restarts

 kubectl -n es-upgrade2 get po | grep packet
packetbeat-beat-packetbeat-2cvwp            1/1     Running     3          4h4m
packetbeat-beat-packetbeat-466t8            1/1     Running     3          4h4m
packetbeat-beat-packetbeat-4q5nx            1/1     Running     4          4h4m
packetbeat-beat-packetbeat-7x8fv            1/1     Running     3          4h4m
packetbeat-beat-packetbeat-98ck4            1/1     Running     3          4h4m
packetbeat-beat-packetbeat-98pl4            1/1     Running     3          4h4m
packetbeat-beat-packetbeat-992z6            1/1     Running     0          4h4m

logs of one of packet beat
kubectl -n es-upgrade2 logs packetbeat-beat-packetbeat-466t8

2021-01-04T17:13:21.339Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":33940,"time":{"ms":20}},"total":{"ticks":203130,"time":{"ms":100},"value":203130},"user":{"ticks":169190,"time":{"ms":80}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"fe687c57-5f12-4bea-9a6b-7707292eeaa2","uptime":{"ms":14553100}},"memstats":{"gc_next":118960976,"memory_alloc":84188360,"memory_total":23846088272},"runtime":{"goroutines":19}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":398,"batches":8,"total":398}},"pipeline":{"clients":3,"events":{"active":4117,"published":398,"total":398},"queue":{"acked":398}}},"system":{"load":{"1":5.24,"15":4.89,"5":4.89,"norm":{"1":0.0655,"15":0.0611,"5":0.0611}}}}}}
2021-01-04T17:13:51.338Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":34100,"time":{"ms":158}},"total":{"ticks":203940,"time":{"ms":810},"value":203940},"user":{"ticks":169840,"time":{"ms":652}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":10},"info":{"ephemeral_id":"fe687c57-5f12-4bea-9a6b-7707292eeaa2","uptime":{"ms":14583100}},"memstats":{"gc_next":103774448,"memory_alloc":55803608,"memory_total":23926411312},"runtime":{"goroutines":19}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":350,"batches":7,"total":350}},"pipeline":{"clients":3,"events":{"active":4117,"failed":21,"published":350,"total":371},"queue":{"acked":350}}},"system":{"load":{"1":4.95,"15":4.88,"5":4.85,"norm":{"1":0.0619,"15":0.061,"5":0.0606}}},"tcp":{"dropped_because_of_gaps":4}}}}

knagasri · January 4, 2021, 5:22pm

when I try to see the previous logs

kubectl -n es-upgrade2 logs packetbeat-beat-packetbeat-466t8 --previous
2021-01-04T13:10:10.136Z        INFO    kibana/client.go:119    Kibana url: https://kibana-config-kb-http.es-upgrade2.svc:5601
2021-01-04T13:10:10.334Z        INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":470,"time":{"ms":476}},"total":{"ticks":870,"time":{"ms":881},"value":870},"user":{"ticks":400,"time":{"ms":405}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"9b096216-30d2-4925-a736-8c9d03cba656","uptime":{"ms":1846}},"memstats":{"gc_next":41963280,"memory_alloc":35809304,"memory_total":38786080,"rss":84750336},"runtime":{"goroutines":15}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":3,"events":{"active":0}}},"system":{"cpu":{"cores":80},"load":{"1":5.05,"15":6.01,"5":5.26,"norm":{"1":0.0631,"15":0.0751,"5":0.0658}}}}}}
2021-01-04T13:10:10.334Z        INFO    [monitoring]    log/log.go:154  Uptime: 1.896042773s
2021-01-04T13:10:10.334Z        INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2021-01-04T13:10:10.334Z        INFO    instance/beat.go:447    packetbeat stopped.
2021-01-04T13:10:10.334Z        ERROR   instance/beat.go:951    Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-config-kb-http.es-upgrade2.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [es-upgrade2-packetbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\... (truncated).
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana-config-kb-http.es-upgrade2.svc:5601/api/status fails: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"[security_exception] unable to authenticate user [es-upgrade2-packetbeat-beat-kb-user] for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0=\"Bearer realm=\\\"security\\... (truncated).

I can see the same in audit and remaining beats as well.

system · February 1, 2021, 7:22pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.