How to send logs to logstash via filebeat

Elastic Stack Beats
1

I am using the below code for deploying file beat in GCP:

I can see 3 or 4 restarts . After then , it was getting stable. Can anyone know why it is getting 3 or 4 restarts and also by default it is sending logs to elasticsearch. How can I configure so that it can send collected logs to logstash.
I just commented elasticsearch reference and added output as logstash

Below is my file-beat yaml file:

apiVersion: beat.k8s.elastic.co/v1beta1
 kind: Beat
 metadata:
  name: filebeat
 spec:
  type: filebeat
  version: 7.9.2
  #elasticsearchRef:
  #  name: elasticsearch-config
  kibanaRef:
    name: kibana-config
  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints:
            enabled: true
            default_config:
              type: container
              paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
    output.logstash:   # ------> to send logs to logstash
      hosts: ['${LOGSTASH_HOST:logstash}:${LOGSTASH_PORT:5044}']    
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: log-filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          # specify resource limits and requests
          resources:
            requests:
              memory: 1Gi
              cpu: 1
            limits:
              memory: 4Gi
              cpu: 2
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
     ```
2

It's not clear what you mean by that sorry.

3

@warkolm, I was talking about these below restarts

k8soptr@jumpbox:~$ kubectl -n es-upgrade2 get po | grep file
filebeat-beat-filebeat-2f8lg                1/1     Running     0          21h
filebeat-beat-filebeat-6h2m8                1/1     Running     1          22h
filebeat-beat-filebeat-7zgwg                1/1     Running     6          21h
filebeat-beat-filebeat-b7wsj                1/1     Running     1          22h

logs of one of the above pod

kubectl -n es-upgrade2 logs filebeat-beat-filebeat-7zgwg
2021-01-07T05:25:32.821Z        ERROR   [autodiscover]  cfgfile/list.go:95      Error creating runner from config: Can only start an input when all related states are finished: {Id: native::6017442-2049, Finished: false, Fileinfo: &{a683d27f949f27346c5174f23b3ea196940011b4c932219c888505298ae3b4ed-json.log 2563844 416 {494329726 63745592993 0x607ff60} {2049 6017442 1 33184 0 0 0 0 2563844 4096 5016 {1609996194 549329282} {1609996193 494329726} {1609996193 494329726} [0 0 0]}}, Source: /var/lib/docker/containers/a683d27f949f27346c5174f23b3ea196940011b4c932219c888505298ae3b4ed/a683d27f949f27346c5174f23b3ea196940011b4c932219c888505298ae3b4ed-json.log, Offset: 621624, Timestamp: 2021-01-07 05:25:24.980505643 +0000 UTC m=+4641.359445773, TTL: -1ns, Type: container, Meta: map[], FileStateOS: 6017442-2049}
2021-01-07T05:25:32.821Z        INFO    log/harvester.go:299    Harvester started for file: /var/lib/docker/containers/e084f5884e485b349013de789c4a49d23f76a5be027ed3b8c9efa28a276af4af/e084f5884e485b349013de789c4a49d23f76a5be027ed3b8c9efa28a276af4af-json.log
2021-01-07T05:25:32.823Z        INFO    log/input.go:157        Configured paths: [/var/lib/docker/containers/3c2919d24ea162665093e005d97c1199c77237c3f957b42104269d8e79245d1d/*-json.log]
2021-01-07T05:25:32.823Z        INFO    log/harvester.go:299    Harvester started for file: /var/lib/docker/containers/3c2919d24ea162665093e005d97c1199c77237c3f957b42104269d8e79245d1d/3c2919d24ea162665093e005d97c1199c77237c3f957b42104269d8e79245d1d-json.log
2021-01-07T05:25:35.734Z        INFO    log/harvester.go:322    File was removed: /var/lib/docker/containers/eab4e70bebaa7391e08ef60c3ff32ae91a04ae2c258b44a4be2d98d679f8ee06/eab4e70bebaa7391e08ef60c3ff32ae91a04ae2c258b44a4be2d98d679f8ee06-json.log. Closing because close_removed is enabled.
2021-01-07T05:25:41.578Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":101460,"time":{"ms":594}},"total":{"ticks":579890,"time":{"ms":4278},"value":579890},"user":{"ticks":478430,"time":{"ms":3684}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":64},"info":{"ephemeral_id":"35a7c355-25d8-4f0f-9b02-b395027810bb","uptime":{"ms":4655590}},"memstats":{"gc_next":135317968,"memory_alloc":67991288,"memory_total":49338528800},"runtime":{"goroutines":2585}},"filebeat":{"events":{"active":13,"added":271,"done":258},"harvester":{"closed":7,"files":{"003a7b2c-8873-4543-9d8d-4950a2292f2b":{"last_event_published_time":"2021-01-07T05:25:36.696Z","last_event_timestamp":"2021-01-07T05:25:31.912Z","read_offset":4041,"size":2508},"05253dc6-e3eb-43f8-bc72-c230d409c0cf":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:31.918Z","read_offset":1767,"size":1622},"05e683c2-9017-4fa3-9223-0f005b8895c9":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:31.914Z","read_offset":962,"size":482},"06a2fda4-90ea-4170-8a80-196d8edc07cc":{"last_event_published_time":"2021-01-07T05:25:15.492Z","last_event_timestamp":"2021-01-07T05:25:05.337Z","read_offset":482,"size":482},"0942b6a7-e9eb-4153-b674-d20591847638":{"last_event_published_time":"2021-01-07T05:25:20.476Z","last_event_timestamp":"2021-01-07T04:04:55.099Z","read_offset":208},"0fad1ff2-62b0-407f-a7cc-eca9db90735f":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T04:52:47.779Z","read_offset":5028,"size":1712},"1080c11c-78cb-431a-8412-08975ddb7ec9":{"last_event_published_time":"2021-01-07T05:25:36.697Z","read_offset":2250,"size":3465},"1a9beb44-f494-400a-86d9-24918874de68":{"last_event_published_time":"2021-01-07T05:25:20.476Z","last_event_timestamp":"2021-01-07T05:25:09.232Z","read_offset":1056,"size":1056},"278345f7-4a62-469a-9594-59d845349cbe":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:16:07.463Z","name":"/var/lib/docker/containers/3c2919d24ea162665093e005d97c1199c77237c3f957b42104269d8e79245d1d/3c2919d24ea162665093e005d97c1199c77237c3f957b42104269d8e79245d1d-json.log","read_offset":1473951,"size":1474085,"start_time":"2021-01-07T05:25:32.823Z"},"2a6dde66-67ab-421d-855b-425a79ffaa6d":{"last_event_published_time":"2021-01-07T05:25:20.476Z","last_event_timestamp":"2021-01-07T05:25:15.316Z","read_offset":895,"size":895},"2d5f3d29-4a1d-468c-8722-a9c921439118":{"last_event_published_time":"2021-01-07T05:25:25.748Z","last_event_timestamp":"2021-01-07T04:44:33.702Z","read_offset":607},"33a741fb-8072-4480-a488-49b94837cd58":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T03:04:39.430Z","read_offset":3494},"36fa715c-fb8b-4675-a464-75781fe4bdf5":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:26.309Z","read_offset":3418,"size":5046},"3f781899-f0a1-46a2-9a29-e36684501db1":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:28.871Z","read_offset":836,"size":1045},"4ab768f6-433f-478e-99b5-6cf5928c2216":{"last_event_published_time":"2021-01-07T05:25:24.594Z","read_offset":204,"size":386324},"55197f8e-b0bc-4ce1-9014-da7eacf00e74":{"last_event_published_time":"2021-01-07T05:25:20.476Z","last_event_timestamp":"2021-01-07T05:25:03.042Z","read_offset":1135,"size":482},"56757473-798d-483e-a8ff-dd8b56c0b307":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T04:35:44.621Z","read_offset":10308},"57905689-8144-4144-a38a-597fdad8879f":{"last_event_published_time":"2021-01-07T05:25:20.142Z","last_event_timestamp":"2021-01-07T05:25:05.337Z","read_offset":482,"size":482},"5bab16b9-ad44-450c-9cd3-62f30eacd07e":{"last_event_published_time":"2021-01-07T05:25:15.492Z","read_offset":201},"5cde14ea-d08f-4208-8035-38ac18523884":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:26.119Z","read_offset":1054,"size":1037},"5d9fdd76-50d9-44ff-9e2b-4cdbfe3b69b8":{"last_event_published_time":"2021-01-07T05:25:24.594Z","last_event_timestamp":"2021-01-07T05:25:19.518Z","name":"/var/lib/docker/containers/0d769cacd1823b63da9a1f9561ce2575751ed967dc075fb9a650a00edafeceb5/0d769cacd1823b63da9a1f9561ce2575751ed967dc075fb9a650a00edafeceb5-json.log","read_offset":690,"size":690,"start_time":"2021-01-07T05:25:22.817Z"},"632bc054-45b1-4f9a-99c1-78e05e917ec2":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:25.897Z","read_offset":251},"6b535788-f60f-4095-8b8e-b9ac6ad71bd0":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T04:16:28.521Z","read_offset":394},"7280ff9e-4305-4416-b8c8-9d02c888ff4c":{"last_event_published_time":"2021-01-07T05:25:15.492Z","read_offset":200},"728ce0c0-e892-4c31-8e74-5e765b098de6":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T04:38:35.094Z","read_offset":13295,"size":17497},"75021a73-c71b-4d0b-af25-6ee80cbfa892":{"last_event_published_time":"2021-01-07T05:25:25.748Z","last_event_timestamp":"2021-01-07T04:02:36.640Z","read_offset":599},"7594ae58-e9d9-4e51-93e8-2492e634475f":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:04:23.884Z","read_offset":1407,"size":277057},"7916e25d-8f46-41c4-911b-d2b7de7cf87d":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:29.025Z","read_offset":558,"size":419},"7b92e773-f727-48d9-9906-a10271ef51cb":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T02:45:39.918Z","read_offset":6621,"size":3473},"9040ba55-227e-473e-b0f1-b193d54a1d5a":{"last_event_published_time":"2021-01-07T05:25:20.476Z","last_event_timestamp":"2021-01-07T05:25:18.726Z","read_offset":2591,"size":650},"97f5c9ec-5bf9-470e-adde-f23ac819f3ca":{"last_event_published_time":"2021-01-07T05:25:15.492Z","last_event_timestamp":"2021-01-07T05:11:50.317Z","read_offset":199},"9c270dd7-22f2-4c5c-8714-5aafad5926d5":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:16:15.044Z","name":"/var/lib/docker/containers/e084f5884e485b349013de789c4a49d23f76a5be027ed3b8c9efa28a276af4af/e084f5884e485b349013de789c4a49d23f76a5be027ed3b8c9efa28a276af4af-json.log","read_offset":509580,"size":510814,"start_time":"2021-01-07T05:25:32.821Z"},"a5f646a4-ba91-4468-9e66-fc8bc456d3e1":{"last_event_published_time":"2021-01-07T05:25:25.748Z","last_event_timestamp":"2021-01-07T05:01:23.056Z","read_offset":208},"a8c7f4c8-09c6-4d14-93ae-67c9f2068a79":{"last_event_published_time":"2021-01-07T05:25:15.492Z","last_event_timestamp":"2021-01-07T05:25:06.800Z","read_offset":1673,"size":2415},"b0fdbd1a-bf6c-49df-9cb1-76f69ee944cf":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T04:27:45.566Z","read_offset":1123,"size":11634},"b4a9a3e3-c112-477d-bb3b-57b307cade58":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T03:33:13.001Z","read_offset":4438,"size":7764},"b8e98396-a0da-41aa-9af7-2f7e6d58f8f0":{"last_event_published_time":"2021-01-07T05:25:20.476Z","last_event_timestamp":"2021-01-07T05:25:12.894Z","read_offset":1126,"size":1054},"b8f997fe-1724-4ab7-8a00-b608833937a8":{"last_event_published_time":"2021-01-07T05:25:15.492Z","last_event_timestamp":"2021-01-07T05:25:05.337Z","read_offset":482,"size":482},"c5bc45ec-b5dc-4c36-a4c0-750e1320c31f":{"last_event_published_time":"2021-01-07T05:25:25.748Z","last_event_timestamp":"2021-01-07T05:25:20.474Z","name":"/var/lib/docker/containers/a693776f724f7ad5829f572424e9d663d5894f427347e70b8e6b25150a78ba6c/a693776f724f7ad5829f572424e9d663d5894f427347e70b8e6b25150a78ba6c-json.log","read_offset":691,"size":691,"start_time":"2021-01-07T05:25:21.895Z"},"c7c0f2c4-6385-421a-8895-e84015c672d0":{"last_event_published_time":"2021-01-07T05:25:25.748Z","last_event_timestamp":"2021-01-07T03:53:34.879Z","read_offset":2420},"cc237d32-4b7a-42be-bae6-f917dfdd9c73":{"last_event_published_time":"2021-01-07T05:25:15.492Z","read_offset":195},"cf6478b8-8bce-4bde-83ed-855f8a15db7b":{"last_event_published_time":"2021-01-07T05:25:25.748Z","last_event_timestamp":"2021-01-07T04:23:03.132Z","read_offset":397},"d8024af4-588b-4091-bdd1-174b11cd4bd5":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:24.372Z","read_offset":230},"dccb79d0-04c8-4c66-966c-6ad4890f19a2":{"last_event_published_time":"2021-01-07T05:25:15.492Z","last_event_timestamp":"2021-01-07T05:25:05.337Z","read_offset":482,"size":482},"e70e2629-d4d7-4edc-abb3-13765a6356a1":{"last_event_published_time":"2021-01-07T05:25:36.697Z","last_event_timestamp":"2021-01-07T05:25:31.021Z","read_offset":1830,"size":912},"eb7a41b6-0f56-43e3-a280-b6138e5dd63d":{"last_event_published_time":"2021-01-07T05:25:36.696Z","last_event_timestamp":"2021-01-07T05:25:24.406Z","name":"/var/lib/docker/containers/ff098df1b347a4652078fe58e8d68bb1285a7f3ed8c24219ff4cd88556f43039/ff098df1b347a4652078fe58e8d68bb1285a7f3ed8c24219ff4cd88556f43039-json.log","read_offset":1596122,"size":1595875,"start_time":"2021-01-07T05:25:22.806Z"},"f01a28ec-601f-4eef-9e54-33f4c7de7fbf":{"last_event_published_time":"2021-01-07T05:25:25.748Z","last_event_timestamp":"2021-01-07T05:07:24.579Z","read_offset":1626,"size":127040}},"open_files":53,"running":53,"started":7}},"libbeat":{"config":{"module":{"running":74,"starts":8,"stops":6}},"output":{"events":{"acked":250,"batches":5,"total":250}},"pipeline":{"clients":94,"events":{"active":4136,"filtered":21,"published":250,"total":271},"queue":{"acked":250}}},"registrar":{"states":{"current":1128,"update":258},"writes":{"success":85,"total":85}},"system":{"load":{"1":103.35,"15":105.01,"5":106.5,"norm":{"1":1.2919,"15":1.3126,"5":1.3313}}}}}}
2021-01-07T05:25:41.992Z        INFO    log/harvester.go:299    Harvester started for file: /var/lib/docker/containers/7385c705ee3b3560adeb8349d917ef9191c975437520379af4e4106762f0172a/7385c705ee3b3560adeb8349d917ef9191c975437520379af4e4106762f0172a-json.log
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.