Filebeat doesn't send logs to Elasticsearch

Pawel_Skadorwa · December 21, 2020, 7:55am

Hi,
I decided to talk to you because I have no idea how to resolve the problem with the logging stack. On kubernetes 1.19 I installed elasticsearch 7.10.1, kibana, metricbeat and filebeat in the same version. Connection between kibana and elasticsearch is established. Metrics beat can also communicate with elasticsearch and sends logs. But in case of filebeat there is a strange problem. The service starts there is no error in logs but neither doesn't create a new index nor sends data. Any ideas? I have tested many different configurtions. I wanted also to send logs to logstash but no way.

filebeat.yaml looks like this:

    ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-logging
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      enabled: true
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      #username: ${ELASTICSEARCH_USERNAME}
      #password: ${ELASTICSEARCH_PASSWORD}
    #output.logstash:
      #enabled: true
      #hosts: ["logstash-service:5044"]
      #username: ${ELASTICSEARCH_USERNAME}
      #password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-logging
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.10.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        #- name: ELASTICSEARCH_USERNAME
        #  value: elastic
        #- name: ELASTICSEARCH_PASSWORD
        #  value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-logging
  labels:
    k8s-app: filebeat
---

And the elasticservice looks like this:

    kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: kube-logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

I would be grateful if you could help me
rgds

warkolm · December 21, 2020, 10:04pm

Welcome to our community!

What do your Filebeat logs show?

Pawel_Skadorwa · December 22, 2020, 7:06am

Pleasure to join the community
The filebeat logs:

    kubectl logs filebeat-rb77x -n kube-logging
2020-12-22T07:03:35.827Z	INFO	instance/beat.go:647	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-12-22T07:03:35.828Z	INFO	instance/beat.go:655	Beat ID: 74238ba0-6ecc-4f90-b366-0ba2e58ddc56
2020-12-22T07:03:35.830Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-12-22T07:03:35.830Z	INFO	[beat]	instance/beat.go:983	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "74238ba0-6ecc-4f90-b366-0ba2e58ddc56"}}}
2020-12-22T07:03:35.830Z	INFO	[beat]	instance/beat.go:992	Build info	{"system_info": {"build": {"commit": "94f7632be5d56a7928595da79f4b829ffe123744", "libbeat": "7.8.1", "time": "2020-07-21T15:12:45.000Z", "version": "7.8.1"}}}
2020-12-22T07:03:35.830Z	INFO	[beat]	instance/beat.go:995	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":6,"version":"go1.13.10"}}}
2020-12-22T07:03:35.831Z	INFO	[beat]	instance/beat.go:999	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-12-21T12:55:21Z","containerized":true,"name":"kubernetes-worker-dev.local","ip":["127.0.0.1/8","::1/128","192.168.10.21/24","fe80::e644:3457:518d:b197/64","172.17.0.1/16","10.244.1.0/32","fe80::3c7f:ecff:fe5b:223a/64","10.244.1.1/24","fe80::d0e9:ebff:fe5d:b94c/64","fe80::4c26:2bff:fe0e:43bb/64","fe80::14e7:cdff:fe94:9053/64","fe80::e0ec:70ff:fe1a:d824/64","fe80::2c32:2ff:fe3f:e95a/64","fe80::d881:f1ff:fe11:42e2/64","fe80::b4ae:53ff:fe97:2a22/64","fe80::5c06:31ff:fee2:d55c/64","fe80::1856:bfff:fe00:5c78/64","fe80::e40a:62ff:fe70:2561/64","fe80::cced:78ff:fe0d:ec8e/64","fe80::881d:fff:fe50:73f0/64","fe80::bc7f:fdff:fe83:6f17/64","fe80::3408:fff:fe92:af68/64","fe80::40b0:38ff:fe17:b2ac/64"],"kernel_version":"3.10.0-1160.6.1.el7.x86_64","mac":["00:0c:29:34:f5:2e","02:42:64:05:5e:3f","3e:7f:ec:5b:22:3a","d2:e9:eb:5d:b9:4c","4e:26:2b:0e:43:bb","16:e7:cd:94:90:53","e2:ec:70:1a:d8:24","2e:32:02:3f:e9:5a","da:81:f1:11:42:e2","b6:ae:53:97:2a:22","5e:06:31:e2:d5:5c","1a:56:bf:00:5c:78","e6:0a:62:70:25:61","ce:ed:78:0d:ec:8e","8a:1d:0f:50:73:f0","be:7f:fd:83:6f:17","36:08:0f:92:af:68","42:b0:38:17:b2:ac"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":8,"patch":2003,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"1a018e03a49f4bfc904c69b0d6c08959"}}}
2020-12-22T07:03:35.832Z	INFO	[beat]	instance/beat.go:1028	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-12-22T07:03:34.730Z"}}}
2020-12-22T07:03:35.832Z	INFO	instance/beat.go:310	Setup Beat: filebeat; Version: 7.8.1
2020-12-22T07:03:35.832Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.8.1' as ILM is enabled.
2020-12-22T07:03:35.832Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://elasticsearch:9200
2020-12-22T07:03:35.832Z	INFO	[publisher]	pipeline/module.go:113	Beat name: kubernetes-worker-dev.local
2020-12-22T07:03:35.833Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-12-22T07:03:35.833Z	INFO	instance/beat.go:463	filebeat start running.
2020-12-22T07:03:35.833Z	INFO	registrar/migrate.go:104	No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-12-22T07:03:35.833Z	INFO	registrar/migrate.go:112	Initialize registry meta file
2020-12-22T07:03:35.834Z	INFO	registrar/registrar.go:108	No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-12-22T07:03:35.834Z	INFO	registrar/registrar.go:145	Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-12-22T07:03:35.834Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 0
2020-12-22T07:03:35.834Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2020-12-22T07:03:35.834Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2020-12-22T07:03:35.836Z	INFO	[autodiscover.pod]	kubernetes/util.go:79	kubernetes: Using node kubernetes-worker-dev.local provided in the config
2020-12-22T07:03:35.836Z	INFO	[autodiscover]	autodiscover/autodiscover.go:113	Starting autodiscover manager
2020-12-22T07:03:35.937Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*715c3a6b1a7df41da4ab0b0f0f84166cd73d198b253f3f0c61f188546af3f90a.log]
2020-12-22T07:03:35.937Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*715c3a6b1a7df41da4ab0b0f0f84166cd73d198b253f3f0c61f188546af3f90a.log]
2020-12-22T07:03:35.937Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*dea2f12f8dbc227d17d166e24a4c3970ba4e03db0da6bf71e17ce0cb9448140c.log]
2020-12-22T07:03:35.938Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*dea2f12f8dbc227d17d166e24a4c3970ba4e03db0da6bf71e17ce0cb9448140c.log]
2020-12-22T07:03:35.938Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*2c832233c882225d97f98f02abd031462b6452b58462f877ebb0656bd2b2bd5f.log]
2020-12-22T07:03:35.938Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*2c832233c882225d97f98f02abd031462b6452b58462f877ebb0656bd2b2bd5f.log]
2020-12-22T07:03:35.938Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*2c832233c882225d97f98f02abd031462b6452b58462f877ebb0656bd2b2bd5f.log]
2020-12-22T07:03:35.938Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*2c832233c882225d97f98f02abd031462b6452b58462f877ebb0656bd2b2bd5f.log]
2020-12-22T07:03:35.939Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*ead7722a79b90d7768802deb0b7ab0463c06f3502cde632197e4be0b46fa1a45.log]
2020-12-22T07:03:35.939Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*ead7722a79b90d7768802deb0b7ab0463c06f3502cde632197e4be0b46fa1a45.log]
2020-12-22T07:03:35.939Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*fbdea31de106d63fe7d915ccbdabd14ff8a88810c486e01b631d44f3dc0e496c.log]
2020-12-22T07:03:35.939Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*fbdea31de106d63fe7d915ccbdabd14ff8a88810c486e01b631d44f3dc0e496c.log]
2020-12-22T07:03:35.940Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*fbdea31de106d63fe7d915ccbdabd14ff8a88810c486e01b631d44f3dc0e496c.log]
2020-12-22T07:03:35.940Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*7893516869d23064b5f580231f937b4cba02f45df04e336cac71a080f1b98e0e.log]
2020-12-22T07:03:35.940Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*7893516869d23064b5f580231f937b4cba02f45df04e336cac71a080f1b98e0e.log]
2020-12-22T07:03:35.940Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*07511132decfbb0c466474de217ae0552c577ac0101f1e5af531757d7f6b2703.log]
2020-12-22T07:03:35.941Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*07511132decfbb0c466474de217ae0552c577ac0101f1e5af531757d7f6b2703.log]
2020-12-22T07:03:35.941Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*c323dc1dbd40e442c73f45e048222522e705cc0bb4f2594da81559af2f30497f.log]
2020-12-22T07:03:35.941Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*c323dc1dbd40e442c73f45e048222522e705cc0bb4f2594da81559af2f30497f.log]
2020-12-22T07:03:35.941Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*88221787224f375135aae4d3e0e33f0f894742e895fc596657cf5f2410eec125.log]
2020-12-22T07:03:35.941Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*88221787224f375135aae4d3e0e33f0f894742e895fc596657cf5f2410eec125.log]
2020-12-22T07:03:35.942Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*f4693cb024e6a26eecca0471ecbbf95d0a7ce73b2154d6623dda3d6315849121.log]
2020-12-22T07:03:35.942Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*f4693cb024e6a26eecca0471ecbbf95d0a7ce73b2154d6623dda3d6315849121.log]
2020-12-22T07:03:35.942Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*9235288713ce27626d383013810d293cfdee87cc967e5d5ffd9fdd9be6566917.log]
2020-12-22T07:03:35.942Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*9235288713ce27626d383013810d293cfdee87cc967e5d5ffd9fdd9be6566917.log]
2020-12-22T07:03:35.943Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*118e0bd40ebde82c379fac1f47513c7b076f782d6458ac05f63ee154ae299774.log]
2020-12-22T07:03:35.943Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*118e0bd40ebde82c379fac1f47513c7b076f782d6458ac05f63ee154ae299774.log]
2020-12-22T07:03:35.943Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*81839137a231a13c0ab633b5f200084f6e1b8c9811cc7a4a9aa62f0246a15d74.log]
2020-12-22T07:03:35.943Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*81839137a231a13c0ab633b5f200084f6e1b8c9811cc7a4a9aa62f0246a15d74.log]
2020-12-22T07:03:35.944Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*5381958c2024ba7f000dbbbf1bc106445f2839e8845ff62297df7e4cc40f596e.log]
2020-12-22T07:03:35.944Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*5381958c2024ba7f000dbbbf1bc106445f2839e8845ff62297df7e4cc40f596e.log]
2020-12-22T07:03:35.944Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*5381958c2024ba7f000dbbbf1bc106445f2839e8845ff62297df7e4cc40f596e.log]
2020-12-22T07:03:35.944Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*5a8655dece3bbd22dcac6d665cff8134696b6b503f984308bcca92c99a656267.log]
2020-12-22T07:03:35.945Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*5a8655dece3bbd22dcac6d665cff8134696b6b503f984308bcca92c99a656267.log]
2020-12-22T07:03:35.945Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*5a8655dece3bbd22dcac6d665cff8134696b6b503f984308bcca92c99a656267.log]
2020-12-22T07:03:35.945Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*027c421ad322a72e2371009e8f08c2379abca5905a380a6e704f063f5d9f4636.log]
2020-12-22T07:03:35.945Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*027c421ad322a72e2371009e8f08c2379abca5905a380a6e704f063f5d9f4636.log]
2020-12-22T07:03:35.946Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*027c421ad322a72e2371009e8f08c2379abca5905a380a6e704f063f5d9f4636.log]
2020-12-22T07:03:35.946Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*da69439c5c2716682c082a44c969a94f07cf3ad9c01afe2bc506df694950a577.log]
2020-12-22T07:03:35.946Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*da69439c5c2716682c082a44c969a94f07cf3ad9c01afe2bc506df694950a577.log]
2020-12-22T07:03:35.978Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*a2a5f33b9ca43d1106e159a60ba7a064a93e9cdfb05579218fc3f3f17cfb8a67.log]
2020-12-22T07:03:35.978Z	INFO	log/input.go:152	Configured paths: [/var/log/containers/*a2a5f33b9ca43d1106e159a60ba7a064a93e9cdfb05579218fc3f3f17cfb8a67.log]
2020-12-22T07:03:38.828Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
|2020-12-22T07:04:05.837Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":54}},"total":{"ticks":130,"time":{"ms":135},"value":130},"user":{"ticks":80,"time":{"ms":81}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"748ad402-c051-40e4-8ed7-55cacb9df5ee","uptime":{"ms":30026}},"memstats":

Pawel_Skadorwa · December 23, 2020, 11:14am

Is it possible that index hasn't been created because app logs hadn't been found? I suppose the filebeat looks for logs in a correct format, doesn't it?

warkolm · December 29, 2020, 8:31pm

Filebeat doesn't look for specific formats unless you use a module.

I'll move this to the Beats category, maybe someone there will have some ideas.

shaunak · December 30, 2020, 12:44pm

Yes, looking at the Filebeat logs you posted this is exactly what's happening. Filebeat lazily connects to Elasticsearch only when it has a batch of events (could be as small as a batch of 1) to send. For some reason it looks like Filebeat thinks there are no log events to send yet.

The very last line in your log (the one with [monitoring] in it) got cut off. Could you post the entire line please? It might give some insights into the state of Filebeat.

Thanks!

Shaunak

Pawel_Skadorwa · January 4, 2021, 10:47am

Thanks for answer. Yes, of course. The line looks like this:

|2021-01-04T10:43:31.018Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":202290,"time":{"ms":13}},"total":{"ticks":425280,"time":{"ms":33},"value":425280},"user":{"ticks":222990,"time":{"ms":20}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d6789240-13c1-42a9-b6b1-388f9fad7c22","uptime":{"ms":530970027}},"memstats":{"gc_next":17646928,"memory_alloc":8877328,"memory_total":15970950432},"runtime":{"goroutines":1273}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":19,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.09,"15":0.11,"5":0.07,"norm":{"1":0.015,"15":0.0183,"5":0.0117}}}}}}|
|---|---|---|---|---|---|
|2021-01-04T10:44:01.018Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":202300,"time":{"ms":12}},"total":{"ticks":425300,"time":{"ms":20},"value":425300},"user":{"ticks":223000,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d6789240-13c1-42a9-b6b1-388f9fad7c22","uptime":{"ms":531000027}},"memstats":{"gc_next":17646928,"memory_alloc":9784248,"memory_total":15971857352},"runtime":{"goroutines":1273}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":19,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.33,"15":0.13,"5":0.13,"norm":{"1":0.055,"15":0.0217,"5":0.0217}}}}}}|
|2021-01-04T10:44:31.018Z|INFO|[monitoring]|log/log.go:145|Non-zero metrics in the last 30s|{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":202310,"time":{"ms":10}},"total":{"ticks":425320,"time":{"ms":16},"value":425320},"user":{"ticks":223010,"time":{"ms":6}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d6789240-13c1-42a9-b6b1-388f9fad7c22","uptime":{"ms":531030027}},"memstats":{"gc_next":17646928,"memory_alloc":10687544,"memory_total":15972760648},"runtime":{"goroutines":1273}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":19,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.41,"15":0.15,"5":0.17,"norm":{"1":0.0683,"15":0.025,"5":0.0283}}}}}}|

And there are also such lines in the logs:

2020-12-30T10:10:31.020Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":37220,"time":{"ms":8}},"total":{"ticks":75500,"time":{"ms":16},"value":75500},"user":{"ticks":38280,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"d6789240-13c1-42a9-b6b1-388f9fad7c22","uptime":{"ms":96990030}},"memstats":{"gc_next":16428208,"memory_alloc":11683136,"memory_total":2911507344},"runtime":{"goroutines":921}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":19,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.14,"5":0.06,"norm":{"1":0,"15":0.0233,"5":0.01}}}}}}

2020-12-30T10:10:45.449Z INFO log/input.go:152 Configured paths: [/var/log/containers/*211cb2222eefb50a395b14ee40472a4a91d8a8db86c814591c47fc70ea79673f.log]

2020-12-30T10:10:45.454Z INFO log/input.go:152 Configured paths: [/var/log/containers/*211cb2222eefb50a395b14ee40472a4a91d8a8db86c814591c47fc70ea79673f.log]

2020-12-30T10:10:45.454Z INFO log/input.go:152 Configured paths: [/var/log/containers/*498775e739ff4268482343ce3ac15c0d7a0d067c6e6cdd51807ca784c5f85705.log]

2020-12-30T10:10:45.454Z INFO log/input.go:152 Configured paths: [/var/log/containers/*498775e739ff4268482343ce3ac15c0d7a0d067c6e6cdd51807ca784c5f85705.log]

2020-12-30T10:10:45.455Z INFO log/input.go:152 Configured paths: [/var/log/containers/*376e70899e04e140f3b0181ccfac0a6161968bb29fc4fc7ce54df8d3766d9606.log]

2020-12-30T10:10:45.455Z INFO log/input.go:152 Configured paths: [/var/log/containers/*376e70899e04e140f3b0181ccfac0a6161968bb29fc4fc7ce54df8d3766d9606.log]

2020-12-30T10:10:45.455Z INFO log/input.go:152 Configured paths: [/var/log/containers/*8476bae6ced4af1254d3a4b9587642355b4ae6bb330e9f195359375d852fa735.log]

2020-12-30T10:10:45.456Z INFO log/input.go:152 Configured paths: [/var/log/containers/*8476bae6ced4af1254d3a4b9587642355b4ae6bb330e9f195359375d852fa735.log]

Pawel_Skadorwa · January 19, 2021, 10:21am

Any other ideas?

system · February 16, 2021, 10:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeats doesn't send logs to Elasticsearch Beats filebeat	1	320	February 29, 2024
Filebeat not sending data to elasticsearch Beats filebeat	3	662	August 8, 2023
Filebeat and Logstash no send Logs Beats filebeat	7	1228	November 22, 2021
Filebeat is not sending Logs to Logstash Beats filebeat	1	324	February 3, 2021
Configure filebeat to ship elasticsearch logs to monitoring cluster in kubernetes Beats elastic-stack-monitoring , filebeat	1	296	March 12, 2021

Filebeat doesn't send logs to Elasticsearch

Related topics