Filebeat not sending data to elasticsearch

Elastic Stack Beats
1

Hi

've installed elasticsearch 8.5 and Kibana 8.5 in my kubernetes cluster simply applying the official helm file in the elastic repo. Now I'm trying to install filebeat with the following conf:

filebeat.inputs:
    - type: container
      paths:
      - "/var/log/app.log"
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            in_cluster: true

and

output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol: https
      ssl.certificate_authorities: ["/usr/share/filebeat/certs/ca.crt"]

Our apps are writing logs in pod container under /var/log/app.log but it seems filebeat does not read the log or not send it to elasticsearch because no index are created in elastic.

How can I solve the problem? What am I doing wrong?

Thanks

2

Is your filebeat running? What do you have in your filebeat logs?

If your filebeat.yml looks like this, then it looks wrong, the indentation for the processors key is wrong, processors should be on the same column as filebeat.inputs.

3

Filebeat is running and there are 12 pods running.

Sorry, but I made a mistake copying and pasting my file.
This is the right one:

filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
      - "/var/log/*.log"
      json.keys_under_root: true
      json.overwrite_keys: true
      json.add_error_key: true
      json.expand_keys: true
        
    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      - add_kubernetes_metadata:
            host: ${NODE_NAME}
            in_cluster: true

In filebeat logs I have a lot of errors saying:

{"log.level":"error","@timestamp":"2023-07-11T21:40:18.493Z","log.logger":"reader_docker_json","log.origin":{"file.name":"readjson/docker_json.go","file.line":231},"message":"Parse line error: parsing CRI timestamp: parsing time \"2023/06/16\" as \"2006-01-02T15:04:05.999999999Z07:00\": cannot parse \"/06/16\" as \"-\"","service.name":"filebeat","ecs.version":"1.6.0"}

No other errors are found (for example reaching elasticsearch master or something else)

Thanks

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.