Is there something about my configuration isn't right? What did I miss?

Elastic Stack Beats
1

Hello there I have some issue to parse kubernetes containers multi lines utilizing filebeat and logstash. kubernetes log document are situated in/var/log/containers/*.log and in a json line structure.

Is there something about my configuration isn't right? What did I miss?

filebeat.yml:

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    -
      paths:
        - /var/log/containers/*.log
      fields: {log_type: containers}
      ignore_older: 5m
      symlinks: true
      json.message_key: log
      json.keys_under_root: true
      json.add_error_key: true
      multiline.pattern: '^\d{4}-\d{2}-\d{2}'
      multiline.match: after
      multiline.negate: true
      document_type: kube-logs
  registry_file: /var/log/containers/filebeat_registry
output:
  logstash:
    hosts: ["logstash-logging:5044"]

logstash.conf:

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "kube-logs" {

    date {
      match => ["time", "ISO8601"]
      remove_field => ["time"]
    }
    json {
      source => "message"
    }

    grok {
      match => [ "log", "<SOME_PATTERN>" ]
      overwrite => [ "message" ]
    }

}

kubernetes container:

{"log":"11:11:17,740 |-INFO in ch.qos.logback.core.joran.action.mapWAR - Attaching appender named [FILE-LOG] to Logger[ROOT]\n","stream":"stdout","time":"2017-05-09T11:11:17.742837362Z"}
{"log":"11:11:17,740 |-INFO in ch.qos.logback.classic.joran.action.mapWAR - End of configuration.\n","stream":"stdout","time":"2017-05-09T11:11:17.742840277Z"}
{"log":"11:11:17,741 |-INFO in ch.qos.logback.classic.joran.mapWAR - Registering current configuration as safe fallback point\n","stream":"stdout","time":"2017-05-09T11:11:17.742843277Z"}
{"log":"\n another line","stream":"stdout","time":"2017-05-09T11:11:17.742846485Z"}
{"log":"09-May-2017 11:11:17.756 INFO [localhost-startStop-1] org.apache.catalina.startup.mapWAR nice","stream":"stderr","time":"2017-05-09T11:11:17.756924376Z"}
{"log":"09-May-2017 11:11:17.757 INFO [localhost-startStop-1] org.apache.catalina.startup.mapWAR great","stream":"stderr","time":"2017-05-09T11:11:17.757465828Z"}

Thanks,
Sailaja.

2

Which filebeat version are you using?

Does filebeat even start with this configuration? I can't find the input type. The document_type configuration has been removed a long time ago.

Better use the docker input. The docker input removes the json envelope before forwarding logs to multiline:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.