Parsing docker container logs

Elastic Stack Logstash
1

Hi @magnusbaeck ,

I'm using filebeat to ship my logs to Logstash and I need help with parsing docker container logs along with few other logs.

My Filebeat configuration:

       filebeat.prospectors:
        - input_type: log
          paths:
            - /var/log/mesos/*.log
            - /var/log/dcos/dcos.log
            - /var/lib/docker/containers/*/*.log
        tail_files: true
        #output.elasticsearch:
        #  hosts: ["http://coordinator.elastic.l4lb.thisdcos.directory:9200"]
        output.logstash:
          hosts: logstash.marathon.mesos:5044
          timeout: 90
          bulk_max_size: 1024

My Logstash config:

input { 
 beats { port=> 5044 }
}

filter {
 grok {
  match => { 'message' => [ '%{SYSLOGLINE}', '%{CISCO_REASON}:%{ISO8601_SECOND}' ] }
 }
}

output {
 if '_grokparsefailure' in [tags] {
  elasticsearch {
   hosts => 'http://coordinator.elastic.l4lb.thisdcos.directory:9200'
   manage_template => false
   index => 'containers-%{+YYYY.MM.dd}'
  }
}
 else {
  elasticsearch {
   hosts => 'http://coordinator.elastic.l4lb.thisdcos.directory:9200'
   manage_template => false
   index => 'dcos-%{+YYYY.MM.dd}'
  }
 } 
}

By using the above configuration, I was expecting to parse the dcos and mesos logs with dcos-* index and container logs with containers-* index as it fails the grok pattern match. I have used the Online Grok debugger to find the patterns, but no luck in finding a unique pattern that parses only the container logs.

How do i define a pattern that is unique to container log such as below.

{"log": - - [08/Mar/2018:21:39:27 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.29.0\" \"-\"\n","stream":"stdout","time":"2018-03-08T21:39:27.310958603Z"}

Is there a way to parse the logs based on the source path ?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.