Configuring Logstash to filter docker logs

[aysala]

Hi,

I have an ELK stack for logging a DCOS cluster. I am trying to parse the container logs from one of the nodes and I would like to ship only the required container logs form the node. Here, I am trying to figure out a way to identify a container by its container name or image name.

So far my Logstash configuration is:

input { 
 beats { port=> 5044 } 
} 

filter { 
 if '/dcos' in [source] { 
  grok { 
  match => { 'message' => [ '%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:message}', '%{GREEDYDATA:message}' ] }
  add_tag => 'dcos_log' } } } 

filter { if '/mesos' in [source] { 
 grok { 
   match => { 'message' => [ '%{GREEDYDATA:message}'] }
   add_tag => 'mesos_log' } } } 

filter { if [docker][name] == 'nginx' {
 grok { 
   match => { 'message' => [ '%{YEAR:year}/%{MONTHNUM:month}/%{MONTHDAY:day} %{TIME:time}', '%{MONTHDAY:date}/%{MONTH:month}/%{YEAR:year}:%{TIME:time}', '%{LOGLEVEL:level}: %{GREEDYDATA:text}', '%{IP:hostip}', '%{GREEDYDATA:message}' ] }
   add_field => { 'timestamp' => '%{year}-%{month}-%{date}T%{time}Z' }
   add_tag => 'container_log' }
 mutate { remove_field => [ 'year', 'month', 'date', 'time' ] } } } 

output {
 if 'dcos_log' in [tags] {
   elasticsearch { hosts => 'ElasticURL' 
   manage_template => false index => 'dcos-%{+YYYY.MM.dd}'} } 
 else if 'mesos_log' in [tags] {
  elasticsearch { hosts => 'ElasticURL' 
  manage_template => false index => 'mesos-%{+YYYY.MM.dd}' } }
 else { 
 elasticsearch { hosts => 'ElasticURL' 
 manage_template => false index => 'containers-%{+YYYY.MM.dd}' } } }

Filebeat Configuration:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/mesos/*.log
    - /var/log/dcos/dcos.log
    - /var/lib/docker/containers/*/*.log
   processors:
     - add_docker_metadata: ~

exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
tail_files: true
#output.elasticsearch:
#  hosts: ["ElasticURL"]
output.logstash:
  hosts: logstashURL:5044
  timeout: 90
  bulk_max_size: 1024

I know there is a way by using the filebeat configuration

filebeat.prospectors:
- type: docker
  containers.ids:
  - '*'
  paths:
  - /var/lib/docker/containers/*.log
  processors:
  - add_docker_metadata: ~

But ideally if I'm handling multiple containers say a 50, I believe this wouldn't look neat.
Can anyone help?

[yaauie (Ry Biesemeyer)]

@aysala

I am trying to parse the container logs from one of the nodes

What do the logs look like?

@aysala

I would like to ship only the required container logs

What is distinct about the logs you'd like to include in an output? A value of a tag? Of a field?

[aysala]

@yaauie

What do the logs look like?

Logs look something like:

{"log":"{:timestamp=\u003e\"2018-03-19T15:57:11.955000+0000\", :message=\u003e\"Adding pattern\", \"ELB_ACCESS_LOG\"=\u003e\"%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} \\\"%{ELB_REQUEST_LINE}\\\"\", :level=\u003e:info}\n","stream":"stdout","time":"2018-03-19T15:57:11.958211264Z"}

@yaauie

What is distinct about the logs you'd like to include in an output? A value of a tag? Of a field?

Tags at the output yes. I can use the 'container_log' tag to distinct the container logs into a 'containers-*' index.

[aysala]

I have resolved this issue by upgrading Filebeat to the latest version (6.2.3). This version allowed me to ship the container logs along with its metadata like container name, image name, Mesos_Task_ID etc, by which I could parse only the required container logs.

Thanks!