Configuring Logstash to filter docker logs

Elastic Stack Logstash
1

Hi,

I have an ELK stack for logging a DCOS cluster. I am trying to parse the container logs from one of the nodes and I would like to ship only the required container logs form the node. Here, I am trying to figure out a way to identify a container by its container name or image name.

So far my Logstash configuration is:

input { 
 beats { port=> 5044 } 
} 

filter { 
 if '/dcos' in [source] { 
  grok { 
  match => { 'message' => [ '%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:message}', '%{GREEDYDATA:message}' ] }
  add_tag => 'dcos_log' } } } 

filter { if '/mesos' in [source] { 
 grok { 
   match => { 'message' => [ '%{GREEDYDATA:message}'] }
   add_tag => 'mesos_log' } } } 

filter { if [docker][name] == 'nginx' {
 grok { 
   match => { 'message' => [ '%{YEAR:year}/%{MONTHNUM:month}/%{MONTHDAY:day} %{TIME:time}', '%{MONTHDAY:date}/%{MONTH:month}/%{YEAR:year}:%{TIME:time}', '%{LOGLEVEL:level}: %{GREEDYDATA:text}', '%{IP:hostip}', '%{GREEDYDATA:message}' ] }
   add_field => { 'timestamp' => '%{year}-%{month}-%{date}T%{time}Z' }
   add_tag => 'container_log' }
 mutate { remove_field => [ 'year', 'month', 'date', 'time' ] } } } 

output {
 if 'dcos_log' in [tags] {
   elasticsearch { hosts => 'ElasticURL' 
   manage_template => false index => 'dcos-%{+YYYY.MM.dd}'} } 
 else if 'mesos_log' in [tags] {
  elasticsearch { hosts => 'ElasticURL' 
  manage_template => false index => 'mesos-%{+YYYY.MM.dd}' } }
 else { 
 elasticsearch { hosts => 'ElasticURL' 
 manage_template => false index => 'containers-%{+YYYY.MM.dd}' } } }

Filebeat Configuration:

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/mesos/*.log
    - /var/log/dcos/dcos.log
    - /var/lib/docker/containers/*/*.log
   processors:
     - add_docker_metadata: ~

exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
tail_files: true
#output.elasticsearch:
#  hosts: ["ElasticURL"]
output.logstash:
  hosts: logstashURL:5044
  timeout: 90
  bulk_max_size: 1024

I know there is a way by using the filebeat configuration

filebeat.prospectors:
- type: docker
  containers.ids:
  - '*'
  paths:
  - /var/lib/docker/containers/*.log
  processors:
  - add_docker_metadata: ~

But ideally if I'm handling multiple containers say a 50, I believe this wouldn't look neat.
Can anyone help?

2

What do the logs look like?

What is distinct about the logs you'd like to include in an output? A value of a tag? Of a field?

3

Logs look something like:

{"log":"{:timestamp=\u003e\"2018-03-19T15:57:11.955000+0000\", :message=\u003e\"Adding pattern\", \"ELB_ACCESS_LOG\"=\u003e\"%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} \\\"%{ELB_REQUEST_LINE}\\\"\", :level=\u003e:info}\n","stream":"stdout","time":"2018-03-19T15:57:11.958211264Z"}

Tags at the output yes. I can use the 'container_log' tag to distinct the container logs into a 'containers-*' index.

4

I have resolved this issue by upgrading Filebeat to the latest version (6.2.3). This version allowed me to ship the container logs along with its metadata like container name, image name, Mesos_Task_ID etc, by which I could parse only the required container logs.

Thanks!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.