# Configuring Logstash to filter docker logs

**URL:** <https://discuss.elastic.co/t/configuring-logstash-to-filter-docker-logs/124633>\
**Category:** Logstash\
**Created:** [March 19, 2018, 7:57pm UTC](https://discuss.elastic.co/t/configuring-logstash-to-filter-docker-logs/124633 "2018-03-19T19:57:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\
**Post date:** [March 19, 2018, 7:57pm UTC](https://discuss.elastic.co/t/configuring-logstash-to-filter-docker-logs/124633/1 "2018-03-19T19:57:10Z")

</div>

Hi,

I have an ELK stack for logging a DCOS cluster. I am trying to parse the container logs from one of the nodes and I would like to ship only the required container logs form the node. Here, I am trying to figure out a way to identify a container by its container name or image name.

So far my Logstash configuration is:

```
input { 
 beats { port=> 5044 } 
} 

filter { 
 if '/dcos' in [source] { 
  grok { 
  match => { 'message' => ['%{SYSLOGTIMESTAMP:timestamp} %{GREEDYDATA:message}', '%{GREEDYDATA:message}'] }
  add_tag => 'dcos_log' } } } 

filter { if '/mesos' in [source] { 
 grok { 
   match => { 'message' => ['%{GREEDYDATA:message}'] }
   add_tag => 'mesos_log' } } } 

filter { if [docker][name] == 'nginx' {
 grok { 
   match => { 'message' => ['%{YEAR:year}/%{MONTHNUM:month}/%{MONTHDAY:day} %{TIME:time}', '%{MONTHDAY:date}/%{MONTH:month}/%{YEAR:year}:%{TIME:time}', '%{LOGLEVEL:level}: %{GREEDYDATA:text}', '%{IP:hostip}', '%{GREEDYDATA:message}'] }
   add_field => { 'timestamp' => '%{year}-%{month}-%{date}T%{time}Z' }
   add_tag => 'container_log' }
 mutate { remove_field => ['year', 'month', 'date', 'time'] } } } 

output {
 if 'dcos_log' in [tags] {
   elasticsearch { hosts => 'ElasticURL' 
   manage_template => false index => 'dcos-%{+YYYY.MM.dd}'} } 
 else if 'mesos_log' in [tags] {
  elasticsearch { hosts => 'ElasticURL' 
  manage_template => false index => 'mesos-%{+YYYY.MM.dd}' } }
 else { 
 elasticsearch { hosts => 'ElasticURL' 
 manage_template => false index => 'containers-%{+YYYY.MM.dd}' } } }

```

Filebeat Configuration:

```
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/mesos/*.log
    - /var/log/dcos/dcos.log
    - /var/lib/docker/containers/*/*.log
   processors:
     - add_docker_metadata: ~

exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
tail_files: true
#output.elasticsearch:
# hosts: ["ElasticURL"]
output.logstash:
  hosts: logstashURL:5044
  timeout: 90
  bulk_max_size: 1024

```

I know there is a way by using the filebeat configuration

```
filebeat.prospectors:
- type: docker
  containers.ids:
  - '*'
  paths:
  - /var/lib/docker/containers/*.log
  processors:
  - add_docker_metadata: ~

```

But ideally if I'm handling multiple containers say a 50, I believe this wouldn't look neat.  
Can anyone help?

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 20, 2018, 7:15am UTC](https://discuss.elastic.co/t/configuring-logstash-to-filter-docker-logs/124633/2 "2018-03-20T07:15:21Z")

</div>

> [@aysala](#):
>
> I am trying to parse the container logs from one of the nodes

What do the logs look like?

> [@aysala](#):
>
> I would like to ship only the required container logs

What is distinct about the logs you'd like to include in an output? A value of a tag? Of a field?

---

<div class="post-metadata">

**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\
**Post date:** [March 20, 2018, 1:48pm UTC](https://discuss.elastic.co/t/configuring-logstash-to-filter-docker-logs/124633/3 "2018-03-20T13:48:23Z")

</div>

> [@yaauie](#):
>
> What do the logs look like?

Logs look something like:

`{"log":"{:timestamp=\u003e\"2018-03-19T15:57:11.955000+0000\", :message=\u003e\"Adding pattern\", \"ELB_ACCESS_LOG\"=\u003e\"%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{INT:response:int} %{INT:backend_response:int} %{INT:received_bytes:int} %{INT:bytes:int} \\\"%{ELB_REQUEST_LINE}\\\"\", :level=\u003e:info}\n","stream":"stdout","time":"2018-03-19T15:57:11.958211264Z"}`

> [@yaauie](#):
>
> What is distinct about the logs you'd like to include in an output? A value of a tag? Of a field?

Tags at the output yes. I can use the 'container\_log' tag to distinct the container logs into a 'containers-\*' index.

---

<div class="post-metadata">

**Author:** ![aysala](https://avatars.discourse-cdn.com/v4/letter/a/47e85d/32.png) [@aysala](https://discuss.elastic.co/u/aysala)\
**Post date:** [March 26, 2018, 6:49pm UTC](https://discuss.elastic.co/t/configuring-logstash-to-filter-docker-logs/124633/4 "2018-03-26T18:49:57Z")

</div>

I have resolved this issue by upgrading Filebeat to the latest version (6.2.3). This version allowed me to ship the container logs along with its metadata like container name, image name, Mesos\_Task\_ID etc, by which I could parse only the required container logs.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 23, 2018, 6:50pm UTC](https://discuss.elastic.co/t/configuring-logstash-to-filter-docker-logs/124633/5 "2018-04-23T18:50:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.